From nobody Sun Apr 12 05:51:24 2026 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=quarantine dis=none) header.from=scaleway.com ARC-Seal: i=1; a=rsa-sha256; t=1771948430; cv=none; d=zohomail.com; s=zohoarc; b=AMYVeT3rq4I5kUuDLKpPuY5JxYoLa3CZbc7+K7hNSm10a10eI63v7hqzxIGAzPQ2ui1Y791UbEUtE2UkuGLuO3qBBb2xsMeZIYM3W5CZMQZp1tJZIdZW/yrlhS6AEMcAgJNOeFHcqYC98PPo+uZyMCsKWH41hofLReBSRPNEUy4= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1771948430; h=Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=HBzph0yd8ODlMrDm3zgIFroU1WmXKib1kj+ismUWb8c=; b=ZQhS9yQ5jB7UGaJUznA76qpGmK1edvb9ScYbAK+Lu+eYtRkSpvd63wMnqYmxrclfnITYnVK12yokpLbadJjVl/6KzmSCWQLRY1R4N5IYpJLsQrTL/ipLzSUn8Iek33bbhR2/9fakTN9ZPTBLL/WzEu0NL0yqKwOi0yAtMH/K7yE= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=quarantine dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1771948430646848.4773620701746; Tue, 24 Feb 2026 07:53:50 -0800 (PST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1vuuj7-0003jv-GA; Tue, 24 Feb 2026 10:53:33 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1vuuj5-0003i9-7v for qemu-devel@nongnu.org; Tue, 24 Feb 2026 10:53:31 -0500 Received: from mail-wm1-x32a.google.com ([2a00:1450:4864:20::32a]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1vuuj1-00084q-I0 for qemu-devel@nongnu.org; Tue, 24 Feb 2026 10:53:30 -0500 Received: by mail-wm1-x32a.google.com with SMTP id 5b1f17b1804b1-4806ce0f97bso46070765e9.0 for ; Tue, 24 Feb 2026 07:53:27 -0800 (PST) Received: from localhost ([2a01:e0a:e9e:c3d0:7a5b:f76a:6cf6:aac2]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-483bd765604sm5827415e9.15.2026.02.24.07.53.25 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 24 Feb 2026 07:53:25 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=scaleway.com; s=google; t=1771948406; x=1772553206; darn=nongnu.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=HBzph0yd8ODlMrDm3zgIFroU1WmXKib1kj+ismUWb8c=; b=I7rX6EREfB9R2/9ez2shqC0DXHpKj7qInHcXGgFpNpUG2O0QZEMBqsUY8FZ48IN9c3 3rJIItiGQX47y5+546wP3bXuCxewKxewLEggURMKY8GOkUzHhSM/PF8mLQFK+4SNofhd kpZ/sOayDCJDFX54G63X7IUyTMSgQLFp5TYo8Xy/m398AvDVJK4ImnPk5jIGBml/1JsR EAAKEER9OdQrKo2DalWVskTMDiOVKK4aElElOHdjAgVbk6Az0BvPpeZre/vz2EHhfY0p GDkr99M1vYjJvvszHMDwtk38T5bM+/nB+XkKchRPnhISJKDSgJV7m9Hn8A+bBpopiWrA J3CQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1771948406; x=1772553206; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=HBzph0yd8ODlMrDm3zgIFroU1WmXKib1kj+ismUWb8c=; b=i48K4oJjpZW2T4gWMa/rwjwaAC2Tx3JX1LPsfSuCkNpDxa55z+njCuKaZyfKpurddB VNBdkNVbDzb3yy4Ithb2V0pndb57IOT2T4EhDdhXYuAQKxMMN/yP/Rkh44SgR2M/Vv09 ChgBPPgTD99cOhZu1e724dYu1+09pKfn+0rusxfvlZHIQKrDMS+tYhfxkejyoHYR+CvZ HYuBJ0ICMleBGPs2GnpWegn14bCH/IL4pCbG0X5ymQZDBXNbcCFy85q0fEnxO5yaJ5OX 5XCeqFU8BtvmKFdVYzUR41uVlNBr1iQ7y/Yw2uwFARpSpjLmVmfkXG6MlkpHtQpn82md X6dQ== X-Gm-Message-State: AOJu0YzbeUXgyyhv3Ybi0fIKMJl0VP11EUivn+gaIarIIrWRzPUqWage Fho2RiCqzo3vi82baPieeESCNhdY6BFimujm/uOcReYZUFuv2tMLx2PkxgbvqpmKsaXUvjJebhe 58YJA X-Gm-Gg: AZuq6aKBOvs1u3873tM7hhkE6XtbR1kFXwxSNi1gkkeyY+91PR0TFCwNl90ZMDr/oeo vKJiAaQBodcl5uboH/yDHL/cKEzCK0GOwGXmy31Ih1F1h8rSZWTH/MGxzg46BQj5P5aF5LlBC2p b/VV7s8cq4d6juRuQLbr2Wx+oVeZwCxRgZRRW8lkMLPA5GCtE1evQWtjSCrmKFse97vMJXG2l0F 9m8hAccs98rBJ8yMQ0KuSFzj8jYIiP4xu0cNTxnXLPZ8Z8Ye75zAzxUiBYBHSgX+ohgIoVkuAl1 xgRoWd4EB5P6J/yXyOKCm6BEuLViICUs8SI4gP/5xaq1LFtA5ORUc/01XcP+V2D0TDtun8oYVIb /h+YBalGmtRBvwdR27ZJ9Jh7phjJAOC1MJc8uLIfAHzj1AehH1MfuQJ7vM+sZAs7zIIwzcy2o6S WEa+9RKN4kU0ys/M28U7KJypE= X-Received: by 2002:a05:600c:8b71:b0:483:7980:4687 with SMTP id 5b1f17b1804b1-483a95dd932mr192661685e9.17.1771948405899; Tue, 24 Feb 2026 07:53:25 -0800 (PST) From: Antoine Damhet To: qemu-devel@nongnu.org Cc: Antoine Damhet , qemu-block@nongnu.org, Kevin Wolf , Hanna Reitz , qemu-stable@nongnu.org Subject: [PATCH v2 1/3] block/curl: fix concurrent completion handling Date: Tue, 24 Feb 2026 16:51:29 +0100 Message-ID: <20260224155314.1658988-2-adamhet@scaleway.com> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260224155314.1658988-1-adamhet@scaleway.com> References: <20260224155314.1658988-1-adamhet@scaleway.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=2a00:1450:4864:20::32a; envelope-from=adamhet@scaleway.com; helo=mail-wm1-x32a.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=unavailable autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: qemu development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @scaleway.com) X-ZM-MESSAGEID: 1771948433712158500 Content-Type: text/plain; charset="utf-8" curl_multi_check_completion would bail upon the first completed transfer even if more completion messages were available thus leaving some in flight IOs stuck. Rework a bit the loop to make the iterations clearer and drop the breaks. The original hang can be somewhat reproduced with the following command: $ qemu-img convert -p -m 16 -O qcow2 -c --image-opts \ 'file.driver=3Dhttps,file.url=3Dhttps://scaleway.testdebit.info/10G.iso,f= ile.readahead=3D1M' \ /tmp/test.qcow2 Fixes: 1f2cead32443 ("curl: Ensure all informationals are checked for compl= etion") Cc: qemu-stable@nongnu.org Signed-off-by: Antoine Damhet Reviewed-by: Kevin Wolf --- block/curl.c | 11 ++--------- 1 file changed, 2 insertions(+), 9 deletions(-) diff --git a/block/curl.c b/block/curl.c index 4e77c93b46e6..6dccf002564e 100644 --- a/block/curl.c +++ b/block/curl.c @@ -324,17 +324,11 @@ curl_find_buf(BDRVCURLState *s, uint64_t start, uint6= 4_t len, CURLAIOCB *acb) static void curl_multi_check_completion(BDRVCURLState *s) { int msgs_in_queue; + CURLMsg *msg; =20 /* Try to find done transfers, so we can free the easy * handle again. */ - for (;;) { - CURLMsg *msg; - msg =3D curl_multi_info_read(s->multi, &msgs_in_queue); - - /* Quit when there are no more completions */ - if (!msg) - break; - + while ((msg =3D curl_multi_info_read(s->multi, &msgs_in_queue))) { if (msg->msg =3D=3D CURLMSG_DONE) { int i; CURLState *state =3D NULL; @@ -397,7 +391,6 @@ static void curl_multi_check_completion(BDRVCURLState *= s) } =20 curl_clean_state(state); - break; } } } --=20 2.53.0 From nobody Sun Apr 12 05:51:24 2026 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=quarantine dis=none) header.from=scaleway.com ARC-Seal: i=1; a=rsa-sha256; t=1771948457; cv=none; d=zohomail.com; s=zohoarc; b=KCvcEkn2HGVWyzJ1Oc8sXrJE7F63COxV9zO7hKd/nO9mKN0Qd3mLssC/uwd19ov6By9WRSF0w8mjWomyBVuLwRgOnffcTdpPOTBcpDj5T7U70Pfhj0XXxCAluxkCd8/VGbGp7rTk3ceBvMRXKnttlMtFhKtE82573FGap0HM7WI= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1771948457; h=Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=J99vWbO+OMptFve7tCC13Qkd2jDgRuTvSKUDgtl9bDs=; b=J1U91fBBkDar+VBh7LHbTaww1PIlDy8snL9tAno81nP8P2i33hWkcUOjm56aeDDBbJLrN9KBt+O1vLRM+wFGYY2DNBa72mJjGNKjEX4kG3R7gZWim+zEu+UDg/8cZRZjc95MMh7tlRVe8OQ7dZHGcO4vt+yiRIqDZXutDFYnONg= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=quarantine dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1771948457299247.09541278275492; Tue, 24 Feb 2026 07:54:17 -0800 (PST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1vuuj6-0003jQ-Pv; Tue, 24 Feb 2026 10:53:32 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1vuuj5-0003i6-4q for qemu-devel@nongnu.org; Tue, 24 Feb 2026 10:53:31 -0500 Received: from mail-wr1-x42f.google.com ([2a00:1450:4864:20::42f]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1vuuj2-00086Q-HK for qemu-devel@nongnu.org; Tue, 24 Feb 2026 10:53:30 -0500 Received: by mail-wr1-x42f.google.com with SMTP id ffacd0b85a97d-4359a16a400so5028909f8f.1 for ; Tue, 24 Feb 2026 07:53:28 -0800 (PST) Received: from localhost ([2a01:e0a:e9e:c3d0:7a5b:f76a:6cf6:aac2]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-43970d4c96csm29621730f8f.30.2026.02.24.07.53.26 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 24 Feb 2026 07:53:26 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=scaleway.com; s=google; t=1771948407; x=1772553207; darn=nongnu.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=J99vWbO+OMptFve7tCC13Qkd2jDgRuTvSKUDgtl9bDs=; b=hO+otsLQSi8sSBBSN+ZCa04LIjkcNM9fZCASKUfBScZbQS2srC0c3dbvGfwWyx4Q0E NkfgJmIjqzisSJbJp3NJ1DtUsUpzldsgKQY1IUfIEU/h1e+yL4D1QHpv+hG+/EnxDfHK UxvNE7lfVodcjtjlI63My5vEtbfEGq4s/Xi2jRp+f5WBTpwUhpD2J8Ywf6wov8OVMo7l +EfdtQ+DpVV2NRrXccYw6Dd8jdmgN8/4coeAKAPe/y2E3EXR9J5qVpqdGFWoafmCCdW8 fy1n9NSD5RIjswLT0SNKER7x8OQZSWx5odpMMkWs44tk99YUSfIsW2oLOLQsGsvM2qKj kRsg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1771948407; x=1772553207; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=J99vWbO+OMptFve7tCC13Qkd2jDgRuTvSKUDgtl9bDs=; b=fyBKF27MUYf9AS1s+ezoyxT75r4zWgiVUOG5TjWY7UT8JeMJuVzFOCqv3SVb7z/Ns+ 8XmvpTEITlUaQe/Zzu4ssTOmgaySWwkQqeVi4AkRCw8/Uz5MlqGno4wGTYjEsMa1kpNR 5hlCXXggrSc8Ta2ACs0OS6SG5AE9aiBKKxQ4SHMJ4JZQyKsbPDTpnQk+hIIyzA+d5UWx ltp7MiNt8asm+p+RLwsq94iQexI4XUOVzbTQ+uGH6EUe3DE+kmu32DUmShCbUdcJ7bWp dWrvftrdIL7dKZalmEeREN6BxtCjfBooKarQf8sHNLkfWv+KCUZac3v4cqeemT+v8wkq waaA== X-Gm-Message-State: AOJu0YxtXtJwK4qi9EVtq0g13RlHV7XCoSz3fdIxBIgTuqs/eZFPJTHK bL2g1KrfhpbthTIFVNjYxKI3Fw1bYeNr+GDpICiOf2ViPD1rMehRh7C9qkfrIRHClJ/B5EWMN50 gZCUh X-Gm-Gg: ATEYQzwTd2/raSPAiR0EDQtHZOAkoet4lHMGZ/RTtadwjbfgCnWhwpvdQxntv1cnUKu ND3Vi0exnjFBI7ZVUiZIhLkTm0b1j3dusG1q5mYqdtOd1+5+Q6Vsbw/APDe80COiB6dRiNBzPuy BI6fGoXXDjZe/jjyHpHgZwgqDhWM7030TdFy9EIsWZrstyDNtajdoIKyRvSbrPqoCpzQ9iB6PO3 Sk93/2N6Xiw/5UzI0B129adVAeuRPYhFfxLT3c38oiz+B0VMJxSH+YnRE6MdVnmVZYoSzwCPXZu g2YG9RbCZxFuAN7LJUUvdMEGWgtVVdHQiiUmkSQ5kVDvkgXdY/Gj5JKVMWJ93cTIS6yAGLTJ/Cj ZVK1whIz6p09mALux6W2alUxK+AM7EMr7OewRqrgiqs9FrZyfhSXIvRjEqvNK8rW34/C1CauSey jhVfHn9XtK+lTXcUZRVVoQ3I8= X-Received: by 2002:a05:6000:3107:b0:436:34d9:4627 with SMTP id ffacd0b85a97d-4396f17baf1mr22792013f8f.37.1771948406945; Tue, 24 Feb 2026 07:53:26 -0800 (PST) From: Antoine Damhet To: qemu-devel@nongnu.org Cc: Antoine Damhet , qemu-block@nongnu.org, Kevin Wolf , Hanna Reitz , Eric Blake , Markus Armbruster Subject: [PATCH v2 2/3] qapi: block: Refactor HTTP(s) common arguments Date: Tue, 24 Feb 2026 16:51:30 +0100 Message-ID: <20260224155314.1658988-3-adamhet@scaleway.com> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260224155314.1658988-1-adamhet@scaleway.com> References: <20260224155314.1658988-1-adamhet@scaleway.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=2a00:1450:4864:20::42f; envelope-from=adamhet@scaleway.com; helo=mail-wr1-x42f.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: qemu development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @scaleway.com) X-ZM-MESSAGEID: 1771948460563158501 Content-Type: text/plain; charset="utf-8" The HTTPs curl block driver is a superset of the HTTP driver, reflect that in the QAPI. Suggested-by: Markus Armbruster Signed-off-by: Antoine Damhet Reviewed-by: Markus Armbruster --- qapi/block-core.json | 13 ++----------- 1 file changed, 2 insertions(+), 11 deletions(-) diff --git a/qapi/block-core.json b/qapi/block-core.json index b82af7425614..a7871705fa69 100644 --- a/qapi/block-core.json +++ b/qapi/block-core.json @@ -4595,23 +4595,14 @@ # Driver specific block device options for HTTPS connections over the # curl backend. URLs must start with "https://". # -# @cookie: List of cookies to set; format is "name1=3Dcontent1; -# name2=3Dcontent2;" as explained by CURLOPT_COOKIE(3). Defaults to -# no cookies. -# # @sslverify: Whether to verify the SSL certificate's validity # (defaults to true) # -# @cookie-secret: ID of a QCryptoSecret object providing the cookie -# data in a secure way. See @cookie for the format. (since 2.10) -# # Since: 2.9 ## { 'struct': 'BlockdevOptionsCurlHttps', - 'base': 'BlockdevOptionsCurlBase', - 'data': { '*cookie': 'str', - '*sslverify': 'bool', - '*cookie-secret': 'str'} } + 'base': 'BlockdevOptionsCurlHttp', + 'data': { '*sslverify': 'bool'} } =20 ## # @BlockdevOptionsCurlFtp: --=20 2.53.0 From nobody Sun Apr 12 05:51:24 2026 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=quarantine dis=none) header.from=scaleway.com ARC-Seal: i=1; a=rsa-sha256; t=1771948432; cv=none; d=zohomail.com; s=zohoarc; b=bWf5eKjG3MZo3qapHDpayILU0482dEekDn6QYCBFXMttdxsJhCXvXKBoHJpcbAn8cCwW/8qyE2iISq1RNxa9r+/K8D0/G5H/FmMBkfL0WPRYJHUz1wPN2Ue9uzMmPbrAGTqIf6HrRzDnCMks29TMdM8frTTwidGcf3GTl8pRmng= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1771948432; h=Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=kj8T6mb1572v2v1TIxJJ9XBRNaQfvElom9Lfk0/2aVA=; b=DwaCLS2iqi1T2fYL7fSHKXm6FlNhwes7/K6EIbjexuzJ/5mzdKnwj4Y2xJLtP6Hu3uKe3Bidn4pNJsq6xHotQIzcBZiISKxc3JK1fG1lVgB8h7PY/wy5smYHBEtLtHITPoz9e7DnuEp6nXMmWR8QUqkG6DM2kF8cpW4PcHrD9Ws= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=quarantine dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1771948432407883.5671829810259; Tue, 24 Feb 2026 07:53:52 -0800 (PST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1vuuj8-0003kN-6j; Tue, 24 Feb 2026 10:53:34 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1vuuj6-0003jc-UI for qemu-devel@nongnu.org; Tue, 24 Feb 2026 10:53:32 -0500 Received: from mail-wr1-x42a.google.com ([2a00:1450:4864:20::42a]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1vuuj3-00087C-Kq for qemu-devel@nongnu.org; Tue, 24 Feb 2026 10:53:32 -0500 Received: by mail-wr1-x42a.google.com with SMTP id ffacd0b85a97d-4398913af88so575412f8f.2 for ; Tue, 24 Feb 2026 07:53:29 -0800 (PST) Received: from localhost ([2a01:e0a:e9e:c3d0:7a5b:f76a:6cf6:aac2]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-43970d54c5csm27753096f8f.38.2026.02.24.07.53.27 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 24 Feb 2026 07:53:27 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=scaleway.com; s=google; t=1771948408; x=1772553208; darn=nongnu.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=kj8T6mb1572v2v1TIxJJ9XBRNaQfvElom9Lfk0/2aVA=; b=YZbfYuf5HanryEjdzGXFOD1QMv4OPdphQeAJRlbwRzOonEgJQ1iodFz6awcUC/xknZ 8OVmHHeuEUp0/yNA8sNkuvY5twfQxSm264oIpFT1Ky8L6V4tHp6q+qkumrjYJkw4729I A5oyCL0B1+Ouqk6X+zMnNlxzHoMFzxZTxk2e4l/gobAlwNiuNnw8TSlaCP8bSeBTLnUu pSucQ4KLzLrteS/E/h1+UK4dh6QCroA14c2TviBQLwUlYAeCp1Qgmb/Qg1wZ3p+91IJe ZbRM4I2Yu2iW7dj2XfF2m68cRIsFwYoD5K3Ph271YM1+bxuVxaEbkFXDy6P1BxYScddj Qliw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1771948408; x=1772553208; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=kj8T6mb1572v2v1TIxJJ9XBRNaQfvElom9Lfk0/2aVA=; b=Vl+LD9bGrpoQC4bnH12jEJgko9MOi5GdLJivMzvBM1LWFa7IDHmv8+iDPlAMTW6mYQ gjGhCLD8cOzusd3jKxmhWczD3XaoRYC11PaDeQsk0pRGLuYOJG3A7MpNGMs9Q7oOD1y6 CxHvE4ispoObYBPPrRmCeUVCS1Qb4O7DCsZLAX93C79EQPSpYh2saQhJhBfgtzzF0Eih cix/mSIh6NBIgebOZKgW1fLA4+05Vppr2fszBKyn3MHO0AEVnn6Gq/zZN1JircBDaFHv w0B0KtmG1514xhwFbQXd21r2hxD6MEqcG5esmj7Ys3MXqMooygXhG7zvu+GQJMCkSYmx xC9w== X-Gm-Message-State: AOJu0Yx41/LzcLJOyXzLlOwtbBf0w0Mne5afjWEQnRUM9ufPnPOlBe4C EaikHB4fRWOINWHLsWCMigEqwfWukySza541AWIrolS9AilpE1gBkPixQxhQlokDGolbaVAqHuH H+Ya9 X-Gm-Gg: ATEYQzz4zUiQNzbsHXcoPgkhONy0TEGaDFxhMANFg+RMIohYlwrsjuP/6Sa6uTr99Yr kt3vD2jG11nQ93IPA/+AfDrbYayj4kz7KsExbi5K4p/Or9FmD0DrImDjZyrMmoYmZxmAfbGUMaM 1Jp+2DOCoLfDA5mcbqrbU1g6kluDxFmUV7oYbR5dENJq/ctQbdYF+XQRh/J5JRoDZTv6lBPWnGw Kb4OSeO4KlSYPx/3ivfN2jGXZamm8drrFj4phU5q31MfjeYLVXOAdIxRso3uDGEW/T1zprLpHPW wLhUzK6f/Qnd1pLJqb4VGBgBAxOTDXsco8JDgCwf0ZNbfq++0/ARZul5TNocmc8tVs9Cjlcaikd pbNCxZoxHxlCdZ9yLrkVFVZNoEHSCS/kvUr1b9rV+N7L9GmVcS4Lk7nobwTZpG9SKdZqffEcb8y Fz+nqqkugP/Yg0t6e9FZa/2NY= X-Received: by 2002:adf:e346:0:b0:437:6efe:94c5 with SMTP id ffacd0b85a97d-4396f1564b2mr17444321f8f.2.1771948407951; Tue, 24 Feb 2026 07:53:27 -0800 (PST) From: Antoine Damhet To: qemu-devel@nongnu.org Cc: Antoine Damhet , qemu-block@nongnu.org, Kevin Wolf , Hanna Reitz , Pierrick Bouvier , Eric Blake , Markus Armbruster Subject: [PATCH v2 3/3] block/curl: add support for S3 presigned URLs Date: Tue, 24 Feb 2026 16:51:31 +0100 Message-ID: <20260224155314.1658988-4-adamhet@scaleway.com> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260224155314.1658988-1-adamhet@scaleway.com> References: <20260224155314.1658988-1-adamhet@scaleway.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=2a00:1450:4864:20::42a; envelope-from=adamhet@scaleway.com; helo=mail-wr1-x42a.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: qemu development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @scaleway.com) X-ZM-MESSAGEID: 1771948434550158500 Content-Type: text/plain; charset="utf-8" S3 presigned URLs are signed for a specific HTTP method (typically GET for our use cases). The curl block driver currently issues a HEAD request to discover the web server features and the file size, which fails with 'HTTP 403' (forbidden). Add a 'force-range' option that skips the HEAD request and instead issues a minimal GET request (querying 1 byte from the server) to extract the file size from the 'Content-Range' response header. To achieve this the 'curl_header_cb' is redesigned to generically parse HTTP headers. $ $QEMU -drive driver=3Dhttps,\ 'url=3Dhttps://s3.example.com/some.img?X-Amz-Security-Token=3D= XXX', force-range=3Dtrue Enabling the 'force-range' option without the web server specified with @url supporting it might cause the server to respond successfully with 'HTTP 200' and attempt to send the whole file body. With the 'CURLOPT_NOBODY' option set the libcurl will skip reading after the headers and close the connection. QEMU still gracefully detects the missing feature. This might waste a small number of TCP packets but is otherwise transparent to the user. Signed-off-by: Antoine Damhet Acked-by: Markus Armbruster --- block/curl.c | 104 ++++++++++++++++++-------- block/trace-events | 1 + docs/system/device-url-syntax.rst.inc | 6 ++ qapi/block-core.json | 9 ++- 4 files changed, 86 insertions(+), 34 deletions(-) diff --git a/block/curl.c b/block/curl.c index 6dccf002564e..66aecfb20ec6 100644 --- a/block/curl.c +++ b/block/curl.c @@ -62,10 +62,12 @@ #define CURL_BLOCK_OPT_PASSWORD_SECRET "password-secret" #define CURL_BLOCK_OPT_PROXY_USERNAME "proxy-username" #define CURL_BLOCK_OPT_PROXY_PASSWORD_SECRET "proxy-password-secret" +#define CURL_BLOCK_OPT_FORCE_RANGE "force-range" =20 #define CURL_BLOCK_OPT_READAHEAD_DEFAULT (256 * 1024) #define CURL_BLOCK_OPT_SSLVERIFY_DEFAULT true #define CURL_BLOCK_OPT_TIMEOUT_DEFAULT 5 +#define CURL_BLOCK_OPT_FORCE_RANGE_DEFAULT false =20 struct BDRVCURLState; struct CURLState; @@ -206,27 +208,33 @@ static size_t curl_header_cb(void *ptr, size_t size, = size_t nmemb, void *opaque) { BDRVCURLState *s =3D opaque; size_t realsize =3D size * nmemb; - const char *p =3D ptr; - const char *end =3D p + realsize; - const char *t =3D "accept-ranges : bytes "; /* A lowercase template */ + g_autofree char *header =3D g_strstrip(g_strndup(ptr, realsize)); + char *val =3D strchr(header, ':'); =20 - /* check if header matches the "t" template */ - for (;;) { - if (*t =3D=3D ' ') { /* space in t matches any amount of isspace i= n p */ - if (p < end && g_ascii_isspace(*p)) { - ++p; - } else { - ++t; - } - } else if (*t && p < end && *t =3D=3D g_ascii_tolower(*p)) { - ++p, ++t; - } else { - break; - } + if (!val) { + return realsize; } =20 - if (!*t && p =3D=3D end) { /* if we managed to reach ends of both stri= ngs */ - s->accept_range =3D true; + *val++ =3D '\0'; + g_strchomp(header); + while (g_ascii_isspace(*val)) { + ++val; + } + + trace_curl_header_cb(header, val); + + if (!g_ascii_strcasecmp(header, "accept-ranges")) { + if (!g_ascii_strcasecmp(val, "bytes")) { + s->accept_range =3D true; + } + } else if (!g_ascii_strcasecmp(header, "Content-Range")) { + /* Content-Range fmt is `bytes begin-end/full_size` */ + val =3D strchr(val, '/'); + if (val) { + if (qemu_strtou64(val + 1, NULL, 10, &s->len) < 0) { + s->len =3D UINT64_MAX; + } + } } =20 return realsize; @@ -668,6 +676,11 @@ static QemuOptsList runtime_opts =3D { .type =3D QEMU_OPT_STRING, .help =3D "ID of secret used as password for HTTP proxy auth", }, + { + .name =3D CURL_BLOCK_OPT_FORCE_RANGE, + .type =3D QEMU_OPT_BOOL, + .help =3D "Assume HTTP range requests are supported", + }, { /* end of list */ } }, }; @@ -690,6 +703,7 @@ static int curl_open(BlockDriverState *bs, QDict *optio= ns, int flags, #endif const char *secretid; const char *protocol_delimiter; + bool force_range; int ret; =20 bdrv_graph_rdlock_main_loop(); @@ -807,35 +821,56 @@ static int curl_open(BlockDriverState *bs, QDict *opt= ions, int flags, } =20 s->accept_range =3D false; + s->len =3D UINT64_MAX; + force_range =3D qemu_opt_get_bool(opts, CURL_BLOCK_OPT_FORCE_RANGE, + CURL_BLOCK_OPT_FORCE_RANGE_DEFAULT); + /* + * When minimal CURL will be bumped to `7.83`, the header callback + m= anual + * parsing can be replaced by `curl_easy_header` calls + */ if (curl_easy_setopt(state->curl, CURLOPT_NOBODY, 1L) || curl_easy_setopt(state->curl, CURLOPT_HEADERFUNCTION, curl_header_= cb) || curl_easy_setopt(state->curl, CURLOPT_HEADERDATA, s)) { - pstrcpy(state->errmsg, CURL_ERROR_SIZE, - "curl library initialization failed."); - goto out; + goto out_init; + } + if (force_range) { + if (curl_easy_setopt(state->curl, CURLOPT_CUSTOMREQUEST, "GET") || + curl_easy_setopt(state->curl, CURLOPT_RANGE, "0-0")) { + goto out_init; + } } + if (curl_easy_perform(state->curl)) goto out; - /* CURL 7.55.0 deprecates CURLINFO_CONTENT_LENGTH_DOWNLOAD in favour of - * the *_T version which returns a more sensible type for content leng= th. - */ + + if (!force_range) { + /* + * CURL 7.55.0 deprecates CURLINFO_CONTENT_LENGTH_DOWNLOAD in favo= ur of + * the *_T version which returns a more sensible type for content + * length. + */ #if LIBCURL_VERSION_NUM >=3D 0x073700 - if (curl_easy_getinfo(state->curl, CURLINFO_CONTENT_LENGTH_DOWNLOAD_T,= &cl)) { - goto out; - } + if (curl_easy_getinfo(state->curl, CURLINFO_CONTENT_LENGTH_DOWNLOA= D_T, + &cl)) { + goto out; + } #else - if (curl_easy_getinfo(state->curl, CURLINFO_CONTENT_LENGTH_DOWNLOAD, &= cl)) { - goto out; - } + if (curl_easy_getinfo(state->curl, CURLINFO_CONTENT_LENGTH_DOWNLOA= D, + &cl)) { + goto out; + } #endif - if (cl < 0) { + if (cl >=3D 0) { + s->len =3D cl; + } + } + + if (s->len =3D=3D UINT64_MAX) { pstrcpy(state->errmsg, CURL_ERROR_SIZE, "Server didn't report file size."); goto out; } =20 - s->len =3D cl; - if ((!strncasecmp(s->url, "http://", strlen("http://")) || !strncasecmp(s->url, "https://", strlen("https://"))) && !s->accept_range) { @@ -856,6 +891,9 @@ static int curl_open(BlockDriverState *bs, QDict *optio= ns, int flags, qemu_opts_del(opts); return 0; =20 +out_init: + pstrcpy(state->errmsg, CURL_ERROR_SIZE, + "curl library initialization failed."); out: error_setg(errp, "CURL: Error opening file: %s", state->errmsg); curl_easy_cleanup(state->curl); diff --git a/block/trace-events b/block/trace-events index c9b4736ff884..d170fc96f15f 100644 --- a/block/trace-events +++ b/block/trace-events @@ -191,6 +191,7 @@ ssh_server_status(int status) "server status=3D%d" curl_timer_cb(long timeout_ms) "timer callback timeout_ms %ld" curl_sock_cb(int action, int fd) "sock action %d on fd %d" curl_read_cb(size_t realsize) "just reading %zu bytes" +curl_header_cb(const char *key, const char *val) "looking at %s: %s" curl_open(const char *file) "opening %s" curl_open_size(uint64_t size) "size =3D %" PRIu64 curl_setup_preadv(uint64_t bytes, uint64_t start, const char *range) "read= ing %" PRIu64 " at %" PRIu64 " (%s)" diff --git a/docs/system/device-url-syntax.rst.inc b/docs/system/device-url= -syntax.rst.inc index aae65d138c00..445e2a0a4157 100644 --- a/docs/system/device-url-syntax.rst.inc +++ b/docs/system/device-url-syntax.rst.inc @@ -179,6 +179,12 @@ These are specified using a special URL syntax. get the size of the image to be downloaded. If not set, the default timeout of 5 seconds is used. =20 + ``force-range`` + Assume the HTTP backend supports range requests and avoid doing + an HTTP HEAD request to discover the feature. Typically S3 + presigned URLs will only support one method and refuse other + request types. + Note that when passing options to qemu explicitly, ``driver`` is the value of . =20 diff --git a/qapi/block-core.json b/qapi/block-core.json index a7871705fa69..50e7078cbec0 100644 --- a/qapi/block-core.json +++ b/qapi/block-core.json @@ -4582,12 +4582,19 @@ # @cookie-secret: ID of a QCryptoSecret object providing the cookie # data in a secure way. See @cookie for the format. (since 2.10) # +# @force-range: Don't issue a HEAD HTTP request to discover if the +# http server supports range requests and rely only on GET +# requests. This is especially useful for S3 presigned URLs where +# HEAD requests are unauthorized. Defaults to false. +# (default: false; since 11.0) +# # Since: 2.9 ## { 'struct': 'BlockdevOptionsCurlHttp', 'base': 'BlockdevOptionsCurlBase', 'data': { '*cookie': 'str', - '*cookie-secret': 'str'} } + '*cookie-secret': 'str', + '*force-range': 'bool'} } =20 ## # @BlockdevOptionsCurlHttps: --=20 2.53.0