From nobody Sun Apr 12 04:21:37 2026 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=quarantine dis=none) header.from=scaleway.com ARC-Seal: i=1; a=rsa-sha256; t=1770913742; cv=none; d=zohomail.com; s=zohoarc; b=Ybh2Gf14cQ2jPWfT8dzNmeAyleR0h4y/bDB1o2rXY52JJ06y7nesdCWjGGBFuHi8L7yfgzsQREcK/+8D7o0PB5P2FpXBf6ckHG/VnkFhIcbLLMm9Q+t1BoDFMmEbwq8tAKv+IhiPd/7mwoZdrkbN9YdOZPiqR+Al7+SLy4MZyd4= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1770913742; h=Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=l7WdD2zHkBagt+yCXYN+euTN8kOLoPQP3BR3Isyndvk=; b=K7dhjzPz3OAFqLXMi3p07FqKDSwjKl2/45f+dArp4cgIk68FoE2+REXnwFWR/rU0wqqMH8dPyYFz+rLHYLCLxD4ZPIEctIyU6X9beuS7iCrxDb05uiZ/6I2AsR5+P3a1k0LojZemlnkIGxBSA9nAygdLXwiIa3tHoaPOscERRhk= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=quarantine dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1770913741709179.59999988927711; Thu, 12 Feb 2026 08:29:01 -0800 (PST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1vqZYT-0001Z4-5K; Thu, 12 Feb 2026 11:28:37 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1vqZY9-0001Pb-3t for qemu-devel@nongnu.org; Thu, 12 Feb 2026 11:28:17 -0500 Received: from mail-wm1-x32e.google.com ([2a00:1450:4864:20::32e]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1vqZY5-0007jN-Qa for qemu-devel@nongnu.org; Thu, 12 Feb 2026 11:28:16 -0500 Received: by mail-wm1-x32e.google.com with SMTP id 5b1f17b1804b1-481188b7760so299295e9.0 for ; Thu, 12 Feb 2026 08:28:10 -0800 (PST) Received: from localhost (710304585.box.freepro.com. [130.180.219.188]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-4836131d484sm33198425e9.27.2026.02.12.08.28.07 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 12 Feb 2026 08:28:07 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=scaleway.com; s=google; t=1770913688; x=1771518488; darn=nongnu.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=l7WdD2zHkBagt+yCXYN+euTN8kOLoPQP3BR3Isyndvk=; b=R8zw03sH7C+rHDXJo22KP+vXFQOZls9Ntkd8iL4V/lSYoJXUJBJdD46snXO+m/mm4b lvlpeSEgWQR6PidhVzvtJz49yZHNsh5iSChQOiBxGrXKtC+SJgzx/8xLl9oUVB1eidRC +8ciufWIS1QoLBD09XGTc2FOPRvS7ul76sb46fDZNxMwfZalpI7K2HPu5myAd6VpsEqM clSdnUPf0ZHNFAn8Merocy+o/MXLo9G67DWsVPy7kUju1G2CItpDsblmgnoMafcZR8ts 8VmJdIt4mARN8h5/3WyZmi022WxnuC1k95txYSVpoeNwjA217dCluwtdkIGuqEfbZMLS cbUQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1770913688; x=1771518488; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=l7WdD2zHkBagt+yCXYN+euTN8kOLoPQP3BR3Isyndvk=; b=fJ99Lqxud/NTq6I0YGOe1eLCtfaCr38Qh3eVnSyhNKgUzdk1MiMrHsatOktoWL3lkD XWx/3Q1N6yKftCv+AGzEqQQJfbx/WJ5PsmLRMjM43w5bW+xkz4rnz/lR7mMEqeT2NAul NeEdBwvNno2RDyPJQbLrd9v8mrWI5fiODDzBzmDT45szOSjsAgFoH1NX7xOElVrp0Zwg EcEkegGeRcCGtmnf3+CObo02dt1XeozNeNWut5a2nmIuBNKKaCH+dQt0E8/eBASJfBng FbmNFi2AJqTQEQEitxVjQ77Q0z+8Hm5jqKWLVOcph0jspGvTrLU5wW3y9flSMMVv7ZYY CODQ== X-Gm-Message-State: AOJu0YwwuG9+v7ksTmFyN9p+BNKfqacPBtixTst/qP282wNYGSyd8vMm p8FU+lLroLFGvrm69vYg22Qtz+DI+45+LQZKKUWDkeVLH7/u2493UMA+Ihfq734iQwm0rffKv3+ ylfeT X-Gm-Gg: AZuq6aKW2k/pu8xTqZ5TpuD6h0z3Rkr4a6ZACfjhnpVAPJ5qeS53RFMi2U0VyTk4QuZ KKn0sRUrOggwWIWNfrzfF2sO3KeKi6IbTq1IEMefe5hUZ55hXYFIAea1cwTcIiB/BoMqO9oNkXr StgyHoOYB0uc2mY1xCFGf9VG+cHOHn0XoD3HuhXqNmsXG7MY36shCmMsZM8zlPY0XGkkceH9Q0B qM8qdEzz6FBvIdbiYb70LdeEB5lG1uAM80iRJR1L0hRsIA2XaPjkuV3vTvcJXfCF0y2Ci4PbqVD NsX1PDB40ZhyurWNVbeZL8UKkDne5+1Yu0kigMnrD0R5HLwfghfJQkA+4NaWjVAW5aQ6uyP8zZA uawVBCI0S0WrtbN/Y1hK2FqZdepaNdoqwj2MBlOHpRv+y63Rdk9Z+X2BRrf5r7zM6mItiQNwOVg y3Xp4GrDNnudwK62Rg/yYduZmAldgKa/AqKJ1iDMByTx/NnTG3 X-Received: by 2002:a05:600c:354c:b0:480:3230:6c89 with SMTP id 5b1f17b1804b1-483656c8246mr51006065e9.12.1770913687845; Thu, 12 Feb 2026 08:28:07 -0800 (PST) From: Antoine Damhet To: qemu-devel@nongnu.org Cc: Antoine Damhet , qemu-block@nongnu.org, Kevin Wolf , Hanna Reitz , qemu-stable@nongnu.org Subject: [PATCH 1/2] block/curl: fix concurrent completion handling Date: Thu, 12 Feb 2026 17:27:24 +0100 Message-ID: <20260212162730.440855-2-adamhet@scaleway.com> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260212162730.440855-1-adamhet@scaleway.com> References: <20260212162730.440855-1-adamhet@scaleway.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=2a00:1450:4864:20::32e; envelope-from=adamhet@scaleway.com; helo=mail-wm1-x32e.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=unavailable autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: qemu development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @scaleway.com) X-ZM-MESSAGEID: 1770913746443158500 Content-Type: text/plain; charset="utf-8" curl_multi_check_completion would bail upon the first completed transfer even if more completion messages were available thus leaving some in flight IOs stuck. Rework a bit the loop to make the iterations clearer and drop the breaks. The original hang can be somewhat reproduced with the following command: $ qemu-img convert -p -m 16 -O qcow2 -c --image-opts \ 'file.driver=3Dhttps,file.url=3Dhttps://scaleway.testdebit.info/10G.iso,f= ile.readahead=3D1M' \ /tmp/test.qcow2 Fixes: 1f2cead32443 ("curl: Ensure all informationals are checked for compl= etion") Cc: qemu-stable@nongnu.org Signed-off-by: Antoine Damhet Reviewed-by: Kevin Wolf --- block/curl.c | 11 ++--------- 1 file changed, 2 insertions(+), 9 deletions(-) diff --git a/block/curl.c b/block/curl.c index 4e77c93b46e6..6dccf002564e 100644 --- a/block/curl.c +++ b/block/curl.c @@ -324,17 +324,11 @@ curl_find_buf(BDRVCURLState *s, uint64_t start, uint6= 4_t len, CURLAIOCB *acb) static void curl_multi_check_completion(BDRVCURLState *s) { int msgs_in_queue; + CURLMsg *msg; =20 /* Try to find done transfers, so we can free the easy * handle again. */ - for (;;) { - CURLMsg *msg; - msg =3D curl_multi_info_read(s->multi, &msgs_in_queue); - - /* Quit when there are no more completions */ - if (!msg) - break; - + while ((msg =3D curl_multi_info_read(s->multi, &msgs_in_queue))) { if (msg->msg =3D=3D CURLMSG_DONE) { int i; CURLState *state =3D NULL; @@ -397,7 +391,6 @@ static void curl_multi_check_completion(BDRVCURLState *= s) } =20 curl_clean_state(state); - break; } } } --=20 2.53.0 From nobody Sun Apr 12 04:21:37 2026 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=quarantine dis=none) header.from=scaleway.com ARC-Seal: i=1; a=rsa-sha256; t=1770913742; cv=none; d=zohomail.com; s=zohoarc; b=ijVE+nPnLORwAejmqonEbJnl1SiG35ByoPo0PHbSRb0AEw5woxM8C+7OIuU5l26tvyIFbtuJoAOQX7wgjoMmrr70K0WPYABT/gp1glBHuTNJyl+1/yGObtrMZdYuCwdhinJM3q8Da6ItGBXAArjFyNfXQgzDZRCg9N22IBe8Sig= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1770913742; h=Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=R8Mg3xLx5AR9QnULNPfNm28Oi0tteTYNIGto3Ebs6+s=; b=CrUnZr7amV8UwQovyR7RrN7hom1EEeuUmK5zrRvze+9D0joA7C3hfx/9zU1z8gisfYixr+POSQU5lAIKA7gSE5sqn2cfVO3i5d2CDDl/FHxeIeBF4NrIKyRree7bjHc+CznZmH2S5O6GA1d5Inzf/Mr1A+j6GWYJT51odP7JfMk= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=quarantine dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1770913742735177.10947682843016; Thu, 12 Feb 2026 08:29:02 -0800 (PST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1vqZYZ-0001eO-M1; Thu, 12 Feb 2026 11:28:43 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1vqZY9-0001Q3-AU for qemu-devel@nongnu.org; Thu, 12 Feb 2026 11:28:17 -0500 Received: from mail-wm1-x329.google.com ([2a00:1450:4864:20::329]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1vqZY3-0007jc-Nf for qemu-devel@nongnu.org; Thu, 12 Feb 2026 11:28:17 -0500 Received: by mail-wm1-x329.google.com with SMTP id 5b1f17b1804b1-4836f363ad2so266635e9.1 for ; Thu, 12 Feb 2026 08:28:10 -0800 (PST) Received: from localhost (710304585.box.freepro.com. [130.180.219.188]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-4835b958b6csm140710445e9.1.2026.02.12.08.28.08 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 12 Feb 2026 08:28:08 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=scaleway.com; s=google; t=1770913689; x=1771518489; darn=nongnu.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=R8Mg3xLx5AR9QnULNPfNm28Oi0tteTYNIGto3Ebs6+s=; b=F1zvCAAwSSoenJNkzD6tOQzdoK0wSTyEZLJQkqxMolH5ZC6cVvFN53NWVvt79gMaqk B/kXfN1/3O/j1WxBK8oey3t2gK3dDjN9wOdgNn0p5tU9PK7NWBcLcriDw9vBAWtfDLAR ClO+tR0HFKzwWGUnuFY+gX09KyoB42Pc5IYfXZRwbh+OsAraPALr5WwRZ1wD+19Pxb22 BueDy5Ke5755Wotwv14n3xTITP5vTauO5wuzdcRnZdhu4zptRKzObwDt/tbZhLP1mxxK NXb1NVlz5kI2J/Q65a27tn7eZkN/yjPHFhjZhCy9pkkYoKgW7NcjW4XH6N4vz9xtM7RE SKEA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1770913689; x=1771518489; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=R8Mg3xLx5AR9QnULNPfNm28Oi0tteTYNIGto3Ebs6+s=; b=STHd+cmjQUbo2zYZlKRyOcku+MUoBpqLoCpSojPOCB0934QrYhEpNaFznLd1wqIfk7 h5xJ4aFsmrqYMvgeCwWOOfz4J6JvDJZJv1sGtNA4apd+kS07GeuWhRMmMczscSSx+1Q3 gaJhDRaQTtEIPCR1K6NCf6vtbolQolnDTy9OkJCGsHBIDYuMtFjrDlsyt7bN7EujvTX8 htvhEvemr7pI/9adw8HHihDISuNR0W/RFjf4uNe2/JJVLbn35Jzu1tlZZv21AEXJdV4X 0f4lKMbYEfwY99k9DXxgLVIhroL17yzcehHtWvZmNkTnYRNBL9Y3OOEJOoBoPnll+eYj CeWQ== X-Gm-Message-State: AOJu0Yz75YA09QKTjS0dZbQqoA9RQ8uuP7zNRbtUpYor30u/KNe/NNTp FX1kvEwimwxWil6YFFmNeAC86/3Fvc+uakpJwC9U+HRl9HFXvcC3SN+Fo/uqw6IHweO0RxRPtTG caqd+ X-Gm-Gg: AZuq6aI010WwowMP8sk7gnXZk8tqtm0Wp/qaX5OPvyIwxFPDNjTlcki7+U4opP4Ie4V 25e9wTHcvsJQ9WHz+pi8Rjuo1kIZDn2mBvauUjZcTulbJ7CbXEYPxjcICswLVzAlLCD7zDuQlmS bqDtxj+d9hDCu2wlDa2QyYJiMYl53Ll4X0QEgLfDiFf4dLYcpwvIhHFFkB/PHlAjilT/t08WzUk cwr45/oHPLHLijDnBemL+/BqysXfTGWi5HXQgtmwTHxIPDBvnMYtMilyWajOAV6Ckd11Dy1tlb7 JrwkabiVj1R/GH8xSUnoYLDO/SBGuIvJT6fC2XDES2YlhcdKcBHrv+AG7kFuTML2o45cE/iT57F ti09eNp7wLxZ7yapbymA3WCcoIGfeoY6t2gcZA3AC+t1IgXHz41oBWbtAwNumebyNa6YOv4PsDa WJWx1DzqFQnqCgMzZ4Fujx7NfSXSWy3oPRZ8qcrjYzdOG/6o6X X-Received: by 2002:a05:600c:3b85:b0:483:6ff1:18b with SMTP id 5b1f17b1804b1-4836ff103a7mr4975605e9.0.1770913689197; Thu, 12 Feb 2026 08:28:09 -0800 (PST) From: Antoine Damhet To: qemu-devel@nongnu.org Cc: Antoine Damhet , qemu-block@nongnu.org, Kevin Wolf , Hanna Reitz , Pierrick Bouvier , Eric Blake , Markus Armbruster Subject: [PATCH 2/2] block/curl: add support for S3 presigned URLs Date: Thu, 12 Feb 2026 17:27:25 +0100 Message-ID: <20260212162730.440855-3-adamhet@scaleway.com> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260212162730.440855-1-adamhet@scaleway.com> References: <20260212162730.440855-1-adamhet@scaleway.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=2a00:1450:4864:20::329; envelope-from=adamhet@scaleway.com; helo=mail-wm1-x329.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: qemu development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @scaleway.com) X-ZM-MESSAGEID: 1770913745932154100 Content-Type: text/plain; charset="utf-8" S3 presigned URLs are signed for a specific HTTP method (typically GET for our use cases). The curl block driver currently issues a HEAD request to discover the backend features and the file size, which fails with 403. Add a 'force-range' option that skips the HEAD request and instead issues a minimal GET request (querying 1 byte from the server) to extract the file size from the 'Content-Range' response header. To achieve this the 'curl_header_cb' is redesigned to generically parse HTTP headers. $ $QEMU -drive driver=3Dhttp,\ 'url=3Dhttps://s3.example.com/some.img?X-Amz-Security-Token=3D= XXX', force-range=3Dtrue Enabling the 'force-range' option without the backend supporting it is undefined behavior and untested but the libcurl should ignore the body and stop reading after the HTTP headers then we would fail with the expected `Server does not support 'range' (byte ranges).` error. Signed-off-by: Antoine Damhet --- block/curl.c | 104 ++++++++++++++++++-------- block/trace-events | 1 + docs/system/device-url-syntax.rst.inc | 6 ++ qapi/block-core.json | 14 +++- 4 files changed, 90 insertions(+), 35 deletions(-) diff --git a/block/curl.c b/block/curl.c index 6dccf002564e..66aecfb20ec6 100644 --- a/block/curl.c +++ b/block/curl.c @@ -62,10 +62,12 @@ #define CURL_BLOCK_OPT_PASSWORD_SECRET "password-secret" #define CURL_BLOCK_OPT_PROXY_USERNAME "proxy-username" #define CURL_BLOCK_OPT_PROXY_PASSWORD_SECRET "proxy-password-secret" +#define CURL_BLOCK_OPT_FORCE_RANGE "force-range" =20 #define CURL_BLOCK_OPT_READAHEAD_DEFAULT (256 * 1024) #define CURL_BLOCK_OPT_SSLVERIFY_DEFAULT true #define CURL_BLOCK_OPT_TIMEOUT_DEFAULT 5 +#define CURL_BLOCK_OPT_FORCE_RANGE_DEFAULT false =20 struct BDRVCURLState; struct CURLState; @@ -206,27 +208,33 @@ static size_t curl_header_cb(void *ptr, size_t size, = size_t nmemb, void *opaque) { BDRVCURLState *s =3D opaque; size_t realsize =3D size * nmemb; - const char *p =3D ptr; - const char *end =3D p + realsize; - const char *t =3D "accept-ranges : bytes "; /* A lowercase template */ + g_autofree char *header =3D g_strstrip(g_strndup(ptr, realsize)); + char *val =3D strchr(header, ':'); =20 - /* check if header matches the "t" template */ - for (;;) { - if (*t =3D=3D ' ') { /* space in t matches any amount of isspace i= n p */ - if (p < end && g_ascii_isspace(*p)) { - ++p; - } else { - ++t; - } - } else if (*t && p < end && *t =3D=3D g_ascii_tolower(*p)) { - ++p, ++t; - } else { - break; - } + if (!val) { + return realsize; } =20 - if (!*t && p =3D=3D end) { /* if we managed to reach ends of both stri= ngs */ - s->accept_range =3D true; + *val++ =3D '\0'; + g_strchomp(header); + while (g_ascii_isspace(*val)) { + ++val; + } + + trace_curl_header_cb(header, val); + + if (!g_ascii_strcasecmp(header, "accept-ranges")) { + if (!g_ascii_strcasecmp(val, "bytes")) { + s->accept_range =3D true; + } + } else if (!g_ascii_strcasecmp(header, "Content-Range")) { + /* Content-Range fmt is `bytes begin-end/full_size` */ + val =3D strchr(val, '/'); + if (val) { + if (qemu_strtou64(val + 1, NULL, 10, &s->len) < 0) { + s->len =3D UINT64_MAX; + } + } } =20 return realsize; @@ -668,6 +676,11 @@ static QemuOptsList runtime_opts =3D { .type =3D QEMU_OPT_STRING, .help =3D "ID of secret used as password for HTTP proxy auth", }, + { + .name =3D CURL_BLOCK_OPT_FORCE_RANGE, + .type =3D QEMU_OPT_BOOL, + .help =3D "Assume HTTP range requests are supported", + }, { /* end of list */ } }, }; @@ -690,6 +703,7 @@ static int curl_open(BlockDriverState *bs, QDict *optio= ns, int flags, #endif const char *secretid; const char *protocol_delimiter; + bool force_range; int ret; =20 bdrv_graph_rdlock_main_loop(); @@ -807,35 +821,56 @@ static int curl_open(BlockDriverState *bs, QDict *opt= ions, int flags, } =20 s->accept_range =3D false; + s->len =3D UINT64_MAX; + force_range =3D qemu_opt_get_bool(opts, CURL_BLOCK_OPT_FORCE_RANGE, + CURL_BLOCK_OPT_FORCE_RANGE_DEFAULT); + /* + * When minimal CURL will be bumped to `7.83`, the header callback + m= anual + * parsing can be replaced by `curl_easy_header` calls + */ if (curl_easy_setopt(state->curl, CURLOPT_NOBODY, 1L) || curl_easy_setopt(state->curl, CURLOPT_HEADERFUNCTION, curl_header_= cb) || curl_easy_setopt(state->curl, CURLOPT_HEADERDATA, s)) { - pstrcpy(state->errmsg, CURL_ERROR_SIZE, - "curl library initialization failed."); - goto out; + goto out_init; + } + if (force_range) { + if (curl_easy_setopt(state->curl, CURLOPT_CUSTOMREQUEST, "GET") || + curl_easy_setopt(state->curl, CURLOPT_RANGE, "0-0")) { + goto out_init; + } } + if (curl_easy_perform(state->curl)) goto out; - /* CURL 7.55.0 deprecates CURLINFO_CONTENT_LENGTH_DOWNLOAD in favour of - * the *_T version which returns a more sensible type for content leng= th. - */ + + if (!force_range) { + /* + * CURL 7.55.0 deprecates CURLINFO_CONTENT_LENGTH_DOWNLOAD in favo= ur of + * the *_T version which returns a more sensible type for content + * length. + */ #if LIBCURL_VERSION_NUM >=3D 0x073700 - if (curl_easy_getinfo(state->curl, CURLINFO_CONTENT_LENGTH_DOWNLOAD_T,= &cl)) { - goto out; - } + if (curl_easy_getinfo(state->curl, CURLINFO_CONTENT_LENGTH_DOWNLOA= D_T, + &cl)) { + goto out; + } #else - if (curl_easy_getinfo(state->curl, CURLINFO_CONTENT_LENGTH_DOWNLOAD, &= cl)) { - goto out; - } + if (curl_easy_getinfo(state->curl, CURLINFO_CONTENT_LENGTH_DOWNLOA= D, + &cl)) { + goto out; + } #endif - if (cl < 0) { + if (cl >=3D 0) { + s->len =3D cl; + } + } + + if (s->len =3D=3D UINT64_MAX) { pstrcpy(state->errmsg, CURL_ERROR_SIZE, "Server didn't report file size."); goto out; } =20 - s->len =3D cl; - if ((!strncasecmp(s->url, "http://", strlen("http://")) || !strncasecmp(s->url, "https://", strlen("https://"))) && !s->accept_range) { @@ -856,6 +891,9 @@ static int curl_open(BlockDriverState *bs, QDict *optio= ns, int flags, qemu_opts_del(opts); return 0; =20 +out_init: + pstrcpy(state->errmsg, CURL_ERROR_SIZE, + "curl library initialization failed."); out: error_setg(errp, "CURL: Error opening file: %s", state->errmsg); curl_easy_cleanup(state->curl); diff --git a/block/trace-events b/block/trace-events index c9b4736ff884..d170fc96f15f 100644 --- a/block/trace-events +++ b/block/trace-events @@ -191,6 +191,7 @@ ssh_server_status(int status) "server status=3D%d" curl_timer_cb(long timeout_ms) "timer callback timeout_ms %ld" curl_sock_cb(int action, int fd) "sock action %d on fd %d" curl_read_cb(size_t realsize) "just reading %zu bytes" +curl_header_cb(const char *key, const char *val) "looking at %s: %s" curl_open(const char *file) "opening %s" curl_open_size(uint64_t size) "size =3D %" PRIu64 curl_setup_preadv(uint64_t bytes, uint64_t start, const char *range) "read= ing %" PRIu64 " at %" PRIu64 " (%s)" diff --git a/docs/system/device-url-syntax.rst.inc b/docs/system/device-url= -syntax.rst.inc index aae65d138c00..e77032e9e4b6 100644 --- a/docs/system/device-url-syntax.rst.inc +++ b/docs/system/device-url-syntax.rst.inc @@ -179,6 +179,12 @@ These are specified using a special URL syntax. get the size of the image to be downloaded. If not set, the default timeout of 5 seconds is used. =20 + ``force-range`` + Assume the HTTP backend supports range requests and avoid doing + a HTTP HEAD request to discover the feature. Typically S3 + presigned URLs will only support one method and refuse other + requests types. + Note that when passing options to qemu explicitly, ``driver`` is the value of . =20 diff --git a/qapi/block-core.json b/qapi/block-core.json index b82af7425614..ff018c2d6bfb 100644 --- a/qapi/block-core.json +++ b/qapi/block-core.json @@ -4582,12 +4582,17 @@ # @cookie-secret: ID of a QCryptoSecret object providing the cookie # data in a secure way. See @cookie for the format. (since 2.10) # +# @force-range: Don't issue a HEAD HTTP request to discover if the +# backend supports range requests and rely only on GET requests. +# This is especially useful for S3 presigned URLs. (since 11.0) +# # Since: 2.9 ## { 'struct': 'BlockdevOptionsCurlHttp', 'base': 'BlockdevOptionsCurlBase', 'data': { '*cookie': 'str', - '*cookie-secret': 'str'} } + '*cookie-secret': 'str', + '*force-range': 'bool'} } =20 ## # @BlockdevOptionsCurlHttps: @@ -4605,13 +4610,18 @@ # @cookie-secret: ID of a QCryptoSecret object providing the cookie # data in a secure way. See @cookie for the format. (since 2.10) # +# @force-range: Don't issue a HEAD HTTP request to discover if the +# backend supports range requests and rely only on GET requests. +# This is especially useful for S3 presigned URLs. (since 11.0) +# # Since: 2.9 ## { 'struct': 'BlockdevOptionsCurlHttps', 'base': 'BlockdevOptionsCurlBase', 'data': { '*cookie': 'str', '*sslverify': 'bool', - '*cookie-secret': 'str'} } + '*cookie-secret': 'str', + '*force-range': 'bool'} } =20 ## # @BlockdevOptionsCurlFtp: --=20 2.53.0