From nobody Fri Apr 4 07:21:18 2025
Delivered-To: importer@patchew.org
Authentication-Results: mx.zohomail.com;
dkim=pass;
spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as
permitted sender)
smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org;
dmarc=pass(p=none dis=none) header.from=gmail.com
ARC-Seal: i=1; a=rsa-sha256; t=1741782418; cv=none;
d=zohomail.com; s=zohoarc;
b=HZ4xF3V/we1Kotvr7juqA8V4Hl5UzdI448yOAY1OsmRyZUmliyax3oViYd0Mkhe5Ny0wsV0VMVi7g7qtSLHnOYqMWEqUOqXGaMIP2Yv3OlTP6ZGqwdRdUORIivvuS9ZFJCoC15hjihcKcmaS3dZb3u7IP14ITs/zKiIwga2NCpg=
ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com;
s=zohoarc;
t=1741782418;
h=Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To;
bh=zIY17leVJs2z4tAUMiT8cAqFys40kyohmzEKVjGlpNM=;
b=Ti9dm0N0KBKyxS6OwOIdv8p5//YKsR/hA7dEP6F6zvLMp0UU8/eeNdcXrgjDnJSzEEWZaMnhf/Cjo2WZS3KALAusgqYFvU7q1vg2Bg09dC95q/NYu+TTksPtgvhQ0MizKVS0OJD0IWw5nUHMfVAgZpbQL2ge5Fw3vyQLjXTCcs4=
ARC-Authentication-Results: i=1; mx.zohomail.com;
dkim=pass;
spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as
permitted sender)
smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org;
dmarc=pass header.from=<npiggin@gmail.com> (p=none dis=none)
Return-Path: <qemu-devel-bounces+importer=patchew.org@nongnu.org>
Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by
mx.zohomail.com
with SMTPS id 1741782418777149.14516047238112;
Wed, 12 Mar 2025 05:26:58 -0700 (PDT)
Received: from localhost ([::1] helo=lists1p.gnu.org)
by lists.gnu.org with esmtp (Exim 4.90_1)
(envelope-from <qemu-devel-bounces@nongnu.org>)
id 1tsLAC-0001eA-U2; Wed, 12 Mar 2025 08:26:20 -0400
Received: from eggs.gnu.org ([2001:470:142:3::10])
by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
(Exim 4.90_1) (envelope-from <npiggin@gmail.com>) id 1tsLAA-0001dd-J3
for qemu-devel@nongnu.org; Wed, 12 Mar 2025 08:26:19 -0400
Received: from mail-pl1-x62c.google.com ([2607:f8b0:4864:20::62c])
by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128)
(Exim 4.90_1) (envelope-from <npiggin@gmail.com>) id 1tsLA8-0001dF-1j
for qemu-devel@nongnu.org; Wed, 12 Mar 2025 08:26:18 -0400
Received: by mail-pl1-x62c.google.com with SMTP id
d9443c01a7336-223f4c06e9fso13585655ad.1
for <qemu-devel@nongnu.org>; Wed, 12 Mar 2025 05:26:13 -0700 (PDT)
Received: from wheely.local0.net ([118.208.151.101])
by smtp.gmail.com with ESMTPSA id
d2e1a72fcca58-736ab33e1d3sm10842408b3a.132.2025.03.12.05.26.08
(version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256);
Wed, 12 Mar 2025 05:26:10 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
d=gmail.com; s=20230601; t=1741782371; x=1742387171; darn=nongnu.org;
h=content-transfer-encoding:mime-version:references:in-reply-to
:message-id:date:subject:cc:to:from:from:to:cc:subject:date
:message-id:reply-to;
bh=zIY17leVJs2z4tAUMiT8cAqFys40kyohmzEKVjGlpNM=;
b=mVegO8oO4JVjpY1xwcltqBCLz2SC6roN48CYvMq8d7oPZVLAiz47T21tmGa0a2tF+I
bUzB8ME3u/sztx5mR4Foi5imM37xqhrLhemlNvvdbt8hbbdmi9tzXuskPKlLcwzN0p21
qj3dJZ+/XbLgZbx9WwRSCBG5Ql4Jq98rI0I2s5FhXO/UUoj25iaIOBYRGhDHhtAhmxQX
cD/dhzx1zQXSMIFh+I3siG0VdBmFyUMzXPC1XQyA/Kn+oLFsl04czfPbwhzoT0q/lU1F
0nDcroq37eWCMP66pRVA5JpOhCDK4pcwtnQGVS+4q+ZQBSywbmbhgEUqnhQe7IBWhf39
h1XA==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
d=1e100.net; s=20230601; t=1741782371; x=1742387171;
h=content-transfer-encoding:mime-version:references:in-reply-to
:message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc
:subject:date:message-id:reply-to;
bh=zIY17leVJs2z4tAUMiT8cAqFys40kyohmzEKVjGlpNM=;
b=gZXmjVtSiG/byKYLmhVdgh61rIWotwOmzM932IAdNsVosRcc7cF7JGqQJejmRfJ9ts
OUg0eOw5OUGm7IFGMZ2+x/xPAVIAOp5pnBHwAqfzodzqe03xNe7UymQ5nhDvJq+DAVo9
m/01F4vxR6OAzNMfRYR7VC6f9THD1j3XRzCfJTz7kJEnLjSMhYkAUQyvDujJs5D7bndv
ILxnPlMRQmAU6PlMlTpjLRpQQBWY2AfqZQeYztWxhmwe8j/wUjeDx8HW33cjTtZEnTeC
PDr9l0EUEwxAxInnCrAlpuY+75GOh/YekXPCzWKvoF9vwEW7R1Mo1UGUKWOwoANvJ6cc
h+8w==
X-Forwarded-Encrypted: i=1;
AJvYcCULHEDTnpfDSIpK3VacWsMOnRfeSubzN2Feq81vzb+ZfraEJ6/FhazMvMXzqS0uRqAbMWRBarPa4mRx@nongnu.org
X-Gm-Message-State: AOJu0Yy51LaIkXy7D4u2p/RyyIGpl8SG6Zv2fuDBKaNWtbUmyiLnvYPG
hdMxyeqJq39mQRIi4Lbl8oHzJvWBymLU+xMiIYebnspdLUGiUB0S
X-Gm-Gg: ASbGncvPciqUI0r9SJY2+Snd9Fm9KbIm4uzEwb8G6FJicMZ7i4L34L64bg/rV/u/31B
8v1lB2G3vNLE2yulyxPxopDnsopqmIHDtOTQtQPktSLo4E6qYHhSxgbqdouDH1PZWKNvM3U3oPI
fjIt81f41ly8k+Xd3BwGa0ZnORyFtiu5HgyTRQVEn/xXJqoDfCTA/w+CZRJcWdxv/s9ZNzjTVMv
yjrCcMLimQJTr6lPRjYih+K7mbWMgKh0JltjDQ85FP1kwd/J1v6EyzYjWO8kFKQn3+LfELvZGYR
pu9NAkHhkKHLSLPFKuVxnWgBxIWJxatAF0afUzM1pQKQHzYKPvg=
X-Google-Smtp-Source:
AGHT+IF+TjI7wJD4rF9NQUlubzjMOWpKVKBr5lDqtHsv+SZJkKC+qq11frN8D58CXiH8N9bWZI5W+g==
X-Received: by 2002:a17:902:f611:b0:21f:53a5:19e0 with SMTP id
d9443c01a7336-22593d78ca5mr99211985ad.12.1741782371247;
Wed, 12 Mar 2025 05:26:11 -0700 (PDT)
From: Nicholas Piggin <npiggin@gmail.com>
To: Thomas Huth <thuth@redhat.com>
Cc: Nicholas Piggin <npiggin@gmail.com>,
=?UTF-8?q?Philippe=20Mathieu-Daud=C3=A9?= <philmd@linaro.org>,
=?UTF-8?q?Daniel=20P=2E=20Berrang=C3=A9?= <berrange@redhat.com>,
qemu-devel@nongnu.org
Subject: [PATCH v2 1/3] tests/functional/asset: Fail assert fetch when retries
are exceeded
Date: Wed, 12 Mar 2025 22:25:56 +1000
Message-ID: <20250312122559.944533-2-npiggin@gmail.com>
X-Mailer: git-send-email 2.47.1
In-Reply-To: <20250312122559.944533-1-npiggin@gmail.com>
References: <20250312122559.944533-1-npiggin@gmail.com>
MIME-Version: 1.0
Content-Transfer-Encoding: quoted-printable
Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17
as permitted sender) client-ip=209.51.188.17;
envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org;
helo=lists.gnu.org;
Received-SPF: pass client-ip=2607:f8b0:4864:20::62c;
envelope-from=npiggin@gmail.com; helo=mail-pl1-x62c.google.com
X-Spam_score_int: -20
X-Spam_score: -2.1
X-Spam_bar: --
X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1,
DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001,
RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001,
SPF_PASS=-0.001 autolearn=ham autolearn_force=no
X-Spam_action: no action
X-BeenThere: qemu-devel@nongnu.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: <qemu-devel.nongnu.org>
List-Unsubscribe: <https://lists.nongnu.org/mailman/options/qemu-devel>,
<mailto:qemu-devel-request@nongnu.org?subject=unsubscribe>
List-Archive: <https://lists.nongnu.org/archive/html/qemu-devel>
List-Post: <mailto:qemu-devel@nongnu.org>
List-Help: <mailto:qemu-devel-request@nongnu.org?subject=help>
List-Subscribe: <https://lists.nongnu.org/mailman/listinfo/qemu-devel>,
<mailto:qemu-devel-request@nongnu.org?subject=subscribe>
Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org
Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org
X-ZohoMail-DKIM: pass (identity @gmail.com)
X-ZM-MESSAGEID: 1741782420803019000
Content-Type: text/plain; charset="utf-8"
Currently the fetch code does not fail gracefully when retry limit is
exceeded, it just falls through the loop with no file, which ends up
hitting other errors.
Add a check for non-existing file, which indicates the retry limit was
exceeded.
Signed-off-by: Nicholas Piggin <npiggin@gmail.com>
Reviewed-by: Daniel P. Berrang=C3=A9 <berrange@redhat.com>
---
tests/functional/qemu_test/asset.py | 3 +++
1 file changed, 3 insertions(+)
diff --git a/tests/functional/qemu_test/asset.py b/tests/functional/qemu_te=
st/asset.py
index f0730695f09..27dd839e705 100644
--- a/tests/functional/qemu_test/asset.py
+++ b/tests/functional/qemu_test/asset.py
@@ -138,6 +138,9 @@ def fetch(self):
tmp_cache_file.unlink()
raise
=20
+ if not os.path.exists(tmp_cache_file):
+ raise Exception("Retries exceeded downloading %s", self.url)
+
try:
# Set these just for informational purposes
os.setxattr(str(tmp_cache_file), "user.qemu-asset-url",
--=20
2.47.1
From nobody Fri Apr 4 07:21:18 2025
Delivered-To: importer@patchew.org
Authentication-Results: mx.zohomail.com;
dkim=pass;
spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as
permitted sender)
smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org;
dmarc=pass(p=none dis=none) header.from=gmail.com
ARC-Seal: i=1; a=rsa-sha256; t=1741782417; cv=none;
d=zohomail.com; s=zohoarc;
b=di5uh1GNdLkOn1vipMLVGFj3IIH2q0b9Dzyqk98n+cd+rMgp429sr3PNWBL/vIYiG1ppzaQEp4TW74kDbwpQUhNSHU9F29A0OO7GAthE3NHCyVE8FcFFuORPZLcI44LlMLcIvg84apdmIseSY4l7kffawPmMoRNHmJ+YhNUraxc=
ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com;
s=zohoarc;
t=1741782417;
h=Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To;
bh=1UMfE3tEvJouIbe7xl+kGrgC94NEdkyZs3CPKKHqiWc=;
b=SrvwhV5EgO6wqTOLK5oc4RwTqG7h7d1QQrUCmjbJpVfCDPokbEmpbBfVOwvx9ZTb1OIxP9NYsbjmycEYu32jdS6/92siRDzKMCF8Pbm5jki3k8ifF5NVE8GoKaZhIi6oyOpjiZ+W59eIc/wH0pYQ9clmpt0uv6mZ4ULDlssfrKM=
ARC-Authentication-Results: i=1; mx.zohomail.com;
dkim=pass;
spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as
permitted sender)
smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org;
dmarc=pass header.from=<npiggin@gmail.com> (p=none dis=none)
Return-Path: <qemu-devel-bounces+importer=patchew.org@nongnu.org>
Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by
mx.zohomail.com
with SMTPS id 1741782417763446.50133068124785;
Wed, 12 Mar 2025 05:26:57 -0700 (PDT)
Received: from localhost ([::1] helo=lists1p.gnu.org)
by lists.gnu.org with esmtp (Exim 4.90_1)
(envelope-from <qemu-devel-bounces@nongnu.org>)
id 1tsLAH-0001fG-N2; Wed, 12 Mar 2025 08:26:25 -0400
Received: from eggs.gnu.org ([2001:470:142:3::10])
by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
(Exim 4.90_1) (envelope-from <npiggin@gmail.com>) id 1tsLAD-0001eG-E9
for qemu-devel@nongnu.org; Wed, 12 Mar 2025 08:26:22 -0400
Received: from mail-pl1-x629.google.com ([2607:f8b0:4864:20::629])
by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128)
(Exim 4.90_1) (envelope-from <npiggin@gmail.com>) id 1tsLA8-0001du-S3
for qemu-devel@nongnu.org; Wed, 12 Mar 2025 08:26:21 -0400
Received: by mail-pl1-x629.google.com with SMTP id
d9443c01a7336-225b5448519so5074665ad.0
for <qemu-devel@nongnu.org>; Wed, 12 Mar 2025 05:26:16 -0700 (PDT)
Received: from wheely.local0.net ([118.208.151.101])
by smtp.gmail.com with ESMTPSA id
d2e1a72fcca58-736ab33e1d3sm10842408b3a.132.2025.03.12.05.26.11
(version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256);
Wed, 12 Mar 2025 05:26:14 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
d=gmail.com; s=20230601; t=1741782375; x=1742387175; darn=nongnu.org;
h=content-transfer-encoding:mime-version:references:in-reply-to
:message-id:date:subject:cc:to:from:from:to:cc:subject:date
:message-id:reply-to;
bh=1UMfE3tEvJouIbe7xl+kGrgC94NEdkyZs3CPKKHqiWc=;
b=OtHC6zMs9uP22FJHaJgEkJAscx3UZ64mB8EJdXKqT1NqdWZze2aRZp3AvPikohDOVg
mTSaymeUAolNZtI2DI08YwiESBwZ1NB/cUmHKAjkeQ+IMmd/rCuvvhZDiEyeR72IXKsZ
gjbdtcbK2ozuedbBN34wNmNDRAi6P6f23aTOTzbCA/M6Od4+I4Lj3YraltvoLqymm8LT
4Iafuk/ta68f9diH+N83lmPhYSVBaNoi858X+VkqticColcamvk5k+ZIF2vAvgeLJ0wp
crhj59scgwSkSsEwQyRUJ0VqXOABeb2DOjocd4ym5xN+cqhL+gBV/FrhXTCUppuo8Bn5
+eRg==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
d=1e100.net; s=20230601; t=1741782375; x=1742387175;
h=content-transfer-encoding:mime-version:references:in-reply-to
:message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc
:subject:date:message-id:reply-to;
bh=1UMfE3tEvJouIbe7xl+kGrgC94NEdkyZs3CPKKHqiWc=;
b=HOCAmbTQHoXpohDNzjEsUEVYUBqs4BKIAEmMP00zMLCFb0i2KCUovTk/9SQFbTXHCg
gJvKcQMkjMqGWi51eyP/3GVJn1acYyrQEF/v5vAM2GVj6rZah35hFjEoNJN3dthjK2ny
xFGg5HoIyx9Cd17iC1PU8rlPWImljGsKcuPNV+bYMIB3e3X1pEqg+iQQnSGYXq91KukQ
78Uh0ODbrPSUw32DqjNvuxj9mOjD6TwT47nkErcTPC4CO00fFrHk58n3UNgei015gRLi
n8YErKZk1DPCp5oKfeC3WzqVOcAMv0t7aeVuHf6RD/WId7lpaHpbv+wN3dtEzbQTIKYl
8x+Q==
X-Forwarded-Encrypted: i=1;
AJvYcCVl+GHyP95H37bLpPvavrpK/Cry1gii0GvyXPynJowuV4gTIDO1C7gEMKDvyq+M1v79tHgJ4HxnTA+K@nongnu.org
X-Gm-Message-State: AOJu0Yw7c9wWyylHMx0p3hEiZ5VIuS316a997OLKBArsJAG7Iw3LzHRE
1iExh6pEPKfvSAvYFUoj1AmrvtN9wFd3ycBmWk/krQ6v7wCO6JIX
X-Gm-Gg: ASbGncsg9ghXe254pJW7R4po8MgyuYrxf6KCTfjp29cutQVGxwnDvP2Ufdu9R5pwxkw
UrbXWoy6zWFWRocyWGEmXjIIM85x5kiUaoMGcEN04vWz+66czxA4kVmx8V7J+I8VZE827VP4djc
jVyrxavJE/O66v3fJw2AJC+ZYozLgFFJJS6DhseZlupqgbGW4l1EBK+AGAGHLv8V4K1rbBL9MVM
iRDnCUsovVOIrPRsOpAif9f91tNa0xlpHeJpUvSYc4uAu0WJDdPAYlyDVRcz1/iluBqxx2OaaCG
w1afkA3uHWrokzrtW/gLyxcoIRv2FW7Mk8eHU83kCNWs2COvhv4=
X-Google-Smtp-Source:
AGHT+IFRU7fcNB0Y/lnYGCg4K2qrIV0JA1CuPxtQdhX6JKCOfcuWUzMT/z/N+Mjx1nZsgIBtSSh8PQ==
X-Received: by 2002:a05:6a00:b4e:b0:736:57cb:f2b6 with SMTP id
d2e1a72fcca58-736aaa1acf0mr30387882b3a.12.1741782374705;
Wed, 12 Mar 2025 05:26:14 -0700 (PDT)
From: Nicholas Piggin <npiggin@gmail.com>
To: Thomas Huth <thuth@redhat.com>
Cc: Nicholas Piggin <npiggin@gmail.com>,
=?UTF-8?q?Philippe=20Mathieu-Daud=C3=A9?= <philmd@linaro.org>,
=?UTF-8?q?Daniel=20P=2E=20Berrang=C3=A9?= <berrange@redhat.com>,
qemu-devel@nongnu.org
Subject: [PATCH v2 2/3] tests/functional/asset: Verify downloaded size
Date: Wed, 12 Mar 2025 22:25:57 +1000
Message-ID: <20250312122559.944533-3-npiggin@gmail.com>
X-Mailer: git-send-email 2.47.1
In-Reply-To: <20250312122559.944533-1-npiggin@gmail.com>
References: <20250312122559.944533-1-npiggin@gmail.com>
MIME-Version: 1.0
Content-Transfer-Encoding: quoted-printable
Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17
as permitted sender) client-ip=209.51.188.17;
envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org;
helo=lists.gnu.org;
Received-SPF: pass client-ip=2607:f8b0:4864:20::629;
envelope-from=npiggin@gmail.com; helo=mail-pl1-x629.google.com
X-Spam_score_int: -20
X-Spam_score: -2.1
X-Spam_bar: --
X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1,
DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001,
RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001,
SPF_PASS=-0.001 autolearn=ham autolearn_force=no
X-Spam_action: no action
X-BeenThere: qemu-devel@nongnu.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: <qemu-devel.nongnu.org>
List-Unsubscribe: <https://lists.nongnu.org/mailman/options/qemu-devel>,
<mailto:qemu-devel-request@nongnu.org?subject=unsubscribe>
List-Archive: <https://lists.nongnu.org/archive/html/qemu-devel>
List-Post: <mailto:qemu-devel@nongnu.org>
List-Help: <mailto:qemu-devel-request@nongnu.org?subject=help>
List-Subscribe: <https://lists.nongnu.org/mailman/listinfo/qemu-devel>,
<mailto:qemu-devel-request@nongnu.org?subject=subscribe>
Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org
Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org
X-ZohoMail-DKIM: pass (identity @gmail.com)
X-ZM-MESSAGEID: 1741782422565019100
Content-Type: text/plain; charset="utf-8"
If the server provides a Content-Length header, use that to verify the
size of the downloaded file. This catches cases where the connection
terminates early, and gives the opportunity to retry. Without this, the
checksum will likely mismatch and fail without retry.
Signed-off-by: Nicholas Piggin <npiggin@gmail.com>
Reviewed-by: Daniel P. Berrang=C3=A9 <berrange@redhat.com>
---
tests/functional/qemu_test/asset.py | 14 ++++++++++++++
1 file changed, 14 insertions(+)
diff --git a/tests/functional/qemu_test/asset.py b/tests/functional/qemu_te=
st/asset.py
index 27dd839e705..6bbfb9e1cad 100644
--- a/tests/functional/qemu_test/asset.py
+++ b/tests/functional/qemu_test/asset.py
@@ -121,6 +121,20 @@ def fetch(self):
with tmp_cache_file.open("xb") as dst:
with urllib.request.urlopen(self.url) as resp:
copyfileobj(resp, dst)
+ length_hdr =3D resp.getheader("Content-Length")
+
+ # Verify downloaded file size against length metadata, if
+ # available.
+ if length_hdr is not None:
+ length =3D int(length_hdr)
+ fsize =3D tmp_cache_file.stat().st_size
+ if fsize !=3D length:
+ self.log.error("Unable to download %s: "
+ "connection closed before "
+ "transfer complete (%d/%d)",
+ self.url, fsize, length)
+ tmp_cache_file.unlink()
+ continue
break
except FileExistsError:
self.log.debug("%s already exists, "
--=20
2.47.1
From nobody Fri Apr 4 07:21:18 2025
Delivered-To: importer@patchew.org
Authentication-Results: mx.zohomail.com;
dkim=pass;
spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as
permitted sender)
smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org;
dmarc=pass(p=none dis=none) header.from=gmail.com
ARC-Seal: i=1; a=rsa-sha256; t=1741782446; cv=none;
d=zohomail.com; s=zohoarc;
b=TMMFK5fXZndodkt84y+QuGFqUBpNeNqHZUyULyKrJdxvzyi5sMhZ3v2MWcGSKS49lMJwPbLGvJJyx6BZbaqTINh8+4iKYZzq+Sn+TXrfD2PxjSQalUzSv/ljXIkhvsMVPZXqJe5NfF1pdzdo5jqa1PImFjn0aeo+FWvJHe+/9mA=
ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com;
s=zohoarc;
t=1741782446;
h=Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To;
bh=B0nfqgJ67/Ws3HeQSwOWQgrBH7dxTFcRp7xc72rc9Ls=;
b=XdLBShVtBmqqGI42RjHqjtY98QrEJunkrcGTj/lIgS1HutSZotBlh62CgnZG2wUgkdANH8bUTQdDhhiljHo0aBthdt+mahNTiLusRagBCO5qGI+hNFUW2xgBQdvx7opJf2r0X15jQv9F8rEjXu8iuf7X9AB9n91glzEvqxH08pQ=
ARC-Authentication-Results: i=1; mx.zohomail.com;
dkim=pass;
spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as
permitted sender)
smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org;
dmarc=pass header.from=<npiggin@gmail.com> (p=none dis=none)
Return-Path: <qemu-devel-bounces+importer=patchew.org@nongnu.org>
Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by
mx.zohomail.com
with SMTPS id 1741782446892944.4983295047916;
Wed, 12 Mar 2025 05:27:26 -0700 (PDT)
Received: from localhost ([::1] helo=lists1p.gnu.org)
by lists.gnu.org with esmtp (Exim 4.90_1)
(envelope-from <qemu-devel-bounces@nongnu.org>)
id 1tsLAI-0001fQ-B4; Wed, 12 Mar 2025 08:26:27 -0400
Received: from eggs.gnu.org ([2001:470:142:3::10])
by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
(Exim 4.90_1) (envelope-from <npiggin@gmail.com>) id 1tsLAE-0001eZ-DG
for qemu-devel@nongnu.org; Wed, 12 Mar 2025 08:26:22 -0400
Received: from mail-pl1-x62f.google.com ([2607:f8b0:4864:20::62f])
by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128)
(Exim 4.90_1) (envelope-from <npiggin@gmail.com>) id 1tsLAC-0001eY-Bl
for qemu-devel@nongnu.org; Wed, 12 Mar 2025 08:26:22 -0400
Received: by mail-pl1-x62f.google.com with SMTP id
d9443c01a7336-224100e9a5cso125766205ad.2
for <qemu-devel@nongnu.org>; Wed, 12 Mar 2025 05:26:19 -0700 (PDT)
Received: from wheely.local0.net ([118.208.151.101])
by smtp.gmail.com with ESMTPSA id
d2e1a72fcca58-736ab33e1d3sm10842408b3a.132.2025.03.12.05.26.15
(version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256);
Wed, 12 Mar 2025 05:26:17 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
d=gmail.com; s=20230601; t=1741782378; x=1742387178; darn=nongnu.org;
h=content-transfer-encoding:mime-version:references:in-reply-to
:message-id:date:subject:cc:to:from:from:to:cc:subject:date
:message-id:reply-to;
bh=B0nfqgJ67/Ws3HeQSwOWQgrBH7dxTFcRp7xc72rc9Ls=;
b=M5UL2FFtbkKXeorzkX7QUwjzvxnUREwYuWtAZdfDw8SERcoX7qOqzES3G60dlFiP6X
g/BHMB8gW6NqeS/A+yirbNIhz2n0fpOV4va9tWixH5IZBk78Ov0eTP7xLv4/dF8ndlaG
TfArW1FBNenr1K6A4dwpDhsoTDbN72ItwvKEJ4xIm4yubE4KFUTmjvhfAN70MV7SiuK7
5runCupIizjTcbKBNsRKNEu8E/JYOEa7En0y2ghx9HKCxM0/DJUgj89no/zZLkjjIlqd
yiDgKXqBzQC3hwlCSMl1Q1SwhBHc5/0aaPIYMSMGXyg2DIVKbXuVcXb6l8fD8h7QFeHy
aUMg==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
d=1e100.net; s=20230601; t=1741782378; x=1742387178;
h=content-transfer-encoding:mime-version:references:in-reply-to
:message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc
:subject:date:message-id:reply-to;
bh=B0nfqgJ67/Ws3HeQSwOWQgrBH7dxTFcRp7xc72rc9Ls=;
b=jDRyDLwLcJhKmP37icIsGuBre9o+6DX9Of5/cV5Gq7A4TXg3IWdBTsqxbvsZadP3zu
eThYnCPLS5kq6+Bqo+obkeGPITz/EWSJQIPl8zwUFGEy2ficuPKbhexD7SoEsn8ZJp7R
O9gP1wKW84D+dlsokvi1qKtHBCTZDZJinLJ750Ak45h5UBCzsZkRRZBkRrL07u2+/540
rZyb1sFAQZGgmNfq1cvPeY1rc7sPrPtT734NHbHUqZu2ZTc+k/zxcfTSKVG3evz2x5Nx
hMhy9A3iDbptVnsdrQlVBt9HbKSXeGE9REhPgiTJqMu3j+J/m++GGJ5P+s+l2ebZh20P
nj+g==
X-Forwarded-Encrypted: i=1;
AJvYcCVVw5JCKcRgoyfWl7G7BfOiJkc2AWcW4w9EcLfpcS1tMxgvkMoJeCHcug/Uz988steHfVwvHP164dyd@nongnu.org
X-Gm-Message-State: AOJu0YxOmGw5v6VhtKavnWjqK1mLlb9hx2AmWC9SyFi3XGQCCWWESI2z
RnKdpN2PEMf/EX23m9cOVdbjrygN9DBvYvMS55OKmldy6ZXFMOCTydcQ9g==
X-Gm-Gg: ASbGncvLb0exuw1y0xptCc7cs8lvyDrY4z/JsE0hHnMto9rEAvdOE8rzuaGPzMUhtbv
rvctjgjT0ffhF0jPulDrq2pJqVmvgCRHBp1GVAJ8WmeJptVBoVlDq2/OahpgzzwpmTEi+nVOqt3
MOGsZmdBoQp331fJBfkUJJBVI7UbrtG9EpJxhLFmE2t0YsWOTUhIot3bmAjCdzjirzRRJ0qGGbj
dwl+YJCHPEhviFnE/Qx3xfm/iTrJlwFeUJGph1HFNtKd3s5chUrUfVsQLRPYyaYrs8Gs3KwIrSb
QIxcLJY2rMFAB0os06My1JgnGq2w8I5UFjDQa51x3UJuuvYTcy4=
X-Google-Smtp-Source:
AGHT+IHxcGnytqA/4fcavW4BXjDMt97/SdXvVg3UzbP0TqlXSaOCA5nCbISBRRcltZ+no0/Dq0cRqA==
X-Received: by 2002:a17:903:40cb:b0:223:44c5:4eb8 with SMTP id
d9443c01a7336-22428c0752cmr338931355ad.32.1741782378072;
Wed, 12 Mar 2025 05:26:18 -0700 (PDT)
From: Nicholas Piggin <npiggin@gmail.com>
To: Thomas Huth <thuth@redhat.com>
Cc: Nicholas Piggin <npiggin@gmail.com>,
=?UTF-8?q?Philippe=20Mathieu-Daud=C3=A9?= <philmd@linaro.org>,
=?UTF-8?q?Daniel=20P=2E=20Berrang=C3=A9?= <berrange@redhat.com>,
qemu-devel@nongnu.org
Subject: [PATCH v2 3/3] tests/functional/asset: Add AssetError exception class
Date: Wed, 12 Mar 2025 22:25:58 +1000
Message-ID: <20250312122559.944533-4-npiggin@gmail.com>
X-Mailer: git-send-email 2.47.1
In-Reply-To: <20250312122559.944533-1-npiggin@gmail.com>
References: <20250312122559.944533-1-npiggin@gmail.com>
MIME-Version: 1.0
Content-Transfer-Encoding: quoted-printable
Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17
as permitted sender) client-ip=209.51.188.17;
envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org;
helo=lists.gnu.org;
Received-SPF: pass client-ip=2607:f8b0:4864:20::62f;
envelope-from=npiggin@gmail.com; helo=mail-pl1-x62f.google.com
X-Spam_score_int: -20
X-Spam_score: -2.1
X-Spam_bar: --
X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1,
DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001,
RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001,
SPF_PASS=-0.001 autolearn=ham autolearn_force=no
X-Spam_action: no action
X-BeenThere: qemu-devel@nongnu.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: <qemu-devel.nongnu.org>
List-Unsubscribe: <https://lists.nongnu.org/mailman/options/qemu-devel>,
<mailto:qemu-devel-request@nongnu.org?subject=unsubscribe>
List-Archive: <https://lists.nongnu.org/archive/html/qemu-devel>
List-Post: <mailto:qemu-devel@nongnu.org>
List-Help: <mailto:qemu-devel-request@nongnu.org?subject=help>
List-Subscribe: <https://lists.nongnu.org/mailman/listinfo/qemu-devel>,
<mailto:qemu-devel-request@nongnu.org?subject=subscribe>
Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org
Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org
X-ZohoMail-DKIM: pass (identity @gmail.com)
X-ZM-MESSAGEID: 1741782448773019100
Content-Type: text/plain; charset="utf-8"
Assets are uniquely identified by human-readable-ish url, so make an
AssetError exception class that prints url with error message.
A property 'transient' is used to capture whether the client may retry
or try again later, or if it is a serious and likely permanent error.
This is used to retain the existing behaviour of treating HTTP errors
other than 404 as 'transient' and not causing precache step to fail.
Additionally, partial-downloads and stale asset caches that fail to
resolve after the retry limit are now treated as transient and do not
cause precache step to fail.
For background: The NetBSD archive is, at the time of writing, failing
with short transfer. Retrying the fetch at that position (as wget does)
results in a "503 backend unavailable" error. We would like to get that
error code directly, but I have not found a way to do that with urllib,
so treating the short-copy as a transient failure covers that case (and
seems like a reasonable way to handle it in general).
Reviewed-by: Thomas Huth <thuth@redhat.com>
Signed-off-by: Nicholas Piggin <npiggin@gmail.com>
Reviewed-by: Daniel P. Berrang=C3=A9 <berrange@redhat.com>
---
roms/skiboot | 2 +-
tests/functional/qemu_test/asset.py | 43 +++++++++++++++++++----------
tests/lcitool/libvirt-ci | 2 +-
3 files changed, 30 insertions(+), 17 deletions(-)
diff --git a/roms/skiboot b/roms/skiboot
index 24a7eb35966..785a5e3070a 160000
--- a/roms/skiboot
+++ b/roms/skiboot
@@ -1 +1 @@
-Subproject commit 24a7eb35966d93455520bc2debdd7954314b638b
+Subproject commit 785a5e3070a86e18521e62fe202b87209de30fa2
diff --git a/tests/functional/qemu_test/asset.py b/tests/functional/qemu_te=
st/asset.py
index 6bbfb9e1cad..704b84d0ea6 100644
--- a/tests/functional/qemu_test/asset.py
+++ b/tests/functional/qemu_test/asset.py
@@ -17,6 +17,14 @@
from shutil import copyfileobj
from urllib.error import HTTPError
=20
+class AssetError(Exception):
+ def __init__(self, asset, msg, transient=3DFalse):
+ self.url =3D asset.url
+ self.msg =3D msg
+ self.transient =3D transient
+
+ def __str__(self):
+ return "%s: %s" % (self.url, self.msg)
=20
# Instances of this class must be declared as class level variables
# starting with a name "ASSET_". This enables the pre-caching logic
@@ -51,7 +59,7 @@ def _check(self, cache_file):
elif len(self.hash) =3D=3D 128:
hl =3D hashlib.sha512()
else:
- raise Exception("unknown hash type")
+ raise AssetError(self, "unknown hash type")
=20
# Calculate the hash of the file:
with open(cache_file, 'rb') as file:
@@ -111,7 +119,8 @@ def fetch(self):
return str(self.cache_file)
=20
if not self.fetchable():
- raise Exception("Asset cache is invalid and downloads disabled=
")
+ raise AssetError(self,
+ "Asset cache is invalid and downloads disable=
d")
=20
self.log.info("Downloading %s to %s...", self.url, self.cache_file)
tmp_cache_file =3D self.cache_file.with_suffix(".download")
@@ -147,13 +156,23 @@ def fetch(self):
tmp_cache_file)
tmp_cache_file.unlink()
continue
+ except HTTPError as e:
+ tmp_cache_file.unlink()
+ self.log.error("Unable to download %s: HTTP error %d",
+ self.url, e.code)
+ # Treat 404 as fatal, since it is highly likely to
+ # indicate a broken test rather than a transient
+ # server or networking problem
+ if e.code =3D=3D 404:
+ raise AssetError(self, "Unable to download: "
+ "HTTP error %d" % e.code)
+ continue
except Exception as e:
- self.log.error("Unable to download %s: %s", self.url, e)
tmp_cache_file.unlink()
- raise
+ raise AssetError(self, "Unable to download: " % e)
=20
if not os.path.exists(tmp_cache_file):
- raise Exception("Retries exceeded downloading %s", self.url)
+ raise AssetError(self, "Download retries exceeded", transient=
=3DTrue)
=20
try:
# Set these just for informational purposes
@@ -167,8 +186,7 @@ def fetch(self):
=20
if not self._check(tmp_cache_file):
tmp_cache_file.unlink()
- raise Exception("Hash of %s does not match %s" %
- (self.url, self.hash))
+ raise AssetError(self, "Hash does not match %s" % self.hash)
tmp_cache_file.replace(self.cache_file)
# Remove write perms to stop tests accidentally modifying them
os.chmod(self.cache_file, stat.S_IRUSR | stat.S_IRGRP)
@@ -190,15 +208,10 @@ def precache_test(test):
log.info("Attempting to cache '%s'" % asset)
try:
asset.fetch()
- except HTTPError as e:
- # Treat 404 as fatal, since it is highly likely to
- # indicate a broken test rather than a transient
- # server or networking problem
- if e.code =3D=3D 404:
+ except AssetError as e:
+ if not e.transient:
raise
-
- log.debug(f"HTTP error {e.code} from {asset.url} " +
- "skipping asset precache")
+ log.error("%s: skipping asset precache" % e)
=20
log.removeHandler(handler)
=20
diff --git a/tests/lcitool/libvirt-ci b/tests/lcitool/libvirt-ci
index 18c4bfe02c4..b6a65806bc9 160000
--- a/tests/lcitool/libvirt-ci
+++ b/tests/lcitool/libvirt-ci
@@ -1 +1 @@
-Subproject commit 18c4bfe02c467e5639bf9a687139735ccd7a3fff
+Subproject commit b6a65806bc9b2b56985f5e97c936b77c7e7a99fc
--=20
2.47.1