From nobody Tue May 7 11:07:49 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1510321485124657.9580103119293; Fri, 10 Nov 2017 05:44:45 -0800 (PST) Received: from localhost ([::1]:41795 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1eD9bs-0002wq-Ci for importer@patchew.org; Fri, 10 Nov 2017 08:44:40 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:54185) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1eD9aJ-000214-Kl for qemu-devel@nongnu.org; Fri, 10 Nov 2017 08:43:08 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1eD9aG-0006Er-E7 for qemu-devel@nongnu.org; Fri, 10 Nov 2017 08:43:03 -0500 Received: from mx1.redhat.com ([209.132.183.28]:46880) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1eD9aG-0006CP-5x for qemu-devel@nongnu.org; Fri, 10 Nov 2017 08:43:00 -0500 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.phx2.redhat.com [10.5.11.12]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 4A8A6C0467D8; Fri, 10 Nov 2017 13:42:59 +0000 (UTC) Received: from lemon.redhat.com (ovpn-12-19.pek2.redhat.com [10.72.12.19]) by smtp.corp.redhat.com (Postfix) with ESMTP id 0B14461984; Fri, 10 Nov 2017 13:42:57 +0000 (UTC) From: Fam Zheng To: qemu-devel@nongnu.org Date: Fri, 10 Nov 2017 21:42:53 +0800 Message-Id: <20171110134254.25031-2-famz@redhat.com> In-Reply-To: <20171110134254.25031-1-famz@redhat.com> References: <20171110134254.25031-1-famz@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.12 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.31]); Fri, 10 Nov 2017 13:42:59 +0000 (UTC) Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 209.132.183.28 Subject: [Qemu-devel] [PULL 1/2] docker: Improved image checksum X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Peter Maydell Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RSF_0 Z_629925259 SPT_0 Content-Type: text/plain; charset="utf-8" When a base image locally defined by QEMU, such as in the debian images, is updated, the dockerfile checksum mechanism in docker.py still skips updating the derived image, because it only looks at the literal content of the dockerfile, without considering changes to the base image. For example we have a recent fix e58c1f9b35e81 that fixed debian-win64-cross by updating its base image, debian8-mxe, but due to above "feature" of docker.py the image in question is automatically NOT rebuilt unless you add NOCACHE=3D1. It is noticed on Shippable: https://app.shippable.com/github/qemu/qemu/runs/541/2/console because after the fix is merged, the error still occurs, and the log shows the container image is, as explained above, not updated. This is because at the time docker.py was written, there wasn't any dependencies between QEMU's docker images. Now improve this to preprocess any "FROM qemu:*" directives in the dockerfiles while doing checksum, and inline the base image's dockerfile content, recursively. This ensures any changes on the depended _QEMU_ images are taken into account. This means for external images that we expect to retrieve from docker registries, we still do it as before. It is not perfect, because registry images can get updated too. Technically we could substitute the image name with its hex ID as obtained with $(docker images $IMAGE --format=3D"{{.Id}}"), but --format is not supported by RHEL 7, so leave it for now. Reported-by: Philippe Mathieu-Daud=C3=A9 Signed-off-by: Fam Zheng Message-Id: <20171103131229.4737-1-famz@redhat.com> Tested-by: Philippe Mathieu-Daud=C3=A9 Reviewed-by: Philippe Mathieu-Daud=C3=A9 Reviewed-by: Alex Benn=C3=A9e Signed-off-by: Fam Zheng --- tests/docker/docker.py | 24 +++++++++++++++++++++++- 1 file changed, 23 insertions(+), 1 deletion(-) diff --git a/tests/docker/docker.py b/tests/docker/docker.py index 08122ca17d..1246ba9578 100755 --- a/tests/docker/docker.py +++ b/tests/docker/docker.py @@ -105,6 +105,28 @@ def _copy_binary_with_libs(src, dest_dir): so_path =3D os.path.dirname(l) _copy_with_mkdir(l , dest_dir, so_path) =20 +def _read_qemu_dockerfile(img_name): + df =3D os.path.join(os.path.dirname(__file__), "dockerfiles", + img_name + ".docker") + return open(df, "r").read() + +def _dockerfile_preprocess(df): + out =3D "" + for l in df.splitlines(): + if len(l.strip()) =3D=3D 0 or l.startswith("#"): + continue + from_pref =3D "FROM qemu:" + if l.startswith(from_pref): + # TODO: Alternatively we could replace this line with "FROM $I= D" + # where $ID is the image's hex id obtained with + # $ docker images $IMAGE --format=3D"{{.Id}}" + # but unfortunately that's not supported by RHEL 7. + inlining =3D _read_qemu_dockerfile(l[len(from_pref):]) + out +=3D _dockerfile_preprocess(inlining) + continue + out +=3D l + "\n" + return out + class Docker(object): """ Running Docker commands """ def __init__(self): @@ -196,7 +218,7 @@ class Docker(object): checksum =3D self.get_image_dockerfile_checksum(tag) except Exception: return False - return checksum =3D=3D _text_checksum(dockerfile) + return checksum =3D=3D _text_checksum(_dockerfile_preprocess(docke= rfile)) =20 def run(self, cmd, keep, quiet): label =3D uuid.uuid1().hex --=20 2.13.6 From nobody Tue May 7 11:07:49 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1510321651372660.4835416805424; Fri, 10 Nov 2017 05:47:31 -0800 (PST) Received: from localhost ([::1]:41810 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1eD9eT-0004vH-Gn for importer@patchew.org; Fri, 10 Nov 2017 08:47:21 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:54207) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1eD9aM-00022f-57 for qemu-devel@nongnu.org; Fri, 10 Nov 2017 08:43:07 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1eD9aI-0006KJ-A5 for qemu-devel@nongnu.org; Fri, 10 Nov 2017 08:43:06 -0500 Received: from mx1.redhat.com ([209.132.183.28]:35296) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1eD9aI-0006I8-2v for qemu-devel@nongnu.org; Fri, 10 Nov 2017 08:43:02 -0500 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.phx2.redhat.com [10.5.11.12]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 2334661462; Fri, 10 Nov 2017 13:43:01 +0000 (UTC) Received: from lemon.redhat.com (ovpn-12-19.pek2.redhat.com [10.72.12.19]) by smtp.corp.redhat.com (Postfix) with ESMTP id D973961984; Fri, 10 Nov 2017 13:42:59 +0000 (UTC) From: Fam Zheng To: qemu-devel@nongnu.org Date: Fri, 10 Nov 2017 21:42:54 +0800 Message-Id: <20171110134254.25031-3-famz@redhat.com> In-Reply-To: <20171110134254.25031-1-famz@redhat.com> References: <20171110134254.25031-1-famz@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.12 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.39]); Fri, 10 Nov 2017 13:43:01 +0000 (UTC) Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 209.132.183.28 Subject: [Qemu-devel] [PULL 2/2] docker: correctly escape $BACKEND in the help output X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Peter Maydell Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RSF_0 Z_629925259 SPT_0 Content-Type: text/plain; charset="utf-8" From: Philippe Mathieu-Daud=C3=A9 In Makefiles the $ must be escaped as $$ in shell uses. Since 8a2390a4f47: $ make docker [...] NETWORK=3D1 Enable virtual network interface with default b= ackend. NETWORK=3DACKEND Enable virtual network interface with ACKEND. Once escaped: $ make docker [...] NETWORK=3D1 Enable virtual network interface with default b= ackend. NETWORK=3D$BACKEND Enable virtual network interface with $BACKEND. Signed-off-by: Philippe Mathieu-Daud=C3=A9 Reviewed-by: Fam Zheng Message-Id: <20171108024719.8389-1-f4bug@amsat.org> Signed-off-by: Fam Zheng --- tests/docker/Makefile.include | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include index f1a398e9fa..de87341528 100644 --- a/tests/docker/Makefile.include +++ b/tests/docker/Makefile.include @@ -109,7 +109,7 @@ docker: @echo ' DEBUG=3D1 Stop and drop to shell in the created c= ontainer' @echo ' before running the command.' @echo ' NETWORK=3D1 Enable virtual network interface with d= efault backend.' - @echo ' NETWORK=3D$BACKEND Enable virtual network interface with $= BACKEND.' + @echo ' NETWORK=3D$$BACKEND Enable virtual network interface with = $$BACKEND.' @echo ' NOUSER Define to disable adding current user to = containers passwd.' @echo ' NOCACHE=3D1 Ignore cache when build images.' @echo ' EXECUTABLE=3D Include executable in image.' --=20 2.13.6