From nobody Thu May 2 01:06:34 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=fail; spf=none (zohomail.com: 192.237.175.120 is neither permitted nor denied by domain of lists.xenproject.org) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=fail(p=none dis=none) header.from=citrix.com ARC-Seal: i=1; a=rsa-sha256; t=1576766615; cv=none; d=zohomail.com; s=zohoarc; b=TFFQxgvRCY+zV6hTGedChplbLlBHuUp9o5y8Z9/e0kBX/c6TEyhkICy4EYSmA2ZvYBfdWGDij8hhqGvLMWhPUPzd1Yc4uGgXwOUMA27/bXAZwMNvGaxxGHD5DprPmW6fY4otQrI05aV3+zDy9kP4ZjzgN1WvV6bypXrNWXIwUKg= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1576766615; h=Content-Type:Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=uaUT3Xcv2xM6EP5xm1fnbg14T0iF7GScf11b6sPcDsA=; b=m8CSEhdZhKX6e26pz4yWovsQmYkSeIcvZx19Ialv7LWWhDd1Fl7C3EQcYrgg8RUjAEeyrplGxmU71nnOSTDSCHhPgREGgDPjeXPpixMtd3dAUPkNQmoWOguGeBImg+Yc7IIkiF+Ez9dlmsS5sQSxWD5JhhEG9eWs0daIiwGbp3Q= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=fail; spf=none (zohomail.com: 192.237.175.120 is neither permitted nor denied by domain of lists.xenproject.org) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) by mx.zohomail.com with SMTPS id 1576766615917901.200877793523; Thu, 19 Dec 2019 06:43:35 -0800 (PST) Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.89) (envelope-from ) id 1ihx0V-0007U4-Jl; Thu, 19 Dec 2019 14:42:27 +0000 Received: from all-amaz-eas1.inumbo.com ([34.197.232.57] helo=us1-amaz-eas2.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.89) (envelope-from ) id 1ihx0V-0007Tz-1S for xen-devel@lists.xenproject.org; Thu, 19 Dec 2019 14:42:27 +0000 Received: from esa2.hc3370-68.iphmx.com (unknown [216.71.145.153]) by us1-amaz-eas2.inumbo.com (Halon) with ESMTPS id c3c5ee34-226d-11ea-91cf-12813bfff9fa; Thu, 19 Dec 2019 14:42:22 +0000 (UTC) X-Inumbo-ID: c3c5ee34-226d-11ea-91cf-12813bfff9fa DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=citrix.com; s=securemail; t=1576766543; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=3l9t9cQAmhU2fPxYL8qfEiIXNeCfCRbtnDPqWRrhiQY=; b=d920oNS88jvVNvAnGTeCHqbdMJSZ2JAV1sCpN56/NJiWjrMJ5B9NTUDQ EA6zFfrzFqyK0PXhrsRRdvJRO8k/SXhaOQgNvrTiOs/A8mRXPJQcmToub O5m1GqpcRDyks8Cl/6ko5juYzyGE3TaiU60WlvcV7B0ceiDiah0AbfdYB 0=; Authentication-Results: esa2.hc3370-68.iphmx.com; dkim=none (message not signed) header.i=none; spf=None smtp.pra=anthony.perard@citrix.com; spf=Pass smtp.mailfrom=anthony.perard@citrix.com; spf=None smtp.helo=postmaster@mail.citrix.com Received-SPF: none (zohomail.com: 192.237.175.120 is neither permitted nor denied by domain of lists.xenproject.org) client-ip=192.237.175.120; envelope-from=xen-devel-bounces@lists.xenproject.org; helo=lists.xenproject.org; Received-SPF: None (esa2.hc3370-68.iphmx.com: no sender authenticity information available from domain of anthony.perard@citrix.com) identity=pra; client-ip=162.221.158.21; receiver=esa2.hc3370-68.iphmx.com; envelope-from="anthony.perard@citrix.com"; x-sender="anthony.perard@citrix.com"; x-conformance=sidf_compatible Received-SPF: Pass (esa2.hc3370-68.iphmx.com: domain of anthony.perard@citrix.com designates 162.221.158.21 as permitted sender) identity=mailfrom; client-ip=162.221.158.21; receiver=esa2.hc3370-68.iphmx.com; envelope-from="anthony.perard@citrix.com"; x-sender="anthony.perard@citrix.com"; x-conformance=sidf_compatible; x-record-type="v=spf1"; x-record-text="v=spf1 ip4:209.167.231.154 ip4:178.63.86.133 ip4:195.66.111.40/30 ip4:85.115.9.32/28 ip4:199.102.83.4 ip4:192.28.146.160 ip4:192.28.146.107 ip4:216.52.6.88 ip4:216.52.6.188 ip4:162.221.158.21 ip4:162.221.156.83 ip4:168.245.78.127 ~all" Received-SPF: None (esa2.hc3370-68.iphmx.com: no sender authenticity information available from domain of postmaster@mail.citrix.com) identity=helo; client-ip=162.221.158.21; receiver=esa2.hc3370-68.iphmx.com; envelope-from="anthony.perard@citrix.com"; x-sender="postmaster@mail.citrix.com"; x-conformance=sidf_compatible IronPort-SDR: +hNBe8+XZlAepiPYhysvAU9xXrJ7MpO34COy+N0v1N4uvbmCJN+0G59fG7K7bJnd1psLj/HcKt sZgxn6JTOixTKNav+hx2/XnerOPup/AfQd/6Y6Ph9D7w5hOOk+bhjnBXuastFoQm7MpaDf7ZFe 8vnThePmrPRFLAp+Cnk2kXSYM5MNPcEJl4HsBsb+xNB5uU/mxNqv1E4yN/XJiIeq7q6WkJ9n+G 3jA/tYKUc98g7qzNO0oGz0Os320WetwnneXbrX8mnRb7+W2c41lOTqK8gYIVODoQLXyYi8KglO nwA= X-SBRS: 2.7 X-MesageID: 9949393 X-Ironport-Server: esa2.hc3370-68.iphmx.com X-Remote-IP: 162.221.158.21 X-Policy: $RELAYED X-IronPort-AV: E=Sophos;i="5.69,332,1571716800"; d="scan'208";a="9949393" From: Anthony PERARD To: Date: Thu, 19 Dec 2019 14:42:16 +0000 Message-ID: <20191219144217.305851-2-anthony.perard@citrix.com> X-Mailer: git-send-email 2.24.1 In-Reply-To: <20191219144217.305851-1-anthony.perard@citrix.com> References: <20191219144217.305851-1-anthony.perard@citrix.com> MIME-Version: 1.0 Subject: [Xen-devel] [XEN PATCH 1/2] tools: Allow to make *-dir-force-update without ./configure X-BeenThere: xen-devel@lists.xenproject.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Cc: Anthony PERARD , Ian Jackson , Doug Goldstein , Wei Liu Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Errors-To: xen-devel-bounces@lists.xenproject.org Sender: "Xen-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) This also allows to run `make src-tarball` without first having to run `./configure`. Signed-off-by: Anthony PERARD Acked-by: Wei Liu --- tools/Rules.mk | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/tools/Rules.mk b/tools/Rules.mk index cf8935d6a3ea..31cf419ef4f5 100644 --- a/tools/Rules.mk +++ b/tools/Rules.mk @@ -239,7 +239,8 @@ subdir-all-% subdir-clean-% subdir-install-% subdir-uni= nstall-%: .phony subdir-distclean-%: .phony $(MAKE) -C $* distclean =20 -ifeq (,$(findstring clean,$(MAKECMDGOALS))) +no-configure-targets :=3D clean subtree-force-update-all %-dir-force-update +ifeq (,$(filter $(no-configure-targets),$(MAKECMDGOALS))) $(XEN_ROOT)/config/Tools.mk: $(error You have to run ./configure before building or installing the too= ls) endif --=20 Anthony PERARD _______________________________________________ Xen-devel mailing list Xen-devel@lists.xenproject.org https://lists.xenproject.org/mailman/listinfo/xen-devel From nobody Thu May 2 01:06:34 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=fail; spf=none (zohomail.com: 192.237.175.120 is neither permitted nor denied by domain of lists.xenproject.org) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=fail(p=none dis=none) header.from=citrix.com ARC-Seal: i=1; a=rsa-sha256; t=1576766600; cv=none; d=zohomail.com; s=zohoarc; b=DkTegEycU5VpJVVm91q9vPudFXALspwCHO5cSePec/U47PTDkCwdEShJvSsJ7xpsfYO2eM5KFkQXkJqF9J3WSymc1MUu1x7SrxcAIk7okeysT/OEhOx+tZVR065g7lNJwSXOjhzY2IkMtP5eHbQ5cEqlt4p91A5VC79q4u/ipjA= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1576766600; h=Content-Type:Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=c7z5Bivbwj1am0vmrekSwXRZDTALZx0wMcZz5Gqp3qY=; b=j8CG5pXd/Isr9/nNXkas18EdURZjUSMg5oNz+e2An6KX+HjlRg9o4aIv6FdVqyfInmvZ+WWEbHi5SOLjKdoo2A/or4WauGUZ064V3wrN7gB/R2vBAk366ix355gTcf7x+mhUYOGn5/S5EcRcrX3tPwt7j5zvg4uxzGKXq6HpFxw= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=fail; spf=none (zohomail.com: 192.237.175.120 is neither permitted nor denied by domain of lists.xenproject.org) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) by mx.zohomail.com with SMTPS id 1576766600415390.2760469657869; Thu, 19 Dec 2019 06:43:20 -0800 (PST) Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.89) (envelope-from ) id 1ihx0a-0007UY-T1; Thu, 19 Dec 2019 14:42:32 +0000 Received: from all-amaz-eas1.inumbo.com ([34.197.232.57] helo=us1-amaz-eas2.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.89) (envelope-from ) id 1ihx0a-0007UQ-1s for xen-devel@lists.xenproject.org; Thu, 19 Dec 2019 14:42:32 +0000 Received: from esa2.hc3370-68.iphmx.com (unknown [216.71.145.153]) by us1-amaz-eas2.inumbo.com (Halon) with ESMTPS id c3c5ee35-226d-11ea-91cf-12813bfff9fa; Thu, 19 Dec 2019 14:42:22 +0000 (UTC) X-Inumbo-ID: c3c5ee35-226d-11ea-91cf-12813bfff9fa DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=citrix.com; s=securemail; t=1576766543; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=mZlcSNva71a8ScphlVLiZaz4clDPGOmGevigrs2OdN4=; b=SkLojATmbbEyCjE/yxy70ACVPtsmppCFW74J413n3/EEJ/h8HFj9YvPU glaboJebhs/xnFN4dI4CbwD8rA4dHt8bXzkAR+wEgIwgThqycq12bW0Dw 9de+VPUxVThAzD7cNP3T1G1wIdchHnbMsdJIVDijgfiswNnaCIx1cdmqN c=; Authentication-Results: esa2.hc3370-68.iphmx.com; dkim=none (message not signed) header.i=none; spf=None smtp.pra=anthony.perard@citrix.com; spf=Pass smtp.mailfrom=anthony.perard@citrix.com; spf=None smtp.helo=postmaster@mail.citrix.com Received-SPF: none (zohomail.com: 192.237.175.120 is neither permitted nor denied by domain of lists.xenproject.org) client-ip=192.237.175.120; envelope-from=xen-devel-bounces@lists.xenproject.org; helo=lists.xenproject.org; Received-SPF: None (esa2.hc3370-68.iphmx.com: no sender authenticity information available from domain of anthony.perard@citrix.com) identity=pra; client-ip=162.221.158.21; receiver=esa2.hc3370-68.iphmx.com; envelope-from="anthony.perard@citrix.com"; x-sender="anthony.perard@citrix.com"; x-conformance=sidf_compatible Received-SPF: Pass (esa2.hc3370-68.iphmx.com: domain of anthony.perard@citrix.com designates 162.221.158.21 as permitted sender) identity=mailfrom; client-ip=162.221.158.21; receiver=esa2.hc3370-68.iphmx.com; envelope-from="anthony.perard@citrix.com"; x-sender="anthony.perard@citrix.com"; x-conformance=sidf_compatible; x-record-type="v=spf1"; x-record-text="v=spf1 ip4:209.167.231.154 ip4:178.63.86.133 ip4:195.66.111.40/30 ip4:85.115.9.32/28 ip4:199.102.83.4 ip4:192.28.146.160 ip4:192.28.146.107 ip4:216.52.6.88 ip4:216.52.6.188 ip4:162.221.158.21 ip4:162.221.156.83 ip4:168.245.78.127 ~all" Received-SPF: None (esa2.hc3370-68.iphmx.com: no sender authenticity information available from domain of postmaster@mail.citrix.com) identity=helo; client-ip=162.221.158.21; receiver=esa2.hc3370-68.iphmx.com; envelope-from="anthony.perard@citrix.com"; x-sender="postmaster@mail.citrix.com"; x-conformance=sidf_compatible IronPort-SDR: aN2HoESqvoWhxaOguG5Y83Qj4cU8dP9gU7ZYqyEh262oXFYb6cK67S03M0dLtsN1YFl6sWBLXR ztcGwxKbW0n60dDoi0w5Qgcdts/gRff8NWGCUlz0WBkTT7jDPfV4jEsOLLnU9UGe0plmpASLe1 WW0zFO/wsvx/qiyboxEIu7qPa9ZLzL0yDmCfYWWlEJs/zmUKHPzSjMJ35NdA0bqNmXBGm/AjXw Bn1TFHrYArcRO1n5gfnEt2sWHW+r1JjzYv4x3CS8URYRHiMSbydERa45RArCFKPpWCvfRVFMfu q2Y= X-SBRS: 2.7 X-MesageID: 9949394 X-Ironport-Server: esa2.hc3370-68.iphmx.com X-Remote-IP: 162.221.158.21 X-Policy: $RELAYED X-IronPort-AV: E=Sophos;i="5.69,332,1571716800"; d="scan'208";a="9949394" From: Anthony PERARD To: Date: Thu, 19 Dec 2019 14:42:17 +0000 Message-ID: <20191219144217.305851-3-anthony.perard@citrix.com> X-Mailer: git-send-email 2.24.1 In-Reply-To: <20191219144217.305851-1-anthony.perard@citrix.com> References: <20191219144217.305851-1-anthony.perard@citrix.com> MIME-Version: 1.0 Subject: [Xen-devel] [XEN PATCH 2/2] automation: Cache sub-project git tree in build jobs X-BeenThere: xen-devel@lists.xenproject.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Cc: Anthony PERARD , Ian Jackson , Doug Goldstein , Wei Liu Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Errors-To: xen-devel-bounces@lists.xenproject.org Sender: "Xen-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) GitLab have a caching capability, see [1]. Let's use it to avoid using Internet too often. The cache is setup so that when xen.git/Config.mk is changed, the cache will need to be recreated. This has been chosen because that is where the information about how to clone sub-project trees is encoded (revisions). That may not work for qemu-xen tree which usually is `master', but that should be fine for now. The cache is populated of "git bundle" which will contain a mirror of the original repo, and can be cloned from. If the bundle exist, the script have the Xen makefiles clone from it, otherwise it will clone from the original URL and the bundles will be created just after. We have more than one runner in GitLab, and no shared cache between them, so every build jobs will be responsible to create the cache. [1] https://docs.gitlab.com/ee/ci/yaml/README.html#cache Signed-off-by: Anthony PERARD --- automation/gitlab-ci/build.yaml | 8 +++++ automation/scripts/prepare-cache.sh | 52 +++++++++++++++++++++++++++++ 2 files changed, 60 insertions(+) create mode 100755 automation/scripts/prepare-cache.sh diff --git a/automation/gitlab-ci/build.yaml b/automation/gitlab-ci/build.y= aml index 1e61d30c8545..8f9f53a4222f 100644 --- a/automation/gitlab-ci/build.yaml +++ b/automation/gitlab-ci/build.yaml @@ -1,6 +1,14 @@ .build-tmpl: &build stage: build image: registry.gitlab.com/xen-project/xen/${CONTAINER} + cache: + key: + files: + - Config.mk + paths: + - ci_cache + before_script: + - ./automation/scripts/prepare-cache.sh script: - ./automation/scripts/build 2>&1 | tee build.log artifacts: diff --git a/automation/scripts/prepare-cache.sh b/automation/scripts/prepa= re-cache.sh new file mode 100755 index 000000000000..017f1b8f0672 --- /dev/null +++ b/automation/scripts/prepare-cache.sh @@ -0,0 +1,52 @@ +#!/bin/bash + +set -ex + +cachedir=3D"${CI_PROJECT_DIR:=3D`pwd`}/ci_cache" +mkdir -p "$cachedir" + +declare -A r +r[extras/mini-os]=3DMINIOS_UPSTREAM_URL +r[tools/qemu-xen-dir]=3DQEMU_UPSTREAM_URL +r[tools/qemu-xen-traditional-dir]=3DQEMU_TRADITIONAL_URL +r[tools/firmware/ovmf-dir]=3DOVMF_UPSTREAM_URL +r[tools/firmware/seabios-dir]=3DSEABIOS_UPSTREAM_URL + +bundle_loc() { + echo "$cachedir/${1//\//_}.git.bundle" +} +for d in ${!r[@]}; do + if [ -e $(bundle_loc $d) ]; then + export ${r[$d]}=3D$(bundle_loc $d) + fi +done + +if ! make subtree-force-update-all; then + # There's maybe an issue with one of the git bundle, just clear the ca= che + # and allow it to be rebuilt by a different jobs. + # Make will reclone missing clones from original URLs instead of from = the + # bundle. + for d in ${!r[@]}; do + rm -f "$(bundle_loc $d)" + done + exit +fi + + +tmpdir=3D$(mktemp -d "$CI_PROJECT_DIR/ci-tmp.XXX") +for d in ${!r[@]}; do + bundle=3D$(bundle_loc $d) + if [ -e $bundle ]; then + # We didn't download anything new + continue + fi + # We create a mirror to be able to create a bundle that is a mirror of + # upstream. Otherwise, the bundle may not have refs that the build sys= tem + # will want, i.e. refs/heads/master would be missing from the bundle. + url=3D$(git --git-dir=3D$d/.git config remote.origin.url) + repo_mirrored=3D"$tmpdir/${d//\//_}" + git clone --bare --mirror --reference "$d" "$url" "$repo_mirrored" + git --git-dir=3D"$repo_mirrored" bundle create $bundle --all + rm -rf "$repo_mirrored" +done +rmdir "$tmpdir" --=20 Anthony PERARD _______________________________________________ Xen-devel mailing list Xen-devel@lists.xenproject.org https://lists.xenproject.org/mailman/listinfo/xen-devel