From nobody Fri Nov 14 00:46:50 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=linaro.org ARC-Seal: i=1; a=rsa-sha256; t=1584023602; cv=none; d=zohomail.com; s=zohoarc; b=HUsv7hAs4Z9pWY9Ld5JWX8Z0lMBLTnayfnPTdMBhaTFatyY2uLFLfF9iJGWYDdmiuUUrRsm3kh2hLgVacQFb0noGh61rkUsCkytT6IxDrUtKDfq68BlnZdDa2sjwneCfLJmrW6j8WqKQbKwGOLRWAEXqabvK0FN9B0/0aCPs6OM= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1584023602; h=Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:Message-ID:References:Sender:Subject:To; bh=PSWsHdPBUl+o86ngz+CHW38RdycFivwM+vzJP6Xfyog=; b=cUXt2k3FSDy8js0ayJjsFT0yunSheYZkN8ArnPWVfDJNLteOPKDbxoXexZujrEojIvD89mYJFA3bdXyKRo1bdlNwBtis2UAg5XmlbDMoLGw+nmwDDTs2zoG9Jjo9azGE2/E0+HJNXR8pL/S9ZpFOmtjw31cAOKy3czbFx2XmdNs= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1584023602812383.1208908210492; Thu, 12 Mar 2020 07:33:22 -0700 (PDT) Received: from localhost ([::1]:42338 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jCOtl-00021Z-Je for importer@patchew.org; Thu, 12 Mar 2020 10:33:21 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:35625) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jCOr7-00076p-Bn for qemu-devel@nongnu.org; Thu, 12 Mar 2020 10:30:38 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1jCOr6-0006Qc-79 for qemu-devel@nongnu.org; Thu, 12 Mar 2020 10:30:37 -0400 Received: from mail-pf1-x434.google.com ([2607:f8b0:4864:20::434]:46739) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1jCOr6-0006OW-1Y for qemu-devel@nongnu.org; Thu, 12 Mar 2020 10:30:36 -0400 Received: by mail-pf1-x434.google.com with SMTP id c19so3346573pfo.13 for ; Thu, 12 Mar 2020 07:30:35 -0700 (PDT) Received: from Rfoley-MA01.hsd1.ma.comcast.net ([2601:199:4480:60c0:845e:b9f6:81a6:8f5e]) by smtp.gmail.com with ESMTPSA id 63sm14832651pfx.132.2020.03.12.07.30.32 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 12 Mar 2020 07:30:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=PSWsHdPBUl+o86ngz+CHW38RdycFivwM+vzJP6Xfyog=; b=wv0FIzbJEPyCv1yZrPo+729Au61CA2reHfcpHul9GRl1A0vkUePVz02W+6B0Ijk/0C f7Eu1qssnHlyrDbXso+9DUfZzCRLX3S0mZZEfPiqB+vMQ/NjQIRx49Ly7A1L4eILKlO1 n6gPVC0LtUkXFeWriyNCwLAfGRqJLFJlsJ0V6J0qc17DUW2UWk8t4rT/VqkkESh66H0f jBj38N5930aOCDmyjLvH7ys9iyPpo81OnhWZ3su/T5xKRMWYF7plG9P4R42JFacq9MBP f/6Tc1aBBC6jZmujfzh0Z64+WbQg1H6rV7ab0ZE0c5FJxsn8ZOU0MkC/GIMOMyFh8ewd lDDQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=PSWsHdPBUl+o86ngz+CHW38RdycFivwM+vzJP6Xfyog=; b=sgykw8yfe/saS8C9WI/FHN6TOQRoSzT3JGaRaUeq7cp+wenrzUiIMb/dkuLKATwopA e7OAkxxRIOWYUDOeiJCr28a65Jmxyd7W1+E7Rsxt5574WsbJz2Aj1jq7kNpRo+qAr0NW MNF5l9vw5/fS45TfyveenTDeC0evNb7QlOBEPt5ywpO9onx4mx39a8/BYkqit7Ed8b6H P+w/S/2zJQffHC2CYr6eqVhPFnksuFa/be9EC7mptNq+HY0z9KpMjswfuC+KpQZv7+4j 7wwvXrMUU84KzUIBESqLY/5ZwoyCeabGUJPa0r3ZM7GvAFu0xJ15hx29JxxiqV3kBchb i9gQ== X-Gm-Message-State: ANhLgQ0ULGqcFUYQhL3bS2lrL7Z/Uh7tif5WTO4XwgM2h2oc9/z6r66h DUBBkDt0wDO5aHM7Z2t+BOoWj3c1Hnw= X-Google-Smtp-Source: ADFU+vtWUlH2ppAgbSUCRHc3qW5hRiUTZE9rDwpY9GCmWQT5AotRRvwlJ8rKJHNl4L570vO+Evdq2Q== X-Received: by 2002:a63:7c1d:: with SMTP id x29mr7694926pgc.197.1584023434454; Thu, 12 Mar 2020 07:30:34 -0700 (PDT) From: Robert Foley To: qemu-devel@nongnu.org Subject: [PATCH v4 01/10] tests/vm: Fix issues in gen_cloud_init_iso Date: Thu, 12 Mar 2020 10:27:19 -0400 Message-Id: <20200312142728.12285-2-robert.foley@linaro.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20200312142728.12285-1-robert.foley@linaro.org> References: <20200312142728.12285-1-robert.foley@linaro.org> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:4864:20::434 X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: philmd@redhat.com, alex.bennee@linaro.org, robert.foley@linaro.org, peter.puhov@linaro.org Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Fixed issue caused by dependencies on a later set of patches that have not been merged yet. Signed-off-by: Robert Foley --- tests/vm/basevm.py | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/tests/vm/basevm.py b/tests/vm/basevm.py index 8400b0e07f..db479a65fd 100644 --- a/tests/vm/basevm.py +++ b/tests/vm/basevm.py @@ -353,23 +353,23 @@ class BaseVM(object): "local-hostname: {}-guest\n".format(name)]) mdata.close() udata =3D open(os.path.join(cidir, "user-data"), "w") - print("guest user:pw {}:{}".format(self._config['guest_user'], - self._config['guest_pass'])) + print("guest user:pw {}:{}".format(self.GUEST_USER, + self.GUEST_PASS)) udata.writelines(["#cloud-config\n", "chpasswd:\n", " list: |\n", - " root:%s\n" % self._config['root_pass'], - " %s:%s\n" % (self._config['guest_user'], - self._config['guest_pass']), + " root:%s\n" % self.ROOT_PASS, + " %s:%s\n" % (self.GUEST_USER, + self.GUEST_PASS), " expire: False\n", "users:\n", - " - name: %s\n" % self._config['guest_user'], + " - name: %s\n" % self.GUEST_USER, " sudo: ALL=3D(ALL) NOPASSWD:ALL\n", " ssh-authorized-keys:\n", - " - %s\n" % self._config['ssh_pub_key'], + " - %s\n" % SSH_PUB_KEY.rstrip(), " - name: root\n", " ssh-authorized-keys:\n", - " - %s\n" % self._config['ssh_pub_key'], + " - %s\n" % SSH_PUB_KEY.rstrip(), "locale: en_US.UTF-8\n"]) proxy =3D os.environ.get("http_proxy") if not proxy is None: --=20 2.17.1 From nobody Fri Nov 14 00:46:50 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=linaro.org ARC-Seal: i=1; a=rsa-sha256; t=1584023745; cv=none; d=zohomail.com; s=zohoarc; b=J4EDcBdt1ydrlll4Yt1ms3QbJYnNgQV2uZb+JcYiufXPH18g+GDMqlxqifxNhBq6SRKJSgrBKCH3hn7pfqiGoxIB58u2C5x8wbB50bjyUJkr21srcZbhIkic+vEOuKfyBsO/Jp/yRPq0iNfYvsDUbrtlOrv4+P9fZkuYLw49/Q0= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1584023745; h=Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:Message-ID:References:Sender:Subject:To; bh=8cjhkat5SI1SyV/p2VUsqhwuExQNXm65M6Nib9b3T5k=; b=na9zi9iYfh8jwrunxKBOhlCyodN6ao0uSKz8KcCdZ3qcYQQ/kwgaNcni2gkEdc2ppVusQ6O6B+o/+K329dbDwj3eJ9a/ISzaLBpEue1/UwO88t2US9pgClFac6//9twp4TU+Zvq7nysWxHYi6RU9eGwp+mBJM0AZsTPcXY5DROI= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 15840237457221019.2247070575247; Thu, 12 Mar 2020 07:35:45 -0700 (PDT) Received: from localhost ([::1]:42388 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jCOw4-0006qe-Jl for importer@patchew.org; Thu, 12 Mar 2020 10:35:44 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:35642) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jCOrA-0007AA-7B for qemu-devel@nongnu.org; Thu, 12 Mar 2020 10:30:42 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1jCOr8-0006SU-6O for qemu-devel@nongnu.org; Thu, 12 Mar 2020 10:30:40 -0400 Received: from mail-pl1-x62f.google.com ([2607:f8b0:4864:20::62f]:46111) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1jCOr7-0006Rm-UR for qemu-devel@nongnu.org; Thu, 12 Mar 2020 10:30:38 -0400 Received: by mail-pl1-x62f.google.com with SMTP id w12so2721964pll.13 for ; Thu, 12 Mar 2020 07:30:37 -0700 (PDT) Received: from Rfoley-MA01.hsd1.ma.comcast.net ([2601:199:4480:60c0:845e:b9f6:81a6:8f5e]) by smtp.gmail.com with ESMTPSA id 63sm14832651pfx.132.2020.03.12.07.30.34 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 12 Mar 2020 07:30:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=8cjhkat5SI1SyV/p2VUsqhwuExQNXm65M6Nib9b3T5k=; b=oXCaFS6Z0bZshOiBHLpxYDVjpe/mA3nr8UKakpXzHqVGZUpYZnl9AymHJc/REp47dt 06y24gDBXEQt925LmGJlfqINL/QvnexrevhckPdXBqZ0Owf8vsedJ+yPgtU5vgc83TxZ rFi0YL50zXXzQUq8oY59ImkWaY6TEttoTO/vlP3Xn3/jmYL5S1TJRG2qlWF5dQoi7w8Q cJRpaEe5+uLdXLTK0IOnHzzG25wLbTQYu72oQEe3MYFfchYrEKZ0PIOmTxhNz1NZZfpQ XOGXscdnbuMwJcYKDB2skkqOx/628Xsr9yyzRnB6dM+mKcLd7G56xt27W4wZEFHYLkFK mvCA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=8cjhkat5SI1SyV/p2VUsqhwuExQNXm65M6Nib9b3T5k=; b=XRnBpC4JrNxsTdWH/or2nDTJnLml6jeEHr5/jsJTcQRx72bZOaW1jQrb9wtufEHLPv 8UZeaWTJ3gIlMj2x8/ZL+qhZIezWTHbU0thrbbkgDE3fTiq7kEsJvrc8+/nlcpRjE1FG gpUFhUHKHego09d/g7AsqBXPCmOUoh/EZdzDpJ+c1vWSdUsZDAKSKKNiCiVBtbT38rQw cywVAFoBh2xMqLg5Vbb9KIy3SteBIBHD9oYQo5OP6QcrC0ZINv56FZT1uFVFGMKHAn2W tZEKcdbEViKzFZKB9ot6XsyRZbGJtyABHXQyXw2sgzDbcvLGrLrsoMAtxupHlm/gi+0d Whsg== X-Gm-Message-State: ANhLgQ0gSxtmPb/fPdcwXDKFSxJBYKlX5PEIX4RtT7l+l1g1+7GgOfBq 6L/+f4OTGU3uNGKPXPrK/Copy3T67ws= X-Google-Smtp-Source: ADFU+vuYWESf6LkcBMxYCnkjnLvAe3kYY5B4IyJIOZ1Qm2diFNHRqwPg1sECNX9tsyKwGaXs4+fopA== X-Received: by 2002:a17:90a:f0cd:: with SMTP id fa13mr4380053pjb.129.1584023436220; Thu, 12 Mar 2020 07:30:36 -0700 (PDT) From: Robert Foley To: qemu-devel@nongnu.org Subject: [PATCH v4 02/10] tests/vm: Add configuration to basevm.py Date: Thu, 12 Mar 2020 10:27:20 -0400 Message-Id: <20200312142728.12285-3-robert.foley@linaro.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20200312142728.12285-1-robert.foley@linaro.org> References: <20200312142728.12285-1-robert.foley@linaro.org> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:4864:20::62f X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: philmd@redhat.com, alex.bennee@linaro.org, robert.foley@linaro.org, peter.puhov@linaro.org Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Added use of a configuration to tests/vm/basevm.py. The configuration provides parameters used to configure a VM. This allows for providing alternate configurations to the VM being created/launched. cpu, machine, memory, and NUMA configuration are all examples of configuration which we might want to vary on the VM being creat= ed or launched. This will for example allow for creating an aarch64 vm. Signed-off-by: Robert Foley --- tests/vm/basevm.py | 159 ++++++++++++++++++++++++++++++++++----------- 1 file changed, 120 insertions(+), 39 deletions(-) diff --git a/tests/vm/basevm.py b/tests/vm/basevm.py index db479a65fd..97c6f625c9 100644 --- a/tests/vm/basevm.py +++ b/tests/vm/basevm.py @@ -30,15 +30,39 @@ import shutil import multiprocessing import traceback =20 -SSH_KEY =3D open(os.path.join(os.path.dirname(__file__), - "..", "keys", "id_rsa")).read() -SSH_PUB_KEY =3D open(os.path.join(os.path.dirname(__file__), - "..", "keys", "id_rsa.pub")).read() - +SSH_KEY_FILE =3D os.path.join(os.path.dirname(__file__), + "..", "keys", "id_rsa") +SSH_PUB_KEY_FILE =3D os.path.join(os.path.dirname(__file__), + "..", "keys", "id_rsa.pub") + +# This is the standard configuration. +# Any or all of these can be overridden by +# passing in a config argument to the VM constructor. +DEFAULT_CONFIG =3D { + 'cpu' : "max", + 'machine' : 'pc', + 'guest_user' : "qemu", + 'guest_pass' : "qemupass", + 'root_pass' : "qemupass", + 'ssh_key_file' : SSH_KEY_FILE, + 'ssh_pub_key_file': SSH_PUB_KEY_FILE, + 'memory' : "4G", + 'extra_args' : [], + 'qemu_args' : "", + 'dns' : "", + 'ssh_port' : 0, + 'install_cmds' : "", + 'boot_dev_type' : "block", + 'ssh_timeout' : 1, +} +BOOT_DEVICE =3D { + 'block' : "-drive file=3D{},if=3Dnone,id=3Ddrive0,cache=3Dwriteback "\ + "-device virtio-blk,drive=3Ddrive0,bootindex=3D0", + 'scsi' : "-device virtio-scsi-device,id=3Dscsi "\ + "-drive file=3D{},format=3Draw,if=3Dnone,id=3Dhd0 "\ + "-device scsi-hd,drive=3Dhd0,bootindex=3D0", +} class BaseVM(object): - GUEST_USER =3D "qemu" - GUEST_PASS =3D "qemupass" - ROOT_PASS =3D "qemupass" =20 envvars =3D [ "https_proxy", @@ -61,19 +85,30 @@ class BaseVM(object): # 4 is arbitrary, but greater than 2, # since we found we need to wait more than twice as long. tcg_ssh_timeout_multiplier =3D 4 - def __init__(self, debug=3DFalse, vcpus=3DNone): + def __init__(self, debug=3DFalse, vcpus=3DNone, config=3DNone): self._guest =3D None + # Allow input config to override defaults. + self._config =3D DEFAULT_CONFIG.copy() + if config !=3D None: + self._config.update(config) + self.validate_ssh_keys() self._tmpdir =3D os.path.realpath(tempfile.mkdtemp(prefix=3D"vm-te= st-", suffix=3D".tmp", dir=3D".")) atexit.register(shutil.rmtree, self._tmpdir) - - self._ssh_key_file =3D os.path.join(self._tmpdir, "id_rsa") - open(self._ssh_key_file, "w").write(SSH_KEY) - subprocess.check_call(["chmod", "600", self._ssh_key_file]) - - self._ssh_pub_key_file =3D os.path.join(self._tmpdir, "id_rsa.pub") - open(self._ssh_pub_key_file, "w").write(SSH_PUB_KEY) + # Copy the key files to a temporary directory. + # Also chmod the key file to agree with ssh requirements. + self._config['ssh_key'] =3D \ + open(self._config['ssh_key_file']).read().rstrip() + self._config['ssh_pub_key'] =3D \ + open(self._config['ssh_pub_key_file']).read().rstrip() + self._ssh_tmp_key_file =3D os.path.join(self._tmpdir, "id_rsa") + open(self._ssh_tmp_key_file, "w").write(self._config['ssh_key']) + subprocess.check_call(["chmod", "600", self._ssh_tmp_key_file]) + + self._ssh_tmp_pub_key_file =3D os.path.join(self._tmpdir, "id_rsa.= pub") + open(self._ssh_tmp_pub_key_file, + "w").write(self._config['ssh_pub_key']) =20 self.debug =3D debug self._stderr =3D sys.stderr @@ -82,11 +117,14 @@ class BaseVM(object): self._stdout =3D sys.stdout else: self._stdout =3D self._devnull + netdev =3D "user,id=3Dvnet,hostfwd=3D:127.0.0.1:{}-:22" self._args =3D [ \ - "-nodefaults", "-m", "4G", - "-cpu", "max", - "-netdev", "user,id=3Dvnet,hostfwd=3D:127.0.0.1:0-:22" + - (",ipv6=3Dno" if not self.ipv6 else ""), + "-nodefaults", "-m", self._config['memory'], + "-cpu", self._config['cpu'], + "-netdev", + netdev.format(self._config['ssh_port']) + + (",ipv6=3Dno" if not self.ipv6 else "") + + (",dns=3D" + self._config['dns'] if self._config['dns'] else "= "), "-device", "virtio-net-pci,netdev=3Dvnet", "-vnc", "127.0.0.1:0,to=3D20"] if vcpus and vcpus > 1: @@ -97,6 +135,45 @@ class BaseVM(object): logging.info("KVM not available, not using -enable-kvm") self._data_args =3D [] =20 + if self._config['qemu_args'] !=3D None: + qemu_args =3D self._config['qemu_args'] + qemu_args =3D qemu_args.replace('\n',' ').replace('\r','') + # Remove any empty strings from list. + self._config['extra_args'] =3D [x for x in qemu_args.split(' '= ) if x] + + def validate_ssh_keys(self): + """Check to see if the ssh key files exist.""" + if 'ssh_key_file' not in self._config or\ + not os.path.exists(self._config['ssh_key_file']): + raise Exception("ssh key file not found.") + if 'ssh_pub_key_file' not in self._config or\ + not os.path.exists(self._config['ssh_pub_key_file']): + raise Exception("ssh pub key file not found.") + + def wait_boot(self, wait_string=3DNone): + """Wait for the standard string we expect + on completion of a normal boot. + The user can also choose to override with an + alternate string to wait for.""" + if wait_string is None: + if self.login_prompt is None: + raise Exception("self.login_prompt not defined") + wait_string =3D self.login_prompt + # Intentionally bump up the default timeout under TCG, + # since the console wait below takes longer. + timeout =3D self.socket_timeout + if not kvm_available(self.arch): + timeout *=3D 8 + self.console_init(timeout=3Dtimeout) + self.console_wait(wait_string) + + def __getattr__(self, name): + # Support direct access to config by key. + # for example, access self._config['cpu'] by self.cpu + if name.lower() in self._config.keys(): + return self._config[name.lower()] + return object.__getattribute__(self, name) + def _download_with_cache(self, url, sha256sum=3DNone, sha512sum=3DNone= ): def check_sha256sum(fname): if not sha256sum: @@ -128,8 +205,9 @@ class BaseVM(object): "-t", "-o", "StrictHostKeyChecking=3Dno", "-o", "UserKnownHostsFile=3D" + os.devnull, - "-o", "ConnectTimeout=3D1", - "-p", self.ssh_port, "-i", self._ssh_key_file] + "-o", + "ConnectTimeout=3D{}".format(self._config["ssh_timeout"= ]), + "-p", self.ssh_port, "-i", self._ssh_tmp_key_file] # If not in debug mode, set ssh to quiet mode to # avoid printing the results of commands. if not self.debug: @@ -178,15 +256,15 @@ class BaseVM(object): "virtio-blk,drive=3D%s,serial=3D%s,bootindex= =3D1" % (name, name)] =20 def boot(self, img, extra_args=3D[]): - args =3D self._args + [ - "-device", "VGA", - "-drive", "file=3D%s,if=3Dnone,id=3Ddrive0,cache=3Dwriteback" = % img, - "-device", "virtio-blk,drive=3Ddrive0,bootindex=3D0"] - args +=3D self._data_args + extra_args + boot_dev =3D BOOT_DEVICE[self._config['boot_dev_type']] + boot_params =3D boot_dev.format(img) + args =3D self._args + boot_params.split(' ') + args +=3D self._data_args + extra_args + self._config['extra_args'] + args +=3D ["-device", "VGA"] logging.debug("QEMU args: %s", " ".join(args)) qemu_bin =3D os.environ.get("QEMU", "qemu-system-" + self.arch) guest =3D QEMUMachine(binary=3Dqemu_bin, args=3Dargs) - guest.set_machine('pc') + guest.set_machine(self._config['machine']) guest.set_console() try: guest.launch() @@ -294,7 +372,8 @@ class BaseVM(object): self.console_send(command) =20 def console_ssh_init(self, prompt, user, pw): - sshkey_cmd =3D "echo '%s' > .ssh/authorized_keys\n" % SSH_PUB_KEY.= rstrip() + sshkey_cmd =3D "echo '%s' > .ssh/authorized_keys\n" \ + % self._config['ssh_pub_key'].rstrip() self.console_wait_send("login:", "%s\n" % user) self.console_wait_send("Password:", "%s\n" % pw) self.console_wait_send(prompt, "mkdir .ssh\n") @@ -353,23 +432,23 @@ class BaseVM(object): "local-hostname: {}-guest\n".format(name)]) mdata.close() udata =3D open(os.path.join(cidir, "user-data"), "w") - print("guest user:pw {}:{}".format(self.GUEST_USER, - self.GUEST_PASS)) + print("guest user:pw {}:{}".format(self._config['guest_user'], + self._config['guest_pass'])) udata.writelines(["#cloud-config\n", "chpasswd:\n", " list: |\n", - " root:%s\n" % self.ROOT_PASS, - " %s:%s\n" % (self.GUEST_USER, - self.GUEST_PASS), + " root:%s\n" % self._config['root_pass'], + " %s:%s\n" % (self._config['guest_user'], + self._config['guest_pass']), " expire: False\n", "users:\n", - " - name: %s\n" % self.GUEST_USER, + " - name: %s\n" % self._config['guest_user'], " sudo: ALL=3D(ALL) NOPASSWD:ALL\n", " ssh-authorized-keys:\n", - " - %s\n" % SSH_PUB_KEY.rstrip(), + " - %s\n" % self._config['ssh_pub_key'], " - name: root\n", " ssh-authorized-keys:\n", - " - %s\n" % SSH_PUB_KEY.rstrip(), + " - %s\n" % self._config['ssh_pub_key'], "locale: en_US.UTF-8\n"]) proxy =3D os.environ.get("http_proxy") if not proxy is None: @@ -422,15 +501,17 @@ def parse_args(vmcls): parser.disable_interspersed_args() return parser.parse_args() =20 -def main(vmcls): +def main(vmcls, config=3DNone): try: + if config =3D=3D None: + config =3D {} args, argv =3D parse_args(vmcls) if not argv and not args.build_qemu and not args.build_image: print("Nothing to do?") return 1 logging.basicConfig(level=3D(logging.DEBUG if args.debug else logging.WARN)) - vm =3D vmcls(debug=3Dargs.debug, vcpus=3Dargs.jobs) + vm =3D vmcls(debug=3Dargs.debug, vcpus=3Dargs.jobs, config=3Dconfi= g) if args.build_image: if os.path.exists(args.image) and not args.force: sys.stderr.writelines(["Image file exists: %s\n" % args.im= age, --=20 2.17.1 From nobody Fri Nov 14 00:46:50 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=linaro.org ARC-Seal: i=1; a=rsa-sha256; t=1584023550; cv=none; d=zohomail.com; s=zohoarc; b=B+KflSBWiBEy4xRTs4Mq4tw0xF9cvnKQLd5TkeQheo+iuBBr1WMPV7DOY/9MeSMTpv9J4nnaVRWM7+t2kNSXp149wxAAsza/Oyf73wgMCyf+12PRBesqgnwpVY8w5BYvXlJeImCHIS0DyXMAN52teAiNxdaFV+q2+Uykyuo4fRw= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1584023550; h=Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:Message-ID:References:Sender:Subject:To; bh=w4JJmuHvLzQg3vaVBI7GHDHOQVzkw/mMvoetRPw77+8=; b=DT8J5O2gqLqWTbiCH4fmWFWj163O94JCHJjrq+GJpr/e9ssHqfFlFkQrjLmqQ9MF6ByHUOjnzyBErH9GCe7L3qy5zoGhLL9pLVys8gRnuPUzGStxNmqP8NcBPxVB+XeLAYIPvrjdM+aiYCAZP+DW/gE2m+vUd7i8RxoFJCq7WfM= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1584023550609366.277680512035; Thu, 12 Mar 2020 07:32:30 -0700 (PDT) Received: from localhost ([::1]:42322 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jCOsv-0000Pf-2V for importer@patchew.org; Thu, 12 Mar 2020 10:32:29 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:35659) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jCOrC-0007Bt-2r for qemu-devel@nongnu.org; Thu, 12 Mar 2020 10:30:43 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1jCOrA-0006U9-2R for qemu-devel@nongnu.org; Thu, 12 Mar 2020 10:30:42 -0400 Received: from mail-pg1-x543.google.com ([2607:f8b0:4864:20::543]:37608) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1jCOr9-0006TQ-SC for qemu-devel@nongnu.org; Thu, 12 Mar 2020 10:30:40 -0400 Received: by mail-pg1-x543.google.com with SMTP id a32so2315308pga.4 for ; Thu, 12 Mar 2020 07:30:39 -0700 (PDT) Received: from Rfoley-MA01.hsd1.ma.comcast.net ([2601:199:4480:60c0:845e:b9f6:81a6:8f5e]) by smtp.gmail.com with ESMTPSA id 63sm14832651pfx.132.2020.03.12.07.30.36 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 12 Mar 2020 07:30:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=w4JJmuHvLzQg3vaVBI7GHDHOQVzkw/mMvoetRPw77+8=; b=W/mnVi8c2KjM/5e2QZ3Ftsnxdjo05uFVABNXcZYmTNenvI5PbD8DkGneoaoo0PN3uQ HNNPcsvB7mxOJhCnPsejFwKv0BuvfJUwWZ+MK2nfVFYAqJw+pkRaXohL8UnjPBFupcVo E7Q+OTkPS57iAxpkJnDlxXDijd1gDc0I+IUtyHZoay2/TH9QNV3He699JLSCfMZwi0vm CcCIHSxp13yiDzQ5uMYnmAUTMF4cabJuQUN90jvdlTWC6ImmCAKhIvRJUs0IzQsYdrHX CVdIqQZbfvy0GYRgVV3djkFiJfPbrOPnNrxv1BgTqZdzQq66pZcMChWtZ1b365u0pY/9 VPOQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=w4JJmuHvLzQg3vaVBI7GHDHOQVzkw/mMvoetRPw77+8=; b=KbJH/+lZt+cuj2G23rabfGcxyVpbp1NVvWj0o0v4Rlk7COobL1ho7UwwlYF34ynB2H 3NGnIcs0hoHvDOJz0LCR8o33/4HKiE/5nswuKeLD2tM8/ESWW8T7wy4Vv6KGKOpegbY+ Lge76+rjMw9vtqy3CYctuhUpQZOHr4n9L7+0f3wYA+jG1M9HunNuepfXlYd6M31UO2mh oryDFk3WxfLe4oPzeoWdWW5kPYIz+D2Pc/zLVDs6DNKYmvRaV/n1q3In+nmb2XYGHKtH WaJgFcYwfG6nKrVR9WUBIymlReZldVeKhFsqJidswu8Pp45norXjAax0XxNKJqHP4mkZ NuEw== X-Gm-Message-State: ANhLgQ0DrcIfXgx4v73TwGOPCWkrmASYCdJ2ZZ4Y5RLIykGW5OsV/TL8 m/BujkGYVXh/YVRR/Pex33r7vs9Dhqg= X-Google-Smtp-Source: ADFU+vsu+g5JSl+gD0SdqXNxdr2liQ3KLg0ldIshtSl9UYQYuzbXutyozg2Vr2P04YW89Q2wDSUqVw== X-Received: by 2002:a63:e007:: with SMTP id e7mr8479101pgh.262.1584023438147; Thu, 12 Mar 2020 07:30:38 -0700 (PDT) From: Robert Foley To: qemu-devel@nongnu.org Subject: [PATCH v4 03/10] tests/vm: Added configuration file support Date: Thu, 12 Mar 2020 10:27:21 -0400 Message-Id: <20200312142728.12285-4-robert.foley@linaro.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20200312142728.12285-1-robert.foley@linaro.org> References: <20200312142728.12285-1-robert.foley@linaro.org> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:4864:20::543 X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: philmd@redhat.com, alex.bennee@linaro.org, robert.foley@linaro.org, peter.puhov@linaro.org Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Changes to tests/vm/basevm.py to allow accepting a configuration file as a parameter. Allows for specifying VM options such as cpu, machine, memory, and arbitrary qemu arguments for specifying options such as NUMA configuration. Also added an example conf_example_aarch64.yml and conf_example_x86.yml. Signed-off-by: Robert Foley --- configure | 9 ++++++ tests/vm/Makefile.include | 6 ++++ tests/vm/basevm.py | 40 +++++++++++++++++++++++- tests/vm/conf_example_aarch64.yml | 51 +++++++++++++++++++++++++++++++ tests/vm/conf_example_x86.yml | 50 ++++++++++++++++++++++++++++++ 5 files changed, 155 insertions(+), 1 deletion(-) create mode 100644 tests/vm/conf_example_aarch64.yml create mode 100644 tests/vm/conf_example_x86.yml diff --git a/configure b/configure index 3c7470096f..739cde2318 100755 --- a/configure +++ b/configure @@ -947,6 +947,13 @@ do fi done =20 +# Check for existence of python3 yaml, needed to +# import yaml config files into vm-build. +python_yaml=3D"no" +if $(python3 -c "import yaml" 2> /dev/null); then + python_yaml=3D"yes" +fi + : ${smbd=3D${SMBD-/usr/sbin/smbd}} =20 # Default objcc to clang if available, otherwise use CC @@ -6584,6 +6591,7 @@ if test "$docs" !=3D "no"; then echo "sphinx-build $sphinx_build" fi echo "genisoimage $genisoimage" +echo "python_yaml $python_yaml" echo "slirp support $slirp $(echo_version $slirp $slirp_version)" if test "$slirp" !=3D "no" ; then echo "smbd $smbd" @@ -7634,6 +7642,7 @@ echo "INSTALL_LIB=3D$install -c -m 0644" >> $config_h= ost_mak echo "PYTHON=3D$python" >> $config_host_mak echo "SPHINX_BUILD=3D$sphinx_build" >> $config_host_mak echo "GENISOIMAGE=3D$genisoimage" >> $config_host_mak +echo "PYTHON_YAML=3D$python_yaml" >> $config_host_mak echo "CC=3D$cc" >> $config_host_mak if $iasl -h > /dev/null 2>&1; then echo "IASL=3D$iasl" >> $config_host_mak diff --git a/tests/vm/Makefile.include b/tests/vm/Makefile.include index 1bf9693d19..78a2de1f82 100644 --- a/tests/vm/Makefile.include +++ b/tests/vm/Makefile.include @@ -43,6 +43,12 @@ endif @echo " V=3D1 - Enable verbose ouput on host and guest commands" @echo " QEMU=3D/path/to/qemu - Change path to QEMU binary" @echo " QEMU_IMG=3D/path/to/qemu-img - Change path to qemu-img tool" +ifeq ($(PYTHON_YAML),yes) + @echo " QEMU_CONFIG=3D/path/conf.yml - Change path to VM configurati= on .yml file." +else + @echo " (install python3-yaml to enable support for yaml file to confi= gure a VM.)" +endif + @echo " See conf_example_*.yml for file= format details." =20 vm-build-all: $(addprefix vm-build-, $(IMAGES)) =20 diff --git a/tests/vm/basevm.py b/tests/vm/basevm.py index 97c6f625c9..dd545d3d1d 100644 --- a/tests/vm/basevm.py +++ b/tests/vm/basevm.py @@ -461,9 +461,43 @@ class BaseVM(object): cwd=3Dcidir, stdin=3Dself._devnull, stdout=3Dself._stdou= t, stderr=3Dself._stdout) - return os.path.join(cidir, "cloud-init.iso") =20 +def parse_config(config, args): + """ Parse yaml config and populate our config structure. + The yaml config allows the user to override the + defaults for VM parameters. In many cases these + defaults can be overridden without rebuilding the VM.""" + if args.config: + config_file =3D args.config + elif 'QEMU_CONFIG' in os.environ: + config_file =3D os.environ['QEMU_CONFIG'] + else: + return config + if not os.path.exists(config_file): + raise Exception("config file {} does not exist".format(config_file= )) + # We gracefully handle importing the yaml module + # since it might not be installed. + # If we are here it means the user supplied a .yml file, + # so if the yaml module is not installed we will exit with error. + try: + import yaml + except ImportError: + print("The python3-yaml package is needed "\ + "to support config.yaml files") + # Instead of raising an exception we exit to avoid + # a raft of messy (expected) errors to stdout. + exit(1) + with open(config_file) as f: + yaml_dict =3D yaml.safe_load(f) + + if 'qemu-conf' in yaml_dict: + config.update(yaml_dict['qemu-conf']) + else: + raise Exception("config file {} is not valid"\ + " missing qemu-conf".format(config_file)) + return config + def parse_args(vmcls): =20 def get_default_jobs(): @@ -498,6 +532,9 @@ def parse_args(vmcls): help=3D"Interactively run command") parser.add_option("--snapshot", "-s", action=3D"store_true", help=3D"run tests with a snapshot") + parser.add_option("--config", "-c", default=3DNone, + help=3D"Provide config yaml for configuration. "\ + "See config_example.yaml for example.") parser.disable_interspersed_args() return parser.parse_args() =20 @@ -509,6 +546,7 @@ def main(vmcls, config=3DNone): if not argv and not args.build_qemu and not args.build_image: print("Nothing to do?") return 1 + config =3D parse_config(config, args) logging.basicConfig(level=3D(logging.DEBUG if args.debug else logging.WARN)) vm =3D vmcls(debug=3Dargs.debug, vcpus=3Dargs.jobs, config=3Dconfi= g) diff --git a/tests/vm/conf_example_aarch64.yml b/tests/vm/conf_example_aarc= h64.yml new file mode 100644 index 0000000000..9d44ae356f --- /dev/null +++ b/tests/vm/conf_example_aarch64.yml @@ -0,0 +1,51 @@ +# +# Example yaml for use by any of the scripts in tests/vm. +# Can be provided as an environment variable QEMU_CONFIG +# +qemu-conf: + + # If any of the below are not provided, we will just use the qemu defa= ults. + + # Login username and password(has to be sudo enabled) + guest_user: qemu + guest_pass: "qemupass" + + # Password for root user can be different from guest. + root_pass: "qemupass" + + # If one key is provided, both must be provided. + #ssh_key: /complete/path/of/your/keyfile/id_rsa + #ssh_pub_key: /complete/path/of/your/keyfile/id_rsa.pub + + cpu: max + machine: virt,gic-version=3Dmax + memory: 16G + + # The below is a example for how to configure NUMA topology with + # 4 NUMA nodes and 2 different NUMA distances. + qemu_args: "-smp cpus=3D16,sockets=3D2,cores=3D8 + -numa node,cpus=3D0-3,nodeid=3D0 -numa node,cpus=3D4-7,nod= eid=3D1 + -numa node,cpus=3D8-11,nodeid=3D2 -numa node,cpus=3D12-15,= nodeid=3D3 + -numa dist,src=3D0,dst=3D1,val=3D15 -numa dist,src=3D2,dst= =3D3,val=3D15 + -numa dist,src=3D0,dst=3D2,val=3D20 -numa dist,src=3D0,dst= =3D3,val=3D20 + -numa dist,src=3D1,dst=3D2,val=3D20 -numa dist,src=3D1,dst= =3D3,val=3D20" + + # By default we do not set the DNS. + # You override the defaults by setting the below. + #dns: 1.234.567.89 + + # By default we will use a "block" device, but + # you can also boot from a "scsi" device. + # Just keep in mind your scripts might need to change + # As you will have /dev/sda instead of /dev/vda (for block device) + boot_dev_type: "block" + + # By default the ssh port is not fixed. + # A fixed ssh port makes it easier for automated tests. + #ssh_port: 5555 + + # To install a different set of packages, provide a command to issue + #install_cmds: "apt-get update ; apt-get build-dep -y qemu" + + # Or to skip the install entirely, just provide "" + #install_cmds: "" diff --git a/tests/vm/conf_example_x86.yml b/tests/vm/conf_example_x86.yml new file mode 100644 index 0000000000..78d3f5830f --- /dev/null +++ b/tests/vm/conf_example_x86.yml @@ -0,0 +1,50 @@ +# +# Example yaml for use by any of the x86 based scripts in tests/vm. +# Can be provided as an environment variable QEMU_CONFIG +# +qemu-conf: + + # If any of the below are not provided, we will just use the qemu defa= ults. + + # Login username and password(has to be sudo enabled) + guest_user: "qemu" + guest_pass: "qemupass" + + # Password for root user can be different from guest. + root_pass: "qemupass" + + # Provide default ssh keys of current user. + # You need to edit the below for your user. + #ssh_key_file: /home//.ssh/id_rsa + #ssh_pub_key_file: /home//.ssh/id_rsa.pub + + cpu: max + machine: pc + memory: 8G + + # The below is a example for how to configure NUMA topology with + # 4 NUMA nodes and 2 different NUMA distances. + qemu_args: "-smp cpus=3D8,sockets=3D2,cores=3D4 + -object memory-backend-ram,size=3D4G,policy=3Dbind,host-no= des=3D0,id=3Dram-node0 + -object memory-backend-ram,size=3D4G,policy=3Dbind,host-no= des=3D0,id=3Dram-node1 + -object memory-backend-ram,size=3D4G,policy=3Dbind,host-no= des=3D1,id=3Dram-node2 + -object memory-backend-ram,size=3D4G,policy=3Dbind,host-no= des=3D1,id=3Dram-node3 + -numa node,cpus=3D0-1,nodeid=3D0 -numa node,cpus=3D2-3,nod= eid=3D1 + -numa node,cpus=3D4-5,nodeid=3D2 -numa node,cpus=3D6-7,nod= eid=3D3 + -numa dist,src=3D0,dst=3D1,val=3D15 -numa dist,src=3D2,dst= =3D3,val=3D15 + -numa dist,src=3D0,dst=3D2,val=3D20 -numa dist,src=3D0,dst= =3D3,val=3D20 + -numa dist,src=3D1,dst=3D2,val=3D20 -numa dist,src=3D1,dst= =3D3,val=3D20" + + # By default we do not set the DNS. + # You override the defaults by setting the below. + #dns: "1.234.567.89" + + # By default we will use a "block" device, but + # you can also boot from a "scsi" device. + # Just keep in mind your scripts might need to change + # As you will have /dev/sda instead of /dev/vda (for block device) + boot_dev_type: "block" + + # By default the ssh port is not fixed. + # A fixed ssh port makes it easier for automated tests. + ssh_port: 5555 --=20 2.17.1 From nobody Fri Nov 14 00:46:50 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=linaro.org ARC-Seal: i=1; a=rsa-sha256; t=1584023911; cv=none; d=zohomail.com; s=zohoarc; b=B4KC4yD/CTMhRlgzNXNeYX0Qy4MqZxF8NbD6lwnlq47yi2FNQbpiOiBFvaAYTsP7oDg6IiDY6UOcrVz+nJZrPo6ubnikmyxo3O9tXY+B9aIctS1tIBtzpS1rEn+7OOjGv0oP8W5Rc9aYmkvhTtr5vL4Hp/KZvb8xPnh48BPj9po= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1584023911; h=Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:Message-ID:References:Sender:Subject:To; bh=Kg9peQoC2sg52IZVPinKGbMeesGBXCKH6J+N63DQI98=; b=DXYVWPcVnUjBeZlKiWLo4Hv0Vlgn4iYFUakXie5UouiWD9kcYHvn6BvdIN5Mb9vBL/di98VPKqsKOmoa5JGtmrAXb+LTL88GvJdB2P+wIzFht7TGh7yv2nXnwbnSKH6JQDPNHykMEDJH99LpB4xoxAIIi4W/Dn2qSdApvACrRE4= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1584023911841759.0405901673012; Thu, 12 Mar 2020 07:38:31 -0700 (PDT) Received: from localhost ([::1]:42532 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jCOyk-0003I0-Ql for importer@patchew.org; Thu, 12 Mar 2020 10:38:30 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:35664) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jCOrC-0007Cc-PO for qemu-devel@nongnu.org; Thu, 12 Mar 2020 10:30:47 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1jCOrB-0006V8-Ei for qemu-devel@nongnu.org; Thu, 12 Mar 2020 10:30:42 -0400 Received: from mail-pg1-x536.google.com ([2607:f8b0:4864:20::536]:33049) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1jCOrB-0006Ug-8i for qemu-devel@nongnu.org; Thu, 12 Mar 2020 10:30:41 -0400 Received: by mail-pg1-x536.google.com with SMTP id m5so3181182pgg.0 for ; Thu, 12 Mar 2020 07:30:41 -0700 (PDT) Received: from Rfoley-MA01.hsd1.ma.comcast.net ([2601:199:4480:60c0:845e:b9f6:81a6:8f5e]) by smtp.gmail.com with ESMTPSA id 63sm14832651pfx.132.2020.03.12.07.30.38 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 12 Mar 2020 07:30:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=Kg9peQoC2sg52IZVPinKGbMeesGBXCKH6J+N63DQI98=; b=IoHMkWn41ef7kLpALelhVQFIr///zjCkq9DW4VuiBLnY/7XyjVca5xJWIjNJSorfja go02LBeI/WFIg8fNltG35vTGmbwka2XnMf2G2Ak+PB4uW8Y8I8xQkM/SkZnYLKuYpZoN 3dYfKtoVcQK57A/GHGDuRVJ7JUgfgIFlzlq/sFXaGYcnRvjbXx5TPgqBwztXqKbeAzkU gBvlJY4oRzdEfRuHA3vZ2vWSX91Cv9ontzwL0vGsgecAHQSaZMrd5JilOW71WwuGmDwq 6A5nvgZeOiPccA1JqAaSKFq6XxJKEcBl4+km7gjA3wShSNtL2dsfcjF5W0IEbv6VHuFf SjWQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=Kg9peQoC2sg52IZVPinKGbMeesGBXCKH6J+N63DQI98=; b=kwrtbo/SM4pIZNc2JvZFUvCNWZ3c6RSFtKpetCbuYOuKyirv6mVXmyjF9lhqgnBbG4 pqIiiIgUIoQif4Aj3rCjidbtWLbICxQcJg7opAlH0iLbho6E4AIV9ecVVO5TO1ZBuJc1 oHk1z4ltEDM5oGp7t8giHtq2qIvH+AP0cLB0WkGu8414qeAkNr5PWIqF0wWUrAGb0pM7 FWPs4y2F4lxtkLG9FIWgyFHuLu/lIiSzcNoKq5kUdWWQkj6w5iADbhTnuetvHMKO/NHv yrvFzvXhk8rSXDlIRoLCF982bD7UGNXjgkRkGZ3Z/i5iZFve3Izqv/hW79USwQcrWKrX hiRg== X-Gm-Message-State: ANhLgQ26MsPb3ALaMs5QXxhbJUWOUJEIlM8YwW/gGEWKE15yrOAmWKHv 1TshqHllPXvUrqEkncjIvRlTK9AHAP0= X-Google-Smtp-Source: ADFU+vvBSrr0+50YAKeTo2ApxdMcWSvfSm5B4wJ9+dvUPLppJ6YbT0jJjhqGe1/UmaAAjzuh4LOOOg== X-Received: by 2002:a05:6a00:48:: with SMTP id i8mr6714715pfk.20.1584023439896; Thu, 12 Mar 2020 07:30:39 -0700 (PDT) From: Robert Foley To: qemu-devel@nongnu.org Subject: [PATCH v4 04/10] tests/vm: add --boot-console switch Date: Thu, 12 Mar 2020 10:27:22 -0400 Message-Id: <20200312142728.12285-5-robert.foley@linaro.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20200312142728.12285-1-robert.foley@linaro.org> References: <20200312142728.12285-1-robert.foley@linaro.org> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:4864:20::536 X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: philmd@redhat.com, alex.bennee@linaro.org, robert.foley@linaro.org, peter.puhov@linaro.org Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Added ability to view console during boot via --boot-console switch to basevm.py. This helps debug issues that occur during the boot sequence. Also added a new special variable to vm-build: BOOT_CONSOLE=3D1 will cause this new --boot-console switch to be set. Signed-off-by: Robert Foley --- tests/vm/Makefile.include | 4 ++++ tests/vm/basevm.py | 11 +++++++++-- 2 files changed, 13 insertions(+), 2 deletions(-) diff --git a/tests/vm/Makefile.include b/tests/vm/Makefile.include index 78a2de1f82..d921ee14cf 100644 --- a/tests/vm/Makefile.include +++ b/tests/vm/Makefile.include @@ -40,6 +40,7 @@ endif @echo ' EXTRA_CONFIGURE_OPTS=3D"..."' @echo " J=3D[0..9]* - Override the -jN parameter for make= commands" @echo " DEBUG=3D1 - Enable verbose output on host and i= nteractive debugging" + @echo " BOOT_CONSOLE=3D1 - Show the console output at boot ti= me. " @echo " V=3D1 - Enable verbose ouput on host and guest commands" @echo " QEMU=3D/path/to/qemu - Change path to QEMU binary" @echo " QEMU_IMG=3D/path/to/qemu-img - Change path to qemu-img tool" @@ -62,6 +63,7 @@ $(IMAGES_DIR)/%.img: $(SRC_PATH)/tests/vm/% \ $(call quiet-command, \ $(PYTHON) $< \ $(if $(V)$(DEBUG), --debug) \ + $(if $(BOOT_CONSOLE),--boot-console) \ --image "$@" \ --force \ --build-image $@, \ @@ -76,6 +78,7 @@ vm-build-%: $(IMAGES_DIR)/%.img $(if $(DEBUG), --interactive) \ $(if $(J),--jobs $(J)) \ $(if $(V),--verbose) \ + $(if $(BOOT_CONSOLE),--boot-console) \ --image "$<" \ $(if $(BUILD_TARGET),--build-target $(BUILD_TARGET)) \ --snapshot \ @@ -96,6 +99,7 @@ vm-boot-ssh-%: $(IMAGES_DIR)/%.img $(call quiet-command, \ $(PYTHON) $(SRC_PATH)/tests/vm/$* \ $(if $(J),--jobs $(J)) \ + $(if $(BOOT_CONSOLE),--boot-console) \ --image "$<" \ --interactive \ false, \ diff --git a/tests/vm/basevm.py b/tests/vm/basevm.py index dd545d3d1d..aab3d98edf 100644 --- a/tests/vm/basevm.py +++ b/tests/vm/basevm.py @@ -85,8 +85,10 @@ class BaseVM(object): # 4 is arbitrary, but greater than 2, # since we found we need to wait more than twice as long. tcg_ssh_timeout_multiplier =3D 4 - def __init__(self, debug=3DFalse, vcpus=3DNone, config=3DNone): + def __init__(self, debug=3DFalse, vcpus=3DNone, config=3DNone, + boot_console=3DNone): self._guest =3D None + self._boot_console =3D boot_console # Allow input config to override defaults. self._config =3D DEFAULT_CONFIG.copy() if config !=3D None: @@ -535,6 +537,8 @@ def parse_args(vmcls): parser.add_option("--config", "-c", default=3DNone, help=3D"Provide config yaml for configuration. "\ "See config_example.yaml for example.") + parser.add_option("--boot-console", action=3D"store_true", + help=3D"Show console during boot. ") parser.disable_interspersed_args() return parser.parse_args() =20 @@ -549,7 +553,8 @@ def main(vmcls, config=3DNone): config =3D parse_config(config, args) logging.basicConfig(level=3D(logging.DEBUG if args.debug else logging.WARN)) - vm =3D vmcls(debug=3Dargs.debug, vcpus=3Dargs.jobs, config=3Dconfi= g) + vm =3D vmcls(debug=3Dargs.debug, vcpus=3Dargs.jobs, config=3Dconfi= g, + boot_console=3Dargs.boot_console) if args.build_image: if os.path.exists(args.image) and not args.force: sys.stderr.writelines(["Image file exists: %s\n" % args.im= age, @@ -569,6 +574,8 @@ def main(vmcls, config=3DNone): if args.snapshot: img +=3D ",snapshot=3Don" vm.boot(img) + if vm._boot_console: + vm.wait_boot() vm.wait_ssh() except Exception as e: if isinstance(e, SystemExit) and e.code =3D=3D 0: --=20 2.17.1 From nobody Fri Nov 14 00:46:50 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=linaro.org ARC-Seal: i=1; a=rsa-sha256; t=1584023552; cv=none; d=zohomail.com; s=zohoarc; b=NbYS6Hcfe3O+OvBe1GrUiuJiOJhtqgCHC/YxLxAbpnToSa6Z52Krc/ExWsS9KmxTOIBwjPiliqaVhVpuCzTPojBWdkUsjYPgAElPl7ZNRn+st8XYgJxv4mKsEKffvoz0hP549OKxtcAE5xLzAa131mKQmXiuMeG1GUqcc3ONmoo= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1584023552; h=Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:Message-ID:References:Sender:Subject:To; bh=YcJtreTuJf2SK2XtbK4xe8EoCvPleFPj5jx6jA6C+SI=; b=Q0NWjfmWqZdvX/8G2UV+I4nmOSdEewxsiBG6Lwt9EcvkYcrawxQZB+9jyOdQc593oZ2d77JcAltfNfFI4tg8kLES9cI3IQWvYRzxv6kVnUJa1ICWgG1R6RRNpGj3r/iF8zWZZU7yyUuMxbmpliK1PU7K+UR/JvDK/CzTXuxEVhM= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1584023552412371.90503141132547; Thu, 12 Mar 2020 07:32:32 -0700 (PDT) Received: from localhost ([::1]:42326 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jCOsx-0000Uw-8e for importer@patchew.org; Thu, 12 Mar 2020 10:32:31 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:35693) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jCOrJ-0007Ie-Cw for qemu-devel@nongnu.org; Thu, 12 Mar 2020 10:30:50 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1jCOrI-0006Y4-0J for qemu-devel@nongnu.org; Thu, 12 Mar 2020 10:30:49 -0400 Received: from mail-pf1-x433.google.com ([2607:f8b0:4864:20::433]:34143) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1jCOrH-0006Xo-Pm for qemu-devel@nongnu.org; Thu, 12 Mar 2020 10:30:47 -0400 Received: by mail-pf1-x433.google.com with SMTP id 23so3372940pfj.1 for ; Thu, 12 Mar 2020 07:30:47 -0700 (PDT) Received: from Rfoley-MA01.hsd1.ma.comcast.net ([2601:199:4480:60c0:845e:b9f6:81a6:8f5e]) by smtp.gmail.com with ESMTPSA id 63sm14832651pfx.132.2020.03.12.07.30.40 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 12 Mar 2020 07:30:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=YcJtreTuJf2SK2XtbK4xe8EoCvPleFPj5jx6jA6C+SI=; b=HCjXo7OM3TsJd/RzpatmNGnyd6EHNKZYLx84oF+ghnMIIegQr6lnJdOgudJ0pVZCHG pesKRxODpJo5V3AC468zUfNrC26qskBT2OKLQmvm6ezkA8xm7tdwE2mQ1XSs8JFU+cTu s3iXo/d3Gj0bBC0GDPAzg7nsTCL/pPydYfF6fTLAm+wu+nIY1AaNWngomX5+XkKvOr5P wcE/LXDFev7dUhWEjtsAUAmyZy6HO2stjF+YJm/1e8X/wcHX3cQtWLDqyuPhgWd0wNlf de/FwdeGk8Vk1vF/sd+p/98ycvw/CoD9Bgal9Oy87pv7oAM/33Ar+h1vRnWdP+3ptTi8 yQSA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=YcJtreTuJf2SK2XtbK4xe8EoCvPleFPj5jx6jA6C+SI=; b=OwAgLOGCbg5EiFsVvMMeq6rQ5oPiVDDoFIph1f/rm4FG3xkmBC1cfiZO8xoRKIOfLg Btj9jzmUj5UqSo9P9CkqP5ndzedH8e2rOUMBsmdHMY4ev7fCJXlrjt9FOk0vdWO0uEBf GOPxKpE5/S6OsiL+foffE1B5MFDNbXifpzC4zSKDQQOMVusX5/DHO5+0xAEhknZmj41M CVnTh8WVLaYhNEyIm5QYYlrmcejZCbFDQpg+haxqCAp6HFQNjd70JaZu3zQNvGVRiUc9 jQL0ii4SLkzpaImnrdZmuwEUMAvCtAkWKTfovS4R2cFkbwKlOF7tt3vTvqy96tvmcYGf +WBw== X-Gm-Message-State: ANhLgQ30pk6sICJl3tv9G3vdvEQDpmhiOePzraWSbIuZ7fvibf2qzTN0 UR4KlCql6W/oWG6CEWKKEVTo9i5iSKE= X-Google-Smtp-Source: ADFU+vuGm7UJzYliZTAYgUFXNmWV67X2YNxQKid59IIWn2Atx1+U56XSiWjudWakd3+ByomRRBK89A== X-Received: by 2002:aa7:8055:: with SMTP id y21mr8298190pfm.151.1584023441664; Thu, 12 Mar 2020 07:30:41 -0700 (PDT) From: Robert Foley To: qemu-devel@nongnu.org Subject: [PATCH v4 05/10] tests/vm: Add ability to select QEMU from current build. Date: Thu, 12 Mar 2020 10:27:23 -0400 Message-Id: <20200312142728.12285-6-robert.foley@linaro.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20200312142728.12285-1-robert.foley@linaro.org> References: <20200312142728.12285-1-robert.foley@linaro.org> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:4864:20::433 X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: philmd@redhat.com, alex.bennee@linaro.org, robert.foley@linaro.org, peter.puhov@linaro.org Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Added a new special variable QEMU_LOCAL=3D1, which will indicate to take the QEMU binary from the current build. Signed-off-by: Robert Foley --- tests/vm/Makefile.include | 4 ++++ tests/vm/basevm.py | 28 ++++++++++++++++++++++++---- 2 files changed, 28 insertions(+), 4 deletions(-) diff --git a/tests/vm/Makefile.include b/tests/vm/Makefile.include index d921ee14cf..667705a91a 100644 --- a/tests/vm/Makefile.include +++ b/tests/vm/Makefile.include @@ -42,6 +42,7 @@ endif @echo " DEBUG=3D1 - Enable verbose output on host and i= nteractive debugging" @echo " BOOT_CONSOLE=3D1 - Show the console output at boot ti= me. " @echo " V=3D1 - Enable verbose ouput on host and guest commands" + @echo " QEMU_LOCAL=3D1 - Use QEMU binary local to this= build." @echo " QEMU=3D/path/to/qemu - Change path to QEMU binary" @echo " QEMU_IMG=3D/path/to/qemu-img - Change path to qemu-img tool" ifeq ($(PYTHON_YAML),yes) @@ -64,6 +65,7 @@ $(IMAGES_DIR)/%.img: $(SRC_PATH)/tests/vm/% \ $(PYTHON) $< \ $(if $(V)$(DEBUG), --debug) \ $(if $(BOOT_CONSOLE),--boot-console) \ + --build-path $(BUILD_DIR)\ --image "$@" \ --force \ --build-image $@, \ @@ -79,6 +81,7 @@ vm-build-%: $(IMAGES_DIR)/%.img $(if $(J),--jobs $(J)) \ $(if $(V),--verbose) \ $(if $(BOOT_CONSOLE),--boot-console) \ + --build-path $(BUILD_DIR)\ --image "$<" \ $(if $(BUILD_TARGET),--build-target $(BUILD_TARGET)) \ --snapshot \ @@ -100,6 +103,7 @@ vm-boot-ssh-%: $(IMAGES_DIR)/%.img $(PYTHON) $(SRC_PATH)/tests/vm/$* \ $(if $(J),--jobs $(J)) \ $(if $(BOOT_CONSOLE),--boot-console) \ + --build-path $(BUILD_DIR)\ --image "$<" \ --interactive \ false, \ diff --git a/tests/vm/basevm.py b/tests/vm/basevm.py index aab3d98edf..3562a33ffa 100644 --- a/tests/vm/basevm.py +++ b/tests/vm/basevm.py @@ -86,8 +86,9 @@ class BaseVM(object): # since we found we need to wait more than twice as long. tcg_ssh_timeout_multiplier =3D 4 def __init__(self, debug=3DFalse, vcpus=3DNone, config=3DNone, - boot_console=3DNone): + boot_console=3DFalse, build_path=3DNone): self._guest =3D None + self._build_path =3D build_path self._boot_console =3D boot_console # Allow input config to override defaults. self._config =3D DEFAULT_CONFIG.copy() @@ -264,8 +265,8 @@ class BaseVM(object): args +=3D self._data_args + extra_args + self._config['extra_args'] args +=3D ["-device", "VGA"] logging.debug("QEMU args: %s", " ".join(args)) - qemu_bin =3D os.environ.get("QEMU", "qemu-system-" + self.arch) - guest =3D QEMUMachine(binary=3Dqemu_bin, args=3Dargs) + qemu_path =3D get_qemu_path(self.arch, self._build_path) + guest =3D QEMUMachine(binary=3Dqemu_path, args=3Dargs) guest.set_machine(self._config['machine']) guest.set_console() try: @@ -465,6 +466,22 @@ class BaseVM(object): stderr=3Dself._stdout) return os.path.join(cidir, "cloud-init.iso") =20 +def get_qemu_path(arch, build_path=3DNone): + """Fetch the path to the qemu binary.""" + qemu_local =3D os.environ.get("QEMU_LOCAL", 0) + # If QEMU environment variable set, it takes precedence + if "QEMU" in os.environ: + qemu_path =3D os.environ["QEMU"] + elif qemu_local: + if not build_path: + raise Exception("--build-path option required with QEMU_LOCAL") + qemu_path =3D os.path.join(build_path, arch + "-softmmu") + qemu_path =3D os.path.join(qemu_path, "qemu-system-" + arch) + else: + # Default is to use system path for qemu. + qemu_path =3D "qemu-system-" + arch + return qemu_path + def parse_config(config, args): """ Parse yaml config and populate our config structure. The yaml config allows the user to override the @@ -539,6 +556,8 @@ def parse_args(vmcls): "See config_example.yaml for example.") parser.add_option("--boot-console", action=3D"store_true", help=3D"Show console during boot. ") + parser.add_option("--build-path", default=3DNone, + help=3D"Path of build directory. ") parser.disable_interspersed_args() return parser.parse_args() =20 @@ -554,7 +573,8 @@ def main(vmcls, config=3DNone): logging.basicConfig(level=3D(logging.DEBUG if args.debug else logging.WARN)) vm =3D vmcls(debug=3Dargs.debug, vcpus=3Dargs.jobs, config=3Dconfi= g, - boot_console=3Dargs.boot_console) + boot_console=3Dargs.boot_console, + build_path=3Dargs.build_path) if args.build_image: if os.path.exists(args.image) and not args.force: sys.stderr.writelines(["Image file exists: %s\n" % args.im= age, --=20 2.17.1 From nobody Fri Nov 14 00:46:50 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=linaro.org ARC-Seal: i=1; a=rsa-sha256; t=1584023673; cv=none; d=zohomail.com; s=zohoarc; b=YAF5GeyExlhOLG3Bqvqtr3n9qKKQ9pQEYmQyYwJRiUTtWMTynCkbsaN/F19ZilJhvthUlrkJTQ1UnAzh8VBx8p7Kp6sOUztGzjsFFn1X9haWlaw8F6Xpq8KaRbnGJjHZa/+TR0BkYk34sYHa5Ee2rkY9cWBJno6hSLnDU+j/Y7k= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1584023673; h=Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:Message-ID:References:Sender:Subject:To; bh=8MBax5bvqnGFn59+X+0C/P+ZVgi3X2sGtV0kV14mTCU=; b=YKqqr7av7NWxrRHd/AS/lhOjx5yv3VZQ/A38rP/o+xybOuZz+6CWvWCn2srmHOjdAMUF09QYl5KsOwIa0t7nUNhogez+TBv9Vh7MsoMT2au1C2ZhvWBd5PoxkHsSiNEGj+2sq/qM65MSaKqIqC5/xqSdWCo4CFXGWLsEAr93Svo= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1584023673261495.9776094511968; Thu, 12 Mar 2020 07:34:33 -0700 (PDT) Received: from localhost ([::1]:42382 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jCOuu-0004um-4a for importer@patchew.org; Thu, 12 Mar 2020 10:34:32 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:35697) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jCOrJ-0007Ig-Qb for qemu-devel@nongnu.org; Thu, 12 Mar 2020 10:30:54 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1jCOrE-0006X7-Vp for qemu-devel@nongnu.org; Thu, 12 Mar 2020 10:30:49 -0400 Received: from mail-pl1-x633.google.com ([2607:f8b0:4864:20::633]:44035) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1jCOrE-0006Wb-QC for qemu-devel@nongnu.org; Thu, 12 Mar 2020 10:30:44 -0400 Received: by mail-pl1-x633.google.com with SMTP id d9so2723175plo.11 for ; Thu, 12 Mar 2020 07:30:44 -0700 (PDT) Received: from Rfoley-MA01.hsd1.ma.comcast.net ([2601:199:4480:60c0:845e:b9f6:81a6:8f5e]) by smtp.gmail.com with ESMTPSA id 63sm14832651pfx.132.2020.03.12.07.30.41 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 12 Mar 2020 07:30:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=8MBax5bvqnGFn59+X+0C/P+ZVgi3X2sGtV0kV14mTCU=; b=J06cYGm3233jvXeQUOsSAkPSWCP9g6/W9r8rJPtGKgW8HSC0UJJ69/ezmPCpWb1mCW lj6cJgOU+ZTpt0xylcPUQw7BtvuOgRuGCNCrUvmYpU/1LkLPsQ4PplbfYwhs2npYtVpB tfU1poP61yi6Ik7alJQmCgH6ImjUBaHzkaekDUQHYovNScNs4lm9PK5gGF5rlB5AWG+e lvD/yJyz4YW5Xc+K9874vRsg2Vl61yrF4CjLegqNDZHreY60OkEGXsCEQ7RIqtQb/2vS STNGO7oMCHyrQ4/8sApQ5Lqyi7+eRw1sIxb+u9x7LykLWXRDUOJlxNthXf+B5IY2liNv JYKA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=8MBax5bvqnGFn59+X+0C/P+ZVgi3X2sGtV0kV14mTCU=; b=dakFVDTHvFmEbIIzgeudmtch17GclsmJ1MHloah6+pxd2V8l62bvWt3DAmUKDwcC8L GcX0jNBV6+PHAZaJYjZHYDEqOw+5i3JstPfq2iKwc+hdV7GasmxWO2KUtvkEGZhH4Mn9 ItU8jWHeNC7fZQGvTX2chGhK5PuqLtGPpzwzVKTnp3+3OuMe9dWqpe7+RaY+4x4/NUgE Uj10Jgux/5R8Lv7rZzMwsvOLotTCOksRb3ieFeVAY73YHBuPRfZQrXCvmzzyBswqqKde Dece0sZYV4YDihlAfapnCRQGU+RMRZXYxLN8BqLsWLGnFUBe3yIIQ+1nQ+Q9To7FSSTP Qcig== X-Gm-Message-State: ANhLgQ0tWD13r50yC/FY8QENDDpUFfwUQ/LIWC9rvJL252uxLv5J/FC0 EWUGO59nKjO0Gdz8NF+ZX4LHKhZC1H0= X-Google-Smtp-Source: ADFU+vu2iqanlfiAPx1vh89F01EoaI3xidL4bcWHHnAGk/uQfPJdXpBueHeY6RcabMll8QPhrVKt1w== X-Received: by 2002:a17:902:9b8a:: with SMTP id y10mr7923687plp.114.1584023443479; Thu, 12 Mar 2020 07:30:43 -0700 (PDT) From: Robert Foley To: qemu-devel@nongnu.org Subject: [PATCH v4 06/10] tests/vm: allow wait_ssh() to specify command Date: Thu, 12 Mar 2020 10:27:24 -0400 Message-Id: <20200312142728.12285-7-robert.foley@linaro.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20200312142728.12285-1-robert.foley@linaro.org> References: <20200312142728.12285-1-robert.foley@linaro.org> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:4864:20::633 X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: philmd@redhat.com, alex.bennee@linaro.org, robert.foley@linaro.org, peter.puhov@linaro.org Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" This allows for waiting for completion of arbitrary commands. Signed-off-by: Robert Foley Reviewed-by: Alex Benn=C3=A9e --- tests/vm/basevm.py | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/tests/vm/basevm.py b/tests/vm/basevm.py index 3562a33ffa..305b839000 100644 --- a/tests/vm/basevm.py +++ b/tests/vm/basevm.py @@ -394,24 +394,24 @@ class BaseVM(object): def print_step(self, text): sys.stderr.write("### %s ...\n" % text) =20 - def wait_ssh(self, wait_root=3DFalse, seconds=3D300): + def wait_ssh(self, wait_root=3DFalse, seconds=3D300, cmd=3D"exit 0"): # Allow more time for VM to boot under TCG. if not kvm_available(self.arch): seconds *=3D self.tcg_ssh_timeout_multiplier starttime =3D datetime.datetime.now() endtime =3D starttime + datetime.timedelta(seconds=3Dseconds) - guest_up =3D False + cmd_success =3D False while datetime.datetime.now() < endtime: - if wait_root and self.ssh_root("exit 0") =3D=3D 0: - guest_up =3D True + if wait_root and self.ssh_root(cmd) =3D=3D 0: + cmd_success =3D True break - elif self.ssh("exit 0") =3D=3D 0: - guest_up =3D True + elif self.ssh(cmd) =3D=3D 0: + cmd_success =3D True break seconds =3D (endtime - datetime.datetime.now()).total_seconds() logging.debug("%ds before timeout", seconds) time.sleep(1) - if not guest_up: + if not cmd_success: raise Exception("Timeout while waiting for guest ssh") =20 def shutdown(self): --=20 2.17.1 From nobody Fri Nov 14 00:46:50 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=linaro.org ARC-Seal: i=1; a=rsa-sha256; t=1584023553; cv=none; d=zohomail.com; s=zohoarc; b=BKGBlWOZlNg+cSO2xykOqByENedf5IgC7okuFPbcZu+kvwU2f4q14zmCFElG5mSBw35y8ZnLUhJQlW4XYvaEfyZKokNIJwZ3lm7BF7kW5KV6EP/H/Ia6qQ6v5BhzJjvGvJeZzRuhN8ddXKzoGiZ91Cy+daFLwfTGk96NVzkcg+4= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1584023553; h=Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:Message-ID:References:Sender:Subject:To; bh=r4N6YwCj4u4jdZ+SFqfYJDZRpWJ35tghF9+9S271aaA=; b=gIb5RRkfQX1sz9IKksgz3xpWsS7mB6ueQqtIBUY5Skug0q/fKi9TnXn9i/VTU/jYcfUi8rY8Xwlp7ZxLWHSbEdvd6ECQZEYamUMMe6esmq5KysEjCGadfDqXsqJVHYfYmPB2Z+wULVzOiBlHgE6qoxWhZiciMRwWOrKyL0f+xOQ= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1584023553633253.71596265930089; Thu, 12 Mar 2020 07:32:33 -0700 (PDT) Received: from localhost ([::1]:42332 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jCOsy-0000Ys-GL for importer@patchew.org; Thu, 12 Mar 2020 10:32:32 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:35711) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jCOrK-0007JY-Qu for qemu-devel@nongnu.org; Thu, 12 Mar 2020 10:30:53 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1jCOrI-0006YN-Ks for qemu-devel@nongnu.org; Thu, 12 Mar 2020 10:30:50 -0400 Received: from mail-pj1-x1032.google.com ([2607:f8b0:4864:20::1032]:37693) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1jCOrI-0006Xd-Bj for qemu-devel@nongnu.org; Thu, 12 Mar 2020 10:30:48 -0400 Received: by mail-pj1-x1032.google.com with SMTP id ca13so2737788pjb.2 for ; Thu, 12 Mar 2020 07:30:48 -0700 (PDT) Received: from Rfoley-MA01.hsd1.ma.comcast.net ([2601:199:4480:60c0:845e:b9f6:81a6:8f5e]) by smtp.gmail.com with ESMTPSA id 63sm14832651pfx.132.2020.03.12.07.30.43 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 12 Mar 2020 07:30:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=r4N6YwCj4u4jdZ+SFqfYJDZRpWJ35tghF9+9S271aaA=; b=mO0n/vRtHPWMVpMD/1C2vb6HUpAjpz6go843+pAVN0Nc+T3k8jj35KzDyANdIl1LrV zAFGlGkntRAgqg0PT93V42Q+Pe+GTKtN/hGuNlj/zd5jKSydr1sP1olAnu4oYhhoGQjI vul+j0bBpW2iVr3Mnx0mkvuQKm6id06nUWik3j4YqRtJKoadBqApWt3CZdXpKIj3Dkh1 2Gi0uUJRe6ebYB9fvMcVVIjLYPfv039wOOL1lYBYuw/NPkB3SaWZdnfeu5YMP0HM2zqe mVSbi8/CixvVcf/rfNornUVVcjI8wZurnGG/FZP4jF0qgVYi5WS9z1anQhJDAw39AzCo 5znA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=r4N6YwCj4u4jdZ+SFqfYJDZRpWJ35tghF9+9S271aaA=; b=iVM+ZYQ+Xhq7a2yUs9LqQEdwVTmf3oy/XtVn/cKqAb/1DPHTZUaBrWj1CuEG7ADAa0 HIBvW8DJNlAU77SJxSQ1Yssc1pEW+TgwfLXYJ0jGcIyGYQbYUBoxYqMlrbJMwuS6Zev0 NZ5LNGMogsSdas68q3IwAaCfekKX1IArjM0jaQt317O+NB4p0zh2hpzip8xqC8jdJjEH x4yvZqaaKbERClnaLFzgmFkrzhIkctCkPGSpLt/Cu81/62ZAhjHFJJGc8oYaIHe8/aKk VBPYpi6EfKkIDkWxLPTu9o92svZOBs6+WjAGigy4ROBQIqE1OCpqFWKtlekAzupDrJ3Q 6X+A== X-Gm-Message-State: ANhLgQ2YufxA7T3fwX93LBrdMJmbsOzSeApZaUo9J7OA/PrVirly/Sjp dN6QzWGylaGEFWnK49tcUE7BkciR/6U= X-Google-Smtp-Source: ADFU+vtvVxa652ppJTCARsjkvo6nqTUZTbNQzPyjgoa/C/Ek19JKR1AIfcOqX33fxj7HvOVzEpOcDQ== X-Received: by 2002:a17:90a:d585:: with SMTP id v5mr4482258pju.4.1584023445207; Thu, 12 Mar 2020 07:30:45 -0700 (PDT) From: Robert Foley To: qemu-devel@nongnu.org Subject: [PATCH v4 07/10] tests/vm: Added a new script for ubuntu.aarch64. Date: Thu, 12 Mar 2020 10:27:25 -0400 Message-Id: <20200312142728.12285-8-robert.foley@linaro.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20200312142728.12285-1-robert.foley@linaro.org> References: <20200312142728.12285-1-robert.foley@linaro.org> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:4864:20::1032 X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: philmd@redhat.com, alex.bennee@linaro.org, robert.foley@linaro.org, peter.puhov@linaro.org Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" ubuntu.aarch64 provides a script to create an Ubuntu 18.04 VM. Another new file is also added aarch64vm.py, which is a module with common methods used by aarch64 VMs, such as how to create the flash images. Signed-off-by: Robert Foley --- configure | 16 ++++++ tests/vm/Makefile.include | 8 +++ tests/vm/aarch64vm.py | 100 ++++++++++++++++++++++++++++++++ tests/vm/basevm.py | 8 +++ tests/vm/ubuntu.aarch64 | 117 ++++++++++++++++++++++++++++++++++++++ 5 files changed, 249 insertions(+) create mode 100644 tests/vm/aarch64vm.py create mode 100755 tests/vm/ubuntu.aarch64 diff --git a/configure b/configure index 739cde2318..81525bbf20 100755 --- a/configure +++ b/configure @@ -947,6 +947,20 @@ do fi done =20 +# Check for files needed by aarch64 VMs. +# Allow user to override the path for efi also. +qemu_efi_aarch64=3D +for fd in $QEMU_EFI_PATH /usr/share/qemu-efi-aarch64/QEMU_EFI.fd +do + if has $fd + then + if [ -f $fd ]; then + qemu_efi_aarch64=3D$fd + fi + break + fi +done + # Check for existence of python3 yaml, needed to # import yaml config files into vm-build. python_yaml=3D"no" @@ -6591,6 +6605,7 @@ if test "$docs" !=3D "no"; then echo "sphinx-build $sphinx_build" fi echo "genisoimage $genisoimage" +echo "qemu_efi_aarch64 $qemu_efi_aarch64" echo "python_yaml $python_yaml" echo "slirp support $slirp $(echo_version $slirp $slirp_version)" if test "$slirp" !=3D "no" ; then @@ -7642,6 +7657,7 @@ echo "INSTALL_LIB=3D$install -c -m 0644" >> $config_h= ost_mak echo "PYTHON=3D$python" >> $config_host_mak echo "SPHINX_BUILD=3D$sphinx_build" >> $config_host_mak echo "GENISOIMAGE=3D$genisoimage" >> $config_host_mak +echo "QEMU_EFI_AARCH64=3D$qemu_efi_aarch64" >> $config_host_mak echo "PYTHON_YAML=3D$python_yaml" >> $config_host_mak echo "CC=3D$cc" >> $config_host_mak if $iasl -h > /dev/null 2>&1; then diff --git a/tests/vm/Makefile.include b/tests/vm/Makefile.include index 667705a91a..9934008bfa 100644 --- a/tests/vm/Makefile.include +++ b/tests/vm/Makefile.include @@ -5,6 +5,9 @@ IMAGES :=3D freebsd netbsd openbsd centos fedora ifneq ($(GENISOIMAGE),) IMAGES +=3D ubuntu.i386 centos +ifneq ($(QEMU_EFI_AARCH64),) +IMAGES +=3D ubuntu.aarch64 +endif endif =20 IMAGES_DIR :=3D $(HOME)/.cache/qemu-vm/images @@ -23,6 +26,11 @@ vm-help vm-test: ifneq ($(GENISOIMAGE),) @echo " vm-build-centos - Build QEMU in CentOS VM, with = Docker" @echo " vm-build-ubuntu.i386 - Build QEMU in ubuntu i386 VM" +ifneq ($(QEMU_EFI_AARCH64),) + @echo " vm-build-ubuntu.aarch64 - Build QEMU in ubuntu aarch64 V= M" +else + @echo " (install qemu-efi-aarch64 to build centos/ubuntu aarch64 images.= )" +endif else @echo " (install genisoimage to build centos/ubuntu images)" endif diff --git a/tests/vm/aarch64vm.py b/tests/vm/aarch64vm.py new file mode 100644 index 0000000000..646dc975f2 --- /dev/null +++ b/tests/vm/aarch64vm.py @@ -0,0 +1,100 @@ +#!/usr/bin/env python3 +# +# VM testing aarch64 library +# +# Copyright 2020 Linaro +# +# Authors: +# Robert Foley +# +# This code is licensed under the GPL version 2 or later. See +# the COPYING file in the top-level directory. +# +import os +import sys +import subprocess +import basevm +from qemu.accel import kvm_available + +# This is the config needed for current version of QEMU. +# This works for both kvm and tcg. +CURRENT_CONFIG =3D { + 'cpu' : "max", + 'machine' : "virt,gic-version=3Dmax", +} + +# The minimum minor version of QEMU we will support with aarch64 VMs is 3. +# QEMU versions less than 3 have various issues running these VMs. +QEMU_AARCH64_MIN_VERSION =3D 3 + +# The DEFAULT_CONFIG will default to a version of +# parameters that works for backwards compatibility. +DEFAULT_CONFIG =3D {'kvm' : {'cpu' : "host", + 'machine' : "virt,gic-version=3Dhost"}, + 'tcg' : {'cpu' : "cortex-a57", + 'machine' : "virt"}, +} + +def get_config_defaults(vmcls, default_config): + """Fetch the configuration defaults for this VM, + taking into consideration the defaults for + aarch64 first, followed by the defaults for this VM.""" + config =3D default_config + config.update(aarch_get_config_defaults(vmcls)) + return config + +def aarch_get_config_defaults(vmcls): + """Set the defaults for current version of QEMU.""" + config =3D CURRENT_CONFIG + args, argv =3D basevm.parse_args(vmcls) + qemu_path =3D basevm.get_qemu_path(vmcls.arch, args.build_path) + qemu_version =3D basevm.get_qemu_version(qemu_path) + if qemu_version < QEMU_AARCH64_MIN_VERSION: + error =3D "\nThis major version of QEMU {} is to old for aarch64 V= Ms.\n"\ + "The major version must be at least {}.\n"\ + "To continue with the current build of QEMU, "\ + "please restart with QEMU_LOCAL=3D1 .\n" + print(error.format(qemu_version, QEMU_AARCH64_MIN_VERSION)) + exit(1) + if qemu_version =3D=3D QEMU_AARCH64_MIN_VERSION: + # We have an older version of QEMU, + # set the config values for backwards compatibility. + if kvm_available('aarch64'): + config.update(DEFAULT_CONFIG['kvm']) + else: + config.update(DEFAULT_CONFIG['tcg']) + return config + +def create_flash_images(flash_dir=3D"./"): + """Creates the appropriate pflash files + for an aarch64 VM.""" + flash0_path =3D get_flash_path(flash_dir, "flash0") + flash1_path =3D get_flash_path(flash_dir, "flash1") + subprocess.check_call(["dd", "if=3D/dev/zero", "of=3D{}".format(flash0= _path), + "bs=3D1M", "count=3D64"]) + # A reliable way to get the QEMU EFI image is via an installed package. + efi_img =3D "/usr/share/qemu-efi-aarch64/QEMU_EFI.fd" + if not os.path.exists(efi_img): + sys.stderr.write("*** {} is missing\n".format(efi_img)) + sys.stderr.write("*** please install qemu-efi-aarch64 package\n") + exit(3) + subprocess.check_call(["dd", "if=3D{}".format(efi_img), + "of=3D{}".format(flash0_path), + "conv=3Dnotrunc"]) + subprocess.check_call(["dd", "if=3D/dev/zero", + "of=3D{}".format(flash1_path), + "bs=3D1M", "count=3D64"]) + +def get_pflash_args(flash_dir=3D"./"): + """Returns a string that can be used to + boot qemu using the appropriate pflash files + for aarch64.""" + flash0_path =3D get_flash_path(flash_dir, "flash0") + flash1_path =3D get_flash_path(flash_dir, "flash1") + pflash_args_str =3D "-drive file=3D{},format=3Draw,if=3Dpflash "\ + "-drive file=3D{},format=3Draw,if=3Dpflash" + pflash_args =3D pflash_args_str.format(flash0_path, flash1_path) + return pflash_args.split(" ") + +def get_flash_path(flash_dir, name): + return os.path.join(flash_dir, "{}.img".format(name)) diff --git a/tests/vm/basevm.py b/tests/vm/basevm.py index 305b839000..a0727d7058 100644 --- a/tests/vm/basevm.py +++ b/tests/vm/basevm.py @@ -482,6 +482,14 @@ def get_qemu_path(arch, build_path=3DNone): qemu_path =3D "qemu-system-" + arch return qemu_path =20 +def get_qemu_version(qemu_path): + """Get the version number from the current QEMU, + and return the major number.""" + output =3D subprocess.check_output([qemu_path, '--version']) + version_line =3D output.decode("utf-8") + version_num =3D re.split(' |\(', version_line)[3].split('.')[0] + return int(version_num) + def parse_config(config, args): """ Parse yaml config and populate our config structure. The yaml config allows the user to override the diff --git a/tests/vm/ubuntu.aarch64 b/tests/vm/ubuntu.aarch64 new file mode 100755 index 0000000000..0413971d33 --- /dev/null +++ b/tests/vm/ubuntu.aarch64 @@ -0,0 +1,117 @@ +#!/usr/bin/env python3 +# +# Ubuntu aarch64 image +# +# Copyright 2020 Linaro +# +# Authors: +# Robert Foley +# Originally based on ubuntu.i386 Fam Zheng +# +# This code is licensed under the GPL version 2 or later. See +# the COPYING file in the top-level directory. +# + +import os +import sys +import subprocess +import basevm +from qemu.accel import kvm_available +import time +import aarch64vm + +DEFAULT_CONFIG =3D { + 'cpu' : "cortex-a57", + 'machine' : "virt,gic-version=3D3", + 'install_cmds' : "apt-get update,"\ + "apt-get build-dep -y qemu,"\ + "apt-get install -y libfdt-dev flex bison", + # We increase beyond the default time since during boot + # it can take some time (many seconds) to log into the VM + # especially using softmmu. + 'ssh_timeout' : 60, +} + +class UbuntuAarch64VM(basevm.BaseVM): + name =3D "ubuntu.aarch64" + arch =3D "aarch64" + image_name =3D "ubuntu-18.04-server-cloudimg-arm64.img" + image_link =3D "https://cloud-images.ubuntu.com/releases/18.04/release= /" + image_name + login_prompt =3D "ubuntu-aarch64-guest login:" + BUILD_SCRIPT =3D """ + set -e; + cd $(mktemp -d); + sudo chmod a+r /dev/vdb; + tar --checkpoint=3D.10 -xf /dev/vdb; + ./configure {configure_opts}; + make --output-sync {target} -j{jobs} {verbose}; + """ + def boot(self, img, extra_args=3DNone): + aarch64vm.create_flash_images(self._tmpdir) + default_args =3D aarch64vm.get_pflash_args(self._tmpdir) + if extra_args: + extra_args.extend(default_args) + else: + extra_args =3D default_args + # We always add these performance tweaks + # because without them, we boot so slowly that we + # can time out finding the boot efi device. + if '-smp' not in extra_args and \ + '-smp' not in self._config['extra_args'] and \ + '-smp' not in self._args: + # Only add if not already there to give caller option to chang= e it. + extra_args.extend(["-smp", "8"]) + + # We have overridden boot() since aarch64 has additional parameter= s. + # Call down to the base class method. + super(UbuntuAarch64VM, self).boot(img, extra_args=3Dextra_args) + + def build_image(self, img): + os_img =3D self._download_with_cache(self.image_link) + img_tmp =3D img + ".tmp" + subprocess.check_call(["cp", "-f", os_img, img_tmp]) + subprocess.check_call(["qemu-img", "resize", img_tmp, "+50G"]) + ci_img =3D self.gen_cloud_init_iso() + + self.boot(img_tmp, extra_args =3D ["-cdrom", ci_img]) + if self._boot_console: + self.wait_boot() + # First command we issue is fix for slow ssh login. + self.wait_ssh(wait_root=3DTrue, + cmd=3D"chmod -x /etc/update-motd.d/*") + # Wait for cloud init to finish + self.wait_ssh(wait_root=3DTrue, + cmd=3D"ls /var/lib/cloud/instance/boot-finished") + self.ssh_root("touch /etc/cloud/cloud-init.disabled") + # Disable auto upgrades. + # We want to keep the VM system state stable. + self.ssh_root('sed -ie \'s/"1"/"0"/g\' /etc/apt/apt.conf.d/20auto-= upgrades') + # If the user chooses *not* to do the second phase, + # then we will jump right to the graceful shutdown + if self._config['install_cmds'] !=3D "": + self.ssh_root("sync") + # Shutdown and then boot it again. + # Allows us to know for sure it is booting (not shutting down) + # before we call wait_ssh(). + self.graceful_shutdown() + self.boot(img_tmp) + if self._boot_console: + self.wait_boot() + self.wait_ssh(wait_root=3DTrue) + self.wait_ssh(wait_root=3DTrue, cmd=3D"locale") + # The previous update sometimes doesn't survive a reboot, so d= o it again + self.ssh_root("sed -ie s/^#\ deb-src/deb-src/g /etc/apt/source= s.list") + + # Issue the install commands. + # This can be overriden by the user in the config .yml. + install_cmds =3D self._config['install_cmds'].split(',') + for cmd in install_cmds: + self.ssh_root(cmd) + self.graceful_shutdown() + self.wait() + os.rename(img_tmp, img) + return 0 + +if __name__ =3D=3D "__main__": + defaults =3D aarch64vm.get_config_defaults(UbuntuAarch64VM, DEFAULT_CO= NFIG) + sys.exit(basevm.main(UbuntuAarch64VM, defaults)) --=20 2.17.1 From nobody Fri Nov 14 00:46:50 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=linaro.org ARC-Seal: i=1; a=rsa-sha256; t=1584023974; cv=none; d=zohomail.com; s=zohoarc; b=F4ZDQ8WNQTSH6bBlRqOZsAEy4IStSqO+RAIFXTiyvzFXxzQNRqxqULMCCG6qI7MptNKKSY3VTAIH/z7HjGqFk/Z/MpoVdPJEG6mZ3BU3hD6xaizKltx6J/HA7k47/0GJ6eRFh9YJk+7rt3ppViWUI0CveAfX1NnHgPYFFcB1Y+s= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1584023974; h=Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:Message-ID:References:Sender:Subject:To; bh=hmG132c25gkwNnOaVm8bO+gX0TNplnYsjboTJaPel+o=; b=NhZ5V16Q29L5MV93Y8OMsNPOO+tV6he6/FoxgYvy8x9/O8vuZK1WOwXo3Rj6WXkp0AHgpOD+6QHQkJR7fOyRn071fPv7D4+UKdo6bOzznK/WilPqGYUpwEmBlsn4VwQn+WwsX39fWobt0AwNk7BK80fg0KEh8Gig4nbyDY/ldfw= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1584023974151328.32884221396034; Thu, 12 Mar 2020 07:39:34 -0700 (PDT) Received: from localhost ([::1]:42562 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jCOzl-0004fg-11 for importer@patchew.org; Thu, 12 Mar 2020 10:39:33 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:35727) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jCOrM-0007LK-GP for qemu-devel@nongnu.org; Thu, 12 Mar 2020 10:30:54 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1jCOrJ-0006Yz-P1 for qemu-devel@nongnu.org; Thu, 12 Mar 2020 10:30:52 -0400 Received: from mail-pj1-x1033.google.com ([2607:f8b0:4864:20::1033]:40187) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1jCOrJ-0006Ya-GX for qemu-devel@nongnu.org; Thu, 12 Mar 2020 10:30:49 -0400 Received: by mail-pj1-x1033.google.com with SMTP id bo3so1306581pjb.5 for ; Thu, 12 Mar 2020 07:30:49 -0700 (PDT) Received: from Rfoley-MA01.hsd1.ma.comcast.net ([2601:199:4480:60c0:845e:b9f6:81a6:8f5e]) by smtp.gmail.com with ESMTPSA id 63sm14832651pfx.132.2020.03.12.07.30.45 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 12 Mar 2020 07:30:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=hmG132c25gkwNnOaVm8bO+gX0TNplnYsjboTJaPel+o=; b=PtAX7Yq8Xe+qap8ljfsuEcEKHdqlT9F9vwrl2WPn6GW60dVA0qhUQO9ZPf32ksT5/F z/rbtnWQ91MebSlLGGdxZjR4NSZdw7k1IlgqzadQ0LrCl1Tb0mUExv8JdZUNUzJCdPec d6FIMvZnPQsQnTnjZqybxA2wNxm4HuzIvU3iBUlT/YbQ3f7zWAfvUiHrerSEIXcqYylq HhcsHDVcTP7cqjRaBHsoStHF1JHa3niM1sScDp+ApbgOdDQTfrozGVsDemmhMrnOSrBP JkhGQlAkH6qAtk9pBWPWMu53GuCyhInCxoh37DmQPcJOd0O7c43qZlC/XLywtk6ZOC6G FOHQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=hmG132c25gkwNnOaVm8bO+gX0TNplnYsjboTJaPel+o=; b=NXwnqZC+PlhA157Q/EDtADdXiLOmn0+ITFki/zV6M0NHanuDmxnMYE4vithqk93R9c BB+N695XhyD+jufpqnvlrR8qFvzq2PYKY/Py++8wXgDtUb3qzxa+YU6DYCvF0Hsa4kQ+ 32KNEqiagktlZBVp1TVgjOZyVzM40wV9hpmViP4Z9ufaoqJjVhfr/6AotCWUAs1+f4WM 9rqjeTC32rTzQd6Fu2zEtwgs8pJE4FxIj+B+03DdWCyiwqE24Dam4DsHRto8I0KRwZhf 8coanDDkY82B6Sbd4Ka4nwQuEHcf7mGu+uEOYVN/g7Tvs3TUskwyYd3jAL2LTP6ZfCTO hBMA== X-Gm-Message-State: ANhLgQ1r2/dLT8/A/JaOqowB998Hb9isHt8PLyMo95h2AcavlT3hO1aB dbKlj2I+XVA3HzSwSsaznaCTN/9bacM= X-Google-Smtp-Source: ADFU+vsFmnkstv0h33Sort0bR2YwH969wUCaMvgB1R/RXA33vtoOwmNIQ/3gTAF1la+ZgBOQhwXQWQ== X-Received: by 2002:a17:90a:1a43:: with SMTP id 3mr4407862pjl.35.1584023447726; Thu, 12 Mar 2020 07:30:47 -0700 (PDT) From: Robert Foley To: qemu-devel@nongnu.org Subject: [PATCH v4 08/10] tests/vm: Added a new script for centos.aarch64. Date: Thu, 12 Mar 2020 10:27:26 -0400 Message-Id: <20200312142728.12285-9-robert.foley@linaro.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20200312142728.12285-1-robert.foley@linaro.org> References: <20200312142728.12285-1-robert.foley@linaro.org> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:4864:20::1033 X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: philmd@redhat.com, alex.bennee@linaro.org, robert.foley@linaro.org, peter.puhov@linaro.org Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" centos.aarch64 creates a CentOS 8 image. Also added a new kickstart script used to build the centos.aarch64 image. Signed-off-by: Robert Foley --- tests/vm/Makefile.include | 3 +- tests/vm/centos-8-aarch64.ks | 51 ++++++++ tests/vm/centos.aarch64 | 226 +++++++++++++++++++++++++++++++++++ 3 files changed, 279 insertions(+), 1 deletion(-) create mode 100644 tests/vm/centos-8-aarch64.ks create mode 100755 tests/vm/centos.aarch64 diff --git a/tests/vm/Makefile.include b/tests/vm/Makefile.include index 9934008bfa..419e0f8613 100644 --- a/tests/vm/Makefile.include +++ b/tests/vm/Makefile.include @@ -6,7 +6,7 @@ IMAGES :=3D freebsd netbsd openbsd centos fedora ifneq ($(GENISOIMAGE),) IMAGES +=3D ubuntu.i386 centos ifneq ($(QEMU_EFI_AARCH64),) -IMAGES +=3D ubuntu.aarch64 +IMAGES +=3D ubuntu.aarch64 centos.aarch64 endif endif =20 @@ -28,6 +28,7 @@ ifneq ($(GENISOIMAGE),) @echo " vm-build-ubuntu.i386 - Build QEMU in ubuntu i386 VM" ifneq ($(QEMU_EFI_AARCH64),) @echo " vm-build-ubuntu.aarch64 - Build QEMU in ubuntu aarch64 V= M" + @echo " vm-build-centos.aarch64 - Build QEMU in CentOS aarch64 V= M" else @echo " (install qemu-efi-aarch64 to build centos/ubuntu aarch64 images.= )" endif diff --git a/tests/vm/centos-8-aarch64.ks b/tests/vm/centos-8-aarch64.ks new file mode 100644 index 0000000000..fd6ebe4d49 --- /dev/null +++ b/tests/vm/centos-8-aarch64.ks @@ -0,0 +1,51 @@ +# CentOS aarch64 image kickstart file. +# This file is used by the CentOS installer to +# script the generation of the image. +# +# Copyright 2020 Linaro +# +ignoredisk --only-use=3Dvda +# System bootloader configuration +bootloader --append=3D" crashkernel=3Dauto" --location=3Dmbr --boot-drive= =3Dvda +autopart --type=3Dplain +# Partition clearing information +clearpart --linux --initlabel --drives=3Dvda +# Use text mode install +text +repo --name=3D"AppStream" --baseurl=3Dfile:///run/install/repo/AppStream +# Use CDROM installation media +cdrom +# Keyboard layouts +keyboard --vckeymap=3Dus --xlayouts=3D'' +# System language +lang en_US.UTF-8 + +# Network information +network --bootproto=3Ddhcp --device=3Denp0s1 --onboot=3Doff --ipv6=3Dauto= --no-activate +network --hostname=3Dlocalhost.localdomain +# Run the Setup Agent on first boot +firstboot --enable +# Do not configure the X Window System +skipx +# System services +services --enabled=3D"chronyd" +# System timezone +timezone America/New_York --isUtc + +# Shutdown after installation is complete. +shutdown + +%packages +@^server-product-environment +kexec-tools + +%end + +%addon com_redhat_kdump --enable --reserve-mb=3D'auto' + +%end +%anaconda +pwpolicy root --minlen=3D6 --minquality=3D1 --notstrict --nochanges --note= mpty +pwpolicy user --minlen=3D6 --minquality=3D1 --notstrict --nochanges --empt= yok +pwpolicy luks --minlen=3D6 --minquality=3D1 --notstrict --nochanges --note= mpty +%end diff --git a/tests/vm/centos.aarch64 b/tests/vm/centos.aarch64 new file mode 100755 index 0000000000..f2aa79e02d --- /dev/null +++ b/tests/vm/centos.aarch64 @@ -0,0 +1,226 @@ +#!/usr/bin/env python3 +# +# Centos aarch64 image +# +# Copyright 2020 Linaro +# +# Authors: +# Robert Foley +# Originally based on ubuntu.aarch64 +# +# This code is licensed under the GPL version 2 or later. See +# the COPYING file in the top-level directory. +# + +import os +import sys +import subprocess +import basevm +import time +import traceback +import aarch64vm + +DEFAULT_CONFIG =3D { + 'cpu' : "max", + 'machine' : "virt,gic-version=3Dmax", + 'install_cmds' : "yum install -y make git python3 gcc gcc-c++ flex bis= on, "\ + "yum install -y glib2-devel pixman-devel zlib-devel, "\ + "yum install -y perl-Test-Harness, "\ + "sudo dnf config-manager "\ + "--add-repo=3Dhttps://download.docker.com/linux/centos/docker-ce.r= epo,"\ + "sudo dnf install --nobest -y docker-ce.aarch64,"\ + "systemctl enable docker", + # We increase beyond the default time since during boot + # it can take some time (many seconds) to log into the VM. + 'ssh_timeout' : 60, +} + +class CentosAarch64VM(basevm.BaseVM): + name =3D "centos.aarch64" + arch =3D "aarch64" + login_prompt =3D "localhost login:" + prompt =3D '[root@localhost ~]#' + image_name =3D "CentOS-8-aarch64-1905-dvd1.iso" + image_link =3D "http://mirrors.usc.edu/pub/linux/distributions/centos/= 8.0.1905/isos/aarch64/" + image_link +=3D image_name + BUILD_SCRIPT =3D """ + set -e; + cd $(mktemp -d); + sudo chmod a+r /dev/vdb; + tar --checkpoint=3D.10 -xf /dev/vdb; + ./configure {configure_opts}; + make --output-sync {target} -j{jobs} {verbose}; + """ + def set_key_perm(self): + """Set permissions properly on certain files to allow + ssh access.""" + self.console_wait_send(self.prompt, + "/usr/sbin/restorecon -R -v /root/.ssh\n") + self.console_wait_send(self.prompt, + "/usr/sbin/restorecon -R -v "\ + "/home/{}/.ssh\n".format(self._config["guest_user"])) + + def create_kickstart(self): + """Generate the kickstart file used to generate the centos image."= "" + # Start with the template for the kickstart. + ks_file =3D "../tests/vm/centos-8-aarch64.ks" + subprocess.check_call("cp {} ./ks.cfg".format(ks_file), shell=3DTr= ue) + # Append the ssh keys to the kickstart file + # as the post processing phase of installation. + with open("ks.cfg", "a") as f: + # Add in the root pw and guest user. + rootpw =3D "rootpw --plaintext {}\n" + f.write(rootpw.format(self._config["root_pass"])) + add_user =3D "user --groups=3Dwheel --name=3D{} "\ + "--password=3D{} --plaintext\n" + f.write(add_user.format(self._config["guest_user"], + self._config["guest_pass"])) + # Add the ssh keys. + f.write("%post --log=3D/root/ks-post.log\n") + f.write("mkdir -p /root/.ssh\n") + addkey =3D 'echo "{}" >> /root/.ssh/authorized_keys\n' + addkey_cmd =3D addkey.format(self._config["ssh_pub_key"]) + f.write(addkey_cmd) + f.write('mkdir -p /home/{}/.ssh\n'.format(self._config["guest_= user"])) + addkey =3D 'echo "{}" >> /home/{}/.ssh/authorized_keys\n' + addkey_cmd =3D addkey.format(self._config["ssh_pub_key"], + self._config["guest_user"]) + f.write(addkey_cmd) + f.write("%end\n") + # Take our kickstart file and create an .iso from it. + # The .iso will be provided to qemu as we boot + # from the install dvd. + # Anaconda will recognize the label "OEMDRV" and will + # start the automated installation. + gen_iso_img =3D 'genisoimage -output ks.iso -volid "OEMDRV" ks.cfg' + subprocess.check_call(gen_iso_img, shell=3DTrue) + + def wait_for_shutdown(self): + """We wait for qemu to shutdown the VM and exit. + While this happens we display the console view + for easier debugging.""" + # The image creation is essentially done, + # so whether or not the wait is successful we want to + # wait for qemu to exit (the self.wait()) before we return. + try: + self.console_wait("reboot: Power down") + except Exception as e: + sys.stderr.write("Exception hit\n") + if isinstance(e, SystemExit) and e.code =3D=3D 0: + return 0 + traceback.print_exc() + finally: + self.wait() + + def build_base_image(self, dest_img): + """Run through the centos installer to create + a base image with name dest_img.""" + # We create the temp image, and only rename + # to destination when we are done. + img =3D dest_img + ".tmp" + # Create an empty image. + # We will provide this as the install destination. + qemu_img_create =3D "qemu-img create {} 50G".format(img) + subprocess.check_call(qemu_img_create, shell=3DTrue) + + # Create our kickstart file to be fed to the installer. + self.create_kickstart() + # Boot the install dvd with the params as our ks.iso + os_img =3D self._download_with_cache(self.image_link) + dvd_iso =3D "centos-8-dvd.iso" + subprocess.check_call(["cp", "-f", os_img, dvd_iso]) + extra_args =3D "-cdrom ks.iso" + extra_args +=3D " -drive file=3D{},if=3Dnone,id=3Ddrive1,cache=3Dw= riteback" + extra_args +=3D " -device virtio-blk,drive=3Ddrive1,bootindex=3D1" + extra_args =3D extra_args.format(dvd_iso).split(" ") + self.boot(img, extra_args=3Dextra_args) + self.console_wait_send("change the selection", "\n") + # We seem to need to hit esc (chr(27)) twice to abort the + # media check, which takes a long time. + # Waiting a bit seems to be more reliable before hitting esc. + self.console_wait("Checking") + time.sleep(5) + self.console_wait_send("Checking", chr(27)) + time.sleep(5) + self.console_wait_send("Checking", chr(27)) + print("Found Checking") + # Give sufficient time for the installer to create the image. + self.console_init(timeout=3D7200) + self.wait_for_shutdown() + os.rename(img, dest_img) + print("Done with base image build: {}".format(dest_img)) + + def check_create_base_img(self, img_base, img_dest): + """Create a base image using the installer. + We will use the base image if it exists. + This helps cut down on install time in case we + need to restart image creation, + since the base image creation can take a long time.""" + if not os.path.exists(img_base): + print("Generate new base image: {}".format(img_base)) + self.build_base_image(img_base); + else: + print("Use existing base image: {}".format(img_base)) + # Save a copy of the base image and copy it to dest. + # which we will use going forward. + subprocess.check_call(["cp", img_base, img_dest]) + + def boot(self, img, extra_args=3DNone): + aarch64vm.create_flash_images(self._tmpdir) + default_args =3D aarch64vm.get_pflash_args(self._tmpdir) + if extra_args: + extra_args.extend(default_args) + else: + extra_args =3D default_args + # We always add these performance tweaks + # because without them, we boot so slowly that we + # can time out finding the boot efi device. + if '-smp' not in extra_args and \ + '-smp' not in self._config['extra_args'] and \ + '-smp' not in self._args: + # Only add if not already there to give caller option to chang= e it. + extra_args.extend(["-smp", "8"]) + # We have overridden boot() since aarch64 has additional parameter= s. + # Call down to the base class method. + super(CentosAarch64VM, self).boot(img, extra_args=3Dextra_args) + + def build_image(self, img): + img_tmp =3D img + ".tmp" + self.check_create_base_img(img + ".base", img_tmp) + + # Boot the new image for the first time to finish installation. + self.boot(img_tmp) + self.console_init() + self.console_wait_send(self.login_prompt, "root\n") + self.console_wait_send("Password:", + "{}\n".format(self._config["root_pass"])) + + self.set_key_perm() + self.console_wait_send(self.prompt, "rpm -q centos-release\n") + enable_adapter =3D "sed -i 's/ONBOOT=3Dno/ONBOOT=3Dyes/g'" \ + " /etc/sysconfig/network-scripts/ifcfg-enp0s1\n" + self.console_wait_send(self.prompt, enable_adapter) + self.console_wait_send(self.prompt, "ifup enp0s1\n") + self.console_wait_send(self.prompt, + 'echo "qemu ALL=3D(ALL) NOPASSWD:ALL" | '\ + 'sudo tee /etc/sudoers.d/qemu\n') + self.console_wait(self.prompt) + + # Rest of the commands we issue through ssh. + self.wait_ssh(wait_root=3DTrue) + + # If the user chooses *not* to do the second phase, + # then we will jump right to the graceful shutdown + if self._config['install_cmds'] !=3D "": + install_cmds =3D self._config['install_cmds'].split(',') + for cmd in install_cmds: + self.ssh_root(cmd) + self.ssh_root("poweroff") + self.wait_for_shutdown() + os.rename(img_tmp, img) + print("image creation complete: {}".format(img)) + return 0 + +if __name__ =3D=3D "__main__": + defaults =3D aarch64vm.get_config_defaults(CentosAarch64VM, DEFAULT_CO= NFIG) + sys.exit(basevm.main(CentosAarch64VM, defaults)) --=20 2.17.1 From nobody Fri Nov 14 00:46:50 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=linaro.org ARC-Seal: i=1; a=rsa-sha256; t=1584023670; cv=none; d=zohomail.com; s=zohoarc; b=IFfyMKimOitDBZBzi2eWH9Zf5p64t66FKOYvwUKWGkQEV4WNZFQKC87S+qXbZsGcOAQmb7LAoowyeo4V5M638DJ7Iq16kNLRytSGI4+YKeuSEUhuh39lcr0Sbp3VBEB4FIkZFfQneg7vJdqc2pgwZ1Zu5u66e1yN/Yk7eM33KvY= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1584023670; h=Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:Message-ID:References:Sender:Subject:To; bh=Zu55w57Oui8Xur6dOmzXIZRXCZIx7GGCMyc1jtFJeG4=; b=fYX3A5ROXMg5hHBwhL7zbynj69/hIrDCBSVH5SFY1t4AsvDRUQT09BWUHFMugNnZAndZAL9ASrXs3JtjonyMCjYd64jvFQ9Wn6YkwTxWZKN2UPfgxt3dMzNKGzHPR5/Tr199GPd8dhu6O5cvMtbszLbtD0QWsKHsjKRxTuyMFL4= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1584023670104123.3247360746185; Thu, 12 Mar 2020 07:34:30 -0700 (PDT) Received: from localhost ([::1]:42379 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jCOuq-0004lf-Uo for importer@patchew.org; Thu, 12 Mar 2020 10:34:28 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:35746) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jCOrO-0007MA-Qu for qemu-devel@nongnu.org; Thu, 12 Mar 2020 10:30:56 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1jCOrM-0006aO-2m for qemu-devel@nongnu.org; Thu, 12 Mar 2020 10:30:54 -0400 Received: from mail-pl1-x644.google.com ([2607:f8b0:4864:20::644]:39871) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1jCOrL-0006Zd-BW for qemu-devel@nongnu.org; Thu, 12 Mar 2020 10:30:51 -0400 Received: by mail-pl1-x644.google.com with SMTP id j20so2735306pll.6 for ; Thu, 12 Mar 2020 07:30:51 -0700 (PDT) Received: from Rfoley-MA01.hsd1.ma.comcast.net ([2601:199:4480:60c0:845e:b9f6:81a6:8f5e]) by smtp.gmail.com with ESMTPSA id 63sm14832651pfx.132.2020.03.12.07.30.47 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 12 Mar 2020 07:30:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=Zu55w57Oui8Xur6dOmzXIZRXCZIx7GGCMyc1jtFJeG4=; b=zAGn4r79ip/B04Fud+G21ePfSvS07yQoL95NXayfbaaaElhRM/PqwAQfZVVohem9JH yO6Ef/gxSqWVBGjdOi/pR7rbDiweswvXo1dPcRX7TT8CpKCMnIHFqfRxpmel+r1PQVbB JC9nXy4+xUqSVnpnsmlGlzLDuFGe8NXBG9PWUINlzqLucXhFsYZGRu2AGC/w9ypZxRVM krBPK1uFWc1F8pVuMDWiKrPXwVLZqDQtgmTy5pCldsGV3TJimI1TQPe/2LMqmuCaGtSd s6I4JoMKJ9USseNpBTuc4PQhC48P0/KlNvZ7izxA+OqQMVt9qU4IN4sQOLKXRU2yNDXv Z9eA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=Zu55w57Oui8Xur6dOmzXIZRXCZIx7GGCMyc1jtFJeG4=; b=jhreq3CmYd/VKgCFdG4alywwO9ItWOn6sSJV/gQe1YsHVB7qgBVdlSoRDjxAiLcUW2 e7+VAETQZHLNo6y+0EFKY4v8j9623i6a3FkZZcjU2e+rEVpSrIv2PvAA9GzxnHIIYJQW Yvg5jMEQneCdOSUnp4eJFkwBFXNPbuNf8Rl73q6jBSds0Rm4IiS2Z6UPhRlxP7h5fxiO T0PoGeeatwImtDxPD0Z2A/iDeuLzlWsx1zQGUMPkb0iiLhg5p1oYKXvI6At6dBKrZUEO 7JXkomXEZR0F5gaYmWaBfdfFCkZEe6qt/2EP/Gq+eg/e4VFW1J+gXXQRrpsXL9YQqaAt uRKA== X-Gm-Message-State: ANhLgQ3YFA1BVIFCOy8pv2/6TWqrC88izsUYWbxqbAOW8ku72iYvEvZT Q3GDAKAapD3lSG5EuuAGsjHLSqE09nI= X-Google-Smtp-Source: ADFU+vuw0vIJoBXTur5txz8MWUz9W/jKNCpbv/UINY3eLSGppJOuO3GGN9y82Q6mZOGVWXyBNnzlsg== X-Received: by 2002:a17:902:9a4c:: with SMTP id x12mr7797042plv.297.1584023449658; Thu, 12 Mar 2020 07:30:49 -0700 (PDT) From: Robert Foley To: qemu-devel@nongnu.org Subject: [PATCH v4 09/10] tests/vm: change scripts to use self._config Date: Thu, 12 Mar 2020 10:27:27 -0400 Message-Id: <20200312142728.12285-10-robert.foley@linaro.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20200312142728.12285-1-robert.foley@linaro.org> References: <20200312142728.12285-1-robert.foley@linaro.org> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:4864:20::644 X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: philmd@redhat.com, alex.bennee@linaro.org, robert.foley@linaro.org, peter.puhov@linaro.org Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" This change converts existing scripts to using for example self.ROOT_PASS, to self._config['root_pass']. We made similar changes for GUEST_USER, and GUEST_PASS. This allows us also to remove the change in basevm.py, which adds __getattr__ for backwards compatibility. Signed-off-by: Robert Foley --- tests/vm/basevm.py | 11 ++--------- tests/vm/fedora | 17 +++++++++-------- tests/vm/freebsd | 16 ++++++++-------- tests/vm/netbsd | 19 ++++++++++--------- tests/vm/openbsd | 17 +++++++++-------- 5 files changed, 38 insertions(+), 42 deletions(-) diff --git a/tests/vm/basevm.py b/tests/vm/basevm.py index a0727d7058..bd4a1f9639 100644 --- a/tests/vm/basevm.py +++ b/tests/vm/basevm.py @@ -170,13 +170,6 @@ class BaseVM(object): self.console_init(timeout=3Dtimeout) self.console_wait(wait_string) =20 - def __getattr__(self, name): - # Support direct access to config by key. - # for example, access self._config['cpu'] by self.cpu - if name.lower() in self._config.keys(): - return self._config[name.lower()] - return object.__getattribute__(self, name) - def _download_with_cache(self, url, sha256sum=3DNone, sha512sum=3DNone= ): def check_sha256sum(fname): if not sha256sum: @@ -226,13 +219,13 @@ class BaseVM(object): return r =20 def ssh(self, *cmd): - return self._ssh_do(self.GUEST_USER, cmd, False) + return self._ssh_do(self._config["guest_user"], cmd, False) =20 def ssh_root(self, *cmd): return self._ssh_do("root", cmd, False) =20 def ssh_check(self, *cmd): - self._ssh_do(self.GUEST_USER, cmd, True) + self._ssh_do(self._config["guest_user"], cmd, True) =20 def ssh_root_check(self, *cmd): self._ssh_do("root", cmd, True) diff --git a/tests/vm/fedora b/tests/vm/fedora index 4843b4175e..6a7023eb71 100755 --- a/tests/vm/fedora +++ b/tests/vm/fedora @@ -108,20 +108,20 @@ class FedoraVM(basevm.BaseVM): =20 self.console_wait_send("7) [!] Root password", "7\n") self.console_wait("Password:") - self.console_send("%s\n" % self.ROOT_PASS) + self.console_send("%s\n" % self._config["root_pass"]) self.console_wait("Password (confirm):") - self.console_send("%s\n" % self.ROOT_PASS) + self.console_send("%s\n" % self._config["root_pass"]) =20 self.console_wait_send("8) [ ] User creation", "8\n") self.console_wait_send("1) [ ] Create user", "1\n") self.console_wait_send("3) User name", "3\n") - self.console_wait_send("ENTER:", "%s\n" % self.GUEST_USER) + self.console_wait_send("ENTER:", "%s\n" % self._config["guest_user= "]) self.console_wait_send("4) [ ] Use password", "4\n") self.console_wait_send("5) Password", "5\n") self.console_wait("Password:") - self.console_send("%s\n" % self.GUEST_PASS) + self.console_send("%s\n" % self._config["guest_pass"]) self.console_wait("Password (confirm):") - self.console_send("%s\n" % self.GUEST_PASS) + self.console_send("%s\n" % self._config["guest_pass"]) self.console_wait_send("7) Groups", "c\n") =20 while True: @@ -139,7 +139,7 @@ class FedoraVM(basevm.BaseVM): if good: break time.sleep(10) - self.console_send("r\n" % self.GUEST_PASS) + self.console_send("r\n" % self._config["guest_pass"]) =20 self.console_wait_send("'b' to begin install", "b\n") =20 @@ -150,12 +150,13 @@ class FedoraVM(basevm.BaseVM): =20 # setup qemu user prompt =3D " ~]$" - self.console_ssh_init(prompt, self.GUEST_USER, self.GUEST_PASS) + self.console_ssh_init(prompt, self._config["guest_user"], + self._config["guest_pass"]) self.console_wait_send(prompt, "exit\n") =20 # setup root user prompt =3D " ~]#" - self.console_ssh_init(prompt, "root", self.ROOT_PASS) + self.console_ssh_init(prompt, "root", self._config["root_pass"]) self.console_sshd_config(prompt) =20 # setup virtio-blk #1 (tarfile) diff --git a/tests/vm/freebsd b/tests/vm/freebsd index 86770878b6..43f14f102d 100755 --- a/tests/vm/freebsd +++ b/tests/vm/freebsd @@ -113,9 +113,9 @@ class FreeBSDVM(basevm.BaseVM): =20 # post-install configuration self.console_wait("New Password:") - self.console_send("%s\n" % self.ROOT_PASS) + self.console_send("%s\n" % self._config["root_pass"]) self.console_wait("Retype New Password:") - self.console_send("%s\n" % self.ROOT_PASS) + self.console_send("%s\n" % self._config["root_pass"]) =20 self.console_wait_send("Network Configuration", "\n") self.console_wait_send("IPv4", "y") @@ -134,9 +134,9 @@ class FreeBSDVM(basevm.BaseVM): # qemu user self.console_wait_send("Add User Accounts", "y") self.console_wait("Username") - self.console_send("%s\n" % self.GUEST_USER) + self.console_send("%s\n" % self._config["guest_user"]) self.console_wait("Full name") - self.console_send("%s\n" % self.GUEST_USER) + self.console_send("%s\n" % self._config["guest_user"]) self.console_wait_send("Uid", "\n") self.console_wait_send("Login group", "\n") self.console_wait_send("Login group", "\n") @@ -148,9 +148,9 @@ class FreeBSDVM(basevm.BaseVM): self.console_wait_send("Use an empty password", "\n") self.console_wait_send("Use a random password", "\n") self.console_wait("Enter password:") - self.console_send("%s\n" % self.GUEST_PASS) + self.console_send("%s\n" % self._config["guest_pass"]) self.console_wait("Enter password again:") - self.console_send("%s\n" % self.GUEST_PASS) + self.console_send("%s\n" % self._config["guest_pass"]) self.console_wait_send("Lock out", "\n") self.console_wait_send("OK", "yes\n") self.console_wait_send("Add another user", "no\n") @@ -164,12 +164,12 @@ class FreeBSDVM(basevm.BaseVM): =20 # setup qemu user prompt =3D "$" - self.console_ssh_init(prompt, self.GUEST_USER, self.GUEST_PASS) + self.console_ssh_init(prompt, self._config["guest_user"], self._co= nfig["guest_pass"]) self.console_wait_send(prompt, "exit\n") =20 # setup root user prompt =3D "root@freebsd:~ #" - self.console_ssh_init(prompt, "root", self.ROOT_PASS) + self.console_ssh_init(prompt, "root", self._config["root_pass"]) self.console_sshd_config(prompt) =20 # setup serial console diff --git a/tests/vm/netbsd b/tests/vm/netbsd index 55590f4601..e953fcd161 100755 --- a/tests/vm/netbsd +++ b/tests/vm/netbsd @@ -131,24 +131,24 @@ class NetBSDVM(basevm.BaseVM): self.console_wait_send("d: Change root password", "d\n") self.console_wait_send("a: Yes", "a\n") self.console_wait("New password:") - self.console_send("%s\n" % self.ROOT_PASS) + self.console_send("%s\n" % self._config["root_pass"]) self.console_wait("New password:") - self.console_send("%s\n" % self.ROOT_PASS) + self.console_send("%s\n" % self._config["root_pass"]) self.console_wait("Retype new password:") - self.console_send("%s\n" % self.ROOT_PASS) + self.console_send("%s\n" % self._config["root_pass"]) =20 self.console_wait_send("o: Add a user", "o\n") self.console_wait("username") - self.console_send("%s\n" % self.GUEST_USER) + self.console_send("%s\n" % self._config["guest_pass"]) self.console_wait("to group wheel") self.console_wait_send("a: Yes", "a\n") self.console_wait_send("a: /bin/sh", "a\n") self.console_wait("New password:") - self.console_send("%s\n" % self.GUEST_PASS) + self.console_send("%s\n" % self._config["guest_pass"]) self.console_wait("New password:") - self.console_send("%s\n" % self.GUEST_PASS) + self.console_send("%s\n" % self._config["guest_pass"]) self.console_wait("Retype new password:") - self.console_send("%s\n" % self.GUEST_PASS) + self.console_send("%s\n" % self._config["guest_pass"]) =20 self.console_wait_send("a: Configure network", "a\n") self.console_wait_send("a: vioif0", "a\n") @@ -181,12 +181,13 @@ class NetBSDVM(basevm.BaseVM): =20 # setup qemu user prompt =3D "localhost$" - self.console_ssh_init(prompt, self.GUEST_USER, self.GUEST_PASS) + self.console_ssh_init(prompt, self._config["guest_user"], + self._config["guest_pass"]) self.console_wait_send(prompt, "exit\n") =20 # setup root user prompt =3D "localhost#" - self.console_ssh_init(prompt, "root", self.ROOT_PASS) + self.console_ssh_init(prompt, "root", self._config["root_pass"]) self.console_sshd_config(prompt) =20 # setup virtio-blk #1 (tarfile) diff --git a/tests/vm/openbsd b/tests/vm/openbsd index ab6abbedab..445ea78516 100755 --- a/tests/vm/openbsd +++ b/tests/vm/openbsd @@ -98,9 +98,9 @@ class OpenBSDVM(basevm.BaseVM): self.console_wait_send("Which network interface", "done\n") self.console_wait_send("DNS domain name", "localnet\n") self.console_wait("Password for root account") - self.console_send("%s\n" % self.ROOT_PASS) + self.console_send("%s\n" % self._config["root_pass"]) self.console_wait("Password for root account") - self.console_send("%s\n" % self.ROOT_PASS) + self.console_send("%s\n" % self._config["root_pass"]) self.console_wait_send("Start sshd(8)", "yes\n") self.console_wait_send("X Window System", "\n") self.console_wait_send("xenodm", "\n") @@ -108,13 +108,13 @@ class OpenBSDVM(basevm.BaseVM): self.console_wait_send("Which speed", "\n") =20 self.console_wait("Setup a user") - self.console_send("%s\n" % self.GUEST_USER) + self.console_send("%s\n" % self._config["guest_user"]) self.console_wait("Full name") - self.console_send("%s\n" % self.GUEST_USER) + self.console_send("%s\n" % self._config["guest_user"]) self.console_wait("Password") - self.console_send("%s\n" % self.GUEST_PASS) + self.console_send("%s\n" % self._config["guest_pass"]) self.console_wait("Password") - self.console_send("%s\n" % self.GUEST_PASS) + self.console_send("%s\n" % self._config["guest_pass"]) =20 self.console_wait_send("Allow root ssh login", "yes\n") self.console_wait_send("timezone", "UTC\n") @@ -135,12 +135,13 @@ class OpenBSDVM(basevm.BaseVM): =20 # setup qemu user prompt =3D "$" - self.console_ssh_init(prompt, self.GUEST_USER, self.GUEST_PASS) + self.console_ssh_init(prompt, self._config["guest_user"], + self._config["guest_pass"]) self.console_wait_send(prompt, "exit\n") =20 # setup root user prompt =3D "openbsd#" - self.console_ssh_init(prompt, "root", self.ROOT_PASS) + self.console_ssh_init(prompt, "root", self._config["root_pass"]) self.console_sshd_config(prompt) =20 # setup virtio-blk #1 (tarfile) --=20 2.17.1 From nobody Fri Nov 14 00:46:50 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=linaro.org ARC-Seal: i=1; a=rsa-sha256; t=1584023817; cv=none; d=zohomail.com; s=zohoarc; b=FhmXpCKL6wOC1Ip6Q5ERoDhHjv31AqeIxtviVrxEDeu+KZvZoFj74f4z+O7bwvz057427okSZUeqFyqeZHQyEZ26QVk4mCxqVHlRnisVzhdEN0tRpAgk5x8S+RLAUrtFReKDHKBv5O6gVk4R9OYUg1VDe9OwN81Mhafi6j1tFZw= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1584023817; h=Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:Message-ID:References:Sender:Subject:To; bh=NSEQACypX/+5FjWaplvQgKNX4xaeLCqc5O2UmEo8r4w=; b=M+d7O86lRwG+Eu7ZiUsLGxEmPWRobrnRi7p2FxQNOoaHzgAcyHPmu0Iifaw2Z+n3SgGRMihbVXzC5piZQSXh8+BV24MVe1nzUfuVM6AMluo1PpasNax6+LyzLx7hI+Os6ud4iOVB0dxCXSgUEkwrpj3mrj4mGz1rjMOPqV9bqCc= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 158402381713077.00842152261714; Thu, 12 Mar 2020 07:36:57 -0700 (PDT) Received: from localhost ([::1]:42476 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jCOxE-0001PM-0v for importer@patchew.org; Thu, 12 Mar 2020 10:36:56 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:35766) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jCOrR-0007S3-HI for qemu-devel@nongnu.org; Thu, 12 Mar 2020 10:30:59 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1jCOrO-0006bi-Sr for qemu-devel@nongnu.org; Thu, 12 Mar 2020 10:30:57 -0400 Received: from mail-pl1-x62d.google.com ([2607:f8b0:4864:20::62d]:34919) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1jCOrO-0006as-Fm for qemu-devel@nongnu.org; Thu, 12 Mar 2020 10:30:54 -0400 Received: by mail-pl1-x62d.google.com with SMTP id g6so2746357plt.2 for ; Thu, 12 Mar 2020 07:30:53 -0700 (PDT) Received: from Rfoley-MA01.hsd1.ma.comcast.net ([2601:199:4480:60c0:845e:b9f6:81a6:8f5e]) by smtp.gmail.com with ESMTPSA id 63sm14832651pfx.132.2020.03.12.07.30.50 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 12 Mar 2020 07:30:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=NSEQACypX/+5FjWaplvQgKNX4xaeLCqc5O2UmEo8r4w=; b=T65tm8lKON3XMVRqElGKNeTqyDzE35tHFVqCJyUqCVR6kJzQvWleky1pOprJnfLskA rHvhdSyBHB2lW8CAjMqnzymKdiMB54jV2OdeZCYNiQ5z+/HqjvIr4HGa7VBPOb76BhJs Jua3kXZ5rZ2R+sTZ6KjCs+WqzhwnWbDlRoASa2rJ3J5c6CJkV30sdyD38qdHClThu95M YgoKfcPVuExO4OmQHEg14XeoLvmcUxeSLNAgq3nSZGgUbjVAXPVyeAJLBHq/8NxQH1Te T35OeG8RM3rAF2DHs3GHCMpycliC6bFos+OGbLytOzJfG42nVx71wCo3zMHiUtUXLm1t Ek/w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=NSEQACypX/+5FjWaplvQgKNX4xaeLCqc5O2UmEo8r4w=; b=IM57xp7L/eDHyE7FiUzL4eTNgsagGiosBpO3UsWsrobN9vnEjUm7tt3BtIRpSnv1jo 8lTrpkXq1EdG6oALF8rz3xhQPTAcGaAoJkqvtZUpmj4D9bTxtgQnRSSFRF7t920zLxMl xC2bk+w9GJEThNHPO9F77rjHys3qYouoFipzpq/dnSY/SZfTAodfsn12thiZIjfPl338 AM8LDCR/4o09ycg38CZ2UVPgLbz15ZmI7kCpJKTMQHwWqJReangKrUAIk3BIUiE1ao51 oqggjeF9TAqhXUPzX4WWokTn6FFJBoySIxMgm+dRii1XJUwyaREBGXzupD+Llda3aGmH ts+Q== X-Gm-Message-State: ANhLgQ1gUfXlZtPQ6l41Tf1TzN9TbVAU9uf9kd87F9StqKNMpnN725m7 +LDRXueHe6+eIvj38pEJRVfNC2mQTbk= X-Google-Smtp-Source: ADFU+vvQQg9+2OlMEUBBTZUz3dudGL7I5zebZUKk5KN4+GextLKxxLdfegg7LdqG4Y0EAr5yJaCrtA== X-Received: by 2002:a17:902:9303:: with SMTP id bc3mr7855470plb.187.1584023451748; Thu, 12 Mar 2020 07:30:51 -0700 (PDT) From: Robert Foley To: qemu-devel@nongnu.org Subject: [PATCH v4 10/10] tests/vm: Add workaround to consume console Date: Thu, 12 Mar 2020 10:27:28 -0400 Message-Id: <20200312142728.12285-11-robert.foley@linaro.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20200312142728.12285-1-robert.foley@linaro.org> References: <20200312142728.12285-1-robert.foley@linaro.org> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:4864:20::62d X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: philmd@redhat.com, alex.bennee@linaro.org, robert.foley@linaro.org, peter.puhov@linaro.org Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" The ConsoleSocket object provides a socket interface which will consume all arriving characters on the socket, but will provide those chars via recv() as would a regular socket. This is a workaround we found was needed since there is a known issue where QEMU will hang waiting for console characters to be consumed. We also add the option of logging the console to a file. Signed-off-by: Robert Foley --- python/qemu/console_socket.py | 162 ++++++++++++++++++++++++++++++++++ python/qemu/machine.py | 23 ++++- tests/vm/Makefile.include | 4 + tests/vm/basevm.py | 24 ++++- 4 files changed, 205 insertions(+), 8 deletions(-) create mode 100644 python/qemu/console_socket.py diff --git a/python/qemu/console_socket.py b/python/qemu/console_socket.py new file mode 100644 index 0000000000..a1f74e60ac --- /dev/null +++ b/python/qemu/console_socket.py @@ -0,0 +1,162 @@ +#!/usr/bin/env python3 +# +# This python module implements a ConsoleSocket object which is +# designed always drain the socket itself, and place +# the bytes into a in memory buffer for later processing. +# +# Optionally a file path can be passed in and we will also +# dump the characters to this file for debug. +# +# Copyright 2020 Linaro +# +# Authors: +# Robert Foley +# +# This code is licensed under the GPL version 2 or later. See +# the COPYING file in the top-level directory. +# +import asyncore +import socket +import threading +import io +import os +import sys +from collections import deque +import time +import traceback + +class ConsoleSocket(asyncore.dispatcher): + + def __init__(self, address, file=3DNone): + self._recv_timeout_sec =3D 300 + self._buffer =3D deque() + self._asyncore_thread =3D None + self._sock =3D socket.socket(socket.AF_UNIX, socket.SOCK_STREAM) + self._sock.connect(address) + self._logfile =3D None + if file: + self._logfile =3D open(file, "w") + asyncore.dispatcher.__init__(self, sock=3Dself._sock) + self._thread_start() + self._open =3D True + + def _thread_start(self): + """Kick off a thread to wait on the asyncore.loop""" + if self._asyncore_thread is not None: + return + self._asyncore_thread =3D threading.Thread(target=3Dasyncore.loop, + kwargs=3D{'timeout':1}) + self._asyncore_thread.daemon =3D True + self._asyncore_thread.start() + + def handle_close(self): + """redirect close to base class""" + # Call the base class close, but not self.close() since + # handle_close() occurs in the context of the thread which + # self.close() attempts to join. + asyncore.dispatcher.close(self) + + def close(self): + """Close the base object and wait for the thread to terminate""" + if self._open: + self._open =3D False + asyncore.dispatcher.close(self) + if self._asyncore_thread is not None: + thread, self._asyncore_thread =3D self._asyncore_thread, N= one + thread.join() + if self._logfile: + self._logfile.close() + self._logfile =3D None + + def handle_read(self): + """process arriving characters into in memory _buffer""" + try: + data =3D asyncore.dispatcher.recv(self, 1) + # latin1 is needed since there are some chars + # we are receiving that cannot be encoded to utf-8 + # such as 0xe2, 0x80, 0xA6. + string =3D data.decode("latin1") + except: + print("Exception seen.") + traceback.print_exc() + return + if self._logfile: + self._logfile.write("{}".format(string)) + self._logfile.flush() + for c in string: + self._buffer.append(c) + + def recv(self, n=3D1): + """Return chars from in memory buffer""" + start_time =3D time.time() + while len(self._buffer) < n: + time.sleep(0.1) + elapsed_sec =3D time.time() - start_time + if elapsed_sec > self._recv_timeout_sec: + raise socket.timeout + chars =3D ''.join([self._buffer.popleft() for i in range(n)]) + # We choose to use latin1 to remain consistent with + # handle_read() and give back the same data as the user would + # receive if they were reading directly from the + # socket w/o our intervention. + return chars.encode("latin1") + + def set_blocking(self): + """Maintain compatibility with socket API""" + pass + + def settimeout(self, seconds): + """Set current timeout on recv""" + self._recv_timeout_sec =3D seconds + +class ByteBuffer(deque): + """Simple in memory buffer with read/write interface""" + def write(self, bytes): + for i in bytes: + self.append(i) + def read(self, n): + return ''.join([self.popleft() for i in range(n)]) + +if __name__ =3D=3D '__main__': + # Brief test to exercise the above code. + # The ConsoleSocket will ship some data to the server, + # the server will echo it back and the client will echo what it receiv= ed. + + # First remove the socket. + address =3D "./test_console_socket" + if os.path.exists(address): + os.unlink(address) + + # Create the server side. + server_socket =3D socket.socket(socket.AF_UNIX, socket.SOCK_STREAM) + server_socket.bind(address) + server_socket.listen(1) + + # Create the object we are trying to test. + console_socket =3D ConsoleSocket(address, file=3D"./logfile.txt") + + # Generate some data and ship it over the socket. + send_data =3D "" + for i in range(10): + send_data +=3D "this is a test message {}\n".format(i) + console_socket.send(send_data.encode('latin1')) + connection, client_address =3D server_socket.accept() + + # Process the data on the server and ship it back. + data =3D connection.recv(len(send_data)) + print("server received: {}".format(data)) + print("server: sending data back to the client") + connection.sendall(data) + + # Client receives teh bytes and displays them. + print("client: receiving bytes") + bytes =3D console_socket.recv(len(data)) + recv_data =3D bytes.decode('latin1') + print("client received: {}".format(recv_data)) + assert(recv_data =3D=3D send_data) + # Close console connection first, then close server. + console_socket.close() + connection.close() + server_socket.close() + print("test successful.") + diff --git a/python/qemu/machine.py b/python/qemu/machine.py index 183d8f3d38..a995b8db3a 100644 --- a/python/qemu/machine.py +++ b/python/qemu/machine.py @@ -24,6 +24,7 @@ import subprocess import shutil import socket import tempfile +from qemu.console_socket import ConsoleSocket =20 from . import qmp =20 @@ -71,7 +72,8 @@ class QEMUMachine(object): =20 def __init__(self, binary, args=3DNone, wrapper=3DNone, name=3DNone, test_dir=3D"/var/tmp", monitor_address=3DNone, - socket_scm_helper=3DNone, sock_dir=3DNone): + socket_scm_helper=3DNone, sock_dir=3DNone, + drain_console=3DFalse, console_log=3DNone): ''' Initialize a QEMUMachine =20 @@ -82,6 +84,9 @@ class QEMUMachine(object): @param test_dir: where to create socket and log file @param monitor_address: address for QMP monitor @param socket_scm_helper: helper program, required for send_fd_scm= () + @param sock_dir: where to create socket (overrides test_dir for so= ck) + @param console_log: (optional) path to console log file + @param drain_console: (optional) True to drain console socket to b= uffer @note: Qemu process is not started until launch() is used. ''' if args is None: @@ -118,6 +123,12 @@ class QEMUMachine(object): self._console_address =3D None self._console_socket =3D None self._remove_files =3D [] + self._console_log_path =3D console_log + if self._console_log_path: + # In order to log the console, buffering needs to be enabled. + self._drain_console =3D True + else: + self._drain_console =3D drain_console =20 # just in case logging wasn't configured by the main script: logging.basicConfig() @@ -566,7 +577,11 @@ class QEMUMachine(object): Returns a socket connected to the console """ if self._console_socket is None: - self._console_socket =3D socket.socket(socket.AF_UNIX, - socket.SOCK_STREAM) - self._console_socket.connect(self._console_address) + if self._drain_console: + self._console_socket =3D ConsoleSocket(self._console_addre= ss, + file=3Dself._console_= log_path) + else: + self._console_socket =3D socket.socket(socket.AF_UNIX, + socket.SOCK_STREAM) + self._console_socket.connect(self._console_address) return self._console_socket diff --git a/tests/vm/Makefile.include b/tests/vm/Makefile.include index 419e0f8613..fb61f10619 100644 --- a/tests/vm/Makefile.include +++ b/tests/vm/Makefile.include @@ -50,6 +50,7 @@ endif @echo " J=3D[0..9]* - Override the -jN parameter for make= commands" @echo " DEBUG=3D1 - Enable verbose output on host and i= nteractive debugging" @echo " BOOT_CONSOLE=3D1 - Show the console output at boot ti= me. " + @echo " LOG_CONSOLE=3D1 - Log console to file in: ~/.cache/qe= mu-vm " @echo " V=3D1 - Enable verbose ouput on host and guest commands" @echo " QEMU_LOCAL=3D1 - Use QEMU binary local to this= build." @echo " QEMU=3D/path/to/qemu - Change path to QEMU binary" @@ -74,6 +75,7 @@ $(IMAGES_DIR)/%.img: $(SRC_PATH)/tests/vm/% \ $(PYTHON) $< \ $(if $(V)$(DEBUG), --debug) \ $(if $(BOOT_CONSOLE),--boot-console) \ + $(if $(LOG_CONSOLE),--log-console) \ --build-path $(BUILD_DIR)\ --image "$@" \ --force \ @@ -90,6 +92,7 @@ vm-build-%: $(IMAGES_DIR)/%.img $(if $(J),--jobs $(J)) \ $(if $(V),--verbose) \ $(if $(BOOT_CONSOLE),--boot-console) \ + $(if $(LOG_CONSOLE),--log-console) \ --build-path $(BUILD_DIR)\ --image "$<" \ $(if $(BUILD_TARGET),--build-target $(BUILD_TARGET)) \ @@ -112,6 +115,7 @@ vm-boot-ssh-%: $(IMAGES_DIR)/%.img $(PYTHON) $(SRC_PATH)/tests/vm/$* \ $(if $(J),--jobs $(J)) \ $(if $(BOOT_CONSOLE),--boot-console) \ + $(if $(LOG_CONSOLE),--log-console) \ --build-path $(BUILD_DIR)\ --image "$<" \ --interactive \ diff --git a/tests/vm/basevm.py b/tests/vm/basevm.py index bd4a1f9639..e39670635a 100644 --- a/tests/vm/basevm.py +++ b/tests/vm/basevm.py @@ -81,12 +81,14 @@ class BaseVM(object): poweroff =3D "poweroff" # enable IPv6 networking ipv6 =3D True + # This is the timeout on the wait for console bytes. + socket_timeout =3D 120 # Scale up some timeouts under TCG. # 4 is arbitrary, but greater than 2, # since we found we need to wait more than twice as long. tcg_ssh_timeout_multiplier =3D 4 def __init__(self, debug=3DFalse, vcpus=3DNone, config=3DNone, - boot_console=3DFalse, build_path=3DNone): + boot_console=3DFalse, build_path=3DNone, log_console=3DFa= lse): self._guest =3D None self._build_path =3D build_path self._boot_console =3D boot_console @@ -113,6 +115,11 @@ class BaseVM(object): open(self._ssh_tmp_pub_key_file, "w").write(self._config['ssh_pub_key']) =20 + self._console_log_path =3D None + if log_console: + self._console_log_path =3D \ + os.path.join(os.path.expanduser("~/.cache/qemu-vm= "), + "{}.install.log".format(self.name)) self.debug =3D debug self._stderr =3D sys.stderr self._devnull =3D open(os.devnull, "w") @@ -259,7 +266,9 @@ class BaseVM(object): args +=3D ["-device", "VGA"] logging.debug("QEMU args: %s", " ".join(args)) qemu_path =3D get_qemu_path(self.arch, self._build_path) - guest =3D QEMUMachine(binary=3Dqemu_path, args=3Dargs) + guest =3D QEMUMachine(binary=3Dqemu_path, args=3Dargs, + console_log=3Dself._console_log_path, + drain_console=3DTrue) guest.set_machine(self._config['machine']) guest.set_console() try: @@ -273,6 +282,8 @@ class BaseVM(object): raise atexit.register(self.shutdown) self._guest =3D guest + # Init console so we can start consuming the chars. + self.console_init() usernet_info =3D guest.qmp("human-monitor-command", command_line=3D"info usernet") self.ssh_port =3D None @@ -284,7 +295,9 @@ class BaseVM(object): raise Exception("Cannot find ssh port from 'info usernet':\n%s= " % \ usernet_info) =20 - def console_init(self, timeout =3D 120): + def console_init(self, timeout =3D None): + if timeout =3D=3D None: + timeout =3D self.socket_timeout vm =3D self._guest vm.console_socket.settimeout(timeout) =20 @@ -559,6 +572,8 @@ def parse_args(vmcls): help=3D"Show console during boot. ") parser.add_option("--build-path", default=3DNone, help=3D"Path of build directory. ") + parser.add_option("--log-console", action=3D"store_true", + help=3D"Log console to file.") parser.disable_interspersed_args() return parser.parse_args() =20 @@ -575,7 +590,8 @@ def main(vmcls, config=3DNone): else logging.WARN)) vm =3D vmcls(debug=3Dargs.debug, vcpus=3Dargs.jobs, config=3Dconfi= g, boot_console=3Dargs.boot_console, - build_path=3Dargs.build_path) + build_path=3Dargs.build_path, + log_console=3Dargs.log_console) if args.build_image: if os.path.exists(args.image) and not args.force: sys.stderr.writelines(["Image file exists: %s\n" % args.im= age, --=20 2.17.1