From nobody Sun May 19 14:14:26 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=linaro.org ARC-Seal: i=1; a=rsa-sha256; t=1588262960; cv=none; d=zohomail.com; s=zohoarc; b=IuJmymXP12vXtMQuQfZ6Oz/SA/xCD1uTQ/4XSYR2TIYdbuGIS15n/0E8fnAWnhfiCmN7kRETojvC4jIL9f34CafdcbofsL60JjH5i6VYrm2XyzO3QvoKaqc+My5ResifDMh7OvJVRSrWGkZehXdyOtXW5uX5SKdLVvm0TVdWv5g= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1588262960; h=Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:Message-ID:References:Sender:Subject:To; bh=IHVxQvHLGmuAsOe7Our5n+V2vriioUviLGm8LCdzEg4=; b=XInTqq/n9qlSEjA8bh7HQSsz99iqwjaqTWAvkUM+QJelB0b0iTo19a1QATw3wH6igN4YamPpv6vYPTnQjKpQR3wc5QOLuGEfw2sIsTDvU8k6RM7GkFwRL1lgBha1gPk1DRJD9Z45cWiPuQakMipy7RJMFdRYqEXGb6hgF6NK20k= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1588262960667757.0670714738344; Thu, 30 Apr 2020 09:09:20 -0700 (PDT) Received: from localhost ([::1]:48948 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jUBkV-0004LY-7i for importer@patchew.org; Thu, 30 Apr 2020 12:09:19 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:45424) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jUBdJ-0000K7-S2 for qemu-devel@nongnu.org; Thu, 30 Apr 2020 12:03:11 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.90_1) (envelope-from ) id 1jUBaA-0003BM-Oc for qemu-devel@nongnu.org; Thu, 30 Apr 2020 12:01:53 -0400 Received: from mail-qt1-x82e.google.com ([2607:f8b0:4864:20::82e]:46585) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1jUBaA-00037m-BE for qemu-devel@nongnu.org; Thu, 30 Apr 2020 11:58:38 -0400 Received: by mail-qt1-x82e.google.com with SMTP id g26so1542944qtv.13 for ; Thu, 30 Apr 2020 08:58:37 -0700 (PDT) Received: from Rfoley-MA01.hsd1.ma.comcast.net ([2601:199:4480:60c0:7489:3328:3806:7c3]) by smtp.gmail.com with ESMTPSA id v16sm305254qkf.80.2020.04.30.08.58.34 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 30 Apr 2020 08:58:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=IHVxQvHLGmuAsOe7Our5n+V2vriioUviLGm8LCdzEg4=; b=MmW/LqAQ/3GA+kw9lylNLGRPSk9fz1/zyjeb3z9ccqa7Qjoo9NAfWIBBqLQ4lLV0n5 6zhbYeTuj9SyAEsH0xhYOTLTD8LQw+zW/2N84U0knUvt1vPWgvMtoTymF5hGiMrBv//r PKH/T7Ldb7sJH9UsmUGVQlHrJCF6xDhAt1iHFxuzEZLiBb5AUj/wY5dQu/kWtKPiTt+9 PFM3wayMyyQ9Ky47wxJ5WCHZOQ20z/pFyk3Fhbl8QjHVmCfc8+HGohf293WdRAhc9Ek3 FFf3HDm/1l2F7HzcLk2xCHAuL4EjewziA23DqLtXMZaZvb2/C84eIaHpgjVBb8NiCk5b 7NAQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=IHVxQvHLGmuAsOe7Our5n+V2vriioUviLGm8LCdzEg4=; b=Uugil+dRGicAjSudnJBQcPC/hSxgVjQ+a7ZcBK9g4i6Ll5Xe4u2+9aqogkhSpZUOJK ajRKWY+yRmlZcTkCzNdzWHGCR1jodx1H1IgQS3duoNFYU9DuH4wfiDSTZb0ZYqnvzwFD EQAtWgGDyMDzilYbxdmjNSHYS1ZREe0jYCYbncNyVbR+iXOB1dyRRfMxqH04Rc3cvfms mf8ZbizXrWElNwx76jXq9tlmrXv595WmVsRN6EYnDBwTbLLW+GxuYv/3oJxzi+yNuT7p LrvpbSSEw65q1XE1akkVDwmFIW2KWeiW4dV/RP4YdNpnmlqk3wU+2p243dyQNc8ZIk36 q8dQ== X-Gm-Message-State: AGi0PuZGq6KmA/IbjHigqOU4NkE5M/mR1Aly7tRhDyOcbQB0jSUNI57m UIolqENF9gIJuZlzMbG8Nh4nBNwo/h/8Dg== X-Google-Smtp-Source: APiQypKamrj7kUvClTb99EPrt5lwyxI2/eWnpgOjxpQUIQMIAYuN9b1UnsS+/MAO0MFYkxpGB9Da+A== X-Received: by 2002:aed:26e2:: with SMTP id q89mr4304756qtd.60.1588262316353; Thu, 30 Apr 2020 08:58:36 -0700 (PDT) From: Robert Foley To: qemu-devel@nongnu.org Subject: [PATCH v5 1/9] tests/vm: Add configuration to basevm.py Date: Thu, 30 Apr 2020 11:57:10 -0400 Message-Id: <20200430155718.212-2-robert.foley@linaro.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20200430155718.212-1-robert.foley@linaro.org> References: <20200430155718.212-1-robert.foley@linaro.org> Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=2607:f8b0:4864:20::82e; envelope-from=robert.foley@linaro.org; helo=mail-qt1-x82e.google.com X-detected-operating-system: by eggs.gnu.org: Error: [-] PROGRAM ABORT : Malformed IPv6 address (bad octet value). Location : parse_addr6(), p0f-client.c:67 X-Received-From: 2607:f8b0:4864:20::82e X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: philmd@redhat.com, alex.bennee@linaro.org, robert.foley@linaro.org, peter.puhov@linaro.org Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Added use of a configuration to tests/vm/basevm.py. The configuration provides parameters used to configure a VM. This allows for providing alternate configurations to the VM being created/launched. cpu, machine, memory, and NUMA configuration are all examples of configuration which we might want to vary on the VM being creat= ed or launched. This will for example allow for creating an aarch64 vm. Signed-off-by: Robert Foley Reviewed-by: Peter Puhov --- tests/vm/basevm.py | 170 ++++++++++++++++++++++++++++++++++----------- 1 file changed, 131 insertions(+), 39 deletions(-) diff --git a/tests/vm/basevm.py b/tests/vm/basevm.py index 756ccf7aca..fb93a111c1 100644 --- a/tests/vm/basevm.py +++ b/tests/vm/basevm.py @@ -29,16 +29,41 @@ import tempfile import shutil import multiprocessing import traceback - -SSH_KEY =3D open(os.path.join(os.path.dirname(__file__), - "..", "keys", "id_rsa")).read() -SSH_PUB_KEY =3D open(os.path.join(os.path.dirname(__file__), - "..", "keys", "id_rsa.pub")).read() - +import shlex + +SSH_KEY_FILE =3D os.path.join(os.path.dirname(__file__), + "..", "keys", "id_rsa") +SSH_PUB_KEY_FILE =3D os.path.join(os.path.dirname(__file__), + "..", "keys", "id_rsa.pub") + +# This is the standard configuration. +# Any or all of these can be overridden by +# passing in a config argument to the VM constructor. +DEFAULT_CONFIG =3D { + 'cpu' : "max", + 'machine' : 'pc', + 'guest_user' : "qemu", + 'guest_pass' : "qemupass", + 'root_pass' : "qemupass", + 'ssh_key_file' : SSH_KEY_FILE, + 'ssh_pub_key_file': SSH_PUB_KEY_FILE, + 'memory' : "4G", + 'extra_args' : [], + 'qemu_args' : "", + 'dns' : "", + 'ssh_port' : 0, + 'install_cmds' : "", + 'boot_dev_type' : "block", + 'ssh_timeout' : 1, +} +BOOT_DEVICE =3D { + 'block' : "-drive file=3D{},if=3Dnone,id=3Ddrive0,cache=3Dwriteback "\ + "-device virtio-blk,drive=3Ddrive0,bootindex=3D0", + 'scsi' : "-device virtio-scsi-device,id=3Dscsi "\ + "-drive file=3D{},format=3Draw,if=3Dnone,id=3Dhd0 "\ + "-device scsi-hd,drive=3Dhd0,bootindex=3D0", +} class BaseVM(object): - GUEST_USER =3D "qemu" - GUEST_PASS =3D "qemupass" - ROOT_PASS =3D "qemupass" =20 envvars =3D [ "https_proxy", @@ -61,19 +86,30 @@ class BaseVM(object): # 4 is arbitrary, but greater than 2, # since we found we need to wait more than twice as long. tcg_ssh_timeout_multiplier =3D 4 - def __init__(self, debug=3DFalse, vcpus=3DNone): + def __init__(self, debug=3DFalse, vcpus=3DNone, config=3DNone): self._guest =3D None + # Allow input config to override defaults. + self._config =3D DEFAULT_CONFIG.copy() + if config !=3D None: + self._config.update(config) + self.validate_ssh_keys() self._tmpdir =3D os.path.realpath(tempfile.mkdtemp(prefix=3D"vm-te= st-", suffix=3D".tmp", dir=3D".")) atexit.register(shutil.rmtree, self._tmpdir) - - self._ssh_key_file =3D os.path.join(self._tmpdir, "id_rsa") - open(self._ssh_key_file, "w").write(SSH_KEY) - subprocess.check_call(["chmod", "600", self._ssh_key_file]) - - self._ssh_pub_key_file =3D os.path.join(self._tmpdir, "id_rsa.pub") - open(self._ssh_pub_key_file, "w").write(SSH_PUB_KEY) + # Copy the key files to a temporary directory. + # Also chmod the key file to agree with ssh requirements. + self._config['ssh_key'] =3D \ + open(self._config['ssh_key_file']).read().rstrip() + self._config['ssh_pub_key'] =3D \ + open(self._config['ssh_pub_key_file']).read().rstrip() + self._ssh_tmp_key_file =3D os.path.join(self._tmpdir, "id_rsa") + open(self._ssh_tmp_key_file, "w").write(self._config['ssh_key']) + subprocess.check_call(["chmod", "600", self._ssh_tmp_key_file]) + + self._ssh_tmp_pub_key_file =3D os.path.join(self._tmpdir, "id_rsa.= pub") + open(self._ssh_tmp_pub_key_file, + "w").write(self._config['ssh_pub_key']) =20 self.debug =3D debug self._stderr =3D sys.stderr @@ -82,11 +118,14 @@ class BaseVM(object): self._stdout =3D sys.stdout else: self._stdout =3D self._devnull + netdev =3D "user,id=3Dvnet,hostfwd=3D:127.0.0.1:{}-:22" self._args =3D [ \ - "-nodefaults", "-m", "4G", - "-cpu", "max", - "-netdev", "user,id=3Dvnet,hostfwd=3D:127.0.0.1:0-:22" + - (",ipv6=3Dno" if not self.ipv6 else ""), + "-nodefaults", "-m", self._config['memory'], + "-cpu", self._config['cpu'], + "-netdev", + netdev.format(self._config['ssh_port']) + + (",ipv6=3Dno" if not self.ipv6 else "") + + (",dns=3D" + self._config['dns'] if self._config['dns'] else "= "), "-device", "virtio-net-pci,netdev=3Dvnet", "-vnc", "127.0.0.1:0,to=3D20"] if vcpus and vcpus > 1: @@ -97,6 +136,55 @@ class BaseVM(object): logging.info("KVM not available, not using -enable-kvm") self._data_args =3D [] =20 + if self._config['qemu_args'] !=3D None: + qemu_args =3D self._config['qemu_args'] + qemu_args =3D qemu_args.replace('\n',' ').replace('\r','') + # shlex groups quoted arguments together + # we need this to keep the quoted args together for when + # the QEMU command is issued later. + args =3D shlex.split(qemu_args) + self._config['extra_args'] =3D [] + for arg in args: + if arg: + # Preserve quotes around arguments. + # shlex above takes them out, so add them in. + if " " in arg: + arg =3D '"{}"'.format(arg) + self._config['extra_args'].append(arg) + + def validate_ssh_keys(self): + """Check to see if the ssh key files exist.""" + if 'ssh_key_file' not in self._config or\ + not os.path.exists(self._config['ssh_key_file']): + raise Exception("ssh key file not found.") + if 'ssh_pub_key_file' not in self._config or\ + not os.path.exists(self._config['ssh_pub_key_file']): + raise Exception("ssh pub key file not found.") + + def wait_boot(self, wait_string=3DNone): + """Wait for the standard string we expect + on completion of a normal boot. + The user can also choose to override with an + alternate string to wait for.""" + if wait_string is None: + if self.login_prompt is None: + raise Exception("self.login_prompt not defined") + wait_string =3D self.login_prompt + # Intentionally bump up the default timeout under TCG, + # since the console wait below takes longer. + timeout =3D self.socket_timeout + if not kvm_available(self.arch): + timeout *=3D 8 + self.console_init(timeout=3Dtimeout) + self.console_wait(wait_string) + + def __getattr__(self, name): + # Support direct access to config by key. + # for example, access self._config['cpu'] by self.cpu + if name.lower() in self._config.keys(): + return self._config[name.lower()] + return object.__getattribute__(self, name) + def _download_with_cache(self, url, sha256sum=3DNone, sha512sum=3DNone= ): def check_sha256sum(fname): if not sha256sum: @@ -128,8 +216,9 @@ class BaseVM(object): "-t", "-o", "StrictHostKeyChecking=3Dno", "-o", "UserKnownHostsFile=3D" + os.devnull, - "-o", "ConnectTimeout=3D1", - "-p", self.ssh_port, "-i", self._ssh_key_file] + "-o", + "ConnectTimeout=3D{}".format(self._config["ssh_timeout"= ]), + "-p", self.ssh_port, "-i", self._ssh_tmp_key_file] # If not in debug mode, set ssh to quiet mode to # avoid printing the results of commands. if not self.debug: @@ -178,14 +267,14 @@ class BaseVM(object): "virtio-blk,drive=3D%s,serial=3D%s,bootindex= =3D1" % (name, name)] =20 def boot(self, img, extra_args=3D[]): - args =3D self._args + [ - "-drive", "file=3D%s,if=3Dnone,id=3Ddrive0,cache=3Dwriteback" = % img, - "-device", "virtio-blk,drive=3Ddrive0,bootindex=3D0"] - args +=3D self._data_args + extra_args + boot_dev =3D BOOT_DEVICE[self._config['boot_dev_type']] + boot_params =3D boot_dev.format(img) + args =3D self._args + boot_params.split(' ') + args +=3D self._data_args + extra_args + self._config['extra_args'] logging.debug("QEMU args: %s", " ".join(args)) qemu_bin =3D os.environ.get("QEMU", "qemu-system-" + self.arch) guest =3D QEMUMachine(binary=3Dqemu_bin, args=3Dargs) - guest.set_machine('pc') + guest.set_machine(self._config['machine']) guest.set_console() try: guest.launch() @@ -299,7 +388,8 @@ class BaseVM(object): self.console_send(command) =20 def console_ssh_init(self, prompt, user, pw): - sshkey_cmd =3D "echo '%s' > .ssh/authorized_keys\n" % SSH_PUB_KEY.= rstrip() + sshkey_cmd =3D "echo '%s' > .ssh/authorized_keys\n" \ + % self._config['ssh_pub_key'].rstrip() self.console_wait_send("login:", "%s\n" % user) self.console_wait_send("Password:", "%s\n" % pw) self.console_wait_send(prompt, "mkdir .ssh\n") @@ -358,23 +448,23 @@ class BaseVM(object): "local-hostname: {}-guest\n".format(name)]) mdata.close() udata =3D open(os.path.join(cidir, "user-data"), "w") - print("guest user:pw {}:{}".format(self.GUEST_USER, - self.GUEST_PASS)) + print("guest user:pw {}:{}".format(self._config['guest_user'], + self._config['guest_pass'])) udata.writelines(["#cloud-config\n", "chpasswd:\n", " list: |\n", - " root:%s\n" % self.ROOT_PASS, - " %s:%s\n" % (self.GUEST_USER, - self.GUEST_PASS), + " root:%s\n" % self._config['root_pass'], + " %s:%s\n" % (self._config['guest_user'], + self._config['guest_pass']), " expire: False\n", "users:\n", - " - name: %s\n" % self.GUEST_USER, + " - name: %s\n" % self._config['guest_user'], " sudo: ALL=3D(ALL) NOPASSWD:ALL\n", " ssh-authorized-keys:\n", - " - %s\n" % SSH_PUB_KEY, + " - %s\n" % self._config['ssh_pub_key'], " - name: root\n", " ssh-authorized-keys:\n", - " - %s\n" % SSH_PUB_KEY, + " - %s\n" % self._config['ssh_pub_key'], "locale: en_US.UTF-8\n"]) proxy =3D os.environ.get("http_proxy") if not proxy is None: @@ -427,15 +517,17 @@ def parse_args(vmcls): parser.disable_interspersed_args() return parser.parse_args() =20 -def main(vmcls): +def main(vmcls, config=3DNone): try: + if config =3D=3D None: + config =3D {} args, argv =3D parse_args(vmcls) if not argv and not args.build_qemu and not args.build_image: print("Nothing to do?") return 1 logging.basicConfig(level=3D(logging.DEBUG if args.debug else logging.WARN)) - vm =3D vmcls(debug=3Dargs.debug, vcpus=3Dargs.jobs) + vm =3D vmcls(debug=3Dargs.debug, vcpus=3Dargs.jobs, config=3Dconfi= g) if args.build_image: if os.path.exists(args.image) and not args.force: sys.stderr.writelines(["Image file exists: %s\n" % args.im= age, --=20 2.17.1 From nobody Sun May 19 14:14:26 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=linaro.org ARC-Seal: i=1; a=rsa-sha256; t=1588263141; cv=none; d=zohomail.com; s=zohoarc; b=J6tc9+4AYFENQekR/lq7pLWwY207qHHPAjIosQQicc0tIb05vsXqYJnQFPtvawYCB8GbZj1JB8o7G80BqVWxLM4We6yEzETKNGg9WRAg6OBvDNAu6FGrpY3oDidAjV+TYEDrN5Km80yCbLG5hX/6MGDnCiUqwTT6UQDgeLLa2N4= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1588263141; h=Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:Message-ID:References:Sender:Subject:To; bh=YCSiLeyYTeL4ASqV7GKK96BGQv7O117PTDXpRucreq4=; b=P39VBXLBLX7OoYqjeoPQ+bhB4KgZCOkQAyUGEEe9RV6DZxPrLYYGmbHTF2oVSfHeixtolY3O8XB028GcF5RdEOOd3KRkDKwk0riip6vJKeMZSxYLeok0B7JmAmib3bIvZmjI3FLLpKyeW6HTrm5aQgFWYxH61GCMLZXUX85BXoo= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1588263141753920.9239656578474; Thu, 30 Apr 2020 09:12:21 -0700 (PDT) Received: from localhost ([::1]:60348 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jUBnQ-0000kH-Aw for importer@patchew.org; Thu, 30 Apr 2020 12:12:20 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:45432) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jUBdT-0000T4-IC for qemu-devel@nongnu.org; Thu, 30 Apr 2020 12:03:13 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.90_1) (envelope-from ) id 1jUBaC-0003C1-7r for qemu-devel@nongnu.org; Thu, 30 Apr 2020 12:02:03 -0400 Received: from mail-qt1-x843.google.com ([2607:f8b0:4864:20::843]:41496) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1jUBaB-0003Bg-Os for qemu-devel@nongnu.org; Thu, 30 Apr 2020 11:58:39 -0400 Received: by mail-qt1-x843.google.com with SMTP id h26so5386033qtu.8 for ; Thu, 30 Apr 2020 08:58:39 -0700 (PDT) Received: from Rfoley-MA01.hsd1.ma.comcast.net ([2601:199:4480:60c0:7489:3328:3806:7c3]) by smtp.gmail.com with ESMTPSA id v16sm305254qkf.80.2020.04.30.08.58.36 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 30 Apr 2020 08:58:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=YCSiLeyYTeL4ASqV7GKK96BGQv7O117PTDXpRucreq4=; b=VZwq++7L4VlPvwUWqQLxL6IBKZQPViQKm1C8e0toYx5CzI2xt2InSarHVNP/S2pxAc jLH4BOLaeGdgs4aXbNNWn9UwP89CV0b871x5UFUUiDk57dMECSzS0DZTgvR5VgX80tam M8vrS1mfLbQYSV5DfR2Tyap7b8PC7tMsT4f1DTAStMnwlqrazSlAFsuUHxIs7hyWE6r0 eWLMqnmSQAmVELhZHCPSwzu/MhZBKjWMvd3kakwl6PUUU1GwcKCmxQf5mQASd0ehmSuH 9MHhYYrlwW0Wg+eh20bfl3OszGW2oYMB6Y5VdLnyX3/TK7O9vao+96R+7IFTz6daDCAk JI9w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=YCSiLeyYTeL4ASqV7GKK96BGQv7O117PTDXpRucreq4=; b=KnrGzx6JTcVKzFRfaoJ0t8dxf4MsoqcgPE2lGy8VJhQJtXM1cRCMWksixLyYrMAoi7 yI/90X7lNJg9Y8MGf3QOL552ElSJpUAc4m/ALbam9MPIxnh5FyA1VLER0Us0TdLHlybf 33Nr1t3v/cKM06kFiwh0VgljoUno3QJhFFu6yNUXPmf5pDgnSI2GNN4YZaC2RyvDge+p KfALJhjtwStShNYmzqehO8Mt+xYfJNL8IBI6BXSmyl0bV5Ww9Hn3hHvAETTW9tVYbRKa qa4HW1XbHTn27yxFjFwXvicON1RPN4y8DvMjRiN6l0vjf6JkqfuAIEEG5EGR0NhjdO78 s9Pw== X-Gm-Message-State: AGi0PuYrLOblZEwEY3DbLwl/E/Yt9xf0EH4Pd29jFXfKFUkz1nKyPixz MQXvdxm5sAUqqbXhz/JLnqTHitBpLP53NA== X-Google-Smtp-Source: APiQypLKJ4QWscw3ACFVbCRbjTFrzoC8voLGqgOYIypNMKbRaLZgqGRv8UPI5zxYU9/Gkhs1hquZyQ== X-Received: by 2002:ac8:6751:: with SMTP id n17mr4560171qtp.16.1588262317813; Thu, 30 Apr 2020 08:58:37 -0700 (PDT) From: Robert Foley To: qemu-devel@nongnu.org Subject: [PATCH v5 2/9] tests/vm: Added configuration file support Date: Thu, 30 Apr 2020 11:57:11 -0400 Message-Id: <20200430155718.212-3-robert.foley@linaro.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20200430155718.212-1-robert.foley@linaro.org> References: <20200430155718.212-1-robert.foley@linaro.org> Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=2607:f8b0:4864:20::843; envelope-from=robert.foley@linaro.org; helo=mail-qt1-x843.google.com X-detected-operating-system: by eggs.gnu.org: Error: [-] PROGRAM ABORT : Malformed IPv6 address (bad octet value). Location : parse_addr6(), p0f-client.c:67 X-Received-From: 2607:f8b0:4864:20::843 X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: philmd@redhat.com, alex.bennee@linaro.org, robert.foley@linaro.org, peter.puhov@linaro.org Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Changes to tests/vm/basevm.py to allow accepting a configuration file as a parameter. Allows for specifying VM options such as cpu, machine, memory, and arbitrary qemu arguments for specifying options such as NUMA configuration. Also added an example conf_example_aarch64.yml and conf_example_x86.yml. Signed-off-by: Robert Foley Reviewed-by: Peter Puhov --- configure | 9 ++++++ tests/vm/Makefile.include | 6 ++++ tests/vm/basevm.py | 40 +++++++++++++++++++++++- tests/vm/conf_example_aarch64.yml | 51 +++++++++++++++++++++++++++++++ tests/vm/conf_example_x86.yml | 50 ++++++++++++++++++++++++++++++ 5 files changed, 155 insertions(+), 1 deletion(-) create mode 100644 tests/vm/conf_example_aarch64.yml create mode 100644 tests/vm/conf_example_x86.yml diff --git a/configure b/configure index 23b5e93752..eec6615b81 100755 --- a/configure +++ b/configure @@ -950,6 +950,13 @@ do fi done =20 +# Check for existence of python3 yaml, needed to +# import yaml config files into vm-build. +python_yaml=3D"no" +if $(python3 -c "import yaml" 2> /dev/null); then + python_yaml=3D"yes" +fi + : ${smbd=3D${SMBD-/usr/sbin/smbd}} =20 # Default objcc to clang if available, otherwise use CC @@ -6581,6 +6588,7 @@ if test "$docs" !=3D "no"; then echo "sphinx-build $sphinx_build" fi echo "genisoimage $genisoimage" +echo "python_yaml $python_yaml" echo "slirp support $slirp $(echo_version $slirp $slirp_version)" if test "$slirp" !=3D "no" ; then echo "smbd $smbd" @@ -7642,6 +7650,7 @@ echo "PYTHON=3D$python" >> $config_host_mak echo "SPHINX_BUILD=3D$sphinx_build" >> $config_host_mak echo "SPHINX_WERROR=3D$sphinx_werror" >> $config_host_mak echo "GENISOIMAGE=3D$genisoimage" >> $config_host_mak +echo "PYTHON_YAML=3D$python_yaml" >> $config_host_mak echo "CC=3D$cc" >> $config_host_mak if $iasl -h > /dev/null 2>&1; then echo "IASL=3D$iasl" >> $config_host_mak diff --git a/tests/vm/Makefile.include b/tests/vm/Makefile.include index 1bf9693d19..78a2de1f82 100644 --- a/tests/vm/Makefile.include +++ b/tests/vm/Makefile.include @@ -43,6 +43,12 @@ endif @echo " V=3D1 - Enable verbose ouput on host and guest commands" @echo " QEMU=3D/path/to/qemu - Change path to QEMU binary" @echo " QEMU_IMG=3D/path/to/qemu-img - Change path to qemu-img tool" +ifeq ($(PYTHON_YAML),yes) + @echo " QEMU_CONFIG=3D/path/conf.yml - Change path to VM configurati= on .yml file." +else + @echo " (install python3-yaml to enable support for yaml file to confi= gure a VM.)" +endif + @echo " See conf_example_*.yml for file= format details." =20 vm-build-all: $(addprefix vm-build-, $(IMAGES)) =20 diff --git a/tests/vm/basevm.py b/tests/vm/basevm.py index fb93a111c1..97f8ac4218 100644 --- a/tests/vm/basevm.py +++ b/tests/vm/basevm.py @@ -477,9 +477,43 @@ class BaseVM(object): cwd=3Dcidir, stdin=3Dself._devnull, stdout=3Dself._stdou= t, stderr=3Dself._stdout) - return os.path.join(cidir, "cloud-init.iso") =20 +def parse_config(config, args): + """ Parse yaml config and populate our config structure. + The yaml config allows the user to override the + defaults for VM parameters. In many cases these + defaults can be overridden without rebuilding the VM.""" + if args.config: + config_file =3D args.config + elif 'QEMU_CONFIG' in os.environ: + config_file =3D os.environ['QEMU_CONFIG'] + else: + return config + if not os.path.exists(config_file): + raise Exception("config file {} does not exist".format(config_file= )) + # We gracefully handle importing the yaml module + # since it might not be installed. + # If we are here it means the user supplied a .yml file, + # so if the yaml module is not installed we will exit with error. + try: + import yaml + except ImportError: + print("The python3-yaml package is needed "\ + "to support config.yaml files") + # Instead of raising an exception we exit to avoid + # a raft of messy (expected) errors to stdout. + exit(1) + with open(config_file) as f: + yaml_dict =3D yaml.safe_load(f) + + if 'qemu-conf' in yaml_dict: + config.update(yaml_dict['qemu-conf']) + else: + raise Exception("config file {} is not valid"\ + " missing qemu-conf".format(config_file)) + return config + def parse_args(vmcls): =20 def get_default_jobs(): @@ -514,6 +548,9 @@ def parse_args(vmcls): help=3D"Interactively run command") parser.add_option("--snapshot", "-s", action=3D"store_true", help=3D"run tests with a snapshot") + parser.add_option("--config", "-c", default=3DNone, + help=3D"Provide config yaml for configuration. "\ + "See config_example.yaml for example.") parser.disable_interspersed_args() return parser.parse_args() =20 @@ -525,6 +562,7 @@ def main(vmcls, config=3DNone): if not argv and not args.build_qemu and not args.build_image: print("Nothing to do?") return 1 + config =3D parse_config(config, args) logging.basicConfig(level=3D(logging.DEBUG if args.debug else logging.WARN)) vm =3D vmcls(debug=3Dargs.debug, vcpus=3Dargs.jobs, config=3Dconfi= g) diff --git a/tests/vm/conf_example_aarch64.yml b/tests/vm/conf_example_aarc= h64.yml new file mode 100644 index 0000000000..9d44ae356f --- /dev/null +++ b/tests/vm/conf_example_aarch64.yml @@ -0,0 +1,51 @@ +# +# Example yaml for use by any of the scripts in tests/vm. +# Can be provided as an environment variable QEMU_CONFIG +# +qemu-conf: + + # If any of the below are not provided, we will just use the qemu defa= ults. + + # Login username and password(has to be sudo enabled) + guest_user: qemu + guest_pass: "qemupass" + + # Password for root user can be different from guest. + root_pass: "qemupass" + + # If one key is provided, both must be provided. + #ssh_key: /complete/path/of/your/keyfile/id_rsa + #ssh_pub_key: /complete/path/of/your/keyfile/id_rsa.pub + + cpu: max + machine: virt,gic-version=3Dmax + memory: 16G + + # The below is a example for how to configure NUMA topology with + # 4 NUMA nodes and 2 different NUMA distances. + qemu_args: "-smp cpus=3D16,sockets=3D2,cores=3D8 + -numa node,cpus=3D0-3,nodeid=3D0 -numa node,cpus=3D4-7,nod= eid=3D1 + -numa node,cpus=3D8-11,nodeid=3D2 -numa node,cpus=3D12-15,= nodeid=3D3 + -numa dist,src=3D0,dst=3D1,val=3D15 -numa dist,src=3D2,dst= =3D3,val=3D15 + -numa dist,src=3D0,dst=3D2,val=3D20 -numa dist,src=3D0,dst= =3D3,val=3D20 + -numa dist,src=3D1,dst=3D2,val=3D20 -numa dist,src=3D1,dst= =3D3,val=3D20" + + # By default we do not set the DNS. + # You override the defaults by setting the below. + #dns: 1.234.567.89 + + # By default we will use a "block" device, but + # you can also boot from a "scsi" device. + # Just keep in mind your scripts might need to change + # As you will have /dev/sda instead of /dev/vda (for block device) + boot_dev_type: "block" + + # By default the ssh port is not fixed. + # A fixed ssh port makes it easier for automated tests. + #ssh_port: 5555 + + # To install a different set of packages, provide a command to issue + #install_cmds: "apt-get update ; apt-get build-dep -y qemu" + + # Or to skip the install entirely, just provide "" + #install_cmds: "" diff --git a/tests/vm/conf_example_x86.yml b/tests/vm/conf_example_x86.yml new file mode 100644 index 0000000000..78d3f5830f --- /dev/null +++ b/tests/vm/conf_example_x86.yml @@ -0,0 +1,50 @@ +# +# Example yaml for use by any of the x86 based scripts in tests/vm. +# Can be provided as an environment variable QEMU_CONFIG +# +qemu-conf: + + # If any of the below are not provided, we will just use the qemu defa= ults. + + # Login username and password(has to be sudo enabled) + guest_user: "qemu" + guest_pass: "qemupass" + + # Password for root user can be different from guest. + root_pass: "qemupass" + + # Provide default ssh keys of current user. + # You need to edit the below for your user. + #ssh_key_file: /home//.ssh/id_rsa + #ssh_pub_key_file: /home//.ssh/id_rsa.pub + + cpu: max + machine: pc + memory: 8G + + # The below is a example for how to configure NUMA topology with + # 4 NUMA nodes and 2 different NUMA distances. + qemu_args: "-smp cpus=3D8,sockets=3D2,cores=3D4 + -object memory-backend-ram,size=3D4G,policy=3Dbind,host-no= des=3D0,id=3Dram-node0 + -object memory-backend-ram,size=3D4G,policy=3Dbind,host-no= des=3D0,id=3Dram-node1 + -object memory-backend-ram,size=3D4G,policy=3Dbind,host-no= des=3D1,id=3Dram-node2 + -object memory-backend-ram,size=3D4G,policy=3Dbind,host-no= des=3D1,id=3Dram-node3 + -numa node,cpus=3D0-1,nodeid=3D0 -numa node,cpus=3D2-3,nod= eid=3D1 + -numa node,cpus=3D4-5,nodeid=3D2 -numa node,cpus=3D6-7,nod= eid=3D3 + -numa dist,src=3D0,dst=3D1,val=3D15 -numa dist,src=3D2,dst= =3D3,val=3D15 + -numa dist,src=3D0,dst=3D2,val=3D20 -numa dist,src=3D0,dst= =3D3,val=3D20 + -numa dist,src=3D1,dst=3D2,val=3D20 -numa dist,src=3D1,dst= =3D3,val=3D20" + + # By default we do not set the DNS. + # You override the defaults by setting the below. + #dns: "1.234.567.89" + + # By default we will use a "block" device, but + # you can also boot from a "scsi" device. + # Just keep in mind your scripts might need to change + # As you will have /dev/sda instead of /dev/vda (for block device) + boot_dev_type: "block" + + # By default the ssh port is not fixed. + # A fixed ssh port makes it easier for automated tests. + ssh_port: 5555 --=20 2.17.1 From nobody Sun May 19 14:14:26 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=linaro.org ARC-Seal: i=1; a=rsa-sha256; t=1588263112; cv=none; d=zohomail.com; s=zohoarc; b=JDGf3J72nv+DsXDh1sETpxSAnk2py1S3awa5a2BZRbrkGBIlDLF/uh22r14sbP3uJ6vTRiRB2mMGEzQuU+VOci8XDiZimzxAVQmObjEeWXvv3HrZnng9yhnNGcYqI0driov1hb/8xO1oDtOFZ4LI09effebFq3mEPjGi6DRas3k= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1588263112; h=Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:Message-ID:References:Sender:Subject:To; bh=7n0OEOUcViSoi8lDFmvS1FDdqucfcjXxY1zrvZFIRGA=; b=S+wcXpv6oBdvdISXAytD7la1YlidIectWO2zWlTpkjSiXLIwzLAiJxVzaYt8kAGF1xLoB7+RrbabYUYpzCvzrcRrIM+GHwqfN5/MwvXk38WlOl/i7j8I2nGNa/HYpJtiWDhLe3d3MRjOa/BnJd1snOqj5dk2+60vXBiG6nSQ3Co= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1588263112629171.53576505243052; Thu, 30 Apr 2020 09:11:52 -0700 (PDT) Received: from localhost ([::1]:57812 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jUBmx-00089C-BY for importer@patchew.org; Thu, 30 Apr 2020 12:11:51 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:45436) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jUBdU-0000UU-Gg for qemu-devel@nongnu.org; Thu, 30 Apr 2020 12:03:13 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.90_1) (envelope-from ) id 1jUBaC-0003CJ-Tu for qemu-devel@nongnu.org; Thu, 30 Apr 2020 12:02:04 -0400 Received: from mail-qt1-x844.google.com ([2607:f8b0:4864:20::844]:40432) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1jUBaC-0003Bs-Eg for qemu-devel@nongnu.org; Thu, 30 Apr 2020 11:58:40 -0400 Received: by mail-qt1-x844.google.com with SMTP id e17so5377787qtp.7 for ; Thu, 30 Apr 2020 08:58:40 -0700 (PDT) Received: from Rfoley-MA01.hsd1.ma.comcast.net ([2601:199:4480:60c0:7489:3328:3806:7c3]) by smtp.gmail.com with ESMTPSA id v16sm305254qkf.80.2020.04.30.08.58.37 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 30 Apr 2020 08:58:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=7n0OEOUcViSoi8lDFmvS1FDdqucfcjXxY1zrvZFIRGA=; b=h5q1pEd1IHxUeE3X2QUkC6hHBI/quzW+osUrMvXMbkP26ag2wX+Rpys/RU1awmfLXZ vEhMmZz2odSY3BefYjilamPgqGPyLWO5R3S697pgNocfh2TIlzoMQDn7Pqo0iJr1p+Kh Q2F8QWja/Hz+X8EpKENKbq8NHroiMtV4eBcibLr2aLojHL1qu9AlNfqP6mdrzYMeKKq0 6TWWpNsXdFPDX32TmgAOE4hxDV4JxNGdO3JdYHvNzu3lO3p8vJwQ/pLtwrnT7Jr46N4f SkLhAyCGaHEh8+2RgImqvslORwIp0XtBhJY8P+2LJcjQMpB0cLruNhypHbDknXGxrGYi oduw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=7n0OEOUcViSoi8lDFmvS1FDdqucfcjXxY1zrvZFIRGA=; b=OS6OvI2j5NI9OHbJ0lK3gFQ344/4VfhCnhayo2sDf775ynn411bhzR9BeyFLPxH7fo cN5x84NXqH2erL6e/XUqhDGzpyXnaGiouG+iK7rmGeIBZvDo9vJKdME2gGPgMsoVO8o4 BEw7xgt80YOVuMXCqrh9yE3brA4n9g1ZLY4eo48qm/QKVo4mFp/J3E/TzvvDQpumhSlq pS7Ltc/0gP7Cn6/A5kKv4S//1GAdutNEPovWmCQKCXzYXGHJAbEpRcOvj3EnqbSfmAs4 ma7m7uCaDzuR6B8shn0Xh/nwaAEKVYcs88ry+M81wmYjtezJTmhqx9WkwqkXBg1fqb/p V2kQ== X-Gm-Message-State: AGi0PuZrFq1nyvYVvAJC6iu1iZ0jqr32eHAVQfaDwzaxNtWNcAMwcClF 4bszj+Bg3SBGKsCOE49X2DIGPMK7lPfj0w== X-Google-Smtp-Source: APiQypJhZKwtSGjR7Jrs14z2utCtaA2QY8s3pbxTEWpJMLZ4QMW5+xJE80iR85jaouamSrApYhG58Q== X-Received: by 2002:aed:3bda:: with SMTP id s26mr4329055qte.261.1588262319113; Thu, 30 Apr 2020 08:58:39 -0700 (PDT) From: Robert Foley To: qemu-devel@nongnu.org Subject: [PATCH v5 3/9] tests/vm: add --boot-console switch Date: Thu, 30 Apr 2020 11:57:12 -0400 Message-Id: <20200430155718.212-4-robert.foley@linaro.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20200430155718.212-1-robert.foley@linaro.org> References: <20200430155718.212-1-robert.foley@linaro.org> Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=2607:f8b0:4864:20::844; envelope-from=robert.foley@linaro.org; helo=mail-qt1-x844.google.com X-detected-operating-system: by eggs.gnu.org: Error: [-] PROGRAM ABORT : Malformed IPv6 address (bad octet value). Location : parse_addr6(), p0f-client.c:67 X-Received-From: 2607:f8b0:4864:20::844 X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: philmd@redhat.com, alex.bennee@linaro.org, robert.foley@linaro.org, peter.puhov@linaro.org Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Added ability to view console during boot via --boot-console switch to basevm.py. This helps debug issues that occur during the boot sequence. Also added a new special variable to vm-build: BOOT_CONSOLE=3D1 will cause this new --boot-console switch to be set. Signed-off-by: Robert Foley Reviewed-by: Peter Puhov --- tests/vm/Makefile.include | 4 ++++ tests/vm/basevm.py | 11 +++++++++-- 2 files changed, 13 insertions(+), 2 deletions(-) diff --git a/tests/vm/Makefile.include b/tests/vm/Makefile.include index 78a2de1f82..d921ee14cf 100644 --- a/tests/vm/Makefile.include +++ b/tests/vm/Makefile.include @@ -40,6 +40,7 @@ endif @echo ' EXTRA_CONFIGURE_OPTS=3D"..."' @echo " J=3D[0..9]* - Override the -jN parameter for make= commands" @echo " DEBUG=3D1 - Enable verbose output on host and i= nteractive debugging" + @echo " BOOT_CONSOLE=3D1 - Show the console output at boot ti= me. " @echo " V=3D1 - Enable verbose ouput on host and guest commands" @echo " QEMU=3D/path/to/qemu - Change path to QEMU binary" @echo " QEMU_IMG=3D/path/to/qemu-img - Change path to qemu-img tool" @@ -62,6 +63,7 @@ $(IMAGES_DIR)/%.img: $(SRC_PATH)/tests/vm/% \ $(call quiet-command, \ $(PYTHON) $< \ $(if $(V)$(DEBUG), --debug) \ + $(if $(BOOT_CONSOLE),--boot-console) \ --image "$@" \ --force \ --build-image $@, \ @@ -76,6 +78,7 @@ vm-build-%: $(IMAGES_DIR)/%.img $(if $(DEBUG), --interactive) \ $(if $(J),--jobs $(J)) \ $(if $(V),--verbose) \ + $(if $(BOOT_CONSOLE),--boot-console) \ --image "$<" \ $(if $(BUILD_TARGET),--build-target $(BUILD_TARGET)) \ --snapshot \ @@ -96,6 +99,7 @@ vm-boot-ssh-%: $(IMAGES_DIR)/%.img $(call quiet-command, \ $(PYTHON) $(SRC_PATH)/tests/vm/$* \ $(if $(J),--jobs $(J)) \ + $(if $(BOOT_CONSOLE),--boot-console) \ --image "$<" \ --interactive \ false, \ diff --git a/tests/vm/basevm.py b/tests/vm/basevm.py index 97f8ac4218..983adbf6ff 100644 --- a/tests/vm/basevm.py +++ b/tests/vm/basevm.py @@ -86,8 +86,10 @@ class BaseVM(object): # 4 is arbitrary, but greater than 2, # since we found we need to wait more than twice as long. tcg_ssh_timeout_multiplier =3D 4 - def __init__(self, debug=3DFalse, vcpus=3DNone, config=3DNone): + def __init__(self, debug=3DFalse, vcpus=3DNone, config=3DNone, + boot_console=3DNone): self._guest =3D None + self._boot_console =3D boot_console # Allow input config to override defaults. self._config =3D DEFAULT_CONFIG.copy() if config !=3D None: @@ -551,6 +553,8 @@ def parse_args(vmcls): parser.add_option("--config", "-c", default=3DNone, help=3D"Provide config yaml for configuration. "\ "See config_example.yaml for example.") + parser.add_option("--boot-console", action=3D"store_true", + help=3D"Show console during boot. ") parser.disable_interspersed_args() return parser.parse_args() =20 @@ -565,7 +569,8 @@ def main(vmcls, config=3DNone): config =3D parse_config(config, args) logging.basicConfig(level=3D(logging.DEBUG if args.debug else logging.WARN)) - vm =3D vmcls(debug=3Dargs.debug, vcpus=3Dargs.jobs, config=3Dconfi= g) + vm =3D vmcls(debug=3Dargs.debug, vcpus=3Dargs.jobs, config=3Dconfi= g, + boot_console=3Dargs.boot_console) if args.build_image: if os.path.exists(args.image) and not args.force: sys.stderr.writelines(["Image file exists: %s\n" % args.im= age, @@ -585,6 +590,8 @@ def main(vmcls, config=3DNone): if args.snapshot: img +=3D ",snapshot=3Don" vm.boot(img) + if vm._boot_console: + vm.wait_boot() vm.wait_ssh() except Exception as e: if isinstance(e, SystemExit) and e.code =3D=3D 0: --=20 2.17.1 From nobody Sun May 19 14:14:26 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=linaro.org ARC-Seal: i=1; a=rsa-sha256; t=1588263748; cv=none; d=zohomail.com; s=zohoarc; b=Rz7PKBvGCipj+QJeN3b1gaEp++OmwWT9HnzswVI3383BiDpjte+ck8en88lTuTN3wnuezV9Qfc/Tz5NklwSCsR3vycH6sjot/PdNqykoCani5MtWK4snzSmXt6gAAcqEK55uB8glPrFLoilbNwoWZpTMiCHjRH5LTLXCCIyo78M= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1588263748; h=Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:Message-ID:References:Sender:Subject:To; bh=YmekmIbOKaWK/Uh2GRis93ucBWqUHLXkmR51MffmvwM=; b=G2SJowT7oox5WloaCiIin63XXh/VXFMngPf+Goi+s+X/qvYyIRFIxtg6Glkcnu2rVF8b+fvfETs6s0IihNnLGVA9egeKX+Z2H2RThPfvA8R1gORsHtq98JgT9/9JM0qnlipkSoI0jS0V5+5Xzd+Av15C28by9E8NLPmHLTN3z1M= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 158826374879167.92243514647816; Thu, 30 Apr 2020 09:22:28 -0700 (PDT) Received: from localhost ([::1]:33188 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jUBxD-0000Xt-H8 for importer@patchew.org; Thu, 30 Apr 2020 12:22:27 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:45440) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jUBdW-0000Wi-Ep for qemu-devel@nongnu.org; Thu, 30 Apr 2020 12:03:14 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.90_1) (envelope-from ) id 1jUBaE-0003Gy-Bn for qemu-devel@nongnu.org; Thu, 30 Apr 2020 12:02:06 -0400 Received: from mail-qt1-x842.google.com ([2607:f8b0:4864:20::842]:44395) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1jUBaD-0003CV-UM for qemu-devel@nongnu.org; Thu, 30 Apr 2020 11:58:41 -0400 Received: by mail-qt1-x842.google.com with SMTP id g16so1429969qtp.11 for ; Thu, 30 Apr 2020 08:58:41 -0700 (PDT) Received: from Rfoley-MA01.hsd1.ma.comcast.net ([2601:199:4480:60c0:7489:3328:3806:7c3]) by smtp.gmail.com with ESMTPSA id v16sm305254qkf.80.2020.04.30.08.58.39 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 30 Apr 2020 08:58:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=YmekmIbOKaWK/Uh2GRis93ucBWqUHLXkmR51MffmvwM=; b=tiYP9aEsBbwABMk8frZpnVcjawPSWZUoCUeuATLvFyBCa0nOC1lfp0mCRBjgR2m/Kg zjzFxrR/oX3dvaevOYPdMhExohTYrUvAoAQAQJAjTzqPpFazEc5SaSn0bGuy58++MXND PXLlXzkfzwDdO6p3TURdXVOR6rUR8XjNKRoxoV6jnQ12Z791mqTT73RCwIn/Hrn8Up+t p6tvtjCnJp1KnBS++7/kh+agbWccyvFg+JbYsiYYfGIXgtzodrUuDdemUgQ92jmLv+NK OdBg5kbyE4OoY0m7fk7iEZP6VPTbyHgO5/Kr9zl3LqAxqzjbbkdwCDjY47eph4W1OfTa 5aAA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=YmekmIbOKaWK/Uh2GRis93ucBWqUHLXkmR51MffmvwM=; b=DHZcIgg20v86xvIfxOIxtac1nqGnVOE4GXEfFPCyjQVr1XV4WJgE1B/ILPcEyQGwWo 9KDi1DPJ2wvvKJ0foPN663hHiWM0tWa28a8oDtPjUjZzkKNIW3Q03wsQ8zPaqkFEQcRY Bafoe8vvJi2/5Wl/1ZUXXEG8cHf2enkecAJ5Fx+xtdHjQqlutoR/U66p+od2KhkE79/j 38El1RLReiN2rKUCZNbMo5Bgqj/55MhRonO1p911OEQNrUYegZr+Pl6V6I4P4aWs+Em1 /oHcqVDRw+lscIzb1MWuacpn8bUDwlOxkFaKCUy6wa6lpbiwUxeN7aBS0CAcicP/RErZ TbDw== X-Gm-Message-State: AGi0Pua57O14sRQyMXplq/UyQmf4gv4u5awzbmcMd6xuWLydI2+QWw9W /Q8b/4/0+tGcJpwZTH7djVXhAMIgqW75pw== X-Google-Smtp-Source: APiQypIUnzf/rDTy6mR5DJ0qGwkV9/JstZnTJKKsi1/im7e0dZZeZ/YmvNTmZOOInHSfzjVcIVE0jA== X-Received: by 2002:ac8:189c:: with SMTP id s28mr4380698qtj.65.1588262320546; Thu, 30 Apr 2020 08:58:40 -0700 (PDT) From: Robert Foley To: qemu-devel@nongnu.org Subject: [PATCH v5 4/9] tests/vm: Add ability to select QEMU from current build. Date: Thu, 30 Apr 2020 11:57:13 -0400 Message-Id: <20200430155718.212-5-robert.foley@linaro.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20200430155718.212-1-robert.foley@linaro.org> References: <20200430155718.212-1-robert.foley@linaro.org> Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=2607:f8b0:4864:20::842; envelope-from=robert.foley@linaro.org; helo=mail-qt1-x842.google.com X-detected-operating-system: by eggs.gnu.org: Error: [-] PROGRAM ABORT : Malformed IPv6 address (bad octet value). Location : parse_addr6(), p0f-client.c:67 X-Received-From: 2607:f8b0:4864:20::842 X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: philmd@redhat.com, alex.bennee@linaro.org, robert.foley@linaro.org, peter.puhov@linaro.org Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Added a new special variable QEMU_LOCAL=3D1, which will indicate to take the QEMU binary from the current build. Signed-off-by: Robert Foley Reviewed-by: Peter Puhov --- tests/vm/Makefile.include | 4 ++++ tests/vm/basevm.py | 30 +++++++++++++++++++++++++----- 2 files changed, 29 insertions(+), 5 deletions(-) diff --git a/tests/vm/Makefile.include b/tests/vm/Makefile.include index d921ee14cf..667705a91a 100644 --- a/tests/vm/Makefile.include +++ b/tests/vm/Makefile.include @@ -42,6 +42,7 @@ endif @echo " DEBUG=3D1 - Enable verbose output on host and i= nteractive debugging" @echo " BOOT_CONSOLE=3D1 - Show the console output at boot ti= me. " @echo " V=3D1 - Enable verbose ouput on host and guest commands" + @echo " QEMU_LOCAL=3D1 - Use QEMU binary local to this= build." @echo " QEMU=3D/path/to/qemu - Change path to QEMU binary" @echo " QEMU_IMG=3D/path/to/qemu-img - Change path to qemu-img tool" ifeq ($(PYTHON_YAML),yes) @@ -64,6 +65,7 @@ $(IMAGES_DIR)/%.img: $(SRC_PATH)/tests/vm/% \ $(PYTHON) $< \ $(if $(V)$(DEBUG), --debug) \ $(if $(BOOT_CONSOLE),--boot-console) \ + --build-path $(BUILD_DIR)\ --image "$@" \ --force \ --build-image $@, \ @@ -79,6 +81,7 @@ vm-build-%: $(IMAGES_DIR)/%.img $(if $(J),--jobs $(J)) \ $(if $(V),--verbose) \ $(if $(BOOT_CONSOLE),--boot-console) \ + --build-path $(BUILD_DIR)\ --image "$<" \ $(if $(BUILD_TARGET),--build-target $(BUILD_TARGET)) \ --snapshot \ @@ -100,6 +103,7 @@ vm-boot-ssh-%: $(IMAGES_DIR)/%.img $(PYTHON) $(SRC_PATH)/tests/vm/$* \ $(if $(J),--jobs $(J)) \ $(if $(BOOT_CONSOLE),--boot-console) \ + --build-path $(BUILD_DIR)\ --image "$<" \ --interactive \ false, \ diff --git a/tests/vm/basevm.py b/tests/vm/basevm.py index 983adbf6ff..7b86ccd018 100644 --- a/tests/vm/basevm.py +++ b/tests/vm/basevm.py @@ -87,8 +87,9 @@ class BaseVM(object): # since we found we need to wait more than twice as long. tcg_ssh_timeout_multiplier =3D 4 def __init__(self, debug=3DFalse, vcpus=3DNone, config=3DNone, - boot_console=3DNone): + boot_console=3DFalse, build_path=3DNone): self._guest =3D None + self._build_path =3D build_path self._boot_console =3D boot_console # Allow input config to override defaults. self._config =3D DEFAULT_CONFIG.copy() @@ -274,15 +275,15 @@ class BaseVM(object): args =3D self._args + boot_params.split(' ') args +=3D self._data_args + extra_args + self._config['extra_args'] logging.debug("QEMU args: %s", " ".join(args)) - qemu_bin =3D os.environ.get("QEMU", "qemu-system-" + self.arch) - guest =3D QEMUMachine(binary=3Dqemu_bin, args=3Dargs) + qemu_path =3D get_qemu_path(self.arch, self._build_path) + guest =3D QEMUMachine(binary=3Dqemu_path, args=3Dargs) guest.set_machine(self._config['machine']) guest.set_console() try: guest.launch() except: logging.error("Failed to launch QEMU, command line:") - logging.error(" ".join([qemu_bin] + args)) + logging.error(" ".join([qemu_path] + args)) logging.error("Log:") logging.error(guest.get_log()) logging.error("QEMU version >=3D 2.10 is required") @@ -481,6 +482,22 @@ class BaseVM(object): stderr=3Dself._stdout) return os.path.join(cidir, "cloud-init.iso") =20 +def get_qemu_path(arch, build_path=3DNone): + """Fetch the path to the qemu binary.""" + qemu_local =3D os.environ.get("QEMU_LOCAL", 0) + # If QEMU environment variable set, it takes precedence + if "QEMU" in os.environ: + qemu_path =3D os.environ["QEMU"] + elif qemu_local: + if not build_path: + raise Exception("--build-path option required with QEMU_LOCAL") + qemu_path =3D os.path.join(build_path, arch + "-softmmu") + qemu_path =3D os.path.join(qemu_path, "qemu-system-" + arch) + else: + # Default is to use system path for qemu. + qemu_path =3D "qemu-system-" + arch + return qemu_path + def parse_config(config, args): """ Parse yaml config and populate our config structure. The yaml config allows the user to override the @@ -555,6 +572,8 @@ def parse_args(vmcls): "See config_example.yaml for example.") parser.add_option("--boot-console", action=3D"store_true", help=3D"Show console during boot. ") + parser.add_option("--build-path", default=3DNone, + help=3D"Path of build directory. ") parser.disable_interspersed_args() return parser.parse_args() =20 @@ -570,7 +589,8 @@ def main(vmcls, config=3DNone): logging.basicConfig(level=3D(logging.DEBUG if args.debug else logging.WARN)) vm =3D vmcls(debug=3Dargs.debug, vcpus=3Dargs.jobs, config=3Dconfi= g, - boot_console=3Dargs.boot_console) + boot_console=3Dargs.boot_console, + build_path=3Dargs.build_path) if args.build_image: if os.path.exists(args.image) and not args.force: sys.stderr.writelines(["Image file exists: %s\n" % args.im= age, --=20 2.17.1 From nobody Sun May 19 14:14:26 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=linaro.org ARC-Seal: i=1; a=rsa-sha256; t=1588263233; cv=none; d=zohomail.com; s=zohoarc; b=UIVh9c9Oof0o2rFeaoJx/QrVYDULLbx9XG8ecL1ZULBsi37oF9k9dHbnuYtcTXKIm+2uF4REjh0JnSHBzCY6m2+NVgvAgj14eUwHQ/BMEUa5aYjl0Oas75g+TXB2S9Pav44R17lvDadIU6d56607KtrtshxjiQdGpGEH525NPCA= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1588263233; h=Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:Message-ID:References:Sender:Subject:To; bh=aF24IKnVFhq1UB6gmNrnCUeMXZezC3NkQoX6jpUZt8c=; b=h+anM71E8tcG+PLxXHSpfMlnSm2ZdaCNvgiPhhJqWtdT3umaLWuGTmVDCVpaopM2Bb8so10aOKjME5Y8FjJXA/1lriP1T4D4PLT0u8vJ/A/FMZ/TXf+iTEPcAz7GrFQdxH+uYA4IodnFcIow1z5jozgHBVOj5dG/05QtMDJkXPE= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1588263233165589.1947665693708; Thu, 30 Apr 2020 09:13:53 -0700 (PDT) Received: from localhost ([::1]:38270 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jUBot-0003hC-Rt for importer@patchew.org; Thu, 30 Apr 2020 12:13:51 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:45444) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jUBdX-0000Y9-8r for qemu-devel@nongnu.org; Thu, 30 Apr 2020 12:03:14 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.90_1) (envelope-from ) id 1jUBaF-0003MX-Ef for qemu-devel@nongnu.org; Thu, 30 Apr 2020 12:02:07 -0400 Received: from mail-qt1-x82a.google.com ([2607:f8b0:4864:20::82a]:34303) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1jUBaE-0003HS-WA for qemu-devel@nongnu.org; Thu, 30 Apr 2020 11:58:43 -0400 Received: by mail-qt1-x82a.google.com with SMTP id b1so5420468qtt.1 for ; Thu, 30 Apr 2020 08:58:42 -0700 (PDT) Received: from Rfoley-MA01.hsd1.ma.comcast.net ([2601:199:4480:60c0:7489:3328:3806:7c3]) by smtp.gmail.com with ESMTPSA id v16sm305254qkf.80.2020.04.30.08.58.40 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 30 Apr 2020 08:58:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=aF24IKnVFhq1UB6gmNrnCUeMXZezC3NkQoX6jpUZt8c=; b=cy6JHVlpNUxfV38LmzXrlZLRwgiMfwqKMzLXkOM80giZPGva/sjsp7XttnRlBN02I5 ml3IlaUEwaaJJmeDCg+kPXE1uehAbXZ71YgAXDiPYs3bFtQLs26PHCVFRdPTqKrc4wMb eyaAmT1H64ZJbKVoXTe3NcOjER7uxy7s82L6OC8HiXxtWniZtk+IkEU1emfia48ytlbM IZrXrxrqcxSeBn978UVeQSDzCHBW+KcC9NJr4BsCTpwJq/VC3gmorfbLVYHHLcC/Xwvy cTErmVnS4DRizUZuKfyfQTEISU2kNQ7BvRw1Elk1uq7ZUMLEWoLLthhxaUeCKN4xmlMS pyCg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=aF24IKnVFhq1UB6gmNrnCUeMXZezC3NkQoX6jpUZt8c=; b=io64806EKT/AaCrW7uq/VQBAo63VOk62RdbEzuZ5drplVgwyCMfA6HhGUKE2Kum+6o ntEc3g9x+zTe3AqxWp8w3Jn/oiIB6k5HHVNBE2JI0Eg4fyo3m1kP5X9wcWK+2/XJkQZr 194G11XntryLJmISWG8qCnfy+hR+dqRZHhe+FBMhDL0s0eA91XEw4pTqMkzxzrxGyB0C QgHX8ERtKQxSqNQ1XXjntrHPNXIGL5vQJEs0bcdf42wqJ/MpRqVdxTrFwjXRDH/RuQVZ 3AmWBSG5TJm2g8jT+yUZUfQV9HVPi1jCliR5ZzErl6GZEO810ntN0QtkV0Va5+iG92AX 47dw== X-Gm-Message-State: AGi0PuZ2yLN5gzVDcYm4iYXv98UMNUSfc/BVN2RTH9g7h4QSVQtulE8P n36s3URbwpPa/42lk/7VbB4SB/SGDWVOZw== X-Google-Smtp-Source: APiQypLDIgBAsh6qAW4H+Ry/jg/6OKc9HeIAwK008JYk0QbltXw27ekaV+e6mTip+PjWKuHJZZm60Q== X-Received: by 2002:ac8:5653:: with SMTP id 19mr4289832qtt.252.1588262321752; Thu, 30 Apr 2020 08:58:41 -0700 (PDT) From: Robert Foley To: qemu-devel@nongnu.org Subject: [PATCH v5 5/9] tests/vm: allow wait_ssh() to specify command Date: Thu, 30 Apr 2020 11:57:14 -0400 Message-Id: <20200430155718.212-6-robert.foley@linaro.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20200430155718.212-1-robert.foley@linaro.org> References: <20200430155718.212-1-robert.foley@linaro.org> Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=2607:f8b0:4864:20::82a; envelope-from=robert.foley@linaro.org; helo=mail-qt1-x82a.google.com X-detected-operating-system: by eggs.gnu.org: Error: [-] PROGRAM ABORT : Malformed IPv6 address (bad octet value). Location : parse_addr6(), p0f-client.c:67 X-Received-From: 2607:f8b0:4864:20::82a X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: philmd@redhat.com, alex.bennee@linaro.org, robert.foley@linaro.org, peter.puhov@linaro.org Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" This allows for waiting for completion of arbitrary commands. Signed-off-by: Robert Foley Reviewed-by: Peter Puhov --- tests/vm/basevm.py | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/tests/vm/basevm.py b/tests/vm/basevm.py index 7b86ccd018..9f06f84974 100644 --- a/tests/vm/basevm.py +++ b/tests/vm/basevm.py @@ -410,24 +410,24 @@ class BaseVM(object): def print_step(self, text): sys.stderr.write("### %s ...\n" % text) =20 - def wait_ssh(self, wait_root=3DFalse, seconds=3D300): + def wait_ssh(self, wait_root=3DFalse, seconds=3D300, cmd=3D"exit 0"): # Allow more time for VM to boot under TCG. if not kvm_available(self.arch): seconds *=3D self.tcg_ssh_timeout_multiplier starttime =3D datetime.datetime.now() endtime =3D starttime + datetime.timedelta(seconds=3Dseconds) - guest_up =3D False + cmd_success =3D False while datetime.datetime.now() < endtime: - if wait_root and self.ssh_root("exit 0") =3D=3D 0: - guest_up =3D True + if wait_root and self.ssh_root(cmd) =3D=3D 0: + cmd_success =3D True break - elif self.ssh("exit 0") =3D=3D 0: - guest_up =3D True + elif self.ssh(cmd) =3D=3D 0: + cmd_success =3D True break seconds =3D (endtime - datetime.datetime.now()).total_seconds() logging.debug("%ds before timeout", seconds) time.sleep(1) - if not guest_up: + if not cmd_success: raise Exception("Timeout while waiting for guest ssh") =20 def shutdown(self): --=20 2.17.1 From nobody Sun May 19 14:14:26 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=linaro.org ARC-Seal: i=1; a=rsa-sha256; t=1588263883; cv=none; d=zohomail.com; s=zohoarc; b=a6aEwlWnbv9fzSjTRVaVDQe7T3T6Uh5qbsM2FfyeGheuBca0FJO3tPUkNTJxSfUnuuFDZHH//fA6ArLQ4E5kK3Xo1ALgNAlY67yKQDQgwHYknRiBpS93dn/jjt0PxqHOhrbHd/o6ZAz/uY6UnV2QeFuW3Uj6vMRDW5z4Jnt9gcU= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1588263883; h=Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:Message-ID:References:Sender:Subject:To; bh=TYO4Zx0Loe04L+x6lPxnHdIAL0X0Jk82WXdBB7mYsRE=; b=bcT+055Rxslxrg9JvGvihgxIMFcTiqjaqb3Xv1XIIUWz0WWjJfSmixkZAK11LB4eytVxpZF9ytMFwRQfKwKppjNFo1T6OFH6MvE/yEDidziy3UmO0l7kYIbesB2Q+LO+sAWYcZh99PnAXGsUf7ajkCaf98/TV9Z3+UK8EClS94s= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1588263883121823.4375328663566; Thu, 30 Apr 2020 09:24:43 -0700 (PDT) Received: from localhost ([::1]:38834 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jUBzN-000414-7D for importer@patchew.org; Thu, 30 Apr 2020 12:24:41 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:45448) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jUBda-0000by-2I for qemu-devel@nongnu.org; Thu, 30 Apr 2020 12:03:16 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.90_1) (envelope-from ) id 1jUBaH-0003Uq-89 for qemu-devel@nongnu.org; Thu, 30 Apr 2020 12:02:09 -0400 Received: from mail-qk1-x735.google.com ([2607:f8b0:4864:20::735]:46641) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1jUBaG-0003Qa-Oz for qemu-devel@nongnu.org; Thu, 30 Apr 2020 11:58:44 -0400 Received: by mail-qk1-x735.google.com with SMTP id g74so6165533qke.13 for ; Thu, 30 Apr 2020 08:58:44 -0700 (PDT) Received: from Rfoley-MA01.hsd1.ma.comcast.net ([2601:199:4480:60c0:7489:3328:3806:7c3]) by smtp.gmail.com with ESMTPSA id v16sm305254qkf.80.2020.04.30.08.58.41 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 30 Apr 2020 08:58:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=TYO4Zx0Loe04L+x6lPxnHdIAL0X0Jk82WXdBB7mYsRE=; b=Je608rteJbKCi888tRaQgduw62n5oWEaHMkAHHu7H496LSlwQBZU0zZLyN4CoeZY/9 Jq6Sl8Ej222UwXT4GsrNfy4xKwHubCHZy+9vtfOY1HgIETYrQ051+nF2iNdU6ox58HvL tcK4bb4BR+ulC1mW1JstFjXDUk1hM2GE5rCiDQdGRxuL22G3dgu2Soj0e2rUZD9s6E0x PBQFOyxQcIHPK9A4Cp6zCqrntLu0U+clK32LQ3sxb8lhzDDkRXJah8I5uXF0qAMvZgl1 9QdpAPfhqlV421PBaLqkX7ijs0mnH0vCyT31S/xxCqmY/JKVnrVQzA20g2sXsFkwVzg6 fRNA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=TYO4Zx0Loe04L+x6lPxnHdIAL0X0Jk82WXdBB7mYsRE=; b=XGcZJStXDdliimEH9Tn8TFkfrC/PW0MhjTpBdD+EhlViMR8jJs8pSVo9X0GIBPey36 ORu+Jc87w96pgWBMb9G9FMCwrLGUFhwMWQvrlw5KahElAbBO4zNUmQkIA4K6s3bMBsmZ yMhg+tAmBysGha35UEli6xlE2BjHc/D8hIYkR7AJAI012Um6UaSXM9zH8EBhUGaYWMSW xA65bs82TMjAZ20wX8zTAFYf4kDBOhAam/dcfLej23G7odf6/3mCLYm74+IhwdsKbJHv u9OGpBcd5GGlXj+bhHfUQ0ELkvVXqdbvDpPDlirpg0hKiP0ywcbifDCxpB5mCyX1ZQRJ jSpQ== X-Gm-Message-State: AGi0PubXyCSM8swIDO7cS8eeJiySfejuA1rZ6ENJsEIBZl6t8/ea4akP yYi5iFTp0G0Cl7MBjNEFOKJDg4ezm8abaw== X-Google-Smtp-Source: APiQypJJtJPZxKEQptuYHAnU5OIqe8+R1F9PDhtp16+zWP/vJLf9ATUBgy3yDkZieCCSuu8tm1yLEQ== X-Received: by 2002:a37:66d8:: with SMTP id a207mr4087875qkc.127.1588262323073; Thu, 30 Apr 2020 08:58:43 -0700 (PDT) From: Robert Foley To: qemu-devel@nongnu.org Subject: [PATCH v5 6/9] tests/vm: Added a new script for ubuntu.aarch64. Date: Thu, 30 Apr 2020 11:57:15 -0400 Message-Id: <20200430155718.212-7-robert.foley@linaro.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20200430155718.212-1-robert.foley@linaro.org> References: <20200430155718.212-1-robert.foley@linaro.org> Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=2607:f8b0:4864:20::735; envelope-from=robert.foley@linaro.org; helo=mail-qk1-x735.google.com X-detected-operating-system: by eggs.gnu.org: Error: [-] PROGRAM ABORT : Malformed IPv6 address (bad octet value). Location : parse_addr6(), p0f-client.c:67 X-Received-From: 2607:f8b0:4864:20::735 X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: philmd@redhat.com, alex.bennee@linaro.org, robert.foley@linaro.org, peter.puhov@linaro.org Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" ubuntu.aarch64 provides a script to create an Ubuntu 20.04 VM. Another new file is also added aarch64vm.py, which is a module with common methods used by aarch64 VMs, such as how to create the flash images. Signed-off-by: Robert Foley Reviewed-by: Peter Puhov --- configure | 16 ++++++ tests/vm/Makefile.include | 8 +++ tests/vm/aarch64vm.py | 105 ++++++++++++++++++++++++++++++++++ tests/vm/basevm.py | 8 +++ tests/vm/ubuntu.aarch64 | 117 ++++++++++++++++++++++++++++++++++++++ 5 files changed, 254 insertions(+) create mode 100644 tests/vm/aarch64vm.py create mode 100755 tests/vm/ubuntu.aarch64 diff --git a/configure b/configure index eec6615b81..00102f2bf0 100755 --- a/configure +++ b/configure @@ -950,6 +950,20 @@ do fi done =20 +# Check for files needed by aarch64 VMs. +# Allow user to override the path for efi also. +qemu_efi_aarch64=3D +for fd in $QEMU_EFI_PATH /usr/share/qemu-efi-aarch64/QEMU_EFI.fd +do + if has $fd + then + if [ -f $fd ]; then + qemu_efi_aarch64=3D$fd + fi + break + fi +done + # Check for existence of python3 yaml, needed to # import yaml config files into vm-build. python_yaml=3D"no" @@ -6588,6 +6602,7 @@ if test "$docs" !=3D "no"; then echo "sphinx-build $sphinx_build" fi echo "genisoimage $genisoimage" +echo "qemu_efi_aarch64 $qemu_efi_aarch64" echo "python_yaml $python_yaml" echo "slirp support $slirp $(echo_version $slirp $slirp_version)" if test "$slirp" !=3D "no" ; then @@ -7650,6 +7665,7 @@ echo "PYTHON=3D$python" >> $config_host_mak echo "SPHINX_BUILD=3D$sphinx_build" >> $config_host_mak echo "SPHINX_WERROR=3D$sphinx_werror" >> $config_host_mak echo "GENISOIMAGE=3D$genisoimage" >> $config_host_mak +echo "QEMU_EFI_AARCH64=3D$qemu_efi_aarch64" >> $config_host_mak echo "PYTHON_YAML=3D$python_yaml" >> $config_host_mak echo "CC=3D$cc" >> $config_host_mak if $iasl -h > /dev/null 2>&1; then diff --git a/tests/vm/Makefile.include b/tests/vm/Makefile.include index 667705a91a..9934008bfa 100644 --- a/tests/vm/Makefile.include +++ b/tests/vm/Makefile.include @@ -5,6 +5,9 @@ IMAGES :=3D freebsd netbsd openbsd centos fedora ifneq ($(GENISOIMAGE),) IMAGES +=3D ubuntu.i386 centos +ifneq ($(QEMU_EFI_AARCH64),) +IMAGES +=3D ubuntu.aarch64 +endif endif =20 IMAGES_DIR :=3D $(HOME)/.cache/qemu-vm/images @@ -23,6 +26,11 @@ vm-help vm-test: ifneq ($(GENISOIMAGE),) @echo " vm-build-centos - Build QEMU in CentOS VM, with = Docker" @echo " vm-build-ubuntu.i386 - Build QEMU in ubuntu i386 VM" +ifneq ($(QEMU_EFI_AARCH64),) + @echo " vm-build-ubuntu.aarch64 - Build QEMU in ubuntu aarch64 V= M" +else + @echo " (install qemu-efi-aarch64 to build centos/ubuntu aarch64 images.= )" +endif else @echo " (install genisoimage to build centos/ubuntu images)" endif diff --git a/tests/vm/aarch64vm.py b/tests/vm/aarch64vm.py new file mode 100644 index 0000000000..3a9807cd43 --- /dev/null +++ b/tests/vm/aarch64vm.py @@ -0,0 +1,105 @@ +#!/usr/bin/env python3 +# +# VM testing aarch64 library +# +# Copyright 2020 Linaro +# +# Authors: +# Robert Foley +# +# This code is licensed under the GPL version 2 or later. See +# the COPYING file in the top-level directory. +# +import os +import sys +import subprocess +import basevm +from qemu.accel import kvm_available + +# This is the config needed for current version of QEMU. +# This works for both kvm and tcg. +CURRENT_CONFIG =3D { + 'cpu' : "max", + 'machine' : "virt,gic-version=3Dmax", +} + +# The minimum minor version of QEMU we will support with aarch64 VMs is 3. +# QEMU versions less than 3 have various issues running these VMs. +QEMU_AARCH64_MIN_VERSION =3D 3 + +# The DEFAULT_CONFIG will default to a version of +# parameters that works for backwards compatibility. +DEFAULT_CONFIG =3D {'kvm' : {'cpu' : "host", + 'machine' : "virt,gic-version=3Dhost"}, + 'tcg' : {'cpu' : "cortex-a57", + 'machine' : "virt"}, +} + +def get_config_defaults(vmcls, default_config): + """Fetch the configuration defaults for this VM, + taking into consideration the defaults for + aarch64 first, followed by the defaults for this VM.""" + config =3D default_config + config.update(aarch_get_config_defaults(vmcls)) + return config + +def aarch_get_config_defaults(vmcls): + """Set the defaults for current version of QEMU.""" + config =3D CURRENT_CONFIG + args, argv =3D basevm.parse_args(vmcls) + qemu_path =3D basevm.get_qemu_path(vmcls.arch, args.build_path) + qemu_version =3D basevm.get_qemu_version(qemu_path) + if qemu_version < QEMU_AARCH64_MIN_VERSION: + error =3D "\nThis major version of QEMU {} is to old for aarch64 V= Ms.\n"\ + "The major version must be at least {}.\n"\ + "To continue with the current build of QEMU, "\ + "please restart with QEMU_LOCAL=3D1 .\n" + print(error.format(qemu_version, QEMU_AARCH64_MIN_VERSION)) + exit(1) + if qemu_version =3D=3D QEMU_AARCH64_MIN_VERSION: + # We have an older version of QEMU, + # set the config values for backwards compatibility. + if kvm_available('aarch64'): + config.update(DEFAULT_CONFIG['kvm']) + else: + config.update(DEFAULT_CONFIG['tcg']) + return config + +def create_flash_images(flash_dir=3D"./"): + """Creates the appropriate pflash files + for an aarch64 VM.""" + flash0_path =3D get_flash_path(flash_dir, "flash0") + flash1_path =3D get_flash_path(flash_dir, "flash1") + fd_null =3D open(os.devnull, 'w') + subprocess.check_call(["dd", "if=3D/dev/zero", "of=3D{}".format(flash0= _path), + "bs=3D1M", "count=3D64"], + stdout=3Dfd_null, stderr=3Dsubprocess.STDOUT) + # A reliable way to get the QEMU EFI image is via an installed package. + efi_img =3D "/usr/share/qemu-efi-aarch64/QEMU_EFI.fd" + if not os.path.exists(efi_img): + sys.stderr.write("*** {} is missing\n".format(efi_img)) + sys.stderr.write("*** please install qemu-efi-aarch64 package\n") + exit(3) + subprocess.check_call(["dd", "if=3D{}".format(efi_img), + "of=3D{}".format(flash0_path), + "conv=3Dnotrunc"], + stdout=3Dfd_null, stderr=3Dsubprocess.STDOUT) + subprocess.check_call(["dd", "if=3D/dev/zero", + "of=3D{}".format(flash1_path), + "bs=3D1M", "count=3D64"], + stdout=3Dfd_null, stderr=3Dsubprocess.STDOUT) + fd_null.close() + +def get_pflash_args(flash_dir=3D"./"): + """Returns a string that can be used to + boot qemu using the appropriate pflash files + for aarch64.""" + flash0_path =3D get_flash_path(flash_dir, "flash0") + flash1_path =3D get_flash_path(flash_dir, "flash1") + pflash_args_str =3D "-drive file=3D{},format=3Draw,if=3Dpflash "\ + "-drive file=3D{},format=3Draw,if=3Dpflash" + pflash_args =3D pflash_args_str.format(flash0_path, flash1_path) + return pflash_args.split(" ") + +def get_flash_path(flash_dir, name): + return os.path.join(flash_dir, "{}.img".format(name)) diff --git a/tests/vm/basevm.py b/tests/vm/basevm.py index 9f06f84974..dddd4711ff 100644 --- a/tests/vm/basevm.py +++ b/tests/vm/basevm.py @@ -498,6 +498,14 @@ def get_qemu_path(arch, build_path=3DNone): qemu_path =3D "qemu-system-" + arch return qemu_path =20 +def get_qemu_version(qemu_path): + """Get the version number from the current QEMU, + and return the major number.""" + output =3D subprocess.check_output([qemu_path, '--version']) + version_line =3D output.decode("utf-8") + version_num =3D re.split(' |\(', version_line)[3].split('.')[0] + return int(version_num) + def parse_config(config, args): """ Parse yaml config and populate our config structure. The yaml config allows the user to override the diff --git a/tests/vm/ubuntu.aarch64 b/tests/vm/ubuntu.aarch64 new file mode 100755 index 0000000000..40bf1110e8 --- /dev/null +++ b/tests/vm/ubuntu.aarch64 @@ -0,0 +1,117 @@ +#!/usr/bin/env python3 +# +# Ubuntu aarch64 image +# +# Copyright 2020 Linaro +# +# Authors: +# Robert Foley +# Originally based on ubuntu.i386 Fam Zheng +# +# This code is licensed under the GPL version 2 or later. See +# the COPYING file in the top-level directory. +# + +import os +import sys +import subprocess +import basevm +from qemu.accel import kvm_available +import time +import aarch64vm + +DEFAULT_CONFIG =3D { + 'cpu' : "cortex-a57", + 'machine' : "virt,gic-version=3D3", + 'install_cmds' : "apt-get update,"\ + "apt-get build-dep -y --arch-only qemu,"\ + "apt-get install -y libfdt-dev flex bison pkg-config", + # We increase beyond the default time since during boot + # it can take some time (many seconds) to log into the VM + # especially using softmmu. + 'ssh_timeout' : 60, +} + +class UbuntuAarch64VM(basevm.BaseVM): + name =3D "ubuntu.aarch64" + arch =3D "aarch64" + image_name =3D "ubuntu-20.04-server-cloudimg-arm64.img" + image_link =3D "https://cloud-images.ubuntu.com/releases/20.04/release= /" + image_name + login_prompt =3D "ubuntu-aarch64-guest login:" + BUILD_SCRIPT =3D """ + set -e; + cd $(mktemp -d); + sudo chmod a+r /dev/vdb; + tar --checkpoint=3D.10 -xf /dev/vdb; + ./configure {configure_opts}; + make --output-sync {target} -j{jobs} {verbose}; + """ + def boot(self, img, extra_args=3DNone): + aarch64vm.create_flash_images(self._tmpdir) + default_args =3D aarch64vm.get_pflash_args(self._tmpdir) + if extra_args: + extra_args.extend(default_args) + else: + extra_args =3D default_args + # We always add these performance tweaks + # because without them, we boot so slowly that we + # can time out finding the boot efi device. + if '-smp' not in extra_args and \ + '-smp' not in self._config['extra_args'] and \ + '-smp' not in self._args: + # Only add if not already there to give caller option to chang= e it. + extra_args.extend(["-smp", "8"]) + + # We have overridden boot() since aarch64 has additional parameter= s. + # Call down to the base class method. + super(UbuntuAarch64VM, self).boot(img, extra_args=3Dextra_args) + + def build_image(self, img): + os_img =3D self._download_with_cache(self.image_link) + img_tmp =3D img + ".tmp" + subprocess.check_call(["cp", "-f", os_img, img_tmp]) + subprocess.check_call(["qemu-img", "resize", img_tmp, "+50G"]) + ci_img =3D self.gen_cloud_init_iso() + + self.boot(img_tmp, extra_args =3D ["-cdrom", ci_img]) + if self._boot_console: + self.wait_boot() + # First command we issue is fix for slow ssh login. + self.wait_ssh(wait_root=3DTrue, + cmd=3D"chmod -x /etc/update-motd.d/*") + # Wait for cloud init to finish + self.wait_ssh(wait_root=3DTrue, + cmd=3D"ls /var/lib/cloud/instance/boot-finished") + self.ssh_root("touch /etc/cloud/cloud-init.disabled") + # Disable auto upgrades. + # We want to keep the VM system state stable. + self.ssh_root('sed -ie \'s/"1"/"0"/g\' /etc/apt/apt.conf.d/20auto-= upgrades') + # If the user chooses *not* to do the second phase, + # then we will jump right to the graceful shutdown + if self._config['install_cmds'] !=3D "": + self.ssh_root("sync") + # Shutdown and then boot it again. + # Allows us to know for sure it is booting (not shutting down) + # before we call wait_ssh(). + self.graceful_shutdown() + self.boot(img_tmp) + if self._boot_console: + self.wait_boot() + self.wait_ssh(wait_root=3DTrue) + self.wait_ssh(wait_root=3DTrue, cmd=3D"locale") + # The previous update sometimes doesn't survive a reboot, so d= o it again + self.ssh_root("sed -ie s/^#\ deb-src/deb-src/g /etc/apt/source= s.list") + + # Issue the install commands. + # This can be overriden by the user in the config .yml. + install_cmds =3D self._config['install_cmds'].split(',') + for cmd in install_cmds: + self.ssh_root(cmd) + self.graceful_shutdown() + self.wait() + os.rename(img_tmp, img) + return 0 + +if __name__ =3D=3D "__main__": + defaults =3D aarch64vm.get_config_defaults(UbuntuAarch64VM, DEFAULT_CO= NFIG) + sys.exit(basevm.main(UbuntuAarch64VM, defaults)) --=20 2.17.1 From nobody Sun May 19 14:14:26 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=linaro.org ARC-Seal: i=1; a=rsa-sha256; t=1588263342; cv=none; d=zohomail.com; s=zohoarc; b=hRQBMSA2Ztj6EKoTjC0V9u0Nox+kwR/rz+2UwaGtb86vw6yQpfK1V5hrv2dGaM/a1z+av/ClAH24vAM8kl1FPl/hxdkQn0Vxe60YdxN6GmwGtxi6n5gYNYdG1tZsiv4agZbVuNWWC51ZF0UbkEZMcaqAUVEnauESEy/u00/fsYg= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1588263342; h=Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:Message-ID:References:Sender:Subject:To; bh=ui+UwFWcWGxKqirrTxQJ7A+TJRx59oMmHjjMlbzkfRE=; b=cLr+q6efaoZ9zZgrojvBB/NUWzdBtobpRgswaaDfEr1TkzqzdwDjUlWnWr/rKfy3Flj+J3OhQNWPBFjnLDaf+HCIRVFKcBVHmu0IZphgaZeii061i39zkyDyCWpJQb9eTI0+c/LvkTZAK8BtR5dsrBVNqzJITaE5PYMcAvY9A8s= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 158826334278698.22508396345268; Thu, 30 Apr 2020 09:15:42 -0700 (PDT) Received: from localhost ([::1]:44994 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jUBqf-0006VE-1v for importer@patchew.org; Thu, 30 Apr 2020 12:15:41 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:45456) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jUBdb-0000e9-KC for qemu-devel@nongnu.org; Thu, 30 Apr 2020 12:03:16 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.90_1) (envelope-from ) id 1jUBaI-0003ai-Ng for qemu-devel@nongnu.org; Thu, 30 Apr 2020 12:02:11 -0400 Received: from mail-qk1-x72e.google.com ([2607:f8b0:4864:20::72e]:35322) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1jUBaI-0003YU-9N for qemu-devel@nongnu.org; Thu, 30 Apr 2020 11:58:46 -0400 Received: by mail-qk1-x72e.google.com with SMTP id c63so6224368qke.2 for ; Thu, 30 Apr 2020 08:58:45 -0700 (PDT) Received: from Rfoley-MA01.hsd1.ma.comcast.net ([2601:199:4480:60c0:7489:3328:3806:7c3]) by smtp.gmail.com with ESMTPSA id v16sm305254qkf.80.2020.04.30.08.58.43 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 30 Apr 2020 08:58:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=ui+UwFWcWGxKqirrTxQJ7A+TJRx59oMmHjjMlbzkfRE=; b=KuR7vqFMO3Ao3eXdKhywR54iWt+ES7GgEj/z0p9Z5cnamawzA1fH/6AVG//7bXYGmJ 28Z2yJUZNJHnsyw4L2HOVyQ9s4Vx3SIdKITdwKETjFCbZYqhC4/A/KRJTl6nuOMM0Jr2 DAi4S5VgSbb6k3vzLgpGyj5RaVUZj9khaF/kgNWphuTduDykCtjcedk60jViUTqcwjWm 8vk5y9Nb40yh4IMglNp+ylr9vfaub4ca2TDpqFpA7naYgsDy6orPkmIXIh7T9+1q55Or vVuuIKAdz+J3p4be+hiYDw+kokDG9UHVym5zv46N96+SE54ejEWNgmEurDbOKArXFs2g CYkA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=ui+UwFWcWGxKqirrTxQJ7A+TJRx59oMmHjjMlbzkfRE=; b=SioOZgGsMBadeM4o+uoBHswP0VnzS8Mj1tn7BysZ/VHIa42MYdVORpQXieOkoZORtf j5Ng0EomfLn8OGqlG5IUK+pzWJ+dgnbBi9bGNQox5fj8Ww2BeJ+Vk75wB/l9pW2kf4bT B03Lhu2AADVZ77RE0yg2e65aNW3aRK9SL4ciQShXQs0LbCZmtTPQr9ALBKmHMeaSM4ED JpJ/HGrr9Iac56jl/zgfVsCPIAUyGfiSQRjo2mYNqrNVP+8xT+J+5ckLqtRBIfG2X7EM n9oX+oAiBjmFoRVzp51+Prd6m4uQFzTn7I5GVM/Z3nk8H6eLR3EyzcTH0TvhDXUX39J6 wT8Q== X-Gm-Message-State: AGi0PubkftXxZYD9bog0qs1HbzxNEO1ZcXfY7YU8DOTbmCCznYlrLBzm 1Ep0r8VWvqb32bASzzpfRXSlim9VzM2SOA== X-Google-Smtp-Source: APiQypL4pUOKU5Fg+Hh0oigaEt2YroPUap8+j0XL+RUOMi7KyaK35D5TeUwr2foUNO98WqHWtcZvyw== X-Received: by 2002:a37:4fd8:: with SMTP id d207mr3849683qkb.345.1588262324460; Thu, 30 Apr 2020 08:58:44 -0700 (PDT) From: Robert Foley To: qemu-devel@nongnu.org Subject: [PATCH v5 7/9] tests/vm: Added a new script for centos.aarch64. Date: Thu, 30 Apr 2020 11:57:16 -0400 Message-Id: <20200430155718.212-8-robert.foley@linaro.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20200430155718.212-1-robert.foley@linaro.org> References: <20200430155718.212-1-robert.foley@linaro.org> Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=2607:f8b0:4864:20::72e; envelope-from=robert.foley@linaro.org; helo=mail-qk1-x72e.google.com X-detected-operating-system: by eggs.gnu.org: Error: [-] PROGRAM ABORT : Malformed IPv6 address (bad octet value). Location : parse_addr6(), p0f-client.c:67 X-Received-From: 2607:f8b0:4864:20::72e X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: philmd@redhat.com, alex.bennee@linaro.org, robert.foley@linaro.org, peter.puhov@linaro.org Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" centos.aarch64 creates a CentOS 8 image. Also added a new kickstart script used to build the centos.aarch64 image. Signed-off-by: Robert Foley Reviewed-by: Peter Puhov --- tests/vm/Makefile.include | 3 +- tests/vm/centos-8-aarch64.ks | 51 ++++++++ tests/vm/centos.aarch64 | 227 +++++++++++++++++++++++++++++++++++ 3 files changed, 280 insertions(+), 1 deletion(-) create mode 100644 tests/vm/centos-8-aarch64.ks create mode 100755 tests/vm/centos.aarch64 diff --git a/tests/vm/Makefile.include b/tests/vm/Makefile.include index 9934008bfa..419e0f8613 100644 --- a/tests/vm/Makefile.include +++ b/tests/vm/Makefile.include @@ -6,7 +6,7 @@ IMAGES :=3D freebsd netbsd openbsd centos fedora ifneq ($(GENISOIMAGE),) IMAGES +=3D ubuntu.i386 centos ifneq ($(QEMU_EFI_AARCH64),) -IMAGES +=3D ubuntu.aarch64 +IMAGES +=3D ubuntu.aarch64 centos.aarch64 endif endif =20 @@ -28,6 +28,7 @@ ifneq ($(GENISOIMAGE),) @echo " vm-build-ubuntu.i386 - Build QEMU in ubuntu i386 VM" ifneq ($(QEMU_EFI_AARCH64),) @echo " vm-build-ubuntu.aarch64 - Build QEMU in ubuntu aarch64 V= M" + @echo " vm-build-centos.aarch64 - Build QEMU in CentOS aarch64 V= M" else @echo " (install qemu-efi-aarch64 to build centos/ubuntu aarch64 images.= )" endif diff --git a/tests/vm/centos-8-aarch64.ks b/tests/vm/centos-8-aarch64.ks new file mode 100644 index 0000000000..fd6ebe4d49 --- /dev/null +++ b/tests/vm/centos-8-aarch64.ks @@ -0,0 +1,51 @@ +# CentOS aarch64 image kickstart file. +# This file is used by the CentOS installer to +# script the generation of the image. +# +# Copyright 2020 Linaro +# +ignoredisk --only-use=3Dvda +# System bootloader configuration +bootloader --append=3D" crashkernel=3Dauto" --location=3Dmbr --boot-drive= =3Dvda +autopart --type=3Dplain +# Partition clearing information +clearpart --linux --initlabel --drives=3Dvda +# Use text mode install +text +repo --name=3D"AppStream" --baseurl=3Dfile:///run/install/repo/AppStream +# Use CDROM installation media +cdrom +# Keyboard layouts +keyboard --vckeymap=3Dus --xlayouts=3D'' +# System language +lang en_US.UTF-8 + +# Network information +network --bootproto=3Ddhcp --device=3Denp0s1 --onboot=3Doff --ipv6=3Dauto= --no-activate +network --hostname=3Dlocalhost.localdomain +# Run the Setup Agent on first boot +firstboot --enable +# Do not configure the X Window System +skipx +# System services +services --enabled=3D"chronyd" +# System timezone +timezone America/New_York --isUtc + +# Shutdown after installation is complete. +shutdown + +%packages +@^server-product-environment +kexec-tools + +%end + +%addon com_redhat_kdump --enable --reserve-mb=3D'auto' + +%end +%anaconda +pwpolicy root --minlen=3D6 --minquality=3D1 --notstrict --nochanges --note= mpty +pwpolicy user --minlen=3D6 --minquality=3D1 --notstrict --nochanges --empt= yok +pwpolicy luks --minlen=3D6 --minquality=3D1 --notstrict --nochanges --note= mpty +%end diff --git a/tests/vm/centos.aarch64 b/tests/vm/centos.aarch64 new file mode 100755 index 0000000000..db61fbbce1 --- /dev/null +++ b/tests/vm/centos.aarch64 @@ -0,0 +1,227 @@ +#!/usr/bin/env python3 +# +# Centos aarch64 image +# +# Copyright 2020 Linaro +# +# Authors: +# Robert Foley +# Originally based on ubuntu.aarch64 +# +# This code is licensed under the GPL version 2 or later. See +# the COPYING file in the top-level directory. +# + +import os +import sys +import subprocess +import basevm +import time +import traceback +import aarch64vm + +DEFAULT_CONFIG =3D { + 'cpu' : "max", + 'machine' : "virt,gic-version=3Dmax", + 'install_cmds' : "yum install -y make git python3 gcc gcc-c++ flex bis= on, "\ + "yum install -y glib2-devel pixman-devel zlib-devel, "\ + "yum install -y perl-Test-Harness, "\ + "alternatives --set python /usr/bin/python3, "\ + "sudo dnf config-manager "\ + "--add-repo=3Dhttps://download.docker.com/linux/centos/docker-ce.r= epo,"\ + "sudo dnf install --nobest -y docker-ce.aarch64,"\ + "systemctl enable docker", + # We increase beyond the default time since during boot + # it can take some time (many seconds) to log into the VM. + 'ssh_timeout' : 60, +} + +class CentosAarch64VM(basevm.BaseVM): + name =3D "centos.aarch64" + arch =3D "aarch64" + login_prompt =3D "localhost login:" + prompt =3D '[root@localhost ~]#' + image_name =3D "CentOS-8-aarch64-1905-dvd1.iso" + image_link =3D "http://mirrors.usc.edu/pub/linux/distributions/centos/= 8.0.1905/isos/aarch64/" + image_link +=3D image_name + BUILD_SCRIPT =3D """ + set -e; + cd $(mktemp -d); + sudo chmod a+r /dev/vdb; + tar --checkpoint=3D.10 -xf /dev/vdb; + ./configure {configure_opts}; + make --output-sync {target} -j{jobs} {verbose}; + """ + def set_key_perm(self): + """Set permissions properly on certain files to allow + ssh access.""" + self.console_wait_send(self.prompt, + "/usr/sbin/restorecon -R -v /root/.ssh\n") + self.console_wait_send(self.prompt, + "/usr/sbin/restorecon -R -v "\ + "/home/{}/.ssh\n".format(self._config["guest_user"])) + + def create_kickstart(self): + """Generate the kickstart file used to generate the centos image."= "" + # Start with the template for the kickstart. + ks_file =3D "../tests/vm/centos-8-aarch64.ks" + subprocess.check_call("cp {} ./ks.cfg".format(ks_file), shell=3DTr= ue) + # Append the ssh keys to the kickstart file + # as the post processing phase of installation. + with open("ks.cfg", "a") as f: + # Add in the root pw and guest user. + rootpw =3D "rootpw --plaintext {}\n" + f.write(rootpw.format(self._config["root_pass"])) + add_user =3D "user --groups=3Dwheel --name=3D{} "\ + "--password=3D{} --plaintext\n" + f.write(add_user.format(self._config["guest_user"], + self._config["guest_pass"])) + # Add the ssh keys. + f.write("%post --log=3D/root/ks-post.log\n") + f.write("mkdir -p /root/.ssh\n") + addkey =3D 'echo "{}" >> /root/.ssh/authorized_keys\n' + addkey_cmd =3D addkey.format(self._config["ssh_pub_key"]) + f.write(addkey_cmd) + f.write('mkdir -p /home/{}/.ssh\n'.format(self._config["guest_= user"])) + addkey =3D 'echo "{}" >> /home/{}/.ssh/authorized_keys\n' + addkey_cmd =3D addkey.format(self._config["ssh_pub_key"], + self._config["guest_user"]) + f.write(addkey_cmd) + f.write("%end\n") + # Take our kickstart file and create an .iso from it. + # The .iso will be provided to qemu as we boot + # from the install dvd. + # Anaconda will recognize the label "OEMDRV" and will + # start the automated installation. + gen_iso_img =3D 'genisoimage -output ks.iso -volid "OEMDRV" ks.cfg' + subprocess.check_call(gen_iso_img, shell=3DTrue) + + def wait_for_shutdown(self): + """We wait for qemu to shutdown the VM and exit. + While this happens we display the console view + for easier debugging.""" + # The image creation is essentially done, + # so whether or not the wait is successful we want to + # wait for qemu to exit (the self.wait()) before we return. + try: + self.console_wait("reboot: Power down") + except Exception as e: + sys.stderr.write("Exception hit\n") + if isinstance(e, SystemExit) and e.code =3D=3D 0: + return 0 + traceback.print_exc() + finally: + self.wait() + + def build_base_image(self, dest_img): + """Run through the centos installer to create + a base image with name dest_img.""" + # We create the temp image, and only rename + # to destination when we are done. + img =3D dest_img + ".tmp" + # Create an empty image. + # We will provide this as the install destination. + qemu_img_create =3D "qemu-img create {} 50G".format(img) + subprocess.check_call(qemu_img_create, shell=3DTrue) + + # Create our kickstart file to be fed to the installer. + self.create_kickstart() + # Boot the install dvd with the params as our ks.iso + os_img =3D self._download_with_cache(self.image_link) + dvd_iso =3D "centos-8-dvd.iso" + subprocess.check_call(["cp", "-f", os_img, dvd_iso]) + extra_args =3D "-cdrom ks.iso" + extra_args +=3D " -drive file=3D{},if=3Dnone,id=3Ddrive1,cache=3Dw= riteback" + extra_args +=3D " -device virtio-blk,drive=3Ddrive1,bootindex=3D1" + extra_args =3D extra_args.format(dvd_iso).split(" ") + self.boot(img, extra_args=3Dextra_args) + self.console_wait_send("change the selection", "\n") + # We seem to need to hit esc (chr(27)) twice to abort the + # media check, which takes a long time. + # Waiting a bit seems to be more reliable before hitting esc. + self.console_wait("Checking") + time.sleep(5) + self.console_wait_send("Checking", chr(27)) + time.sleep(5) + self.console_wait_send("Checking", chr(27)) + print("Found Checking") + # Give sufficient time for the installer to create the image. + self.console_init(timeout=3D7200) + self.wait_for_shutdown() + os.rename(img, dest_img) + print("Done with base image build: {}".format(dest_img)) + + def check_create_base_img(self, img_base, img_dest): + """Create a base image using the installer. + We will use the base image if it exists. + This helps cut down on install time in case we + need to restart image creation, + since the base image creation can take a long time.""" + if not os.path.exists(img_base): + print("Generate new base image: {}".format(img_base)) + self.build_base_image(img_base); + else: + print("Use existing base image: {}".format(img_base)) + # Save a copy of the base image and copy it to dest. + # which we will use going forward. + subprocess.check_call(["cp", img_base, img_dest]) + + def boot(self, img, extra_args=3DNone): + aarch64vm.create_flash_images(self._tmpdir) + default_args =3D aarch64vm.get_pflash_args(self._tmpdir) + if extra_args: + extra_args.extend(default_args) + else: + extra_args =3D default_args + # We always add these performance tweaks + # because without them, we boot so slowly that we + # can time out finding the boot efi device. + if '-smp' not in extra_args and \ + '-smp' not in self._config['extra_args'] and \ + '-smp' not in self._args: + # Only add if not already there to give caller option to chang= e it. + extra_args.extend(["-smp", "8"]) + # We have overridden boot() since aarch64 has additional parameter= s. + # Call down to the base class method. + super(CentosAarch64VM, self).boot(img, extra_args=3Dextra_args) + + def build_image(self, img): + img_tmp =3D img + ".tmp" + self.check_create_base_img(img + ".base", img_tmp) + + # Boot the new image for the first time to finish installation. + self.boot(img_tmp) + self.console_init() + self.console_wait_send(self.login_prompt, "root\n") + self.console_wait_send("Password:", + "{}\n".format(self._config["root_pass"])) + + self.set_key_perm() + self.console_wait_send(self.prompt, "rpm -q centos-release\n") + enable_adapter =3D "sed -i 's/ONBOOT=3Dno/ONBOOT=3Dyes/g'" \ + " /etc/sysconfig/network-scripts/ifcfg-enp0s1\n" + self.console_wait_send(self.prompt, enable_adapter) + self.console_wait_send(self.prompt, "ifup enp0s1\n") + self.console_wait_send(self.prompt, + 'echo "qemu ALL=3D(ALL) NOPASSWD:ALL" | '\ + 'sudo tee /etc/sudoers.d/qemu\n') + self.console_wait(self.prompt) + + # Rest of the commands we issue through ssh. + self.wait_ssh(wait_root=3DTrue) + + # If the user chooses *not* to do the second phase, + # then we will jump right to the graceful shutdown + if self._config['install_cmds'] !=3D "": + install_cmds =3D self._config['install_cmds'].split(',') + for cmd in install_cmds: + self.ssh_root(cmd) + self.ssh_root("poweroff") + self.wait_for_shutdown() + os.rename(img_tmp, img) + print("image creation complete: {}".format(img)) + return 0 + +if __name__ =3D=3D "__main__": + defaults =3D aarch64vm.get_config_defaults(CentosAarch64VM, DEFAULT_CO= NFIG) + sys.exit(basevm.main(CentosAarch64VM, defaults)) --=20 2.17.1 From nobody Sun May 19 14:14:26 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=linaro.org ARC-Seal: i=1; a=rsa-sha256; t=1588263957; cv=none; d=zohomail.com; s=zohoarc; b=Kxfd11AE0tF3hlI+aSDVM8jpW1DByJM+GlWgV220ZH5YaSw3xxIY2R5gLtPdfIvwo+jBhtjMrAbjyvE3Hp1Tv1NAuXGaqcix3SqW2sIwZ+H6IIQI4vUcPgtWuKzqcsZ6ZUk0us1d2MGKrPGLYehoJNWhhvMoMQRcTOH6xWHhW8c= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1588263957; h=Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:Message-ID:References:Sender:Subject:To; bh=semgx67YJQ/DRrCJe76N29NtkhEf2cXk846eTkDv8ro=; b=BRgDa6AXFhB6KAX6wVb00u/1fbFjVbmQlU4N79r8XNU7QE9KHvIX5DyXpdsguSFbTT9HRV5JX6flPMSo8T29hjb1zp61Oc72ubss1hHcZkghZc5l9vTgM4+Dvo681XzWaxsgkVPC6YYSLVUinJdxh+BQnF3T5+B/gxj06d9l1F0= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1588263957248724.9124206508181; Thu, 30 Apr 2020 09:25:57 -0700 (PDT) Received: from localhost ([::1]:41718 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jUC0Z-0005GE-RU for importer@patchew.org; Thu, 30 Apr 2020 12:25:55 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:45464) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jUBdg-0000fY-Qe for qemu-devel@nongnu.org; Thu, 30 Apr 2020 12:03:17 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.90_1) (envelope-from ) id 1jUBaJ-0003e4-KC for qemu-devel@nongnu.org; Thu, 30 Apr 2020 12:02:16 -0400 Received: from mail-qk1-x741.google.com ([2607:f8b0:4864:20::741]:41977) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1jUBaJ-0003Zj-4Q for qemu-devel@nongnu.org; Thu, 30 Apr 2020 11:58:47 -0400 Received: by mail-qk1-x741.google.com with SMTP id n143so6202441qkn.8 for ; Thu, 30 Apr 2020 08:58:46 -0700 (PDT) Received: from Rfoley-MA01.hsd1.ma.comcast.net ([2601:199:4480:60c0:7489:3328:3806:7c3]) by smtp.gmail.com with ESMTPSA id v16sm305254qkf.80.2020.04.30.08.58.44 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 30 Apr 2020 08:58:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=semgx67YJQ/DRrCJe76N29NtkhEf2cXk846eTkDv8ro=; b=co00Qa2HiBfBiaGGAmnwQNucIdza0PydCHJLSBb4wt890/zaEpmml/WiUgwXVt6wyN 2e1mg80rHRhPCEQIl7QFMA+46HddRrrurHExqcaHFodERVxzOIOTSxmITxBjbVudSSLf BSNtSqtAU1tcZAB7+AJMpD7J1sEYkmnMc+FL8PqjK54B7ydMt0CRgifkruHLbON8lAZl 9JY6IUFH3YyQnI9FQjZZUfPLXkuTmk0Qy8GT+QojE4xwO43QAU/5eROrTwtztQBmBCST IHk0eUpbELZSZv2M18A2akuJCpqMhtIkS/aQsDJm48ccORpn9NZwMXfI9/pcRW2D0dcN w6tQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=semgx67YJQ/DRrCJe76N29NtkhEf2cXk846eTkDv8ro=; b=sOhACmVwrV8RhxYIn/OebTZq6JE2igEmoYnOc43EoMTnJRjazBN30hERmYigU1Uiz9 LTmVbl9InIcXM1vXI0jRm7ldg5xpDo4BTvnSO400nMZ7uCwkjQF2KIUt2/Dcj2emqahM XMG/T5oXpcXpD4qP5DpQr4gU6gtNnxv3LU1Zd/FanHmukha77qJR/+BCjf1z1GgXKJ+A 4S1ABD9zP5RVpMpxQxeoyQz4M3RBT8PY+HewDOuyqvy7mEFp2MMR8UItEn0/mGFuiZzR iskGVs6U4gmIW9tr69f3fEdD2hcaAwDPLp8Td3+u4j5QYP5P37xsOslzcOO3h0kK79n0 epQA== X-Gm-Message-State: AGi0Puak9z9bcihIvsmSFZg/Eon4SAfmsZODLOprbrgUhA+x3jA7bFJS lMcOPJdKahIn/El4JcANd5xSFISXoWXHTg== X-Google-Smtp-Source: APiQypKnw2H3LPgM3isGT8xgwQIcCoHa+kkQqFWcCFZVTCd0mgh8PAFOGS4mkBjbRDSHVskpOOengg== X-Received: by 2002:a37:7ac2:: with SMTP id v185mr4042557qkc.386.1588262325656; Thu, 30 Apr 2020 08:58:45 -0700 (PDT) From: Robert Foley To: qemu-devel@nongnu.org Subject: [PATCH v5 8/9] tests/vm: change scripts to use self._config Date: Thu, 30 Apr 2020 11:57:17 -0400 Message-Id: <20200430155718.212-9-robert.foley@linaro.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20200430155718.212-1-robert.foley@linaro.org> References: <20200430155718.212-1-robert.foley@linaro.org> Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=2607:f8b0:4864:20::741; envelope-from=robert.foley@linaro.org; helo=mail-qk1-x741.google.com X-detected-operating-system: by eggs.gnu.org: Error: [-] PROGRAM ABORT : Malformed IPv6 address (bad octet value). Location : parse_addr6(), p0f-client.c:67 X-Received-From: 2607:f8b0:4864:20::741 X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: philmd@redhat.com, alex.bennee@linaro.org, robert.foley@linaro.org, peter.puhov@linaro.org Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" This change converts existing scripts to using for example self.ROOT_PASS, to self._config['root_pass']. We made similar changes for GUEST_USER, and GUEST_PASS. This allows us also to remove the change in basevm.py, which adds __getattr__ for backwards compatibility. Signed-off-by: Robert Foley Reviewed-by: Peter Puhov --- tests/vm/basevm.py | 11 ++--------- tests/vm/fedora | 17 +++++++++-------- tests/vm/freebsd | 16 ++++++++-------- tests/vm/netbsd | 19 ++++++++++--------- tests/vm/openbsd | 17 +++++++++-------- 5 files changed, 38 insertions(+), 42 deletions(-) diff --git a/tests/vm/basevm.py b/tests/vm/basevm.py index dddd4711ff..5dcae07b7f 100644 --- a/tests/vm/basevm.py +++ b/tests/vm/basevm.py @@ -181,13 +181,6 @@ class BaseVM(object): self.console_init(timeout=3Dtimeout) self.console_wait(wait_string) =20 - def __getattr__(self, name): - # Support direct access to config by key. - # for example, access self._config['cpu'] by self.cpu - if name.lower() in self._config.keys(): - return self._config[name.lower()] - return object.__getattribute__(self, name) - def _download_with_cache(self, url, sha256sum=3DNone, sha512sum=3DNone= ): def check_sha256sum(fname): if not sha256sum: @@ -237,13 +230,13 @@ class BaseVM(object): return r =20 def ssh(self, *cmd): - return self._ssh_do(self.GUEST_USER, cmd, False) + return self._ssh_do(self._config["guest_user"], cmd, False) =20 def ssh_root(self, *cmd): return self._ssh_do("root", cmd, False) =20 def ssh_check(self, *cmd): - self._ssh_do(self.GUEST_USER, cmd, True) + self._ssh_do(self._config["guest_user"], cmd, True) =20 def ssh_root_check(self, *cmd): self._ssh_do("root", cmd, True) diff --git a/tests/vm/fedora b/tests/vm/fedora index bd9c6cf295..f536a92678 100755 --- a/tests/vm/fedora +++ b/tests/vm/fedora @@ -109,20 +109,20 @@ class FedoraVM(basevm.BaseVM): =20 self.console_wait_send("7) [!] Root password", "7\n") self.console_wait("Password:") - self.console_send("%s\n" % self.ROOT_PASS) + self.console_send("%s\n" % self._config["root_pass"]) self.console_wait("Password (confirm):") - self.console_send("%s\n" % self.ROOT_PASS) + self.console_send("%s\n" % self._config["root_pass"]) =20 self.console_wait_send("8) [ ] User creation", "8\n") self.console_wait_send("1) [ ] Create user", "1\n") self.console_wait_send("3) User name", "3\n") - self.console_wait_send("ENTER:", "%s\n" % self.GUEST_USER) + self.console_wait_send("ENTER:", "%s\n" % self._config["guest_user= "]) self.console_wait_send("4) [ ] Use password", "4\n") self.console_wait_send("5) Password", "5\n") self.console_wait("Password:") - self.console_send("%s\n" % self.GUEST_PASS) + self.console_send("%s\n" % self._config["guest_pass"]) self.console_wait("Password (confirm):") - self.console_send("%s\n" % self.GUEST_PASS) + self.console_send("%s\n" % self._config["guest_pass"]) self.console_wait_send("7) Groups", "c\n") =20 while True: @@ -140,7 +140,7 @@ class FedoraVM(basevm.BaseVM): if good: break time.sleep(10) - self.console_send("r\n" % self.GUEST_PASS) + self.console_send("r\n" % self._config["guest_pass"]) =20 self.console_wait_send("'b' to begin install", "b\n") =20 @@ -151,12 +151,13 @@ class FedoraVM(basevm.BaseVM): =20 # setup qemu user prompt =3D " ~]$" - self.console_ssh_init(prompt, self.GUEST_USER, self.GUEST_PASS) + self.console_ssh_init(prompt, self._config["guest_user"], + self._config["guest_pass"]) self.console_wait_send(prompt, "exit\n") =20 # setup root user prompt =3D " ~]#" - self.console_ssh_init(prompt, "root", self.ROOT_PASS) + self.console_ssh_init(prompt, "root", self._config["root_pass"]) self.console_sshd_config(prompt) =20 # setup virtio-blk #1 (tarfile) diff --git a/tests/vm/freebsd b/tests/vm/freebsd index 298967fe9c..898393b58d 100755 --- a/tests/vm/freebsd +++ b/tests/vm/freebsd @@ -114,9 +114,9 @@ class FreeBSDVM(basevm.BaseVM): =20 # post-install configuration self.console_wait("New Password:") - self.console_send("%s\n" % self.ROOT_PASS) + self.console_send("%s\n" % self._config["root_pass"]) self.console_wait("Retype New Password:") - self.console_send("%s\n" % self.ROOT_PASS) + self.console_send("%s\n" % self._config["root_pass"]) =20 self.console_wait_send("Network Configuration", "\n") self.console_wait_send("IPv4", "y") @@ -135,9 +135,9 @@ class FreeBSDVM(basevm.BaseVM): # qemu user self.console_wait_send("Add User Accounts", "y") self.console_wait("Username") - self.console_send("%s\n" % self.GUEST_USER) + self.console_send("%s\n" % self._config["guest_user"]) self.console_wait("Full name") - self.console_send("%s\n" % self.GUEST_USER) + self.console_send("%s\n" % self._config["guest_user"]) self.console_wait_send("Uid", "\n") self.console_wait_send("Login group", "\n") self.console_wait_send("Login group", "\n") @@ -149,9 +149,9 @@ class FreeBSDVM(basevm.BaseVM): self.console_wait_send("Use an empty password", "\n") self.console_wait_send("Use a random password", "\n") self.console_wait("Enter password:") - self.console_send("%s\n" % self.GUEST_PASS) + self.console_send("%s\n" % self._config["guest_pass"]) self.console_wait("Enter password again:") - self.console_send("%s\n" % self.GUEST_PASS) + self.console_send("%s\n" % self._config["guest_pass"]) self.console_wait_send("Lock out", "\n") self.console_wait_send("OK", "yes\n") self.console_wait_send("Add another user", "no\n") @@ -165,12 +165,12 @@ class FreeBSDVM(basevm.BaseVM): =20 # setup qemu user prompt =3D "$" - self.console_ssh_init(prompt, self.GUEST_USER, self.GUEST_PASS) + self.console_ssh_init(prompt, self._config["guest_user"], self._co= nfig["guest_pass"]) self.console_wait_send(prompt, "exit\n") =20 # setup root user prompt =3D "root@freebsd:~ #" - self.console_ssh_init(prompt, "root", self.ROOT_PASS) + self.console_ssh_init(prompt, "root", self._config["root_pass"]) self.console_sshd_config(prompt) =20 # setup serial console diff --git a/tests/vm/netbsd b/tests/vm/netbsd index b10c9d429d..7e54fe8322 100755 --- a/tests/vm/netbsd +++ b/tests/vm/netbsd @@ -121,24 +121,24 @@ class NetBSDVM(basevm.BaseVM): self.console_wait_send("d: Change root password", "d\n") self.console_wait_send("a: Yes", "a\n") self.console_wait("New password:") - self.console_send("%s\n" % self.ROOT_PASS) + self.console_send("%s\n" % self._config["root_pass"]) self.console_wait("New password:") - self.console_send("%s\n" % self.ROOT_PASS) + self.console_send("%s\n" % self._config["root_pass"]) self.console_wait("Retype new password:") - self.console_send("%s\n" % self.ROOT_PASS) + self.console_send("%s\n" % self._config["root_pass"]) =20 self.console_wait_send("o: Add a user", "o\n") self.console_wait("username") - self.console_send("%s\n" % self.GUEST_USER) + self.console_send("%s\n" % self._config["guest_pass"]) self.console_wait("to group wheel") self.console_wait_send("a: Yes", "a\n") self.console_wait_send("a: /bin/sh", "a\n") self.console_wait("New password:") - self.console_send("%s\n" % self.GUEST_PASS) + self.console_send("%s\n" % self._config["guest_pass"]) self.console_wait("New password:") - self.console_send("%s\n" % self.GUEST_PASS) + self.console_send("%s\n" % self._config["guest_pass"]) self.console_wait("Retype new password:") - self.console_send("%s\n" % self.GUEST_PASS) + self.console_send("%s\n" % self._config["guest_pass"]) =20 self.console_wait_send("a: Configure network", "a\n") self.console_wait_send("a: vioif0", "a\n") @@ -171,12 +171,13 @@ class NetBSDVM(basevm.BaseVM): =20 # setup qemu user prompt =3D "localhost$" - self.console_ssh_init(prompt, self.GUEST_USER, self.GUEST_PASS) + self.console_ssh_init(prompt, self._config["guest_user"], + self._config["guest_pass"]) self.console_wait_send(prompt, "exit\n") =20 # setup root user prompt =3D "localhost#" - self.console_ssh_init(prompt, "root", self.ROOT_PASS) + self.console_ssh_init(prompt, "root", self._config["root_pass"]) self.console_sshd_config(prompt) =20 # setup virtio-blk #1 (tarfile) diff --git a/tests/vm/openbsd b/tests/vm/openbsd index 0b705f4945..04a532c763 100755 --- a/tests/vm/openbsd +++ b/tests/vm/openbsd @@ -99,9 +99,9 @@ class OpenBSDVM(basevm.BaseVM): self.console_wait_send("Which network interface", "done\n") self.console_wait_send("DNS domain name", "localnet\n") self.console_wait("Password for root account") - self.console_send("%s\n" % self.ROOT_PASS) + self.console_send("%s\n" % self._config["root_pass"]) self.console_wait("Password for root account") - self.console_send("%s\n" % self.ROOT_PASS) + self.console_send("%s\n" % self._config["root_pass"]) self.console_wait_send("Start sshd(8)", "yes\n") self.console_wait_send("X Window System", "\n") self.console_wait_send("xenodm", "\n") @@ -109,13 +109,13 @@ class OpenBSDVM(basevm.BaseVM): self.console_wait_send("Which speed", "\n") =20 self.console_wait("Setup a user") - self.console_send("%s\n" % self.GUEST_USER) + self.console_send("%s\n" % self._config["guest_user"]) self.console_wait("Full name") - self.console_send("%s\n" % self.GUEST_USER) + self.console_send("%s\n" % self._config["guest_user"]) self.console_wait("Password") - self.console_send("%s\n" % self.GUEST_PASS) + self.console_send("%s\n" % self._config["guest_pass"]) self.console_wait("Password") - self.console_send("%s\n" % self.GUEST_PASS) + self.console_send("%s\n" % self._config["guest_pass"]) =20 self.console_wait_send("Allow root ssh login", "yes\n") self.console_wait_send("timezone", "UTC\n") @@ -136,12 +136,13 @@ class OpenBSDVM(basevm.BaseVM): =20 # setup qemu user prompt =3D "$" - self.console_ssh_init(prompt, self.GUEST_USER, self.GUEST_PASS) + self.console_ssh_init(prompt, self._config["guest_user"], + self._config["guest_pass"]) self.console_wait_send(prompt, "exit\n") =20 # setup root user prompt =3D "openbsd#" - self.console_ssh_init(prompt, "root", self.ROOT_PASS) + self.console_ssh_init(prompt, "root", self._config["root_pass"]) self.console_sshd_config(prompt) =20 # setup virtio-blk #1 (tarfile) --=20 2.17.1 From nobody Sun May 19 14:14:26 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=linaro.org ARC-Seal: i=1; a=rsa-sha256; t=1588263374; cv=none; d=zohomail.com; s=zohoarc; b=KdjuubDEiGcZrJCLM4aib992d30BXaV1PWcfTtzl4GDxLNm6s3OlJUogiuUYY/keHTsyBwMVK+m7RClMMYsLTwgcTsyJTYQSzogohoji6/9BTVSHjRIr4SH3osShAVwNZB3N9RBOFk9RNM8sKhj97f7I/k5QLszjHUrw7XyZeQk= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1588263374; h=Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:Message-ID:References:Sender:Subject:To; bh=nN+S0/0+qWidt3Axw93y5S26kGU8LdXlgWZCTrxq+qI=; b=KCDPkf/dOyKL9CYU42VqhmwZNrDhiL27rgximlCQga6PuwYEcerJwTlfQmBjiiKIKjoAMqxW2Z9C3FOJnBc97it5xjfRxr2XJEkcY90b6Bg5BrVcNfyHqadvSUUyMRkUS84NnYPHCzs7xZdyFDxJr/C1vlZjgAfMuSUSMbqez98= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1588263374295376.80756556287633; Thu, 30 Apr 2020 09:16:14 -0700 (PDT) Received: from localhost ([::1]:46286 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jUBrA-000761-Tn for importer@patchew.org; Thu, 30 Apr 2020 12:16:12 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:45474) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jUBdi-0000iP-84 for qemu-devel@nongnu.org; Thu, 30 Apr 2020 12:03:18 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.90_1) (envelope-from ) id 1jUBaL-0003iu-7X for qemu-devel@nongnu.org; Thu, 30 Apr 2020 12:02:18 -0400 Received: from mail-qk1-x744.google.com ([2607:f8b0:4864:20::744]:42597) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1jUBaK-0003g9-PI for qemu-devel@nongnu.org; Thu, 30 Apr 2020 11:58:48 -0400 Received: by mail-qk1-x744.google.com with SMTP id b188so6185059qkd.9 for ; Thu, 30 Apr 2020 08:58:48 -0700 (PDT) Received: from Rfoley-MA01.hsd1.ma.comcast.net ([2601:199:4480:60c0:7489:3328:3806:7c3]) by smtp.gmail.com with ESMTPSA id v16sm305254qkf.80.2020.04.30.08.58.45 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 30 Apr 2020 08:58:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=nN+S0/0+qWidt3Axw93y5S26kGU8LdXlgWZCTrxq+qI=; b=ruKCgbXtR0eFReK0JKQDOLJ26jrJ1osiFzr59TDE7Wl7F3YfsMbelCFNpzjfIUmVKr XtPWOYU3SQrhET9s8AsLix32x3NMcexzVLB/tEjviJIJawiGdj6iBf4nsAUPrHrThip+ jda0g6CXl5kHA6gnG7flPEIFhf0Amm4dg8xGxGt8Q+sre/mmZC7W8pGpLGL3a7Bu6jNV qD8T9awoJcnOX+ApEJ938/JnlSUj+SJ63tCFsJY7+hSu1nCgevsEANTJpSdHnNbi03Lq MT5K+Hvn09Nv+KyOHboxu4SrnGMWWyRnAONuZWNup6PwId+baskqDlMok+6OFJviwECr Do/Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=nN+S0/0+qWidt3Axw93y5S26kGU8LdXlgWZCTrxq+qI=; b=hgDevIUmrHS97Jbn9tNapnMgbrZ6o0mJ6Dj4ckpK7X5N3nvNVWqWQs/h77zDtFTnrl w8OTClBGrB9UXxCRkjrTyBFKHnHHsI3gdCldUi6wLgLY0UlpTSW3GDyqEbloShgtwfFz XStTEML0FwDXcGObbIQP2n5qpjfXzFQInPZxtUfGlwLridS9pfhNzFIJLexa3DDT1zhC OK57ahnObwHBtYoE7xabEr/1N/wR5iAanQ5HmmbHn12H1zS7HTx1FTfyxTGJt/Feq1JQ sMRqcHBlCFCdjje8eoto3UkAm386A9PMdrOZ+s9luYB4d2m6IXI2S+nJMKNH58kf+TZ3 ws3g== X-Gm-Message-State: AGi0Pub2ci+U+uYgdsuEBkNJ9kef629eHCiaVy2QHoU7O8Pjh30bBhrI +yazk3jDzve5yTqHUSqpXMq93I8DBnMhhQ== X-Google-Smtp-Source: APiQypJRHlhWx6QTfcXqPYGr9GMe5DAonOQIVA4sJlHUnvqa86ZnWsovH/aUANEMEy/2reeuh/xGyg== X-Received: by 2002:ae9:e896:: with SMTP id a144mr4427455qkg.146.1588262326983; Thu, 30 Apr 2020 08:58:46 -0700 (PDT) From: Robert Foley To: qemu-devel@nongnu.org Subject: [PATCH v5 9/9] tests/vm: Add workaround to consume console Date: Thu, 30 Apr 2020 11:57:18 -0400 Message-Id: <20200430155718.212-10-robert.foley@linaro.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20200430155718.212-1-robert.foley@linaro.org> References: <20200430155718.212-1-robert.foley@linaro.org> Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=2607:f8b0:4864:20::744; envelope-from=robert.foley@linaro.org; helo=mail-qk1-x744.google.com X-detected-operating-system: by eggs.gnu.org: Error: [-] PROGRAM ABORT : Malformed IPv6 address (bad octet value). Location : parse_addr6(), p0f-client.c:67 X-Received-From: 2607:f8b0:4864:20::744 X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: philmd@redhat.com, alex.bennee@linaro.org, robert.foley@linaro.org, peter.puhov@linaro.org Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" The ConsoleSocket object provides a socket interface which will consume all arriving characters on the socket, but will provide those chars via recv() as would a regular socket. This is a workaround we found was needed since there is a known issue where QEMU will hang waiting for console characters to be consumed. We also add the option of logging the console to a file. Signed-off-by: Robert Foley Reviewed-by: Peter Puhov --- python/qemu/console_socket.py | 162 ++++++++++++++++++++++++++++++++++ python/qemu/machine.py | 23 ++++- tests/vm/Makefile.include | 4 + tests/vm/basevm.py | 24 ++++- 4 files changed, 205 insertions(+), 8 deletions(-) create mode 100644 python/qemu/console_socket.py diff --git a/python/qemu/console_socket.py b/python/qemu/console_socket.py new file mode 100644 index 0000000000..a1f74e60ac --- /dev/null +++ b/python/qemu/console_socket.py @@ -0,0 +1,162 @@ +#!/usr/bin/env python3 +# +# This python module implements a ConsoleSocket object which is +# designed always drain the socket itself, and place +# the bytes into a in memory buffer for later processing. +# +# Optionally a file path can be passed in and we will also +# dump the characters to this file for debug. +# +# Copyright 2020 Linaro +# +# Authors: +# Robert Foley +# +# This code is licensed under the GPL version 2 or later. See +# the COPYING file in the top-level directory. +# +import asyncore +import socket +import threading +import io +import os +import sys +from collections import deque +import time +import traceback + +class ConsoleSocket(asyncore.dispatcher): + + def __init__(self, address, file=3DNone): + self._recv_timeout_sec =3D 300 + self._buffer =3D deque() + self._asyncore_thread =3D None + self._sock =3D socket.socket(socket.AF_UNIX, socket.SOCK_STREAM) + self._sock.connect(address) + self._logfile =3D None + if file: + self._logfile =3D open(file, "w") + asyncore.dispatcher.__init__(self, sock=3Dself._sock) + self._thread_start() + self._open =3D True + + def _thread_start(self): + """Kick off a thread to wait on the asyncore.loop""" + if self._asyncore_thread is not None: + return + self._asyncore_thread =3D threading.Thread(target=3Dasyncore.loop, + kwargs=3D{'timeout':1}) + self._asyncore_thread.daemon =3D True + self._asyncore_thread.start() + + def handle_close(self): + """redirect close to base class""" + # Call the base class close, but not self.close() since + # handle_close() occurs in the context of the thread which + # self.close() attempts to join. + asyncore.dispatcher.close(self) + + def close(self): + """Close the base object and wait for the thread to terminate""" + if self._open: + self._open =3D False + asyncore.dispatcher.close(self) + if self._asyncore_thread is not None: + thread, self._asyncore_thread =3D self._asyncore_thread, N= one + thread.join() + if self._logfile: + self._logfile.close() + self._logfile =3D None + + def handle_read(self): + """process arriving characters into in memory _buffer""" + try: + data =3D asyncore.dispatcher.recv(self, 1) + # latin1 is needed since there are some chars + # we are receiving that cannot be encoded to utf-8 + # such as 0xe2, 0x80, 0xA6. + string =3D data.decode("latin1") + except: + print("Exception seen.") + traceback.print_exc() + return + if self._logfile: + self._logfile.write("{}".format(string)) + self._logfile.flush() + for c in string: + self._buffer.append(c) + + def recv(self, n=3D1): + """Return chars from in memory buffer""" + start_time =3D time.time() + while len(self._buffer) < n: + time.sleep(0.1) + elapsed_sec =3D time.time() - start_time + if elapsed_sec > self._recv_timeout_sec: + raise socket.timeout + chars =3D ''.join([self._buffer.popleft() for i in range(n)]) + # We choose to use latin1 to remain consistent with + # handle_read() and give back the same data as the user would + # receive if they were reading directly from the + # socket w/o our intervention. + return chars.encode("latin1") + + def set_blocking(self): + """Maintain compatibility with socket API""" + pass + + def settimeout(self, seconds): + """Set current timeout on recv""" + self._recv_timeout_sec =3D seconds + +class ByteBuffer(deque): + """Simple in memory buffer with read/write interface""" + def write(self, bytes): + for i in bytes: + self.append(i) + def read(self, n): + return ''.join([self.popleft() for i in range(n)]) + +if __name__ =3D=3D '__main__': + # Brief test to exercise the above code. + # The ConsoleSocket will ship some data to the server, + # the server will echo it back and the client will echo what it receiv= ed. + + # First remove the socket. + address =3D "./test_console_socket" + if os.path.exists(address): + os.unlink(address) + + # Create the server side. + server_socket =3D socket.socket(socket.AF_UNIX, socket.SOCK_STREAM) + server_socket.bind(address) + server_socket.listen(1) + + # Create the object we are trying to test. + console_socket =3D ConsoleSocket(address, file=3D"./logfile.txt") + + # Generate some data and ship it over the socket. + send_data =3D "" + for i in range(10): + send_data +=3D "this is a test message {}\n".format(i) + console_socket.send(send_data.encode('latin1')) + connection, client_address =3D server_socket.accept() + + # Process the data on the server and ship it back. + data =3D connection.recv(len(send_data)) + print("server received: {}".format(data)) + print("server: sending data back to the client") + connection.sendall(data) + + # Client receives teh bytes and displays them. + print("client: receiving bytes") + bytes =3D console_socket.recv(len(data)) + recv_data =3D bytes.decode('latin1') + print("client received: {}".format(recv_data)) + assert(recv_data =3D=3D send_data) + # Close console connection first, then close server. + console_socket.close() + connection.close() + server_socket.close() + print("test successful.") + diff --git a/python/qemu/machine.py b/python/qemu/machine.py index b9a98e2c86..30e7a16b92 100644 --- a/python/qemu/machine.py +++ b/python/qemu/machine.py @@ -24,6 +24,7 @@ import subprocess import shutil import socket import tempfile +from qemu.console_socket import ConsoleSocket =20 from . import qmp =20 @@ -71,7 +72,8 @@ class QEMUMachine(object): =20 def __init__(self, binary, args=3DNone, wrapper=3DNone, name=3DNone, test_dir=3D"/var/tmp", monitor_address=3DNone, - socket_scm_helper=3DNone, sock_dir=3DNone): + socket_scm_helper=3DNone, sock_dir=3DNone, + drain_console=3DFalse, console_log=3DNone): ''' Initialize a QEMUMachine =20 @@ -82,6 +84,9 @@ class QEMUMachine(object): @param test_dir: where to create socket and log file @param monitor_address: address for QMP monitor @param socket_scm_helper: helper program, required for send_fd_scm= () + @param sock_dir: where to create socket (overrides test_dir for so= ck) + @param console_log: (optional) path to console log file + @param drain_console: (optional) True to drain console socket to b= uffer @note: Qemu process is not started until launch() is used. ''' if args is None: @@ -118,6 +123,12 @@ class QEMUMachine(object): self._console_address =3D None self._console_socket =3D None self._remove_files =3D [] + self._console_log_path =3D console_log + if self._console_log_path: + # In order to log the console, buffering needs to be enabled. + self._drain_console =3D True + else: + self._drain_console =3D drain_console =20 # just in case logging wasn't configured by the main script: logging.basicConfig() @@ -568,7 +579,11 @@ class QEMUMachine(object): Returns a socket connected to the console """ if self._console_socket is None: - self._console_socket =3D socket.socket(socket.AF_UNIX, - socket.SOCK_STREAM) - self._console_socket.connect(self._console_address) + if self._drain_console: + self._console_socket =3D ConsoleSocket(self._console_addre= ss, + file=3Dself._console_l= og_path) + else: + self._console_socket =3D socket.socket(socket.AF_UNIX, + socket.SOCK_STREAM) + self._console_socket.connect(self._console_address) return self._console_socket diff --git a/tests/vm/Makefile.include b/tests/vm/Makefile.include index 419e0f8613..fb61f10619 100644 --- a/tests/vm/Makefile.include +++ b/tests/vm/Makefile.include @@ -50,6 +50,7 @@ endif @echo " J=3D[0..9]* - Override the -jN parameter for make= commands" @echo " DEBUG=3D1 - Enable verbose output on host and i= nteractive debugging" @echo " BOOT_CONSOLE=3D1 - Show the console output at boot ti= me. " + @echo " LOG_CONSOLE=3D1 - Log console to file in: ~/.cache/qe= mu-vm " @echo " V=3D1 - Enable verbose ouput on host and guest commands" @echo " QEMU_LOCAL=3D1 - Use QEMU binary local to this= build." @echo " QEMU=3D/path/to/qemu - Change path to QEMU binary" @@ -74,6 +75,7 @@ $(IMAGES_DIR)/%.img: $(SRC_PATH)/tests/vm/% \ $(PYTHON) $< \ $(if $(V)$(DEBUG), --debug) \ $(if $(BOOT_CONSOLE),--boot-console) \ + $(if $(LOG_CONSOLE),--log-console) \ --build-path $(BUILD_DIR)\ --image "$@" \ --force \ @@ -90,6 +92,7 @@ vm-build-%: $(IMAGES_DIR)/%.img $(if $(J),--jobs $(J)) \ $(if $(V),--verbose) \ $(if $(BOOT_CONSOLE),--boot-console) \ + $(if $(LOG_CONSOLE),--log-console) \ --build-path $(BUILD_DIR)\ --image "$<" \ $(if $(BUILD_TARGET),--build-target $(BUILD_TARGET)) \ @@ -112,6 +115,7 @@ vm-boot-ssh-%: $(IMAGES_DIR)/%.img $(PYTHON) $(SRC_PATH)/tests/vm/$* \ $(if $(J),--jobs $(J)) \ $(if $(BOOT_CONSOLE),--boot-console) \ + $(if $(LOG_CONSOLE),--log-console) \ --build-path $(BUILD_DIR)\ --image "$<" \ --interactive \ diff --git a/tests/vm/basevm.py b/tests/vm/basevm.py index 5dcae07b7f..258c0d5482 100644 --- a/tests/vm/basevm.py +++ b/tests/vm/basevm.py @@ -82,12 +82,14 @@ class BaseVM(object): poweroff =3D "poweroff" # enable IPv6 networking ipv6 =3D True + # This is the timeout on the wait for console bytes. + socket_timeout =3D 120 # Scale up some timeouts under TCG. # 4 is arbitrary, but greater than 2, # since we found we need to wait more than twice as long. tcg_ssh_timeout_multiplier =3D 4 def __init__(self, debug=3DFalse, vcpus=3DNone, config=3DNone, - boot_console=3DFalse, build_path=3DNone): + boot_console=3DFalse, build_path=3DNone, log_console=3DFa= lse): self._guest =3D None self._build_path =3D build_path self._boot_console =3D boot_console @@ -114,6 +116,11 @@ class BaseVM(object): open(self._ssh_tmp_pub_key_file, "w").write(self._config['ssh_pub_key']) =20 + self._console_log_path =3D None + if log_console: + self._console_log_path =3D \ + os.path.join(os.path.expanduser("~/.cache/qemu-vm= "), + "{}.install.log".format(self.name)) self.debug =3D debug self._stderr =3D sys.stderr self._devnull =3D open(os.devnull, "w") @@ -269,7 +276,9 @@ class BaseVM(object): args +=3D self._data_args + extra_args + self._config['extra_args'] logging.debug("QEMU args: %s", " ".join(args)) qemu_path =3D get_qemu_path(self.arch, self._build_path) - guest =3D QEMUMachine(binary=3Dqemu_path, args=3Dargs) + guest =3D QEMUMachine(binary=3Dqemu_path, args=3Dargs, + console_log=3Dself._console_log_path, + drain_console=3DTrue) guest.set_machine(self._config['machine']) guest.set_console() try: @@ -283,6 +292,8 @@ class BaseVM(object): raise atexit.register(self.shutdown) self._guest =3D guest + # Init console so we can start consuming the chars. + self.console_init() usernet_info =3D guest.qmp("human-monitor-command", command_line=3D"info usernet") self.ssh_port =3D None @@ -294,7 +305,9 @@ class BaseVM(object): raise Exception("Cannot find ssh port from 'info usernet':\n%s= " % \ usernet_info) =20 - def console_init(self, timeout =3D 120): + def console_init(self, timeout =3D None): + if timeout =3D=3D None: + timeout =3D self.socket_timeout vm =3D self._guest vm.console_socket.settimeout(timeout) self.console_raw_path =3D os.path.join(vm._temp_dir, @@ -575,6 +588,8 @@ def parse_args(vmcls): help=3D"Show console during boot. ") parser.add_option("--build-path", default=3DNone, help=3D"Path of build directory. ") + parser.add_option("--log-console", action=3D"store_true", + help=3D"Log console to file.") parser.disable_interspersed_args() return parser.parse_args() =20 @@ -591,7 +606,8 @@ def main(vmcls, config=3DNone): else logging.WARN)) vm =3D vmcls(debug=3Dargs.debug, vcpus=3Dargs.jobs, config=3Dconfi= g, boot_console=3Dargs.boot_console, - build_path=3Dargs.build_path) + build_path=3Dargs.build_path, + log_console=3Dargs.log_console) if args.build_image: if os.path.exists(args.image) and not args.force: sys.stderr.writelines(["Image file exists: %s\n" % args.im= age, --=20 2.17.1