From nobody Sat Jun 1 02:33:29 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=linaro.org ARC-Seal: i=1; a=rsa-sha256; t=1589312112; cv=none; d=zohomail.com; s=zohoarc; b=bK6BmElSzmigwroe99CqcOWJkqYwtzekkXv8EwaBE0Faixq2Jp+aiY1GAnQpvJIv7dRaIzFRuC53+ZDH+KB5eP9dGSVfHx2dCLOWtbLHPeL0x0SgWiwS8Yn+OYLQwTHgIYRPZAKnpDqoj+PatA+tuMPyZ+QeYOWUPJ9ehl+GCtk= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1589312112; h=Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:Message-ID:References:Sender:Subject:To; bh=IHVxQvHLGmuAsOe7Our5n+V2vriioUviLGm8LCdzEg4=; b=dvSWcljoQLPcsD2pgXqwVTIfjV7kzd6I8SeuRjs9AHWy7ffdCm3VPC3Jqv+4NrgjSGQHA7p52O7qJbsn33RTstiDt1TTcxrjGi6h4pr+mwtAAIajjiT5FclraUAJkRdU0OlafPL6O2EwCkS2He9vxGljCsx2ynNVKtotSTlOFD0= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1589312112815465.14612252549193; Tue, 12 May 2020 12:35:12 -0700 (PDT) Received: from localhost ([::1]:41378 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jYagJ-0001s7-9J for importer@patchew.org; Tue, 12 May 2020 15:35:11 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:47242) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1jYaej-0000K8-L0 for qemu-devel@nongnu.org; Tue, 12 May 2020 15:33:34 -0400 Received: from mail-qk1-x72d.google.com ([2607:f8b0:4864:20::72d]:41612) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1jYaeh-0001q7-Pm for qemu-devel@nongnu.org; Tue, 12 May 2020 15:33:33 -0400 Received: by mail-qk1-x72d.google.com with SMTP id n14so14913763qke.8 for ; Tue, 12 May 2020 12:33:31 -0700 (PDT) Received: from Rfoley-MA01.hsd1.ma.comcast.net ([2601:199:4480:60c0:d31:9f53:b3eb:9b53]) by smtp.gmail.com with ESMTPSA id j45sm2100998qtk.14.2020.05.12.12.33.28 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 12 May 2020 12:33:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=IHVxQvHLGmuAsOe7Our5n+V2vriioUviLGm8LCdzEg4=; b=VNobTaSht+DGyk/GTwbQcYUVC/3h5zk4hkLIhzb/lhD65yDyrAKQEKPXImz3Kp4B3u CuLulFQLxYm6FPaKukYJESwODCK+Jwaz8QwDvk8VhpJ6ipun7vuz+miAqegMNNLIrwak uIU1j8gou1XkaJjnV3GUSDlmOJMpeSvUYwDdNzRTeQeZiuivEE7dUQHgOh01WC5i902R KIYGMhoZPsMw06gDKwltZfKRW/bdbVpcIiVHluNINTAebvNqTk+NTZ1gruOa55v8VQIr kZbHyWNHDJGCN6ENnRK66VCDhLDS3DBGC6h4JPcsTTa96lhYI2ha/GwdlpyC2x6npIE8 frMQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=IHVxQvHLGmuAsOe7Our5n+V2vriioUviLGm8LCdzEg4=; b=QXkic65z4RgQSwYfiKZc7ei202LkjBp5LrgJIZeo28HgxNdmmLsf971Y1qATcenHxg kynqGNJ4HUQzl0zhJa/nHVq9erDJH3pcQVn8jm5V736db0BKuIBgF95ncuyMuKlzUNGN N59Ivm6mmz7fI7i20VzmTWN6FL1qeZHV42ul0yhJcviNH/eQd0oBVqBe3++sFMwzx/s/ Xhi4WqEk9Ncj/zauPx4PGyfaHTeVGP+5ww9g9sfC2BafxWgLwHQP2evPIu07w9FySQba Q+co6QdEIJ/xjOmSgbMdGeYxyA44LT65WOV3j+8ExmjDWDlhdmA5udA8vj6tnZU32X3c y8aA== X-Gm-Message-State: AGi0PuY9a3/+J1oIpCv1Sz5r0WL9ULIQ4WcoG4lXQgten3pgxsoe188k kO/lRD7/2fmdu5MwEbi3MP66zdA2X/Vy7w== X-Google-Smtp-Source: APiQypIjxhT8M4F13SFKY/4mGL8MggqFy9blAYcZM+h+fH1GShI48rfl9zQaoKcb446LBwsoVqN0YA== X-Received: by 2002:a37:8902:: with SMTP id l2mr21610049qkd.127.1589312009833; Tue, 12 May 2020 12:33:29 -0700 (PDT) From: Robert Foley To: qemu-devel@nongnu.org Subject: [PATCH v6 1/9] tests/vm: Add configuration to basevm.py Date: Tue, 12 May 2020 15:33:32 -0400 Message-Id: <20200512193340.265-2-robert.foley@linaro.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20200512193340.265-1-robert.foley@linaro.org> References: <20200512193340.265-1-robert.foley@linaro.org> Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=2607:f8b0:4864:20::72d; envelope-from=robert.foley@linaro.org; helo=mail-qk1-x72d.google.com X-detected-operating-system: by eggs.gnu.org: No matching host in p0f cache. That's all we know. X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001 autolearn=_AUTOLEARN X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: philmd@redhat.com, alex.bennee@linaro.org, robert.foley@linaro.org, peter.puhov@linaro.org Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Added use of a configuration to tests/vm/basevm.py. The configuration provides parameters used to configure a VM. This allows for providing alternate configurations to the VM being created/launched. cpu, machine, memory, and NUMA configuration are all examples of configuration which we might want to vary on the VM being creat= ed or launched. This will for example allow for creating an aarch64 vm. Signed-off-by: Robert Foley Reviewed-by: Peter Puhov --- tests/vm/basevm.py | 170 ++++++++++++++++++++++++++++++++++----------- 1 file changed, 131 insertions(+), 39 deletions(-) diff --git a/tests/vm/basevm.py b/tests/vm/basevm.py index 756ccf7aca..fb93a111c1 100644 --- a/tests/vm/basevm.py +++ b/tests/vm/basevm.py @@ -29,16 +29,41 @@ import tempfile import shutil import multiprocessing import traceback - -SSH_KEY =3D open(os.path.join(os.path.dirname(__file__), - "..", "keys", "id_rsa")).read() -SSH_PUB_KEY =3D open(os.path.join(os.path.dirname(__file__), - "..", "keys", "id_rsa.pub")).read() - +import shlex + +SSH_KEY_FILE =3D os.path.join(os.path.dirname(__file__), + "..", "keys", "id_rsa") +SSH_PUB_KEY_FILE =3D os.path.join(os.path.dirname(__file__), + "..", "keys", "id_rsa.pub") + +# This is the standard configuration. +# Any or all of these can be overridden by +# passing in a config argument to the VM constructor. +DEFAULT_CONFIG =3D { + 'cpu' : "max", + 'machine' : 'pc', + 'guest_user' : "qemu", + 'guest_pass' : "qemupass", + 'root_pass' : "qemupass", + 'ssh_key_file' : SSH_KEY_FILE, + 'ssh_pub_key_file': SSH_PUB_KEY_FILE, + 'memory' : "4G", + 'extra_args' : [], + 'qemu_args' : "", + 'dns' : "", + 'ssh_port' : 0, + 'install_cmds' : "", + 'boot_dev_type' : "block", + 'ssh_timeout' : 1, +} +BOOT_DEVICE =3D { + 'block' : "-drive file=3D{},if=3Dnone,id=3Ddrive0,cache=3Dwriteback "\ + "-device virtio-blk,drive=3Ddrive0,bootindex=3D0", + 'scsi' : "-device virtio-scsi-device,id=3Dscsi "\ + "-drive file=3D{},format=3Draw,if=3Dnone,id=3Dhd0 "\ + "-device scsi-hd,drive=3Dhd0,bootindex=3D0", +} class BaseVM(object): - GUEST_USER =3D "qemu" - GUEST_PASS =3D "qemupass" - ROOT_PASS =3D "qemupass" =20 envvars =3D [ "https_proxy", @@ -61,19 +86,30 @@ class BaseVM(object): # 4 is arbitrary, but greater than 2, # since we found we need to wait more than twice as long. tcg_ssh_timeout_multiplier =3D 4 - def __init__(self, debug=3DFalse, vcpus=3DNone): + def __init__(self, debug=3DFalse, vcpus=3DNone, config=3DNone): self._guest =3D None + # Allow input config to override defaults. + self._config =3D DEFAULT_CONFIG.copy() + if config !=3D None: + self._config.update(config) + self.validate_ssh_keys() self._tmpdir =3D os.path.realpath(tempfile.mkdtemp(prefix=3D"vm-te= st-", suffix=3D".tmp", dir=3D".")) atexit.register(shutil.rmtree, self._tmpdir) - - self._ssh_key_file =3D os.path.join(self._tmpdir, "id_rsa") - open(self._ssh_key_file, "w").write(SSH_KEY) - subprocess.check_call(["chmod", "600", self._ssh_key_file]) - - self._ssh_pub_key_file =3D os.path.join(self._tmpdir, "id_rsa.pub") - open(self._ssh_pub_key_file, "w").write(SSH_PUB_KEY) + # Copy the key files to a temporary directory. + # Also chmod the key file to agree with ssh requirements. + self._config['ssh_key'] =3D \ + open(self._config['ssh_key_file']).read().rstrip() + self._config['ssh_pub_key'] =3D \ + open(self._config['ssh_pub_key_file']).read().rstrip() + self._ssh_tmp_key_file =3D os.path.join(self._tmpdir, "id_rsa") + open(self._ssh_tmp_key_file, "w").write(self._config['ssh_key']) + subprocess.check_call(["chmod", "600", self._ssh_tmp_key_file]) + + self._ssh_tmp_pub_key_file =3D os.path.join(self._tmpdir, "id_rsa.= pub") + open(self._ssh_tmp_pub_key_file, + "w").write(self._config['ssh_pub_key']) =20 self.debug =3D debug self._stderr =3D sys.stderr @@ -82,11 +118,14 @@ class BaseVM(object): self._stdout =3D sys.stdout else: self._stdout =3D self._devnull + netdev =3D "user,id=3Dvnet,hostfwd=3D:127.0.0.1:{}-:22" self._args =3D [ \ - "-nodefaults", "-m", "4G", - "-cpu", "max", - "-netdev", "user,id=3Dvnet,hostfwd=3D:127.0.0.1:0-:22" + - (",ipv6=3Dno" if not self.ipv6 else ""), + "-nodefaults", "-m", self._config['memory'], + "-cpu", self._config['cpu'], + "-netdev", + netdev.format(self._config['ssh_port']) + + (",ipv6=3Dno" if not self.ipv6 else "") + + (",dns=3D" + self._config['dns'] if self._config['dns'] else "= "), "-device", "virtio-net-pci,netdev=3Dvnet", "-vnc", "127.0.0.1:0,to=3D20"] if vcpus and vcpus > 1: @@ -97,6 +136,55 @@ class BaseVM(object): logging.info("KVM not available, not using -enable-kvm") self._data_args =3D [] =20 + if self._config['qemu_args'] !=3D None: + qemu_args =3D self._config['qemu_args'] + qemu_args =3D qemu_args.replace('\n',' ').replace('\r','') + # shlex groups quoted arguments together + # we need this to keep the quoted args together for when + # the QEMU command is issued later. + args =3D shlex.split(qemu_args) + self._config['extra_args'] =3D [] + for arg in args: + if arg: + # Preserve quotes around arguments. + # shlex above takes them out, so add them in. + if " " in arg: + arg =3D '"{}"'.format(arg) + self._config['extra_args'].append(arg) + + def validate_ssh_keys(self): + """Check to see if the ssh key files exist.""" + if 'ssh_key_file' not in self._config or\ + not os.path.exists(self._config['ssh_key_file']): + raise Exception("ssh key file not found.") + if 'ssh_pub_key_file' not in self._config or\ + not os.path.exists(self._config['ssh_pub_key_file']): + raise Exception("ssh pub key file not found.") + + def wait_boot(self, wait_string=3DNone): + """Wait for the standard string we expect + on completion of a normal boot. + The user can also choose to override with an + alternate string to wait for.""" + if wait_string is None: + if self.login_prompt is None: + raise Exception("self.login_prompt not defined") + wait_string =3D self.login_prompt + # Intentionally bump up the default timeout under TCG, + # since the console wait below takes longer. + timeout =3D self.socket_timeout + if not kvm_available(self.arch): + timeout *=3D 8 + self.console_init(timeout=3Dtimeout) + self.console_wait(wait_string) + + def __getattr__(self, name): + # Support direct access to config by key. + # for example, access self._config['cpu'] by self.cpu + if name.lower() in self._config.keys(): + return self._config[name.lower()] + return object.__getattribute__(self, name) + def _download_with_cache(self, url, sha256sum=3DNone, sha512sum=3DNone= ): def check_sha256sum(fname): if not sha256sum: @@ -128,8 +216,9 @@ class BaseVM(object): "-t", "-o", "StrictHostKeyChecking=3Dno", "-o", "UserKnownHostsFile=3D" + os.devnull, - "-o", "ConnectTimeout=3D1", - "-p", self.ssh_port, "-i", self._ssh_key_file] + "-o", + "ConnectTimeout=3D{}".format(self._config["ssh_timeout"= ]), + "-p", self.ssh_port, "-i", self._ssh_tmp_key_file] # If not in debug mode, set ssh to quiet mode to # avoid printing the results of commands. if not self.debug: @@ -178,14 +267,14 @@ class BaseVM(object): "virtio-blk,drive=3D%s,serial=3D%s,bootindex= =3D1" % (name, name)] =20 def boot(self, img, extra_args=3D[]): - args =3D self._args + [ - "-drive", "file=3D%s,if=3Dnone,id=3Ddrive0,cache=3Dwriteback" = % img, - "-device", "virtio-blk,drive=3Ddrive0,bootindex=3D0"] - args +=3D self._data_args + extra_args + boot_dev =3D BOOT_DEVICE[self._config['boot_dev_type']] + boot_params =3D boot_dev.format(img) + args =3D self._args + boot_params.split(' ') + args +=3D self._data_args + extra_args + self._config['extra_args'] logging.debug("QEMU args: %s", " ".join(args)) qemu_bin =3D os.environ.get("QEMU", "qemu-system-" + self.arch) guest =3D QEMUMachine(binary=3Dqemu_bin, args=3Dargs) - guest.set_machine('pc') + guest.set_machine(self._config['machine']) guest.set_console() try: guest.launch() @@ -299,7 +388,8 @@ class BaseVM(object): self.console_send(command) =20 def console_ssh_init(self, prompt, user, pw): - sshkey_cmd =3D "echo '%s' > .ssh/authorized_keys\n" % SSH_PUB_KEY.= rstrip() + sshkey_cmd =3D "echo '%s' > .ssh/authorized_keys\n" \ + % self._config['ssh_pub_key'].rstrip() self.console_wait_send("login:", "%s\n" % user) self.console_wait_send("Password:", "%s\n" % pw) self.console_wait_send(prompt, "mkdir .ssh\n") @@ -358,23 +448,23 @@ class BaseVM(object): "local-hostname: {}-guest\n".format(name)]) mdata.close() udata =3D open(os.path.join(cidir, "user-data"), "w") - print("guest user:pw {}:{}".format(self.GUEST_USER, - self.GUEST_PASS)) + print("guest user:pw {}:{}".format(self._config['guest_user'], + self._config['guest_pass'])) udata.writelines(["#cloud-config\n", "chpasswd:\n", " list: |\n", - " root:%s\n" % self.ROOT_PASS, - " %s:%s\n" % (self.GUEST_USER, - self.GUEST_PASS), + " root:%s\n" % self._config['root_pass'], + " %s:%s\n" % (self._config['guest_user'], + self._config['guest_pass']), " expire: False\n", "users:\n", - " - name: %s\n" % self.GUEST_USER, + " - name: %s\n" % self._config['guest_user'], " sudo: ALL=3D(ALL) NOPASSWD:ALL\n", " ssh-authorized-keys:\n", - " - %s\n" % SSH_PUB_KEY, + " - %s\n" % self._config['ssh_pub_key'], " - name: root\n", " ssh-authorized-keys:\n", - " - %s\n" % SSH_PUB_KEY, + " - %s\n" % self._config['ssh_pub_key'], "locale: en_US.UTF-8\n"]) proxy =3D os.environ.get("http_proxy") if not proxy is None: @@ -427,15 +517,17 @@ def parse_args(vmcls): parser.disable_interspersed_args() return parser.parse_args() =20 -def main(vmcls): +def main(vmcls, config=3DNone): try: + if config =3D=3D None: + config =3D {} args, argv =3D parse_args(vmcls) if not argv and not args.build_qemu and not args.build_image: print("Nothing to do?") return 1 logging.basicConfig(level=3D(logging.DEBUG if args.debug else logging.WARN)) - vm =3D vmcls(debug=3Dargs.debug, vcpus=3Dargs.jobs) + vm =3D vmcls(debug=3Dargs.debug, vcpus=3Dargs.jobs, config=3Dconfi= g) if args.build_image: if os.path.exists(args.image) and not args.force: sys.stderr.writelines(["Image file exists: %s\n" % args.im= age, --=20 2.17.1 From nobody Sat Jun 1 02:33:29 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=linaro.org ARC-Seal: i=1; a=rsa-sha256; t=1589312252; cv=none; d=zohomail.com; s=zohoarc; b=KT/jtbN7CyM1kqTApn1C7OUKU1fhALgIM39+EPw4PZsBCGcEsfFK7kG/yYDo9ibLB5RcVRPXksulDT20AvOxey56TZAhlgReDVcIq66jrPoaDogfplbnacRsfNiX0H/NwCpctAAV1IFTLmu4WDAjH09YCg4C9tXLjwoC7i2BP4k= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1589312252; h=Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:Message-ID:References:Sender:Subject:To; bh=GDWBC+xXvVZvAssx4rV2vfXTQ7ubWVSFuWNRHvIlL8s=; b=O/Xi7Pif9SXn/gz4yO3joV9aKBZtgZ/D2wt5iKPkDE/w0dVsf4TmtwC8MV5XyfUSzEVgz0hjvFQrdgGG06cVC/UyvBR98Sl690QbTCzTSbgPp0J0IFtLVPD4bn4sePfciQczGxYuHN87byOmXOvPizsSogm/yI5FBJRMA0JtFew= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1589312252205812.8531385599613; Tue, 12 May 2020 12:37:32 -0700 (PDT) Received: from localhost ([::1]:49174 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jYaiY-0005lc-Nq for importer@patchew.org; Tue, 12 May 2020 15:37:30 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:47252) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1jYaek-0000KY-WC for qemu-devel@nongnu.org; Tue, 12 May 2020 15:33:35 -0400 Received: from mail-qk1-x741.google.com ([2607:f8b0:4864:20::741]:34393) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1jYaei-0001qO-Kq for qemu-devel@nongnu.org; Tue, 12 May 2020 15:33:34 -0400 Received: by mail-qk1-x741.google.com with SMTP id 190so9346651qki.1 for ; Tue, 12 May 2020 12:33:32 -0700 (PDT) Received: from Rfoley-MA01.hsd1.ma.comcast.net ([2601:199:4480:60c0:d31:9f53:b3eb:9b53]) by smtp.gmail.com with ESMTPSA id j45sm2100998qtk.14.2020.05.12.12.33.29 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 12 May 2020 12:33:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=GDWBC+xXvVZvAssx4rV2vfXTQ7ubWVSFuWNRHvIlL8s=; b=SlAOYg4GqtQYC0gJ3SDV7CwYIEqVye/OilqqpmwKhN728g2rSt2Rn66N0w+xiZAvax 7aW3D/O4ZmPzIS5hlkejQQeG4tVlJHAGrZc/noFqLKeSzSIc6gwkRx+DQlNntDdTX6js t0/brXHtcnxlEk5JWY+vB6dXwH+ScZ/M0SFJn+rSJbhQeJ3eqmWZ4+/vgRmlaMyL3WAo 5u0T87zAq5i9oLkW0gzek8JFblmDUsWRbIjkSLx2g4F144Py6R8x/mnkHthQAs1+YX4z vUTiaZNxYnhDWzRoSMvp0OL78TrmKyrkCdjjSmqF+FL7S7CeBMY6/dlQ2Jq2lrZjyKJg NWjg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=GDWBC+xXvVZvAssx4rV2vfXTQ7ubWVSFuWNRHvIlL8s=; b=mol5pE15+L1vt7xkUk1vAbAg/hSJ1jxmkCiJNxXCbbDIyRqU439+qDnQJ9ZWg3CkgN T4pAiKfYpXTHZTGWGG0VKWI3tBlD5pMGwsg2tfiJ81kIaAsBu2rl65y88jD3TDq5oElt bhv9HUGXmoYWAdIu+TUoxbrmifOPpTx1Ug2yEjmVUvV6voPHEMBOxXFMCPXM1+3J8WS2 JQU1Duhccn2xPg/cx4wFph2m7Bfh0Rb+BLL6fcF1JpOE/JKGRYGbjYkFACmVLmu358Ys 37oBhqBoWhVrN2sP5SYkS5+wlP0XpaG8wz6ZMY22K49HUrMC1YOQ0iCUym7R5G+WN2sC RwPg== X-Gm-Message-State: AGi0PuacuWMLk+93dITRt+WLtClV4/VwxQAWH2tD1Wdx3iDFZ3YtSaUQ 4RWLnSgKFUOBmPFLzRbCW+rX6ro4ETyeiQ== X-Google-Smtp-Source: APiQypKZ9wR94NTf5QHhZNN0/7BMkBwMQZ7TPhTUIcb1Kb5gpvMNwzM9qneM+aed11uaW8AJTOXDrg== X-Received: by 2002:a37:9bcb:: with SMTP id d194mr22349304qke.16.1589312010923; Tue, 12 May 2020 12:33:30 -0700 (PDT) From: Robert Foley To: qemu-devel@nongnu.org Subject: [PATCH v6 2/9] tests/vm: Added configuration file support Date: Tue, 12 May 2020 15:33:33 -0400 Message-Id: <20200512193340.265-3-robert.foley@linaro.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20200512193340.265-1-robert.foley@linaro.org> References: <20200512193340.265-1-robert.foley@linaro.org> Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=2607:f8b0:4864:20::741; envelope-from=robert.foley@linaro.org; helo=mail-qk1-x741.google.com X-detected-operating-system: by eggs.gnu.org: No matching host in p0f cache. That's all we know. X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001 autolearn=_AUTOLEARN X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: philmd@redhat.com, alex.bennee@linaro.org, robert.foley@linaro.org, peter.puhov@linaro.org Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Changes to tests/vm/basevm.py to allow accepting a configuration file as a parameter. Allows for specifying VM options such as cpu, machine, memory, and arbitrary qemu arguments for specifying options such as NUMA configuration. Also added an example conf_example_aarch64.yml and conf_example_x86.yml. Signed-off-by: Robert Foley Reviewed-by: Peter Puhov --- configure | 9 ++++++ tests/vm/Makefile.include | 6 ++++ tests/vm/basevm.py | 40 +++++++++++++++++++++++- tests/vm/conf_example_aarch64.yml | 51 +++++++++++++++++++++++++++++++ tests/vm/conf_example_x86.yml | 50 ++++++++++++++++++++++++++++++ 5 files changed, 155 insertions(+), 1 deletion(-) create mode 100644 tests/vm/conf_example_aarch64.yml create mode 100644 tests/vm/conf_example_x86.yml diff --git a/configure b/configure index 0d69c360c0..b3cf32f556 100755 --- a/configure +++ b/configure @@ -950,6 +950,13 @@ do fi done =20 +# Check for existence of python3 yaml, needed to +# import yaml config files into vm-build. +python_yaml=3D"no" +if $(python3 -c "import yaml" 2> /dev/null); then + python_yaml=3D"yes" +fi + : ${smbd=3D${SMBD-/usr/sbin/smbd}} =20 # Default objcc to clang if available, otherwise use CC @@ -6581,6 +6588,7 @@ if test "$docs" !=3D "no"; then echo "sphinx-build $sphinx_build" fi echo "genisoimage $genisoimage" +echo "python_yaml $python_yaml" echo "slirp support $slirp $(echo_version $slirp $slirp_version)" if test "$slirp" !=3D "no" ; then echo "smbd $smbd" @@ -7642,6 +7650,7 @@ echo "PYTHON=3D$python" >> $config_host_mak echo "SPHINX_BUILD=3D$sphinx_build" >> $config_host_mak echo "SPHINX_WERROR=3D$sphinx_werror" >> $config_host_mak echo "GENISOIMAGE=3D$genisoimage" >> $config_host_mak +echo "PYTHON_YAML=3D$python_yaml" >> $config_host_mak echo "CC=3D$cc" >> $config_host_mak if $iasl -h > /dev/null 2>&1; then echo "IASL=3D$iasl" >> $config_host_mak diff --git a/tests/vm/Makefile.include b/tests/vm/Makefile.include index 1bf9693d19..78a2de1f82 100644 --- a/tests/vm/Makefile.include +++ b/tests/vm/Makefile.include @@ -43,6 +43,12 @@ endif @echo " V=3D1 - Enable verbose ouput on host and guest commands" @echo " QEMU=3D/path/to/qemu - Change path to QEMU binary" @echo " QEMU_IMG=3D/path/to/qemu-img - Change path to qemu-img tool" +ifeq ($(PYTHON_YAML),yes) + @echo " QEMU_CONFIG=3D/path/conf.yml - Change path to VM configurati= on .yml file." +else + @echo " (install python3-yaml to enable support for yaml file to confi= gure a VM.)" +endif + @echo " See conf_example_*.yml for file= format details." =20 vm-build-all: $(addprefix vm-build-, $(IMAGES)) =20 diff --git a/tests/vm/basevm.py b/tests/vm/basevm.py index fb93a111c1..97f8ac4218 100644 --- a/tests/vm/basevm.py +++ b/tests/vm/basevm.py @@ -477,9 +477,43 @@ class BaseVM(object): cwd=3Dcidir, stdin=3Dself._devnull, stdout=3Dself._stdou= t, stderr=3Dself._stdout) - return os.path.join(cidir, "cloud-init.iso") =20 +def parse_config(config, args): + """ Parse yaml config and populate our config structure. + The yaml config allows the user to override the + defaults for VM parameters. In many cases these + defaults can be overridden without rebuilding the VM.""" + if args.config: + config_file =3D args.config + elif 'QEMU_CONFIG' in os.environ: + config_file =3D os.environ['QEMU_CONFIG'] + else: + return config + if not os.path.exists(config_file): + raise Exception("config file {} does not exist".format(config_file= )) + # We gracefully handle importing the yaml module + # since it might not be installed. + # If we are here it means the user supplied a .yml file, + # so if the yaml module is not installed we will exit with error. + try: + import yaml + except ImportError: + print("The python3-yaml package is needed "\ + "to support config.yaml files") + # Instead of raising an exception we exit to avoid + # a raft of messy (expected) errors to stdout. + exit(1) + with open(config_file) as f: + yaml_dict =3D yaml.safe_load(f) + + if 'qemu-conf' in yaml_dict: + config.update(yaml_dict['qemu-conf']) + else: + raise Exception("config file {} is not valid"\ + " missing qemu-conf".format(config_file)) + return config + def parse_args(vmcls): =20 def get_default_jobs(): @@ -514,6 +548,9 @@ def parse_args(vmcls): help=3D"Interactively run command") parser.add_option("--snapshot", "-s", action=3D"store_true", help=3D"run tests with a snapshot") + parser.add_option("--config", "-c", default=3DNone, + help=3D"Provide config yaml for configuration. "\ + "See config_example.yaml for example.") parser.disable_interspersed_args() return parser.parse_args() =20 @@ -525,6 +562,7 @@ def main(vmcls, config=3DNone): if not argv and not args.build_qemu and not args.build_image: print("Nothing to do?") return 1 + config =3D parse_config(config, args) logging.basicConfig(level=3D(logging.DEBUG if args.debug else logging.WARN)) vm =3D vmcls(debug=3Dargs.debug, vcpus=3Dargs.jobs, config=3Dconfi= g) diff --git a/tests/vm/conf_example_aarch64.yml b/tests/vm/conf_example_aarc= h64.yml new file mode 100644 index 0000000000..9d44ae356f --- /dev/null +++ b/tests/vm/conf_example_aarch64.yml @@ -0,0 +1,51 @@ +# +# Example yaml for use by any of the scripts in tests/vm. +# Can be provided as an environment variable QEMU_CONFIG +# +qemu-conf: + + # If any of the below are not provided, we will just use the qemu defa= ults. + + # Login username and password(has to be sudo enabled) + guest_user: qemu + guest_pass: "qemupass" + + # Password for root user can be different from guest. + root_pass: "qemupass" + + # If one key is provided, both must be provided. + #ssh_key: /complete/path/of/your/keyfile/id_rsa + #ssh_pub_key: /complete/path/of/your/keyfile/id_rsa.pub + + cpu: max + machine: virt,gic-version=3Dmax + memory: 16G + + # The below is a example for how to configure NUMA topology with + # 4 NUMA nodes and 2 different NUMA distances. + qemu_args: "-smp cpus=3D16,sockets=3D2,cores=3D8 + -numa node,cpus=3D0-3,nodeid=3D0 -numa node,cpus=3D4-7,nod= eid=3D1 + -numa node,cpus=3D8-11,nodeid=3D2 -numa node,cpus=3D12-15,= nodeid=3D3 + -numa dist,src=3D0,dst=3D1,val=3D15 -numa dist,src=3D2,dst= =3D3,val=3D15 + -numa dist,src=3D0,dst=3D2,val=3D20 -numa dist,src=3D0,dst= =3D3,val=3D20 + -numa dist,src=3D1,dst=3D2,val=3D20 -numa dist,src=3D1,dst= =3D3,val=3D20" + + # By default we do not set the DNS. + # You override the defaults by setting the below. + #dns: 1.234.567.89 + + # By default we will use a "block" device, but + # you can also boot from a "scsi" device. + # Just keep in mind your scripts might need to change + # As you will have /dev/sda instead of /dev/vda (for block device) + boot_dev_type: "block" + + # By default the ssh port is not fixed. + # A fixed ssh port makes it easier for automated tests. + #ssh_port: 5555 + + # To install a different set of packages, provide a command to issue + #install_cmds: "apt-get update ; apt-get build-dep -y qemu" + + # Or to skip the install entirely, just provide "" + #install_cmds: "" diff --git a/tests/vm/conf_example_x86.yml b/tests/vm/conf_example_x86.yml new file mode 100644 index 0000000000..78d3f5830f --- /dev/null +++ b/tests/vm/conf_example_x86.yml @@ -0,0 +1,50 @@ +# +# Example yaml for use by any of the x86 based scripts in tests/vm. +# Can be provided as an environment variable QEMU_CONFIG +# +qemu-conf: + + # If any of the below are not provided, we will just use the qemu defa= ults. + + # Login username and password(has to be sudo enabled) + guest_user: "qemu" + guest_pass: "qemupass" + + # Password for root user can be different from guest. + root_pass: "qemupass" + + # Provide default ssh keys of current user. + # You need to edit the below for your user. + #ssh_key_file: /home//.ssh/id_rsa + #ssh_pub_key_file: /home//.ssh/id_rsa.pub + + cpu: max + machine: pc + memory: 8G + + # The below is a example for how to configure NUMA topology with + # 4 NUMA nodes and 2 different NUMA distances. + qemu_args: "-smp cpus=3D8,sockets=3D2,cores=3D4 + -object memory-backend-ram,size=3D4G,policy=3Dbind,host-no= des=3D0,id=3Dram-node0 + -object memory-backend-ram,size=3D4G,policy=3Dbind,host-no= des=3D0,id=3Dram-node1 + -object memory-backend-ram,size=3D4G,policy=3Dbind,host-no= des=3D1,id=3Dram-node2 + -object memory-backend-ram,size=3D4G,policy=3Dbind,host-no= des=3D1,id=3Dram-node3 + -numa node,cpus=3D0-1,nodeid=3D0 -numa node,cpus=3D2-3,nod= eid=3D1 + -numa node,cpus=3D4-5,nodeid=3D2 -numa node,cpus=3D6-7,nod= eid=3D3 + -numa dist,src=3D0,dst=3D1,val=3D15 -numa dist,src=3D2,dst= =3D3,val=3D15 + -numa dist,src=3D0,dst=3D2,val=3D20 -numa dist,src=3D0,dst= =3D3,val=3D20 + -numa dist,src=3D1,dst=3D2,val=3D20 -numa dist,src=3D1,dst= =3D3,val=3D20" + + # By default we do not set the DNS. + # You override the defaults by setting the below. + #dns: "1.234.567.89" + + # By default we will use a "block" device, but + # you can also boot from a "scsi" device. + # Just keep in mind your scripts might need to change + # As you will have /dev/sda instead of /dev/vda (for block device) + boot_dev_type: "block" + + # By default the ssh port is not fixed. + # A fixed ssh port makes it easier for automated tests. + ssh_port: 5555 --=20 2.17.1 From nobody Sat Jun 1 02:33:29 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=linaro.org ARC-Seal: i=1; a=rsa-sha256; t=1589312377; cv=none; d=zohomail.com; s=zohoarc; b=Qp+LXS0zLUhRw6ODhcftUntJ/q5bvygpTqyzBgPFa7vfG2Una5dDOU6rGx8aqvzNh/SnmamHMHtc/IeFr0M+zVr2PrQmqpn24330ZXSju6yOdeZrK5nQLcY1qdzraRFqkfMK4ZX9s4rKak1WwqF9oiTT5VX0vfLKCx04QW3hGgQ= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1589312377; h=Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:Message-ID:References:Sender:Subject:To; bh=4XbTsG1xyE6Gxli83WzwDvPJ23B7Sb+sDM/S8AQT0kE=; b=kCK/dCLONh5vPpKf2UL+3PT+fdtV832BAhKsCmQf29kJqSRXp2M7D6IpBfIywWk96hNKQUaUjYmld0YsFhE6EocTnSGlcCNexwFGZhuTYUEXBCpFv/+2Qs2jhNQATafK8zBYK0OTCpY2C3ykHvdeLRBbK7MJMPkEWuUVhZn3DUs= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1589312377724242.72193604971653; Tue, 12 May 2020 12:39:37 -0700 (PDT) Received: from localhost ([::1]:57564 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jYaka-000177-DV for importer@patchew.org; Tue, 12 May 2020 15:39:36 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:47254) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1jYael-0000Kg-CW for qemu-devel@nongnu.org; Tue, 12 May 2020 15:33:35 -0400 Received: from mail-qk1-x742.google.com ([2607:f8b0:4864:20::742]:37575) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1jYaej-0001qb-9S for qemu-devel@nongnu.org; Tue, 12 May 2020 15:33:34 -0400 Received: by mail-qk1-x742.google.com with SMTP id s186so13074878qkd.4 for ; Tue, 12 May 2020 12:33:32 -0700 (PDT) Received: from Rfoley-MA01.hsd1.ma.comcast.net ([2601:199:4480:60c0:d31:9f53:b3eb:9b53]) by smtp.gmail.com with ESMTPSA id j45sm2100998qtk.14.2020.05.12.12.33.31 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 12 May 2020 12:33:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=4XbTsG1xyE6Gxli83WzwDvPJ23B7Sb+sDM/S8AQT0kE=; b=YulGLjtUHkItKXBp2mvpG0BSnoD7x5Abn5xaxyttQ9x46Pr6Bhxy1ULYU+FRfbbf++ N5RNOlVMetpf3SopdQt0MP+pckmbuzlg92D55iICB3RzIOomnhkiCnvwstSSDeeP6jbx S+cLKZ9h8e2NAp6lyd0l8ly1kh7tehgNlCWaT+A/kZI+b183MneujKOoH81g15gIzKVy qXW0TMyO2IHmMWumofCHmrsFE6W2ck/oS5yVh9R/H8+EDv7i8nEPdGOdLv9Kkwcvn0L/ iFeruBhlpSC4Vcl4qR0R9ofVizmTkyYaWUwI5h/LP28ykPnw2WdgL7HDEvQZAKYqK8NL G0KA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=4XbTsG1xyE6Gxli83WzwDvPJ23B7Sb+sDM/S8AQT0kE=; b=K3bT1oYm4sV4hXBVjVpjWC9BeNaVEEKrURp6X08A0bUraqUlmJl/FKVdGlGv+94sNZ nsj3kD1VBY5/tMhDiaTslsHGU9uoVJL5crfS4ZSijYyjew7b9nWjd2Xe/POeQfNVKeZ+ UU36O8+Q9asGKuWMVOLoytiZ9EckpvDfEJvDr0G4nOKGRtP8DVdD4f7nq4v/qAPyilRY kTiWMuMGl2tot7Z60g1F9RjKMQmplb7gx7aYXcTGzsoL72+8XkGKTfWUS3xS5Uugkwwg RX6hIxzXvmyOx5/6SUQ3AllgGfwljm9d3rQduAjoU9rbFO6x72tHxU2CkTZshG9SDxgq R44w== X-Gm-Message-State: AGi0PuYveh8SL93cmsM5qtOViuCsP46Lavip0SOYQAls8HYKN7z+IjLO F1mPhSlkZdwtdvfXruMuz1KrZup+zzuk1A== X-Google-Smtp-Source: APiQypLldCQOX0gsM8DlUHod5SeNX0iZ9lCPg6rysW8wYNqIbI4NxGPNMd82tUvnRyYZBknRIMvzEw== X-Received: by 2002:a37:7dc4:: with SMTP id y187mr14833780qkc.412.1589312011969; Tue, 12 May 2020 12:33:31 -0700 (PDT) From: Robert Foley To: qemu-devel@nongnu.org Subject: [PATCH v6 3/9] tests/vm: Pass --debug through for vm-boot-ssh. Date: Tue, 12 May 2020 15:33:34 -0400 Message-Id: <20200512193340.265-4-robert.foley@linaro.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20200512193340.265-1-robert.foley@linaro.org> References: <20200512193340.265-1-robert.foley@linaro.org> Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=2607:f8b0:4864:20::742; envelope-from=robert.foley@linaro.org; helo=mail-qk1-x742.google.com X-detected-operating-system: by eggs.gnu.org: No matching host in p0f cache. That's all we know. X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001 autolearn=_AUTOLEARN X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: philmd@redhat.com, alex.bennee@linaro.org, robert.foley@linaro.org, peter.puhov@linaro.org Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" This helps debug issues that occur during the boot sequence. Signed-off-by: Robert Foley Reviewed-by: Peter Puhov --- tests/vm/Makefile.include | 1 + 1 file changed, 1 insertion(+) diff --git a/tests/vm/Makefile.include b/tests/vm/Makefile.include index 78a2de1f82..4e041103b7 100644 --- a/tests/vm/Makefile.include +++ b/tests/vm/Makefile.include @@ -96,6 +96,7 @@ vm-boot-ssh-%: $(IMAGES_DIR)/%.img $(call quiet-command, \ $(PYTHON) $(SRC_PATH)/tests/vm/$* \ $(if $(J),--jobs $(J)) \ + $(if $(V)$(DEBUG), --debug) \ --image "$<" \ --interactive \ false, \ --=20 2.17.1 From nobody Sat Jun 1 02:33:29 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=linaro.org ARC-Seal: i=1; a=rsa-sha256; t=1589312260; cv=none; d=zohomail.com; s=zohoarc; b=LenSPOF6uxDY9VuLUJ+8x9eSTzQQOpdo8TDxW7LHcEoFTuB3Qj4dxoawW9zuto9PIlM6kiHlYDuKqR7rUO2Fj53HDcojydkORbZ+Ya9qsRg0GijavBpbF/NRnKK5l4SIgJvQJODEQDAER70yOeglM4xiwWDOd4+wvTXS8TBRwss= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1589312260; h=Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:Message-ID:References:Sender:Subject:To; bh=gSaM358UHMm/zfpDXaif/Gr0958wC6l/oGYNTNtJTnA=; b=EkCC6Hny5pOnXRGvQkwxoLRxvzMgrCeRlB3AsgaZiylc54iIEya4/d8VXx3ti+zGfIIP/8c+wQAUcczoB/J1hACVTKc+Njlqrp7Id8WgcXdLlva4oqvk1RpcnUvb9G8NVjBBYaBe/G1ZM0Kti0aSQEX7pY/BFGDFLI6l9xPty9Y= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1589312260490392.3263322142452; Tue, 12 May 2020 12:37:40 -0700 (PDT) Received: from localhost ([::1]:49946 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jYaig-00064X-Vh for importer@patchew.org; Tue, 12 May 2020 15:37:39 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:47260) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1jYaem-0000La-6M for qemu-devel@nongnu.org; Tue, 12 May 2020 15:33:36 -0400 Received: from mail-qk1-x744.google.com ([2607:f8b0:4864:20::744]:35151) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1jYaek-0001r5-Fu for qemu-devel@nongnu.org; Tue, 12 May 2020 15:33:35 -0400 Received: by mail-qk1-x744.google.com with SMTP id f13so14322791qkh.2 for ; Tue, 12 May 2020 12:33:34 -0700 (PDT) Received: from Rfoley-MA01.hsd1.ma.comcast.net ([2601:199:4480:60c0:d31:9f53:b3eb:9b53]) by smtp.gmail.com with ESMTPSA id j45sm2100998qtk.14.2020.05.12.12.33.32 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 12 May 2020 12:33:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=gSaM358UHMm/zfpDXaif/Gr0958wC6l/oGYNTNtJTnA=; b=lNCGE9uJt3+H8wgGwYXd3UtUtu/SEZkokAzjo54QXLBfK8I/TgyiMBsjs5drEXRjft ZC88xh+dg7KGr+UF8PSm18PGBaRUg9/ufUQ581XSnSLdhc9+3NsFFDL/DWmzmTeX2/yS YYWsl1r5oq9k3x77SXUf2c8anKpWeWjthszE5nb/5+ju4EXAtZvjipYzBTrNUHM6j6X7 8VXXiuF6I1pxoK5wROdBYe/4zVfeUbdEZZdJ1/tu+ErBVNz/5+SkR7hSDtPRXWq6NxDe 1kStjZ1t7DW7yx2bYP15wGPqR8oGw6zyxyY0WtGR1A1tINfo217/j7gt72UYUMYfNfiZ xP0g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=gSaM358UHMm/zfpDXaif/Gr0958wC6l/oGYNTNtJTnA=; b=fqZfBNgkY2XW1BL6CD+jTJZ5VKc7BdGRU6gF2T3oGV+FcLfwp6JksGhawtZUZVvSWZ HGkUjrO8aClzmoRVy2oicpeeuwYnsNPIV1zvDNvMnzsrDyV23BRo/zCQ8RLPp5MKMPyf 7FAPst/LdZZyEB5aC7vccfJLSJuDcMKVy9R+7KH2CINm4WCBvJK86KNpaekzaRYhhPP1 RODix0XCZSkEVN604hGGqaIk7BmabMMhsLEUJOGgSqSO4dnMahrCSseGz+PAUBfNX2Aw dt2DRf/qcT+Guup4VZC6V/tcgmTTfK9QBe/F3iPLeyEfLISqqx7V8DQ7APWm7Rn6HlO0 BgzQ== X-Gm-Message-State: AGi0PuaoB8DOPHw/dZiB5ZYsxj7UFb+OaPHxnEynjypXP1Kr649ElhPT Jcmen4Qj36y7mZVvYVzBe9Ftkozt1k7Ewg== X-Google-Smtp-Source: APiQypIFyWdZ7Eslx9K7cRbApKEFiehlwChXuKMlRWJ6UCTBmgEXgBChym/VvW+9ip0VUZ3PIna8cQ== X-Received: by 2002:a37:a603:: with SMTP id p3mr21814549qke.133.1589312013104; Tue, 12 May 2020 12:33:33 -0700 (PDT) From: Robert Foley To: qemu-devel@nongnu.org Subject: [PATCH v6 4/9] tests/vm: Add ability to select QEMU from current build. Date: Tue, 12 May 2020 15:33:35 -0400 Message-Id: <20200512193340.265-5-robert.foley@linaro.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20200512193340.265-1-robert.foley@linaro.org> References: <20200512193340.265-1-robert.foley@linaro.org> Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=2607:f8b0:4864:20::744; envelope-from=robert.foley@linaro.org; helo=mail-qk1-x744.google.com X-detected-operating-system: by eggs.gnu.org: No matching host in p0f cache. That's all we know. X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001 autolearn=_AUTOLEARN X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: philmd@redhat.com, alex.bennee@linaro.org, robert.foley@linaro.org, peter.puhov@linaro.org Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Added a new special variable QEMU_LOCAL=3D1, which will indicate to take the QEMU binary from the current build. Signed-off-by: Robert Foley Reviewed-by: Peter Puhov --- tests/vm/Makefile.include | 4 ++++ tests/vm/basevm.py | 31 ++++++++++++++++++++++++++----- 2 files changed, 30 insertions(+), 5 deletions(-) diff --git a/tests/vm/Makefile.include b/tests/vm/Makefile.include index 4e041103b7..99c06832ac 100644 --- a/tests/vm/Makefile.include +++ b/tests/vm/Makefile.include @@ -41,6 +41,7 @@ endif @echo " J=3D[0..9]* - Override the -jN parameter for make= commands" @echo " DEBUG=3D1 - Enable verbose output on host and i= nteractive debugging" @echo " V=3D1 - Enable verbose ouput on host and guest commands" + @echo " QEMU_LOCAL=3D1 - Use QEMU binary local to this= build." @echo " QEMU=3D/path/to/qemu - Change path to QEMU binary" @echo " QEMU_IMG=3D/path/to/qemu-img - Change path to qemu-img tool" ifeq ($(PYTHON_YAML),yes) @@ -62,6 +63,7 @@ $(IMAGES_DIR)/%.img: $(SRC_PATH)/tests/vm/% \ $(call quiet-command, \ $(PYTHON) $< \ $(if $(V)$(DEBUG), --debug) \ + --build-path $(BUILD_DIR)\ --image "$@" \ --force \ --build-image $@, \ @@ -76,6 +78,7 @@ vm-build-%: $(IMAGES_DIR)/%.img $(if $(DEBUG), --interactive) \ $(if $(J),--jobs $(J)) \ $(if $(V),--verbose) \ + --build-path $(BUILD_DIR)\ --image "$<" \ $(if $(BUILD_TARGET),--build-target $(BUILD_TARGET)) \ --snapshot \ @@ -97,6 +100,7 @@ vm-boot-ssh-%: $(IMAGES_DIR)/%.img $(PYTHON) $(SRC_PATH)/tests/vm/$* \ $(if $(J),--jobs $(J)) \ $(if $(V)$(DEBUG), --debug) \ + --build-path $(BUILD_DIR)\ --image "$<" \ --interactive \ false, \ diff --git a/tests/vm/basevm.py b/tests/vm/basevm.py index 97f8ac4218..1cb104c402 100644 --- a/tests/vm/basevm.py +++ b/tests/vm/basevm.py @@ -86,8 +86,10 @@ class BaseVM(object): # 4 is arbitrary, but greater than 2, # since we found we need to wait more than twice as long. tcg_ssh_timeout_multiplier =3D 4 - def __init__(self, debug=3DFalse, vcpus=3DNone, config=3DNone): + def __init__(self, debug=3DFalse, vcpus=3DNone, config=3DNone, + build_path=3DNone): self._guest =3D None + self._build_path =3D build_path # Allow input config to override defaults. self._config =3D DEFAULT_CONFIG.copy() if config !=3D None: @@ -272,15 +274,15 @@ class BaseVM(object): args =3D self._args + boot_params.split(' ') args +=3D self._data_args + extra_args + self._config['extra_args'] logging.debug("QEMU args: %s", " ".join(args)) - qemu_bin =3D os.environ.get("QEMU", "qemu-system-" + self.arch) - guest =3D QEMUMachine(binary=3Dqemu_bin, args=3Dargs) + qemu_path =3D get_qemu_path(self.arch, self._build_path) + guest =3D QEMUMachine(binary=3Dqemu_path, args=3Dargs) guest.set_machine(self._config['machine']) guest.set_console() try: guest.launch() except: logging.error("Failed to launch QEMU, command line:") - logging.error(" ".join([qemu_bin] + args)) + logging.error(" ".join([qemu_path] + args)) logging.error("Log:") logging.error(guest.get_log()) logging.error("QEMU version >=3D 2.10 is required") @@ -479,6 +481,22 @@ class BaseVM(object): stderr=3Dself._stdout) return os.path.join(cidir, "cloud-init.iso") =20 +def get_qemu_path(arch, build_path=3DNone): + """Fetch the path to the qemu binary.""" + qemu_local =3D os.environ.get("QEMU_LOCAL", 0) + # If QEMU environment variable set, it takes precedence + if "QEMU" in os.environ: + qemu_path =3D os.environ["QEMU"] + elif qemu_local: + if not build_path: + raise Exception("--build-path option required with QEMU_LOCAL") + qemu_path =3D os.path.join(build_path, arch + "-softmmu") + qemu_path =3D os.path.join(qemu_path, "qemu-system-" + arch) + else: + # Default is to use system path for qemu. + qemu_path =3D "qemu-system-" + arch + return qemu_path + def parse_config(config, args): """ Parse yaml config and populate our config structure. The yaml config allows the user to override the @@ -551,6 +569,8 @@ def parse_args(vmcls): parser.add_option("--config", "-c", default=3DNone, help=3D"Provide config yaml for configuration. "\ "See config_example.yaml for example.") + parser.add_option("--build-path", default=3DNone, + help=3D"Path of build directory. ") parser.disable_interspersed_args() return parser.parse_args() =20 @@ -565,7 +585,8 @@ def main(vmcls, config=3DNone): config =3D parse_config(config, args) logging.basicConfig(level=3D(logging.DEBUG if args.debug else logging.WARN)) - vm =3D vmcls(debug=3Dargs.debug, vcpus=3Dargs.jobs, config=3Dconfi= g) + vm =3D vmcls(debug=3Dargs.debug, vcpus=3Dargs.jobs, config=3Dconfi= g, + build_path=3Dargs.build_path) if args.build_image: if os.path.exists(args.image) and not args.force: sys.stderr.writelines(["Image file exists: %s\n" % args.im= age, --=20 2.17.1 From nobody Sat Jun 1 02:33:29 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=linaro.org ARC-Seal: i=1; a=rsa-sha256; t=1589312162; cv=none; d=zohomail.com; s=zohoarc; b=V6xTCswEKAkXKatvXoc+gGEfdX2SRnP4N0pbtZlgtUL4e+RRkuS8ZFhtzzDdsOKh17of7i2VtnwCM4OeDYXT1BWKz8h54wW60KO8LCv2dfZx6VkvjrLLjSYHqY/a2k2fQiqV3CL2F4oErTdHKJDWSYqJNXfdrMT0oJ8NlFHN8F0= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1589312162; h=Content-Type:Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=BTKcYeUaFGhya1RL8BEXIQFfl5d3zBX2OKJUpLkqnTc=; b=d2Ck1E3RHTihj2euzm/PT5So562dT2ZFsrs5YVRm+ZiddykJ4KBVeqzBUAB1jbJjdGf/YYGrKSQHqF++acPGktMw4r1V31C0nJ5KqbvuxN63QyUk0sRSbs00HgjhW89EZRGPsM6S5oRkztqHzLqhVodAqQ0QFrKLwwmrINQc1xo= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1589312162342609.0655814747632; Tue, 12 May 2020 12:36:02 -0700 (PDT) Received: from localhost ([::1]:43940 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jYah6-0003Ly-Ul for importer@patchew.org; Tue, 12 May 2020 15:36:00 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:47264) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1jYaen-0000NX-Cn for qemu-devel@nongnu.org; Tue, 12 May 2020 15:33:37 -0400 Received: from mail-qk1-x734.google.com ([2607:f8b0:4864:20::734]:41619) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1jYael-0001rZ-DJ for qemu-devel@nongnu.org; Tue, 12 May 2020 15:33:37 -0400 Received: by mail-qk1-x734.google.com with SMTP id n14so14913960qke.8 for ; Tue, 12 May 2020 12:33:35 -0700 (PDT) Received: from Rfoley-MA01.hsd1.ma.comcast.net ([2601:199:4480:60c0:d31:9f53:b3eb:9b53]) by smtp.gmail.com with ESMTPSA id j45sm2100998qtk.14.2020.05.12.12.33.33 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 12 May 2020 12:33:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=BTKcYeUaFGhya1RL8BEXIQFfl5d3zBX2OKJUpLkqnTc=; b=NjpZiQRdMt5gv9uZV8Bp/WU1fBG4c49Q8pvOU3hZnfF3+Y3n1hwa7QH5FE/yoAmwtG b3rEG3iNb7Ulf7w0bzVzYq11OYLcOwa+s9qca2o16VD2mGjS7RMVwFrT6i01Od7AFMm1 K8lJtEAMUR5cdnXxHdm2qYpROog9gTVfyMZUUbeqPs686y4hGnqkpVtKIrOnv07ZkzsE c1Rn42BYaTRw2NlXxNULFnF/h+fciYAXXDxRU7e+TcGdD5Ze+/SoXrfjSPcyZk1gmRlo NJuZpY7VTLRfMCOsVPjDPonZNoORb/RPEWgMuzDuqCIAOH/dtgFIKID+oESQLPZWFAOV cmAg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=BTKcYeUaFGhya1RL8BEXIQFfl5d3zBX2OKJUpLkqnTc=; b=HWDNrDRgI+mj1TkMK+OPCvJW3Td/6feg8pKSx1TiuMbCaohyLHoKVB6fStyHlklOD2 D0vFq+5tm3aVE4hhdhOo9sOkfQJwn9amzmURP61/ewlQREq+Eny6PMulD9RhiPt/r/VF 7VymdTpM8AFO75mJIB23LTUcPTwFsycTIF13FBKGsCFTm5DpTlO2CAOdQmjRMvBb0pdM o0pbzwfjEhq+l8uAaOGzjiHFWNhEGT1KU9MLzusApYZ5p6cTYoxIi/OI5qtf2VDbowz6 DLKL3hZ+wsq74FZs4raCWs4utiY2lgGVadHza1yxsM0rcvyfrJuXTnTOFGi1cRsV/xcP xlWQ== X-Gm-Message-State: AGi0PubFgdxwlPSuMLyfllPRXNOcfRlUmfOB5F1FuICITX94T1236V0I j+Kcu7BFquIQg2K6KQICZ/mk1LbYCliN6w== X-Google-Smtp-Source: APiQypI+AibBl1mfgamP98/Kx10dJWp75OESPEwEN5+8uHHdsQvQa9wi+US8b+Tsp0Hx/FqDF19u3w== X-Received: by 2002:a37:b185:: with SMTP id a127mr21996182qkf.87.1589312014195; Tue, 12 May 2020 12:33:34 -0700 (PDT) From: Robert Foley To: qemu-devel@nongnu.org Subject: [PATCH v6 5/9] tests/vm: allow wait_ssh() to specify command Date: Tue, 12 May 2020 15:33:36 -0400 Message-Id: <20200512193340.265-6-robert.foley@linaro.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20200512193340.265-1-robert.foley@linaro.org> References: <20200512193340.265-1-robert.foley@linaro.org> MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=2607:f8b0:4864:20::734; envelope-from=robert.foley@linaro.org; helo=mail-qk1-x734.google.com X-detected-operating-system: by eggs.gnu.org: No matching host in p0f cache. That's all we know. X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001 autolearn=_AUTOLEARN X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: philmd@redhat.com, alex.bennee@linaro.org, robert.foley@linaro.org, peter.puhov@linaro.org Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) This allows for waiting for completion of arbitrary commands. Signed-off-by: Robert Foley Reviewed-by: Peter Puhov Reviewed-by: Alex Benn=C3=A9e --- tests/vm/basevm.py | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/tests/vm/basevm.py b/tests/vm/basevm.py index 1cb104c402..4fa1ea8fde 100644 --- a/tests/vm/basevm.py +++ b/tests/vm/basevm.py @@ -409,24 +409,24 @@ class BaseVM(object): def print_step(self, text): sys.stderr.write("### %s ...\n" % text) =20 - def wait_ssh(self, wait_root=3DFalse, seconds=3D300): + def wait_ssh(self, wait_root=3DFalse, seconds=3D300, cmd=3D"exit 0"): # Allow more time for VM to boot under TCG. if not kvm_available(self.arch): seconds *=3D self.tcg_ssh_timeout_multiplier starttime =3D datetime.datetime.now() endtime =3D starttime + datetime.timedelta(seconds=3Dseconds) - guest_up =3D False + cmd_success =3D False while datetime.datetime.now() < endtime: - if wait_root and self.ssh_root("exit 0") =3D=3D 0: - guest_up =3D True + if wait_root and self.ssh_root(cmd) =3D=3D 0: + cmd_success =3D True break - elif self.ssh("exit 0") =3D=3D 0: - guest_up =3D True + elif self.ssh(cmd) =3D=3D 0: + cmd_success =3D True break seconds =3D (endtime - datetime.datetime.now()).total_seconds() logging.debug("%ds before timeout", seconds) time.sleep(1) - if not guest_up: + if not cmd_success: raise Exception("Timeout while waiting for guest ssh") =20 def shutdown(self): --=20 2.17.1 From nobody Sat Jun 1 02:33:29 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=linaro.org ARC-Seal: i=1; a=rsa-sha256; t=1589312302; cv=none; d=zohomail.com; s=zohoarc; b=SsdcC9LUETkxLZ4grl5BasSCP9BGdOeM5i4jX6bSNFPn3blWIgG5npey1tQKtc8RP9rNJGgUPA4lwIN3zjhcpc90upi8Ef01D54iZVUrWiDcTHHcvkxHXgBUD6pCOnfHETdBlpbgJrYROj3/J5xlT9/N/GFTkRPQH3TyqPStwP0= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1589312302; h=Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:Message-ID:References:Sender:Subject:To; bh=0hIqa3lVQCKwVL1VaqBT09Jl1bMFB1GWgiSzpA9FZzQ=; b=adZojOl8E5lQU95rDbAM2FjmPnVXVjopg+XlXy4dxGcNlQq6sVcwvdhwvZqLTfi0YzGlfPeW9ZHBqz1DRcgCbCPVqjn//2e0Egvqm03XMcwY6pAntQ4fh/XZV7Kni7NYcEipqc21Ew+DMrEbbZM8yrtaOnsRBivWLf103TVGSP4= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1589312302878943.1217927146524; Tue, 12 May 2020 12:38:22 -0700 (PDT) Received: from localhost ([::1]:52532 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jYajN-00076k-EP for importer@patchew.org; Tue, 12 May 2020 15:38:21 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:47280) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1jYaeq-0000PP-2A for qemu-devel@nongnu.org; Tue, 12 May 2020 15:33:41 -0400 Received: from mail-qk1-x72f.google.com ([2607:f8b0:4864:20::72f]:33574) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1jYaen-0001sU-Do for qemu-devel@nongnu.org; Tue, 12 May 2020 15:33:39 -0400 Received: by mail-qk1-x72f.google.com with SMTP id z80so9389384qka.0 for ; Tue, 12 May 2020 12:33:36 -0700 (PDT) Received: from Rfoley-MA01.hsd1.ma.comcast.net ([2601:199:4480:60c0:d31:9f53:b3eb:9b53]) by smtp.gmail.com with ESMTPSA id j45sm2100998qtk.14.2020.05.12.12.33.34 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 12 May 2020 12:33:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=0hIqa3lVQCKwVL1VaqBT09Jl1bMFB1GWgiSzpA9FZzQ=; b=illLbH/YmNEHLGLviWrWWRhZt49G2Y4dMy/2iCQ6sIvk/uYXMg5dkI2frgKUcj3a/Z VlPMWr0uw7Js/4r2InQJxR7qHuPdPgJlWtWxvlWxGY7SMIVpZFWzz2odjbrNIHWtw4d0 yi4OpkRKIDHZxoGFboXYsT/QXQswh+aGvx1X5bbf5yqOhOf9YF9cDnO1+ZaON51ATm7J QNkXm805G7dyrU2bst0iX2VgAwdFiWQetB5glJUo6Q/nzhV2QIkrmqkQjRoqoeaS9FSL AYmQGRDip8R+0QGbNhsiqHxMuSvMQZnlHrqWw4eheUsIAUjo7bZZ5r1uD37H3xRQTiH4 mqag== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=0hIqa3lVQCKwVL1VaqBT09Jl1bMFB1GWgiSzpA9FZzQ=; b=PJEkhXBebRWAwuh7ryXltvZ9dLcikVTUwlSQaWLGXHzq2GaDM5LtLJSfBfECph0s43 9DgleLos9KhXeXgaSenmreirq1JkOtoCKQNNcIKeUyN2oxzhzOCjmtLjORFglRHaR89+ j5p4RFzdI2ZAyx98NhExsgw31cSX4yPq+gE1O2AGqy4Wb4Z8VyBaWFKkS0xAHnxNn9I/ tmzVS7vSFkV8eYTdi7ge1Xb2K3nXbUCfVk8rCnom+0DTXTDGEfX396FDRmBuUQhLmx+j foxZ+z3Vd15qGTqKsSCIGMNQp9ZTMDwWyr8CSa01QRO0kBt2Ztq+8z9aNxrWwVQenHen EBnA== X-Gm-Message-State: AOAM530T30rS8sN7rpQdwzmt+iyEYO0t/WCc+R9dU1MXJtiLlYWDNPgo QWQ6/A+GILumQNj1TAx66OuUn4xpaTjkOA== X-Google-Smtp-Source: ABdhPJz8FmeMAwjgQyi7UM6F6I21qLTGR6DBmS0dj5VfryWURr1yM/6yFOpubcvYgXc9piBI/pdXAA== X-Received: by 2002:a05:620a:1266:: with SMTP id b6mr5859236qkl.427.1589312015417; Tue, 12 May 2020 12:33:35 -0700 (PDT) From: Robert Foley To: qemu-devel@nongnu.org Subject: [PATCH v6 6/9] tests/vm: Added a new script for ubuntu.aarch64. Date: Tue, 12 May 2020 15:33:37 -0400 Message-Id: <20200512193340.265-7-robert.foley@linaro.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20200512193340.265-1-robert.foley@linaro.org> References: <20200512193340.265-1-robert.foley@linaro.org> Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=2607:f8b0:4864:20::72f; envelope-from=robert.foley@linaro.org; helo=mail-qk1-x72f.google.com X-detected-operating-system: by eggs.gnu.org: No matching host in p0f cache. That's all we know. X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001 autolearn=_AUTOLEARN X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: philmd@redhat.com, alex.bennee@linaro.org, robert.foley@linaro.org, peter.puhov@linaro.org Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" ubuntu.aarch64 provides a script to create an Ubuntu 18.04 VM. Another new file is also added aarch64vm.py, which is a module with common methods used by aarch64 VMs, such as how to create the flash images. Signed-off-by: Robert Foley Reviewed-by: Peter Puhov --- configure | 16 ++++++ tests/vm/Makefile.include | 8 +++ tests/vm/aarch64vm.py | 105 ++++++++++++++++++++++++++++++++++ tests/vm/basevm.py | 8 +++ tests/vm/ubuntu.aarch64 | 117 ++++++++++++++++++++++++++++++++++++++ 5 files changed, 254 insertions(+) create mode 100644 tests/vm/aarch64vm.py create mode 100755 tests/vm/ubuntu.aarch64 diff --git a/configure b/configure index b3cf32f556..26f8561557 100755 --- a/configure +++ b/configure @@ -950,6 +950,20 @@ do fi done =20 +# Check for files needed by aarch64 VMs. +# Allow user to override the path for efi also. +qemu_efi_aarch64=3D +for fd in $QEMU_EFI_PATH /usr/share/qemu-efi-aarch64/QEMU_EFI.fd +do + if has $fd + then + if [ -f $fd ]; then + qemu_efi_aarch64=3D$fd + fi + break + fi +done + # Check for existence of python3 yaml, needed to # import yaml config files into vm-build. python_yaml=3D"no" @@ -6588,6 +6602,7 @@ if test "$docs" !=3D "no"; then echo "sphinx-build $sphinx_build" fi echo "genisoimage $genisoimage" +echo "qemu_efi_aarch64 $qemu_efi_aarch64" echo "python_yaml $python_yaml" echo "slirp support $slirp $(echo_version $slirp $slirp_version)" if test "$slirp" !=3D "no" ; then @@ -7650,6 +7665,7 @@ echo "PYTHON=3D$python" >> $config_host_mak echo "SPHINX_BUILD=3D$sphinx_build" >> $config_host_mak echo "SPHINX_WERROR=3D$sphinx_werror" >> $config_host_mak echo "GENISOIMAGE=3D$genisoimage" >> $config_host_mak +echo "QEMU_EFI_AARCH64=3D$qemu_efi_aarch64" >> $config_host_mak echo "PYTHON_YAML=3D$python_yaml" >> $config_host_mak echo "CC=3D$cc" >> $config_host_mak if $iasl -h > /dev/null 2>&1; then diff --git a/tests/vm/Makefile.include b/tests/vm/Makefile.include index 99c06832ac..d918d13f50 100644 --- a/tests/vm/Makefile.include +++ b/tests/vm/Makefile.include @@ -5,6 +5,9 @@ IMAGES :=3D freebsd netbsd openbsd centos fedora ifneq ($(GENISOIMAGE),) IMAGES +=3D ubuntu.i386 centos +ifneq ($(QEMU_EFI_AARCH64),) +IMAGES +=3D ubuntu.aarch64 +endif endif =20 IMAGES_DIR :=3D $(HOME)/.cache/qemu-vm/images @@ -23,6 +26,11 @@ vm-help vm-test: ifneq ($(GENISOIMAGE),) @echo " vm-build-centos - Build QEMU in CentOS VM, with = Docker" @echo " vm-build-ubuntu.i386 - Build QEMU in ubuntu i386 VM" +ifneq ($(QEMU_EFI_AARCH64),) + @echo " vm-build-ubuntu.aarch64 - Build QEMU in ubuntu aarch64 V= M" +else + @echo " (install qemu-efi-aarch64 to build centos/ubuntu aarch64 images.= )" +endif else @echo " (install genisoimage to build centos/ubuntu images)" endif diff --git a/tests/vm/aarch64vm.py b/tests/vm/aarch64vm.py new file mode 100644 index 0000000000..3a9807cd43 --- /dev/null +++ b/tests/vm/aarch64vm.py @@ -0,0 +1,105 @@ +#!/usr/bin/env python3 +# +# VM testing aarch64 library +# +# Copyright 2020 Linaro +# +# Authors: +# Robert Foley +# +# This code is licensed under the GPL version 2 or later. See +# the COPYING file in the top-level directory. +# +import os +import sys +import subprocess +import basevm +from qemu.accel import kvm_available + +# This is the config needed for current version of QEMU. +# This works for both kvm and tcg. +CURRENT_CONFIG =3D { + 'cpu' : "max", + 'machine' : "virt,gic-version=3Dmax", +} + +# The minimum minor version of QEMU we will support with aarch64 VMs is 3. +# QEMU versions less than 3 have various issues running these VMs. +QEMU_AARCH64_MIN_VERSION =3D 3 + +# The DEFAULT_CONFIG will default to a version of +# parameters that works for backwards compatibility. +DEFAULT_CONFIG =3D {'kvm' : {'cpu' : "host", + 'machine' : "virt,gic-version=3Dhost"}, + 'tcg' : {'cpu' : "cortex-a57", + 'machine' : "virt"}, +} + +def get_config_defaults(vmcls, default_config): + """Fetch the configuration defaults for this VM, + taking into consideration the defaults for + aarch64 first, followed by the defaults for this VM.""" + config =3D default_config + config.update(aarch_get_config_defaults(vmcls)) + return config + +def aarch_get_config_defaults(vmcls): + """Set the defaults for current version of QEMU.""" + config =3D CURRENT_CONFIG + args, argv =3D basevm.parse_args(vmcls) + qemu_path =3D basevm.get_qemu_path(vmcls.arch, args.build_path) + qemu_version =3D basevm.get_qemu_version(qemu_path) + if qemu_version < QEMU_AARCH64_MIN_VERSION: + error =3D "\nThis major version of QEMU {} is to old for aarch64 V= Ms.\n"\ + "The major version must be at least {}.\n"\ + "To continue with the current build of QEMU, "\ + "please restart with QEMU_LOCAL=3D1 .\n" + print(error.format(qemu_version, QEMU_AARCH64_MIN_VERSION)) + exit(1) + if qemu_version =3D=3D QEMU_AARCH64_MIN_VERSION: + # We have an older version of QEMU, + # set the config values for backwards compatibility. + if kvm_available('aarch64'): + config.update(DEFAULT_CONFIG['kvm']) + else: + config.update(DEFAULT_CONFIG['tcg']) + return config + +def create_flash_images(flash_dir=3D"./"): + """Creates the appropriate pflash files + for an aarch64 VM.""" + flash0_path =3D get_flash_path(flash_dir, "flash0") + flash1_path =3D get_flash_path(flash_dir, "flash1") + fd_null =3D open(os.devnull, 'w') + subprocess.check_call(["dd", "if=3D/dev/zero", "of=3D{}".format(flash0= _path), + "bs=3D1M", "count=3D64"], + stdout=3Dfd_null, stderr=3Dsubprocess.STDOUT) + # A reliable way to get the QEMU EFI image is via an installed package. + efi_img =3D "/usr/share/qemu-efi-aarch64/QEMU_EFI.fd" + if not os.path.exists(efi_img): + sys.stderr.write("*** {} is missing\n".format(efi_img)) + sys.stderr.write("*** please install qemu-efi-aarch64 package\n") + exit(3) + subprocess.check_call(["dd", "if=3D{}".format(efi_img), + "of=3D{}".format(flash0_path), + "conv=3Dnotrunc"], + stdout=3Dfd_null, stderr=3Dsubprocess.STDOUT) + subprocess.check_call(["dd", "if=3D/dev/zero", + "of=3D{}".format(flash1_path), + "bs=3D1M", "count=3D64"], + stdout=3Dfd_null, stderr=3Dsubprocess.STDOUT) + fd_null.close() + +def get_pflash_args(flash_dir=3D"./"): + """Returns a string that can be used to + boot qemu using the appropriate pflash files + for aarch64.""" + flash0_path =3D get_flash_path(flash_dir, "flash0") + flash1_path =3D get_flash_path(flash_dir, "flash1") + pflash_args_str =3D "-drive file=3D{},format=3Draw,if=3Dpflash "\ + "-drive file=3D{},format=3Draw,if=3Dpflash" + pflash_args =3D pflash_args_str.format(flash0_path, flash1_path) + return pflash_args.split(" ") + +def get_flash_path(flash_dir, name): + return os.path.join(flash_dir, "{}.img".format(name)) diff --git a/tests/vm/basevm.py b/tests/vm/basevm.py index 4fa1ea8fde..e94b2d0356 100644 --- a/tests/vm/basevm.py +++ b/tests/vm/basevm.py @@ -497,6 +497,14 @@ def get_qemu_path(arch, build_path=3DNone): qemu_path =3D "qemu-system-" + arch return qemu_path =20 +def get_qemu_version(qemu_path): + """Get the version number from the current QEMU, + and return the major number.""" + output =3D subprocess.check_output([qemu_path, '--version']) + version_line =3D output.decode("utf-8") + version_num =3D re.split(' |\(', version_line)[3].split('.')[0] + return int(version_num) + def parse_config(config, args): """ Parse yaml config and populate our config structure. The yaml config allows the user to override the diff --git a/tests/vm/ubuntu.aarch64 b/tests/vm/ubuntu.aarch64 new file mode 100755 index 0000000000..81d3b2f5ed --- /dev/null +++ b/tests/vm/ubuntu.aarch64 @@ -0,0 +1,117 @@ +#!/usr/bin/env python3 +# +# Ubuntu aarch64 image +# +# Copyright 2020 Linaro +# +# Authors: +# Robert Foley +# Originally based on ubuntu.i386 Fam Zheng +# +# This code is licensed under the GPL version 2 or later. See +# the COPYING file in the top-level directory. +# + +import os +import sys +import subprocess +import basevm +from qemu.accel import kvm_available +import time +import aarch64vm + +DEFAULT_CONFIG =3D { + 'cpu' : "cortex-a57", + 'machine' : "virt,gic-version=3D3", + 'install_cmds' : "apt-get update,"\ + "apt-get build-dep -y --arch-only qemu,"\ + "apt-get install -y libfdt-dev flex bison pkg-config", + # We increase beyond the default time since during boot + # it can take some time (many seconds) to log into the VM + # especially using softmmu. + 'ssh_timeout' : 60, +} + +class UbuntuAarch64VM(basevm.BaseVM): + name =3D "ubuntu.aarch64" + arch =3D "aarch64" + image_name =3D "ubuntu-18.04-server-cloudimg-arm64.img" + image_link =3D "https://cloud-images.ubuntu.com/releases/18.04/release= /" + image_name + login_prompt =3D "ubuntu-aarch64-guest login:" + BUILD_SCRIPT =3D """ + set -e; + cd $(mktemp -d); + sudo chmod a+r /dev/vdb; + tar --checkpoint=3D.10 -xf /dev/vdb; + ./configure {configure_opts}; + make --output-sync {target} -j{jobs} {verbose}; + """ + def boot(self, img, extra_args=3DNone): + aarch64vm.create_flash_images(self._tmpdir) + default_args =3D aarch64vm.get_pflash_args(self._tmpdir) + if extra_args: + extra_args.extend(default_args) + else: + extra_args =3D default_args + # We always add these performance tweaks + # because without them, we boot so slowly that we + # can time out finding the boot efi device. + if '-smp' not in extra_args and \ + '-smp' not in self._config['extra_args'] and \ + '-smp' not in self._args: + # Only add if not already there to give caller option to chang= e it. + extra_args.extend(["-smp", "8"]) + + # We have overridden boot() since aarch64 has additional parameter= s. + # Call down to the base class method. + super(UbuntuAarch64VM, self).boot(img, extra_args=3Dextra_args) + + def build_image(self, img): + os_img =3D self._download_with_cache(self.image_link) + img_tmp =3D img + ".tmp" + subprocess.check_call(["cp", "-f", os_img, img_tmp]) + subprocess.check_call(["qemu-img", "resize", img_tmp, "+50G"]) + ci_img =3D self.gen_cloud_init_iso() + + self.boot(img_tmp, extra_args =3D ["-cdrom", ci_img]) + if self.debug: + self.wait_boot() + # First command we issue is fix for slow ssh login. + self.wait_ssh(wait_root=3DTrue, + cmd=3D"chmod -x /etc/update-motd.d/*") + # Wait for cloud init to finish + self.wait_ssh(wait_root=3DTrue, + cmd=3D"ls /var/lib/cloud/instance/boot-finished") + self.ssh_root("touch /etc/cloud/cloud-init.disabled") + # Disable auto upgrades. + # We want to keep the VM system state stable. + self.ssh_root('sed -ie \'s/"1"/"0"/g\' /etc/apt/apt.conf.d/20auto-= upgrades') + # If the user chooses *not* to do the second phase, + # then we will jump right to the graceful shutdown + if self._config['install_cmds'] !=3D "": + self.ssh_root("sync") + # Shutdown and then boot it again. + # Allows us to know for sure it is booting (not shutting down) + # before we call wait_ssh(). + self.graceful_shutdown() + self.boot(img_tmp) + if self.debug: + self.wait_boot() + self.wait_ssh(wait_root=3DTrue) + self.wait_ssh(wait_root=3DTrue, cmd=3D"locale") + # The previous update sometimes doesn't survive a reboot, so d= o it again + self.ssh_root("sed -ie s/^#\ deb-src/deb-src/g /etc/apt/source= s.list") + + # Issue the install commands. + # This can be overriden by the user in the config .yml. + install_cmds =3D self._config['install_cmds'].split(',') + for cmd in install_cmds: + self.ssh_root(cmd) + self.graceful_shutdown() + self.wait() + os.rename(img_tmp, img) + return 0 + +if __name__ =3D=3D "__main__": + defaults =3D aarch64vm.get_config_defaults(UbuntuAarch64VM, DEFAULT_CO= NFIG) + sys.exit(basevm.main(UbuntuAarch64VM, defaults)) --=20 2.17.1 From nobody Sat Jun 1 02:33:29 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=linaro.org ARC-Seal: i=1; a=rsa-sha256; t=1589312388; cv=none; d=zohomail.com; s=zohoarc; b=B3pDnSqm6phfkTC08zFJHCWWHT3Wzegh3ll0aEQ1BgVmwcTNto31gba75s8R91/k3pAaIobNSc5sPwmWpPBBL/XutBvVlYnpezpXHYtSqjQPPdJ9U+Dvv53Ium/lM9KGa+YhffRIm4LnuN5KzXi3qFm8LWrnKmcEnvLTkkNqWzM= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1589312388; h=Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:Message-ID:References:Sender:Subject:To; bh=Lia0V8+23Rwgz2blgfWdLoTz66wiNS+AnW6w6e53X7I=; b=EOvd+6S+aYTNrU4TgnTApJc7tsRx5QZfJAqGA5q8+g8wI4EEhvDUTQSNKh2ZcJI2o3MrBWIt7ZHYgF2RqFEh3e0sh280tLdADxEs6lMsBkq+e7A53GSYKwdjpRJIVku1LubryYMw9K+IpQ9HQkfDvoPmBQuyswiIE9Qgy0NDoXg= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1589312388157657.597183939849; Tue, 12 May 2020 12:39:48 -0700 (PDT) Received: from localhost ([::1]:58430 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jYakk-0001Vf-O2 for importer@patchew.org; Tue, 12 May 2020 15:39:46 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:47306) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1jYaew-0000RJ-Gj for qemu-devel@nongnu.org; Tue, 12 May 2020 15:33:46 -0400 Received: from mail-qk1-x732.google.com ([2607:f8b0:4864:20::732]:33577) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1jYaep-0001ss-SM for qemu-devel@nongnu.org; Tue, 12 May 2020 15:33:46 -0400 Received: by mail-qk1-x732.google.com with SMTP id z80so9389437qka.0 for ; Tue, 12 May 2020 12:33:38 -0700 (PDT) Received: from Rfoley-MA01.hsd1.ma.comcast.net ([2601:199:4480:60c0:d31:9f53:b3eb:9b53]) by smtp.gmail.com with ESMTPSA id j45sm2100998qtk.14.2020.05.12.12.33.35 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 12 May 2020 12:33:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=Lia0V8+23Rwgz2blgfWdLoTz66wiNS+AnW6w6e53X7I=; b=g66ox0XhScWtKr6YUm0uIj8L9V7CXQxWRxz5aKEEIo56ElWlCL+5IAAeiJLV8WAdnX o0YpC0eTLyhFtgfjoeFklarFC8f7sb1PjiuCCRFvfZJEUjk2/ec50ky2Tf3bX2QIKxV/ 0XPOnEhaFYgevsNx8cuA8g/2GN32uDQ2kL/82zuVxVvVPtKbndz++ynqb4Npy60zevuY O5dsjuYQdX9jesYdopxRHF8pel+nFMhaV92JI8z+DYnX9DdEGp6Ct6jgyFxNU5yIwIzi aOdamBnhRY/hIVdxKoSBby1WHy4KOyBLUXMJOOXcSUv0qZfPY+gKU4LZATAt3v33N/mG Y0XA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=Lia0V8+23Rwgz2blgfWdLoTz66wiNS+AnW6w6e53X7I=; b=IbaPvE+RhPbQ16TDbN5cp0xOSRYOjnxGjdp9bc4uCMGwbq4oQEdUor1ZK1OfU3VZ25 5FBoPsjGEbqc+3Drpauxu7TSKQvCGJcqKmqfJkXiQF2+QmTeV9Upgww80gDMpZCDzrUy tKOeaf9l9I9yrmZQ+i8L8C6ptt27zJKOJFSHSL/ucPtt2KJ+xw0V9afb+noMgkJMYHaP aWJZv3vxyFqQDl0Y2CUpXUFpoYKOOaD450a9lSAd8WPGIHdRGPDRRy+3HysZQho1VPWH qtq8WInVhY/Si5TVJ1dzpID4GR+MErOGLtcmsXnJwJYO0Z2ZzarYFmTGbtI5SmOAJd9t 3MzQ== X-Gm-Message-State: AGi0PuYB+orYRfy9Q8UD3wqLVQwzBg944YqsWqX17rao6kgUVZaL9Bp5 UsOHcf5jnGa3JPCrqVVc3Xq0AOXN8qazwg== X-Google-Smtp-Source: APiQypIfhyKu81KEhawsHZeALveTTGQyFcns7gRj38RDSrVU/j1LoaJa+pXDg88ebWdvh5pN7nUSUg== X-Received: by 2002:a37:66d5:: with SMTP id a204mr22303939qkc.188.1589312016727; Tue, 12 May 2020 12:33:36 -0700 (PDT) From: Robert Foley To: qemu-devel@nongnu.org Subject: [PATCH v6 7/9] tests/vm: Added a new script for centos.aarch64. Date: Tue, 12 May 2020 15:33:38 -0400 Message-Id: <20200512193340.265-8-robert.foley@linaro.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20200512193340.265-1-robert.foley@linaro.org> References: <20200512193340.265-1-robert.foley@linaro.org> Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=2607:f8b0:4864:20::732; envelope-from=robert.foley@linaro.org; helo=mail-qk1-x732.google.com X-detected-operating-system: by eggs.gnu.org: No matching host in p0f cache. That's all we know. X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001 autolearn=_AUTOLEARN X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: philmd@redhat.com, alex.bennee@linaro.org, robert.foley@linaro.org, peter.puhov@linaro.org Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" centos.aarch64 creates a CentOS 8 image. Also added a new kickstart script used to build the centos.aarch64 image. Signed-off-by: Robert Foley Reviewed-by: Peter Puhov --- tests/vm/Makefile.include | 3 +- tests/vm/centos-8-aarch64.ks | 51 ++++++++ tests/vm/centos.aarch64 | 227 +++++++++++++++++++++++++++++++++++ 3 files changed, 280 insertions(+), 1 deletion(-) create mode 100644 tests/vm/centos-8-aarch64.ks create mode 100755 tests/vm/centos.aarch64 diff --git a/tests/vm/Makefile.include b/tests/vm/Makefile.include index d918d13f50..62502c13c2 100644 --- a/tests/vm/Makefile.include +++ b/tests/vm/Makefile.include @@ -6,7 +6,7 @@ IMAGES :=3D freebsd netbsd openbsd centos fedora ifneq ($(GENISOIMAGE),) IMAGES +=3D ubuntu.i386 centos ifneq ($(QEMU_EFI_AARCH64),) -IMAGES +=3D ubuntu.aarch64 +IMAGES +=3D ubuntu.aarch64 centos.aarch64 endif endif =20 @@ -28,6 +28,7 @@ ifneq ($(GENISOIMAGE),) @echo " vm-build-ubuntu.i386 - Build QEMU in ubuntu i386 VM" ifneq ($(QEMU_EFI_AARCH64),) @echo " vm-build-ubuntu.aarch64 - Build QEMU in ubuntu aarch64 V= M" + @echo " vm-build-centos.aarch64 - Build QEMU in CentOS aarch64 V= M" else @echo " (install qemu-efi-aarch64 to build centos/ubuntu aarch64 images.= )" endif diff --git a/tests/vm/centos-8-aarch64.ks b/tests/vm/centos-8-aarch64.ks new file mode 100644 index 0000000000..fd6ebe4d49 --- /dev/null +++ b/tests/vm/centos-8-aarch64.ks @@ -0,0 +1,51 @@ +# CentOS aarch64 image kickstart file. +# This file is used by the CentOS installer to +# script the generation of the image. +# +# Copyright 2020 Linaro +# +ignoredisk --only-use=3Dvda +# System bootloader configuration +bootloader --append=3D" crashkernel=3Dauto" --location=3Dmbr --boot-drive= =3Dvda +autopart --type=3Dplain +# Partition clearing information +clearpart --linux --initlabel --drives=3Dvda +# Use text mode install +text +repo --name=3D"AppStream" --baseurl=3Dfile:///run/install/repo/AppStream +# Use CDROM installation media +cdrom +# Keyboard layouts +keyboard --vckeymap=3Dus --xlayouts=3D'' +# System language +lang en_US.UTF-8 + +# Network information +network --bootproto=3Ddhcp --device=3Denp0s1 --onboot=3Doff --ipv6=3Dauto= --no-activate +network --hostname=3Dlocalhost.localdomain +# Run the Setup Agent on first boot +firstboot --enable +# Do not configure the X Window System +skipx +# System services +services --enabled=3D"chronyd" +# System timezone +timezone America/New_York --isUtc + +# Shutdown after installation is complete. +shutdown + +%packages +@^server-product-environment +kexec-tools + +%end + +%addon com_redhat_kdump --enable --reserve-mb=3D'auto' + +%end +%anaconda +pwpolicy root --minlen=3D6 --minquality=3D1 --notstrict --nochanges --note= mpty +pwpolicy user --minlen=3D6 --minquality=3D1 --notstrict --nochanges --empt= yok +pwpolicy luks --minlen=3D6 --minquality=3D1 --notstrict --nochanges --note= mpty +%end diff --git a/tests/vm/centos.aarch64 b/tests/vm/centos.aarch64 new file mode 100755 index 0000000000..db61fbbce1 --- /dev/null +++ b/tests/vm/centos.aarch64 @@ -0,0 +1,227 @@ +#!/usr/bin/env python3 +# +# Centos aarch64 image +# +# Copyright 2020 Linaro +# +# Authors: +# Robert Foley +# Originally based on ubuntu.aarch64 +# +# This code is licensed under the GPL version 2 or later. See +# the COPYING file in the top-level directory. +# + +import os +import sys +import subprocess +import basevm +import time +import traceback +import aarch64vm + +DEFAULT_CONFIG =3D { + 'cpu' : "max", + 'machine' : "virt,gic-version=3Dmax", + 'install_cmds' : "yum install -y make git python3 gcc gcc-c++ flex bis= on, "\ + "yum install -y glib2-devel pixman-devel zlib-devel, "\ + "yum install -y perl-Test-Harness, "\ + "alternatives --set python /usr/bin/python3, "\ + "sudo dnf config-manager "\ + "--add-repo=3Dhttps://download.docker.com/linux/centos/docker-ce.r= epo,"\ + "sudo dnf install --nobest -y docker-ce.aarch64,"\ + "systemctl enable docker", + # We increase beyond the default time since during boot + # it can take some time (many seconds) to log into the VM. + 'ssh_timeout' : 60, +} + +class CentosAarch64VM(basevm.BaseVM): + name =3D "centos.aarch64" + arch =3D "aarch64" + login_prompt =3D "localhost login:" + prompt =3D '[root@localhost ~]#' + image_name =3D "CentOS-8-aarch64-1905-dvd1.iso" + image_link =3D "http://mirrors.usc.edu/pub/linux/distributions/centos/= 8.0.1905/isos/aarch64/" + image_link +=3D image_name + BUILD_SCRIPT =3D """ + set -e; + cd $(mktemp -d); + sudo chmod a+r /dev/vdb; + tar --checkpoint=3D.10 -xf /dev/vdb; + ./configure {configure_opts}; + make --output-sync {target} -j{jobs} {verbose}; + """ + def set_key_perm(self): + """Set permissions properly on certain files to allow + ssh access.""" + self.console_wait_send(self.prompt, + "/usr/sbin/restorecon -R -v /root/.ssh\n") + self.console_wait_send(self.prompt, + "/usr/sbin/restorecon -R -v "\ + "/home/{}/.ssh\n".format(self._config["guest_user"])) + + def create_kickstart(self): + """Generate the kickstart file used to generate the centos image."= "" + # Start with the template for the kickstart. + ks_file =3D "../tests/vm/centos-8-aarch64.ks" + subprocess.check_call("cp {} ./ks.cfg".format(ks_file), shell=3DTr= ue) + # Append the ssh keys to the kickstart file + # as the post processing phase of installation. + with open("ks.cfg", "a") as f: + # Add in the root pw and guest user. + rootpw =3D "rootpw --plaintext {}\n" + f.write(rootpw.format(self._config["root_pass"])) + add_user =3D "user --groups=3Dwheel --name=3D{} "\ + "--password=3D{} --plaintext\n" + f.write(add_user.format(self._config["guest_user"], + self._config["guest_pass"])) + # Add the ssh keys. + f.write("%post --log=3D/root/ks-post.log\n") + f.write("mkdir -p /root/.ssh\n") + addkey =3D 'echo "{}" >> /root/.ssh/authorized_keys\n' + addkey_cmd =3D addkey.format(self._config["ssh_pub_key"]) + f.write(addkey_cmd) + f.write('mkdir -p /home/{}/.ssh\n'.format(self._config["guest_= user"])) + addkey =3D 'echo "{}" >> /home/{}/.ssh/authorized_keys\n' + addkey_cmd =3D addkey.format(self._config["ssh_pub_key"], + self._config["guest_user"]) + f.write(addkey_cmd) + f.write("%end\n") + # Take our kickstart file and create an .iso from it. + # The .iso will be provided to qemu as we boot + # from the install dvd. + # Anaconda will recognize the label "OEMDRV" and will + # start the automated installation. + gen_iso_img =3D 'genisoimage -output ks.iso -volid "OEMDRV" ks.cfg' + subprocess.check_call(gen_iso_img, shell=3DTrue) + + def wait_for_shutdown(self): + """We wait for qemu to shutdown the VM and exit. + While this happens we display the console view + for easier debugging.""" + # The image creation is essentially done, + # so whether or not the wait is successful we want to + # wait for qemu to exit (the self.wait()) before we return. + try: + self.console_wait("reboot: Power down") + except Exception as e: + sys.stderr.write("Exception hit\n") + if isinstance(e, SystemExit) and e.code =3D=3D 0: + return 0 + traceback.print_exc() + finally: + self.wait() + + def build_base_image(self, dest_img): + """Run through the centos installer to create + a base image with name dest_img.""" + # We create the temp image, and only rename + # to destination when we are done. + img =3D dest_img + ".tmp" + # Create an empty image. + # We will provide this as the install destination. + qemu_img_create =3D "qemu-img create {} 50G".format(img) + subprocess.check_call(qemu_img_create, shell=3DTrue) + + # Create our kickstart file to be fed to the installer. + self.create_kickstart() + # Boot the install dvd with the params as our ks.iso + os_img =3D self._download_with_cache(self.image_link) + dvd_iso =3D "centos-8-dvd.iso" + subprocess.check_call(["cp", "-f", os_img, dvd_iso]) + extra_args =3D "-cdrom ks.iso" + extra_args +=3D " -drive file=3D{},if=3Dnone,id=3Ddrive1,cache=3Dw= riteback" + extra_args +=3D " -device virtio-blk,drive=3Ddrive1,bootindex=3D1" + extra_args =3D extra_args.format(dvd_iso).split(" ") + self.boot(img, extra_args=3Dextra_args) + self.console_wait_send("change the selection", "\n") + # We seem to need to hit esc (chr(27)) twice to abort the + # media check, which takes a long time. + # Waiting a bit seems to be more reliable before hitting esc. + self.console_wait("Checking") + time.sleep(5) + self.console_wait_send("Checking", chr(27)) + time.sleep(5) + self.console_wait_send("Checking", chr(27)) + print("Found Checking") + # Give sufficient time for the installer to create the image. + self.console_init(timeout=3D7200) + self.wait_for_shutdown() + os.rename(img, dest_img) + print("Done with base image build: {}".format(dest_img)) + + def check_create_base_img(self, img_base, img_dest): + """Create a base image using the installer. + We will use the base image if it exists. + This helps cut down on install time in case we + need to restart image creation, + since the base image creation can take a long time.""" + if not os.path.exists(img_base): + print("Generate new base image: {}".format(img_base)) + self.build_base_image(img_base); + else: + print("Use existing base image: {}".format(img_base)) + # Save a copy of the base image and copy it to dest. + # which we will use going forward. + subprocess.check_call(["cp", img_base, img_dest]) + + def boot(self, img, extra_args=3DNone): + aarch64vm.create_flash_images(self._tmpdir) + default_args =3D aarch64vm.get_pflash_args(self._tmpdir) + if extra_args: + extra_args.extend(default_args) + else: + extra_args =3D default_args + # We always add these performance tweaks + # because without them, we boot so slowly that we + # can time out finding the boot efi device. + if '-smp' not in extra_args and \ + '-smp' not in self._config['extra_args'] and \ + '-smp' not in self._args: + # Only add if not already there to give caller option to chang= e it. + extra_args.extend(["-smp", "8"]) + # We have overridden boot() since aarch64 has additional parameter= s. + # Call down to the base class method. + super(CentosAarch64VM, self).boot(img, extra_args=3Dextra_args) + + def build_image(self, img): + img_tmp =3D img + ".tmp" + self.check_create_base_img(img + ".base", img_tmp) + + # Boot the new image for the first time to finish installation. + self.boot(img_tmp) + self.console_init() + self.console_wait_send(self.login_prompt, "root\n") + self.console_wait_send("Password:", + "{}\n".format(self._config["root_pass"])) + + self.set_key_perm() + self.console_wait_send(self.prompt, "rpm -q centos-release\n") + enable_adapter =3D "sed -i 's/ONBOOT=3Dno/ONBOOT=3Dyes/g'" \ + " /etc/sysconfig/network-scripts/ifcfg-enp0s1\n" + self.console_wait_send(self.prompt, enable_adapter) + self.console_wait_send(self.prompt, "ifup enp0s1\n") + self.console_wait_send(self.prompt, + 'echo "qemu ALL=3D(ALL) NOPASSWD:ALL" | '\ + 'sudo tee /etc/sudoers.d/qemu\n') + self.console_wait(self.prompt) + + # Rest of the commands we issue through ssh. + self.wait_ssh(wait_root=3DTrue) + + # If the user chooses *not* to do the second phase, + # then we will jump right to the graceful shutdown + if self._config['install_cmds'] !=3D "": + install_cmds =3D self._config['install_cmds'].split(',') + for cmd in install_cmds: + self.ssh_root(cmd) + self.ssh_root("poweroff") + self.wait_for_shutdown() + os.rename(img_tmp, img) + print("image creation complete: {}".format(img)) + return 0 + +if __name__ =3D=3D "__main__": + defaults =3D aarch64vm.get_config_defaults(CentosAarch64VM, DEFAULT_CO= NFIG) + sys.exit(basevm.main(CentosAarch64VM, defaults)) --=20 2.17.1 From nobody Sat Jun 1 02:33:30 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=linaro.org ARC-Seal: i=1; a=rsa-sha256; t=1589312532; cv=none; d=zohomail.com; s=zohoarc; b=euWUvWRilpthdDAxXABU54LggIKglo3r11y49N+sdXbzr+H3On3Ze2SHU5kr54PsJRYxa+7NL0rGMEPqPY1C6ETHbFMvhzLc9Vwq0GGYvGRUlSUwWv3ECXhpLnCsnGTbdKwhmufvIO4GNr7qr9GTaooc3dytZpQLgUKRNABpZVs= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1589312532; h=Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:Message-ID:References:Sender:Subject:To; bh=/s//tYI4BgCcobGdVgIcQsFlBmIupQaLWvMZYw32pNg=; b=E0OslyTlikyhsynrWYt08EnB3EmmRHMeard1rErvFY6SfOZPVjzLP+41m293euC/axZtsjfFY4JVKUynDvIfIhgVWNIqkwvxd9tynnjyYaQo5ymn5bWVOwbJACqPchk2m7HJB1epyhqDhQ1DoL52SWB/0/Gv2yZyYcpp22UtAQQ= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1589312532303176.3122853256865; Tue, 12 May 2020 12:42:12 -0700 (PDT) Received: from localhost ([::1]:37782 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jYan4-0004wk-Pa for importer@patchew.org; Tue, 12 May 2020 15:42:10 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:47302) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1jYaew-0000RF-EL for qemu-devel@nongnu.org; Tue, 12 May 2020 15:33:46 -0400 Received: from mail-qk1-x741.google.com ([2607:f8b0:4864:20::741]:39426) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1jYaep-0001tG-Qa for qemu-devel@nongnu.org; Tue, 12 May 2020 15:33:43 -0400 Received: by mail-qk1-x741.google.com with SMTP id a136so5957860qkg.6 for ; Tue, 12 May 2020 12:33:38 -0700 (PDT) Received: from Rfoley-MA01.hsd1.ma.comcast.net ([2601:199:4480:60c0:d31:9f53:b3eb:9b53]) by smtp.gmail.com with ESMTPSA id j45sm2100998qtk.14.2020.05.12.12.33.36 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 12 May 2020 12:33:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=/s//tYI4BgCcobGdVgIcQsFlBmIupQaLWvMZYw32pNg=; b=M7k0kKaVCvM58fESIjIefSEWXIoY7spL3fQeRU95LWoydJB6GR5zB5l3jDGmmLB1Gj Svv2ATfCEwoeAH09kz50LypPkD9HfIsWA6qtmB06rNybmDDO+9Kt8IHmWmeWWDpqeAFl KcieA0OVPqaqiC2K16Au52FQloYq4riQ4nnl7tFDAooiZCftUyQSNjfn+ms2vSeBgreE IHKZkgTSDT5lVr8EwqGKeeH0ZMDtPJtcTRAzekJa9/QpjPJkLuIOb4Ok3YVFbHgRZPwm SvnG9umIVg8j/ydPKvl6F9wGLHysMnVYkBD59GZCetzrlxfsJN2lmTEykAwO3eogDfqj OM4w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=/s//tYI4BgCcobGdVgIcQsFlBmIupQaLWvMZYw32pNg=; b=l85woTQt0Szm+cd2v2m9TeZ+QcXB8Y+LmqBIYvSua9FbEqZz8DDgdWdtqJ0C9BNsrO JHV/uycLm7p9FCesB0fZkD5nXtK4HUKTdzmsrBT9Pb7LD4OdCdkLJefXvtKVzW29vqxy HUxVNOkoFbRxFoLOB2gRX7FXjKnrP9FsTK4q3PFrlOBtTb5ybAAMJcM1Q1UK3GPsAQ/F lO37J+F3hXdOeBKUMR22sbPe/ZNq5o86Ga5blV7R1HqBJJzKwWgW2IzpS8u+pAhwynPj eZPnbVR5vxNmCnkL8l2oDY7efgC/5vcW9t9PQnZwMNQ9FZrOd4lkYzoZOvti1L1YFOfV LR2Q== X-Gm-Message-State: AGi0PuafhZ/s9V86V1lonfQmGWBnhCs4dnXe7Lib8XXMnP+02XmLNdP4 FQ2Wjy11jdUVNUpGkIcuLiUnr7bvCeU5Aw== X-Google-Smtp-Source: APiQypJ0wPZQpnOqkQ9kQXWUypE/H9trw55qiUN2JPXpe4UG57/XfZA+8qjIGRwBHLXiPWLOTaO5aw== X-Received: by 2002:a37:4b0c:: with SMTP id y12mr21783463qka.43.1589312017781; Tue, 12 May 2020 12:33:37 -0700 (PDT) From: Robert Foley To: qemu-devel@nongnu.org Subject: [PATCH v6 8/9] tests/vm: change scripts to use self._config Date: Tue, 12 May 2020 15:33:39 -0400 Message-Id: <20200512193340.265-9-robert.foley@linaro.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20200512193340.265-1-robert.foley@linaro.org> References: <20200512193340.265-1-robert.foley@linaro.org> Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=2607:f8b0:4864:20::741; envelope-from=robert.foley@linaro.org; helo=mail-qk1-x741.google.com X-detected-operating-system: by eggs.gnu.org: No matching host in p0f cache. That's all we know. X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001 autolearn=_AUTOLEARN X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: philmd@redhat.com, alex.bennee@linaro.org, robert.foley@linaro.org, peter.puhov@linaro.org Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" This change converts existing scripts to using for example self.ROOT_PASS, to self._config['root_pass']. We made similar changes for GUEST_USER, and GUEST_PASS. This allows us also to remove the change in basevm.py, which adds __getattr__ for backwards compatibility. Signed-off-by: Robert Foley Reviewed-by: Peter Puhov --- tests/vm/basevm.py | 11 ++--------- tests/vm/fedora | 17 +++++++++-------- tests/vm/freebsd | 16 ++++++++-------- tests/vm/netbsd | 19 ++++++++++--------- tests/vm/openbsd | 17 +++++++++-------- 5 files changed, 38 insertions(+), 42 deletions(-) diff --git a/tests/vm/basevm.py b/tests/vm/basevm.py index e94b2d0356..bb20d8ca54 100644 --- a/tests/vm/basevm.py +++ b/tests/vm/basevm.py @@ -180,13 +180,6 @@ class BaseVM(object): self.console_init(timeout=3Dtimeout) self.console_wait(wait_string) =20 - def __getattr__(self, name): - # Support direct access to config by key. - # for example, access self._config['cpu'] by self.cpu - if name.lower() in self._config.keys(): - return self._config[name.lower()] - return object.__getattribute__(self, name) - def _download_with_cache(self, url, sha256sum=3DNone, sha512sum=3DNone= ): def check_sha256sum(fname): if not sha256sum: @@ -236,13 +229,13 @@ class BaseVM(object): return r =20 def ssh(self, *cmd): - return self._ssh_do(self.GUEST_USER, cmd, False) + return self._ssh_do(self._config["guest_user"], cmd, False) =20 def ssh_root(self, *cmd): return self._ssh_do("root", cmd, False) =20 def ssh_check(self, *cmd): - self._ssh_do(self.GUEST_USER, cmd, True) + self._ssh_do(self._config["guest_user"], cmd, True) =20 def ssh_root_check(self, *cmd): self._ssh_do("root", cmd, True) diff --git a/tests/vm/fedora b/tests/vm/fedora index bd9c6cf295..f536a92678 100755 --- a/tests/vm/fedora +++ b/tests/vm/fedora @@ -109,20 +109,20 @@ class FedoraVM(basevm.BaseVM): =20 self.console_wait_send("7) [!] Root password", "7\n") self.console_wait("Password:") - self.console_send("%s\n" % self.ROOT_PASS) + self.console_send("%s\n" % self._config["root_pass"]) self.console_wait("Password (confirm):") - self.console_send("%s\n" % self.ROOT_PASS) + self.console_send("%s\n" % self._config["root_pass"]) =20 self.console_wait_send("8) [ ] User creation", "8\n") self.console_wait_send("1) [ ] Create user", "1\n") self.console_wait_send("3) User name", "3\n") - self.console_wait_send("ENTER:", "%s\n" % self.GUEST_USER) + self.console_wait_send("ENTER:", "%s\n" % self._config["guest_user= "]) self.console_wait_send("4) [ ] Use password", "4\n") self.console_wait_send("5) Password", "5\n") self.console_wait("Password:") - self.console_send("%s\n" % self.GUEST_PASS) + self.console_send("%s\n" % self._config["guest_pass"]) self.console_wait("Password (confirm):") - self.console_send("%s\n" % self.GUEST_PASS) + self.console_send("%s\n" % self._config["guest_pass"]) self.console_wait_send("7) Groups", "c\n") =20 while True: @@ -140,7 +140,7 @@ class FedoraVM(basevm.BaseVM): if good: break time.sleep(10) - self.console_send("r\n" % self.GUEST_PASS) + self.console_send("r\n" % self._config["guest_pass"]) =20 self.console_wait_send("'b' to begin install", "b\n") =20 @@ -151,12 +151,13 @@ class FedoraVM(basevm.BaseVM): =20 # setup qemu user prompt =3D " ~]$" - self.console_ssh_init(prompt, self.GUEST_USER, self.GUEST_PASS) + self.console_ssh_init(prompt, self._config["guest_user"], + self._config["guest_pass"]) self.console_wait_send(prompt, "exit\n") =20 # setup root user prompt =3D " ~]#" - self.console_ssh_init(prompt, "root", self.ROOT_PASS) + self.console_ssh_init(prompt, "root", self._config["root_pass"]) self.console_sshd_config(prompt) =20 # setup virtio-blk #1 (tarfile) diff --git a/tests/vm/freebsd b/tests/vm/freebsd index 298967fe9c..898393b58d 100755 --- a/tests/vm/freebsd +++ b/tests/vm/freebsd @@ -114,9 +114,9 @@ class FreeBSDVM(basevm.BaseVM): =20 # post-install configuration self.console_wait("New Password:") - self.console_send("%s\n" % self.ROOT_PASS) + self.console_send("%s\n" % self._config["root_pass"]) self.console_wait("Retype New Password:") - self.console_send("%s\n" % self.ROOT_PASS) + self.console_send("%s\n" % self._config["root_pass"]) =20 self.console_wait_send("Network Configuration", "\n") self.console_wait_send("IPv4", "y") @@ -135,9 +135,9 @@ class FreeBSDVM(basevm.BaseVM): # qemu user self.console_wait_send("Add User Accounts", "y") self.console_wait("Username") - self.console_send("%s\n" % self.GUEST_USER) + self.console_send("%s\n" % self._config["guest_user"]) self.console_wait("Full name") - self.console_send("%s\n" % self.GUEST_USER) + self.console_send("%s\n" % self._config["guest_user"]) self.console_wait_send("Uid", "\n") self.console_wait_send("Login group", "\n") self.console_wait_send("Login group", "\n") @@ -149,9 +149,9 @@ class FreeBSDVM(basevm.BaseVM): self.console_wait_send("Use an empty password", "\n") self.console_wait_send("Use a random password", "\n") self.console_wait("Enter password:") - self.console_send("%s\n" % self.GUEST_PASS) + self.console_send("%s\n" % self._config["guest_pass"]) self.console_wait("Enter password again:") - self.console_send("%s\n" % self.GUEST_PASS) + self.console_send("%s\n" % self._config["guest_pass"]) self.console_wait_send("Lock out", "\n") self.console_wait_send("OK", "yes\n") self.console_wait_send("Add another user", "no\n") @@ -165,12 +165,12 @@ class FreeBSDVM(basevm.BaseVM): =20 # setup qemu user prompt =3D "$" - self.console_ssh_init(prompt, self.GUEST_USER, self.GUEST_PASS) + self.console_ssh_init(prompt, self._config["guest_user"], self._co= nfig["guest_pass"]) self.console_wait_send(prompt, "exit\n") =20 # setup root user prompt =3D "root@freebsd:~ #" - self.console_ssh_init(prompt, "root", self.ROOT_PASS) + self.console_ssh_init(prompt, "root", self._config["root_pass"]) self.console_sshd_config(prompt) =20 # setup serial console diff --git a/tests/vm/netbsd b/tests/vm/netbsd index b10c9d429d..7e54fe8322 100755 --- a/tests/vm/netbsd +++ b/tests/vm/netbsd @@ -121,24 +121,24 @@ class NetBSDVM(basevm.BaseVM): self.console_wait_send("d: Change root password", "d\n") self.console_wait_send("a: Yes", "a\n") self.console_wait("New password:") - self.console_send("%s\n" % self.ROOT_PASS) + self.console_send("%s\n" % self._config["root_pass"]) self.console_wait("New password:") - self.console_send("%s\n" % self.ROOT_PASS) + self.console_send("%s\n" % self._config["root_pass"]) self.console_wait("Retype new password:") - self.console_send("%s\n" % self.ROOT_PASS) + self.console_send("%s\n" % self._config["root_pass"]) =20 self.console_wait_send("o: Add a user", "o\n") self.console_wait("username") - self.console_send("%s\n" % self.GUEST_USER) + self.console_send("%s\n" % self._config["guest_pass"]) self.console_wait("to group wheel") self.console_wait_send("a: Yes", "a\n") self.console_wait_send("a: /bin/sh", "a\n") self.console_wait("New password:") - self.console_send("%s\n" % self.GUEST_PASS) + self.console_send("%s\n" % self._config["guest_pass"]) self.console_wait("New password:") - self.console_send("%s\n" % self.GUEST_PASS) + self.console_send("%s\n" % self._config["guest_pass"]) self.console_wait("Retype new password:") - self.console_send("%s\n" % self.GUEST_PASS) + self.console_send("%s\n" % self._config["guest_pass"]) =20 self.console_wait_send("a: Configure network", "a\n") self.console_wait_send("a: vioif0", "a\n") @@ -171,12 +171,13 @@ class NetBSDVM(basevm.BaseVM): =20 # setup qemu user prompt =3D "localhost$" - self.console_ssh_init(prompt, self.GUEST_USER, self.GUEST_PASS) + self.console_ssh_init(prompt, self._config["guest_user"], + self._config["guest_pass"]) self.console_wait_send(prompt, "exit\n") =20 # setup root user prompt =3D "localhost#" - self.console_ssh_init(prompt, "root", self.ROOT_PASS) + self.console_ssh_init(prompt, "root", self._config["root_pass"]) self.console_sshd_config(prompt) =20 # setup virtio-blk #1 (tarfile) diff --git a/tests/vm/openbsd b/tests/vm/openbsd index 0b705f4945..04a532c763 100755 --- a/tests/vm/openbsd +++ b/tests/vm/openbsd @@ -99,9 +99,9 @@ class OpenBSDVM(basevm.BaseVM): self.console_wait_send("Which network interface", "done\n") self.console_wait_send("DNS domain name", "localnet\n") self.console_wait("Password for root account") - self.console_send("%s\n" % self.ROOT_PASS) + self.console_send("%s\n" % self._config["root_pass"]) self.console_wait("Password for root account") - self.console_send("%s\n" % self.ROOT_PASS) + self.console_send("%s\n" % self._config["root_pass"]) self.console_wait_send("Start sshd(8)", "yes\n") self.console_wait_send("X Window System", "\n") self.console_wait_send("xenodm", "\n") @@ -109,13 +109,13 @@ class OpenBSDVM(basevm.BaseVM): self.console_wait_send("Which speed", "\n") =20 self.console_wait("Setup a user") - self.console_send("%s\n" % self.GUEST_USER) + self.console_send("%s\n" % self._config["guest_user"]) self.console_wait("Full name") - self.console_send("%s\n" % self.GUEST_USER) + self.console_send("%s\n" % self._config["guest_user"]) self.console_wait("Password") - self.console_send("%s\n" % self.GUEST_PASS) + self.console_send("%s\n" % self._config["guest_pass"]) self.console_wait("Password") - self.console_send("%s\n" % self.GUEST_PASS) + self.console_send("%s\n" % self._config["guest_pass"]) =20 self.console_wait_send("Allow root ssh login", "yes\n") self.console_wait_send("timezone", "UTC\n") @@ -136,12 +136,13 @@ class OpenBSDVM(basevm.BaseVM): =20 # setup qemu user prompt =3D "$" - self.console_ssh_init(prompt, self.GUEST_USER, self.GUEST_PASS) + self.console_ssh_init(prompt, self._config["guest_user"], + self._config["guest_pass"]) self.console_wait_send(prompt, "exit\n") =20 # setup root user prompt =3D "openbsd#" - self.console_ssh_init(prompt, "root", self.ROOT_PASS) + self.console_ssh_init(prompt, "root", self._config["root_pass"]) self.console_sshd_config(prompt) =20 # setup virtio-blk #1 (tarfile) --=20 2.17.1 From nobody Sat Jun 1 02:33:30 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=linaro.org ARC-Seal: i=1; a=rsa-sha256; t=1589312443; cv=none; d=zohomail.com; s=zohoarc; b=SZmj2vXacNMr/NsByMgMFW88HuLxiHw9sNaxSGGcvlTCG4VsZjOXykjqPbPJMdI9TSFUIbqvvh5WxbyWWPne8YTiRaCsKCcoqZqYgyvuLb0xkAlYD38Lxdgy3e6aU6vkADKqXuk2gmAiCeMrZyy4tbKvjq4ayWHluUdoWYqls6Y= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1589312443; h=Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:Message-ID:References:Sender:Subject:To; bh=aNR1JmgVz46Qvpg0icRk9M4DFTNu1hcHi8bAg4H6vGg=; b=UYGvdJaWFFhurjVeOO37dmZyQ0IoNaM8DNpbKoAZ7WFVwH0wyrIao7yY5Zq9Tm8ZToxmcefxFKzYB2f4xJh0oV+n5shpc3Qv6htq4J6M0u4Az0ZkROY/LchE09inQbCuIjFM8JSxuaCh2xiHDZnoFNM4YUS2iCvx+FxV1zdEbBI= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 158931244302879.01100354683638; Tue, 12 May 2020 12:40:43 -0700 (PDT) Received: from localhost ([::1]:60916 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jYald-0002mq-Iq for importer@patchew.org; Tue, 12 May 2020 15:40:41 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:47314) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1jYaez-0000TN-1v for qemu-devel@nongnu.org; Tue, 12 May 2020 15:33:49 -0400 Received: from mail-qk1-x743.google.com ([2607:f8b0:4864:20::743]:34398) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1jYaer-0001uG-Id for qemu-devel@nongnu.org; Tue, 12 May 2020 15:33:48 -0400 Received: by mail-qk1-x743.google.com with SMTP id 190so9347208qki.1 for ; Tue, 12 May 2020 12:33:40 -0700 (PDT) Received: from Rfoley-MA01.hsd1.ma.comcast.net ([2601:199:4480:60c0:d31:9f53:b3eb:9b53]) by smtp.gmail.com with ESMTPSA id j45sm2100998qtk.14.2020.05.12.12.33.37 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 12 May 2020 12:33:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=aNR1JmgVz46Qvpg0icRk9M4DFTNu1hcHi8bAg4H6vGg=; b=Per6B54NJeJqh2V4COctkhI6NRkRrKNuOwbeEDfzzONC5J/WpCJ/KhuKPe1Tpk4NVN cUilz3940OrWEh3nXJjJcWrmSdI4hgJNPWRqlqRUa4mLHNS7aq/t0XYc/725phhjFcDt pWjZrMZZgChayQgp3Gntn4Vmj6qFaUtURqkR72znmcIQYbwZjv8fmXy65rQTjLIlxRBH VOMIt/uf0B8CbzbCeIjZuTWc4A49XxfkKcjGiJS9p+8NMIOkGw+qcBAO4mtMeBvca4Xn i4knQactc8WJy9qPGWcFkrtbuy5txre39hWt40O52V9D3y+hzDPtLK5j0XwvfoLs7VQe i+5Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=aNR1JmgVz46Qvpg0icRk9M4DFTNu1hcHi8bAg4H6vGg=; b=NdejrZciyjV1CpOC6MVqt7xlpzCvX1NW11GcED8c6bbtGE5fJqUdvOIVvUrNQOz7JU nwBCa9p3QuhH2az7BBzJpMeh13zkExgdgqECliJVsD/GnIiIlKFesNMQMNob0X98PnX7 96umsW2hstCeGuF/lSfEiDW3G1kIFNX5aQ/nvzzi5XTH79gdoQxWdFSbDik/WPVwht35 p/4XQA2pHpcLkfbNge3bpIrR6VnWZLmJh/RhlJ+g34UoPJai/1MEWaMGM+3k3FkCtwFh RzOafB0+c2V4ugaAEXEgD+cbzZJs4/gsxp1VwFQFpG3C0C7r8J4kWWH9fyZE+wdOpzsy Maiw== X-Gm-Message-State: AGi0PuYnLbsETit15nKqfD6mOqZYZiB0oO0RjE5d3oyTpj+RBy74UKpG lR8yCZHggenlf8a6npymCaT+45tmrnu3YA== X-Google-Smtp-Source: APiQypKifRpgxCaWW/ZIQn/NBeeh0gG8x6e+rAeZpGFYZarph2PV3kxEEyBxYTS04k3RDddzOmEcgQ== X-Received: by 2002:a37:50d4:: with SMTP id e203mr21669657qkb.61.1589312018945; Tue, 12 May 2020 12:33:38 -0700 (PDT) From: Robert Foley To: qemu-devel@nongnu.org Subject: [PATCH v6 9/9] tests/vm: Add workaround to consume console Date: Tue, 12 May 2020 15:33:40 -0400 Message-Id: <20200512193340.265-10-robert.foley@linaro.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20200512193340.265-1-robert.foley@linaro.org> References: <20200512193340.265-1-robert.foley@linaro.org> Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=2607:f8b0:4864:20::743; envelope-from=robert.foley@linaro.org; helo=mail-qk1-x743.google.com X-detected-operating-system: by eggs.gnu.org: No matching host in p0f cache. That's all we know. X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001 autolearn=_AUTOLEARN X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: philmd@redhat.com, alex.bennee@linaro.org, robert.foley@linaro.org, peter.puhov@linaro.org Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" The ConsoleSocket object provides a socket interface which will consume all arriving characters on the socket, but will provide those chars via recv() as would a regular socket. This is a workaround we found was needed since there is a known issue where QEMU will hang waiting for console characters to be consumed. We also add the option of logging the console to a file. Signed-off-by: Robert Foley Reviewed-by: Peter Puhov --- python/qemu/console_socket.py | 162 ++++++++++++++++++++++++++++++++++ python/qemu/machine.py | 23 ++++- tests/vm/Makefile.include | 4 + tests/vm/basevm.py | 24 ++++- 4 files changed, 205 insertions(+), 8 deletions(-) create mode 100644 python/qemu/console_socket.py diff --git a/python/qemu/console_socket.py b/python/qemu/console_socket.py new file mode 100644 index 0000000000..a1f74e60ac --- /dev/null +++ b/python/qemu/console_socket.py @@ -0,0 +1,162 @@ +#!/usr/bin/env python3 +# +# This python module implements a ConsoleSocket object which is +# designed always drain the socket itself, and place +# the bytes into a in memory buffer for later processing. +# +# Optionally a file path can be passed in and we will also +# dump the characters to this file for debug. +# +# Copyright 2020 Linaro +# +# Authors: +# Robert Foley +# +# This code is licensed under the GPL version 2 or later. See +# the COPYING file in the top-level directory. +# +import asyncore +import socket +import threading +import io +import os +import sys +from collections import deque +import time +import traceback + +class ConsoleSocket(asyncore.dispatcher): + + def __init__(self, address, file=3DNone): + self._recv_timeout_sec =3D 300 + self._buffer =3D deque() + self._asyncore_thread =3D None + self._sock =3D socket.socket(socket.AF_UNIX, socket.SOCK_STREAM) + self._sock.connect(address) + self._logfile =3D None + if file: + self._logfile =3D open(file, "w") + asyncore.dispatcher.__init__(self, sock=3Dself._sock) + self._thread_start() + self._open =3D True + + def _thread_start(self): + """Kick off a thread to wait on the asyncore.loop""" + if self._asyncore_thread is not None: + return + self._asyncore_thread =3D threading.Thread(target=3Dasyncore.loop, + kwargs=3D{'timeout':1}) + self._asyncore_thread.daemon =3D True + self._asyncore_thread.start() + + def handle_close(self): + """redirect close to base class""" + # Call the base class close, but not self.close() since + # handle_close() occurs in the context of the thread which + # self.close() attempts to join. + asyncore.dispatcher.close(self) + + def close(self): + """Close the base object and wait for the thread to terminate""" + if self._open: + self._open =3D False + asyncore.dispatcher.close(self) + if self._asyncore_thread is not None: + thread, self._asyncore_thread =3D self._asyncore_thread, N= one + thread.join() + if self._logfile: + self._logfile.close() + self._logfile =3D None + + def handle_read(self): + """process arriving characters into in memory _buffer""" + try: + data =3D asyncore.dispatcher.recv(self, 1) + # latin1 is needed since there are some chars + # we are receiving that cannot be encoded to utf-8 + # such as 0xe2, 0x80, 0xA6. + string =3D data.decode("latin1") + except: + print("Exception seen.") + traceback.print_exc() + return + if self._logfile: + self._logfile.write("{}".format(string)) + self._logfile.flush() + for c in string: + self._buffer.append(c) + + def recv(self, n=3D1): + """Return chars from in memory buffer""" + start_time =3D time.time() + while len(self._buffer) < n: + time.sleep(0.1) + elapsed_sec =3D time.time() - start_time + if elapsed_sec > self._recv_timeout_sec: + raise socket.timeout + chars =3D ''.join([self._buffer.popleft() for i in range(n)]) + # We choose to use latin1 to remain consistent with + # handle_read() and give back the same data as the user would + # receive if they were reading directly from the + # socket w/o our intervention. + return chars.encode("latin1") + + def set_blocking(self): + """Maintain compatibility with socket API""" + pass + + def settimeout(self, seconds): + """Set current timeout on recv""" + self._recv_timeout_sec =3D seconds + +class ByteBuffer(deque): + """Simple in memory buffer with read/write interface""" + def write(self, bytes): + for i in bytes: + self.append(i) + def read(self, n): + return ''.join([self.popleft() for i in range(n)]) + +if __name__ =3D=3D '__main__': + # Brief test to exercise the above code. + # The ConsoleSocket will ship some data to the server, + # the server will echo it back and the client will echo what it receiv= ed. + + # First remove the socket. + address =3D "./test_console_socket" + if os.path.exists(address): + os.unlink(address) + + # Create the server side. + server_socket =3D socket.socket(socket.AF_UNIX, socket.SOCK_STREAM) + server_socket.bind(address) + server_socket.listen(1) + + # Create the object we are trying to test. + console_socket =3D ConsoleSocket(address, file=3D"./logfile.txt") + + # Generate some data and ship it over the socket. + send_data =3D "" + for i in range(10): + send_data +=3D "this is a test message {}\n".format(i) + console_socket.send(send_data.encode('latin1')) + connection, client_address =3D server_socket.accept() + + # Process the data on the server and ship it back. + data =3D connection.recv(len(send_data)) + print("server received: {}".format(data)) + print("server: sending data back to the client") + connection.sendall(data) + + # Client receives teh bytes and displays them. + print("client: receiving bytes") + bytes =3D console_socket.recv(len(data)) + recv_data =3D bytes.decode('latin1') + print("client received: {}".format(recv_data)) + assert(recv_data =3D=3D send_data) + # Close console connection first, then close server. + console_socket.close() + connection.close() + server_socket.close() + print("test successful.") + diff --git a/python/qemu/machine.py b/python/qemu/machine.py index b9a98e2c86..30e7a16b92 100644 --- a/python/qemu/machine.py +++ b/python/qemu/machine.py @@ -24,6 +24,7 @@ import subprocess import shutil import socket import tempfile +from qemu.console_socket import ConsoleSocket =20 from . import qmp =20 @@ -71,7 +72,8 @@ class QEMUMachine(object): =20 def __init__(self, binary, args=3DNone, wrapper=3DNone, name=3DNone, test_dir=3D"/var/tmp", monitor_address=3DNone, - socket_scm_helper=3DNone, sock_dir=3DNone): + socket_scm_helper=3DNone, sock_dir=3DNone, + drain_console=3DFalse, console_log=3DNone): ''' Initialize a QEMUMachine =20 @@ -82,6 +84,9 @@ class QEMUMachine(object): @param test_dir: where to create socket and log file @param monitor_address: address for QMP monitor @param socket_scm_helper: helper program, required for send_fd_scm= () + @param sock_dir: where to create socket (overrides test_dir for so= ck) + @param console_log: (optional) path to console log file + @param drain_console: (optional) True to drain console socket to b= uffer @note: Qemu process is not started until launch() is used. ''' if args is None: @@ -118,6 +123,12 @@ class QEMUMachine(object): self._console_address =3D None self._console_socket =3D None self._remove_files =3D [] + self._console_log_path =3D console_log + if self._console_log_path: + # In order to log the console, buffering needs to be enabled. + self._drain_console =3D True + else: + self._drain_console =3D drain_console =20 # just in case logging wasn't configured by the main script: logging.basicConfig() @@ -568,7 +579,11 @@ class QEMUMachine(object): Returns a socket connected to the console """ if self._console_socket is None: - self._console_socket =3D socket.socket(socket.AF_UNIX, - socket.SOCK_STREAM) - self._console_socket.connect(self._console_address) + if self._drain_console: + self._console_socket =3D ConsoleSocket(self._console_addre= ss, + file=3Dself._console_= log_path) + else: + self._console_socket =3D socket.socket(socket.AF_UNIX, + socket.SOCK_STREAM) + self._console_socket.connect(self._console_address) return self._console_socket diff --git a/tests/vm/Makefile.include b/tests/vm/Makefile.include index 62502c13c2..942062a53b 100644 --- a/tests/vm/Makefile.include +++ b/tests/vm/Makefile.include @@ -49,6 +49,7 @@ endif @echo ' EXTRA_CONFIGURE_OPTS=3D"..."' @echo " J=3D[0..9]* - Override the -jN parameter for make= commands" @echo " DEBUG=3D1 - Enable verbose output on host and i= nteractive debugging" + @echo " LOG_CONSOLE=3D1 - Log console to file in: ~/.cache/qe= mu-vm " @echo " V=3D1 - Enable verbose ouput on host and guest commands" @echo " QEMU_LOCAL=3D1 - Use QEMU binary local to this= build." @echo " QEMU=3D/path/to/qemu - Change path to QEMU binary" @@ -72,6 +73,7 @@ $(IMAGES_DIR)/%.img: $(SRC_PATH)/tests/vm/% \ $(call quiet-command, \ $(PYTHON) $< \ $(if $(V)$(DEBUG), --debug) \ + $(if $(LOG_CONSOLE),--log-console) \ --build-path $(BUILD_DIR)\ --image "$@" \ --force \ @@ -87,6 +89,7 @@ vm-build-%: $(IMAGES_DIR)/%.img $(if $(DEBUG), --interactive) \ $(if $(J),--jobs $(J)) \ $(if $(V),--verbose) \ + $(if $(LOG_CONSOLE),--log-console) \ --build-path $(BUILD_DIR)\ --image "$<" \ $(if $(BUILD_TARGET),--build-target $(BUILD_TARGET)) \ @@ -109,6 +112,7 @@ vm-boot-ssh-%: $(IMAGES_DIR)/%.img $(PYTHON) $(SRC_PATH)/tests/vm/$* \ $(if $(J),--jobs $(J)) \ $(if $(V)$(DEBUG), --debug) \ + $(if $(LOG_CONSOLE),--log-console) \ --build-path $(BUILD_DIR)\ --image "$<" \ --interactive \ diff --git a/tests/vm/basevm.py b/tests/vm/basevm.py index bb20d8ca54..c71e059616 100644 --- a/tests/vm/basevm.py +++ b/tests/vm/basevm.py @@ -82,12 +82,14 @@ class BaseVM(object): poweroff =3D "poweroff" # enable IPv6 networking ipv6 =3D True + # This is the timeout on the wait for console bytes. + socket_timeout =3D 120 # Scale up some timeouts under TCG. # 4 is arbitrary, but greater than 2, # since we found we need to wait more than twice as long. tcg_ssh_timeout_multiplier =3D 4 def __init__(self, debug=3DFalse, vcpus=3DNone, config=3DNone, - build_path=3DNone): + build_path=3DNone, log_console=3DFalse): self._guest =3D None self._build_path =3D build_path # Allow input config to override defaults. @@ -113,6 +115,11 @@ class BaseVM(object): open(self._ssh_tmp_pub_key_file, "w").write(self._config['ssh_pub_key']) =20 + self._console_log_path =3D None + if log_console: + self._console_log_path =3D \ + os.path.join(os.path.expanduser("~/.cache/qemu-vm= "), + "{}.install.log".format(self.name)) self.debug =3D debug self._stderr =3D sys.stderr self._devnull =3D open(os.devnull, "w") @@ -268,7 +275,9 @@ class BaseVM(object): args +=3D self._data_args + extra_args + self._config['extra_args'] logging.debug("QEMU args: %s", " ".join(args)) qemu_path =3D get_qemu_path(self.arch, self._build_path) - guest =3D QEMUMachine(binary=3Dqemu_path, args=3Dargs) + guest =3D QEMUMachine(binary=3Dqemu_path, args=3Dargs, + console_log=3Dself._console_log_path, + drain_console=3DTrue) guest.set_machine(self._config['machine']) guest.set_console() try: @@ -282,6 +291,8 @@ class BaseVM(object): raise atexit.register(self.shutdown) self._guest =3D guest + # Init console so we can start consuming the chars. + self.console_init() usernet_info =3D guest.qmp("human-monitor-command", command_line=3D"info usernet") self.ssh_port =3D None @@ -293,7 +304,9 @@ class BaseVM(object): raise Exception("Cannot find ssh port from 'info usernet':\n%s= " % \ usernet_info) =20 - def console_init(self, timeout =3D 120): + def console_init(self, timeout =3D None): + if timeout =3D=3D None: + timeout =3D self.socket_timeout vm =3D self._guest vm.console_socket.settimeout(timeout) self.console_raw_path =3D os.path.join(vm._temp_dir, @@ -572,6 +585,8 @@ def parse_args(vmcls): "See config_example.yaml for example.") parser.add_option("--build-path", default=3DNone, help=3D"Path of build directory. ") + parser.add_option("--log-console", action=3D"store_true", + help=3D"Log console to file.") parser.disable_interspersed_args() return parser.parse_args() =20 @@ -587,7 +602,8 @@ def main(vmcls, config=3DNone): logging.basicConfig(level=3D(logging.DEBUG if args.debug else logging.WARN)) vm =3D vmcls(debug=3Dargs.debug, vcpus=3Dargs.jobs, config=3Dconfi= g, - build_path=3Dargs.build_path) + build_path=3Dargs.build_path, + log_console=3Dargs.log_console) if args.build_image: if os.path.exists(args.image) and not args.force: sys.stderr.writelines(["Image file exists: %s\n" % args.im= age, --=20 2.17.1