From nobody Tue Feb 10 10:03:38 2026 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=oracle.com ARC-Seal: i=1; a=rsa-sha256; t=1586166751; cv=none; d=zohomail.com; s=zohoarc; b=bgABe9wleuPkuM6MCQfae/A1aQmc7wUnIex90bvuWgWgBjR6V4IjpAuhk4GaxJIix7n9GbTHO8miulRWKO9W/pPYs7guiFlfKlXkV6tEhubNI5glOl5jfpy9F+QQbSokfN1xk4kL1esahgh6RWX9GfK0Ya6SGHnrXw/O++4if54= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1586166751; h=Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=ZocP0cSRsfVsVQd13v1loLCOLD58vhqoOPrPurUZea8=; b=LGZuevRZA+SxLmd4bmdVs2uX8PKteJLBHEE4rdzq3g5ZnzcaiS81pSECW52OdzgJunoI18DQW2zQXAI+4sXoQulYGJ36WBGfmCxJjPA2KI6Vs0pVE0BOD2ivePPEVr4/rOTxYWNChKbsctfn+aVWU+4gUpiK9yT4NMrqQCn2R4w= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1586166751623416.5584203713081; Mon, 6 Apr 2020 02:52:31 -0700 (PDT) Received: from localhost ([::1]:57492 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jLOQf-0003l7-SW for importer@patchew.org; Mon, 06 Apr 2020 05:52:29 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:44560) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jLOHG-0003OY-NJ for qemu-devel@nongnu.org; Mon, 06 Apr 2020 05:42:48 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1jLOHF-0002fx-02 for qemu-devel@nongnu.org; Mon, 06 Apr 2020 05:42:46 -0400 Received: from userp2130.oracle.com ([156.151.31.86]:35426) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1jLOHE-0002fG-Ne for qemu-devel@nongnu.org; Mon, 06 Apr 2020 05:42:44 -0400 Received: from pps.filterd (userp2130.oracle.com [127.0.0.1]) by userp2130.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 0369dpr4089419; Mon, 6 Apr 2020 09:42:39 GMT Received: from aserp3020.oracle.com (aserp3020.oracle.com [141.146.126.70]) by userp2130.oracle.com with ESMTP id 306hnqwu13-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 06 Apr 2020 09:42:39 +0000 Received: from pps.filterd (aserp3020.oracle.com [127.0.0.1]) by aserp3020.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 0369fOcd066204; Mon, 6 Apr 2020 09:42:38 GMT Received: from userv0122.oracle.com (userv0122.oracle.com [156.151.31.75]) by aserp3020.oracle.com with ESMTP id 307419xjct-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 06 Apr 2020 09:42:38 +0000 Received: from abhmp0016.oracle.com (abhmp0016.oracle.com [141.146.116.22]) by userv0122.oracle.com (8.14.4/8.14.4) with ESMTP id 0369gZUS013926; Mon, 6 Apr 2020 09:42:35 GMT Received: from flaka.hsd1.ca.comcast.net (/67.180.143.163) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Mon, 06 Apr 2020 02:42:31 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=corp-2020-01-29; bh=ZocP0cSRsfVsVQd13v1loLCOLD58vhqoOPrPurUZea8=; b=UQNgX5m4gNp0FqT+IzHkFtyE93PSSnJDUgg5lMiOMLXTb2gaElXaYI+ziNjvaAp3C9lP l1nOrhfBVigBe7GGFcUv0KN4wzmiTCSAJVBOzo2FNBRg/TkRY1h8whlcuTbflAsYWlgr SSr27hRZ74x8YNrmh+rI891jSFC4G+sjUe9e5r+HuCrBWSYQcHETxqt0AYb8dbfImMqF odf6hl6HRgYS7bFStsMqrUJIiiQz+Wr0b5zgKxV7UpEVqpRbmiJpqzz1vipShUHjiijU U0WPzeKtB++DemJe9L3uK4xEFwmKtgJXkNXjKDhRMAalVmA0H+jI3sBhE7FwZj8KTT+a zQ== From: elena.ufimtseva@oracle.com To: qemu-devel@nongnu.org Subject: [PATCH v6 36/36] multi-process: add configure and usage information Date: Mon, 6 Apr 2020 02:41:26 -0700 Message-Id: X-Mailer: git-send-email 2.25.GIT In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Proofpoint-Virus-Version: vendor=nai engine=6000 definitions=9582 signatures=668685 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=1 phishscore=0 malwarescore=0 bulkscore=0 spamscore=0 adultscore=0 mlxlogscore=999 mlxscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2003020000 definitions=main-2004060084 X-Proofpoint-Virus-Version: vendor=nai engine=6000 definitions=9582 signatures=668685 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 bulkscore=0 phishscore=0 adultscore=0 priorityscore=1501 mlxscore=0 malwarescore=0 mlxlogscore=999 lowpriorityscore=0 spamscore=0 impostorscore=0 suspectscore=1 clxscore=1015 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2003020000 definitions=main-2004060083 X-detected-operating-system: by eggs.gnu.org: GNU/Linux 3.x [generic] [fuzzy] X-Received-From: 156.151.31.86 X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: elena.ufimtseva@oracle.com, fam@euphon.net, swapnil.ingle@nutanix.com, john.g.johnson@oracle.com, kraxel@redhat.com, jag.raman@oracle.com, quintela@redhat.com, mst@redhat.com, armbru@redhat.com, kanth.ghatraju@oracle.com, felipe@nutanix.com, thuth@redhat.com, ehabkost@redhat.com, konrad.wilk@oracle.com, dgilbert@redhat.com, liran.alon@oracle.com, stefanha@redhat.com, thanos.makatos@nutanix.com, rth@twiddle.net, kwolf@redhat.com, berrange@redhat.com, mreitz@redhat.com, ross.lagerwall@citrix.com, marcandre.lureau@gmail.com, pbonzini@redhat.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) Content-Type: text/plain; charset="utf-8" From: Elena Ufimtseva Signed-off-by: Elena Ufimtseva Signed-off-by: Jagannathan Raman Signed-off-by: John G Johnson --- docs/multi-process.rst | 85 +++++++++++++++++++++++++ scripts/mpqemu-launcher-perf-mode.py | 93 ++++++++++++++++++++++++++++ scripts/mpqemu-launcher.py | 54 ++++++++++++++++ 3 files changed, 232 insertions(+) create mode 100644 docs/multi-process.rst create mode 100755 scripts/mpqemu-launcher-perf-mode.py create mode 100755 scripts/mpqemu-launcher.py diff --git a/docs/multi-process.rst b/docs/multi-process.rst new file mode 100644 index 0000000000..9d53667b4e --- /dev/null +++ b/docs/multi-process.rst @@ -0,0 +1,85 @@ +Multi-process QEMU +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + +This document describes how to configure and use multi-process qemu. +For the design document refer to docs/devel/qemu-multiprocess. + +1) Configuration +---------------- + +To enable support for multi-process add --enable-mpqemu +to the list of options for the "configure" script. + + +2) Usage +-------- + +Multi-process QEMU requires an orchestrator to launch. Please refer to a +light-weight python based orchestrator for mpqemu in +scripts/mpqemu-launcher.py to lauch QEMU in multi-process mode. + +scripts/mpqemu-launcher-perf-mode.py launches in "perf" mode. In this mode, +the same QEMU process connects to multiple remote devices, each emulated in +a separate process. + +As of now, we only support the emulation of lsi53c895a in a separate proce= ss. + +Following is a description of command-line used to launch mpqemu. + +* Orchestrator: + + - The Orchestrator creates a unix socketpair + + - It launches the remote process and passes one of the + sockets to it via command-line. + + - It then launches QEMU and specifies the other socket as an option + to the Proxy device object + +* Remote Process: + + - The first command-line option in the remote process is one of the + sockets created by the Orchestrator + + - The remaining options are no different from how one launches QEMU with + devices. The only other requirement is each PCI device must have a + unique ID specified to it. This is needed to pair remote device with t= he + Proxy object. + + - Example command-line for the remote process is as follows: + + /usr/bin/qemu-scsu-dev 4 \ + -device lsi53c895a,id=3Dlsi0 = \ + -drive id=3Ddrive_image2,file=3D/build/ol7-nvme-test-1.qcow2 = \ + -device scsi-hd,id=3Ddrive2,drive=3Ddrive_image2,bus=3Dlsi0.0,scsi-i= d=3D0 + +* QEMU: + + - Since parts of the RAM are shared between QEMU & remote process, a + memory-backend-file is required to facilitate this, as follows: + + -object memory-backend-file,id=3Dmem,mem-path=3D/dev/shm/,size=3D4096M= ,share=3Don + + - A "pci-proxy-dev" device is created for each of the PCI devices emulat= ed + in the remote process. A "socket" sub-option specifies the other end of + unix channel created by orchestrator. The "id" sub-option must be spec= ified + and should be the same as the "id" specified for the remote PCI device + + - Example commandline for QEMU is as follows: + + -device pci-proxy-dev,id=3Dlsi0,socket=3D3 + +* Monitor / QMP: + + - The remote process supports QEMU monitor. It could be specified using = the + "-monitor" or "-qmp" command-line options + + - As an example, one could connect to the monitor by adding the following + to the command-line of the remote process + + -monitor unix:/home/qmp-sock,server,nowait + + - The user could connect to the monitor using the qmp script or using + "socat" as outlined below: + + socat /home/qmp-sock stdio diff --git a/scripts/mpqemu-launcher-perf-mode.py b/scripts/mpqemu-launcher= -perf-mode.py new file mode 100755 index 0000000000..a9be82faf5 --- /dev/null +++ b/scripts/mpqemu-launcher-perf-mode.py @@ -0,0 +1,93 @@ +#!/usr/bin/env python3 + +import socket +import os +import subprocess +import time + +PROC_QEMU=3D'/usr/bin/qemu-system-x86_64' + +PROC_REMOTE=3D'/usr/bin/qemu-scsi-dev' + +proxy_1, remote_1 =3D socket.socketpair(socket.AF_UNIX, socket.SOCK_STREAM) +proxy_2, remote_2 =3D socket.socketpair(socket.AF_UNIX, socket.SOCK_STREAM) +proxy_3, remote_3 =3D socket.socketpair(socket.AF_UNIX, socket.SOCK_STREAM) + +remote_cmd_1 =3D [ PROC_REMOTE, = \ + str(remote_1.fileno()), = \ + '-device', 'lsi53c895a,id=3Dlsi1', = \ + '-drive', 'id=3Ddrive_image1,' = \ + 'file=3D/build/ol7-nvme-test-1.qcow2', = \ + '-device', 'scsi-hd,id=3Ddrive1,drive=3Ddrive_image1,' = \ + 'bus=3Dlsi1.0,scsi-id=3D0', = \ + ] + +remote_cmd_2 =3D [ PROC_REMOTE, = \ + str(remote_2.fileno()), = \ + '-device', 'lsi53c895a,id=3Dlsi2', = \ + '-drive', 'id=3Ddrive_image2,' = \ + 'file=3D/build/ol7-nvme-test-2.qcow2', = \ + '-device', 'scsi-hd,id=3Ddrive2,drive=3Ddrive_image2,' = \ + 'bus=3Dlsi2.0,scsi-id=3D0' = \ + ] + +remote_cmd_3 =3D [ PROC_REMOTE, = \ + str(remote_3.fileno()), = \ + '-device', 'lsi53c895a,id=3Dlsi3', = \ + '-drive', 'id=3Ddrive_image3,' = \ + 'file=3D/build/ol7-nvme-test-3.qcow2', = \ + '-device', 'scsi-hd,id=3Ddrive3,drive=3Ddrive_image3,' = \ + 'bus=3Dlsi3.0,scsi-id=3D0' = \ + ] + +proxy_cmd =3D [ PROC_QEMU, = \ + '-name', 'OL7.4', = \ + '-machine', 'q35,accel=3Dkvm', = \ + '-smp', 'sockets=3D1,cores=3D1,threads=3D1', = \ + '-m', '2048', = \ + '-object', 'memory-backend-file,id=3Dsysmem-file,' = \ + 'mem-path=3D/dev/shm/test-mem,size=3D2G,share= =3Don', \ + '-numa', 'node,memdev=3Dsysmem-file', = \ + '-device', 'virtio-scsi-pci,id=3Dvirtio_scsi_pci0', = \ + '-drive', 'id=3Ddrive_image1,if=3Dnone,format=3Dqcow2,' = \ + 'file=3D/home/ol7-hdd-1.qcow2', = \ + '-device', 'scsi-hd,id=3Dimage1,drive=3Ddrive_image1,' = \ + 'bus=3Dvirtio_scsi_pci0.0', = \ + '-boot', 'd', = \ + '-vnc', ':0', = \ + '-device', 'pci-proxy-dev,id=3Dlsi1,' = \ + 'socket=3D'+str(proxy_1.fileno()), = \ + '-device', 'pci-proxy-dev,id=3Dlsi2,' = \ + 'socket=3D'+str(proxy_2.fileno()), = \ + '-device', 'pci-proxy-dev,id=3Dlsi3,' = \ + 'socket=3D'+str(proxy_3.fileno()) = \ + ] + + +pid =3D os.fork(); +if pid =3D=3D 0: + # In remote_1 + print('Launching Remote process 1'); + process =3D subprocess.Popen(remote_cmd_1, pass_fds=3D[remote_1.fileno= ()]) + os._exit(0) + + +pid =3D os.fork(); +if pid =3D=3D 0: + # In remote_2 + print('Launching Remote process 2'); + process =3D subprocess.Popen(remote_cmd_2, pass_fds=3D[remote_2.fileno= ()]) + os._exit(0) + + +pid =3D os.fork(); +if pid =3D=3D 0: + # In remote_3 + print('Launching Remote process 3'); + process =3D subprocess.Popen(remote_cmd_3, pass_fds=3D[remote_3.fileno= ()]) + os._exit(0) + + +print('Launching Proxy process'); +process =3D subprocess.Popen(proxy_cmd, pass_fds=3D[proxy_1.fileno(), = \ + proxy_2.fileno(), proxy_3.fileno()]) diff --git a/scripts/mpqemu-launcher.py b/scripts/mpqemu-launcher.py new file mode 100755 index 0000000000..28d6aa441f --- /dev/null +++ b/scripts/mpqemu-launcher.py @@ -0,0 +1,54 @@ +#!/usr/bin/env python3 +import socket +import os +import subprocess +import time + +PROC_QEMU=3D'/usr/bin/qemu-system-x86_64' + +PROC_REMOTE=3D'/usr/bin/qemu-scsi-dev' + +proxy, remote =3D socket.socketpair(socket.AF_UNIX, socket.SOCK_STREAM) + +remote_cmd =3D [ PROC_REMOTE, = \ + str(remote.fileno()), = \ + '-device', 'lsi53c895a,id=3Dlsi1', = \ + '-drive', 'id=3Ddrive_image1,file=3D/build/ol7-nvme-test-1.= qcow2', \ + '-device', 'scsi-hd,id=3Ddrive1,drive=3Ddrive_image1,bus=3D= lsi1.0,' \ + 'scsi-id=3D0', = \ + '-device', 'lsi53c895a,id=3Dlsi2', = \ + '-drive', 'id=3Ddrive_image2,file=3D/build/ol7-nvme-test-2.= qcow2', \ + '-device', 'scsi-hd,id=3Ddrive2,drive=3Ddrive_image2,bus=3D= lsi2.0,' \ + 'scsi-id=3D0' = \ + ] + +proxy_cmd =3D [ PROC_QEMU, = \ + '-name', 'OL7.4', = \ + '-machine', 'q35,accel=3Dkvm', = \ + '-smp', 'sockets=3D1,cores=3D1,threads=3D1', = \ + '-m', '2048', = \ + '-object', 'memory-backend-file,id=3Dsysmem-file,' = \ + 'mem-path=3D/dev/shm/test-mem,size=3D2G,share= =3Don', \ + '-numa', 'node,memdev=3Dsysmem-file', = \ + '-device', 'virtio-scsi-pci,id=3Dvirtio_scsi_pci0', = \ + '-drive', 'id=3Ddrive_image1,if=3Dnone,format=3Dqcow2,' = \ + 'file=3D/home/ol7-hdd-1.qcow2', = \ + '-device', 'scsi-hd,id=3Dimage1,drive=3Ddrive_image1,' = \ + 'bus=3Dvirtio_scsi_pci0.0', = \ + '-boot', 'd', = \ + '-vnc', ':0', = \ + '-device', 'pci-proxy-dev,id=3Dlsi1,socket=3D'+str(proxy.fil= eno()), \ + '-device', 'pci-proxy-dev,id=3Dlsi2,socket=3D'+str(proxy.fil= eno()) \ + ] + + +pid =3D os.fork(); + +if pid: + # In Proxy + print('Launching QEMU with Proxy object'); + process =3D subprocess.Popen(proxy_cmd, pass_fds=3D[proxy.fileno()]) +else: + # In remote + print('Launching Remote process'); + process =3D subprocess.Popen(remote_cmd, pass_fds=3D[remote.fileno()]) --=20 2.25.GIT