Hello,

This is the v12 of the patchset. Thank you very much for the
review of the v11 of the series.

We made changes to the following patches in this version:
  - Moved patches 18 & 19 in v11 to the front of the series based
    on feedback from Phil
  - [PATCH v12 02/19 ] multi-process: add configure and usage information
  - [PATCH v12 04/19 ] multi-process: Add config option for multi-process QEMU
  - [PATCH v12 08/19] multi-process: define MPQemuMsg format and transmission
    functions.

In summary, we replaced "scripts/mpqemu-launcher.py" with
"tests/multiprocess/multiprocess-lsi53c895a.py". We tested this test on
x86_64 and aarch64 architectures, which we have access to.

We changed the name of the config variable called CONFIG_MPQEMU
to CONFIG_MULTIPROCESS. We also moved the config variable
definition out of the "configure" script, and into the Kconfig
system. Previously, the user specified if multiprocess was enabled
using the "--enable-mpqemu" argument to the configure script.
In this version, we changed that. The multiprocess support is enabled
automatically if Kconfig system detects KVM support. This is needed
to run acceptance tests in the future.

We are working on acceptance tests (tests/acceptance/) for this
project. However, we have hit a roadblock and are working with the
avocado-devel community to resolve the issue.

We noticed that checkpatch.pl script flagged a warning for Patch 4 for
this series, but we don't believe that's a valid concern. We generated
the patches using QEMU's git orderfile
(git format-patch -O scripts/git.orderfile ...).

To touch upon the history of this project, we posted the Proof Of
Concept patches before the BoF session in 2018. Subsequently, we have
posted 11 versions on the qemu-devel mailing list. You can find them
by following the links below ([1] - [11]).Following people contributed
to the design and implementation of this project:
Jagannathan Raman <jag.raman@oracle.com>
Elena Ufimtseva <elena.ufimtseva@oracle.com>
John G Johnson <john.g.johnson@oracle.com>
Stefan Hajnoczi <stefanha@redhat.com>
Konrad Wilk <konrad.wilk@oracle.com>
Kanth Ghatraju <kanth.ghatraju@oracle.com>

We would like to thank QEMU community for your feedback in the
design and implementation of this project.Qemu wiki page:
https://wiki.qemu.org/Features/MultiProcessQEMU

For the full concept writeup about QEMU multi-process, please
refer to docs/devel/qemu-multiprocess.rst. Also see
docs/qemu-multiprocess.txt for usage information. We welcome
all your ideas, concerns, and questions for this patchset.

Thank you!

[POC]: https://www.mail-archive.com/qemu-devel@nongnu.org/msg566538.html
[1]: https://www.mail-archive.com/qemu-devel@nongnu.org/msg602285.html
[2]: https://www.mail-archive.com/qemu-devel@nongnu.org/msg624877.html
[3]: https://www.mail-archive.com/qemu-devel@nongnu.org/msg642000.html
[4]: https://www.mail-archive.com/qemu-devel@nongnu.org/msg655118.html
[5]: https://www.mail-archive.com/qemu-devel@nongnu.org/msg682429.html
[6]: https://www.mail-archive.com/qemu-devel@nongnu.org/msg697484.html
[7]: https://patchew.org/QEMU/cover.1593273671.git.elena.ufimtseva@oracle.com/
[8]: https://www.mail-archive.com/qemu-devel@nongnu.org/msg727007.html
[9]: https://www.mail-archive.com/qemu-devel@nongnu.org/msg734275.html
[10]: https://www.mail-archive.com/qemu-devel@nongnu.org/msg747638.html
[11]: https://www.mail-archive.com/qemu-devel@nongnu.org/msg750972.htmlThank you!


Elena Ufimtseva (7):
  multi-process: add configure and usage information
  multi-process: add qio channel function to transmit data and fds
  multi-process: define MPQemuMsg format and transmission functions
  multi-process: introduce proxy object
  multi-process: add proxy communication functions
  multi-process: Forward PCI config space acceses to the remote process
  multi-process: perform device reset in the remote process

Jagannathan Raman (11):
  memory: alloc RAM from file at offset
  multi-process: Add config option for multi-process QEMU
  multi-process: setup PCI host bridge for remote device
  multi-process: setup a machine object for remote device process
  multi-process: Initialize message handler in remote device
  multi-process: Associate fd of a PCIDevice with its object
  multi-process: setup memory manager for remote device
  multi-process: PCI BAR read/write handling for proxy & remote
    endpoints
  multi-process: Synchronize remote memory
  multi-process: create IOHUB object to handle irq
  multi-process: Retrieve PCI info from remote process

John G Johnson (1):
  multi-process: add the concept description to
    docs/devel/qemu-multiprocess

 docs/devel/index.rst                          |   1 +
 docs/devel/multi-process.rst                  | 966 ++++++++++++++++++++++++++
 docs/multi-process.rst                        |  66 ++
 include/exec/memory.h                         |   2 +
 include/exec/ram_addr.h                       |   2 +-
 include/hw/pci-host/remote.h                  |  31 +
 include/hw/pci/pci_ids.h                      |   3 +
 include/hw/remote/iohub.h                     |  42 ++
 include/hw/remote/machine.h                   |  40 ++
 include/hw/remote/memory-sync.h               |  27 +
 include/hw/remote/memory.h                    |  19 +
 include/hw/remote/mpqemu-link.h               |  98 +++
 include/hw/remote/proxy.h                     |  53 ++
 include/hw/remote/remote-obj.h                |  42 ++
 include/io/channel.h                          |  24 +
 include/qemu/mmap-alloc.h                     |   3 +-
 backends/hostmem-memfd.c                      |   2 +-
 hw/misc/ivshmem.c                             |   3 +-
 hw/pci-host/remote.c                          |  75 ++
 hw/remote/iohub.c                             | 123 ++++
 hw/remote/machine.c                           |  79 +++
 hw/remote/memory-sync.c                       | 210 ++++++
 hw/remote/memory.c                            |  58 ++
 hw/remote/message.c                           | 241 +++++++
 hw/remote/mpqemu-link.c                       | 308 ++++++++
 hw/remote/proxy.c                             | 378 ++++++++++
 hw/remote/remote-obj.c                        | 154 ++++
 io/channel.c                                  |  45 ++
 softmmu/memory.c                              |   3 +-
 softmmu/physmem.c                             |  11 +-
 util/mmap-alloc.c                             |   7 +-
 util/oslib-posix.c                            |   2 +-
 MAINTAINERS                                   |  25 +
 accel/Kconfig                                 |   1 +
 hw/Kconfig                                    |   1 +
 hw/meson.build                                |   1 +
 hw/pci-host/Kconfig                           |   3 +
 hw/pci-host/meson.build                       |   1 +
 hw/remote/Kconfig                             |   4 +
 hw/remote/meson.build                         |  13 +
 tests/multiprocess/multiprocess-lsi53c895a.py |  92 +++
 41 files changed, 3246 insertions(+), 13 deletions(-)
 create mode 100644 docs/devel/multi-process.rst
 create mode 100644 docs/multi-process.rst
 create mode 100644 include/hw/pci-host/remote.h
 create mode 100644 include/hw/remote/iohub.h
 create mode 100644 include/hw/remote/machine.h
 create mode 100644 include/hw/remote/memory-sync.h
 create mode 100644 include/hw/remote/memory.h
 create mode 100644 include/hw/remote/mpqemu-link.h
 create mode 100644 include/hw/remote/proxy.h
 create mode 100644 include/hw/remote/remote-obj.h
 create mode 100644 hw/pci-host/remote.c
 create mode 100644 hw/remote/iohub.c
 create mode 100644 hw/remote/machine.c
 create mode 100644 hw/remote/memory-sync.c
 create mode 100644 hw/remote/memory.c
 create mode 100644 hw/remote/message.c
 create mode 100644 hw/remote/mpqemu-link.c
 create mode 100644 hw/remote/proxy.c
 create mode 100644 hw/remote/remote-obj.c
 create mode 100644 hw/remote/Kconfig
 create mode 100644 hw/remote/meson.build
 create mode 100755 tests/multiprocess/multiprocess-lsi53c895a.py

-- 
1.8.3.1

On Tue, Dec 01, 2020 at 03:22:35PM -0500, Jagannathan Raman wrote:
> This is the v12 of the patchset. Thank you very much for the
> review of the v11 of the series.

I'm in favor of merging this for QEMU 6.0. The command-line interface
has the x- prefix so QEMU is not committing to a stable interface.
Changes needed to support additional device types or to switch to the
vfio-user protocol can be made later.

Jag, Elena, JJ: I suggest getting your GPG key to Peter Maydell so you
can send multi-process QEMU pull requests.

Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>

On Thu, Dec 03, 2020 at 09:14:04AM +0000, Stefan Hajnoczi wrote:
> On Tue, Dec 01, 2020 at 03:22:35PM -0500, Jagannathan Raman wrote:
> > This is the v12 of the patchset. Thank you very much for the
> > review of the v11 of the series.
> 
> I'm in favor of merging this for QEMU 6.0. The command-line interface
> has the x- prefix so QEMU is not committing to a stable interface.
> Changes needed to support additional device types or to switch to the
> vfio-user protocol can be made later.
> 

Woot! Thank you Stefan!

> Jag, Elena, JJ: I suggest getting your GPG key to Peter Maydell so you
> can send multi-process QEMU pull requests.
> 
> Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>

In progress.
Do we need to add some tagging for the PULL patches?
Should we include the git repo and have the proper tag as well?

Elena

On Thu, 3 Dec 2020 at 09:51, Stefan Hajnoczi <stefanha@redhat.com> wrote:
>
> On Tue, Dec 01, 2020 at 03:22:35PM -0500, Jagannathan Raman wrote:
> > This is the v12 of the patchset. Thank you very much for the
> > review of the v11 of the series.
>
> I'm in favor of merging this for QEMU 6.0. The command-line interface
> has the x- prefix so QEMU is not committing to a stable interface.
> Changes needed to support additional device types or to switch to the
> vfio-user protocol can be made later.
>
> Jag, Elena, JJ: I suggest getting your GPG key to Peter Maydell so you
> can send multi-process QEMU pull requests.

I would prefer to see this going through the tree of an
established QEMU developer who's already sending pullrequests,
at least initially.

thanks
-- PMM

On Thu, Dec 03, 2020 at 08:40:11PM +0000, Peter Maydell wrote:
> On Thu, 3 Dec 2020 at 09:51, Stefan Hajnoczi <stefanha@redhat.com> wrote:
> >
> > On Tue, Dec 01, 2020 at 03:22:35PM -0500, Jagannathan Raman wrote:
> > > This is the v12 of the patchset. Thank you very much for the
> > > review of the v11 of the series.
> >
> > I'm in favor of merging this for QEMU 6.0. The command-line interface
> > has the x- prefix so QEMU is not committing to a stable interface.
> > Changes needed to support additional device types or to switch to the
> > vfio-user protocol can be made later.
> >
> > Jag, Elena, JJ: I suggest getting your GPG key to Peter Maydell so you
> > can send multi-process QEMU pull requests.
> 
> I would prefer to see this going through the tree of an
> established QEMU developer who's already sending pullrequests,
> at least initially.

Once the discussion has completed I can send the patches in a pull
request.

I don't want to be the bottleneck for all multi-process QEMU patches in
the future though. That's why I think the authors should be able to send
pull requests on their own after the initial code is merged. Much of
this work is isolated an only affects multi-process QEMU and the feature
is marked experimental. There is little risk of introducing instability
for non-multi-process QEMU users/developers. Hence why this is a new
subsystem and has MAINTAINERS files entries.

Does that sound good?

Stefan

On Thu, 10 Dec 2020 at 11:14, Stefan Hajnoczi <stefanha@redhat.com> wrote:
> On Thu, Dec 03, 2020 at 08:40:11PM +0000, Peter Maydell wrote:
> > I would prefer to see this going through the tree of an
> > established QEMU developer who's already sending pullrequests,
> > at least initially.
>
> Once the discussion has completed I can send the patches in a pull
> request.
>
> I don't want to be the bottleneck for all multi-process QEMU patches in
> the future though. That's why I think the authors should be able to send
> pull requests on their own after the initial code is merged. Much of
> this work is isolated an only affects multi-process QEMU and the feature
> is marked experimental. There is little risk of introducing instability
> for non-multi-process QEMU users/developers. Hence why this is a new
> subsystem and has MAINTAINERS files entries.

My reasoning is basically that new pull-request senders are more
work for me, because I have to make sure they have a GPG key set
up, and then examine pull requests pretty carefully to check they're
well-formed, all the sign-offs are correct, the changes aren't
touching areas of the codebase that they shouldn't, and so on.
That's particularly painful if the first pull request that comes
through is a massive one rather than "here's a small number of
patches with some bug fixes".

thanks
-- PMM

On Thu, Dec 10, 2020 at 11:24:46AM +0000, Peter Maydell wrote:
> On Thu, 10 Dec 2020 at 11:14, Stefan Hajnoczi <stefanha@redhat.com> wrote:
> > On Thu, Dec 03, 2020 at 08:40:11PM +0000, Peter Maydell wrote:
> > > I would prefer to see this going through the tree of an
> > > established QEMU developer who's already sending pullrequests,
> > > at least initially.
> >
> > Once the discussion has completed I can send the patches in a pull
> > request.
> >
> > I don't want to be the bottleneck for all multi-process QEMU patches in
> > the future though. That's why I think the authors should be able to send
> > pull requests on their own after the initial code is merged. Much of
> > this work is isolated an only affects multi-process QEMU and the feature
> > is marked experimental. There is little risk of introducing instability
> > for non-multi-process QEMU users/developers. Hence why this is a new
> > subsystem and has MAINTAINERS files entries.
> 
> My reasoning is basically that new pull-request senders are more
> work for me, because I have to make sure they have a GPG key set
> up, and then examine pull requests pretty carefully to check they're
> well-formed, all the sign-offs are correct, the changes aren't
> touching areas of the codebase that they shouldn't, and so on.
> That's particularly painful if the first pull request that comes
> through is a massive one rather than "here's a small number of
> patches with some bug fixes".

Thanks for explaining. I will merge this series when review has
finished and send you a pull request.

Stefan