[v2] docs/s390x/pcidevices: document pci devices on s390x

[PATCH v2] docs/s390x/pcidevices: document pci devices on s390x

Posted by Sebastian Mitterle 1 year, 3 months ago

Add some documentation about the zpci device and how
to use it with pci devices on s390x.

Used source: Cornelia Huck's blog post
https://people.redhat.com/~cohuck/2018/02/19/notes-on-pci-on-s390x.html

Signed-off-by: Sebastian Mitterle <smitterl@redhat.com>
---
v2: move section below 'Device support'
---
 docs/system/s390x/pcidevices.rst | 34 ++++++++++++++++++++++++++++++++
 docs/system/target-s390x.rst     |  1 +
 2 files changed, 35 insertions(+)
 create mode 100644 docs/system/s390x/pcidevices.rst

diff --git a/docs/system/s390x/pcidevices.rst b/docs/system/s390x/pcidevices.rst
new file mode 100644
index 0000000000..fec905d6e6
--- /dev/null
+++ b/docs/system/s390x/pcidevices.rst
@@ -0,0 +1,34 @@
+PCI devices on s390x
+====================
+
+PCI devices on s390x work differently than on other architectures.
+
+To start with, using a PCI device requires the additional ``zpci`` device. For example,
+in order to pass a PCI device ``0000:00:00.0`` through you'd specify::
+
+ qemu-system-s390x ... \
+                   -device zpci,uid=1,fid=0,target=hostdev0,id=zpci1 \
+                   -device vfio-pci,host=0000:00:00.0,id=hostdev0
+
+Here, the zpci device is joined with the PCI device via the ``target`` property.
+
+Note that we don't set bus, slot or function here for the guest as is common in other
+PCI implementations. Topology information is not available on s390x. Instead, ``uid``
+and ``fid`` determine how the device is presented to the guest operating system.
+
+In case of Linux, ``uid`` will be used in the ``domain`` part of the PCI identifier, and
+``fid`` identifies the physical slot, i.e.::
+
+ qemu-system-s390x ... \
+                   -device zpci,uid=7,fid=8,target=hostdev0,id=zpci1 \
+                   ...
+
+will be presented in the guest as::
+
+ # lspci -v
+ 0007:00:00.0 ...
+ Physical Slot: 00000008
+ ...
+
+Finally, note that you might have to enable the ``zpci`` feature in the cpu model in oder to use
+it.
diff --git a/docs/system/target-s390x.rst b/docs/system/target-s390x.rst
index c636f64113..f6f11433c7 100644
--- a/docs/system/target-s390x.rst
+++ b/docs/system/target-s390x.rst
@@ -26,6 +26,7 @@ or vfio-ap is also available.
    s390x/css
    s390x/3270
    s390x/vfio-ccw
+   s390x/pcidevices
 
 Architectural features
 ======================
-- 
2.37.3

Re: [PATCH v2] docs/s390x/pcidevices: document pci devices on s390x

Posted by Cédric Le Goater 1 year, 3 months ago

On 1/27/23 09:46, Sebastian Mitterle wrote:
> Add some documentation about the zpci device and how
> to use it with pci devices on s390x.
> 
> Used source: Cornelia Huck's blog post
> https://people.redhat.com/~cohuck/2018/02/19/notes-on-pci-on-s390x.html
> 
> Signed-off-by: Sebastian Mitterle <smitterl@redhat.com>


Reviewed-by: Cédric Le Goater <clg@redhat.com>

Thanks,

C.


> ---
> v2: move section below 'Device support'
> ---
>   docs/system/s390x/pcidevices.rst | 34 ++++++++++++++++++++++++++++++++
>   docs/system/target-s390x.rst     |  1 +
>   2 files changed, 35 insertions(+)
>   create mode 100644 docs/system/s390x/pcidevices.rst
> 
> diff --git a/docs/system/s390x/pcidevices.rst b/docs/system/s390x/pcidevices.rst
> new file mode 100644
> index 0000000000..fec905d6e6
> --- /dev/null
> +++ b/docs/system/s390x/pcidevices.rst
> @@ -0,0 +1,34 @@
> +PCI devices on s390x
> +====================
> +
> +PCI devices on s390x work differently than on other architectures.
> +
> +To start with, using a PCI device requires the additional ``zpci`` device. For example,
> +in order to pass a PCI device ``0000:00:00.0`` through you'd specify::
> +
> + qemu-system-s390x ... \
> +                   -device zpci,uid=1,fid=0,target=hostdev0,id=zpci1 \
> +                   -device vfio-pci,host=0000:00:00.0,id=hostdev0
> +
> +Here, the zpci device is joined with the PCI device via the ``target`` property.
> +
> +Note that we don't set bus, slot or function here for the guest as is common in other
> +PCI implementations. Topology information is not available on s390x. Instead, ``uid``
> +and ``fid`` determine how the device is presented to the guest operating system.
> +
> +In case of Linux, ``uid`` will be used in the ``domain`` part of the PCI identifier, and
> +``fid`` identifies the physical slot, i.e.::
> +
> + qemu-system-s390x ... \
> +                   -device zpci,uid=7,fid=8,target=hostdev0,id=zpci1 \
> +                   ...
> +
> +will be presented in the guest as::
> +
> + # lspci -v
> + 0007:00:00.0 ...
> + Physical Slot: 00000008
> + ...
> +
> +Finally, note that you might have to enable the ``zpci`` feature in the cpu model in oder to use
> +it.
> diff --git a/docs/system/target-s390x.rst b/docs/system/target-s390x.rst
> index c636f64113..f6f11433c7 100644
> --- a/docs/system/target-s390x.rst
> +++ b/docs/system/target-s390x.rst
> @@ -26,6 +26,7 @@ or vfio-ap is also available.
>      s390x/css
>      s390x/3270
>      s390x/vfio-ccw
> +   s390x/pcidevices
>   
>   Architectural features
>   ======================

Re: [PATCH v2] docs/s390x/pcidevices: document pci devices on s390x

Posted by Cornelia Huck 1 year, 3 months ago

On Fri, Jan 27 2023, Sebastian Mitterle <smitterl@redhat.com> wrote:

> Add some documentation about the zpci device and how
> to use it with pci devices on s390x.
>
> Used source: Cornelia Huck's blog post
> https://people.redhat.com/~cohuck/2018/02/19/notes-on-pci-on-s390x.html
>
> Signed-off-by: Sebastian Mitterle <smitterl@redhat.com>
> ---
> v2: move section below 'Device support'
> ---
>  docs/system/s390x/pcidevices.rst | 34 ++++++++++++++++++++++++++++++++
>  docs/system/target-s390x.rst     |  1 +
>  2 files changed, 35 insertions(+)
>  create mode 100644 docs/system/s390x/pcidevices.rst
>
> diff --git a/docs/system/s390x/pcidevices.rst b/docs/system/s390x/pcidevices.rst
> new file mode 100644
> index 0000000000..fec905d6e6
> --- /dev/null
> +++ b/docs/system/s390x/pcidevices.rst
> @@ -0,0 +1,34 @@
> +PCI devices on s390x
> +====================
> +
> +PCI devices on s390x work differently than on other architectures.

add "and need to be configured in a slightly different way." ?

> +
> +To start with, using a PCI device requires the additional ``zpci`` device. For example,

I think the "zpci" device is not technically "required" (ISTR that we
autogenerate it, if needed); however, you need it if you actually want
to specify uid/fid/... what about:

"Every PCI device is linked with an additional ``zpci`` device. While
the ``zpci`` device will be autogenerated if not specified, it is
recommended to specify it explicitly so that you can pass s390-specific
PCI configuration."

?

> +in order to pass a PCI device ``0000:00:00.0`` through you'd specify::
> +
> + qemu-system-s390x ... \
> +                   -device zpci,uid=1,fid=0,target=hostdev0,id=zpci1 \
> +                   -device vfio-pci,host=0000:00:00.0,id=hostdev0
> +
> +Here, the zpci device is joined with the PCI device via the ``target`` property.
> +
> +Note that we don't set bus, slot or function here for the guest as is common in other
> +PCI implementations. Topology information is not available on s390x. Instead, ``uid``

"Topology information is not available on s390x, and the guest will not
see any of the bus/slot/function information specified on the command
line."

?

> +and ``fid`` determine how the device is presented to the guest operating system.
> +
> +In case of Linux, ``uid`` will be used in the ``domain`` part of the PCI identifier, and
> +``fid`` identifies the physical slot, i.e.::
> +
> + qemu-system-s390x ... \
> +                   -device zpci,uid=7,fid=8,target=hostdev0,id=zpci1 \
> +                   ...
> +
> +will be presented in the guest as::
> +
> + # lspci -v
> + 0007:00:00.0 ...
> + Physical Slot: 00000008
> + ...
> +
> +Finally, note that you might have to enable the ``zpci`` feature in the cpu model in oder to use
> +it.

I'm wondering what the current state of that feature is -- is it present
by default in the newer named models? (My original blog entry was
written nearly five years ago ;)

Re: [PATCH v2] docs/s390x/pcidevices: document pci devices on s390x

Posted by Sebastian Mitterle 1 year, 3 months ago

On Fri, Jan 27, 2023 at 11:30 AM Cornelia Huck <cohuck@redhat.com> wrote:
>
> On Fri, Jan 27 2023, Sebastian Mitterle <smitterl@redhat.com> wrote:
>
> > Add some documentation about the zpci device and how
> > to use it with pci devices on s390x.
> >
> > Used source: Cornelia Huck's blog post
> > https://people.redhat.com/~cohuck/2018/02/19/notes-on-pci-on-s390x.html
> >
> > Signed-off-by: Sebastian Mitterle <smitterl@redhat.com>
> > ---
> > v2: move section below 'Device support'
> > ---
> >  docs/system/s390x/pcidevices.rst | 34 ++++++++++++++++++++++++++++++++
> >  docs/system/target-s390x.rst     |  1 +
> >  2 files changed, 35 insertions(+)
> >  create mode 100644 docs/system/s390x/pcidevices.rst
> >
> > diff --git a/docs/system/s390x/pcidevices.rst b/docs/system/s390x/pcidevices.rst
> > new file mode 100644
> > index 0000000000..fec905d6e6
> > --- /dev/null
> > +++ b/docs/system/s390x/pcidevices.rst
> > @@ -0,0 +1,34 @@
> > +PCI devices on s390x
> > +====================
> > +
> > +PCI devices on s390x work differently than on other architectures.
>
> add "and need to be configured in a slightly different way." ?
>
> > +
> > +To start with, using a PCI device requires the additional ``zpci`` device. For example,
>
> I think the "zpci" device is not technically "required" (ISTR that we
> autogenerate it, if needed); however, you need it if you actually want
> to specify uid/fid/... what about:
>
> "Every PCI device is linked with an additional ``zpci`` device. While
> the ``zpci`` device will be autogenerated if not specified, it is
> recommended to specify it explicitly so that you can pass s390-specific
> PCI configuration."
>
> ?
>
> > +in order to pass a PCI device ``0000:00:00.0`` through you'd specify::
> > +
> > + qemu-system-s390x ... \
> > +                   -device zpci,uid=1,fid=0,target=hostdev0,id=zpci1 \
> > +                   -device vfio-pci,host=0000:00:00.0,id=hostdev0
> > +
> > +Here, the zpci device is joined with the PCI device via the ``target`` property.
> > +
> > +Note that we don't set bus, slot or function here for the guest as is common in other
> > +PCI implementations. Topology information is not available on s390x. Instead, ``uid``
>
> "Topology information is not available on s390x, and the guest will not
> see any of the bus/slot/function information specified on the command
> line."
>
> ?
>
> > +and ``fid`` determine how the device is presented to the guest operating system.
> > +
> > +In case of Linux, ``uid`` will be used in the ``domain`` part of the PCI identifier, and
> > +``fid`` identifies the physical slot, i.e.::
> > +
> > + qemu-system-s390x ... \
> > +                   -device zpci,uid=7,fid=8,target=hostdev0,id=zpci1 \
> > +                   ...
> > +
> > +will be presented in the guest as::
> > +
> > + # lspci -v
> > + 0007:00:00.0 ...
> > + Physical Slot: 00000008
> > + ...
> > +
> > +Finally, note that you might have to enable the ``zpci`` feature in the cpu model in oder to use
> > +it.
>
> I'm wondering what the current state of that feature is -- is it present
> by default in the newer named models? (My original blog entry was
> written nearly five years ago ;)
>
David Hildenbrand confirmed the feature is default-on in the qemu
model since qemu 4.0 and also for the host model if supported by the
modern hardware. Therefore, I'll omit this sentence.

Re: [PATCH v2] docs/s390x/pcidevices: document pci devices on s390x

Posted by Thomas Huth 1 year, 3 months ago

On 27/01/2023 09.46, Sebastian Mitterle wrote:
> Add some documentation about the zpci device and how
> to use it with pci devices on s390x.

Thanks for tackling this! ... some comments below...

> Used source: Cornelia Huck's blog post
> https://people.redhat.com/~cohuck/2018/02/19/notes-on-pci-on-s390x.html
> 
> Signed-off-by: Sebastian Mitterle <smitterl@redhat.com>
> ---
> v2: move section below 'Device support'
> ---
>   docs/system/s390x/pcidevices.rst | 34 ++++++++++++++++++++++++++++++++
>   docs/system/target-s390x.rst     |  1 +
>   2 files changed, 35 insertions(+)
>   create mode 100644 docs/system/s390x/pcidevices.rst
> 
> diff --git a/docs/system/s390x/pcidevices.rst b/docs/system/s390x/pcidevices.rst
> new file mode 100644
> index 0000000000..fec905d6e6
> --- /dev/null
> +++ b/docs/system/s390x/pcidevices.rst
> @@ -0,0 +1,34 @@
> +PCI devices on s390x
> +====================
> +
> +PCI devices on s390x work differently than on other architectures.
> +
> +To start with, using a PCI device requires the additional ``zpci`` device. For example,

Please wrap lines at 80 columns (if possible)

> +in order to pass a PCI device ``0000:00:00.0`` through you'd specify::

I'd suggest to be more explicit here:

in order to pass a host PCI device ``0000:00:00.0`` through to the guest, 
you would specify::

> + qemu-system-s390x ... \
> +                   -device zpci,uid=1,fid=0,target=hostdev0,id=zpci1 \
> +                   -device vfio-pci,host=0000:00:00.0,id=hostdev0
> +
> +Here, the zpci device is joined with the PCI device via the ``target`` property.
> +
> +Note that we don't set bus, slot or function here for the guest as is common in other

"as *it* is common" ?

> +PCI implementations. Topology information is not available on s390x. Instead, ``uid``
> +and ``fid`` determine how the device is presented to the guest operating system.
> +
> +In case of Linux, ``uid`` will be used in the ``domain`` part of the PCI identifier, and
> +``fid`` identifies the physical slot, i.e.::
> +
> + qemu-system-s390x ... \
> +                   -device zpci,uid=7,fid=8,target=hostdev0,id=zpci1 \
> +                   ...
> +
> +will be presented in the guest as::
> +
> + # lspci -v
> + 0007:00:00.0 ...
> + Physical Slot: 00000008
> + ...
> +
> +Finally, note that you might have to enable the ``zpci`` feature in the cpu model in oder to use

s/oder/order/

(and it's a very long line again, please wrap at 80 columns)

> +it.

Should we also add some information about virtio devices? (can also be added 
later, not necessarily in your patch already)

  Thomas

Re: [PATCH v2] docs/s390x/pcidevices: document pci devices on s390x

Posted by Sebastian Mitterle 1 year, 3 months ago

On Fri, Jan 27, 2023 at 10:24 AM Thomas Huth <thuth@redhat.com> wrote:
>
> On 27/01/2023 09.46, Sebastian Mitterle wrote:
> > Add some documentation about the zpci device and how
> > to use it with pci devices on s390x.
>
> Thanks for tackling this! ... some comments below...
>
> > Used source: Cornelia Huck's blog post
> > https://people.redhat.com/~cohuck/2018/02/19/notes-on-pci-on-s390x.html
> >
> > Signed-off-by: Sebastian Mitterle <smitterl@redhat.com>
> > ---
> > v2: move section below 'Device support'
> > ---
> >   docs/system/s390x/pcidevices.rst | 34 ++++++++++++++++++++++++++++++++
> >   docs/system/target-s390x.rst     |  1 +
> >   2 files changed, 35 insertions(+)
> >   create mode 100644 docs/system/s390x/pcidevices.rst
> >
> > diff --git a/docs/system/s390x/pcidevices.rst b/docs/system/s390x/pcidevices.rst
> > new file mode 100644
> > index 0000000000..fec905d6e6
> > --- /dev/null
> > +++ b/docs/system/s390x/pcidevices.rst
> > @@ -0,0 +1,34 @@
> > +PCI devices on s390x
> > +====================
> > +
> > +PCI devices on s390x work differently than on other architectures.
> > +
> > +To start with, using a PCI device requires the additional ``zpci`` device. For example,
>
> Please wrap lines at 80 columns (if possible)
>
> > +in order to pass a PCI device ``0000:00:00.0`` through you'd specify::
>
> I'd suggest to be more explicit here:
>
> in order to pass a host PCI device ``0000:00:00.0`` through to the guest,
> you would specify::
>
> > + qemu-system-s390x ... \
> > +                   -device zpci,uid=1,fid=0,target=hostdev0,id=zpci1 \
> > +                   -device vfio-pci,host=0000:00:00.0,id=hostdev0
> > +
> > +Here, the zpci device is joined with the PCI device via the ``target`` property.
> > +
> > +Note that we don't set bus, slot or function here for the guest as is common in other
>
> "as *it* is common" ?
>
I checked linguee.com and it seems to me it's correct to omit the 'it' here.
> > +PCI implementations. Topology information is not available on s390x. Instead, ``uid``
> > +and ``fid`` determine how the device is presented to the guest operating system.
> > +
> > +In case of Linux, ``uid`` will be used in the ``domain`` part of the PCI identifier, and
> > +``fid`` identifies the physical slot, i.e.::
> > +
> > + qemu-system-s390x ... \
> > +                   -device zpci,uid=7,fid=8,target=hostdev0,id=zpci1 \
> > +                   ...
> > +
> > +will be presented in the guest as::
> > +
> > + # lspci -v
> > + 0007:00:00.0 ...
> > + Physical Slot: 00000008
> > + ...
> > +
> > +Finally, note that you might have to enable the ``zpci`` feature in the cpu model in oder to use
>
> s/oder/order/
>
> (and it's a very long line again, please wrap at 80 columns)
>
> > +it.
>
> Should we also add some information about virtio devices? (can also be added
> later, not necessarily in your patch already)
>
I don't have experience with virtio pci devices on s390x. Therefore I
prefer not to add this information this time.
>   Thomas
>