Add some information on how pci address work on s390x.

Signed-off-by: Cornelia Huck <cohuck@redhat.com>
---
 docs/pci-addresses.rst | 63 ++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 63 insertions(+)

diff --git a/docs/pci-addresses.rst b/docs/pci-addresses.rst
index 923783a151b0..9e241a24fcfb 100644
--- a/docs/pci-addresses.rst
+++ b/docs/pci-addresses.rst
@@ -184,3 +184,66 @@ guest OS rather than as ``0001:08:00.1``, which is the address of the
 device on the host.
 
 Of course, all the rules and behaviors described above still apply.
+
+zPCI addresses
+==============
+
+For s390x machines, PCI addresses are handled yet differently. No
+topology information is relayed in the PCI addresses; instead, the
+``fid`` and ``uid`` elements of the ``zpci`` device convey information.
+In the simplest case, the following XML snippet
+
+::
+
+      <controller type='pci' index='0' model='pci-root'/>
+      <controller type='pci' index='1' model='pci-bridge'>
+        <model name='pci-bridge'/>
+        <target chassisNr='1'/>
+        <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0'>
+          <zpci uid='0x0002' fid='0x00000001'/>
+        </address>
+      </controller>
+      <interface type='bridge'>
+        <mac address='02:ca:fe:fa:ce:04'/>
+        <source bridge='virbr0'/>
+        <model type='virtio'/>
+        <address type='pci' domain='0x0000' bus='0x01' slot='0x01' function='0x0'>
+          <zpci uid='0x0001' fid='0x00000000'/>
+        </address>
+      </interface>
+
+will result in the following in a Linux guest::
+
+  0001:00:00.0 Ethernet controller: Red Hat, Inc. Virtio network device
+
+Note that the PCI bridge is not visible in the guest; s390x always has a flat
+topology.
+
+Neither are any changes in the PCI address visible in the guest; replacing
+the PCI address for the virtio-net device with
+
+::
+
+  <address type='pci' domain='0x0000' bus='0x01' slot='0x07' function='0x3'>
+
+will result in the exactly same view in the guest, as the addresses there
+are generated from the information provided via the ``zpci`` element (in
+fact, from the ``uid``).
+
+Therefore, replacing the virtio-net device definition with the following XML
+snippet
+
+::
+
+      <interface type='bridge'>
+        <mac address='02:ca:fe:fa:ce:04'/>
+        <source bridge='virbr0'/>
+        <model type='virtio'/>
+        <address type='pci' domain='0x0000' bus='0x01' slot='0x07' function='0x3'>
+          <zpci uid='0x0007' fid='0x00000003'/>
+        </address>
+      </interface>
+
+will yield the following result in a Linux guest::
+
+  0007:00:00.0 Ethernet controller: Red Hat, Inc. Virtio network device
-- 
2.21.1

On 4/15/20 12:51 PM, Cornelia Huck wrote:
> Add some information on how pci address work on s390x.
> 
> Signed-off-by: Cornelia Huck <cohuck@redhat.com>
Conny, thanks for catching this "wacky case"... :)
I took the liberty to remove "completely" because there needs to be room 
for pci multifunction support... :D

> ---
>   docs/pci-addresses.rst | 63 ++++++++++++++++++++++++++++++++++++++++++
>   1 file changed, 63 insertions(+)
> 
> diff --git a/docs/pci-addresses.rst b/docs/pci-addresses.rst
> index 923783a151b0..9e241a24fcfb 100644
> --- a/docs/pci-addresses.rst
> +++ b/docs/pci-addresses.rst
> @@ -184,3 +184,66 @@ guest OS rather than as ``0001:08:00.1``, which is the address of the
>   device on the host.
>   
>   Of course, all the rules and behaviors described above still apply.
> +
> +zPCI addresses
> +==============
> +
> +For s390x machines, PCI addresses are handled yet differently. No
> +topology information is relayed in the PCI addresses; instead, the
> +``fid`` and ``uid`` elements of the ``zpci`` device convey information.
Should it be mentioned that qemu uses the pci address internally to plug 
the device into its pci bus since the pci address as far as I know must 
still be properly provided or generated.


> +In the simplest case, the following XML snippet
> +
> +::
> +
> +      <controller type='pci' index='0' model='pci-root'/>
> +      <controller type='pci' index='1' model='pci-bridge'>
> +        <model name='pci-bridge'/>
> +        <target chassisNr='1'/>
> +        <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0'>
> +          <zpci uid='0x0002' fid='0x00000001'/>
> +        </address>
> +      </controller>
> +      <interface type='bridge'>
> +        <mac address='02:ca:fe:fa:ce:04'/>
> +        <source bridge='virbr0'/>
> +        <model type='virtio'/>
> +        <address type='pci' domain='0x0000' bus='0x01' slot='0x01' function='0x0'>
> +          <zpci uid='0x0001' fid='0x00000000'/>
> +        </address>
> +      </interface>
> +
> +will result in the following in a Linux guest::
> +
> +  0001:00:00.0 Ethernet controller: Red Hat, Inc. Virtio network device
> +
> +Note that the PCI bridge is not visible in the guest; s390x always has a flat
> +topology.
> +
> +Neither are any changes in the PCI address visible in the guest; replacing
> +the PCI address for the virtio-net device with
> +
> +::
> +
> +  <address type='pci' domain='0x0000' bus='0x01' slot='0x07' function='0x3'>
> +
> +will result in the exactly same view in the guest, as the addresses there
> +are generated from the information provided via the ``zpci`` element (in
> +fact, from the ``uid``).
> +
This explains that the uid is used by the guest to define the pci domain 
of the guest device.
How about an addition for the fid? Something like:

The function id 'fid' defines the slot address of the pci card in the 
guest. It can be observed in the guest at /sys/bus/pci/slots. In the 
example given above the card would be at /sys/bus/pci/slots/00000000.


> +Therefore, replacing the virtio-net device definition with the following XML
> +snippet
> +
> +::
> +
> +      <interface type='bridge'>
> +        <mac address='02:ca:fe:fa:ce:04'/>
> +        <source bridge='virbr0'/>
> +        <model type='virtio'/>
> +        <address type='pci' domain='0x0000' bus='0x01' slot='0x07' function='0x3'>
> +          <zpci uid='0x0007' fid='0x00000003'/>
> +        </address>
> +      </interface>
> +
> +will yield the following result in a Linux guest::
> +
> +  0007:00:00.0 Ethernet controller: Red Hat, Inc. Virtio network device
> 
and the card would be at /sys/bus/pci/slots/00000003.


-- 
Mit freundlichen Grüßen/Kind regards
    Boris Fiuczynski

IBM Deutschland Research & Development GmbH
Vorsitzender des Aufsichtsrats: Gregor Pillen
Geschäftsführung: Dirk Wittkopp
Sitz der Gesellschaft: Böblingen
Registergericht: Amtsgericht Stuttgart, HRB 243294

Sorry about the time travel.
I will try to give up that super power... :(

On 4/14/20 11:06 PM, Boris Fiuczynski wrote:
> On 4/15/20 12:51 PM, Cornelia Huck wrote:
>> Add some information on how pci address work on s390x.
>>
>> Signed-off-by: Cornelia Huck <cohuck@redhat.com>
> Conny, thanks for catching this "wacky case"... :)
> I took the liberty to remove "completely" because there needs to be room 
> for pci multifunction support... :D
> 
>> ---
>>   docs/pci-addresses.rst | 63 ++++++++++++++++++++++++++++++++++++++++++
>>   1 file changed, 63 insertions(+)
>>
>> diff --git a/docs/pci-addresses.rst b/docs/pci-addresses.rst
>> index 923783a151b0..9e241a24fcfb 100644
>> --- a/docs/pci-addresses.rst
>> +++ b/docs/pci-addresses.rst
>> @@ -184,3 +184,66 @@ guest OS rather than as ``0001:08:00.1``, which 
>> is the address of the
>>   device on the host.
>>   Of course, all the rules and behaviors described above still apply.
>> +
>> +zPCI addresses
>> +==============
>> +
>> +For s390x machines, PCI addresses are handled yet differently. No
>> +topology information is relayed in the PCI addresses; instead, the
>> +``fid`` and ``uid`` elements of the ``zpci`` device convey information.
> Should it be mentioned that qemu uses the pci address internally to plug 
> the device into its pci bus since the pci address as far as I know must 
> still be properly provided or generated.
> 
> 
>> +In the simplest case, the following XML snippet
>> +
>> +::
>> +
>> +      <controller type='pci' index='0' model='pci-root'/>
>> +      <controller type='pci' index='1' model='pci-bridge'>
>> +        <model name='pci-bridge'/>
>> +        <target chassisNr='1'/>
>> +        <address type='pci' domain='0x0000' bus='0x00' slot='0x01' 
>> function='0x0'>
>> +          <zpci uid='0x0002' fid='0x00000001'/>
>> +        </address>
>> +      </controller>
>> +      <interface type='bridge'>
>> +        <mac address='02:ca:fe:fa:ce:04'/>
>> +        <source bridge='virbr0'/>
>> +        <model type='virtio'/>
>> +        <address type='pci' domain='0x0000' bus='0x01' slot='0x01' 
>> function='0x0'>
>> +          <zpci uid='0x0001' fid='0x00000000'/>
>> +        </address>
>> +      </interface>
>> +
>> +will result in the following in a Linux guest::
>> +
>> +  0001:00:00.0 Ethernet controller: Red Hat, Inc. Virtio network device
>> +
>> +Note that the PCI bridge is not visible in the guest; s390x always 
>> has a flat
>> +topology.
>> +
>> +Neither are any changes in the PCI address visible in the guest; 
>> replacing
>> +the PCI address for the virtio-net device with
>> +
>> +::
>> +
>> +  <address type='pci' domain='0x0000' bus='0x01' slot='0x07' 
>> function='0x3'>
>> +
>> +will result in the exactly same view in the guest, as the addresses 
>> there
>> +are generated from the information provided via the ``zpci`` element (in
>> +fact, from the ``uid``).
>> +
> This explains that the uid is used by the guest to define the pci domain 
> of the guest device.
> How about an addition for the fid? Something like:
> 
> The function id 'fid' defines the slot address of the pci card in the 
> guest. It can be observed in the guest at /sys/bus/pci/slots. In the 
> example given above the card would be at /sys/bus/pci/slots/00000000.
> 
> 
>> +Therefore, replacing the virtio-net device definition with the 
>> following XML
>> +snippet
>> +
>> +::
>> +
>> +      <interface type='bridge'>
>> +        <mac address='02:ca:fe:fa:ce:04'/>
>> +        <source bridge='virbr0'/>
>> +        <model type='virtio'/>
>> +        <address type='pci' domain='0x0000' bus='0x01' slot='0x07' 
>> function='0x3'>
>> +          <zpci uid='0x0007' fid='0x00000003'/>
>> +        </address>
>> +      </interface>
>> +
>> +will yield the following result in a Linux guest::
>> +
>> +  0007:00:00.0 Ethernet controller: Red Hat, Inc. Virtio network device
>>
> and the card would be at /sys/bus/pci/slots/00000003.
> 
> 


-- 
Mit freundlichen Grüßen/Kind regards
    Boris Fiuczynski

IBM Deutschland Research & Development GmbH
Vorsitzender des Aufsichtsrats: Gregor Pillen
Geschäftsführung: Dirk Wittkopp
Sitz der Gesellschaft: Böblingen
Registergericht: Amtsgericht Stuttgart, HRB 243294

On Tue, 14 Apr 2020 23:06:47 +0200
Boris Fiuczynski <fiuczy@linux.ibm.com> wrote:

> On 4/15/20 12:51 PM, Cornelia Huck wrote:

> > +zPCI addresses
> > +==============
> > +
> > +For s390x machines, PCI addresses are handled yet differently. No
> > +topology information is relayed in the PCI addresses; instead, the
> > +``fid`` and ``uid`` elements of the ``zpci`` device convey information.  
> Should it be mentioned that qemu uses the pci address internally to plug 
> the device into its pci bus since the pci address as far as I know must 
> still be properly provided or generated.

Not sure how much is autogenerated; however, as this document talks
about how pci addresses in the xml translate to pci addresses in the
guest, I think we can simply ignore it?

> 
> 
> > +In the simplest case, the following XML snippet
> > +
> > +::
> > +
> > +      <controller type='pci' index='0' model='pci-root'/>
> > +      <controller type='pci' index='1' model='pci-bridge'>
> > +        <model name='pci-bridge'/>
> > +        <target chassisNr='1'/>
> > +        <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0'>
> > +          <zpci uid='0x0002' fid='0x00000001'/>
> > +        </address>
> > +      </controller>
> > +      <interface type='bridge'>
> > +        <mac address='02:ca:fe:fa:ce:04'/>
> > +        <source bridge='virbr0'/>
> > +        <model type='virtio'/>
> > +        <address type='pci' domain='0x0000' bus='0x01' slot='0x01' function='0x0'>
> > +          <zpci uid='0x0001' fid='0x00000000'/>
> > +        </address>
> > +      </interface>
> > +
> > +will result in the following in a Linux guest::
> > +
> > +  0001:00:00.0 Ethernet controller: Red Hat, Inc. Virtio network device
> > +
> > +Note that the PCI bridge is not visible in the guest; s390x always has a flat
> > +topology.
> > +
> > +Neither are any changes in the PCI address visible in the guest; replacing
> > +the PCI address for the virtio-net device with
> > +
> > +::
> > +
> > +  <address type='pci' domain='0x0000' bus='0x01' slot='0x07' function='0x3'>
> > +
> > +will result in the exactly same view in the guest, as the addresses there
> > +are generated from the information provided via the ``zpci`` element (in
> > +fact, from the ``uid``).
> > +  
> This explains that the uid is used by the guest to define the pci domain 
> of the guest device.
> How about an addition for the fid? Something like:
> 
> The function id 'fid' defines the slot address of the pci card in the 
> guest. It can be observed in the guest at /sys/bus/pci/slots. In the 
> example given above the card would be at /sys/bus/pci/slots/00000000.

Hm, is it intentional that this does not show up in the actual pci
address? But maybe I'm confused.

> 
> 
> > +Therefore, replacing the virtio-net device definition with the following XML
> > +snippet
> > +
> > +::
> > +
> > +      <interface type='bridge'>
> > +        <mac address='02:ca:fe:fa:ce:04'/>
> > +        <source bridge='virbr0'/>
> > +        <model type='virtio'/>
> > +        <address type='pci' domain='0x0000' bus='0x01' slot='0x07' function='0x3'>
> > +          <zpci uid='0x0007' fid='0x00000003'/>
> > +        </address>
> > +      </interface>
> > +
> > +will yield the following result in a Linux guest::
> > +
> > +  0007:00:00.0 Ethernet controller: Red Hat, Inc. Virtio network device
> >   
> and the card would be at /sys/bus/pci/slots/00000003.

Do you want to explain the fid/slot stuff in a patch on top? Despite
your attempts at time travel, this patch has already been pushed :)

On 4/15/20 3:42 PM, Cornelia Huck wrote:
> On Tue, 14 Apr 2020 23:06:47 +0200
> Boris Fiuczynski <fiuczy@linux.ibm.com> wrote:
> 
>> On 4/15/20 12:51 PM, Cornelia Huck wrote:
> 
>>> +zPCI addresses
>>> +==============
>>> +
>>> +For s390x machines, PCI addresses are handled yet differently. No
>>> +topology information is relayed in the PCI addresses; instead, the
>>> +``fid`` and ``uid`` elements of the ``zpci`` device convey information.
>> Should it be mentioned that qemu uses the pci address internally to plug
>> the device into its pci bus since the pci address as far as I know must
>> still be properly provided or generated.
> 
> Not sure how much is autogenerated; however, as this document talks
> about how pci addresses in the xml translate to pci addresses in the
> guest, I think we can simply ignore it?
ok. I just wanted to explain why the pci address part still exists and 
why libvirt also will autogenerate all other required bridges.

> 
>>
>>
>>> +In the simplest case, the following XML snippet
>>> +
>>> +::
>>> +
>>> +      <controller type='pci' index='0' model='pci-root'/>
>>> +      <controller type='pci' index='1' model='pci-bridge'>
>>> +        <model name='pci-bridge'/>
>>> +        <target chassisNr='1'/>
>>> +        <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0'>
>>> +          <zpci uid='0x0002' fid='0x00000001'/>
>>> +        </address>
>>> +      </controller>
>>> +      <interface type='bridge'>
>>> +        <mac address='02:ca:fe:fa:ce:04'/>
>>> +        <source bridge='virbr0'/>
>>> +        <model type='virtio'/>
>>> +        <address type='pci' domain='0x0000' bus='0x01' slot='0x01' function='0x0'>
>>> +          <zpci uid='0x0001' fid='0x00000000'/>
>>> +        </address>
>>> +      </interface>
>>> +
>>> +will result in the following in a Linux guest::
>>> +
>>> +  0001:00:00.0 Ethernet controller: Red Hat, Inc. Virtio network device
>>> +
>>> +Note that the PCI bridge is not visible in the guest; s390x always has a flat
>>> +topology.
>>> +
>>> +Neither are any changes in the PCI address visible in the guest; replacing
>>> +the PCI address for the virtio-net device with
>>> +
>>> +::
>>> +
>>> +  <address type='pci' domain='0x0000' bus='0x01' slot='0x07' function='0x3'>
>>> +
>>> +will result in the exactly same view in the guest, as the addresses there
>>> +are generated from the information provided via the ``zpci`` element (in
>>> +fact, from the ``uid``).
>>> +
>> This explains that the uid is used by the guest to define the pci domain
>> of the guest device.
>> How about an addition for the fid? Something like:
>>
>> The function id 'fid' defines the slot address of the pci card in the
>> guest. It can be observed in the guest at /sys/bus/pci/slots. In the
>> example given above the card would be at /sys/bus/pci/slots/00000000.
> 
> Hm, is it intentional that this does not show up in the actual pci
> address? But maybe I'm confused.
Are you referring to the slot of the pci address?

If yes, the pci slot does not provided the required address space for 
the function id. Also we once said that the pci bus in qemu would be 
like the pci bus of the CEC and the firmware does provide a a function 
id for every pci function. The zpci fid does one allow to do the same.

> 
>>
>>
>>> +Therefore, replacing the virtio-net device definition with the following XML
>>> +snippet
>>> +
>>> +::
>>> +
>>> +      <interface type='bridge'>
>>> +        <mac address='02:ca:fe:fa:ce:04'/>
>>> +        <source bridge='virbr0'/>
>>> +        <model type='virtio'/>
>>> +        <address type='pci' domain='0x0000' bus='0x01' slot='0x07' function='0x3'>
>>> +          <zpci uid='0x0007' fid='0x00000003'/>
>>> +        </address>
>>> +      </interface>
>>> +
>>> +will yield the following result in a Linux guest::
>>> +
>>> +  0007:00:00.0 Ethernet controller: Red Hat, Inc. Virtio network device
>>>    
>> and the card would be at /sys/bus/pci/slots/00000003.
> 
> Do you want to explain the fid/slot stuff in a patch on top? Despite
> your attempts at time travel, this patch has already been pushed :) 
It seems to be a very pushy time... :( no matter how many breaks are 
produced... ;)  11 minutes from patch production time to commit time. 
Why even send it for review? :D
Once I understand you confusion above I will provide a patch...


-- 
Mit freundlichen Grüßen/Kind regards
    Boris Fiuczynski

IBM Deutschland Research & Development GmbH
Vorsitzender des Aufsichtsrats: Gregor Pillen
Geschäftsführung: Dirk Wittkopp
Sitz der Gesellschaft: Böblingen
Registergericht: Amtsgericht Stuttgart, HRB 243294

On Wed, 15 Apr 2020 16:23:46 +0200
Boris Fiuczynski <fiuczy@linux.ibm.com> wrote:

> On 4/15/20 3:42 PM, Cornelia Huck wrote:
> > On Tue, 14 Apr 2020 23:06:47 +0200
> > Boris Fiuczynski <fiuczy@linux.ibm.com> wrote:
> >   
> >> On 4/15/20 12:51 PM, Cornelia Huck wrote:  

> >>> +In the simplest case, the following XML snippet
> >>> +
> >>> +::
> >>> +
> >>> +      <controller type='pci' index='0' model='pci-root'/>
> >>> +      <controller type='pci' index='1' model='pci-bridge'>
> >>> +        <model name='pci-bridge'/>
> >>> +        <target chassisNr='1'/>
> >>> +        <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0'>
> >>> +          <zpci uid='0x0002' fid='0x00000001'/>
> >>> +        </address>
> >>> +      </controller>
> >>> +      <interface type='bridge'>
> >>> +        <mac address='02:ca:fe:fa:ce:04'/>
> >>> +        <source bridge='virbr0'/>
> >>> +        <model type='virtio'/>
> >>> +        <address type='pci' domain='0x0000' bus='0x01' slot='0x01' function='0x0'>
> >>> +          <zpci uid='0x0001' fid='0x00000000'/>
> >>> +        </address>
> >>> +      </interface>
> >>> +
> >>> +will result in the following in a Linux guest::
> >>> +
> >>> +  0001:00:00.0 Ethernet controller: Red Hat, Inc. Virtio network device
> >>> +
> >>> +Note that the PCI bridge is not visible in the guest; s390x always has a flat
> >>> +topology.
> >>> +
> >>> +Neither are any changes in the PCI address visible in the guest; replacing
> >>> +the PCI address for the virtio-net device with
> >>> +
> >>> +::
> >>> +
> >>> +  <address type='pci' domain='0x0000' bus='0x01' slot='0x07' function='0x3'>
> >>> +
> >>> +will result in the exactly same view in the guest, as the addresses there
> >>> +are generated from the information provided via the ``zpci`` element (in
> >>> +fact, from the ``uid``).
> >>> +  
> >> This explains that the uid is used by the guest to define the pci domain
> >> of the guest device.
> >> How about an addition for the fid? Something like:
> >>
> >> The function id 'fid' defines the slot address of the pci card in the
> >> guest. It can be observed in the guest at /sys/bus/pci/slots. In the
> >> example given above the card would be at /sys/bus/pci/slots/00000000.  
> > 
> > Hm, is it intentional that this does not show up in the actual pci
> > address? But maybe I'm confused.  
> Are you referring to the slot of the pci address?

Yes.

> 
> If yes, the pci slot does not provided the required address space for 
> the function id. Also we once said that the pci bus in qemu would be 
> like the pci bus of the CEC and the firmware does provide a a function 
> id for every pci function. The zpci fid does one allow to do the same.

Ok, now I dimly remember something like that. Still confusing, but
makes sense.

> 
> >   
> >>
> >>  
> >>> +Therefore, replacing the virtio-net device definition with the following XML
> >>> +snippet
> >>> +
> >>> +::
> >>> +
> >>> +      <interface type='bridge'>
> >>> +        <mac address='02:ca:fe:fa:ce:04'/>
> >>> +        <source bridge='virbr0'/>
> >>> +        <model type='virtio'/>
> >>> +        <address type='pci' domain='0x0000' bus='0x01' slot='0x07' function='0x3'>
> >>> +          <zpci uid='0x0007' fid='0x00000003'/>
> >>> +        </address>
> >>> +      </interface>
> >>> +
> >>> +will yield the following result in a Linux guest::
> >>> +
> >>> +  0007:00:00.0 Ethernet controller: Red Hat, Inc. Virtio network device
> >>>      
> >> and the card would be at /sys/bus/pci/slots/00000003.  
> > 
> > Do you want to explain the fid/slot stuff in a patch on top? Despite
> > your attempts at time travel, this patch has already been pushed :)   
> It seems to be a very pushy time... :( no matter how many breaks are 
> produced... ;)  11 minutes from patch production time to commit time. 
> Why even send it for review? :D

Well, I don't have commit rights O:-)

> Once I understand you confusion above I will provide a patch...

I'd say just go ahead.

On Wed, 2020-04-15 at 16:45 +0200, Cornelia Huck wrote:
> On Wed, 15 Apr 2020 16:23:46 +0200
> Boris Fiuczynski <fiuczy@linux.ibm.com> wrote:
> > Once I understand you confusion above I will provide a patch...
> 
> I'd say just go ahead.

While I appreciate the sentiment, I would rather not have an
excessive amount of detail added to this page.

The idea behind it was to show users that they shouldn't expect the
address in the domain XML to match the one in the guest OS, with a
few real-life examples to illustrate the point. So, I don't think
the details of how exactly zPCI IDs translate to guest-visible PCI
addresses is in scope.

It would be great information to have in some other document, though!
Is there something like that in qemu.git, or in the QEMU wiki? Those
are the places where I would expect it to live, since it's not really
tied to libvirt...

-- 
Andrea Bolognani / Red Hat / Virtualization

On Wed, 15 Apr 2020 18:38:06 +0200
Andrea Bolognani <abologna@redhat.com> wrote:

> On Wed, 2020-04-15 at 16:45 +0200, Cornelia Huck wrote:
> > On Wed, 15 Apr 2020 16:23:46 +0200
> > Boris Fiuczynski <fiuczy@linux.ibm.com> wrote:  
> > > Once I understand you confusion above I will provide a patch...  
> > 
> > I'd say just go ahead.  
> 
> While I appreciate the sentiment, I would rather not have an
> excessive amount of detail added to this page.
> 
> The idea behind it was to show users that they shouldn't expect the
> address in the domain XML to match the one in the guest OS, with a
> few real-life examples to illustrate the point. So, I don't think
> the details of how exactly zPCI IDs translate to guest-visible PCI
> addresses is in scope.
> 
> It would be great information to have in some other document, though!
> Is there something like that in qemu.git, or in the QEMU wiki? Those
> are the places where I would expect it to live, since it's not really
> tied to libvirt...

I don't think there's much in QEMU documentation yet; someone(tm) wrote
https://virtualpenguins.blogspot.com/2018/02/notes-on-pci-on-s390x.html,
which might provide a starting point.

On 4/15/20 6:38 PM, Andrea Bolognani wrote:
> On Wed, 2020-04-15 at 16:45 +0200, Cornelia Huck wrote:
>> On Wed, 15 Apr 2020 16:23:46 +0200
>> Boris Fiuczynski <fiuczy@linux.ibm.com> wrote:
>>> Once I understand you confusion above I will provide a patch...
>>
>> I'd say just go ahead.
> 
> While I appreciate the sentiment, I would rather not have an
> excessive amount of detail added to this page.
> 
> The idea behind it was to show users that they shouldn't expect the
> address in the domain XML to match the one in the guest OS, with a
> few real-life examples to illustrate the point. So, I don't think
> the details of how exactly zPCI IDs translate to guest-visible PCI
> addresses is in scope.
> 
> It would be great information to have in some other document, though!
> Is there something like that in qemu.git, or in the QEMU wiki? Those
> are the places where I would expect it to live, since it's not really
> tied to libvirt...
> 

I disagree because fid is a parameter of the pci address just like 
domain, bus, slot, function and uid.
Besides the fact that one of the first sentences in the document is
"When discussing PCI addresses, it's important to understand the 
relationship between the addresses that can be seen in the domain XML 
and those that are visible inside the guest OS."
as a document reader/user of libvirt I would not expect to search around 
in other documentation for fid if all other parameters are correlated here.

So how about a short explanatory sentence for fid like:
"The slot for the PCI device in the guest OS is defined by the fid 
(function id)."

-- 
Mit freundlichen Grüßen/Kind regards
    Boris Fiuczynski

IBM Deutschland Research & Development GmbH
Vorsitzender des Aufsichtsrats: Gregor Pillen
Geschäftsführung: Dirk Wittkopp
Sitz der Gesellschaft: Böblingen
Registergericht: Amtsgericht Stuttgart, HRB 243294

On Thu, 2020-04-16 at 11:46 +0200, Boris Fiuczynski wrote:
> On 4/15/20 6:38 PM, Andrea Bolognani wrote:
> > The idea behind it was to show users that they shouldn't expect the
> > address in the domain XML to match the one in the guest OS, with a
> > few real-life examples to illustrate the point. So, I don't think
> > the details of how exactly zPCI IDs translate to guest-visible PCI
> > addresses is in scope.
> > 
> > It would be great information to have in some other document, though!
> > Is there something like that in qemu.git, or in the QEMU wiki? Those
> > are the places where I would expect it to live, since it's not really
> > tied to libvirt...
> 
> I disagree because fid is a parameter of the pci address just like 
> domain, bus, slot, function and uid.
> Besides the fact that one of the first sentences in the document is
> "When discussing PCI addresses, it's important to understand the 
> relationship between the addresses that can be seen in the domain XML 
> and those that are visible inside the guest OS."

You're right, the current language is not really explaining the
purpose of the document correctly - it's making it sound like it's
the complete guide to all things PCI addresses, which it's definitely
not intended to be.

What about

  Looking at the configuration for a guest, it would be reasonable
  to expect that each PCI device would show up in the guest OS with
  a PCI address that matches the one present in the corresponding
  ``<address>`` element of the domain XML, but that's not guaranteed
  to happen and will in fact not be the case in all but the simplest
  scenarios.

?

> as a document reader/user of libvirt I would not expect to search around 
> in other documentation for fid if all other parameters are correlated here.
> 
> So how about a short explanatory sentence for fid like:
> "The slot for the PCI device in the guest OS is defined by the fid 
> (function id)."

The document already contains the following sentence:

  [...] the addresses there are generated from the information
  provided via the zpci element (in fact, from the uid).

I'm not sure what "slot" means in the sentence you suggest. It
doesn't seem to be the same as slot in domain:bus:slot.function,
because in the second zPCI example that was removed

  <zpci uid='0x0007' fid='0x00000003'/>

would result in

  0007:00:00.0 Ethernet controller: Red Hat, Inc. Virtio network device

with the fid nowhere to be seen. Can you explain what "slot" means
in this context?

Anyway, we can tweak the existing sentence to read something like

  [...] the addresses there are generated from the information
  provided via the zpci element: the uid is used as PCI domain, and
  the fid is used as [your explanation here].

Does that sound good?

-- 
Andrea Bolognani / Red Hat / Virtualization

On 4/16/20 2:21 PM, Andrea Bolognani wrote:
> On Thu, 2020-04-16 at 11:46 +0200, Boris Fiuczynski wrote:
>> On 4/15/20 6:38 PM, Andrea Bolognani wrote:
>>> The idea behind it was to show users that they shouldn't expect the
>>> address in the domain XML to match the one in the guest OS, with a
>>> few real-life examples to illustrate the point. So, I don't think
>>> the details of how exactly zPCI IDs translate to guest-visible PCI
>>> addresses is in scope.
>>>
>>> It would be great information to have in some other document, though!
>>> Is there something like that in qemu.git, or in the QEMU wiki? Those
>>> are the places where I would expect it to live, since it's not really
>>> tied to libvirt...
>>
>> I disagree because fid is a parameter of the pci address just like
>> domain, bus, slot, function and uid.
>> Besides the fact that one of the first sentences in the document is
>> "When discussing PCI addresses, it's important to understand the
>> relationship between the addresses that can be seen in the domain XML
>> and those that are visible inside the guest OS."
> 
> You're right, the current language is not really explaining the
> purpose of the document correctly - it's making it sound like it's
> the complete guide to all things PCI addresses, which it's definitely
> not intended to be.
> 
> What about
> 
>    Looking at the configuration for a guest, it would be reasonable
>    to expect that each PCI device would show up in the guest OS with
>    a PCI address that matches the one present in the corresponding
>    ``<address>`` element of the domain XML, but that's not guaranteed
>    to happen and will in fact not be the case in all but the simplest
>    scenarios.
> 
> ?
Yes, it does bring the documents intention across better.

> 
>> as a document reader/user of libvirt I would not expect to search around
>> in other documentation for fid if all other parameters are correlated here.
>>
>> So how about a short explanatory sentence for fid like:
>> "The slot for the PCI device in the guest OS is defined by the fid
>> (function id)."
> 
> The document already contains the following sentence:
> 
>    [...] the addresses there are generated from the information
>    provided via the zpci element (in fact, from the uid).
> 
> I'm not sure what "slot" means in the sentence you suggest. It
> doesn't seem to be the same as slot in domain:bus:slot.function,
> because in the second zPCI example that was removed
> 
>    <zpci uid='0x0007' fid='0x00000003'/>
> 
> would result in
> 
>    0007:00:00.0 Ethernet controller: Red Hat, Inc. Virtio network device
> I added the sentence "Also fid does not define slot or function of the 
PCI address." following the "Note...topology." sentence.

> with the fid nowhere to be seen. Can you explain what "slot" means
> in this context?
I guess that you now ask to explain it makes it most likely that it 
should be explained here, correct?

The fid (function id) is not used within the PCI address which the guest 
OS generates for the PCI device. Only the uid is used in the PCI address 
to define the PCI domain. PCI bus, slot and function address elements 
are always 0. This might change in the future BUT it would/will still be 
completely unrelated to the PCI attributes bus, slot and function of the 
address element.
The fid is used in the sysfs (/sys/bus/pci/slots/...) to represent the 
PCI card such e.g. you can power on or off the PCI card.
This is what results in guest OS.
root@qemus390x:~# ls /sys/bus/pci/slots/
00000003
root@qemus390x:~# cat /sys/bus/pci/devices/0007\:00\:00.0/function_id
0x00000003

> 
> Anyway, we can tweak the existing sentence to read something like
> 
>    [...] the addresses there are generated from the information
>    provided via the zpci element: the uid is used as PCI domain, and
>    the fid is used as [your explanation here].
[my explanation]
the fid is used as the PCI devices slot in the sysfs.

Does that together with the additional sentence above make more "wacky" 
sense?

> 
> Does that sound good?
> 
Sure.
Since I started editing in the pci-addresses file already, I would try 
to also include these changes if you agree to them.
If you want I can also include your changes in the document introduction.


-- 
Mit freundlichen Grüßen/Kind regards
    Boris Fiuczynski

IBM Deutschland Research & Development GmbH
Vorsitzender des Aufsichtsrats: Gregor Pillen
Geschäftsführung: Dirk Wittkopp
Sitz der Gesellschaft: Böblingen
Registergericht: Amtsgericht Stuttgart, HRB 243294

On a Wednesday in 2020, Cornelia Huck wrote:
>Add some information on how pci address work on s390x.
>
>Signed-off-by: Cornelia Huck <cohuck@redhat.com>
>---
> docs/pci-addresses.rst | 63 ++++++++++++++++++++++++++++++++++++++++++
> 1 file changed, 63 insertions(+)
>
>diff --git a/docs/pci-addresses.rst b/docs/pci-addresses.rst
>index 923783a151b0..9e241a24fcfb 100644
>--- a/docs/pci-addresses.rst
>+++ b/docs/pci-addresses.rst
>@@ -184,3 +184,66 @@ guest OS rather than as ``0001:08:00.1``, which is the address of the
> device on the host.
>
> Of course, all the rules and behaviors described above still apply.
>+

I added one extra blank line to match what we use elsehwere in this file
for this level of headings.

>+zPCI addresses
>+==============
>+

Reviewed-by: Ján Tomko <jtomko@redhat.com>

Jano

On Wed, 2020-04-15 at 12:51 +0200, Cornelia Huck wrote:
> Add some information on how pci address work on s390x.
> 
> Signed-off-by: Cornelia Huck <cohuck@redhat.com>
> ---
>  docs/pci-addresses.rst | 63 ++++++++++++++++++++++++++++++++++++++++++
>  1 file changed, 63 insertions(+)

There are a few comments that I didn't have a chance to raise before
the patch was pushed, so I just posted a follow-up series addressing
them:

  https://www.redhat.com/archives/libvir-list/2020-April/msg00749.html

Please take a look and let me know whether you're okay with them.

-- 
Andrea Bolognani / Red Hat / Virtualization