Add a basic example for passing a dasd via vfio-ccw.

Signed-off-by: Cornelia Huck <cohuck@redhat.com>
---
 docs/system/s390x/vfio-ccw.rst | 58 ++++++++++++++++++++++++++++++++++
 docs/system/target-s390x.rst   |  1 +
 2 files changed, 59 insertions(+)
 create mode 100644 docs/system/s390x/vfio-ccw.rst

diff --git a/docs/system/s390x/vfio-ccw.rst b/docs/system/s390x/vfio-ccw.rst
new file mode 100644
index 000000000000..3b465578971f
--- /dev/null
+++ b/docs/system/s390x/vfio-ccw.rst
@@ -0,0 +1,58 @@
+Subchannel passthrough via vfio-ccw
+===================================
+
+vfio-ccw (based upon the mediated vfio device infrastructure) allows to
+make certain I/O subchannels and their devices available to a guest. The
+host will not interact with those subchannels/devices any more.
+
+Note that while vfio-ccw should work with most non-QDIO devices, only ECKD
+DASD have really been tested.
+
+Example configuration
+---------------------
+
+Step 1: configure the host device
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Note: it is recommended to use the ``mdevctl`` tool for this step. If this
+is not possible or wanted, follow the manual procedure below.
+
+* Locate the subchannel for the device (in this example, ``0.0.2b09``)::
+
+    [root@host ~]# lscss | grep 0.0.2b09 | awk '{print $2}'
+    0.0.0313
+
+* Unbind the subchannel (in this example, ``0.0.0313``) from the standard
+  I/O subchannel driver and bind it to the vfio-ccw driver::
+
+    [root@host ~]# echo 0.0.0313 > /sys/bus/css/devices/0.0.0313/driver/unbind
+    [root@host ~]# echo 0.0.0313 > /sys/bus/css/drivers/vfio_ccw/bind
+
+* Create the mediated device (identified by a uuid)::
+
+    [root@host ~]# uuidgen
+    7e270a25-e163-4922-af60-757fc8ed48c6
+    [root@host ~]# echo "7e270a25-e163-4922-af60-757fc8ed48c6" > /sys/bus/css/devices/0.0.0313/mdev_supported_types/vfio_ccw-io/create
+
+Step 2: configure QEMU
+~~~~~~~~~~~~~~~~~~~~~~
+
+* Reference the created mediated device and (optionally) pick a device id to
+  be presented in the guest (here, ``fe.0.1234``, which will end up visible
+  in the guest as ``0.0.1234``::
+
+    -device vfio-ccw,devno=fe.0.1234,sysfsdev=/sys/bus/mdev/devices/7e270a25-e163-4922-af60-757fc8ed48c6
+
+* Start the guest. The device (here, ``0.0.1234``) should now be usable::
+
+    [root@guest ~]# lscss -d 0.0.1234
+    Device   Subchan.  DevType CU Type Use  PIM PAM POM  CHPIDs           
+    ----------------------------------------------------------------------
+    0.0.1234 0.0.0007  3390/0e 3990/e9      f0  f0  ff   1a2a3a0a 00000000
+    [root@guest ~]# chccwdev -e 0.0.1234
+    Setting device 0.0.1234 online
+    [  197.011652] dasd-eckd 0.0.1234: A channel path to the device has become operational
+    [  197.014468] dasd-eckd 0.0.1234: New DASD 3390/0E (CU 3990/01) with 10017 cylinders, 15 heads, 224 sectors
+    [  197.045606] dasd-eckd 0.0.1234: DASD with 4 KB/block, 7212240 KB total size, 48 KB/track, compatible disk layout
+    [  197.049034]  dasda:VOL1/  0X2B09: dasda1
+    Done
diff --git a/docs/system/target-s390x.rst b/docs/system/target-s390x.rst
index 2592a05303ef..644e404ef9fd 100644
--- a/docs/system/target-s390x.rst
+++ b/docs/system/target-s390x.rst
@@ -25,6 +25,7 @@ or vfio-ap is also available.
    s390x/vfio-ap
    s390x/css
    s390x/3270
+   s390x/vfio-ccw
 
 Architectural features
 ======================
-- 
2.21.3

On 05/05/2020 15.50, Cornelia Huck wrote:
> Add a basic example for passing a dasd via vfio-ccw.
> 
> Signed-off-by: Cornelia Huck <cohuck@redhat.com>
> ---
>  docs/system/s390x/vfio-ccw.rst | 58 ++++++++++++++++++++++++++++++++++
>  docs/system/target-s390x.rst   |  1 +
>  2 files changed, 59 insertions(+)
>  create mode 100644 docs/system/s390x/vfio-ccw.rst
> 
> diff --git a/docs/system/s390x/vfio-ccw.rst b/docs/system/s390x/vfio-ccw.rst
> new file mode 100644
> index 000000000000..3b465578971f
> --- /dev/null
> +++ b/docs/system/s390x/vfio-ccw.rst
> @@ -0,0 +1,58 @@
> +Subchannel passthrough via vfio-ccw
> +===================================
> +
> +vfio-ccw (based upon the mediated vfio device infrastructure) allows to
> +make certain I/O subchannels and their devices available to a guest. The
> +host will not interact with those subchannels/devices any more.
> +
> +Note that while vfio-ccw should work with most non-QDIO devices, only ECKD
> +DASD have really been tested.

s/DASD/DASDs/ ?

> +Example configuration
> +---------------------
> +
> +Step 1: configure the host device
> +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> +
> +Note: it is recommended to use the ``mdevctl`` tool for this step.

Can you also give an example on how to use it?

> If this
> +is not possible or wanted, follow the manual procedure below.
> +
> +* Locate the subchannel for the device (in this example, ``0.0.2b09``)::
> +
> +    [root@host ~]# lscss | grep 0.0.2b09 | awk '{print $2}'
> +    0.0.0313

I'd remove the "[root@host ~]" part from all examples.

> +* Unbind the subchannel (in this example, ``0.0.0313``) from the standard
> +  I/O subchannel driver and bind it to the vfio-ccw driver::
> +
> +    [root@host ~]# echo 0.0.0313 > /sys/bus/css/devices/0.0.0313/driver/unbind
> +    [root@host ~]# echo 0.0.0313 > /sys/bus/css/drivers/vfio_ccw/bind
> +
> +* Create the mediated device (identified by a uuid)::
> +
> +    [root@host ~]# uuidgen
> +    7e270a25-e163-4922-af60-757fc8ed48c6
> +    [root@host ~]# echo "7e270a25-e163-4922-af60-757fc8ed48c6" > /sys/bus/css/devices/0.0.0313/mdev_supported_types/vfio_ccw-io/create

Maybe break the long line with a backslash?

> +Step 2: configure QEMU
> +~~~~~~~~~~~~~~~~~~~~~~
> +
> +* Reference the created mediated device and (optionally) pick a device id to
> +  be presented in the guest (here, ``fe.0.1234``, which will end up visible
> +  in the guest as ``0.0.1234``::
> +
> +    -device vfio-ccw,devno=fe.0.1234,sysfsdev=/sys/bus/mdev/devices/7e270a25-e163-4922-af60-757fc8ed48c6
> +
> +* Start the guest. The device (here, ``0.0.1234``) should now be usable::
> +
> +    [root@guest ~]# lscss -d 0.0.1234
> +    Device   Subchan.  DevType CU Type Use  PIM PAM POM  CHPIDs           
> +    ----------------------------------------------------------------------
> +    0.0.1234 0.0.0007  3390/0e 3990/e9      f0  f0  ff   1a2a3a0a 00000000
> +    [root@guest ~]# chccwdev -e 0.0.1234
> +    Setting device 0.0.1234 online
Here I'd add a sentence saying that you could run "dmesg -t" now to
check the kernel log. Then omit the timestamp in the example output
below, that will shorten the lines quite a bit.

> +    [  197.011652] dasd-eckd 0.0.1234: A channel path to the device has become operational
> +    [  197.014468] dasd-eckd 0.0.1234: New DASD 3390/0E (CU 3990/01) with 10017 cylinders, 15 heads, 224 sectors
> +    [  197.045606] dasd-eckd 0.0.1234: DASD with 4 KB/block, 7212240 KB total size, 48 KB/track, compatible disk layout
> +    [  197.049034]  dasda:VOL1/  0X2B09: dasda1

 Thomas

On Fri, 15 May 2020 09:54:44 +0200
Thomas Huth <thuth@redhat.com> wrote:

> On 05/05/2020 15.50, Cornelia Huck wrote:
> > Add a basic example for passing a dasd via vfio-ccw.
> > 
> > Signed-off-by: Cornelia Huck <cohuck@redhat.com>
> > ---
> >  docs/system/s390x/vfio-ccw.rst | 58 ++++++++++++++++++++++++++++++++++
> >  docs/system/target-s390x.rst   |  1 +
> >  2 files changed, 59 insertions(+)
> >  create mode 100644 docs/system/s390x/vfio-ccw.rst
> > 
> > diff --git a/docs/system/s390x/vfio-ccw.rst b/docs/system/s390x/vfio-ccw.rst
> > new file mode 100644
> > index 000000000000..3b465578971f
> > --- /dev/null
> > +++ b/docs/system/s390x/vfio-ccw.rst
> > @@ -0,0 +1,58 @@
> > +Subchannel passthrough via vfio-ccw
> > +===================================
> > +
> > +vfio-ccw (based upon the mediated vfio device infrastructure) allows to
> > +make certain I/O subchannels and their devices available to a guest. The
> > +host will not interact with those subchannels/devices any more.
> > +
> > +Note that while vfio-ccw should work with most non-QDIO devices, only ECKD
> > +DASD have really been tested.  
> 
> s/DASD/DASDs/ ?

I've seen 'DASD' used both as singular and as plural. Can use 'DASDs'
if that is less confusing.

> 
> > +Example configuration
> > +---------------------
> > +
> > +Step 1: configure the host device
> > +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> > +
> > +Note: it is recommended to use the ``mdevctl`` tool for this step.  
> 
> Can you also give an example on how to use it?

Maybe

"To define the same device as configured below to be started
automatically, use

::

   [root@host ~]+ driverctl -b css set-override 0.0.0313 vfio_ccw
   [root@host ~]# mdevctl define -u 7e270a25-e163-4922-af60-757fc8ed48c6 \
                  -p 0.0.0313 -t vfio-ccw_io -a
"

> 
> > If this
> > +is not possible or wanted, follow the manual procedure below.
> > +
> > +* Locate the subchannel for the device (in this example, ``0.0.2b09``)::
> > +
> > +    [root@host ~]# lscss | grep 0.0.2b09 | awk '{print $2}'
> > +    0.0.0313  
> 
> I'd remove the "[root@host ~]" part from all examples.

I'd rather keep them, just to make it clear where each command is
issued (there are commands issued in the guest in step 2.)

> 
> > +* Unbind the subchannel (in this example, ``0.0.0313``) from the standard
> > +  I/O subchannel driver and bind it to the vfio-ccw driver::
> > +
> > +    [root@host ~]# echo 0.0.0313 > /sys/bus/css/devices/0.0.0313/driver/unbind
> > +    [root@host ~]# echo 0.0.0313 > /sys/bus/css/drivers/vfio_ccw/bind
> > +
> > +* Create the mediated device (identified by a uuid)::
> > +
> > +    [root@host ~]# uuidgen
> > +    7e270a25-e163-4922-af60-757fc8ed48c6
> > +    [root@host ~]# echo "7e270a25-e163-4922-af60-757fc8ed48c6" > /sys/bus/css/devices/0.0.0313/mdev_supported_types/vfio_ccw-io/create  
> 
> Maybe break the long line with a backslash?

Makes sense.

> 
> > +Step 2: configure QEMU
> > +~~~~~~~~~~~~~~~~~~~~~~
> > +
> > +* Reference the created mediated device and (optionally) pick a device id to
> > +  be presented in the guest (here, ``fe.0.1234``, which will end up visible
> > +  in the guest as ``0.0.1234``::
> > +
> > +    -device vfio-ccw,devno=fe.0.1234,sysfsdev=/sys/bus/mdev/devices/7e270a25-e163-4922-af60-757fc8ed48c6
> > +
> > +* Start the guest. The device (here, ``0.0.1234``) should now be usable::
> > +
> > +    [root@guest ~]# lscss -d 0.0.1234
> > +    Device   Subchan.  DevType CU Type Use  PIM PAM POM  CHPIDs           
> > +    ----------------------------------------------------------------------
> > +    0.0.1234 0.0.0007  3390/0e 3990/e9      f0  f0  ff   1a2a3a0a 00000000
> > +    [root@guest ~]# chccwdev -e 0.0.1234
> > +    Setting device 0.0.1234 online  
> Here I'd add a sentence saying that you could run "dmesg -t" now to
> check the kernel log. Then omit the timestamp in the example output
> below, that will shorten the lines quite a bit.

Makes sense as well.

> 
> > +    [  197.011652] dasd-eckd 0.0.1234: A channel path to the device has become operational
> > +    [  197.014468] dasd-eckd 0.0.1234: New DASD 3390/0E (CU 3990/01) with 10017 cylinders, 15 heads, 224 sectors
> > +    [  197.045606] dasd-eckd 0.0.1234: DASD with 4 KB/block, 7212240 KB total size, 48 KB/track, compatible disk layout
> > +    [  197.049034]  dasda:VOL1/  0X2B09: dasda1  
> 
>  Thomas