hw/arm/virt.c | 26 ++++++++++++++++++++++++++ hw/i386/amd_iommu.c | 2 +- hw/i386/intel_iommu.c | 2 +- hw/intc/arm_gicv3_its_common.c | 2 +- hw/intc/arm_gicv3_its_kvm.c | 2 +- hw/pci-host/gpex.c | 6 ++++++ hw/pci/msi.c | 2 +- hw/pci/pci.c | 25 +++++++++++++++++++++++++ include/exec/memattrs.h | 4 ++-- include/hw/arm/virt.h | 1 + include/hw/intc/arm_gicv3_its_common.h | 2 +- include/hw/pci-host/gpex.h | 2 ++ include/hw/pci/pci.h | 8 ++++++++ kvm-all.c | 4 ++-- 14 files changed, 78 insertions(+), 10 deletions(-)
The NXP DPAA2 is a hardware architecture designed for high-speeed network packet processing. The DPAA2 hardware components are managed by a hardware component called the Management Complex (or MC) which provides an object-base abstraction for software drivers to use the DPAA2 hardware. For more details you can see: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/staging/fsl-mc/README.txt?h=v4.10 The interrupts generated by the DPAA2 hardware components are MSIs. We will add support for direct assigning these DPAA2 components/objects to a virtual machine. However, this will add the need to expand the MSI usage in QEMU. Currently the MSIs in QEMU are pretty much tied to PCI. For ARM the GIC ITS is using a device ID for interrupt translation. Currently, for PCI, the requester ID is used as device ID. This will not work when we add another entity that needs also a device ID which is supposed to be unique across the system. My proposal is to add a static allocation in the virt machine. I considered that this allocation is specific to each machine/platform. Currently only virt machine has it, but other implementations may use the same mechanism as well. So, I used a static allocation with this formula: DeviceID = zero_extend( RequesterID[15:0] ) + 0x10000 * Constant This formula was taken from SBSA spec (Appendix I: DeviceID generation and ITS groups). In case of QEMU the constant will be different for each entity. In this way a unique DeviceID will be generated and the device ID will be derived from a requesterID (in case of PCI) or other means in case of other entities. The implementation is generic as there might be in the future other non-pci devices that are using MSIs or IOMMU. Any architecture can use it, though currently only the ARM architecture is using the function that retrieves the stream ID. I did not change all the replacements of the pci_requester_id (with pci_stream_id) in the code (although if the constant is 0, the stream_id is equal with requester_id). The other architectures (e.g. intel iommu code) assume that the ID is the requester ID. Tested on NXP LS2080 platform. History: v1->v2 ------ - the stream ID was added as a field in the pci device structure in order not to traverse the PCI hierarchy each time a MSI is sent. Diana Craciun (2): Increased the size of requester_id field from MemTxAttrs Add a unique ID in the virt machine to be used as device ID hw/arm/virt.c | 26 ++++++++++++++++++++++++++ hw/i386/amd_iommu.c | 2 +- hw/i386/intel_iommu.c | 2 +- hw/intc/arm_gicv3_its_common.c | 2 +- hw/intc/arm_gicv3_its_kvm.c | 2 +- hw/pci-host/gpex.c | 6 ++++++ hw/pci/msi.c | 2 +- hw/pci/pci.c | 25 +++++++++++++++++++++++++ include/exec/memattrs.h | 4 ++-- include/hw/arm/virt.h | 1 + include/hw/intc/arm_gicv3_its_common.h | 2 +- include/hw/pci-host/gpex.h | 2 ++ include/hw/pci/pci.h | 8 ++++++++ kvm-all.c | 4 ++-- 14 files changed, 78 insertions(+), 10 deletions(-) -- 2.5.5
On 23 May 2017 at 12:12, Diana Craciun <diana.craciun@nxp.com> wrote: > The NXP DPAA2 is a hardware architecture designed for high-speeed network > packet processing. The DPAA2 hardware components are managed by a hardware > component called the Management Complex (or MC) which provides an > object-base abstraction for software drivers to use the DPAA2 hardware. > For more details you can see: > https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/staging/fsl-mc/README.txt?h=v4.10 > > The interrupts generated by the DPAA2 hardware components are MSIs. We will add > support for direct assigning these DPAA2 components/objects to a virtual > machine. However, this will add the need to expand the MSI usage in QEMU. > > Currently the MSIs in QEMU are pretty much tied to PCI. For ARM the > GIC ITS is using a device ID for interrupt translation. Currently, for > PCI, the requester ID is used as device ID. This will not work when > we add another entity that needs also a device ID which is supposed to > be unique across the system. > > My proposal is to add a static allocation in the virt machine. I considered > that this allocation is specific to each machine/platform. Currently only > virt machine has it, but other implementations may use the same mechanism > as well. > So, I used a static allocation with this formula: > > DeviceID = zero_extend( RequesterID[15:0] ) + 0x10000 * Constant > > This formula was taken from SBSA spec (Appendix I: DeviceID generation and > ITS groups). In case of QEMU the constant will be different for each entity. > In this way a unique DeviceID will be generated and the device ID will be > derived from a requesterID (in case of PCI) or other means in case of other > entities. > > The implementation is generic as there might be in the future other non-pci devices > that are using MSIs or IOMMU. Any architecture can use it, though currently > only the ARM architecture is using the function that retrieves the stream ID. I > did not change all the replacements of the pci_requester_id (with pci_stream_id) > in the code (although if the constant is 0, the stream_id is equal with requester_id). > The other architectures (e.g. intel iommu code) assume that the ID is the > requester ID. > > Tested on NXP LS2080 platform. So I'm still a bit confused about what this is for and what the relationship is with how the equivalent task would be performed in real hardware. Can you explain how that works, please? Would there be a system MMU involved somewhere that would be allocating these IDs somehow? The patchset also seems to be introducing a fair amount of mechanism without also implementing the actual use for it -- yes, if we had two PCI controllers in the virt board we would need to sort something out I guess, but we don't, we only have one. Evaluating whether the mechanism makes sense without its major user is always trickier. Essentially, I know very little about this use case, but my default position is that if in doubt we should aim to match how real hardware handles something. Virtualization-specific code, and especially something specific to a niche use case like this, is something I'd really prefer to avoid. The ideal is to have something like PCI device passthrough, where the guest can automatically probe for things and QEMU doesn't need to get involved. I was hoping Eric would review this, since he's more up to speed on direct hardware passthrough than I am, but I think he's just gone off on holiday... thanks -- PMM
Peter, thanks for feedback. Diana will return from vacation next week, I am trying clarify some of the questions. >From: Peter Maydell <peter.maydell@linaro.org> >Sent: Monday, July 10, 2017 8:10 PM >To: Diana Madalina Craciun >Cc: QEMU Developers; Michael S. Tsirkin; Eric Auger; Mike Caraman; qemu-arm; Marcel Apfelbaum; Bharat Bhushan; Christoffer Dall; Laurentiu Tudor >Subject: Re: [Qemu-devel] [PATCH v2 0/2] Add global device ID in virt machine > >On 23 May 2017 at 12:12, Diana Craciun <diana.craciun@nxp.com> wrote: >> The NXP DPAA2 is a hardware architecture designed for high-speeed network >> packet processing. The DPAA2 hardware components are managed by a hardware >> component called the Management Complex (or MC) which provides an >> object-base abstraction for software drivers to use the DPAA2 hardware. >> For more details you can see: >> https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgit.kernel.org%2Fpub%2Fscm%2Flinux%2Fkernel%2Fgit%2Ftorvalds%2Flinux.git%2Ftree%2Fdrivers%2Fstaging%2Ffsl-mc%2FREADME.txt%3Fh%3Dv4.10&data=01%7C01%7Cmike.caraman%40nxp.com%7Ccdb80ed25c4a4a71d86708d4c7b68cd8%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0&sdata=JfYha1w2rZ12URBU3LHgo7OwtEPEDqcGWjdAUiRCP4k%3D&reserved=0 >> >> The interrupts generated by the DPAA2 hardware components are MSIs. We will add >> support for direct assigning these DPAA2 components/objects to a virtual >> machine. However, this will add the need to expand the MSI usage in QEMU. >> >> Currently the MSIs in QEMU are pretty much tied to PCI. For ARM the >> GIC ITS is using a device ID for interrupt translation. Currently, for >> PCI, the requester ID is used as device ID. This will not work when >> we add another entity that needs also a device ID which is supposed to >> be unique across the system. >> >> My proposal is to add a static allocation in the virt machine. I considered >> that this allocation is specific to each machine/platform. Currently only >> virt machine has it, but other implementations may use the same mechanism >> as well. >> So, I used a static allocation with this formula: >> >> DeviceID = zero_extend( RequesterID[15:0] ) + 0x10000 * Constant >> >> This formula was taken from SBSA spec (Appendix I: DeviceID generation and >> ITS groups). In case of QEMU the constant will be different for each entity. >> In this way a unique DeviceID will be generated and the device ID will be >> derived from a requesterID (in case of PCI) or other means in case of other >> entities. >> >> The implementation is generic as there might be in the future other non-pci devices >> that are using MSIs or IOMMU. Any architecture can use it, though currently >> only the ARM architecture is using the function that retrieves the stream ID. I >> did not change all the replacements of the pci_requester_id (with pci_stream_id) >> in the code (although if the constant is 0, the stream_id is equal with requester_id). >> The other architectures (e.g. intel iommu code) assume that the ID is the >> requester ID. >> >> Tested on NXP LS2080 platform. > >So I'm still a bit confused about what this is for and what the >relationship is with how the equivalent task would be performed >in real hardware. Can you explain how that works, please? >Would there be a system MMU involved somewhere that would be >allocating these IDs somehow? This is the first step to accommodate device pass-though with MSI support, required for non-PCI/heterogeneous buses on ARM64 platforms with GIC-ITS. We reached the point where our mc-bus infrastructure got the ack and it is ready to get out of staging: https://lkml.org/lkml/2017/7/18/708 The problem we want to address is related to guest DeviceIDs allocation for GIC-ITS emulation. On real hardware the boot loader, in our case u-boot, allocates the DeviceIDs and populates the PCI msi-map device tree binding. We have a similar binding mechanism for MC devices: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/Documentation/devicetree/bindings/pci/pci-msi.txt http://git.denx.de/?p=u-boot.git;a=blob;f=drivers/pci/pcie_layerscape_fixup.c;h=9e6c2f5dfcf62e8b06935aa9ae3c40a734f9571d;hb=HEAD#l118 Qemu does not populate yet the msi-map and the DeviceIDs for PCI devices defaults to bdf at run-time: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/of/irq.c#n599 http://elixir.free-electrons.com/linux/latest/source/drivers/pci/msi.c#L1488 The proposal is to allocate the DeviceIDs as described in SBSA and to populate the device tree, as oppose of generating them at runtime, in order to stick to device binding definitions and to avoid clashes. >The patchset also seems to be introducing a fair amount of >mechanism without also implementing the actual use for it -- yes, >if we had two PCI controllers in the virt board we would need to >sort something out I guess, but we don't, we only have one. >Evaluating whether the mechanism makes sense without its major >user is always trickier. > >Essentially, I know very little about this use case, but my >default position is that if in doubt we should aim to match >how real hardware handles something. Virtualization-specific >code, and especially something specific to a niche use case like >this, is something I'd really prefer to avoid. The ideal is to >have something like PCI device passthrough, where the guest can >automatically probe for things and QEMU doesn't need to get >involved. > >I was hoping Eric would review this, since he's more up to speed >on direct hardware passthrough than I am, but I think he's just >gone off on holiday... > >thanks >-- PMM > Regards, Mike
On Tue, May 23, 2017 at 02:12:43PM +0300, Diana Craciun wrote: > The NXP DPAA2 is a hardware architecture designed for high-speeed network > packet processing. The DPAA2 hardware components are managed by a hardware > component called the Management Complex (or MC) which provides an > object-base abstraction for software drivers to use the DPAA2 hardware. > For more details you can see: > https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/staging/fsl-mc/README.txt?h=v4.10 > > The interrupts generated by the DPAA2 hardware components are MSIs. We will add > support for direct assigning these DPAA2 components/objects to a virtual > machine. However, this will add the need to expand the MSI usage in QEMU. > > Currently the MSIs in QEMU are pretty much tied to PCI. For ARM the > GIC ITS is using a device ID for interrupt translation. Currently, for > PCI, the requester ID is used as device ID. This will not work when > we add another entity that needs also a device ID which is supposed to > be unique across the system. > > My proposal is to add a static allocation in the virt machine. I considered > that this allocation is specific to each machine/platform. Currently only > virt machine has it, but other implementations may use the same mechanism > as well. > So, I used a static allocation with this formula: > > DeviceID = zero_extend( RequesterID[15:0] ) + 0x10000 * Constant > > This formula was taken from SBSA spec (Appendix I: DeviceID generation and > ITS groups). In case of QEMU the constant will be different for each entity. > In this way a unique DeviceID will be generated and the device ID will be > derived from a requesterID (in case of PCI) or other means in case of other > entities. > > The implementation is generic as there might be in the future other non-pci devices > that are using MSIs or IOMMU. Any architecture can use it, though currently > only the ARM architecture is using the function that retrieves the stream ID. I > did not change all the replacements of the pci_requester_id (with pci_stream_id) > in the code (although if the constant is 0, the stream_id is equal with requester_id). > The other architectures (e.g. intel iommu code) assume that the ID is the > requester ID. > > Tested on NXP LS2080 platform. > > History: I am confused. I get it that non-PCI things want something else in their requester ID, but why require it for PCI devices? How about using Constant == 0 for PCI? This way you do not need to touch PCI at all as DeviceID == RequesterID ... > v1->v2 > ------ > - the stream ID was added as a field in the pci device structure in order > not to traverse the PCI hierarchy each time a MSI is sent. > > > Diana Craciun (2): > Increased the size of requester_id field from MemTxAttrs > Add a unique ID in the virt machine to be used as device ID > > hw/arm/virt.c | 26 ++++++++++++++++++++++++++ > hw/i386/amd_iommu.c | 2 +- > hw/i386/intel_iommu.c | 2 +- > hw/intc/arm_gicv3_its_common.c | 2 +- > hw/intc/arm_gicv3_its_kvm.c | 2 +- > hw/pci-host/gpex.c | 6 ++++++ > hw/pci/msi.c | 2 +- > hw/pci/pci.c | 25 +++++++++++++++++++++++++ > include/exec/memattrs.h | 4 ++-- > include/hw/arm/virt.h | 1 + > include/hw/intc/arm_gicv3_its_common.h | 2 +- > include/hw/pci-host/gpex.h | 2 ++ > include/hw/pci/pci.h | 8 ++++++++ > kvm-all.c | 4 ++-- > 14 files changed, 78 insertions(+), 10 deletions(-) > > -- > 2.5.5
© 2016 - 2024 Red Hat, Inc.