From: Rahul Singh <rahul.singh@arm.com>
Setting CONFIG_PCI_PASSTHROUGH=y will enable PCI passthrough on ARM,
even though the feature is not yet complete in the current upstream
codebase. The purpose of this is to make it easier to enable the
necessary configs (HAS_PCI, HAS_VPCI) for testing and development of PCI
passthrough on ARM, and to build it in CI.
Since PCI passthrough on ARM is still work in progress at this time,
make it depend on EXPERT.
Signed-off-by: Rahul Singh <rahul.singh@arm.com>
Signed-off-by: Stefano Stabellini <stefano.stabellini@amd.com>
Signed-off-by: Stewart Hildebrand <stewart.hildebrand@amd.com>
---
The alpine-3.18-gcc-arm64-randconfig CI job should pick up the new
config option, but if we wanted it to be built more consistently in CI
we could add CONFIG_PCI_PASSTHROUGH=y to one of the arm jobs in
automation/gitlab-ci/build.yaml, e.g. alpine-3.18-gcc-debug-arm64.
v7->v8:
* remove select HAS_VPCI as HAS_VPCI_GUEST_SUPPORT already does so
* move select HAS_PCI to HAS_VPCI in common
* move remaining selects under config ARM_64
* add HAS_PASSTHROUGH
* remove "default n" since it is redundant
v6->v7:
* rebase
* send patch separately from series [3]
* add HAS_VPCI_GUEST_SUPPORT since it's upstream now
* drop Julien's A-b due to changes and was given several releases ago
v5->v6:
* no change
v4->v5:
* no change
v3->v4:
* no change
v2->v3:
* add Julien's A-b
v1->v2:
* drop "ARM" naming since it is already in an ARM category
* depend on EXPERT instead of UNSUPPORTED
Changes from downstream to v1:
* depends on ARM_64 (Stefano)
* Don't select HAS_VPCI_GUEST_SUPPORT since this config option is not currently
used in the upstream codebase. This will want to be re-added here once the
vpci series [2] is merged.
* Don't select ARM_SMMU_V3 since this option can already be selected
independently. While PCI passthrough on ARM depends on an SMMU, it does not
depend on a particular version or variant of an SMMU.
* Don't select HAS_ITS since this option can already be selected independently.
HAS_ITS may want to be added here once the MSI series [1] is merged.
* Don't select LATE_HWDOM since this option is unrelated to PCI passthrough.
[1] https://gitlab.com/xen-project/people/bmarquis/xen-arm-poc/-/commits/poc/pci-passthrough
[2] https://lists.xenproject.org/archives/html/xen-devel/2023-10/msg00660.html
[3] https://lore.kernel.org/xen-devel/20231113222118.825758-1-stewart.hildebrand@amd.com/T/#t
(cherry picked from commit 9a08f1f7ce28ec619640ba9ce11018bf443e9a0e from the
downstream branch [1])
---
xen/arch/arm/Kconfig | 8 ++++++++
xen/drivers/Kconfig | 1 +
2 files changed, 9 insertions(+)
diff --git a/xen/arch/arm/Kconfig b/xen/arch/arm/Kconfig
index 3f25da3ca5fd..95a2cd3d006d 100644
--- a/xen/arch/arm/Kconfig
+++ b/xen/arch/arm/Kconfig
@@ -8,6 +8,8 @@ config ARM_64
depends on !ARM_32
select 64BIT
select HAS_FAST_MULTIPLY
+ select HAS_VPCI_GUEST_SUPPORT if PCI_PASSTHROUGH
+ select HAS_PASSTHROUGH if PCI_PASSTHROUGH
config ARM
def_bool y
@@ -258,6 +260,12 @@ config PARTIAL_EMULATION
source "arch/arm/firmware/Kconfig"
+config PCI_PASSTHROUGH
+ bool "PCI passthrough" if EXPERT
+ depends on ARM_64
+ help
+ This option enables PCI device passthrough
+
endmenu
menu "ARM errata workaround via the alternative framework"
diff --git a/xen/drivers/Kconfig b/xen/drivers/Kconfig
index 20050e9bb8b3..d42314bb1904 100644
--- a/xen/drivers/Kconfig
+++ b/xen/drivers/Kconfig
@@ -14,6 +14,7 @@ source "drivers/video/Kconfig"
config HAS_VPCI
bool
+ select HAS_PCI
config HAS_VPCI_GUEST_SUPPORT
bool
base-commit: 229a11aac7bf52f4532ab732ed10f173bd332cd0
--
2.49.0On 13.06.2025 17:17, Stewart Hildebrand wrote: > --- a/xen/arch/arm/Kconfig > +++ b/xen/arch/arm/Kconfig > @@ -8,6 +8,8 @@ config ARM_64 > depends on !ARM_32 > select 64BIT > select HAS_FAST_MULTIPLY > + select HAS_VPCI_GUEST_SUPPORT if PCI_PASSTHROUGH > + select HAS_PASSTHROUGH if PCI_PASSTHROUGH As I just learned, this change (or maybe it was the "select HAS_PCI" further down) has exposed the quarantining Kconfig option prompt, which (aiui) is entirely meaningless on Arm. IOW I think further adjustments are necessary. Jan
On 7/23/25 06:18, Jan Beulich wrote: > On 13.06.2025 17:17, Stewart Hildebrand wrote: >> --- a/xen/arch/arm/Kconfig >> +++ b/xen/arch/arm/Kconfig >> @@ -8,6 +8,8 @@ config ARM_64 >> depends on !ARM_32 >> select 64BIT >> select HAS_FAST_MULTIPLY >> + select HAS_VPCI_GUEST_SUPPORT if PCI_PASSTHROUGH >> + select HAS_PASSTHROUGH if PCI_PASSTHROUGH > > As I just learned, this change (or maybe it was the "select HAS_PCI" > further down) has exposed the quarantining Kconfig option prompt, which > (aiui) is entirely meaningless on Arm. IOW I think further adjustments > are necessary. > > Jan Not entirely meaningless - the choice between "none" and "basic" still seems relevant. Just "scratch page" quarantining hasn't been implemented in any of the Arm iommu drivers. Perhaps just the IOMMU_QUARANTINE_SCRATCH_PAGE option should be hidden on Arm (or only exposed on x86)? E.g.: diff --git a/xen/drivers/passthrough/Kconfig b/xen/drivers/passthrough/Kconfig index 561f9694b2a6..51c54ed530b0 100644 --- a/xen/drivers/passthrough/Kconfig +++ b/xen/drivers/passthrough/Kconfig @@ -90,4 +90,5 @@ choice bool "basic" config IOMMU_QUARANTINE_SCRATCH_PAGE bool "scratch page" + depends on X86 endchoice
On 23.07.2025 16:26, Stewart Hildebrand wrote: > On 7/23/25 06:18, Jan Beulich wrote: >> On 13.06.2025 17:17, Stewart Hildebrand wrote: >>> --- a/xen/arch/arm/Kconfig >>> +++ b/xen/arch/arm/Kconfig >>> @@ -8,6 +8,8 @@ config ARM_64 >>> depends on !ARM_32 >>> select 64BIT >>> select HAS_FAST_MULTIPLY >>> + select HAS_VPCI_GUEST_SUPPORT if PCI_PASSTHROUGH >>> + select HAS_PASSTHROUGH if PCI_PASSTHROUGH >> >> As I just learned, this change (or maybe it was the "select HAS_PCI" >> further down) has exposed the quarantining Kconfig option prompt, which >> (aiui) is entirely meaningless on Arm. IOW I think further adjustments >> are necessary. > > Not entirely meaningless - the choice between "none" and "basic" still > seems relevant. Just "scratch page" quarantining hasn't been implemented > in any of the Arm iommu drivers. Is there support for "basic"? For x86, most of the involved code lives in the vendor-specific drivers, and I can't find anything related in Arm's. Note in particular the hook iommu_quarantine_dev_init() calls, which isn't provided by any of the Arm drivers afaict. Jan
On 7/23/25 10:59, Jan Beulich wrote: > On 23.07.2025 16:26, Stewart Hildebrand wrote: >> On 7/23/25 06:18, Jan Beulich wrote: >>> On 13.06.2025 17:17, Stewart Hildebrand wrote: >>>> --- a/xen/arch/arm/Kconfig >>>> +++ b/xen/arch/arm/Kconfig >>>> @@ -8,6 +8,8 @@ config ARM_64 >>>> depends on !ARM_32 >>>> select 64BIT >>>> select HAS_FAST_MULTIPLY >>>> + select HAS_VPCI_GUEST_SUPPORT if PCI_PASSTHROUGH >>>> + select HAS_PASSTHROUGH if PCI_PASSTHROUGH >>> >>> As I just learned, this change (or maybe it was the "select HAS_PCI" >>> further down) has exposed the quarantining Kconfig option prompt, which >>> (aiui) is entirely meaningless on Arm. IOW I think further adjustments >>> are necessary. >> >> Not entirely meaningless - the choice between "none" and "basic" still >> seems relevant. Just "scratch page" quarantining hasn't been implemented >> in any of the Arm iommu drivers. > > Is there support for "basic"? For x86, most of the involved code lives > in the vendor-specific drivers, and I can't find anything related in > Arm's. Note in particular the hook iommu_quarantine_dev_init() calls, > which isn't provided by any of the Arm drivers afaict. Quoting xen/drivers/passthrough/Kconfig IOMMU_QUARANTINE_* help text: "... basic form, all in-flight DMA will simply be forced to encounter IOMMU faults." My understanding of "basic" is that after destruction of a domU with a PCI device assigned, the PCI device gets assigned to domIO. We have special casing for ( d == dom_io ) in some instances, but otherwise it has relatively little to do with the individual iommu drivers. I believe assigning to domIO should be enough for the Arm iommus to generate faults, since the iommu identifies the PCI device's DMA via sideband information (AXI stream ID).
On 7/23/25 18:16, Stewart Hildebrand wrote:
> On 7/23/25 10:59, Jan Beulich wrote:
>> On 23.07.2025 16:26, Stewart Hildebrand wrote:
>>> On 7/23/25 06:18, Jan Beulich wrote:
>>>> On 13.06.2025 17:17, Stewart Hildebrand wrote:
>>>>> --- a/xen/arch/arm/Kconfig
>>>>> +++ b/xen/arch/arm/Kconfig
>>>>> @@ -8,6 +8,8 @@ config ARM_64
>>>>> depends on !ARM_32
>>>>> select 64BIT
>>>>> select HAS_FAST_MULTIPLY
>>>>> + select HAS_VPCI_GUEST_SUPPORT if PCI_PASSTHROUGH
>>>>> + select HAS_PASSTHROUGH if PCI_PASSTHROUGH
>>>>
>>>> As I just learned, this change (or maybe it was the "select HAS_PCI"
>>>> further down) has exposed the quarantining Kconfig option prompt, which
>>>> (aiui) is entirely meaningless on Arm. IOW I think further adjustments
>>>> are necessary.
>>>
>>> Not entirely meaningless - the choice between "none" and "basic" still
>>> seems relevant. Just "scratch page" quarantining hasn't been implemented
>>> in any of the Arm iommu drivers.
>>
>> Is there support for "basic"? For x86, most of the involved code lives
>> in the vendor-specific drivers, and I can't find anything related in
>> Arm's. Note in particular the hook iommu_quarantine_dev_init() calls,
>> which isn't provided by any of the Arm drivers afaict.
>
> Quoting xen/drivers/passthrough/Kconfig IOMMU_QUARANTINE_* help text:
>
> "... basic form, all in-flight DMA will simply be forced to encounter
> IOMMU faults."
>
> My understanding of "basic" is that after destruction of a domU with a
> PCI device assigned, the PCI device gets assigned to domIO. We have
> special casing for ( d == dom_io ) in some instances, but otherwise it
> has relatively little to do with the individual iommu drivers. I believe
> assigning to domIO should be enough for the Arm iommus to generate
> faults, since the iommu identifies the PCI device's DMA via sideband
> information (AXI stream ID).
Oh, and also note the commit messages in
63919fc4d1ca ("xen/arm: smmuv3: Add PCI devices support for SMMUv3")
and
ca8f6ffeb6e3 ("xen/arm: smmuv2: Add PCI devices support for SMMUv2")
"Implement basic quarantining."
On 24.07.2025 00:26, Stewart Hildebrand wrote:
> On 7/23/25 18:16, Stewart Hildebrand wrote:
>> On 7/23/25 10:59, Jan Beulich wrote:
>>> On 23.07.2025 16:26, Stewart Hildebrand wrote:
>>>> On 7/23/25 06:18, Jan Beulich wrote:
>>>>> On 13.06.2025 17:17, Stewart Hildebrand wrote:
>>>>>> --- a/xen/arch/arm/Kconfig
>>>>>> +++ b/xen/arch/arm/Kconfig
>>>>>> @@ -8,6 +8,8 @@ config ARM_64
>>>>>> depends on !ARM_32
>>>>>> select 64BIT
>>>>>> select HAS_FAST_MULTIPLY
>>>>>> + select HAS_VPCI_GUEST_SUPPORT if PCI_PASSTHROUGH
>>>>>> + select HAS_PASSTHROUGH if PCI_PASSTHROUGH
>>>>>
>>>>> As I just learned, this change (or maybe it was the "select HAS_PCI"
>>>>> further down) has exposed the quarantining Kconfig option prompt, which
>>>>> (aiui) is entirely meaningless on Arm. IOW I think further adjustments
>>>>> are necessary.
>>>>
>>>> Not entirely meaningless - the choice between "none" and "basic" still
>>>> seems relevant. Just "scratch page" quarantining hasn't been implemented
>>>> in any of the Arm iommu drivers.
>>>
>>> Is there support for "basic"? For x86, most of the involved code lives
>>> in the vendor-specific drivers, and I can't find anything related in
>>> Arm's. Note in particular the hook iommu_quarantine_dev_init() calls,
>>> which isn't provided by any of the Arm drivers afaict.
>>
>> Quoting xen/drivers/passthrough/Kconfig IOMMU_QUARANTINE_* help text:
>>
>> "... basic form, all in-flight DMA will simply be forced to encounter
>> IOMMU faults."
>>
>> My understanding of "basic" is that after destruction of a domU with a
>> PCI device assigned, the PCI device gets assigned to domIO. We have
>> special casing for ( d == dom_io ) in some instances, but otherwise it
>> has relatively little to do with the individual iommu drivers. I believe
>> assigning to domIO should be enough for the Arm iommus to generate
>> faults, since the iommu identifies the PCI device's DMA via sideband
>> information (AXI stream ID).
>
> Oh, and also note the commit messages in
> 63919fc4d1ca ("xen/arm: smmuv3: Add PCI devices support for SMMUv3")
> and
> ca8f6ffeb6e3 ("xen/arm: smmuv2: Add PCI devices support for SMMUv2")
> "Implement basic quarantining."
I'm fine with the proposed adjustment if things work on Arm. I'll demand
an Arm maintainer ack then though, I think.
Jan
On 13.06.2025 17:17, Stewart Hildebrand wrote: > --- a/xen/arch/arm/Kconfig > +++ b/xen/arch/arm/Kconfig > @@ -8,6 +8,8 @@ config ARM_64 > depends on !ARM_32 > select 64BIT > select HAS_FAST_MULTIPLY > + select HAS_VPCI_GUEST_SUPPORT if PCI_PASSTHROUGH > + select HAS_PASSTHROUGH if PCI_PASSTHROUGH Seeing this, I like this as little as I liked ... > @@ -258,6 +260,12 @@ config PARTIAL_EMULATION > > source "arch/arm/firmware/Kconfig" > > +config PCI_PASSTHROUGH > + bool "PCI passthrough" if EXPERT > + depends on ARM_64 ... the form with the select-s put here. I'll (obviously) leave it to the Arm maintainers to judge, but my recommendation would be to simply drop this patch. As per the description it's merely "make it easier ...", which imo doesn't warrant such an abuse of HAS_*. As a nit: The select-s also would better be sorted alphabetically. Jan
On 6/16/25 02:42, Jan Beulich wrote:
> On 13.06.2025 17:17, Stewart Hildebrand wrote:
>> --- a/xen/arch/arm/Kconfig
>> +++ b/xen/arch/arm/Kconfig
>> @@ -8,6 +8,8 @@ config ARM_64
>> depends on !ARM_32
>> select 64BIT
>> select HAS_FAST_MULTIPLY
>> + select HAS_VPCI_GUEST_SUPPORT if PCI_PASSTHROUGH
>> + select HAS_PASSTHROUGH if PCI_PASSTHROUGH
>
> Seeing this, I like this as little as I liked ...
>
>> @@ -258,6 +260,12 @@ config PARTIAL_EMULATION
>>
>> source "arch/arm/firmware/Kconfig"
>>
>> +config PCI_PASSTHROUGH
>> + bool "PCI passthrough" if EXPERT
>> + depends on ARM_64
>
> ... the form with the select-s put here. I'll (obviously) leave it to the
> Arm maintainers to judge, but my recommendation would be to simply drop
> this patch. As per the description it's merely "make it easier ...",
> which imo doesn't warrant such an abuse of HAS_*.
"easier" was a poor choice of word. "possible" is more accurate. This
patch addresses a real issue: currently the PCI and vPCI bits can't be
built for Arm, allowing build issues to go unnoticed. E.g. see
4ce671963eb1 ("xen/arm: fix build with HAS_PCI").
On 16.06.2025 12:37, Stewart Hildebrand wrote:
> On 6/16/25 02:42, Jan Beulich wrote:
>> On 13.06.2025 17:17, Stewart Hildebrand wrote:
>>> --- a/xen/arch/arm/Kconfig
>>> +++ b/xen/arch/arm/Kconfig
>>> @@ -8,6 +8,8 @@ config ARM_64
>>> depends on !ARM_32
>>> select 64BIT
>>> select HAS_FAST_MULTIPLY
>>> + select HAS_VPCI_GUEST_SUPPORT if PCI_PASSTHROUGH
>>> + select HAS_PASSTHROUGH if PCI_PASSTHROUGH
>>
>> Seeing this, I like this as little as I liked ...
>>
>>> @@ -258,6 +260,12 @@ config PARTIAL_EMULATION
>>>
>>> source "arch/arm/firmware/Kconfig"
>>>
>>> +config PCI_PASSTHROUGH
>>> + bool "PCI passthrough" if EXPERT
>>> + depends on ARM_64
>>
>> ... the form with the select-s put here. I'll (obviously) leave it to the
>> Arm maintainers to judge, but my recommendation would be to simply drop
>> this patch. As per the description it's merely "make it easier ...",
>> which imo doesn't warrant such an abuse of HAS_*.
>
> "easier" was a poor choice of word. "possible" is more accurate. This
> patch addresses a real issue: currently the PCI and vPCI bits can't be
> built for Arm, allowing build issues to go unnoticed. E.g. see
> 4ce671963eb1 ("xen/arm: fix build with HAS_PCI").
Which gets us back to the question of whether to use "depends on
HAS_PASSTHROUGH" (I think yes then) and where to put the remaining select
(might then better move back to the new option).
Jan
On Mon, 16 Jun 2025, Jan Beulich wrote:
> On 16.06.2025 12:37, Stewart Hildebrand wrote:
> > On 6/16/25 02:42, Jan Beulich wrote:
> >> On 13.06.2025 17:17, Stewart Hildebrand wrote:
> >>> --- a/xen/arch/arm/Kconfig
> >>> +++ b/xen/arch/arm/Kconfig
> >>> @@ -8,6 +8,8 @@ config ARM_64
> >>> depends on !ARM_32
> >>> select 64BIT
> >>> select HAS_FAST_MULTIPLY
> >>> + select HAS_VPCI_GUEST_SUPPORT if PCI_PASSTHROUGH
> >>> + select HAS_PASSTHROUGH if PCI_PASSTHROUGH
> >>
> >> Seeing this, I like this as little as I liked ...
> >>
> >>> @@ -258,6 +260,12 @@ config PARTIAL_EMULATION
> >>>
> >>> source "arch/arm/firmware/Kconfig"
> >>>
> >>> +config PCI_PASSTHROUGH
> >>> + bool "PCI passthrough" if EXPERT
> >>> + depends on ARM_64
> >>
> >> ... the form with the select-s put here. I'll (obviously) leave it to the
> >> Arm maintainers to judge, but my recommendation would be to simply drop
> >> this patch. As per the description it's merely "make it easier ...",
> >> which imo doesn't warrant such an abuse of HAS_*.
> >
> > "easier" was a poor choice of word. "possible" is more accurate. This
> > patch addresses a real issue: currently the PCI and vPCI bits can't be
> > built for Arm, allowing build issues to go unnoticed. E.g. see
> > 4ce671963eb1 ("xen/arm: fix build with HAS_PCI").
>
> Which gets us back to the question of whether to use "depends on
> HAS_PASSTHROUGH" (I think yes then) and where to put the remaining select
> (might then better move back to the new option).
In my opinion, HAS_ options should not be user-configurable but rather
properties of the architecture. Therefore, I would add HAS_PASSTHROUGH
to ARM_64 unconditionally. Then I would make PASSTHROUGH
user-configurable, but dependent on HAS_PASSTHROUGH.
In the rest of the code, we would update the checks to be based on
PASSTHROUGH instead of HAS_PASSTHROUGH.
That said, this patch is simpler while my suggestion is more invasive.
Also this patch is at v8 and we shouldn't keep increasing the scope of
the work for contributors. Finally, I am not certain all maintainers
would agree with my view I just outlined.
So based on the above, and based on the fact that we certainly need this
patch or something like it:
Acked-by: Stefano Stabellini <sstabellini@kernel.org>
© 2016 - 2025 Red Hat, Inc.