From: Stewart Hildebrand
> Sent: 17 July 2024 19:31
...
> > For more normal hardware just ensuring that two separate targets don't share
> > a page while allowing (eg) two 1k BAR to reside in the same 64k page would
> > give some security.
> 
> Allow me to understand this better, with an example:
> 
> PCI Device A
>     BAR 1 (1k)
>     BAR 2 (1k)
> 
> PCI Device B
>     BAR 1 (1k)
>     BAR 2 (1k)
> 
> We align all BARs to 4k. Additionally, are you saying it would be ok to
> let both device A BARs to reside in the same 64k page, while device B
> BARs would need to reside in a separate 64k page? I.e. having two levels
> of alignment: PAGE_SIZE on a per-device basis, and 4k on a per-BAR
> basis?
> 
> If I understand you correctly, there's currently no logic in the PCI
> subsystem to easily support this, so that is a rather large ask. I'm
> also not sure that it's necessary.

That is what I was thinking, but it probably doesn't matter.
It would only be necessary if the system would otherwise run out
of PCI(e) address space.

Even after I reduced our FPGAs BARs from 32MB to 'only' 4MB (1MB + 1MB + 8k)
we still get issues with some PC bios failing to allocate the resources
in some slots - but these are old x86-64 systems that might have been expected
to run 32bit windows.
The requirement to use a separate BAR for MSIX pretty much doubles the
required address space.

As an aside, if a PCIe device asks for:
	BAR-0 (4k)
	BAR-1 (8k)
	BAR-2 (4k)
(which is a bit silly)
does it get packed into 16k with no padding by assigning BAR-2 between
BAR-0 and BAR-1, or is it all padded out to 32k.
I'd probably add a comment to say it isn't done :-)

	David

-
Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK
Registration No: 1397386 (Wales)

On 7/18/24 06:01, David Laight wrote:
> From: Stewart Hildebrand
>> Sent: 17 July 2024 19:31
> ...
>>> For more normal hardware just ensuring that two separate targets don't share
>>> a page while allowing (eg) two 1k BAR to reside in the same 64k page would
>>> give some security.
>>
>> Allow me to understand this better, with an example:
>>
>> PCI Device A
>>     BAR 1 (1k)
>>     BAR 2 (1k)
>>
>> PCI Device B
>>     BAR 1 (1k)
>>     BAR 2 (1k)
>>
>> We align all BARs to 4k. Additionally, are you saying it would be ok to
>> let both device A BARs to reside in the same 64k page, while device B
>> BARs would need to reside in a separate 64k page? I.e. having two levels
>> of alignment: PAGE_SIZE on a per-device basis, and 4k on a per-BAR
>> basis?
>>
>> If I understand you correctly, there's currently no logic in the PCI
>> subsystem to easily support this, so that is a rather large ask. I'm
>> also not sure that it's necessary.
> 
> That is what I was thinking, but it probably doesn't matter.
> It would only be necessary if the system would otherwise run out
> of PCI(e) address space.
> 
> Even after I reduced our FPGAs BARs from 32MB to 'only' 4MB (1MB + 1MB + 8k)
> we still get issues with some PC bios failing to allocate the resources
> in some slots - but these are old x86-64 systems that might have been expected
> to run 32bit windows.

I expect this series will not make any difference with that particular
scenario since the BARs are >4k (and PAGE_SIZE == 4k on x86).

> The requirement to use a separate BAR for MSIX pretty much doubles the
> required address space.

4k region, not BAR.

> As an aside, if a PCIe device asks for:
> 	BAR-0 (4k)
> 	BAR-1 (8k)
> 	BAR-2 (4k)
> (which is a bit silly)
> does it get packed into 16k with no padding by assigning BAR-2 between
> BAR-0 and BAR-1, or is it all padded out to 32k.
> I'd probably add a comment to say it isn't done :-)

On a system with 4k page size, this series should not affect the example
you've provided since those BARs are all 4k or larger.

If you are testing with this series applied to your kernel and notice
any regression, please let me know.