From: Ankit Agrawal <ankita@nvidia.com>

The GPU device memory is reported to the VM as a BAR. The device memory
may not be aligned to the power-of-2, but the QEMU expects the PCI BAR to
be. Align the reported device memory size to the next power-of-2 before
QEMU does an mmap.

Signed-off-by: Ankit Agrawal <ankita@nvidia.com>
---
 hw/vfio/common.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/hw/vfio/common.c b/hw/vfio/common.c
index 4d01ea3515..bb49200458 100644
--- a/hw/vfio/common.c
+++ b/hw/vfio/common.c
@@ -2061,7 +2061,7 @@ int vfio_region_setup(Object *obj, VFIODevice *vbasedev, VFIORegion *region,
 
     region->vbasedev = vbasedev;
     region->flags = info->flags;
-    region->size = info->size;
+    region->size = info->size ? pow2ceil(info->size) : info->size;
     region->fd_offset = info->offset;
     region->nr = index;
 
-- 
2.17.1

On Mon, 5 Jun 2023 16:50:05 -0700
<ankita@nvidia.com> wrote:

> From: Ankit Agrawal <ankita@nvidia.com>
> 
> The GPU device memory is reported to the VM as a BAR. The device memory
> may not be aligned to the power-of-2, but the QEMU expects the PCI BAR to
> be. Align the reported device memory size to the next power-of-2 before
> QEMU does an mmap.
> 
> Signed-off-by: Ankit Agrawal <ankita@nvidia.com>
> ---
>  hw/vfio/common.c | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/hw/vfio/common.c b/hw/vfio/common.c
> index 4d01ea3515..bb49200458 100644
> --- a/hw/vfio/common.c
> +++ b/hw/vfio/common.c
> @@ -2061,7 +2061,7 @@ int vfio_region_setup(Object *obj, VFIODevice *vbasedev, VFIORegion *region,
>  
>      region->vbasedev = vbasedev;
>      region->flags = info->flags;
> -    region->size = info->size;
> +    region->size = info->size ? pow2ceil(info->size) : info->size;
>      region->fd_offset = info->offset;
>      region->nr = index;
>  

Nak, this means our kernel emulation of the BAR is broken, a BAR that
is not naturally aligned is not a PCI BAR.  PCI BAR sizing through the
BAR register still needs to work via the kernel interface alone.  It's
clear now how the kernel resizing the vma on mmap was a hack around
userspace mangling the region size.

Maybe this needs to be exposed as a device specific region, which then
userspace emulates as a BAR for the VM facing device rather than the
kernel emulating it as a BAR.  Thanks,

Alex

On 6/6/23 14:54, Alex Williamson wrote:
> On Mon, 5 Jun 2023 16:50:05 -0700
> <ankita@nvidia.com> wrote:
> 
>> From: Ankit Agrawal <ankita@nvidia.com>
>>
>> The GPU device memory is reported to the VM as a BAR. The device memory
>> may not be aligned to the power-of-2, but the QEMU expects the PCI BAR to
>> be. Align the reported device memory size to the next power-of-2 before
>> QEMU does an mmap.
>>
>> Signed-off-by: Ankit Agrawal <ankita@nvidia.com>
>> ---
>>   hw/vfio/common.c | 2 +-
>>   1 file changed, 1 insertion(+), 1 deletion(-)
>>
>> diff --git a/hw/vfio/common.c b/hw/vfio/common.c
>> index 4d01ea3515..bb49200458 100644
>> --- a/hw/vfio/common.c
>> +++ b/hw/vfio/common.c
>> @@ -2061,7 +2061,7 @@ int vfio_region_setup(Object *obj, VFIODevice *vbasedev, VFIORegion *region,
>>   
>>       region->vbasedev = vbasedev;
>>       region->flags = info->flags;
>> -    region->size = info->size;
>> +    region->size = info->size ? pow2ceil(info->size) : info->size;
>>       region->fd_offset = info->offset;
>>       region->nr = index;
>>   
> 
> Nak, this means our kernel emulation of the BAR is broken, a BAR that
> is not naturally aligned is not a PCI BAR.

Right. So the common code could check this value is correct, like:

   assert(is_power_of_2(->size));

Or less violet using error_report :)

On 6/6/23 01:50, ankita@nvidia.com wrote:
> From: Ankit Agrawal <ankita@nvidia.com>
> 
> The GPU device memory is reported to the VM as a BAR. The device memory
> may not be aligned to the power-of-2, but the QEMU expects the PCI BAR to
> be. Align the reported device memory size to the next power-of-2 before
> QEMU does an mmap.
> 
> Signed-off-by: Ankit Agrawal <ankita@nvidia.com>
> ---
>   hw/vfio/common.c | 2 +-
>   1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/hw/vfio/common.c b/hw/vfio/common.c
> index 4d01ea3515..bb49200458 100644
> --- a/hw/vfio/common.c
> +++ b/hw/vfio/common.c
> @@ -2061,7 +2061,7 @@ int vfio_region_setup(Object *obj, VFIODevice *vbasedev, VFIORegion *region,
>   
>       region->vbasedev = vbasedev;
>       region->flags = info->flags;
> -    region->size = info->size;
> +    region->size = info->size ? pow2ceil(info->size) : info->size;

        region->size = [REAL_]HOST_PAGE_ALIGN(info->size)?

>       region->fd_offset = info->offset;
>       region->nr = index;
>