s small RFC patch series is really a hack on what I need from qemu rather
than a proper implementation. I'm hoping to get some guidance from the list on
how to implement this correctly for qemu upstream. Thank you!

The patch series provides support for the ACPI Generic Port support that's
defined by ACPI spec 6.5 5.2.16.7 (Generic Port Affinity Structure). The
series also adds a genport object that allows locality data to be injected via
qemu commandline to the HMAT tables. The generic port support is to allow a hot
plugged CXL memory device to calculate the locality data from the CPU to
the CXL device. The generic port related data provides the locality data from
the CPU to the CXL host bridge (latency and bandwidth). These data in
addition to the PCIe link data, CDAT from device, and CXL switch CDAT if switch
exist, provides the locality data for the entire path.

Patch1: Adds Generic Port Affinity Structure sub-tables to the SRAT. For
each CXL Host Bridge (HB) a GPAS entry is created with a unique proximity
domain. For example, if the system is created with 4 proximity domains (PXM) for
system memory, then the next GPAS will get PXM 4 and so on.

Patch2: Add the json support for generic port. Split out because
clang-format really clobbers the json files.

Patch3: Add a generic port object. The intention here is to allow setup of
numa nodes, add hmat-lb data and node distance for the generic targets. I had to
add a hack in qemu_create_cli_devices() to realize the genport objects. I need
guidance on where and how to do this properly so the genport objects
realize at the correct place and time.

Example of genport setup:
-object genport,id=$X -numa node,genport=genport$X,nodeid=$Y,initiator=$Z
-numa hmat-lb,initiator=$Z,target=$X,hierarchy=memory,data-type=access-latency,latency=$latency
-numa hmat-lb,initiator=$Z,target=$X,hierarchy=memory,data-type=access-bandwidth,bandwidth=$bandwidthM
for ((i = 0; i < total_nodes; i++)); do
        for ((j = 0; j < cxl_hbs; j++ )); do        # 2 CXL HBs
                -numa dist,src=$i,dst=$X,val=$dist
        done
done
Linux kernel support:
https://lore.kernel.org/linux-cxl/168088732996.1441063.10107817505475386072.stgit@djiang5-mobl3/T/#t

---

Dave Jiang (3):
      hw/acpi: Add support for Generic Port Affinity Structure to SRAT
      genport: Add json support for generic port
      acpi: add generic port device object


 hw/acpi/aml-build.c         | 21 +++++++++++++
 hw/acpi/genport.c           | 61 +++++++++++++++++++++++++++++++++++++
 hw/acpi/meson.build         |  1 +
 hw/i386/acpi-build.c        | 45 +++++++++++++++++++++++++++
 include/hw/acpi/aml-build.h | 27 ++++++++++++++++
 qapi/machine.json           |  3 +-
 qapi/qom.json               | 12 ++++++++
 softmmu/vl.c                | 26 ++++++++++++++++
 8 files changed, 195 insertions(+), 1 deletion(-)
 create mode 100644 hw/acpi/genport.c

--

On Tue, 18 Apr 2023 15:21:36 -0700
Dave Jiang <dave.jiang@intel.com> wrote:

> s small RFC patch series is really a hack on what I need from qemu rather
> than a proper implementation. I'm hoping to get some guidance from the list on
> how to implement this correctly for qemu upstream. Thank you!
> 
> The patch series provides support for the ACPI Generic Port support that's
> defined by ACPI spec 6.5 5.2.16.7 (Generic Port Affinity Structure). The
> series also adds a genport object that allows locality data to be injected via
> qemu commandline to the HMAT tables. The generic port support is to allow a hot
> plugged CXL memory device to calculate the locality data from the CPU to
> the CXL device. The generic port related data provides the locality data from
> the CPU to the CXL host bridge (latency and bandwidth). These data in
> addition to the PCIe link data, CDAT from device, and CXL switch CDAT if switch
> exist, provides the locality data for the entire path.
> 
> Patch1: Adds Generic Port Affinity Structure sub-tables to the SRAT. For
> each CXL Host Bridge (HB) a GPAS entry is created with a unique proximity
> domain. For example, if the system is created with 4 proximity domains (PXM) for
> system memory, then the next GPAS will get PXM 4 and so on.

I may be going crazy but I'm not seeing an increment on the numa node. So I think
they all get 4 at the moment. Found it increment in patch 3.

> 
> Patch2: Add the json support for generic port. Split out because
> clang-format really clobbers the json files.
> 
> Patch3: Add a generic port object. The intention here is to allow setup of
> numa nodes, add hmat-lb data and node distance for the generic targets. I had to
> add a hack in qemu_create_cli_devices() to realize the genport objects. I need
> guidance on where and how to do this properly so the genport objects
> realize at the correct place and time.
> 
> Example of genport setup:
> -object genport,id=$X -numa node,genport=genport$X,nodeid=$Y,initiator=$Z
> -numa hmat-lb,initiator=$Z,target=$X,hierarchy=memory,data-type=access-latency,latency=$latency
> -numa hmat-lb,initiator=$Z,target=$X,hierarchy=memory,data-type=access-bandwidth,bandwidth=$bandwidthM

I think we should be using some links to the host bridges in here.
So I don't think there should be an explicit genport object at all.
Instead we should be able to point at the pxb itself.  Perhaps also
allowing other port types in future.

Something like

-device pxb-cxl,id=cxl1.1
-numa node,genport=cxl1.1,nodeid=$X
-numa hmat-lb,initiator=$Z,target=$X,...
-numa hmat-lb,initiator=$X,target=$Y,...
//as generic port goes both ways.

As we are currently using bus_nr for UID (which is admittedly a somewhat dirty hack that
just happened to be convenient) the ACPI building code can use that to fill in the SRAT
entry at appropriate point.

I haven't tried implementing it so there may well be some ordering issues that
require some late binding etc, but it should be possible to make it work.

> for ((i = 0; i < total_nodes; i++)); do
>         for ((j = 0; j < cxl_hbs; j++ )); do        # 2 CXL HBs
>                 -numa dist,src=$i,dst=$X,val=$dist
>         done
> done
> Linux kernel support:
> https://lore.kernel.org/linux-cxl/168088732996.1441063.10107817505475386072.stgit@djiang5-mobl3/T/#t
> 
> ---
> 
> Dave Jiang (3):
>       hw/acpi: Add support for Generic Port Affinity Structure to SRAT
>       genport: Add json support for generic port
>       acpi: add generic port device object
> 
> 
>  hw/acpi/aml-build.c         | 21 +++++++++++++
>  hw/acpi/genport.c           | 61 +++++++++++++++++++++++++++++++++++++
>  hw/acpi/meson.build         |  1 +
>  hw/i386/acpi-build.c        | 45 +++++++++++++++++++++++++++
>  include/hw/acpi/aml-build.h | 27 ++++++++++++++++
>  qapi/machine.json           |  3 +-
>  qapi/qom.json               | 12 ++++++++
>  softmmu/vl.c                | 26 ++++++++++++++++
>  8 files changed, 195 insertions(+), 1 deletion(-)
>  create mode 100644 hw/acpi/genport.c
> 
> --
> 
>

On 5/3/23 3:42 AM, Jonathan Cameron wrote:
> On Tue, 18 Apr 2023 15:21:36 -0700
> Dave Jiang <dave.jiang@intel.com> wrote:
> 
>> s small RFC patch series is really a hack on what I need from qemu rather
>> than a proper implementation. I'm hoping to get some guidance from the list on
>> how to implement this correctly for qemu upstream. Thank you!
>>
>> The patch series provides support for the ACPI Generic Port support that's
>> defined by ACPI spec 6.5 5.2.16.7 (Generic Port Affinity Structure). The
>> series also adds a genport object that allows locality data to be injected via
>> qemu commandline to the HMAT tables. The generic port support is to allow a hot
>> plugged CXL memory device to calculate the locality data from the CPU to
>> the CXL device. The generic port related data provides the locality data from
>> the CPU to the CXL host bridge (latency and bandwidth). These data in
>> addition to the PCIe link data, CDAT from device, and CXL switch CDAT if switch
>> exist, provides the locality data for the entire path.
>>
>> Patch1: Adds Generic Port Affinity Structure sub-tables to the SRAT. For
>> each CXL Host Bridge (HB) a GPAS entry is created with a unique proximity
>> domain. For example, if the system is created with 4 proximity domains (PXM) for
>> system memory, then the next GPAS will get PXM 4 and so on.
> 
> I may be going crazy but I'm not seeing an increment on the numa node. So I think
> they all get 4 at the moment. Found it increment in patch 3.

Sorry about that. There are some changes for 1/3 strayed into 3/3.

> 
>>
>> Patch2: Add the json support for generic port. Split out because
>> clang-format really clobbers the json files.
>>
>> Patch3: Add a generic port object. The intention here is to allow setup of
>> numa nodes, add hmat-lb data and node distance for the generic targets. I had to
>> add a hack in qemu_create_cli_devices() to realize the genport objects. I need
>> guidance on where and how to do this properly so the genport objects
>> realize at the correct place and time.
>>
>> Example of genport setup:
>> -object genport,id=$X -numa node,genport=genport$X,nodeid=$Y,initiator=$Z
>> -numa hmat-lb,initiator=$Z,target=$X,hierarchy=memory,data-type=access-latency,latency=$latency
>> -numa hmat-lb,initiator=$Z,target=$X,hierarchy=memory,data-type=access-bandwidth,bandwidth=$bandwidthM
> 
> I think we should be using some links to the host bridges in here.
> So I don't think there should be an explicit genport object at all.
> Instead we should be able to point at the pxb itself.  Perhaps also
> allowing other port types in future.
> 
> Something like
> 
> -device pxb-cxl,id=cxl1.1
> -numa node,genport=cxl1.1,nodeid=$X

Ok I think that makes sense. So now there's a relation between genport 
being constructed and the passed in numa node. When we are building the 
SRAT, I assume there's a way to get hold of the parsed numa nodes 
attributes and iterate through to attempt a match?

> -numa hmat-lb,initiator=$Z,target=$X,...
> -numa hmat-lb,initiator=$X,target=$Y,...
> //as generic port goes both ways.
> 
> As we are currently using bus_nr for UID (which is admittedly a somewhat dirty hack that
> just happened to be convenient) the ACPI building code can use that to fill in the SRAT
> entry at appropriate point.
> 
> I haven't tried implementing it so there may well be some ordering issues that
> require some late binding etc, but it should be possible to make it work.
> 
>> for ((i = 0; i < total_nodes; i++)); do
>>          for ((j = 0; j < cxl_hbs; j++ )); do        # 2 CXL HBs
>>                  -numa dist,src=$i,dst=$X,val=$dist
>>          done
>> done
>> Linux kernel support:
>> https://lore.kernel.org/linux-cxl/168088732996.1441063.10107817505475386072.stgit@djiang5-mobl3/T/#t
>>
>> ---
>>
>> Dave Jiang (3):
>>        hw/acpi: Add support for Generic Port Affinity Structure to SRAT
>>        genport: Add json support for generic port
>>        acpi: add generic port device object
>>
>>
>>   hw/acpi/aml-build.c         | 21 +++++++++++++
>>   hw/acpi/genport.c           | 61 +++++++++++++++++++++++++++++++++++++
>>   hw/acpi/meson.build         |  1 +
>>   hw/i386/acpi-build.c        | 45 +++++++++++++++++++++++++++
>>   include/hw/acpi/aml-build.h | 27 ++++++++++++++++
>>   qapi/machine.json           |  3 +-
>>   qapi/qom.json               | 12 ++++++++
>>   softmmu/vl.c                | 26 ++++++++++++++++
>>   8 files changed, 195 insertions(+), 1 deletion(-)
>>   create mode 100644 hw/acpi/genport.c
>>
>> --
>>
>>
>

On 5/3/23 3:42 AM, Jonathan Cameron wrote:
> On Tue, 18 Apr 2023 15:21:36 -0700
> Dave Jiang <dave.jiang@intel.com> wrote:
> 
>> s small RFC patch series is really a hack on what I need from qemu rather
>> than a proper implementation. I'm hoping to get some guidance from the list on
>> how to implement this correctly for qemu upstream. Thank you!
>>
>> The patch series provides support for the ACPI Generic Port support that's
>> defined by ACPI spec 6.5 5.2.16.7 (Generic Port Affinity Structure). The
>> series also adds a genport object that allows locality data to be injected via
>> qemu commandline to the HMAT tables. The generic port support is to allow a hot
>> plugged CXL memory device to calculate the locality data from the CPU to
>> the CXL device. The generic port related data provides the locality data from
>> the CPU to the CXL host bridge (latency and bandwidth). These data in
>> addition to the PCIe link data, CDAT from device, and CXL switch CDAT if switch
>> exist, provides the locality data for the entire path.
>>
>> Patch1: Adds Generic Port Affinity Structure sub-tables to the SRAT. For
>> each CXL Host Bridge (HB) a GPAS entry is created with a unique proximity
>> domain. For example, if the system is created with 4 proximity domains (PXM) for
>> system memory, then the next GPAS will get PXM 4 and so on.
> 
> I may be going crazy but I'm not seeing an increment on the numa node. So I think
> they all get 4 at the moment. Found it increment in patch 3.
> 
>>
>> Patch2: Add the json support for generic port. Split out because
>> clang-format really clobbers the json files.
>>
>> Patch3: Add a generic port object. The intention here is to allow setup of
>> numa nodes, add hmat-lb data and node distance for the generic targets. I had to
>> add a hack in qemu_create_cli_devices() to realize the genport objects. I need
>> guidance on where and how to do this properly so the genport objects
>> realize at the correct place and time.
>>
>> Example of genport setup:
>> -object genport,id=$X -numa node,genport=genport$X,nodeid=$Y,initiator=$Z
>> -numa hmat-lb,initiator=$Z,target=$X,hierarchy=memory,data-type=access-latency,latency=$latency
>> -numa hmat-lb,initiator=$Z,target=$X,hierarchy=memory,data-type=access-bandwidth,bandwidth=$bandwidthM
> 
> I think we should be using some links to the host bridges in here.
> So I don't think there should be an explicit genport object at all.
> Instead we should be able to point at the pxb itself.  Perhaps also
> allowing other port types in future.
> 
> Something like
> 
> -device pxb-cxl,id=cxl1.1
> -numa node,genport=cxl1.1,nodeid=$X
> -numa hmat-lb,initiator=$Z,target=$X,...
> -numa hmat-lb,initiator=$X,target=$Y,...
> //as generic port goes both ways.
> 
> As we are currently using bus_nr for UID (which is admittedly a somewhat dirty hack that
> just happened to be convenient) the ACPI building code can use that to fill in the SRAT
> entry at appropriate point.
> 
> I haven't tried implementing it so there may well be some ordering issues that
> require some late binding etc, but it should be possible to make it work.

Ok thanks for the suggestion Jonathan! I appreciate you looking over 
this. I'll take a look when I get a chance.

> 
>> for ((i = 0; i < total_nodes; i++)); do
>>          for ((j = 0; j < cxl_hbs; j++ )); do        # 2 CXL HBs
>>                  -numa dist,src=$i,dst=$X,val=$dist
>>          done
>> done
>> Linux kernel support:
>> https://lore.kernel.org/linux-cxl/168088732996.1441063.10107817505475386072.stgit@djiang5-mobl3/T/#t
>>
>> ---
>>
>> Dave Jiang (3):
>>        hw/acpi: Add support for Generic Port Affinity Structure to SRAT
>>        genport: Add json support for generic port
>>        acpi: add generic port device object
>>
>>
>>   hw/acpi/aml-build.c         | 21 +++++++++++++
>>   hw/acpi/genport.c           | 61 +++++++++++++++++++++++++++++++++++++
>>   hw/acpi/meson.build         |  1 +
>>   hw/i386/acpi-build.c        | 45 +++++++++++++++++++++++++++
>>   include/hw/acpi/aml-build.h | 27 ++++++++++++++++
>>   qapi/machine.json           |  3 +-
>>   qapi/qom.json               | 12 ++++++++
>>   softmmu/vl.c                | 26 ++++++++++++++++
>>   8 files changed, 195 insertions(+), 1 deletion(-)
>>   create mode 100644 hw/acpi/genport.c
>>
>> --
>>
>>
>