From: Anisa Su <anisa.su@samsung.com>
This patchset adds support for 6 FM API DCD Management commands (0x5600-0x5605)
according to the CXL r3.2 Spec.
The code was tested with libcxlmi, which runs in the QEMU VM and sends 56xxh
commands to the device (QEMU-emulated) through MCTP messages over USB.
Test Configuration:
- Kernel:
To perform end-to-end tests, both MCTP and DCD support are needed for the kernel.
The kernel version used is Ira's latest DCD branch which is based on 6.15-rc2,
which includes the upstreamed support for MCTP over USB:
https://github.com/weiny2/linux-kernel/tree/dcd-v6-2025-04-13
- QEMU:
To enable MCTP over USB in QEMU, Jonathan's RFC patches are applied on ToT master branch
(https://lore.kernel.org/linux-cxl/20250609163334.922346-1-Jonathan.Cameron@huawei.com/T/#m21b9e0dfc689cb1890bb4d961710c23379e04902)
For the tests of commands 0x5600 (Get DCD Info), 0x5601 (Get Host DC Region
Config), and 0x5603 (Get DC Region Extent Lists), DCD kernel code is not involved.
The libcxlmi test program is used to send the command to the device and results
are collected and verified.
For command 0x5602 (Set DC Region Config): device creates an event record with type
DC_EVENT_REGION_CONFIG_UPDATED and triggers an interrupt to the host
if the configuration changes as a result of the command. Currently, the kernel
version used to test this only supports Add/Release type events. Thus, this
request essentially gets ignored but did not cause problems besides the host
not knowing about the configuration change when tested.
For the command 0x5604 (Initiate DC Add) and 0x5605 (Initiate DC Release), the
tests involve libcxlmi test program (acting as the FM), kernel DCD
code (host) and QEMU device. The test workflow follows that in CXL r3.2 section
7.6.7.6.5 and 7.6.7.6.6. More specifically, the tests involve following
steps:
1. Start a VM with CXL topology:
'-device usb-ehci,id=ehci \
-object memory-backend-file,id=cxl-mem1,mem-path=/tmp/t3_cxl1.raw,size=4G \
-object memory-backend-file,id=cxl-lsa1,mem-path=/tmp/t3_lsa1.raw,size=1M \
-device pxb-cxl,bus_nr=12,bus=pcie.0,id=cxl.1,hdm_for_passthrough=true \
-device cxl-rp,port=0,bus=cxl.1,id=cxl_rp_port0,chassis=0,slot=2 \
-device cxl-upstream,port=2,sn=1234,bus=cxl_rp_port0,id=us0,addr=0.0,multifunction=on, \
-device cxl-switch-mailbox-cci,bus=cxl_rp_port0,addr=0.1,target=us0 \
-device cxl-downstream,port=0,bus=us0,id=swport0,chassis=0,slot=4 \
-device cxl-type3,bus=swport0,volatile-dc-memdev=cxl-mem1,id=cxl-dcd0,lsa=cxl-lsa1,num-dc-regions=2,sn=99 \
-device usb-cxl-mctp,bus=ehci.0,id=usb0,target=us0 \
-device usb-cxl-mctp,bus=ehci.0,id=usb1,target=cxl-dcd0\
-machine cxl-fmw.0.targets.0=cxl.1,cxl-fmw.0.size=4G,cxl-fmw.0.interleave-granularity=1k'
2. Load the CXL related drivers in the VM & configure MCTP endpoints:
3. Create a DC region for the DCD device attached:
cxl create-region -m mem0 -d decoder0.0 -s 1G -t dynamic_ram_a
4. Add/release DC extents by sending 0x5604 and 0x5605 respectively through
the following libcxlmi test program:
https://github.com/computexpresslink/libcxlmi/blob/main/examples/fmapi-mctp.c
5. Check and verify the extents by retrieving the extents list through
command 0x5603 in the test program.
6. Create a DAX Device from the extents added:
daxctl create-device -r region0
daxctl reconfigure-device dax0.1 -m system-ram
The remaining 3 commands in this series (0x5606-0x5608) are related to tags
and sharing, thus have not been implemented.
v1: https://lore.kernel.org/linux-cxl/20250317164204.2299371-1-anisa.su887@gmail.com/
v2: https://lore.kernel.org/linux-cxl/aD3jkRBHmbdc9QmD@deb-101020-bm01.eng.stellus.in/T/#t
v3: https://lore.kernel.org/linux-cxl/20250605234227.970187-1-anisa.su887@gmail.com/
v3 -> v4 Changes
================================================================================
1. Rebased on upstream ToT and retested with MCTP-over-UCB
2. cxl-mailbox-utils: 0x5602 - FMAPI Set DC Region Config
- Move condition to return success if new block size == current block size
after bitmap checks because list of what should fail with unsupported doesn't
allow an exception for a noop request according to the spec
- Picked up Fan's review tag
3. hw/cxl: create helper function to create DC Event Records from extents
Based on Fan's feedback on the FM Add/Release patches from v3, reuse code from
qmp_cxl_process_dynamic_capacity_prescriptive() to use in both qmp interface
and mailbox commands.
Created helper function to create DC event recs and insert into log from
qmp_cxl_process_dynamic_capacity_prescriptive().
4. cxl-mailbox-utils: 0x5604 - FMAPI Initiate DC Add
- Fix local tmp array (CXLUpdateDCExtentListInPl *list) to be declared with
g_autofree/g_malloc0
- Use helper method from refactored QMP code to create/insert DC Event Records
remove previously defined helper method cxl_mbox_create_dc_event_records_for_extents
- Remove previously defined helper method cxl_mbox_dc_add_to_pending() because
it is not used by any other methods.
- Removed unreachable return statement after default case in switch statement
5. cxl-mailbox-utils: 0x5605 - FMAPI Initiate DC Release
- Fix local tmp array (CXLUpdateDCExtentListInPl *list) to be declared with
g_autofree/g_malloc0
- Use helper method from refactored QMP code to create/insert DC Event Records
- Fix CXL_EXTENT_REMOVAL_POLICY_MASK definition to be 0x0f instead of 0x7.
The removal policy should be bits[3:0] per the spec. 0x7 only covers bits[2:0]
- Add missed check for if extent in the list covers a pending extent if Forced
Removal Flag is not set
- Removed unreachable return statement after default case in switch statement
Status:
First 6 patches have Fan's review tag
(cxl-mailbox-utils: 0x5600 - FMAPI Get DCD Info ^.. cxl-mailbox-utils: 0x5603 - FMAPI Get DC Region Extent Lists)
Last 3 still need review tag.
Anisa Su (10):
cxl-mailbox-utils: 0x5600 - FMAPI Get DCD Info
cxl/type3: Add dsmas_flags to CXLDCRegion struct
cxl-mailbox-utils: 0x5601 - FMAPI Get Host Region Config
cxl_events.h: Move definition for dynamic_capacity_uuid and enum for
DC event types
hw/cxl_type3: Add DC Region bitmap lock
cxl-mailbox-utils: 0x5602 - FMAPI Set DC Region Config
cxl-mailbox-utils: 0x5603 - FMAPI Get DC Region Extent Lists
hw/cxl: create helper function to create DC Event Records from extents
cxl-mailbox-utils: 0x5604 - FMAPI Initiate DC Add
cxl-mailbox-utils: 0x5605 - FMAPI Initiate DC Release
hw/cxl/cxl-events.c | 38 +++
hw/cxl/cxl-mailbox-utils.c | 494 ++++++++++++++++++++++++++++++++++-
hw/mem/cxl_type3.c | 82 ++----
include/hw/cxl/cxl.h | 1 +
include/hw/cxl/cxl_device.h | 28 ++
include/hw/cxl/cxl_events.h | 15 ++
include/hw/cxl/cxl_mailbox.h | 6 +
7 files changed, 604 insertions(+), 60 deletions(-)
--
2.47.2