From nobody Sat Nov 15 14:54:43 2025 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1750134049918493.90993588810215; Mon, 16 Jun 2025 21:20:49 -0700 (PDT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1uRNo3-0006gY-Uo; Tue, 17 Jun 2025 00:20:19 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1uRNo2-0006gN-33 for qemu-devel@nongnu.org; Tue, 17 Jun 2025 00:20:18 -0400 Received: from zg8tmtyylji0my4xnjqumte4.icoremail.net ([162.243.164.118]) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1uRNny-0005Gl-1D for qemu-devel@nongnu.org; Tue, 17 Jun 2025 00:20:17 -0400 Received: from prodtpl.icoremail.net (unknown [10.12.1.20]) by hzbj-icmmx-7 (Coremail) with SMTP id AQAAfwDn7n757FBoQr04Aw--.543S2; Tue, 17 Jun 2025 12:20:09 +0800 (CST) Received: from phytium.com.cn (unknown [218.76.62.144]) by mail (Coremail) with SMTP id AQAAfwC3Tyny7FBoIwhWAA--.24S3; Tue, 17 Jun 2025 12:20:03 +0800 (CST) From: wangyuquan To: rad@semihalf.com, peter.maydell@linaro.org, leif.lindholm@oss.qualcomm.com, jonathan.cameron@huawei.com Cc: qemu-devel@nongnu.org, linux-cxl@vger.kernel.org, Yuquan Wang Subject: [RFC PATCH v6] hw/arm/sbsa-ref: Support CXL Host Bridge & CFMW Date: Tue, 17 Jun 2025 12:19:46 +0800 Message-Id: <20250617041946.82587-1-wangyuquan1236@phytium.com.cn> X-Mailer: git-send-email 2.34.1 MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-CM-TRANSID: AQAAfwC3Tyny7FBoIwhWAA--.24S3 X-CM-SenderInfo: 5zdqw5pxtxt0arstlqxsk13x1xpou0fpof0/1tbiAQADAWhPIu4E8QAgs9 Authentication-Results: hzbj-icmmx-7; spf=neutral smtp.mail=wangyuquan 1236@phytium.com.cn; X-Coremail-Antispam: 1Uk129KBjvAXoWfGFWrZrW8Xr18KFy3Ar15CFg_yoW8Gr1xKo WIqFs5CF48Kw4SqF10kFZrtrW7XFZ8KFn3JF45CF4Yka1UA3yDJa4fKws7JwsxJr4rtF13 XFZrtr9xW34DJF97n29KB7ZKAUJUUUU8529EdanIXcx71UUUUU7KY7ZEXasCq-sGcSsGvf J3UbIjqfuFe4nvWSU8nxnvy29KBjDU0xBIdaVrnUUvcSsGvfC2KfnxnUUI43ZEXa7xR_UU UUUUUUU== Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=162.243.164.118; envelope-from=wangyuquan1236@phytium.com.cn; helo=zg8tmtyylji0my4xnjqumte4.icoremail.net X-Spam_score_int: -25 X-Spam_score: -2.6 X-Spam_bar: -- X-Spam_report: (-2.6 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_VALIDITY_CERTIFIED_BLOCKED=0.001, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZM-MESSAGEID: 1750134052443116600 Content-Type: text/plain; charset="utf-8" From: Yuquan Wang This creates a specific CXL host bridge (0001:00) with two cxl root ports on sbsa-ref. And the memory layout provides separate space windows for the cxl host bridge in the sbsa-ref memmap: - 64K CXL Host Bridge Component Registers (CHBCR) - 64K CXL_PIO - 128M CXL_MMIO - 256M CXL_ECAM - 4G CXL_MMIO_HIGH To provide CFMWs on sbsa-ref, this extends 1TB space from the hole above RAM Memory [SBSA_MEM] for CXL Fixed Memory Window: - 1T CXL_FIXED_WINDOW Signed-off-by: Yuquan Wang --- v5 -> v6: - Change the CXL root ports to 4 Background =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D Currently the base CXL support for arm platforms is only on Jonathan's patc= hes[2]. SBSA-REF can be more like a real machine, thus my initial purpose is to sup= port the simplest CXL VH topology on sbsa-ref to verify the basic CXL function usage, therefore, some real machine could refer the CXL running result on sbsa-ref= . =20 This series leverages Jonathan's patches to design [SBSA_CXL_CHBCR] and [SBSA_CXL_FIXED_WINDOW] spaces for sbsa-ref layout.=20 Regard to the burden of edk2 firmware, I try to build a static CEDT table a= nd add acpi0016, acpi0017 and other CXL relevant contents into acpi tables[3][4]. = Hence it doesn't need to communicate CXL contents via DT to edk2.=20 The New CXL HOST =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D This patch will use the new CXL host bridge to establish the CXL topology[5= ]. CXL FIXED WINDOW design =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D 0xA0000000000 is chosen as the base address of this space because of 3 reas= ons: 1) It is more suitable to choose a static address instead of that implementation in virt, since a dynamic address space layout of sbsa-ref is not appropriate for its original purpose as a reference platform. 2) The Hotplug Memory address range should in the range of maximum addressable range of sbsa-ref platform(0x10000000000-0x80ffffffffff). It is satisfied the requirements of memory hotplug in linux kernel. 3) The start pfn of CFMW should exceed the reserved_pfn_range for onlined numa node. Usage of CXL on sbsa-ref =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D With the 'create_cxl()' and 'create_cxl_fixed_window()', users don't need t= o input '-device pxb-cxl' , '-device cxl-rp' and '-M cxl-fmw' parameters. Thus, to run sbsa-ref with a CXL device could use: qemu-system-aarch64 \ -object memory-backend-file,id=3Dmem2,mem-path=3D/tmp/mem2,size=3D256M,shar= e=3Dtrue \ -device cxl-type3,bus=3Dcxl.0,volatile-memdev=3Dmem2,id=3Dcxl-mem1 \ Incompatibility problem =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D Although the new CXL host bridge has been separated from the original pcie = host, the incompatibility problem of "-device qemu-xhci" is not resolved. Because the= new device to plug by qemu command would be enumerated at the largest domain(0001), fo= r example, if we add "-device qemu-xhci" to boot sbsa-ref with CXL, the lspci would sh= ow: root@ubuntu-jammy-arm64:~# lspci 0000:00:00.0 Host bridge: Red Hat, Inc. QEMU PCIe Host bridge 0000:00:01.0 Ethernet controller: Intel Corporation 82574L Gigabit Net= work Connection 0000:00:02.0 Display controller: Device 1234:1111 (rev 02) 0001:00:00.0 PCI bridge: Intel Corporation Device 7075 0001:00:01.0 PCI bridge: Intel Corporation Device 7075 0001:00:02.0 PCI bridge: Intel Corporation Device 7075 0001:00:03.0 PCI bridge: Intel Corporation Device 7075 0001:00:04.0 USB controller: Red Hat, Inc. QEMU XHCI Host Controller (= rev 01) 0001:01:00.0 Memory controller [0502]: Intel Corporation Device 0d93 (= rev 01) 0001:02:00.0 Memory controller [0502]: Intel Corporation Device 0d93 (= rev 01) 0001:03:00.0 Memory controller [0502]: Intel Corporation Device 0d93 (= rev 01) 0001:04:00.0 Memory controller [0502]: Intel Corporation Device 0d93 (= rev 01) Hence we should add "bus=3Dpcie.0" when we want to plug some devices on the= original pcie bus, for example: -device qemu-xhci,bus=3Dpcie.0 \ or -device nvme,serial=3Ddeadbeef,bus=3Dpcie.0,drive=3Dhdd \ -drive file=3D../disk/hdd.qcow2,format=3Dqcow2,id=3Dhdd,if=3Dnone \ Dynamic CXL topology problem =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D Actually the ideal expectation is sbsa-ref could also have a dynamic CXL to= pology by user parameters. According to my knowledge, it should pass a dtb to firmware to = match the required address space. I'm currently trying to solve this problem. I am looking for= suggestions on if there are better ways to do it. This series patches are here to hopefully some comments to guide me! Link: [1]: https://lists.nongnu.org/archive/html/qemu-arm/2024-12/msg00350.html [2]: https://lore.kernel.org/linux-cxl/20220616141950.23374-1-Jonathan.Came= ron@huawei.com/ [3]: https://edk2.groups.io/g/devel/message/120851 [4]: https://edk2.groups.io/g/devel/topic/rfc_patch_edk2_platforms_v4/11002= 3229 [5]: https://lore.kernel.org/linux-cxl/20250617040649.81303-1-wangyuquan123= 6@phytium.com.cn/T/#t docs/system/arm/sbsa.rst | 4 ++ hw/arm/Kconfig | 1 + hw/arm/sbsa-ref.c | 137 ++++++++++++++++++++++++++++++++++++++- 3 files changed, 141 insertions(+), 1 deletion(-) diff --git a/docs/system/arm/sbsa.rst b/docs/system/arm/sbsa.rst index 2bf3fc8d59..21b88e88e7 100644 --- a/docs/system/arm/sbsa.rst +++ b/docs/system/arm/sbsa.rst @@ -28,6 +28,7 @@ The ``sbsa-ref`` board supports: - E1000E ethernet card on PCIe bus - Bochs display adapter on PCIe bus - A generic SBSA watchdog device + - CXL host bridge and CXL fixed memory window =20 =20 Board to firmware interface @@ -92,3 +93,6 @@ Platform version changes: =20 0.4 CPU topology information is present in devicetree. + +0.5 + CXL host bridge and CXL fixed memory window are supported. diff --git a/hw/arm/Kconfig b/hw/arm/Kconfig index f543d944c3..b7247f83bc 100644 --- a/hw/arm/Kconfig +++ b/hw/arm/Kconfig @@ -192,6 +192,7 @@ config SBSA_REF select GPIO_KEY select PCI_EXPRESS select PCI_EXPRESS_GENERIC_BRIDGE + select CXL_HOST_BRIDGE select PFLASH_CFI01 select PL011 # UART select PL031 # RTC diff --git a/hw/arm/sbsa-ref.c b/hw/arm/sbsa-ref.c index deae5cf986..7082d83da0 100644 --- a/hw/arm/sbsa-ref.c +++ b/hw/arm/sbsa-ref.c @@ -36,11 +36,15 @@ #include "hw/arm/smmuv3.h" #include "hw/block/flash.h" #include "hw/boards.h" +#include "hw/cxl/cxl.h" +#include "hw/cxl/cxl_host.h" #include "hw/ide/ide-bus.h" #include "hw/ide/ahci-sysbus.h" #include "hw/intc/arm_gicv3_common.h" #include "hw/intc/arm_gicv3_its_common.h" #include "hw/loader.h" +#include "hw/pci/pcie_port.h" +#include "hw/pci-host/cxl_host_bridge.h" #include "hw/pci-host/gpex.h" #include "hw/qdev-properties.h" #include "hw/usb.h" @@ -94,6 +98,13 @@ enum { SBSA_SECURE_MEM, SBSA_AHCI, SBSA_XHCI, + SBSA_CXL, + SBSA_CXL_CHBCR, + SBSA_CXL_MMIO, + SBSA_CXL_MMIO_HIGH, + SBSA_CXL_PIO, + SBSA_CXL_ECAM, + SBSA_CXL_FIXED_WINDOW, }; =20 struct SBSAMachineState { @@ -105,6 +116,7 @@ struct SBSAMachineState { int psci_conduit; DeviceState *gic; PFlashCFI01 *flash[2]; + CXLState cxl_devices_state; }; =20 #define TYPE_SBSA_MACHINE MACHINE_TYPE_NAME("sbsa-ref") @@ -132,6 +144,14 @@ static const MemMapEntry sbsa_ref_memmap[] =3D { /* Space here reserved for more SMMUs */ [SBSA_AHCI] =3D { 0x60100000, 0x00010000 }, [SBSA_XHCI] =3D { 0x60110000, 0x00010000 }, + /* 64K CXL Host Bridge Registers space */ + [SBSA_CXL_CHBCR] =3D { 0x60200000, 0x00010000 }, + /* 64K CXL PIO space */ + [SBSA_CXL_PIO] =3D { 0x60300000, 0x00010000 }, + /* 128M CXL 32-bit MMIO space */ + [SBSA_CXL_MMIO] =3D { 0x60400000, 0x08000000 }, + /* 256M CXL ECAM space */ + [SBSA_CXL_ECAM] =3D { 0x68500000, 0x10000000 }, /* Space here reserved for other devices */ [SBSA_PCIE_PIO] =3D { 0x7fff0000, 0x00010000 }, /* 32-bit address PCIE MMIO space */ @@ -141,6 +161,10 @@ static const MemMapEntry sbsa_ref_memmap[] =3D { /* ~1TB PCIE MMIO space (4GB to 1024GB boundary) */ [SBSA_PCIE_MMIO_HIGH] =3D { 0x100000000ULL, 0xFF00000000ULL }, [SBSA_MEM] =3D { 0x10000000000ULL, RAMLIMIT_BYTES }, + /* 4G CXL 64-bit MMIO space */ + [SBSA_CXL_MMIO_HIGH] =3D { 0x90000000000ULL, 0x100000000ULL }, + /* 1TB CXL FIXED WINDOW space */ + [SBSA_CXL_FIXED_WINDOW] =3D { 0xA0000000000ULL, 0x10000000000ULL }, }; =20 static const int sbsa_ref_irqmap[] =3D { @@ -154,6 +178,7 @@ static const int sbsa_ref_irqmap[] =3D { [SBSA_XHCI] =3D 11, [SBSA_SMMU] =3D 12, /* ... to 15 */ [SBSA_GWDT_WS0] =3D 16, + [SBSA_CXL] =3D 17, /* ... to 20 */ }; =20 static uint64_t sbsa_ref_cpu_mp_affinity(SBSAMachineState *sms, int idx) @@ -216,7 +241,7 @@ static void create_fdt(SBSAMachineState *sms) * fw compatibility. */ qemu_fdt_setprop_cell(fdt, "/", "machine-version-major", 0); - qemu_fdt_setprop_cell(fdt, "/", "machine-version-minor", 4); + qemu_fdt_setprop_cell(fdt, "/", "machine-version-minor", 5); =20 if (ms->numa_state->have_numa_distance) { int size =3D nb_numa_nodes * nb_numa_nodes * 3 * sizeof(uint32_t); @@ -631,6 +656,114 @@ static void create_smmu(const SBSAMachineState *sms, = PCIBus *bus) } } =20 +static void create_cxl_fixed_window(SBSAMachineState *sms, + MemoryRegion *mem, CXLHostBridge *host) +{ + PCIBus *cxlbus =3D PCI_HOST_BRIDGE(host)->bus; + char *cxl_host =3D object_get_canonical_path(OBJECT(cxlbus)); + hwaddr base =3D sbsa_ref_memmap[SBSA_CXL_FIXED_WINDOW].base; + GList *it; + strList host_target =3D { NULL, cxl_host }; + CXLFixedMemoryWindowOptions sbsa_ref_cfmwoptions =3D { + .size =3D 1 * TiB, + .has_interleave_granularity =3D false, + .targets =3D &host_target, + }; + CXLFixedWindow *fw; + + cxl_fixed_memory_window_config(&sms->cxl_devices_state, + &sbsa_ref_cfmwoptions, &error_fatal); + + it =3D sms->cxl_devices_state.fixed_windows; + fw =3D it->data; + fw->base =3D base; + fw->target_hbs[0] =3D OBJECT(host); + + memory_region_init_io(&fw->mr, OBJECT(sms), &cfmws_ops, fw, + "cxl-fixed-memory-region", fw->size); + + memory_region_add_subregion(mem, fw->base, &fw->mr); +} + +static void create_cxl(SBSAMachineState *sms) +{ + hwaddr base_ecam =3D sbsa_ref_memmap[SBSA_CXL_ECAM].base; + hwaddr size_ecam =3D sbsa_ref_memmap[SBSA_CXL_ECAM].size; + hwaddr base_mmio =3D sbsa_ref_memmap[SBSA_CXL_MMIO].base; + hwaddr size_mmio =3D sbsa_ref_memmap[SBSA_CXL_MMIO].size; + hwaddr base_mmio_high =3D sbsa_ref_memmap[SBSA_CXL_MMIO_HIGH].base; + hwaddr size_mmio_high =3D sbsa_ref_memmap[SBSA_CXL_MMIO_HIGH].size; + hwaddr base_pio =3D sbsa_ref_memmap[SBSA_CXL_PIO].base; + hwaddr base_chbcr =3D sbsa_ref_memmap[SBSA_CXL_CHBCR].base; + hwaddr size_chbcr =3D sbsa_ref_memmap[SBSA_CXL_CHBCR].size; + int irq =3D sbsa_ref_irqmap[SBSA_CXL]; + MemoryRegion *mmio_alias, *mmio_alias_high, *mmio_reg; + MemoryRegion *ecam_alias, *ecam_reg; + MemoryRegion *sysmem =3D get_system_memory(); + MemoryRegion *chbcr =3D &sms->cxl_devices_state.host_mr; + DeviceState *dev; + CXLHostBridge *host; + PCIHostState *cxl; + PCIDevice *cxlrp; + PCIEPort *p; + PCIESlot *s; + int i; + + dev =3D qdev_new(TYPE_CXL_HOST); + sysbus_realize_and_unref(SYS_BUS_DEVICE(dev), &error_fatal); + sms->cxl_devices_state.is_enabled =3D true; + + /* Map CXL ECAM space */ + ecam_alias =3D g_new0(MemoryRegion, 1); + ecam_reg =3D sysbus_mmio_get_region(SYS_BUS_DEVICE(dev), 1); + memory_region_init_alias(ecam_alias, OBJECT(dev), "cxl-ecam", + ecam_reg, 0, size_ecam); + memory_region_add_subregion(get_system_memory(), base_ecam, ecam_alias= ); + + /* Map CXL MMIO space */ + mmio_alias =3D g_new0(MemoryRegion, 1); + mmio_reg =3D sysbus_mmio_get_region(SYS_BUS_DEVICE(dev), 2); + memory_region_init_alias(mmio_alias, OBJECT(dev), "cxl-mmio", + mmio_reg, base_mmio, size_mmio); + memory_region_add_subregion(get_system_memory(), base_mmio, mmio_alias= ); + + /* Map CXL MMIO_HIGH space */ + mmio_alias_high =3D g_new0(MemoryRegion, 1); + memory_region_init_alias(mmio_alias_high, OBJECT(dev), "cxl-mmio-high", + mmio_reg, base_mmio_high, size_mmio_high); + memory_region_add_subregion(get_system_memory(), + base_mmio_high, mmio_alias_high); + + /* Map CXL IO port space */ + sysbus_mmio_map(SYS_BUS_DEVICE(dev), 3, base_pio); + + for (i =3D 0; i < PCI_NUM_PINS; i++) { + sysbus_connect_irq(SYS_BUS_DEVICE(dev), i, + qdev_get_gpio_in(sms->gic, irq + i)); + cxl_host_set_irq_num(CXL_HOST(dev), i, irq + i); + } + + /* Map CXL CHBCR space */ + memory_region_init(chbcr, OBJECT(sms), "cxl_host_reg", size_chbcr); + memory_region_add_subregion(sysmem, base_chbcr, chbcr); + + cxl =3D PCI_HOST_BRIDGE(dev); + + for (i =3D 0; i < 4; i++) { + cxlrp =3D pci_new(-1, "cxl-rp"); + p =3D PCIE_PORT(cxlrp); + s =3D PCIE_SLOT(cxlrp); + p->port =3D i; + s->slot =3D i; + pci_realize_and_unref(cxlrp, cxl->bus, &error_fatal); + } + + host =3D CXL_HOST(dev); + cxl_host_hook_up_registers(&sms->cxl_devices_state, host); + + create_cxl_fixed_window(sms, sysmem, host); +} + static void create_pcie(SBSAMachineState *sms) { hwaddr base_ecam =3D sbsa_ref_memmap[SBSA_PCIE_ECAM].base; @@ -823,6 +956,8 @@ static void sbsa_ref_init(MachineState *machine) =20 create_pcie(sms); =20 + create_cxl(sms); + create_secure_ec(secure_sysmem); =20 sms->bootinfo.ram_size =3D machine->ram_size; --=20 2.34.1