From nobody Mon Feb 9 05:41:43 2026 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=nongnu.org ARC-Seal: i=1; a=rsa-sha256; t=1655391377; cv=none; d=zohomail.com; s=zohoarc; b=DWFNH9Wn1Y91qlwUf6ZmFIBAmM88rCm5XWIlkwrJnqsTBrVXaC1Ml2YZjwIEuwu2+7EVTaipfpphlcuuwEkmeCGvZnOcVmk7HLa93ZvkyflV/ALyIFu7xM8GQ2Wa91kSoltqxrlMCARv9eSjvi1As5dK8Vbnoek4WQdwEZf4eWs= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1655391377; h=Content-Type:Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:Reply-To:References:Sender:Subject:To; bh=cwnWLnVcMUcN0vtSJL+daNvkArruGvFZpQFAhX1oFiI=; b=ZSLFq07SX4wM3Od0YGcNZc3VHY+ysD/lrRcfmyUpgU5fci4wRFbxnmLbYJ5J3L3VkdGz11no0UlNnfN0EQFHPBM5Wx5FkoWYxQK9ywauupC7pdJZaegPIzvBAY+ZFjPyrgU2HiYdrsLx5Eq4al7YRuyTpuMqetOeJ2kJt+sJidU= ARC-Authentication-Results: i=1; mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1655391377110603.1019335001197; Thu, 16 Jun 2022 07:56:17 -0700 (PDT) Received: from localhost ([::1]:55378 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1o1qut-0006XI-H5 for importer@patchew.org; Thu, 16 Jun 2022 10:56:15 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:45290) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1o1qre-0002I9-0Y for qemu-devel@nongnu.org; Thu, 16 Jun 2022 10:52:54 -0400 Received: from frasgout.his.huawei.com ([185.176.79.56]:2628) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1o1qrb-0006Kl-Su for qemu-devel@nongnu.org; Thu, 16 Jun 2022 10:52:53 -0400 Received: from fraeml745-chm.china.huawei.com (unknown [172.18.147.206]) by frasgout.his.huawei.com (SkyGuard) with ESMTP id 4LP4qf1qDSz67b2m; Thu, 16 Jun 2022 22:51:10 +0800 (CST) Received: from lhreml710-chm.china.huawei.com (10.201.108.61) by fraeml745-chm.china.huawei.com (10.206.15.226) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.24; Thu, 16 Jun 2022 16:52:50 +0200 Received: from SecurePC-101-06.china.huawei.com (10.122.247.231) by lhreml710-chm.china.huawei.com (10.201.108.61) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.24; Thu, 16 Jun 2022 15:52:49 +0100 To: , "Michael S . Tsirkin" , "Ben Widawsky" CC: Paolo Bonzini , , , , Peter Maydell , Marcel Apfelbaum , "Igor Mammedov" , Markus Armbruster , "Mark Cave-Ayland" , Adam Manzanares , Tong Zhang , "Shameerali Kolothum Thodi" Subject: [PATCH v11 3/3] docs/cxl: Add switch documentation Date: Thu, 16 Jun 2022 15:51:26 +0100 Message-ID: <20220616145126.8002-4-Jonathan.Cameron@huawei.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20220616145126.8002-1-Jonathan.Cameron@huawei.com> References: <20220616145126.8002-1-Jonathan.Cameron@huawei.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Originating-IP: [10.122.247.231] X-ClientProxiedBy: lhreml741-chm.china.huawei.com (10.201.108.191) To lhreml710-chm.china.huawei.com (10.201.108.61) X-CFilter-Loop: Reflected Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=185.176.79.56; envelope-from=jonathan.cameron@huawei.com; helo=frasgout.his.huawei.com X-Spam_score_int: -25 X-Spam_score: -2.6 X-Spam_bar: -- X-Spam_report: (-2.6 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" Reply-to: Jonathan Cameron From: Jonathan Cameron via X-ZM-MESSAGEID: 1655391379213100001 Content-Type: text/plain; charset="utf-8" Switches were already introduced, but now we support them update the documentation to provide an example in diagram and qemu command line parameter forms. Signed-off-by: Jonathan Cameron --- docs/system/devices/cxl.rst | 88 ++++++++++++++++++++++++++++++++++++- 1 file changed, 86 insertions(+), 2 deletions(-) diff --git a/docs/system/devices/cxl.rst b/docs/system/devices/cxl.rst index bcbfe8c490..a57e4c4e5c 100644 --- a/docs/system/devices/cxl.rst +++ b/docs/system/devices/cxl.rst @@ -118,8 +118,6 @@ and associated component register access via PCI bars. =20 CXL Switch ~~~~~~~~~~ -Not yet implemented in QEMU. - Here we consider a simple CXL switch with only a single virtual hierarchy. Whilst more complex devices exist, their visibility to a particular host is generally the same as for @@ -137,6 +135,10 @@ BARs. The Upstream Port has the configuration interfa= ces for the HDM decoders which route incoming memory accesses to the appropriate downstream port. =20 +A CXL switch is created in a similar fashion to PCI switches +by creating an upstream port (cxl-upstream) and a number of +downstream ports on the internal switch bus (cxl-downstream). + CXL Memory Devices - Type 3 ~~~~~~~~~~~~~~~~~~~~~~~~~~~ CXL type 3 devices use a PCI class code and are intended to be supported @@ -240,6 +242,62 @@ Notes: they will take the Host Physical Addresses of accesses and map them to their own local Device Physical Address Space (DPA). =20 +Example topology involving a switch:: + + |<------------------SYSTEM PHYSICAL ADDRESS MAP (1)----------------->| + | __________ __________________________________ __________ | + | | | | | | | | + | | CFMW 0 | | CXL Fixed Memory Window 1 | | CFMW 1 | | + | | HB0 only | | Configured to interleave memory | | HB1 only | | + | | | | memory accesses across HB0/HB1 | | | | + | |____x_____| |__________________________________| |__________| | + | | | | + | | | | + | | | + Interleave Decoder | | | + Matches this HB | | | + \_____________| |_____________/ + __________|__________ _____|_______________ + | | | | + | CXL HB 0 | | CXL HB 1 | + | HB IntLv Decoders | | HB IntLv Decoders | + | PCI/CXL Root Bus 0c | | PCI/CXL Root Bus 0d | + | | | | + |___x_________________| |_____________________| + | | | | + | + A HB 0 HDM Decoder + matches this Port + ___________|___ + | Root Port 0 | + | Appears in | + | PCI topology | + | As 0c:00.0 | + |___________x___| + | + | + \_____________________ + | + | + --------------------------------------------------- + | Switch 0 USP as PCI 0d:00.0 | + | USP has HDM decoder which direct traffic to | + | appropiate downstream port | + | Switch BUS appears as 0e | + |x__________________________________________________| + | | | | + | | | | + _____|_________ ______|______ ______|_____ ______|_______ + (4)| x | | | | | | | + | CXL Type3 0 | | CXL Type3 1 | | CXL type3 2| | CLX Type 3 3 | + | | | | | | | | + | PMEM0(Vol LSA)| | PMEM1 (...) | | PMEM2 (...)| | PMEM3 (...) | + | Decoder to go | | | | | | | + | from host PA | | PCI 10:00.0 | | PCI 11:00.0| | PCI 12:00.0 | + | to device PA | | | | | | | + | PCI as 0f:00.0| | | | | | | + |_______________| |_____________| |____________| |______________| + Example command lines --------------------- A very simple setup with just one directly attached CXL Type 3 device:: @@ -279,6 +337,32 @@ the CXL Type3 device directly attached (no switches).:: -device cxl-type3,bus=3Droot_port16,memdev=3Dcxl-mem4,lsa=3Dcxl-lsa4,id= =3Dcxl-pmem3 \ -M cxl-fmw.0.targets.0=3Dcxl.1,cxl-fmw.0.targets.1=3Dcxl.2,cxl-fmw.0.siz= e=3D4G,cxl-fmw.0.interleave-granularity=3D8k =20 +An example of 4 devices below a switch suitable for 1, 2 or 4 way interlea= ve:: + + qemu-system-aarch64 -M virt,gic-version=3D3,cxl=3Don -m 4g,maxmem=3D8G,s= lots=3D8 -cpu max \ + ... + -object memory-backend-file,id=3Dcxl-mem0,share=3Don,mem-path=3D/tmp/cxl= test.raw,size=3D256M \ + -object memory-backend-file,id=3Dcxl-mem1,share=3Don,mem-path=3D/tmp/cxl= test1.raw,size=3D256M \ + -object memory-backend-file,id=3Dcxl-mem2,share=3Don,mem-path=3D/tmp/cxl= test2.raw,size=3D256M \ + -object memory-backend-file,id=3Dcxl-mem3,share=3Don,mem-path=3D/tmp/cxl= test3.raw,size=3D256M \ + -object memory-backend-file,id=3Dcxl-lsa0,share=3Don,mem-path=3D/tmp/lsa= 0.raw,size=3D256M \ + -object memory-backend-file,id=3Dcxl-lsa1,share=3Don,mem-path=3D/tmp/lsa= 1.raw,size=3D256M \ + -object memory-backend-file,id=3Dcxl-lsa2,share=3Don,mem-path=3D/tmp/lsa= 2.raw,size=3D256M \ + -object memory-backend-file,id=3Dcxl-lsa3,share=3Don,mem-path=3D/tmp/lsa= 3.raw,size=3D256M \ + -device pxb-cxl,bus_nr=3D12,bus=3Dpcie.0,id=3Dcxl.1 \ + -device cxl-rp,port=3D0,bus=3Dcxl.1,id=3Droot_port0,chassis=3D0,slot=3D0= \ + -device cxl-rp,port=3D1,bus=3Dcxl.1,id=3Droot_port1,chassis=3D0,slot=3D1= \ + -device cxl-upstream,bus=3Droot_port0,id=3Dus0 \ + -device cxl-downstream,port=3D0,bus=3Dus0,id=3Dswport0,chassis=3D0,slot= =3D4 \ + -device cxl-type3,bus=3Dswport0,memdev=3Dcxl-mem0,lsa=3Dcxl-lsa0,id=3Dcx= l-pmem0,size=3D256M \ + -device cxl-downstream,port=3D1,bus=3Dus0,id=3Dswport1,chassis=3D0,slot= =3D5 \ + -device cxl-type3,bus=3Dswport1,memdev=3Dcxl-mem1,lsa=3Dcxl-lsa1,id=3Dcx= l-pmem1,size=3D256M \ + -device cxl-downstream,port=3D2,bus=3Dus0,id=3Dswport2,chassis=3D0,slot= =3D6 \ + -device cxl-type3,bus=3Dswport2,memdev=3Dcxl-mem2,lsa=3Dcxl-lsa2,id=3Dcx= l-pmem2,size=3D256M \ + -device cxl-downstream,port=3D3,bus=3Dus0,id=3Dswport3,chassis=3D0,slot= =3D7 \ + -device cxl-type3,bus=3Dswport3,memdev=3Dcxl-mem3,lsa=3Dcxl-lsa3,id=3Dcx= l-pmem3,size=3D256M \ + -M cxl-fmw.0.targets.0=3Dcxl.1,cxl-fmw.0.size=3D4G,cxl-fmw.0.interleave-= granularity=3D4k + Kernel Configuration Options ---------------------------- =20 --=20 2.32.0