From nobody Mon Feb 9 23:39:22 2026 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=nongnu.org ARC-Seal: i=1; a=rsa-sha256; t=1647618682; cv=none; d=zohomail.com; s=zohoarc; b=ddRNfJS6tMMl3iPKYJiMQdD59yUWEiwxJwB3bxTf34EylaBdxJtBiErHrKQc30zbRSyz+QNAlRhU5uf46b8xRusFIEZWZjH7jHjzpvIDREI5nbgfDKNvLdA5B2/C7lL0x5B3ovvQfv3CB810OMSnwoESbzKLzbiYBQVNXqBKMYs= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1647618682; h=Content-Type:Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:Reply-To:References:Sender:Subject:To; bh=KQc41azvdEO9wnCQ/l+bOcwwaQOYYD4nul0ge15jXLM=; b=efbYT44NKHpEu2FVzi6HneyXSx327TvIlyr3ZMT1m5ySMr4PqlVS8acJ/hKfRflWRjt39IzUB9CMiv3BGBX2+q4dsY9ummVLgNVG7wg2CC+MGmw18n7LH93x/EcWDMAegNop+UDy1H0B78MIwD/b1BXhBvS1vBc71LbVW3lQ/Jo= ARC-Authentication-Results: i=1; mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 16476186826038.064580013019508; Fri, 18 Mar 2022 08:51:22 -0700 (PDT) Received: from localhost ([::1]:46494 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1nVEsr-00041n-9e for importer@patchew.org; Fri, 18 Mar 2022 11:51:21 -0400 Received: from eggs.gnu.org ([209.51.188.92]:47286) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nVEYN-0006lZ-N4 for qemu-devel@nongnu.org; Fri, 18 Mar 2022 11:30:13 -0400 Received: from frasgout.his.huawei.com ([185.176.79.56]:2455) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nVEYK-00071w-DX for qemu-devel@nongnu.org; Fri, 18 Mar 2022 11:30:11 -0400 Received: from fraeml735-chm.china.huawei.com (unknown [172.18.147.226]) by frasgout.his.huawei.com (SkyGuard) with ESMTP id 4KKnvp4n0vz67KSc; Fri, 18 Mar 2022 23:28:06 +0800 (CST) Received: from lhreml710-chm.china.huawei.com (10.201.108.61) by fraeml735-chm.china.huawei.com (10.206.15.216) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.24; Fri, 18 Mar 2022 16:30:05 +0100 Received: from SecurePC-101-06.china.huawei.com (10.122.247.231) by lhreml710-chm.china.huawei.com (10.201.108.61) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2308.21; Fri, 18 Mar 2022 15:30:04 +0000 To: , , =?UTF-8?q?Alex=20Benn=C3=A9e?= , Marcel Apfelbaum , "Michael S . Tsirkin" , Igor Mammedov , Markus Armbruster CC: , Ben Widawsky , "Peter Maydell" , Shameerali Kolothum Thodi , =?UTF-8?q?Philippe=20Mathieu-Daud=C3=A9?= , Peter Xu , David Hildenbrand , Paolo Bonzini , Saransh Gupta1 , Shreyas Shah , Chris Browy , "Samarth Saxena" , Dan Williams , "Mark Cave-Ayland" Subject: [PATCH v8 46/46] docs/cxl: Add switch documentation Date: Fri, 18 Mar 2022 15:06:35 +0000 Message-ID: <20220318150635.24600-47-Jonathan.Cameron@huawei.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20220318150635.24600-1-Jonathan.Cameron@huawei.com> References: <20220318150635.24600-1-Jonathan.Cameron@huawei.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Originating-IP: [10.122.247.231] X-ClientProxiedBy: lhreml732-chm.china.huawei.com (10.201.108.83) To lhreml710-chm.china.huawei.com (10.201.108.61) X-CFilter-Loop: Reflected Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=185.176.79.56; envelope-from=jonathan.cameron@huawei.com; helo=frasgout.his.huawei.com X-Spam_score_int: -41 X-Spam_score: -4.2 X-Spam_bar: ---- X-Spam_report: (-4.2 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H5=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" Reply-to: Jonathan Cameron From: Jonathan Cameron via X-ZM-MESSAGEID: 1647618682874100001 Content-Type: text/plain; charset="utf-8" Switches were already introduced, but now we support them update the documentation to provide an example in diagram and qemu command line parameter forms. Signed-off-by: Jonathan Cameron --- docs/system/devices/cxl.rst | 88 ++++++++++++++++++++++++++++++++++++- 1 file changed, 86 insertions(+), 2 deletions(-) diff --git a/docs/system/devices/cxl.rst b/docs/system/devices/cxl.rst index 6871c26efd..ebc5fb3862 100644 --- a/docs/system/devices/cxl.rst +++ b/docs/system/devices/cxl.rst @@ -118,8 +118,6 @@ and associated component register access via PCI bars. =20 CXL Switch ~~~~~~~~~~ -Not yet implemented in QEMU. - Here we consider a simple CXL switch with only a single virtual hierarchy. Whilst more complex devices exist, their visibility to a particular host is generally the same as for @@ -137,6 +135,10 @@ BARs. The Upstream Port has the configuration interfa= ces for the HDM decoders which route incoming memory accesses to the appropriate downstream port. =20 +A CXL switch is created in a similar fashion to PCI switches +by creating an upstream port (cxl-upstream) and a number of +downstream ports on the internal switch bus (cxl-downstream). + CXL Memory Devices - Type 3 ~~~~~~~~~~~~~~~~~~~~~~~~~~~ CXL type 3 devices use a PCI class code and are intended to be supported @@ -240,6 +242,62 @@ Notes: they will take the Host Physical Addresses of accesses and map them to their own local Device Physical Address Space (DPA). =20 +Example topology involving a switch:: + + |<------------------SYSTEM PHYSICAL ADDRESS MAP (1)----------------->| + | __________ __________________________________ __________ | + | | | | | | | | + | | CFMW 0 | | CXL Fixed Memory Window 1 | | CFMW 1 | | + | | HB0 only | | Configured to interleave memory | | HB1 only | | + | | | | memory accesses across HB0/HB1 | | | | + | |____x_____| |__________________________________| |__________| | + | | | | + | | | | + | | | + Interleave Decoder | | | + Matches this HB | | | + \_____________| |_____________/ + __________|__________ _____|_______________ + | | | | + | CXL HB 0 | | CXL HB 1 | + | HB IntLv Decoders | | HB IntLv Decoders | + | PCI/CXL Root Bus 0c | | PCI/CXL Root Bus 0d | + | | | | + |___x_________________| |_____________________| + | | | | + | + A HB 0 HDM Decoder + matches this Port + ___________|___ + | Root Port 0 | + | Appears in | + | PCI topology | + | As 0c:00.0 | + |___________x___| + | + | + \_____________________ + | + | + --------------------------------------------------- + | Switch 0 USP as PCI 0d:00.0 | + | USP has HDM decoder which direct traffic to | + | appropiate downstream port | + | Switch BUS appears as 0e | + |x__________________________________________________| + | | | | + | | | | + _____|_________ ______|______ ______|_____ ______|_______ + (4)| x | | | | | | | + | CXL Type3 0 | | CXL Type3 1 | | CXL type3 2| | CLX Type 3 3 | + | | | | | | | | + | PMEM0(Vol LSA)| | PMEM1 (...) | | PMEM2 (...)| | PMEM3 (...) | + | Decoder to go | | | | | | | + | from host PA | | PCI 10:00.0 | | PCI 11:00.0| | PCI 12:00.0 | + | to device PA | | | | | | | + | PCI as 0f:00.0| | | | | | | + |_______________| |_____________| |____________| |______________| + Example command lines --------------------- A very simple setup with just one directly attached CXL Type 3 device:: @@ -279,6 +337,32 @@ the CXL Type3 device directly attached (no switches).:: -device cxl-type3,bus=3Droot_port16,memdev=3Dcxl-mem4,lsa=3Dcxl-lsa4,id= =3Dcxl-pmem3,size=3D256M \ -cxl-fixed-memory-window targets.0=3Dcxl.1,targets.1=3Dcxl.2,size=3D4G,i= nterleave-granularity=3D8k =20 +An example of 4 devices below a switch suitable for 1, 2 or 4 way interlea= ve:: + + qemu-system-aarch64 -M virt,gic-version=3D3,cxl=3Don -m 4g,maxmem=3D8G,s= lots=3D8 -cpu max \ + ... + -object memory-backend-file,id=3Dcxl-mem0,share=3Don,mem-path=3D/tmp/cxl= test.raw,size=3D256M \ + -object memory-backend-file,id=3Dcxl-mem1,share=3Don,mem-path=3D/tmp/cxl= test1.raw,size=3D256M \ + -object memory-backend-file,id=3Dcxl-mem2,share=3Don,mem-path=3D/tmp/cxl= test2.raw,size=3D256M \ + -object memory-backend-file,id=3Dcxl-mem3,share=3Don,mem-path=3D/tmp/cxl= test3.raw,size=3D256M \ + -object memory-backend-file,id=3Dcxl-lsa0,share=3Don,mem-path=3D/tmp/lsa= 0.raw,size=3D256M \ + -object memory-backend-file,id=3Dcxl-lsa1,share=3Don,mem-path=3D/tmp/lsa= 1.raw,size=3D256M \ + -object memory-backend-file,id=3Dcxl-lsa2,share=3Don,mem-path=3D/tmp/lsa= 2.raw,size=3D256M \ + -object memory-backend-file,id=3Dcxl-lsa3,share=3Don,mem-path=3D/tmp/lsa= 3.raw,size=3D256M \ + -device pxb-cxl,bus_nr=3D12,bus=3Dpcie.0,id=3Dcxl.1 \ + -device cxl-rp,port=3D0,bus=3Dcxl.1,id=3Droot_port0,chassis=3D0,slot=3D0= \ + -device cxl-rp,port=3D1,bus=3Dcxl.1,id=3Droot_port1,chassis=3D0,slot=3D1= \ + -device cxl-upstream,bus=3Droot_port0,id=3Dus0 \ + -device cxl-downstream,port=3D0,bus=3Dus0,id=3Dswport0,chassis=3D0,slot= =3D4 \ + -device cxl-type3,bus=3Dswport0,memdev=3Dcxl-mem0,lsa=3Dcxl-lsa0,id=3Dcx= l-pmem0,size=3D256M \ + -device cxl-downstream,port=3D1,bus=3Dus0,id=3Dswport1,chassis=3D0,slot= =3D5 \ + -device cxl-type3,bus=3Dswport1,memdev=3Dcxl-mem1,lsa=3Dcxl-lsa1,id=3Dcx= l-pmem1,size=3D256M \ + -device cxl-downstream,port=3D2,bus=3Dus0,id=3Dswport2,chassis=3D0,slot= =3D6 \ + -device cxl-type3,bus=3Dswport2,memdev=3Dcxl-mem2,lsa=3Dcxl-lsa2,id=3Dcx= l-pmem2,size=3D256M \ + -device cxl-downstream,port=3D3,bus=3Dus0,id=3Dswport3,chassis=3D0,slot= =3D7 \ + -device cxl-type3,bus=3Dswport3,memdev=3Dcxl-mem3,lsa=3Dcxl-lsa3,id=3Dcx= l-pmem3,size=3D256M \ + -cxl-fixed-memory-window targets.0=3Dcxl.1,size=3D4G,interleave-granular= ity=3D4k + =20 Kernel Configuration Options ---------------------------- =20 --=20 2.32.0