From nobody Wed Sep 3 02:21:44 2025 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=nongnu.org ARC-Seal: i=1; a=rsa-sha256; t=1649087372; cv=none; d=zohomail.com; s=zohoarc; b=bHqzQXaiMc0TiWxQsNMN2kA6/ISQtzX6lmLPMrfhHLfR6iP/N//Ozg0ftHg4lSzJ3fo1XQrQPlaeCJicTR0CP2uZwGLcoDpGmREUGmZld90hfDs7zfodmVH/nC8IN6hFStzBM9oYoLMM8pBN3RVx94O4L9Uh8/gjzbyf87SONgs= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1649087372; h=Content-Type:Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:Reply-To:References:Sender:Subject:To; bh=2/HAj5u48iFKmVYItxoz4ei9tWQ0L855Acu4x4MCDZs=; b=lgtjsKEaQ4t61kfFL4ejcAEUmYyKbO0hGbyWRC87wohqyIOVKTZXT2YTbp1/Qt3F4KlfJtvgII3bg/+WNCRESxJMGOUNh4fby4lsOoxsEvyonqmFbDa67ZQO/Zj0HcByVKSTnRBljH55tu0/uAU5qknY9FkefQ8GBrAop93Pk4o= ARC-Authentication-Results: i=1; mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1649087372641812.8964632579701; Mon, 4 Apr 2022 08:49:32 -0700 (PDT) Received: from localhost ([::1]:43746 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1nbOxP-0007Ke-96 for importer@patchew.org; Mon, 04 Apr 2022 11:49:31 -0400 Received: from eggs.gnu.org ([209.51.188.92]:42090) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nbOm3-0002eY-U3 for qemu-devel@nongnu.org; Mon, 04 Apr 2022 11:37:47 -0400 Received: from frasgout.his.huawei.com ([185.176.79.56]:2512) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nbOm2-0007Un-3D for qemu-devel@nongnu.org; Mon, 04 Apr 2022 11:37:47 -0400 Received: from fraeml711-chm.china.huawei.com (unknown [172.18.147.201]) by frasgout.his.huawei.com (SkyGuard) with ESMTP id 4KXFGl4tw3z67NNV; Mon, 4 Apr 2022 23:35:43 +0800 (CST) Received: from lhreml710-chm.china.huawei.com (10.201.108.61) by fraeml711-chm.china.huawei.com (10.206.15.60) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.24; Mon, 4 Apr 2022 17:37:43 +0200 Received: from SecurePC-101-06.china.huawei.com (10.122.247.231) by lhreml710-chm.china.huawei.com (10.201.108.61) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.24; Mon, 4 Apr 2022 16:37:42 +0100 To: , , =?UTF-8?q?Alex=20Benn=C3=A9e?= , Marcel Apfelbaum , "Michael S . Tsirkin" , Igor Mammedov , Markus Armbruster , "Mark Cave-Ayland" , Adam Manzanares CC: , Ben Widawsky , "Peter Maydell" , Shameerali Kolothum Thodi , =?UTF-8?q?Philippe=20Mathieu-Daud=C3=A9?= , Peter Xu , David Hildenbrand , Paolo Bonzini , Saransh Gupta1 , Shreyas Shah , Chris Browy , "Samarth Saxena" , Dan Williams , "k . jensen @ samsung . com" , Tong Zhang , , Alison Schofield Subject: [PATCH v9 45/45] docs/cxl: Add switch documentation Date: Mon, 4 Apr 2022 16:14:45 +0100 Message-ID: <20220404151445.10955-46-Jonathan.Cameron@huawei.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20220404151445.10955-1-Jonathan.Cameron@huawei.com> References: <20220404151445.10955-1-Jonathan.Cameron@huawei.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Originating-IP: [10.122.247.231] X-ClientProxiedBy: lhreml717-chm.china.huawei.com (10.201.108.68) To lhreml710-chm.china.huawei.com (10.201.108.61) X-CFilter-Loop: Reflected Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=185.176.79.56; envelope-from=jonathan.cameron@huawei.com; helo=frasgout.his.huawei.com X-Spam_score_int: -25 X-Spam_score: -2.6 X-Spam_bar: -- X-Spam_report: (-2.6 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" Reply-to: Jonathan Cameron From: Jonathan Cameron via X-ZM-MESSAGEID: 1649087375136100003 Content-Type: text/plain; charset="utf-8" Switches were already introduced, but now we support them update the documentation to provide an example in diagram and qemu command line parameter forms. Signed-off-by: Jonathan Cameron --- docs/system/devices/cxl.rst | 88 ++++++++++++++++++++++++++++++++++++- 1 file changed, 86 insertions(+), 2 deletions(-) diff --git a/docs/system/devices/cxl.rst b/docs/system/devices/cxl.rst index 9293cbf01a..abf7c1f243 100644 --- a/docs/system/devices/cxl.rst +++ b/docs/system/devices/cxl.rst @@ -118,8 +118,6 @@ and associated component register access via PCI bars. =20 CXL Switch ~~~~~~~~~~ -Not yet implemented in QEMU. - Here we consider a simple CXL switch with only a single virtual hierarchy. Whilst more complex devices exist, their visibility to a particular host is generally the same as for @@ -137,6 +135,10 @@ BARs. The Upstream Port has the configuration interfa= ces for the HDM decoders which route incoming memory accesses to the appropriate downstream port. =20 +A CXL switch is created in a similar fashion to PCI switches +by creating an upstream port (cxl-upstream) and a number of +downstream ports on the internal switch bus (cxl-downstream). + CXL Memory Devices - Type 3 ~~~~~~~~~~~~~~~~~~~~~~~~~~~ CXL type 3 devices use a PCI class code and are intended to be supported @@ -240,6 +242,62 @@ Notes: they will take the Host Physical Addresses of accesses and map them to their own local Device Physical Address Space (DPA). =20 +Example topology involving a switch:: + + |<------------------SYSTEM PHYSICAL ADDRESS MAP (1)----------------->| + | __________ __________________________________ __________ | + | | | | | | | | + | | CFMW 0 | | CXL Fixed Memory Window 1 | | CFMW 1 | | + | | HB0 only | | Configured to interleave memory | | HB1 only | | + | | | | memory accesses across HB0/HB1 | | | | + | |____x_____| |__________________________________| |__________| | + | | | | + | | | | + | | | + Interleave Decoder | | | + Matches this HB | | | + \_____________| |_____________/ + __________|__________ _____|_______________ + | | | | + | CXL HB 0 | | CXL HB 1 | + | HB IntLv Decoders | | HB IntLv Decoders | + | PCI/CXL Root Bus 0c | | PCI/CXL Root Bus 0d | + | | | | + |___x_________________| |_____________________| + | | | | + | + A HB 0 HDM Decoder + matches this Port + ___________|___ + | Root Port 0 | + | Appears in | + | PCI topology | + | As 0c:00.0 | + |___________x___| + | + | + \_____________________ + | + | + --------------------------------------------------- + | Switch 0 USP as PCI 0d:00.0 | + | USP has HDM decoder which direct traffic to | + | appropiate downstream port | + | Switch BUS appears as 0e | + |x__________________________________________________| + | | | | + | | | | + _____|_________ ______|______ ______|_____ ______|_______ + (4)| x | | | | | | | + | CXL Type3 0 | | CXL Type3 1 | | CXL type3 2| | CLX Type 3 3 | + | | | | | | | | + | PMEM0(Vol LSA)| | PMEM1 (...) | | PMEM2 (...)| | PMEM3 (...) | + | Decoder to go | | | | | | | + | from host PA | | PCI 10:00.0 | | PCI 11:00.0| | PCI 12:00.0 | + | to device PA | | | | | | | + | PCI as 0f:00.0| | | | | | | + |_______________| |_____________| |____________| |______________| + Example command lines --------------------- A very simple setup with just one directly attached CXL Type 3 device:: @@ -279,6 +337,32 @@ the CXL Type3 device directly attached (no switches).:: -device cxl-type3,bus=3Droot_port16,memdev=3Dcxl-mem4,lsa=3Dcxl-lsa4,id= =3Dcxl-pmem3 \ -cxl-fixed-memory-window targets.0=3Dcxl.1,targets.1=3Dcxl.2,size=3D4G,i= nterleave-granularity=3D8k =20 +An example of 4 devices below a switch suitable for 1, 2 or 4 way interlea= ve:: + + qemu-system-aarch64 -M virt,gic-version=3D3,cxl=3Don -m 4g,maxmem=3D8G,s= lots=3D8 -cpu max \ + ... + -object memory-backend-file,id=3Dcxl-mem0,share=3Don,mem-path=3D/tmp/cxl= test.raw,size=3D256M \ + -object memory-backend-file,id=3Dcxl-mem1,share=3Don,mem-path=3D/tmp/cxl= test1.raw,size=3D256M \ + -object memory-backend-file,id=3Dcxl-mem2,share=3Don,mem-path=3D/tmp/cxl= test2.raw,size=3D256M \ + -object memory-backend-file,id=3Dcxl-mem3,share=3Don,mem-path=3D/tmp/cxl= test3.raw,size=3D256M \ + -object memory-backend-file,id=3Dcxl-lsa0,share=3Don,mem-path=3D/tmp/lsa= 0.raw,size=3D256M \ + -object memory-backend-file,id=3Dcxl-lsa1,share=3Don,mem-path=3D/tmp/lsa= 1.raw,size=3D256M \ + -object memory-backend-file,id=3Dcxl-lsa2,share=3Don,mem-path=3D/tmp/lsa= 2.raw,size=3D256M \ + -object memory-backend-file,id=3Dcxl-lsa3,share=3Don,mem-path=3D/tmp/lsa= 3.raw,size=3D256M \ + -device pxb-cxl,bus_nr=3D12,bus=3Dpcie.0,id=3Dcxl.1 \ + -device cxl-rp,port=3D0,bus=3Dcxl.1,id=3Droot_port0,chassis=3D0,slot=3D0= \ + -device cxl-rp,port=3D1,bus=3Dcxl.1,id=3Droot_port1,chassis=3D0,slot=3D1= \ + -device cxl-upstream,bus=3Droot_port0,id=3Dus0 \ + -device cxl-downstream,port=3D0,bus=3Dus0,id=3Dswport0,chassis=3D0,slot= =3D4 \ + -device cxl-type3,bus=3Dswport0,memdev=3Dcxl-mem0,lsa=3Dcxl-lsa0,id=3Dcx= l-pmem0,size=3D256M \ + -device cxl-downstream,port=3D1,bus=3Dus0,id=3Dswport1,chassis=3D0,slot= =3D5 \ + -device cxl-type3,bus=3Dswport1,memdev=3Dcxl-mem1,lsa=3Dcxl-lsa1,id=3Dcx= l-pmem1,size=3D256M \ + -device cxl-downstream,port=3D2,bus=3Dus0,id=3Dswport2,chassis=3D0,slot= =3D6 \ + -device cxl-type3,bus=3Dswport2,memdev=3Dcxl-mem2,lsa=3Dcxl-lsa2,id=3Dcx= l-pmem2,size=3D256M \ + -device cxl-downstream,port=3D3,bus=3Dus0,id=3Dswport3,chassis=3D0,slot= =3D7 \ + -device cxl-type3,bus=3Dswport3,memdev=3Dcxl-mem3,lsa=3Dcxl-lsa3,id=3Dcx= l-pmem3,size=3D256M \ + -cxl-fixed-memory-window targets.0=3Dcxl.1,size=3D4G,interleave-granular= ity=3D4k + Kernel Configuration Options ---------------------------- =20 --=20 2.32.0