From nobody Mon Feb 9 08:11:51 2026 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1667220822; cv=none; d=zohomail.com; s=zohoarc; b=Hrkv23Wku5Rr5np4vtI8LUaSSx/7+ncPwNtY8LyHe4DpQMcenfBOIabXp1FncnjmzO3gDKt9tspd35w00Du5swAh5wnPFdXtRw7qaWpVC5D0NdK6CUGJbGsfCaO7goHCsMt9Mh33QdLrUjvWGyjjG47V0IWZmAgfrRFrnYHkb2o= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1667220822; h=Content-Type:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=ETyx33uCkrggVRTo3HiD7Tje2UJXCck/b2Qi00flxlo=; b=G+Z2izwIzsBCMc7xCPwM1fWlZQLU/LUnDCjiD6Z5MTeoaJWu4ASvM2HO1UN/JwRR744s/GNKA2vAJ7qWVe7H8PdHezCICvAmPGbMdff7Cn6z2bia7KC0F1QybagHBsbng3AXfA7BlnTYL4ZKg2DN/tsZWd0lCBg4jk6hn/HQbuM= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1667220822426207.66493407333485; Mon, 31 Oct 2022 05:53:42 -0700 (PDT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1opUHf-0006G2-BV; Mon, 31 Oct 2022 08:52:55 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1opUHC-0002ld-Sy for qemu-devel@nongnu.org; Mon, 31 Oct 2022 08:52:27 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1opUH4-0002ah-Rh for qemu-devel@nongnu.org; Mon, 31 Oct 2022 08:52:26 -0400 Received: from mail-wm1-f71.google.com (mail-wm1-f71.google.com [209.85.128.71]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_128_GCM_SHA256) id us-mta-178-0fo0s_wkNiCaNZFWlMLYgA-1; Mon, 31 Oct 2022 08:52:16 -0400 Received: by mail-wm1-f71.google.com with SMTP id s7-20020a1cf207000000b003cf56bad2e2so2212155wmc.9 for ; Mon, 31 Oct 2022 05:52:15 -0700 (PDT) Received: from redhat.com ([2.52.15.189]) by smtp.gmail.com with ESMTPSA id k18-20020adfe3d2000000b00236705daefesm7012407wrm.39.2022.10.31.05.52.13 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 31 Oct 2022 05:52:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1667220737; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=ETyx33uCkrggVRTo3HiD7Tje2UJXCck/b2Qi00flxlo=; b=YpxQIu4KG6OucQidIFlPw7Js+Boif222W/0zS42stKsZP+++eLAtLQkUOSUNyjAH0PSqzE AqDLtyxY9aCzaxI7lYpSsD1XWXgsiU17EdPvqqbIzjbMLKk7e3DkNEafG3HvMExVZmiWNK CPReB3gmP3D8Suo6OFBbHlDfa5nrSSg= X-MC-Unique: 0fo0s_wkNiCaNZFWlMLYgA-1 X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=ETyx33uCkrggVRTo3HiD7Tje2UJXCck/b2Qi00flxlo=; b=o3fDrsF6qyzgp948ia5Hd1ND8EmEQGBx3+KC9QNFoIdb7PgZNHDnaKWffnQ3AvMaOu WOiYBtruQTeKZb3VFlZHhTywjydS2RpNsgh4cDrF3K4iMAc4/cWVGBaXy8D/BPWC62za MRYG8Gz2Gav1HUBcK9ID6erGX+NEcRWoiqUuddexvli1Z8SayZdY8sLe0PR2dp4sHo3Q HoXvuxPOhvrn1RvB/O4OdKsOejwLZiAMzVnET58HppmddYZIrl64cgmA2VnNZ8HkFc/C FFWwDCTR/YLeI31EOIanQpONXtY7+jthrZ2ER1i4fP4C2QTaixHO/c95xeF3WfWqTnGn 5b4Q== X-Gm-Message-State: ACrzQf2jtzXG2DphwAqK3gV2Sr7Pt+Hex7ncuZjbn8YrDOU72LaOsTEl zNKiKBgFDGWd17pD7zeIErlTrcKNCQ5agMHVHB1G33P+yOeEvtR7wJxreCG5NNXtZasVR0hrtk0 +/9mRntiwutzykxzSPZEk6ODIQ44T74Qce0uSKihc0C3Fm8rhUZg2ExGs52Yt X-Received: by 2002:a05:600c:3d13:b0:3cf:4d63:243d with SMTP id bh19-20020a05600c3d1300b003cf4d63243dmr8050022wmb.24.1667220734629; Mon, 31 Oct 2022 05:52:14 -0700 (PDT) X-Google-Smtp-Source: AMsMyM6fYp9GpeO0lOTQUdVR+IGFiC478BGA3OJ8DqwOJ1slI4OB8BikE4llz0/u3ElZ1rhQX5HEeA== X-Received: by 2002:a05:600c:3d13:b0:3cf:4d63:243d with SMTP id bh19-20020a05600c3d1300b003cf4d63243dmr8050002wmb.24.1667220734270; Mon, 31 Oct 2022 05:52:14 -0700 (PDT) Date: Mon, 31 Oct 2022 08:52:11 -0400 From: "Michael S. Tsirkin" To: qemu-devel@nongnu.org Cc: Peter Maydell , Jonathan Cameron , Marcel Apfelbaum , Ben Widawsky Subject: [PULL 31/86] hw/pci-bridge/cxl-upstream: Add a CDAT table access DOE Message-ID: <20221031124928.128475-32-mst@redhat.com> References: <20221031124928.128475-1-mst@redhat.com> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <20221031124928.128475-1-mst@redhat.com> X-Mailer: git-send-email 2.27.0.106.g8ac3dc51b1 X-Mutt-Fcc: =sent Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=170.10.133.124; envelope-from=mst@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -30 X-Spam_score: -3.1 X-Spam_bar: --- X-Spam_report: (-3.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-1.048, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "Qemu-devel" Errors-To: qemu-devel-bounces+importer=patchew.org+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @redhat.com) X-ZM-MESSAGEID: 1667220824500100013 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Jonathan Cameron This Data Object Exchange Mailbox allows software to query the latency and bandwidth between ports on the switch. For now only provide information on routes between the upstream port and each downstream port (not p2p). Signed-off-by: Jonathan Cameron -- Changes since v8: Mostly to match the type 3 equivalent - Move enum out of function and give it a more descriptive namespace. Message-Id: <20221014151045.24781-6-Jonathan.Cameron@huawei.com> Reviewed-by: Michael S. Tsirkin Signed-off-by: Michael S. Tsirkin --- include/hw/cxl/cxl_cdat.h | 1 + hw/pci-bridge/cxl_upstream.c | 195 ++++++++++++++++++++++++++++++++++- 2 files changed, 195 insertions(+), 1 deletion(-) diff --git a/include/hw/cxl/cxl_cdat.h b/include/hw/cxl/cxl_cdat.h index 52c232e912..e9eda00142 100644 --- a/include/hw/cxl/cxl_cdat.h +++ b/include/hw/cxl/cxl_cdat.h @@ -131,6 +131,7 @@ typedef struct CDATSslbisHeader { uint64_t entry_base_unit; } QEMU_PACKED CDATSslbisHeader; =20 +#define CDAT_PORT_ID_USP 0x100 /* Switch Scoped Latency and Bandwidth Entry - CDAT Table 10 */ typedef struct CDATSslbe { uint16_t port_x_id; diff --git a/hw/pci-bridge/cxl_upstream.c b/hw/pci-bridge/cxl_upstream.c index a83a3e81e4..9b8b57df9d 100644 --- a/hw/pci-bridge/cxl_upstream.c +++ b/hw/pci-bridge/cxl_upstream.c @@ -10,11 +10,12 @@ =20 #include "qemu/osdep.h" #include "qemu/log.h" +#include "hw/qdev-properties.h" #include "hw/pci/msi.h" #include "hw/pci/pcie.h" #include "hw/pci/pcie_port.h" =20 -#define CXL_UPSTREAM_PORT_MSI_NR_VECTOR 1 +#define CXL_UPSTREAM_PORT_MSI_NR_VECTOR 2 =20 #define CXL_UPSTREAM_PORT_MSI_OFFSET 0x70 #define CXL_UPSTREAM_PORT_PCIE_CAP_OFFSET 0x90 @@ -28,6 +29,7 @@ typedef struct CXLUpstreamPort { =20 /*< public >*/ CXLComponentState cxl_cstate; + DOECap doe_cdat; } CXLUpstreamPort; =20 CXLComponentState *cxl_usp_to_cstate(CXLUpstreamPort *usp) @@ -60,6 +62,9 @@ static void cxl_usp_dvsec_write_config(PCIDevice *dev, ui= nt32_t addr, static void cxl_usp_write_config(PCIDevice *d, uint32_t address, uint32_t val, int len) { + CXLUpstreamPort *usp =3D CXL_USP(d); + + pcie_doe_write_config(&usp->doe_cdat, address, val, len); pci_bridge_write_config(d, address, val, len); pcie_cap_flr_write_config(d, address, val, len); pcie_aer_write_config(d, address, val, len); @@ -67,6 +72,18 @@ static void cxl_usp_write_config(PCIDevice *d, uint32_t = address, cxl_usp_dvsec_write_config(d, address, val, len); } =20 +static uint32_t cxl_usp_read_config(PCIDevice *d, uint32_t address, int le= n) +{ + CXLUpstreamPort *usp =3D CXL_USP(d); + uint32_t val; + + if (pcie_doe_read_config(&usp->doe_cdat, address, len, &val)) { + return val; + } + + return pci_default_read_config(d, address, len); +} + static void latch_registers(CXLUpstreamPort *usp) { uint32_t *reg_state =3D usp->cxl_cstate.crb.cache_mem_registers; @@ -119,6 +136,167 @@ static void build_dvsecs(CXLComponentState *cxl) REG_LOC_DVSEC_REVID, dvsec); } =20 +static bool cxl_doe_cdat_rsp(DOECap *doe_cap) +{ + CDATObject *cdat =3D &CXL_USP(doe_cap->pdev)->cxl_cstate.cdat; + uint16_t ent; + void *base; + uint32_t len; + CDATReq *req =3D pcie_doe_get_write_mbox_ptr(doe_cap); + CDATRsp rsp; + + cxl_doe_cdat_update(&CXL_USP(doe_cap->pdev)->cxl_cstate, &error_fatal); + assert(cdat->entry_len); + + /* Discard if request length mismatched */ + if (pcie_doe_get_obj_len(req) < + DIV_ROUND_UP(sizeof(CDATReq), sizeof(uint32_t))) { + return false; + } + + ent =3D req->entry_handle; + base =3D cdat->entry[ent].base; + len =3D cdat->entry[ent].length; + + rsp =3D (CDATRsp) { + .header =3D { + .vendor_id =3D CXL_VENDOR_ID, + .data_obj_type =3D CXL_DOE_TABLE_ACCESS, + .reserved =3D 0x0, + .length =3D DIV_ROUND_UP((sizeof(rsp) + len), sizeof(uint32_t)= ), + }, + .rsp_code =3D CXL_DOE_TAB_RSP, + .table_type =3D CXL_DOE_TAB_TYPE_CDAT, + .entry_handle =3D (ent < cdat->entry_len - 1) ? + ent + 1 : CXL_DOE_TAB_ENT_MAX, + }; + + memcpy(doe_cap->read_mbox, &rsp, sizeof(rsp)); + memcpy(doe_cap->read_mbox + DIV_ROUND_UP(sizeof(rsp), sizeof(uint3= 2_t)), + base, len); + + doe_cap->read_mbox_len +=3D rsp.header.length; + + return true; +} + +static DOEProtocol doe_cdat_prot[] =3D { + { CXL_VENDOR_ID, CXL_DOE_TABLE_ACCESS, cxl_doe_cdat_rsp }, + { } +}; + +enum { + CXL_USP_CDAT_SSLBIS_LAT, + CXL_USP_CDAT_SSLBIS_BW, + CXL_USP_CDAT_NUM_ENTRIES +}; + +static int build_cdat_table(CDATSubHeader ***cdat_table, void *priv) +{ + g_autofree CDATSslbis *sslbis_latency =3D NULL; + g_autofree CDATSslbis *sslbis_bandwidth =3D NULL; + CXLUpstreamPort *us =3D CXL_USP(priv); + PCIBus *bus =3D &PCI_BRIDGE(us)->sec_bus; + int devfn, sslbis_size, i; + int count =3D 0; + uint16_t port_ids[256]; + + for (devfn =3D 0; devfn < ARRAY_SIZE(bus->devices); devfn++) { + PCIDevice *d =3D bus->devices[devfn]; + PCIEPort *port; + + if (!d || !pci_is_express(d) || !d->exp.exp_cap) { + continue; + } + + /* + * Whilst the PCI express spec doesn't allow anything other than + * downstream ports on this bus, let us be a little paranoid + */ + if (!object_dynamic_cast(OBJECT(d), TYPE_PCIE_PORT)) { + continue; + } + + port =3D PCIE_PORT(d); + port_ids[count] =3D port->port; + count++; + } + + /* May not yet have any ports - try again later */ + if (count =3D=3D 0) { + return 0; + } + + sslbis_size =3D sizeof(CDATSslbis) + sizeof(*sslbis_latency->sslbe) * = count; + sslbis_latency =3D g_malloc(sslbis_size); + if (!sslbis_latency) { + return -ENOMEM; + } + *sslbis_latency =3D (CDATSslbis) { + .sslbis_header =3D { + .header =3D { + .type =3D CDAT_TYPE_SSLBIS, + .length =3D sslbis_size, + }, + .data_type =3D HMATLB_DATA_TYPE_ACCESS_LATENCY, + .entry_base_unit =3D 10000, + }, + }; + + for (i =3D 0; i < count; i++) { + sslbis_latency->sslbe[i] =3D (CDATSslbe) { + .port_x_id =3D CDAT_PORT_ID_USP, + .port_y_id =3D port_ids[i], + .latency_bandwidth =3D 15, /* 150ns */ + }; + } + + sslbis_bandwidth =3D g_malloc(sslbis_size); + if (!sslbis_bandwidth) { + return 0; + } + *sslbis_bandwidth =3D (CDATSslbis) { + .sslbis_header =3D { + .header =3D { + .type =3D CDAT_TYPE_SSLBIS, + .length =3D sslbis_size, + }, + .data_type =3D HMATLB_DATA_TYPE_ACCESS_BANDWIDTH, + .entry_base_unit =3D 1000, + }, + }; + + for (i =3D 0; i < count; i++) { + sslbis_bandwidth->sslbe[i] =3D (CDATSslbe) { + .port_x_id =3D CDAT_PORT_ID_USP, + .port_y_id =3D port_ids[i], + .latency_bandwidth =3D 16, /* 16 GB/s */ + }; + } + + *cdat_table =3D g_malloc0(sizeof(*cdat_table) * CXL_USP_CDAT_NUM_ENTRI= ES); + if (!*cdat_table) { + return -ENOMEM; + } + + /* Header always at start of structure */ + (*cdat_table)[CXL_USP_CDAT_SSLBIS_LAT] =3D g_steal_pointer(&sslbis_lat= ency); + (*cdat_table)[CXL_USP_CDAT_SSLBIS_BW] =3D g_steal_pointer(&sslbis_band= width); + + return CXL_USP_CDAT_NUM_ENTRIES; +} + +static void free_default_cdat_table(CDATSubHeader **cdat_table, int num, + void *priv) +{ + int i; + + for (i =3D 0; i < num; i++) { + g_free(cdat_table[i]); + } + g_free(cdat_table); +} + static void cxl_usp_realize(PCIDevice *d, Error **errp) { PCIEPort *p =3D PCIE_PORT(d); @@ -161,6 +339,14 @@ static void cxl_usp_realize(PCIDevice *d, Error **errp) PCI_BASE_ADDRESS_MEM_TYPE_64, component_bar); =20 + pcie_doe_init(d, &usp->doe_cdat, cxl_cstate->dvsec_offset, doe_cdat_pr= ot, + true, 1); + + cxl_cstate->cdat.build_cdat_table =3D build_cdat_table; + cxl_cstate->cdat.free_cdat_table =3D free_default_cdat_table; + cxl_cstate->cdat.private =3D d; + cxl_doe_cdat_init(cxl_cstate, errp); + return; =20 err_cap: @@ -179,6 +365,11 @@ static void cxl_usp_exitfn(PCIDevice *d) pci_bridge_exitfn(d); } =20 +static Property cxl_upstream_props[] =3D { + DEFINE_PROP_STRING("cdat", CXLUpstreamPort, cxl_cstate.cdat.filename), + DEFINE_PROP_END_OF_LIST() +}; + static void cxl_upstream_class_init(ObjectClass *oc, void *data) { DeviceClass *dc =3D DEVICE_CLASS(oc); @@ -186,6 +377,7 @@ static void cxl_upstream_class_init(ObjectClass *oc, vo= id *data) =20 k->is_bridge =3D true; k->config_write =3D cxl_usp_write_config; + k->config_read =3D cxl_usp_read_config; k->realize =3D cxl_usp_realize; k->exit =3D cxl_usp_exitfn; k->vendor_id =3D 0x19e5; /* Huawei */ @@ -194,6 +386,7 @@ static void cxl_upstream_class_init(ObjectClass *oc, vo= id *data) set_bit(DEVICE_CATEGORY_BRIDGE, dc->categories); dc->desc =3D "CXL Switch Upstream Port"; dc->reset =3D cxl_usp_reset; + device_class_set_props(dc, cxl_upstream_props); } =20 static const TypeInfo cxl_usp_info =3D { --=20 MST