From nobody Fri May 17 02:41:19 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=gmail.com ARC-Seal: i=1; a=rsa-sha256; t=1689791609; cv=none; d=zohomail.com; s=zohoarc; b=btgX/lZ0ETQLz2so3P3ltpQHrhctGT8UtPBzq+YA7rLhooz/vF26LxGvc6Sg/cCOQkbtOarxe2LRmtJk3W98vnFrHa2kXA6Eqpb5XPCFhbYyuWallnGFaxM2DnLzbxFy0oPsxrtbngPZwUMbqafN3qwev4CTRAk1ki8f7zwnQgM= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1689791609; h=Content-Transfer-Encoding:Cc:Date:From:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:Sender:Subject:To; bh=dZNO0lAS38XkkxRCJG38vZhvleKkSXD1W1EGlNTFzi0=; b=StQVJGWqOrYNecnzBzzUGAJKMEnqt9QcxdZdhz0DFTz6hLxCUAt8RIxe+ixE13AA3matSawJ6GEhAEHctNi2X1l32Oy2ZGkgI2T37toyXQGsKal3hQBG34LDp7JXPHKAisTTiOXsGKRzqHs/fmvDLeX0P92IA3fZFGqKY2zjY44= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1689791609611358.32439116972967; Wed, 19 Jul 2023 11:33:29 -0700 (PDT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1qMByh-0002uS-U7; Wed, 19 Jul 2023 14:32:50 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qMByb-0002ry-Mq for qemu-devel@nongnu.org; Wed, 19 Jul 2023 14:32:42 -0400 Received: from mail-yb1-xb43.google.com ([2607:f8b0:4864:20::b43]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1qMByZ-00053B-Al for qemu-devel@nongnu.org; Wed, 19 Jul 2023 14:32:41 -0400 Received: by mail-yb1-xb43.google.com with SMTP id 3f1490d57ef6-ca9804dc6e4so53621276.0 for ; Wed, 19 Jul 2023 11:32:38 -0700 (PDT) Received: from fedora.mshome.net (pool-173-79-56-208.washdc.fios.verizon.net. [173.79.56.208]) by smtp.gmail.com with ESMTPSA id y80-20020a253253000000b00c4e1589475esm1005053yby.27.2023.07.19.11.32.36 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 19 Jul 2023 11:32:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1689791557; x=1690396357; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date:message-id:reply-to; bh=dZNO0lAS38XkkxRCJG38vZhvleKkSXD1W1EGlNTFzi0=; b=FwyxyfwUr5gOA738WI6fOV/88x+fPhoGOBrTexUy9nm/zDNuYc1d86UfARRShePJoI kDQWpM44BNxm+NyIQd7O6x1b9lFQW19b+nlubzJVuj3lAC83o+5k0YjjTxZPk0dhZdnK y7ithA3VVqGp92AYBemfDrBqfni8YG27PLNDZYVnFgOGdIhnVxdbizNLOfPo7SNHKkKW 3Tjd3yiGIhbA5IjDBcnWpjqZWREMAMVkafoLHLB2f1buEsTHdzFYemrQUvDB9AV4g4I7 dT9TBqdj/35Tn1YLT+P8tq4C4CA+EDK9cmhXBAPwmSwRnHe/AtwMhdqfy4FUPEcsMSwh JntQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1689791557; x=1690396357; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=dZNO0lAS38XkkxRCJG38vZhvleKkSXD1W1EGlNTFzi0=; b=lfYWXdhtDjGxJy4lQlQ9BQpxZEO+l/C6qnC2RTyG0axnxlVLNv7m5OvTFN6JYrD8vB PLr3qKkCQN011QcnQzO8eqk2xshz6d+8WCW5Y0jpmHUYmeuNX45TEMMWBLcL1/nC32yw WI+ZqlRfeFXmdKGIBTJlgvJb6Sb81iwyCPb0QWsqTG2fZt+N34sNfIwz9RluH3PwiEn7 IpvcUutCXfnDFYJ/Zq63ecX1+YcYp5tki+B99baP8a2U8v5p3/Bj/fTNslEiM/24R2SZ vw6Ubr6ChIdIVnYif2CTM5pHyljdxtTK5CEfypxZKXuFFbT0h1hhOkn1Jo1s2LgQ/Hw2 b9Cg== X-Gm-Message-State: ABy/qLZIjKbGb6mLBq2DWsaTyEYYgNXWK5qJamzCAyK8It1aB1fMf0QL Qwu9R/WH2PRdpAvIWnRQVmNvN1PJbnpM X-Google-Smtp-Source: APBJJlEi7akNhCM3CYRV5xw0Zp3j+/QfKtdWXhZWOIxpeR376H/0vtRvBRppO6iC0Jl/tHxUM/Pi+Q== X-Received: by 2002:a25:abec:0:b0:c75:593f:d4ab with SMTP id v99-20020a25abec000000b00c75593fd4abmr3309559ybi.22.1689791557375; Wed, 19 Jul 2023 11:32:37 -0700 (PDT) From: Gregory Price X-Google-Original-From: Gregory Price To: qemu-devel@nongnu.org Cc: jonathan.cameron@huawei.com, linux-cxl@vger.kernel.org, Gregory Price Subject: [RFC] cxl/type3: minimum MHD cci support Date: Wed, 19 Jul 2023 14:32:19 -0400 Message-Id: <20230719183219.8093-1-gregory.price@memverge.com> X-Mailer: git-send-email 2.39.1 MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=2607:f8b0:4864:20::b43; envelope-from=gourry.memverge@gmail.com; helo=mail-yb1-xb43.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @gmail.com) X-ZM-MESSAGEID: 1689791612286100003 Content-Type: text/plain; charset="utf-8" Implement the MHD GET_INFO cci command and add a shared memory region to the type3 device to host the information. Add a helper program to initialize this shared memory region. For now, limit the number of LD's to the number of heads. Later, this limitation will need to be lifted for MH-MLDs. Intended use case: 1. Create the shared memory region 2. Format the shared memory region 3. Launch QEMU with `is_mhd=3Dtrue,mhd_head=3DN,mhd_shmid=3D$shmid` shmid=3D`ipcmk -M 4096 | grep -o -E '[0-9]+' | head -1` cxl_mhd_init 4 $shmid qemu-system-x86_64 \ -nographic \ -accel kvm \ -drive file=3D./mhd.qcow2,format=3Dqcow2,index=3D0,media=3Ddisk,id=3Dhd \ -m 4G,slots=3D4,maxmem=3D8G \ -smp 4 \ -machine type=3Dq35,cxl=3Don,hmat=3Don \ -device pxb-cxl,id=3Dcxl.0,bus=3Dpcie.0,bus_nr=3D52 \ -device cxl-rp,id=3Drp0,bus=3Dcxl.0,chassis=3D0,port=3D0,slot=3D0 \ -object memory-backend-file,id=3Dmem0,mem-path=3D/tmp/mem0,size=3D4G,shar= e=3Dtrue \ -device cxl-type3,bus=3Drp0,volatile-memdev=3Dmem0,id=3Dcxl-mem0,sn=3D666= 66,is_mhd=3Dtrue,mhd_head=3D0,mhd_shmid=3D$shmid \ -M cxl-fmw.0.targets.0=3Dcxl.0,cxl-fmw.0.size=3D4G Comments: base repo: https://gitlab.com/jic23/qemu base branch: cxl-2023-07-17 Originally I wanted to add this as a separate CCI, but I realized this wouldn't work as intended because that would require a separate pci/cxl device in /dev/ to tunnel messages though. This is not how this will present on real devices, so I went away from that. Next I wanted to simply *dynamically* add the command to the existing CCI in the type3 device, but these are statically defined in cxl-mailbox. I settled for simply adding the cci command to the type 3 device by default, and checking for whether `is_mhd` is set in the command. Ultimately, for MHD, they are likely to have a bunch of vendor specific commands associated with them *and* a bunch of vendor specific state. It would be nice to able to have something like "cci_add_command_set()" to the cci-mailbox, and an interface to override certain type3 functions such as read/write (but this is an exercise for a later patch set). --- hw/cxl/cxl-mailbox-utils.c | 53 +++++++++++++++++++++++++++++++ hw/mem/cxl_type3.c | 50 +++++++++++++++++++++++++++++ include/hw/cxl/cxl_device.h | 12 +++++++ tools/cxl/cxl_mhd_init.c | 63 +++++++++++++++++++++++++++++++++++++ tools/cxl/meson.build | 3 ++ tools/meson.build | 1 + 6 files changed, 182 insertions(+) create mode 100644 tools/cxl/cxl_mhd_init.c create mode 100644 tools/cxl/meson.build diff --git a/hw/cxl/cxl-mailbox-utils.c b/hw/cxl/cxl-mailbox-utils.c index 2819914e8d..9ef4d7f5e0 100644 --- a/hw/cxl/cxl-mailbox-utils.c +++ b/hw/cxl/cxl-mailbox-utils.c @@ -84,6 +84,8 @@ enum { #define GET_PHYSICAL_PORT_STATE 0x1 TUNNEL =3D 0x53, #define MANAGEMENT_COMMAND 0x0 + MHD =3D 0x55, + #define GET_MHD_INFO 0x0 }; =20 /* CCI Message Format CXL r3.0 Figure 7-19 */ @@ -1155,6 +1157,56 @@ static CXLRetCode cmd_media_clear_poison(const struc= t cxl_cmd *cmd, return CXL_MBOX_SUCCESS; } =20 +static CXLRetCode cmd_mhd_get_info(const struct cxl_cmd *cmd, + uint8_t *payload_in, + size_t len_in, + uint8_t *payload_out, + size_t *len_out, + CXLCCI *cci) +{ + CXLType3Dev *ct3d =3D CXL_TYPE3(cci->d); + struct { + uint8_t start_ld; + uint8_t ldmap_len; + } QEMU_PACKED *input =3D (void *)payload_in; + + struct { + uint8_t nr_lds; + uint8_t nr_heads; + uint16_t resv1; + uint8_t start_ld; + uint8_t ldmap_len; + uint16_t resv2; + uint8_t ldmap[]; + } QEMU_PACKED *output =3D (void *)payload_out; + + uint8_t start_ld =3D input->start_ld; + uint8_t ldmap_len =3D input->ldmap_len; + uint8_t i; + + if (!ct3d->is_mhd) { + return CXL_MBOX_UNSUPPORTED; + } + + if (start_ld >=3D ct3d->mhd_state->nr_lds) { + return CXL_MBOX_INVALID_INPUT; + } + + output->nr_lds =3D ct3d->mhd_state->nr_lds; + output->nr_heads =3D ct3d->mhd_state->nr_heads; + output->resv1 =3D 0; + output->start_ld =3D start_ld; + output->resv2 =3D 0; + + for (i =3D 0; i < ldmap_len && (start_ld + i) < output->nr_lds; i++) { + output->ldmap[i] =3D ct3d->mhd_state->ldmap[start_ld + i]; + } + output->ldmap_len =3D i; + + *len_out =3D sizeof(*output) + output->ldmap_len; + return CXL_MBOX_SUCCESS; +} + #define IMMEDIATE_CONFIG_CHANGE (1 << 1) #define IMMEDIATE_DATA_CHANGE (1 << 2) #define IMMEDIATE_POLICY_CHANGE (1 << 3) @@ -1195,6 +1247,7 @@ static const struct cxl_cmd cxl_cmd_set[256][256] =3D= { cmd_media_inject_poison, 8, 0 }, [MEDIA_AND_POISON][CLEAR_POISON] =3D { "MEDIA_AND_POISON_CLEAR_POISON", cmd_media_clear_poison, 72, 0 }, + [MHD][GET_MHD_INFO] =3D {"GET_MULTI_HEADED_INFO", cmd_mhd_get_info, 2,= 0}, }; =20 static const struct cxl_cmd cxl_cmd_set_sw[256][256] =3D { diff --git a/hw/mem/cxl_type3.c b/hw/mem/cxl_type3.c index efb7dece80..98282aabab 100644 --- a/hw/mem/cxl_type3.c +++ b/hw/mem/cxl_type3.c @@ -18,6 +18,7 @@ #include "hw/cxl/cxl.h" #include "hw/pci/msix.h" #include "hw/pci/spdm.h" +#include =20 #define DWORD_BYTE 4 =20 @@ -794,6 +795,45 @@ static DOEProtocol doe_spdm_prot[] =3D { { } }; =20 +static bool cxl_setup_mhd(CXLType3Dev *ct3d, Error **errp) +{ + if (ct3d->is_mhd && (!ct3d->mhd_shmid || (ct3d->mhd_head =3D=3D ~(0)))= ) { + error_setg(errp, "is_mhd requires a shared memory region (mhd_shmi= d) and mhd_head < 32"); + return false; + } else if (!ct3d->is_mhd && (ct3d->mhd_shmid || (ct3d->mhd_head =3D=3D= ~(0)))) { + error_setg(errp, "(is_mhd,mhd_head,mhd_shmid) fields must be used = together"); + return false; + } else if (!ct3d->is_mhd) { + return true; + } + + if (ct3d->mhd_head >=3D 32) { + error_setg(errp, "MHD Head ID must be between 0-31"); + return false; + } + + ct3d->mhd_state =3D shmat(ct3d->mhd_shmid, NULL, 0); + if (ct3d->mhd_state =3D=3D (void*)-1) { + ct3d->mhd_state =3D NULL; + error_setg(errp, "Unable to attach MHD State. Check ipcs is valid"= ); + return false; + } + + /* For now, limit the number of heads to the number of LD's (SLD) */ + if (ct3d->mhd_state->nr_heads <=3D ct3d->mhd_head) { + error_setg(errp, "Invalid head ID for multiheaded device."); + return false; + } + + if (ct3d->mhd_state->nr_lds <=3D ct3d->mhd_head) { + error_setg(errp, "MHD Shared state does not have sufficient lds."); + return false; + } + + ct3d->mhd_state->ldmap[ct3d->mhd_head] =3D ct3d->mhd_head; + return true; +} + static void ct3_realize(PCIDevice *pci_dev, Error **errp) { CXLType3Dev *ct3d =3D CXL_TYPE3(pci_dev); @@ -806,6 +846,10 @@ static void ct3_realize(PCIDevice *pci_dev, Error **er= rp) =20 QTAILQ_INIT(&ct3d->error_list); =20 + if (!cxl_setup_mhd(ct3d, errp)) { + return; + } + if (!cxl_setup_memory(ct3d, errp)) { return; } @@ -910,6 +954,9 @@ static void ct3_exit(PCIDevice *pci_dev) if (ct3d->hostvmem) { address_space_destroy(&ct3d->hostvmem_as); } + if (ct3d->mhd_state) { + shmdt(ct3d->mhd_state); + } } =20 static bool cxl_type3_dpa(CXLType3Dev *ct3d, hwaddr host_addr, uint64_t *d= pa) @@ -1067,6 +1114,9 @@ static Property ct3_props[] =3D { DEFINE_PROP_UINT64("sn", CXLType3Dev, sn, UI64_NULL), DEFINE_PROP_STRING("cdat", CXLType3Dev, cxl_cstate.cdat.filename), DEFINE_PROP_UINT16("spdm", CXLType3Dev, spdm_port, 0), + DEFINE_PROP_BOOL("is_mhd", CXLType3Dev, is_mhd, false), + DEFINE_PROP_UINT32("mhd_head", CXLType3Dev, mhd_head, 0), + DEFINE_PROP_UINT32("mhd_shmid", CXLType3Dev, mhd_shmid, 0), DEFINE_PROP_END_OF_LIST(), }; =20 diff --git a/include/hw/cxl/cxl_device.h b/include/hw/cxl/cxl_device.h index c68981b618..ebaa613c7b 100644 --- a/include/hw/cxl/cxl_device.h +++ b/include/hw/cxl/cxl_device.h @@ -406,6 +406,12 @@ typedef struct CXLPoison { typedef QLIST_HEAD(, CXLPoison) CXLPoisonList; #define CXL_POISON_LIST_LIMIT 256 =20 +struct CXLMHD_SharedState { + uint8_t nr_heads; + uint8_t nr_lds; + uint8_t ldmap[]; +}; + struct CXLType3Dev { /* Private */ PCIDevice parent_obj; @@ -440,6 +446,12 @@ struct CXLType3Dev { unsigned int poison_list_cnt; bool poison_list_overflowed; uint64_t poison_list_overflow_ts; + + /* Multi-headed Device */ + bool is_mhd; + uint32_t mhd_head; + uint32_t mhd_shmid; + struct CXLMHD_SharedState *mhd_state; }; =20 #define TYPE_CXL_TYPE3 "cxl-type3" diff --git a/tools/cxl/cxl_mhd_init.c b/tools/cxl/cxl_mhd_init.c new file mode 100644 index 0000000000..1303aa9494 --- /dev/null +++ b/tools/cxl/cxl_mhd_init.c @@ -0,0 +1,63 @@ +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +struct mhd_state { + uint8_t nr_heads; + uint8_t nr_lds; + uint8_t ldmap[]; +}; + +int main(int argc, char *argv[]) { + int shmid =3D 0; + uint32_t heads =3D 0; + struct mhd_state* mhd_state =3D 0; + uint8_t i; + + if (argc !=3D 3) { + printf("usage: cxl_mhd_init \n" + "\theads : number of heads on the device\n" + "\tshmid : /tmp/mytoken.tmp\n"); + return -1; + } + + // must have at least 1 head + heads =3D (uint32_t)atoi(argv[1]); + if (heads =3D=3D 0 || heads > 32) { + printf("bad heads argument (1-32)\n"); + return -1; + } + + shmid =3D (uint32_t)atoi(argv[2]); + if (shmid=3D=3D 0) { + printf("bad shmid argument\n"); + return -1; + } + + mhd_state =3D shmat(shmid, NULL, 0); + if (mhd_state =3D=3D (void*)-1) { + printf("Unable to attach to shared memory\n"); + return -1; + } + + // Initialize the mhd_state + size_t mhd_state_size =3D sizeof(struct mhd_state) + (sizeof(uint8_t) = * heads); + memset(mhd_state, 0, mhd_state_size); + mhd_state->nr_heads =3D heads; + mhd_state->nr_lds =3D heads; + + // Head ID =3D=3D LD ID for now + for (i =3D 0; i < heads; i++) + mhd_state->ldmap[i] =3D i; + + printf("mhd initialized\n"); + shmdt(mhd_state); + return 0; +} diff --git a/tools/cxl/meson.build b/tools/cxl/meson.build new file mode 100644 index 0000000000..218658fe69 --- /dev/null +++ b/tools/cxl/meson.build @@ -0,0 +1,3 @@ +executable('cxl_mhd_init', files('cxl_mhd_init.c'), + install: true, + install_dir: get_option('libexecdir')) diff --git a/tools/meson.build b/tools/meson.build index e69de29bb2..91a1d788cb 100644 --- a/tools/meson.build +++ b/tools/meson.build @@ -0,0 +1 @@ +subdir('cxl') --=20 2.39.1