From nobody Sat May 18 07:48:33 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=gmail.com ARC-Seal: i=1; a=rsa-sha256; t=1666745373; cv=none; d=zohomail.com; s=zohoarc; b=C9UBQsfSZHmyS1p6nwKjI5MHOHFZOxPzaz9Se3HrW6CjQ4jD9wBGTYL0js36I2FghzrEkmBIC7VJQsGJs9kPLtmx+KMRv7JnavcBp5wY+uk9Z7KYmTelQrvW8YQ5PdEhQjTdTI7sEy/zlnkS2KJuPE6X0TV9uEjWGolqDymA40k= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1666745373; h=Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=d7/73U88aWKwNDfroPuUtEdVueHFIV0vL1R64OOolus=; b=CZPUm+gjevJ1muAXBq3MY+SbiwCnidl1KYZCZig9s9rU2oSaLdImov3DSLP9QlBVYo43WvGdTlmA4pO2XIpH8zLONwAu2vglQvq5ZkKNwhm+fEW8GU/8uquc1cnWVOilrrpx44jQfUaAW3DvkaBdFeo6aTYHf7iXCpsxfixfaaY= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1666745373052107.20493340090184; Tue, 25 Oct 2022 17:49:33 -0700 (PDT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1onUbU-0004AG-AN; Tue, 25 Oct 2022 20:49:08 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1onUbT-000411-3f for qemu-devel@nongnu.org; Tue, 25 Oct 2022 20:49:07 -0400 Received: from mail-qt1-x842.google.com ([2607:f8b0:4864:20::842]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1onUbR-0003YP-FZ for qemu-devel@nongnu.org; Tue, 25 Oct 2022 20:49:06 -0400 Received: by mail-qt1-x842.google.com with SMTP id c23so8901953qtw.8 for ; Tue, 25 Oct 2022 17:49:05 -0700 (PDT) Received: from fedora.mshome.net (pool-173-79-56-208.washdc.fios.verizon.net. [173.79.56.208]) by smtp.gmail.com with ESMTPSA id j15-20020ac8550f000000b0039a610a04b1sm2408651qtq.37.2022.10.25.17.48.42 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 25 Oct 2022 17:48:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=d7/73U88aWKwNDfroPuUtEdVueHFIV0vL1R64OOolus=; b=FKmI7MFmg0vUcBItLfBUZyX5abdLhN+mCsU8k+WVfoqdr5hHERIc4Wvq3lZPrstRXg UU/idLJ5EniJ8I8PEU5jOTkhYDuLRapTw3SvwHkTc47NGprU07JIINHnCgL6OS9KkR44 vjFhw/GBNtDqLloY8Y9yKrAMJL+N00be2BQVNo1P7bPZGnw8xBlVJt8Pzyyg0/NoXIJH Z0aKwO26IYifO0lEu7wQCKDzIwVosxDU7NfS8uVYgVhO6YiRQ62OMeQNrv+hVNem51D5 VlMp54OW5qcLwmsyfB/Bz7PfKRfnLp72A6+FQUOCUXmIL+VJ8unKmOnEQPKTU/7vA68h kd4w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=d7/73U88aWKwNDfroPuUtEdVueHFIV0vL1R64OOolus=; b=Ck0ikaqM0oVr7H5bCQUx4grwrLXXATilI+d0L+qw2Z3e+QV8N8Aj81g5h7QC2UyMkw jgyqmEp4lI6UjshJStOkw0pL/PTtdxPtxXYvdb5hAl/gpyszhwDrT7PRT93CsIGA/L0A 73Bkdbq+xTbD7HeiF8oxoBKcKw9+yC7S1LCC6vDzabK6CUc+HeoaTNAGjK+w5b6cL6kN ddh7iNoHPzXAXpsLIH1whqhH3l6XotpxqvNBdId1nwGvJ4cVcp6Pe+CH7Zt4tEWRZPdG g5TtixLQ77v+Uph1ay4DWdpkPCdLKud+PmeS1wdP0r2VT+uM5RPwNE0RD9+RxRW3IBPQ VuIQ== X-Gm-Message-State: ACrzQf1UdJmjrzpLfV733j0PT2tJOK1spYmxb+SRBMrpPW+Nob+FsID0 mGxBMoPXDAEuxjrckj/gDFsM2TBnxw== X-Google-Smtp-Source: AMsMyM6TE3OJhxhBREvSayYLxzwSr++OBb9psECz2cRe05REWMbChQNHncI+KtTroSDsIzP9zSKLZQ== X-Received: by 2002:ac8:7d0d:0:b0:39c:c5e0:37c0 with SMTP id g13-20020ac87d0d000000b0039cc5e037c0mr33442427qtb.537.1666745343935; Tue, 25 Oct 2022 17:49:03 -0700 (PDT) From: Gregory Price X-Google-Original-From: Gregory Price To: qemu-devel@nongnu.org Cc: jonathan.cameron@huawei.com, linux-cxl@vger.kernel.org, mst@redhat.com, marcel.apfelbaum@gmail.com, imammedo@redhat.com, ani@anisinha.ca, alison.schofield@intel.com, dave@stgolabs.net, a.manzanares@samsung.com, bwidawsk@kernel.org, gregory.price@memverge.com, hchkuo@avery-design.com.tw, cbrowy@avery-design.com, ira.weiny@intel.com Subject: [PATCH 1/4] hw/i386/pc.c: CXL Fixed Memory Window should not reserve e820 in bios Date: Tue, 25 Oct 2022 20:47:34 -0400 Message-Id: <20221026004737.3646-2-gregory.price@memverge.com> X-Mailer: git-send-email 2.37.3 In-Reply-To: <20221026004737.3646-1-gregory.price@memverge.com> References: <20221026004737.3646-1-gregory.price@memverge.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=2607:f8b0:4864:20::842; envelope-from=gourry.memverge@gmail.com; helo=mail-qt1-x842.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "Qemu-devel" Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @gmail.com) X-ZM-MESSAGEID: 1666745375354100007 Content-Type: text/plain; charset="utf-8" Early-boot e820 records will be inserted by the bios/efi/early boot software and be reported to the kernel via insert_resource. Later, when CXL drivers iterate through the regions again, they will insert another resource and make the RESERVED memory area a child. This RESERVED memory area causes the memory region to become unusable, and as a result attempting to create memory regions with `cxl create-region ...` Will fail due to the RESERVED area intersecting with the CXL window. During boot the following traceback is observed: 0xffffffff81101650 in insert_resource_expand_to_fit () 0xffffffff83d964c5 in e820__reserve_resources_late () 0xffffffff83e03210 in pcibios_resource_survey () 0xffffffff83e04f4a in pcibios_init () Which produces a call to reserve the CFMWS area: (gdb) p *new $54 =3D {start =3D 0x290000000, end =3D 0x2cfffffff, name =3D "Reserved", flags =3D 0x200, desc =3D 0x7, parent =3D 0x0, sibling =3D 0x0, child =3D 0x0} Later the Kernel parses ACPI tables and reserves the exact same area as the CXL Fixed Memory Window: 0xffffffff811016a4 in insert_resource_conflict () insert_resource () 0xffffffff81a81389 in cxl_parse_cfmws () 0xffffffff818c4a81 in call_handler () acpi_parse_entries_array () (gdb) p/x *new $59 =3D {start =3D 0x290000000, end =3D 0x2cfffffff, name =3D "CXL Window 0= ", flags =3D 0x200, desc =3D 0x0, parent =3D 0x0, sibling =3D 0x0, child =3D 0x0} This produces the following output in /proc/iomem: 590000000-68fffffff : CXL Window 0 590000000-68fffffff : Reserved This reserved area causes `get_free_mem_region()` to fail due to a check against `__region_intersects()`. Due to this reserved area, the intersect check will only ever return REGION_INTERSECTS, which causes `cxl create-region` to always fail. Signed-off-by: Gregory Price --- hw/i386/pc.c | 2 -- 1 file changed, 2 deletions(-) diff --git a/hw/i386/pc.c b/hw/i386/pc.c index 768982ae9a..203c90fedb 100644 --- a/hw/i386/pc.c +++ b/hw/i386/pc.c @@ -1062,7 +1062,6 @@ void pc_memory_init(PCMachineState *pcms, hwaddr cxl_size =3D MiB; =20 cxl_base =3D pc_get_cxl_range_start(pcms); - e820_add_entry(cxl_base, cxl_size, E820_RESERVED); memory_region_init(mr, OBJECT(machine), "cxl_host_reg", cxl_size); memory_region_add_subregion(system_memory, cxl_base, mr); cxl_resv_end =3D cxl_base + cxl_size; @@ -1078,7 +1077,6 @@ void pc_memory_init(PCMachineState *pcms, memory_region_init_io(&fw->mr, OBJECT(machine), &cfmws_ops= , fw, "cxl-fixed-memory-region", fw->size); memory_region_add_subregion(system_memory, fw->base, &fw->= mr); - e820_add_entry(fw->base, fw->size, E820_RESERVED); cxl_fmw_base +=3D fw->size; cxl_resv_end =3D cxl_fmw_base; } --=20 2.37.3 From nobody Sat May 18 07:48:33 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=gmail.com ARC-Seal: i=1; a=rsa-sha256; t=1666745531; cv=none; d=zohomail.com; s=zohoarc; b=lMnq77ZdPPiD9J1ottvQuh86UgPqlrwinM1qMKGBujC4fvRr6NMjtJJ84jHS2eSSgTMFcBPl6NP6PZ3D9yMTjqnOvv3xqZt0h+fjHDWtz/PNtjFUtuTTG+x7tAJzc4Yo+k3maiXOU+ji2uX+kffwvi/vOc64vC2vF5pdleJQsFI= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1666745531; h=Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=/u9Srr6RXjECMW/z6fGCEplaACCtFUVAvHpD6VsFGYI=; b=DKDprojJrZwlQrG4y9KJnWBmAlMiTuli5PX/QhA89cTsfttZNi8f8tvYKhDcBJ4o+XdtquiEx3Xf3M0wFLHEIlNE1KhlqgtUVlavxAnVQ/7piT9AoErBjMaUka+030KRVjOcxMOIl4ahjqPo9qjtGZW6lXC6wsEp7u0RVkVqRk4= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1666745531255654.1277707558231; Tue, 25 Oct 2022 17:52:11 -0700 (PDT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1onUbl-0005UR-L2; Tue, 25 Oct 2022 20:49:25 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1onUbg-0005LR-By for qemu-devel@nongnu.org; Tue, 25 Oct 2022 20:49:21 -0400 Received: from mail-qv1-xf42.google.com ([2607:f8b0:4864:20::f42]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1onUbe-0003aS-M9 for qemu-devel@nongnu.org; Tue, 25 Oct 2022 20:49:19 -0400 Received: by mail-qv1-xf42.google.com with SMTP id n18so4889419qvt.11 for ; Tue, 25 Oct 2022 17:49:18 -0700 (PDT) Received: from fedora.mshome.net (pool-173-79-56-208.washdc.fios.verizon.net. [173.79.56.208]) by smtp.gmail.com with ESMTPSA id j15-20020ac8550f000000b0039a610a04b1sm2408651qtq.37.2022.10.25.17.49.09 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 25 Oct 2022 17:49:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=/u9Srr6RXjECMW/z6fGCEplaACCtFUVAvHpD6VsFGYI=; b=lb2uPwdsVV8+Qr+6+gRs0GsA/2YzAY3YGpgCAt07Nih5ihzeki8P8UUaaBA6BrAHgB roB4jvw+wtgqFnpXt3hOdgnZePM+QxjVDjSPavlhDDHdvFQAeHMKOELFRI2RKdOBAw3K bF6sJEv31boKoe66WJpxmYhCE59Z6dE9Y7HxK+NGDqxpKCd2pzylEYJ2UF6456wUNUdj haYjnui/Kbv/N4m7ALzWSVCEnbLMET/M3a0GkJbp/P6YE2Tgcf9kEXPBQ+365T9a0CfK 1AbcQfoyNWwR7O+ltUkROuB2g3kDTIUwuYzV+rFqkSW+8ef5L3kks98s+uanuyB4VivB M47g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=/u9Srr6RXjECMW/z6fGCEplaACCtFUVAvHpD6VsFGYI=; b=RdqmNLeO/Gv0uLeaPox3Tm3kT57gYobH2FiInFQ11Xjuc+GlT1G4mHFiBELiy3J6mH Hh7z9gFTwQCV22jBOV+ddTaG+0at1W4o4N8ZKfski6QHtlSNO1yPYGN4W1RUgEZhZKw9 Hrpo05Nn9X6ADIP6Vg5qVXGcfrV5RaclCvCMNbQmVFDQBX+KU71htoU9lLri3nIwdQA6 JcTYT7Zz5Jea/d5jVsNdoeTVcwbRGUYxUgM9Z9XdoWHykhNbhOkkrgYQ6l7joLmqWGCR /VawJwHTIZqdkzn87WiB2hSgwwxnQLzYMjZwQbLLNBZ9g+UMKvTrsAS6uUgzjA0Jk5A4 ITiA== X-Gm-Message-State: ACrzQf1CVXl44bZNojgmtsrz0DlSNF7v9BDz9PH8UdOv9x3BMqeosSBH DDC01WYVcPicJPvqVbFj9iy966OAvQ== X-Google-Smtp-Source: AMsMyM7kWNgW/Qc08xbI2ipjZ7uhFkt3FM7NaxfvqlUw8LQ6GxNqwnQ5fjbsSSstj5XOfsinCWFTng== X-Received: by 2002:ad4:5de8:0:b0:4b1:85db:3509 with SMTP id jn8-20020ad45de8000000b004b185db3509mr34393355qvb.121.1666745356253; Tue, 25 Oct 2022 17:49:16 -0700 (PDT) From: Gregory Price X-Google-Original-From: Gregory Price To: qemu-devel@nongnu.org Cc: jonathan.cameron@huawei.com, linux-cxl@vger.kernel.org, mst@redhat.com, marcel.apfelbaum@gmail.com, imammedo@redhat.com, ani@anisinha.ca, alison.schofield@intel.com, dave@stgolabs.net, a.manzanares@samsung.com, bwidawsk@kernel.org, gregory.price@memverge.com, hchkuo@avery-design.com.tw, cbrowy@avery-design.com, ira.weiny@intel.com Subject: [PATCH 2/4] hw/cxl: Add CXL_CAPACITY_MULTIPLIER definition Date: Tue, 25 Oct 2022 20:47:35 -0400 Message-Id: <20221026004737.3646-3-gregory.price@memverge.com> X-Mailer: git-send-email 2.37.3 In-Reply-To: <20221026004737.3646-1-gregory.price@memverge.com> References: <20221026004737.3646-1-gregory.price@memverge.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=2607:f8b0:4864:20::f42; envelope-from=gourry.memverge@gmail.com; helo=mail-qv1-xf42.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "Qemu-devel" Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @gmail.com) X-ZM-MESSAGEID: 1666745532019100001 Content-Type: text/plain; charset="utf-8" Remove usage of magic numbers when accessing capacity fields and replace with CXL_CAPACITY_MULTIPLIER, matching the kernel definition. Signed-off-by: Gregory Price Reviewed-by: Davidlohr Bueso --- hw/cxl/cxl-mailbox-utils.c | 14 ++++++++------ 1 file changed, 8 insertions(+), 6 deletions(-) diff --git a/hw/cxl/cxl-mailbox-utils.c b/hw/cxl/cxl-mailbox-utils.c index 3e23d29e2d..d7543fd5b4 100644 --- a/hw/cxl/cxl-mailbox-utils.c +++ b/hw/cxl/cxl-mailbox-utils.c @@ -15,6 +15,8 @@ #include "qemu/log.h" #include "qemu/uuid.h" =20 +#define CXL_CAPACITY_MULTIPLIER 0x10000000 /* SZ_256M */ + /* * How to add a new command, example. The command set FOO, with cmd BAR. * 1. Add the command set and cmd to the enum. @@ -267,7 +269,7 @@ static ret_code cmd_firmware_update_get_info(struct cxl= _cmd *cmd, } QEMU_PACKED *fw_info; QEMU_BUILD_BUG_ON(sizeof(*fw_info) !=3D 0x50); =20 - if (cxl_dstate->pmem_size < (256 << 20)) { + if (cxl_dstate->pmem_size < CXL_CAPACITY_MULTIPLIER) { return CXL_MBOX_INTERNAL_ERROR; } =20 @@ -412,7 +414,7 @@ static ret_code cmd_identify_memory_device(struct cxl_c= md *cmd, CXLType3Class *cvc =3D CXL_TYPE3_GET_CLASS(ct3d); uint64_t size =3D cxl_dstate->pmem_size; =20 - if (!QEMU_IS_ALIGNED(size, 256 << 20)) { + if (!QEMU_IS_ALIGNED(size, CXL_CAPACITY_MULTIPLIER)) { return CXL_MBOX_INTERNAL_ERROR; } =20 @@ -422,8 +424,8 @@ static ret_code cmd_identify_memory_device(struct cxl_c= md *cmd, /* PMEM only */ snprintf(id->fw_revision, 0x10, "BWFW VERSION %02d", 0); =20 - id->total_capacity =3D size / (256 << 20); - id->persistent_capacity =3D size / (256 << 20); + id->total_capacity =3D size / CXL_CAPACITY_MULTIPLIER; + id->persistent_capacity =3D size / CXL_CAPACITY_MULTIPLIER; id->lsa_size =3D cvc->get_lsa_size(ct3d); id->poison_list_max_mer[1] =3D 0x1; /* 256 poison records */ =20 @@ -444,14 +446,14 @@ static ret_code cmd_ccls_get_partition_info(struct cx= l_cmd *cmd, QEMU_BUILD_BUG_ON(sizeof(*part_info) !=3D 0x20); uint64_t size =3D cxl_dstate->pmem_size; =20 - if (!QEMU_IS_ALIGNED(size, 256 << 20)) { + if (!QEMU_IS_ALIGNED(size, CXL_CAPACITY_MULTIPLIER)) { return CXL_MBOX_INTERNAL_ERROR; } =20 /* PMEM only */ part_info->active_vmem =3D 0; part_info->next_vmem =3D 0; - part_info->active_pmem =3D size / (256 << 20); + part_info->active_pmem =3D size / CXL_CAPACITY_MULTIPLIER; part_info->next_pmem =3D 0; =20 *len =3D sizeof(*part_info); --=20 2.37.3 From nobody Sat May 18 07:48:33 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=gmail.com ARC-Seal: i=1; a=rsa-sha256; t=1666745471; cv=none; d=zohomail.com; s=zohoarc; b=J+HXMzOkqmFZzZqWC5qQfSVxWN6Yx7H3JsutOjQl27W6Sjl4CkRzsMGKWJ403KoCvlanu/wvq0rfZ+pFfSDvhPVGd4m4FmL6fxLSMq+9owg6zsyJLXgC3FPp+IYTJcHNtrQhkbP7/rvzdPo5W1xaAezePHtdV0NJTYSuGD21CH0= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1666745471; h=Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=3eCdkblp9N10CpvwBlCvOOWSX9hk0WPXXWNuoMHXung=; b=ehLeT3sluxJuoli8RZUeVMCoImOXw+PYnHeRkTlphh8Ek5EtNO/jNKzxvmo39xi48uWHKaI+G9f0b4QSTLvzsFSBDjXwff7xGs2yvKh0WAsABK8Er+p+U49XZrjhSOdL/GjgIF5WxEWtXVD/+r0atsY6Yt8bpQz04XVttuYP8LU= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1666745471957801.7628690879324; Tue, 25 Oct 2022 17:51:11 -0700 (PDT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1onUbs-0005tn-8U; Tue, 25 Oct 2022 20:49:32 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1onUbq-0005me-VS for qemu-devel@nongnu.org; Tue, 25 Oct 2022 20:49:31 -0400 Received: from mail-qt1-x843.google.com ([2607:f8b0:4864:20::843]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1onUbn-0003bG-SS for qemu-devel@nongnu.org; Tue, 25 Oct 2022 20:49:30 -0400 Received: by mail-qt1-x843.google.com with SMTP id bb5so8888601qtb.11 for ; Tue, 25 Oct 2022 17:49:27 -0700 (PDT) Received: from fedora.mshome.net (pool-173-79-56-208.washdc.fios.verizon.net. [173.79.56.208]) by smtp.gmail.com with ESMTPSA id j15-20020ac8550f000000b0039a610a04b1sm2408651qtq.37.2022.10.25.17.49.18 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 25 Oct 2022 17:49:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=3eCdkblp9N10CpvwBlCvOOWSX9hk0WPXXWNuoMHXung=; b=W+dtK43wYZKfPY1up96M7gKPiJYFd8iwcrFUuMybsCXNlHKyg/SUGOUZlXi5vB7JyI pXAWEvjiHlHIC4t8JxXZdncwYoHheg0ZIj0UOMpkHEAWrPAY+GXMwGTsfuH7gScKmDvH o0o+hzrtUTp3zm9VP+o1Pzqs94ljVpZSmb8Q2cKWBiB50nOpVID6Vs0/DbwJMOaUb6HT kDgn1VRH/b1foa7SG9dSmRjpiUNzTzHVSLVyYUVb6oFo/WaSRIcBYIfViPr9LsHZ1Tur RnyFA9wNVwBDAUZA6sNPyiFHMUGK2vx7dA6yvlg1uiYEiQ5ve/XI9L3BflXTxapr4bRV lmaA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=3eCdkblp9N10CpvwBlCvOOWSX9hk0WPXXWNuoMHXung=; b=iVIhIMNCLYwEUghrUbXuoYYFW2SQhbOEe1uj14abe7PqPRWFJKqB8p6REA1dijLcJC zweutfMG6/FbxjMeI5ube75Crkgr2rUuLeMDcPHoNGvejp+0hNGuDRldisEq8wpGLHrq QjuXweW8gyWwKeLUSW9sz8geU/cfjWFlSaERjRquO8pa1T2eEw5XZpFB87LsSmW6I4MN 53JWWOsl2aUW4MmTbMX/03RBx2bpw1MFoLutwwevg4psk+1ByCEqXYX524T5aD4Kt3YR 7imCeOHuWe9Vgdbcf7msAKSfhhugmtdUVoFzhw6ksEMXI12/Y+ZCgcXYnqNzcL8EKVgo mcSA== X-Gm-Message-State: ACrzQf1wH6bZn/0jSfuYKFIodsM97RiCVgifEHhfKCScbSlmFd2X8aZl roz7GQBhfsq+eD/bKqws8LLe9N1tag== X-Google-Smtp-Source: AMsMyM4KdvUGjoK1qeOBgzRxPNIzo2UXmA4fZ6uat3Ig2AevStpprICpMTj37vtwKCuwZNDk5PT3Qw== X-Received: by 2002:ac8:7f54:0:b0:39c:bb57:a1aa with SMTP id g20-20020ac87f54000000b0039cbb57a1aamr34896534qtk.382.1666745366008; Tue, 25 Oct 2022 17:49:26 -0700 (PDT) From: Gregory Price X-Google-Original-From: Gregory Price To: qemu-devel@nongnu.org Cc: jonathan.cameron@huawei.com, linux-cxl@vger.kernel.org, mst@redhat.com, marcel.apfelbaum@gmail.com, imammedo@redhat.com, ani@anisinha.ca, alison.schofield@intel.com, dave@stgolabs.net, a.manzanares@samsung.com, bwidawsk@kernel.org, gregory.price@memverge.com, hchkuo@avery-design.com.tw, cbrowy@avery-design.com, ira.weiny@intel.com Subject: [PATCH 3/4] hw/cxl: Multi-Region CXL Type-3 Devices (Volatile and Persistent) Date: Tue, 25 Oct 2022 20:47:36 -0400 Message-Id: <20221026004737.3646-4-gregory.price@memverge.com> X-Mailer: git-send-email 2.37.3 In-Reply-To: <20221026004737.3646-1-gregory.price@memverge.com> References: <20221026004737.3646-1-gregory.price@memverge.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=2607:f8b0:4864:20::843; envelope-from=gourry.memverge@gmail.com; helo=mail-qt1-x843.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "Qemu-devel" Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @gmail.com) X-ZM-MESSAGEID: 1666745473829100003 Content-Type: text/plain; charset="utf-8" This commit enables each CXL Type-3 device to contain one volatile memory region and one persistent region. Two new properties have been added to cxl-type3 device initialization: [volatile-memdev] and [persistent-memdev] The existing [memdev] property has been deprecated and will default the memory region to a persistent memory region (although a user may assign the region to a ram or file backed region). It cannot be used in combination with the new [persistent-memdev] property. Partitioning volatile memory from persistent memory is not yet supported. Volatile memory is mapped at DPA(0x0), while Persistent memory is mapped at DPA(vmem->size), per CXL Spec 8.2.9.8.2.0 - Get Partition Info. Signed-off-by: Gregory Price --- docs/system/devices/cxl.rst | 53 +++++-- hw/cxl/cxl-mailbox-utils.c | 21 ++- hw/mem/cxl_type3.c | 274 +++++++++++++++++++++++++++--------- include/hw/cxl/cxl_device.h | 11 +- tests/qtest/cxl-test.c | 111 +++++++++++---- 5 files changed, 348 insertions(+), 122 deletions(-) diff --git a/docs/system/devices/cxl.rst b/docs/system/devices/cxl.rst index f25783a4ec..9e165064c8 100644 --- a/docs/system/devices/cxl.rst +++ b/docs/system/devices/cxl.rst @@ -300,15 +300,36 @@ Example topology involving a switch:: =20 Example command lines --------------------- -A very simple setup with just one directly attached CXL Type 3 device:: +A very simple setup with just one directly attached CXL Type 3 Persistent = Memory device:: =20 qemu-system-aarch64 -M virt,gic-version=3D3,cxl=3Don -m 4g,maxmem=3D8G,s= lots=3D8 -cpu max \ ... - -object memory-backend-file,id=3Dcxl-mem1,share=3Don,mem-path=3D/tmp/cxl= test.raw,size=3D256M \ - -object memory-backend-file,id=3Dcxl-lsa1,share=3Don,mem-path=3D/tmp/lsa= .raw,size=3D256M \ + -object memory-backend-file,pmem=3Dtrue,id=3Dpmem0,share=3Don,mem-path= =3D/tmp/cxltest.raw,size=3D256M \ + -object memory-backend-file,pmem=3Dtrue,id=3Dcxl-lsa0,share=3Don,mem-pat= h=3D/tmp/lsa.raw,size=3D256M \ + -device pxb-cxl,bus_nr=3D12,bus=3Dpcie.0,id=3Dcxl.1 \ + -device cxl-rp,port=3D0,bus=3Dcxl.1,id=3Droot_port13,chassis=3D0,slot=3D= 2 \ + -device cxl-type3,bus=3Droot_port13,persistent-memdev=3Dpmem0,lsa=3Dcxl-= lsa1,id=3Dcxl-pmem0 \ + -M cxl-fmw.0.targets.0=3Dcxl.1,cxl-fmw.0.size=3D4G + +A very simple setup with just one directly attached CXL Type 3 Volatile Me= mory device:: + + qemu-system-aarch64 -M virt,gic-version=3D3,cxl=3Don -m 4g,maxmem=3D8G,s= lots=3D8 -cpu max \ + ... + -object memory-backend-ram,id=3Dvmem0,share=3Don,size=3D256M \ -device pxb-cxl,bus_nr=3D12,bus=3Dpcie.0,id=3Dcxl.1 \ -device cxl-rp,port=3D0,bus=3Dcxl.1,id=3Droot_port13,chassis=3D0,slot=3D= 2 \ - -device cxl-type3,bus=3Droot_port13,memdev=3Dcxl-mem1,lsa=3Dcxl-lsa1,id= =3Dcxl-pmem0 \ + -device cxl-type3,bus=3Droot_port13,volatile-memdev=3Dvmem0,id=3Dcxl-vme= m0 \ + -M cxl-fmw.0.targets.0=3Dcxl.1,cxl-fmw.0.size=3D4G + +The same volatile setup may optionally include an LSA region:: + + qemu-system-aarch64 -M virt,gic-version=3D3,cxl=3Don -m 4g,maxmem=3D8G,s= lots=3D8 -cpu max \ + ... + -object memory-backend-ram,id=3Dvmem0,share=3Don,size=3D256M \ + -object memory-backend-file,id=3Dcxl-lsa0,share=3Don,mem-path=3D/tmp/lsa= .raw,size=3D256M \ + -device pxb-cxl,bus_nr=3D12,bus=3Dpcie.0,id=3Dcxl.1 \ + -device cxl-rp,port=3D0,bus=3Dcxl.1,id=3Droot_port13,chassis=3D0,slot=3D= 2 \ + -device cxl-type3,bus=3Droot_port13,volatile-memdev=3Dvmem0,lsa=3Dcxl-ls= a0,id=3Dcxl-vmem0 \ -M cxl-fmw.0.targets.0=3Dcxl.1,cxl-fmw.0.size=3D4G =20 A setup suitable for 4 way interleave. Only one fixed window provided, to = enable 2 way @@ -328,13 +349,13 @@ the CXL Type3 device directly attached (no switches).= :: -device pxb-cxl,bus_nr=3D12,bus=3Dpcie.0,id=3Dcxl.1 \ -device pxb-cxl,bus_nr=3D222,bus=3Dpcie.0,id=3Dcxl.2 \ -device cxl-rp,port=3D0,bus=3Dcxl.1,id=3Droot_port13,chassis=3D0,slot=3D= 2 \ - -device cxl-type3,bus=3Droot_port13,memdev=3Dcxl-mem1,lsa=3Dcxl-lsa1,id= =3Dcxl-pmem0 \ + -device cxl-type3,bus=3Droot_port13,persistent-memdev=3Dcxl-mem1,lsa=3Dc= xl-lsa1,id=3Dcxl-pmem0 \ -device cxl-rp,port=3D1,bus=3Dcxl.1,id=3Droot_port14,chassis=3D0,slot=3D= 3 \ - -device cxl-type3,bus=3Droot_port14,memdev=3Dcxl-mem2,lsa=3Dcxl-lsa2,id= =3Dcxl-pmem1 \ + -device cxl-type3,bus=3Droot_port14,persistent-memdev=3Dcxl-mem2,lsa=3Dc= xl-lsa2,id=3Dcxl-pmem1 \ -device cxl-rp,port=3D0,bus=3Dcxl.2,id=3Droot_port15,chassis=3D0,slot=3D= 5 \ - -device cxl-type3,bus=3Droot_port15,memdev=3Dcxl-mem3,lsa=3Dcxl-lsa3,id= =3Dcxl-pmem2 \ + -device cxl-type3,bus=3Droot_port15,persistent-memdev=3Dcxl-mem3,lsa=3Dc= xl-lsa3,id=3Dcxl-pmem2 \ -device cxl-rp,port=3D1,bus=3Dcxl.2,id=3Droot_port16,chassis=3D0,slot=3D= 6 \ - -device cxl-type3,bus=3Droot_port16,memdev=3Dcxl-mem4,lsa=3Dcxl-lsa4,id= =3Dcxl-pmem3 \ + -device cxl-type3,bus=3Droot_port16,persistent-memdev=3Dcxl-mem4,lsa=3Dc= xl-lsa4,id=3Dcxl-pmem3 \ -M cxl-fmw.0.targets.0=3Dcxl.1,cxl-fmw.0.targets.1=3Dcxl.2,cxl-fmw.0.siz= e=3D4G,cxl-fmw.0.interleave-granularity=3D8k =20 An example of 4 devices below a switch suitable for 1, 2 or 4 way interlea= ve:: @@ -354,15 +375,23 @@ An example of 4 devices below a switch suitable for 1= , 2 or 4 way interleave:: -device cxl-rp,port=3D1,bus=3Dcxl.1,id=3Droot_port1,chassis=3D0,slot=3D1= \ -device cxl-upstream,bus=3Droot_port0,id=3Dus0 \ -device cxl-downstream,port=3D0,bus=3Dus0,id=3Dswport0,chassis=3D0,slot= =3D4 \ - -device cxl-type3,bus=3Dswport0,memdev=3Dcxl-mem0,lsa=3Dcxl-lsa0,id=3Dcx= l-pmem0,size=3D256M \ + -device cxl-type3,bus=3Dswport0,persistent-memdev=3Dcxl-mem0,lsa=3Dcxl-l= sa0,id=3Dcxl-pmem0,size=3D256M \ -device cxl-downstream,port=3D1,bus=3Dus0,id=3Dswport1,chassis=3D0,slot= =3D5 \ - -device cxl-type3,bus=3Dswport1,memdev=3Dcxl-mem1,lsa=3Dcxl-lsa1,id=3Dcx= l-pmem1,size=3D256M \ + -device cxl-type3,bus=3Dswport1,persistent-memdev=3Dcxl-mem1,lsa=3Dcxl-l= sa1,id=3Dcxl-pmem1,size=3D256M \ -device cxl-downstream,port=3D2,bus=3Dus0,id=3Dswport2,chassis=3D0,slot= =3D6 \ - -device cxl-type3,bus=3Dswport2,memdev=3Dcxl-mem2,lsa=3Dcxl-lsa2,id=3Dcx= l-pmem2,size=3D256M \ + -device cxl-type3,bus=3Dswport2,persistent-memdev=3Dcxl-mem2,lsa=3Dcxl-l= sa2,id=3Dcxl-pmem2,size=3D256M \ -device cxl-downstream,port=3D3,bus=3Dus0,id=3Dswport3,chassis=3D0,slot= =3D7 \ - -device cxl-type3,bus=3Dswport3,memdev=3Dcxl-mem3,lsa=3Dcxl-lsa3,id=3Dcx= l-pmem3,size=3D256M \ + -device cxl-type3,bus=3Dswport3,persistent-memdev=3Dcxl-mem3,lsa=3Dcxl-l= sa3,id=3Dcxl-pmem3,size=3D256M \ -M cxl-fmw.0.targets.0=3Dcxl.1,cxl-fmw.0.size=3D4G,cxl-fmw.0.interleave-= granularity=3D4k =20 +Deprecations +------------ + +The Type 3 device [memdev] attribute has been deprecated in favor +of the [persistent-memdev] and [volatile-memdev] attributes. [memdev] +will default to a persistent memory device for backward compatibility +and is incapable of being used in combination with [persistent-memdev]. + Kernel Configuration Options ---------------------------- =20 diff --git a/hw/cxl/cxl-mailbox-utils.c b/hw/cxl/cxl-mailbox-utils.c index d7543fd5b4..c1183614b9 100644 --- a/hw/cxl/cxl-mailbox-utils.c +++ b/hw/cxl/cxl-mailbox-utils.c @@ -269,7 +269,7 @@ static ret_code cmd_firmware_update_get_info(struct cxl= _cmd *cmd, } QEMU_PACKED *fw_info; QEMU_BUILD_BUG_ON(sizeof(*fw_info) !=3D 0x50); =20 - if (cxl_dstate->pmem_size < CXL_CAPACITY_MULTIPLIER) { + if (cxl_dstate->mem_size < CXL_CAPACITY_MULTIPLIER) { return CXL_MBOX_INTERNAL_ERROR; } =20 @@ -412,20 +412,20 @@ static ret_code cmd_identify_memory_device(struct cxl= _cmd *cmd, =20 CXLType3Dev *ct3d =3D container_of(cxl_dstate, CXLType3Dev, cxl_dstate= ); CXLType3Class *cvc =3D CXL_TYPE3_GET_CLASS(ct3d); - uint64_t size =3D cxl_dstate->pmem_size; =20 - if (!QEMU_IS_ALIGNED(size, CXL_CAPACITY_MULTIPLIER)) { + if ((!QEMU_IS_ALIGNED(cxl_dstate->vmem_size, CXL_CAPACITY_MULTIPLIER))= || + (!QEMU_IS_ALIGNED(cxl_dstate->pmem_size, CXL_CAPACITY_MULTIPLIER))= ) { return CXL_MBOX_INTERNAL_ERROR; } =20 id =3D (void *)cmd->payload; memset(id, 0, sizeof(*id)); =20 - /* PMEM only */ snprintf(id->fw_revision, 0x10, "BWFW VERSION %02d", 0); =20 - id->total_capacity =3D size / CXL_CAPACITY_MULTIPLIER; - id->persistent_capacity =3D size / CXL_CAPACITY_MULTIPLIER; + id->total_capacity =3D cxl_dstate->mem_size / CXL_CAPACITY_MULTIPLIER; + id->persistent_capacity =3D cxl_dstate->pmem_size / CXL_CAPACITY_MULTI= PLIER; + id->volatile_capacity =3D cxl_dstate->vmem_size / CXL_CAPACITY_MULTIPL= IER; id->lsa_size =3D cvc->get_lsa_size(ct3d); id->poison_list_max_mer[1] =3D 0x1; /* 256 poison records */ =20 @@ -444,16 +444,15 @@ static ret_code cmd_ccls_get_partition_info(struct cx= l_cmd *cmd, uint64_t next_pmem; } QEMU_PACKED *part_info =3D (void *)cmd->payload; QEMU_BUILD_BUG_ON(sizeof(*part_info) !=3D 0x20); - uint64_t size =3D cxl_dstate->pmem_size; =20 - if (!QEMU_IS_ALIGNED(size, CXL_CAPACITY_MULTIPLIER)) { + if ((!QEMU_IS_ALIGNED(cxl_dstate->vmem_size, CXL_CAPACITY_MULTIPLIER))= || + (!QEMU_IS_ALIGNED(cxl_dstate->pmem_size, CXL_CAPACITY_MULTIPLIER))= ) { return CXL_MBOX_INTERNAL_ERROR; } =20 - /* PMEM only */ - part_info->active_vmem =3D 0; + part_info->active_vmem =3D cxl_dstate->vmem_size / CXL_CAPACITY_MULTIP= LIER; part_info->next_vmem =3D 0; - part_info->active_pmem =3D size / CXL_CAPACITY_MULTIPLIER; + part_info->active_pmem =3D cxl_dstate->pmem_size / CXL_CAPACITY_MULTIP= LIER; part_info->next_pmem =3D 0; =20 *len =3D sizeof(*part_info); diff --git a/hw/mem/cxl_type3.c b/hw/mem/cxl_type3.c index 0317bd96a6..21e866dcaf 100644 --- a/hw/mem/cxl_type3.c +++ b/hw/mem/cxl_type3.c @@ -32,7 +32,8 @@ enum { }; =20 static int ct3_build_cdat_entries_for_mr(CDATSubHeader **cdat_table, - int dsmad_handle, MemoryRegion *m= r) + int dsmad_handle, MemoryRegion *m= r, + bool is_pmem, uint64_t dpa_base) { g_autofree CDATDsmas *dsmas =3D NULL; g_autofree CDATDslbis *dslbis0 =3D NULL; @@ -51,8 +52,8 @@ static int ct3_build_cdat_entries_for_mr(CDATSubHeader **= cdat_table, .length =3D sizeof(*dsmas), }, .DSMADhandle =3D dsmad_handle, - .flags =3D CDAT_DSMAS_FLAG_NV, - .DPA_base =3D 0, + .flags =3D is_pmem ? CDAT_DSMAS_FLAG_NV : 0, + .DPA_base =3D dpa_base, .DPA_length =3D int128_get64(mr->size), }; =20 @@ -151,33 +152,67 @@ static int ct3_build_cdat_entries_for_mr(CDATSubHeade= r **cdat_table, static int ct3_build_cdat_table(CDATSubHeader ***cdat_table, void *priv) { g_autofree CDATSubHeader **table =3D NULL; - MemoryRegion *nonvolatile_mr; CXLType3Dev *ct3d =3D priv; + MemoryRegion *volatile_mr =3D NULL, *nonvolatile_mr =3D NULL; int dsmad_handle =3D 0; + int cur_ent =3D 0; + int len =3D 0; int rc; =20 - if (!ct3d->hostmem) { + if (!ct3d->hostpmem && !ct3d->hostvmem) { return 0; } =20 - nonvolatile_mr =3D host_memory_backend_get_memory(ct3d->hostmem); - if (!nonvolatile_mr) { - return -EINVAL; + if (ct3d->hostvmem) { + volatile_mr =3D host_memory_backend_get_memory(ct3d->hostvmem); + if (!volatile_mr) { + return -EINVAL; + } + len +=3D CT3_CDAT_NUM_ENTRIES; + } + + if (ct3d->hostpmem) { + nonvolatile_mr =3D host_memory_backend_get_memory(ct3d->hostpmem); + if (!nonvolatile_mr) { + return -EINVAL; + } + len +=3D CT3_CDAT_NUM_ENTRIES; } =20 - table =3D g_malloc0(CT3_CDAT_NUM_ENTRIES * sizeof(*table)); + table =3D g_malloc0(len * sizeof(*table)); if (!table) { return -ENOMEM; } =20 - rc =3D ct3_build_cdat_entries_for_mr(table, dsmad_handle++, nonvolatil= e_mr); - if (rc < 0) { - return rc; + /* Now fill them in */ + if (volatile_mr) { + rc =3D ct3_build_cdat_entries_for_mr(table, dsmad_handle++, volati= le_mr, + true, 0); + if (rc < 0) { + return rc; + } + cur_ent =3D CT3_CDAT_NUM_ENTRIES; + } + + if (nonvolatile_mr) { + rc =3D ct3_build_cdat_entries_for_mr(&(table[cur_ent]), dsmad_hand= le++, + nonvolatile_mr, true, (volatile_mr ? volatile_mr->size : 0= )); + if (rc < 0) { + goto error_cleanup; + } + cur_ent +=3D CT3_CDAT_NUM_ENTRIES; } + assert(len =3D=3D cur_ent); =20 *cdat_table =3D g_steal_pointer(&table); =20 - return CT3_CDAT_NUM_ENTRIES; + return len; +error_cleanup: + int i; + for (i =3D 0; i < cur_ent; i++) { + g_free(*cdat_table[i]); + } + return rc; } =20 static void ct3_free_cdat_table(CDATSubHeader **cdat_table, int num, void = *priv) @@ -378,16 +413,48 @@ static void build_dvsecs(CXLType3Dev *ct3d) CXLDVSECRegisterLocator *regloc_dvsec; uint8_t *dvsec; int i; + uint32_t range1_size_hi =3D 0, range1_size_lo =3D 0, + range1_base_hi =3D 0, range1_base_lo =3D 0, + range2_size_hi =3D 0, range2_size_lo =3D 0, + range2_base_hi =3D 0, range2_base_lo =3D 0; + + /* + * Volatile memory is mapped as (0x0) + * Persistent memory is mapped at (volatile->size) + */ + if (ct3d->hostvmem && ct3d->hostpmem) { + range1_size_hi =3D ct3d->hostvmem->size >> 32; + range1_size_lo =3D (2 << 5) | (2 << 2) | 0x3 | + (ct3d->hostvmem->size & 0xF0000000); + range1_base_hi =3D 0; + range1_base_lo =3D 0; + range2_size_hi =3D ct3d->hostpmem->size >> 32; + range2_size_lo =3D (2 << 5) | (2 << 2) | 0x3 | + (ct3d->hostpmem->size & 0xF0000000); + range2_base_hi =3D ct3d->hostvmem->size >> 32; + range2_base_lo =3D ct3d->hostvmem->size & 0xF0000000; + } else { + HostMemoryBackend *hmbe =3D ct3d->hostvmem ? + ct3d->hostvmem : ct3d->hostpmem; + range1_size_hi =3D hmbe->size >> 32; + range1_size_lo =3D (2 << 5) | (2 << 2) | 0x3 | + (hmbe->size & 0xF0000000); + range1_base_hi =3D 0; + range1_base_lo =3D 0; + } =20 dvsec =3D (uint8_t *)&(CXLDVSECDevice){ .cap =3D 0x1e, .ctrl =3D 0x2, .status2 =3D 0x2, - .range1_size_hi =3D ct3d->hostmem->size >> 32, - .range1_size_lo =3D (2 << 5) | (2 << 2) | 0x3 | - (ct3d->hostmem->size & 0xF0000000), - .range1_base_hi =3D 0, - .range1_base_lo =3D 0, + .range1_size_hi =3D range1_size_hi, + .range1_size_lo =3D range1_size_lo, + .range1_base_hi =3D range1_base_hi, + .range1_base_lo =3D range1_base_lo, + .range2_size_hi =3D range2_size_hi, + .range2_size_lo =3D range2_size_lo, + .range2_base_hi =3D range2_base_hi, + .range2_base_lo =3D range2_base_lo }; cxl_component_create_dvsec(cxl_cstate, CXL2_TYPE3_DEVICE, PCIE_CXL_DEVICE_DVSEC_LENGTH, @@ -475,33 +542,62 @@ static bool cxl_setup_memory(CXLType3Dev *ct3d, Error= **errp) MemoryRegion *mr; char *name; =20 - if (!ct3d->hostmem) { - error_setg(errp, "memdev property must be set"); + if (!ct3d->hostmem && !ct3d->hostvmem && !ct3d->hostpmem) { + error_setg(errp, "at least one memdev property must be set"); return false; + } else if (ct3d->hostmem && ct3d->hostpmem) { + error_setg(errp, "[memdev] cannot be used with new " + "[persistent-memdev] property"); + return false; + } else if (ct3d->hostmem) { + /* Use of hostmem property implies pmem */ + ct3d->hostpmem =3D ct3d->hostmem; + ct3d->hostmem =3D NULL; } =20 - mr =3D host_memory_backend_get_memory(ct3d->hostmem); - if (!mr) { - error_setg(errp, "memdev property must be set"); + if (ct3d->hostpmem && !ct3d->lsa) { + error_setg(errp, "lsa property must be set for persistent devices"= ); return false; } - memory_region_set_nonvolatile(mr, true); - memory_region_set_enabled(mr, true); - host_memory_backend_set_mapped(ct3d->hostmem, true); =20 - if (ds->id) { - name =3D g_strdup_printf("cxl-type3-dpa-space:%s", ds->id); - } else { - name =3D g_strdup("cxl-type3-dpa-space"); + if (ct3d->hostvmem) { + mr =3D host_memory_backend_get_memory(ct3d->hostvmem); + if (!mr) { + error_setg(errp, "volatile memdev must have backing device"); + return false; + } + memory_region_set_nonvolatile(mr, false); + memory_region_set_enabled(mr, true); + host_memory_backend_set_mapped(ct3d->hostvmem, true); + if (ds->id) { + name =3D g_strdup_printf("cxl-type3-dpa-vmem-space:%s", ds->id= ); + } else { + name =3D g_strdup("cxl-type3-dpa-vmem-space"); + } + address_space_init(&ct3d->hostvmem_as, mr, name); + ct3d->cxl_dstate.vmem_size =3D mr->size; + ct3d->cxl_dstate.mem_size +=3D mr->size; + g_free(name); } - address_space_init(&ct3d->hostmem_as, mr, name); - g_free(name); - - ct3d->cxl_dstate.pmem_size =3D ct3d->hostmem->size; =20 - if (!ct3d->lsa) { - error_setg(errp, "lsa property must be set"); - return false; + if (ct3d->hostpmem) { + mr =3D host_memory_backend_get_memory(ct3d->hostpmem); + if (!mr) { + error_setg(errp, "persistent memdev must have backing device"); + return false; + } + memory_region_set_nonvolatile(mr, true); + memory_region_set_enabled(mr, true); + host_memory_backend_set_mapped(ct3d->hostpmem, true); + if (ds->id) { + name =3D g_strdup_printf("cxl-type3-dpa-pmem-space:%s", ds->id= ); + } else { + name =3D g_strdup("cxl-type3-dpa-pmem-space"); + } + address_space_init(&ct3d->hostpmem_as, mr, name); + ct3d->cxl_dstate.pmem_size =3D mr->size; + ct3d->cxl_dstate.mem_size +=3D mr->size; + g_free(name); } =20 return true; @@ -609,7 +705,12 @@ err_free_spdm_socket: err_free_special_ops: g_free(regs->special_ops); err_address_space_free: - address_space_destroy(&ct3d->hostmem_as); + if (ct3d->hostvmem) { + address_space_destroy(&ct3d->hostvmem_as); + } + if (ct3d->hostpmem) { + address_space_destroy(&ct3d->hostpmem_as); + } return; } =20 @@ -623,7 +724,12 @@ static void ct3_exit(PCIDevice *pci_dev) cxl_doe_cdat_release(cxl_cstate); spdm_sock_fini(ct3d->doe_spdm.socket); g_free(regs->special_ops); - address_space_destroy(&ct3d->hostmem_as); + if (ct3d->hostvmem) { + address_space_destroy(&ct3d->hostvmem_as); + } + if (ct3d->hostpmem) { + address_space_destroy(&ct3d->hostpmem_as); + } } =20 /* TODO: Support multiple HDM decoders and DPA skip */ @@ -663,11 +769,17 @@ MemTxResult cxl_type3_read(PCIDevice *d, hwaddr host_= addr, uint64_t *data, { CXLType3Dev *ct3d =3D CXL_TYPE3(d); uint64_t dpa_offset; - MemoryRegion *mr; + MemoryRegion *vmr =3D NULL, *pmr =3D NULL; + AddressSpace *as; + + if (ct3d->hostvmem) { + vmr =3D host_memory_backend_get_memory(ct3d->hostvmem); + } + if (ct3d->hostpmem) { + pmr =3D host_memory_backend_get_memory(ct3d->hostpmem); + } =20 - /* TODO support volatile region */ - mr =3D host_memory_backend_get_memory(ct3d->hostmem); - if (!mr) { + if (!vmr && !pmr) { return MEMTX_ERROR; } =20 @@ -675,11 +787,13 @@ MemTxResult cxl_type3_read(PCIDevice *d, hwaddr host_= addr, uint64_t *data, return MEMTX_ERROR; } =20 - if (dpa_offset > int128_get64(mr->size)) { + if (dpa_offset > int128_get64(ct3d->cxl_dstate.mem_size)) { return MEMTX_ERROR; } =20 - return address_space_read(&ct3d->hostmem_as, dpa_offset, attrs, data, = size); + as =3D (vmr && (dpa_offset <=3D int128_get64(vmr->size))) ? + &ct3d->hostvmem_as : &ct3d->hostpmem_as; + return address_space_read(as, dpa_offset, attrs, data, size); } =20 MemTxResult cxl_type3_write(PCIDevice *d, hwaddr host_addr, uint64_t data, @@ -687,10 +801,17 @@ MemTxResult cxl_type3_write(PCIDevice *d, hwaddr host= _addr, uint64_t data, { CXLType3Dev *ct3d =3D CXL_TYPE3(d); uint64_t dpa_offset; - MemoryRegion *mr; + MemoryRegion *vmr =3D NULL, *pmr =3D NULL; + AddressSpace *as; + + if (ct3d->hostvmem) { + vmr =3D host_memory_backend_get_memory(ct3d->hostvmem); + } + if (ct3d->hostpmem) { + pmr =3D host_memory_backend_get_memory(ct3d->hostpmem); + } =20 - mr =3D host_memory_backend_get_memory(ct3d->hostmem); - if (!mr) { + if (!vmr && !pmr) { return MEMTX_OK; } =20 @@ -698,11 +819,13 @@ MemTxResult cxl_type3_write(PCIDevice *d, hwaddr host= _addr, uint64_t data, return MEMTX_OK; } =20 - if (dpa_offset > int128_get64(mr->size)) { + if (dpa_offset > int128_get64(ct3d->cxl_dstate.mem_size)) { return MEMTX_OK; } - return address_space_write(&ct3d->hostmem_as, dpa_offset, attrs, - &data, size); + + as =3D (vmr && (dpa_offset <=3D int128_get64(vmr->size))) ? + &ct3d->hostvmem_as : &ct3d->hostpmem_as; + return address_space_write(as, dpa_offset, attrs, &data, size); } =20 static void ct3d_reset(DeviceState *dev) @@ -717,7 +840,11 @@ static void ct3d_reset(DeviceState *dev) =20 static Property ct3_props[] =3D { DEFINE_PROP_LINK("memdev", CXLType3Dev, hostmem, TYPE_MEMORY_BACKEND, - HostMemoryBackend *), + HostMemoryBackend *), /* for backward compatibility */ + DEFINE_PROP_LINK("persistent-memdev", CXLType3Dev, hostpmem, + TYPE_MEMORY_BACKEND, HostMemoryBackend *), + DEFINE_PROP_LINK("volatile-memdev", CXLType3Dev, hostvmem, + TYPE_MEMORY_BACKEND, HostMemoryBackend *), DEFINE_PROP_LINK("lsa", CXLType3Dev, lsa, TYPE_MEMORY_BACKEND, HostMemoryBackend *), DEFINE_PROP_UINT64("sn", CXLType3Dev, sn, UI64_NULL), @@ -728,10 +855,12 @@ static Property ct3_props[] =3D { =20 static uint64_t get_lsa_size(CXLType3Dev *ct3d) { - MemoryRegion *mr; - - mr =3D host_memory_backend_get_memory(ct3d->lsa); - return memory_region_size(mr); + MemoryRegion *mr =3D NULL; + if (ct3d->lsa) { + mr =3D host_memory_backend_get_memory(ct3d->lsa); + return memory_region_size(mr); + } + return 0; } =20 static void validate_lsa_access(MemoryRegion *mr, uint64_t size, @@ -744,30 +873,35 @@ static void validate_lsa_access(MemoryRegion *mr, uin= t64_t size, static uint64_t get_lsa(CXLType3Dev *ct3d, void *buf, uint64_t size, uint64_t offset) { - MemoryRegion *mr; + MemoryRegion *mr =3D NULL; void *lsa; =20 - mr =3D host_memory_backend_get_memory(ct3d->lsa); - validate_lsa_access(mr, size, offset); + if (ct3d->lsa) { + mr =3D host_memory_backend_get_memory(ct3d->lsa); + validate_lsa_access(mr, size, offset); =20 - lsa =3D memory_region_get_ram_ptr(mr) + offset; - memcpy(buf, lsa, size); + lsa =3D memory_region_get_ram_ptr(mr) + offset; + memcpy(buf, lsa, size); + return size; + } =20 - return size; + return 0; } =20 static void set_lsa(CXLType3Dev *ct3d, const void *buf, uint64_t size, uint64_t offset) { - MemoryRegion *mr; - void *lsa; + MemoryRegion *mr =3D NULL; + void *lsa =3D NULL; =20 - mr =3D host_memory_backend_get_memory(ct3d->lsa); - validate_lsa_access(mr, size, offset); + if (ct3d->lsa) { + mr =3D host_memory_backend_get_memory(ct3d->lsa); + validate_lsa_access(mr, size, offset); =20 - lsa =3D memory_region_get_ram_ptr(mr) + offset; - memcpy(lsa, buf, size); - memory_region_set_dirty(mr, offset, size); + lsa =3D memory_region_get_ram_ptr(mr) + offset; + memcpy(lsa, buf, size); + memory_region_set_dirty(mr, offset, size); + } =20 /* * Just like the PMEM, if the guest is not allowed to exit gracefully,= label @@ -978,7 +1112,7 @@ static void ct3_class_init(ObjectClass *oc, void *data) pc->config_read =3D ct3d_config_read; =20 set_bit(DEVICE_CATEGORY_STORAGE, dc->categories); - dc->desc =3D "CXL PMEM Device (Type 3)"; + dc->desc =3D "CXL Memory Device (Type 3)"; dc->reset =3D ct3d_reset; device_class_set_props(dc, ct3_props); =20 diff --git a/include/hw/cxl/cxl_device.h b/include/hw/cxl/cxl_device.h index 1cd71afcb6..1b366b739c 100644 --- a/include/hw/cxl/cxl_device.h +++ b/include/hw/cxl/cxl_device.h @@ -180,8 +180,10 @@ typedef struct cxl_device_state { uint64_t host_set; } timestamp; =20 - /* memory region for persistent memory, HDM */ + /* memory region size, HDM */ + uint64_t mem_size; uint64_t pmem_size; + uint64_t vmem_size; =20 struct cxl_cmd (*cxl_cmd_set)[256]; /* Move me later */ @@ -311,12 +313,15 @@ struct CXLType3Dev { PCIDevice parent_obj; =20 /* Properties */ - HostMemoryBackend *hostmem; + HostMemoryBackend *hostmem; /* deprecated */ + HostMemoryBackend *hostvmem; + HostMemoryBackend *hostpmem; HostMemoryBackend *lsa; uint64_t sn; =20 /* State */ - AddressSpace hostmem_as; + AddressSpace hostvmem_as; + AddressSpace hostpmem_as; CXLComponentState cxl_cstate; CXLDeviceState cxl_dstate; =20 diff --git a/tests/qtest/cxl-test.c b/tests/qtest/cxl-test.c index 6baed747fa..a05ddc0c7b 100644 --- a/tests/qtest/cxl-test.c +++ b/tests/qtest/cxl-test.c @@ -34,29 +34,46 @@ "-device cxl-rp,id=3Drp2,bus=3Dcxl.1,chassis=3D0,slot=3D2= " \ "-device cxl-rp,id=3Drp3,bus=3Dcxl.1,chassis=3D0,slot=3D3= " =20 -#define QEMU_T3D "-object memory-backend-file,id=3Dcxl-mem0,mem-path=3D%s,= size=3D256M " \ - "-object memory-backend-file,id=3Dlsa0,mem-path=3D%s,size= =3D256M " \ - "-device cxl-type3,bus=3Drp0,memdev=3Dcxl-mem0,lsa=3Dlsa0= ,id=3Dcxl-pmem0 " - -#define QEMU_2T3D "-object memory-backend-file,id=3Dcxl-mem0,mem-path=3D%s= ,size=3D256M " \ - "-object memory-backend-file,id=3Dlsa0,mem-path=3D%s,siz= e=3D256M " \ - "-device cxl-type3,bus=3Drp0,memdev=3Dcxl-mem0,lsa=3Dlsa= 0,id=3Dcxl-pmem0 " \ - "-object memory-backend-file,id=3Dcxl-mem1,mem-path=3D%s= ,size=3D256M " \ - "-object memory-backend-file,id=3Dlsa1,mem-path=3D%s,siz= e=3D256M " \ - "-device cxl-type3,bus=3Drp1,memdev=3Dcxl-mem1,lsa=3Dlsa= 1,id=3Dcxl-pmem1 " - -#define QEMU_4T3D "-object memory-backend-file,id=3Dcxl-mem0,mem-path=3D%s= ,size=3D256M " \ - "-object memory-backend-file,id=3Dlsa0,mem-path=3D%s,siz= e=3D256M " \ - "-device cxl-type3,bus=3Drp0,memdev=3Dcxl-mem0,lsa=3Dlsa= 0,id=3Dcxl-pmem0 " \ - "-object memory-backend-file,id=3Dcxl-mem1,mem-path=3D%s= ,size=3D256M " \ - "-object memory-backend-file,id=3Dlsa1,mem-path=3D%s,siz= e=3D256M " \ - "-device cxl-type3,bus=3Drp1,memdev=3Dcxl-mem1,lsa=3Dlsa= 1,id=3Dcxl-pmem1 " \ - "-object memory-backend-file,id=3Dcxl-mem2,mem-path=3D%s= ,size=3D256M " \ - "-object memory-backend-file,id=3Dlsa2,mem-path=3D%s,siz= e=3D256M " \ - "-device cxl-type3,bus=3Drp2,memdev=3Dcxl-mem2,lsa=3Dlsa= 2,id=3Dcxl-pmem2 " \ - "-object memory-backend-file,id=3Dcxl-mem3,mem-path=3D%s= ,size=3D256M " \ - "-object memory-backend-file,id=3Dlsa3,mem-path=3D%s,siz= e=3D256M " \ - "-device cxl-type3,bus=3Drp3,memdev=3Dcxl-mem3,lsa=3Dlsa= 3,id=3Dcxl-pmem3 " +#define QEMU_T3D_DEPRECATED \ + "-object memory-backend-file,id=3Dcxl-mem0,mem-path=3D%s,size=3D256M " \ + "-object memory-backend-file,id=3Dlsa0,mem-path=3D%s,size=3D256M " \ + "-device cxl-type3,bus=3Drp0,memdev=3Dcxl-mem0,lsa=3Dlsa0,id=3Dcxl-pmem0= " + +#define QEMU_T3D_PMEM \ + "-object memory-backend-file,id=3Dm0,mem-path=3D%s,size=3D256M " \ + "-object memory-backend-file,id=3Dlsa0,mem-path=3D%s,size=3D256M " \ + "-device cxl-type3,bus=3Drp0,persistent-memdev=3Dcxl-m0,lsa=3Dlsa0,id=3D= pmem0 " + +#define QEMU_T3D_VMEM \ + "-object memory-backend-ram,id=3Dmem0,size=3D256M " \ + "-device cxl-type3,bus=3Drp0,volatile-memdev=3Dmem0,id=3Dmem0 " + +#define QEMU_T3D_VMEM_LSA \ + "-object memory-backend-ram,id=3Dmem0,size=3D256M " \ + "-object memory-backend-file,id=3Dlsa0,mem-path=3D%s,size=3D256M " \ + "-device cxl-type3,bus=3Drp0,volatile-memdev=3Dmem0,lsa=3Dlsa0,id=3Dmem0= " + +#define QEMU_2T3D \ + "-object memory-backend-file,id=3Dcxl-mem0,mem-path=3D%s,size=3D256M " = \ + "-object memory-backend-file,id=3Dlsa0,mem-path=3D%s,size=3D256M " \ + "-device cxl-type3,bus=3Drp0,persistent-memdev=3Dcxl-mem0,lsa=3Dlsa0,id= =3Dpmem0 " \ + "-object memory-backend-file,id=3Dcxl-mem1,mem-path=3D%s,size=3D256M " = \ + "-object memory-backend-file,id=3Dlsa1,mem-path=3D%s,size=3D256M " \ + "-device cxl-type3,bus=3Drp1,persistent-memdev=3Dcxl-mem1,lsa=3Dlsa1,id= =3Dpmem1 " + +#define QEMU_4T3D \ + "-object memory-backend-file,id=3Dcxl-mem0,mem-path=3D%s,size=3D256M " \ + "-object memory-backend-file,id=3Dlsa0,mem-path=3D%s,size=3D256M " \ + "-device cxl-type3,bus=3Drp0,persistent-memdev=3Dcxl-mem0,lsa=3Dlsa0,id= =3Dpmem0 " \ + "-object memory-backend-file,id=3Dcxl-mem1,mem-path=3D%s,size=3D256M " = \ + "-object memory-backend-file,id=3Dlsa1,mem-path=3D%s,size=3D256M " \ + "-device cxl-type3,bus=3Drp1,persistent-memdev=3Dcxl-mem1,lsa=3Dlsa1,id= =3Dpmem1 " \ + "-object memory-backend-file,id=3Dcxl-mem2,mem-path=3D%s,size=3D256M " = \ + "-object memory-backend-file,id=3Dlsa2,mem-path=3D%s,size=3D256M " \ + "-device cxl-type3,bus=3Drp2,persistent-memdev=3Dcxl-mem2,lsa=3Dlsa2,id= =3Dpmem2 " \ + "-object memory-backend-file,id=3Dcxl-mem3,mem-path=3D%s,size=3D256M " = \ + "-object memory-backend-file,id=3Dlsa3,mem-path=3D%s,size=3D256M " \ + "-device cxl-type3,bus=3Drp3,persistent-memdev=3Dcxl-mem3,lsa=3Dlsa3,id= =3Dpmem3 " =20 static void cxl_basic_hb(void) { @@ -95,14 +112,53 @@ static void cxl_2root_port(void) } =20 #ifdef CONFIG_POSIX -static void cxl_t3d(void) +static void cxl_t3d_deprecated(void) { g_autoptr(GString) cmdline =3D g_string_new(NULL); g_autofree const char *tmpfs =3D NULL; =20 tmpfs =3D g_dir_make_tmp("cxl-test-XXXXXX", NULL); =20 - g_string_printf(cmdline, QEMU_PXB_CMD QEMU_RP QEMU_T3D, tmpfs, tmpfs); + g_string_printf(cmdline, QEMU_PXB_CMD QEMU_RP QEMU_T3D_DEPRECATED, + tmpfs, tmpfs); + + qtest_start(cmdline->str); + qtest_end(); +} + +static void cxl_t3d_persistent(void) +{ + g_autoptr(GString) cmdline =3D g_string_new(NULL); + g_autofree const char *tmpfs =3D NULL; + + tmpfs =3D g_dir_make_tmp("cxl-test-XXXXXX", NULL); + + g_string_printf(cmdline, QEMU_PXB_CMD QEMU_RP QEMU_T3D_PMEM, + tmpfs, tmpfs); + + qtest_start(cmdline->str); + qtest_end(); +} + +static void cxl_t3d_volatile(void) +{ + g_autoptr(GString) cmdline =3D g_string_new(NULL); + + g_string_printf(cmdline, QEMU_PXB_CMD QEMU_RP QEMU_T3D_VMEM); + + qtest_start(cmdline->str); + qtest_end(); +} + +static void cxl_t3d_volatile_lsa(void) +{ + g_autoptr(GString) cmdline =3D g_string_new(NULL); + g_autofree const char *tmpfs =3D NULL; + + tmpfs =3D g_dir_make_tmp("cxl-test-XXXXXX", NULL); + + g_string_printf(cmdline, QEMU_PXB_CMD QEMU_RP QEMU_T3D_VMEM_LSA, + tmpfs); =20 qtest_start(cmdline->str); qtest_end(); @@ -167,7 +223,10 @@ int main(int argc, char **argv) qtest_add_func("/pci/cxl/rp", cxl_root_port); qtest_add_func("/pci/cxl/rp_x2", cxl_2root_port); #ifdef CONFIG_POSIX - qtest_add_func("/pci/cxl/type3_device", cxl_t3d); + qtest_add_func("/pci/cxl/type3_device", cxl_t3d_deprecated); + qtest_add_func("/pci/cxl/type3_device_pmem", cxl_t3d_persistent); + qtest_add_func("/pci/cxl/type3_device_vmem", cxl_t3d_volatile); + qtest_add_func("/pci/cxl/type3_device_vmem_lsa", cxl_t3d_volatile_= lsa); qtest_add_func("/pci/cxl/rp_x2_type3_x2", cxl_1pxb_2rp_2t3d); qtest_add_func("/pci/cxl/pxb_x2_root_port_x4_type3_x4", cxl_2pxb_4rp_4t3d); --=20 2.37.3 From nobody Sat May 18 07:48:33 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=gmail.com ARC-Seal: i=1; a=rsa-sha256; t=1666745444; cv=none; d=zohomail.com; s=zohoarc; b=AFaAoAnd4BFid8uaASl42sU6qBjN3eEXX1DUlfUrDdX545TI4EqLRbE4B54+untSVzAWo7qJYQsaCALMxSKtILujo2FbuwJpxqQVLsJTbyuvFuiUn3zIi2BVbNb+X9udm7Mp25snuK3uARygGgwrmM+ipFaerC5HTvK7kn2fMPI= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1666745444; h=Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=itdJjoWTzr0ylNc2qCCbqq49u8Zk8OI8xM4xXE0AET8=; b=AWJJ/RouAfSSd5vY/O7LlCFJAlAua/Ju+UonwSXXi+nK8QKGzmBfNIoZtYx++jr78lrE1xR7Rq4daposUCIBYKrQxJC5woJgKQU1gt7w+Mh8LNmaCtf3XUXkyxV5DwpOGfuQc0vqRxmFW9ieOskdX/WdXNn7k0Z0clTss4B5BJI= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1666745444707925.4894660264451; Tue, 25 Oct 2022 17:50:44 -0700 (PDT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1onUc3-0006DR-KN; Tue, 25 Oct 2022 20:49:43 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1onUc1-0006AP-I4 for qemu-devel@nongnu.org; Tue, 25 Oct 2022 20:49:41 -0400 Received: from mail-qv1-xf2b.google.com ([2607:f8b0:4864:20::f2b]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1onUbz-0003cT-PD for qemu-devel@nongnu.org; Tue, 25 Oct 2022 20:49:41 -0400 Received: by mail-qv1-xf2b.google.com with SMTP id i12so10252819qvs.2 for ; Tue, 25 Oct 2022 17:49:39 -0700 (PDT) Received: from fedora.mshome.net (pool-173-79-56-208.washdc.fios.verizon.net. [173.79.56.208]) by smtp.gmail.com with ESMTPSA id j15-20020ac8550f000000b0039a610a04b1sm2408651qtq.37.2022.10.25.17.49.27 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 25 Oct 2022 17:49:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=itdJjoWTzr0ylNc2qCCbqq49u8Zk8OI8xM4xXE0AET8=; b=DErcIejy+jTfZFrrHblUaN3ryaHEn6mgd2TCHyOHrRofiS/ZHNaoRKG8PfnL8yQP0X 2Yv6HFnPkkC+29wcpuzRi6ItVra8KrfgsNUwmlM8yephllXnY0cOTYRsLLbqGL2wMXdn cx7m7O0X4TxiYd+mLeC1m5Oq/wqLB0BA7Nmbjd3u92rDjmGYZf87nlJx4EfKhQTKv89g hcYuwn+YVnQikqdrgPe51EEYdRQv9FIrjG7HCieEIEyLLDaievMubaB+tMFBzj9y9fG2 bnIUG9Ym7bnLWSHGesGLhhYOt0F7Z7EKGPsp+u+L5kNcpexCXeDgW3ZBmu7ME3ZaQFt6 +NGg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=itdJjoWTzr0ylNc2qCCbqq49u8Zk8OI8xM4xXE0AET8=; b=n9SlyEkLBg2Gf++99yQe1OxQvIkDfe8fyvKqcUECIEXEiEQa8DtGhJpoVuZXUDEqJs etyITjolgnKiOaeO4WU1YkgOJHc/88R1NF3rFPflHJ+cLTCjEQSzz8vvbPr46CiZ6vY9 UAUFr+axuJVc0ptb+QltEhRu3HYUiP1pO8ECzeGzlPAF2J28zod4nVEJHugFF97pZQqO tPLpSCA6JQmG4PdjTwafszM5WN+0H4Nvj7UfXi8//7JA6njah867wIxycO3DadRHOTnE ymdGWTSyK/eMAoGIAXUynHjmx0qeqdhgXFq/upDubSySwnhlVAZKbfVEX6/MCXw42bQa 7HrA== X-Gm-Message-State: ACrzQf0D2IZ1hAMCfBoO26CcWfAenFXkAFpNNWT6dx/QT9uYAjoNh2ZO NRK46XTfiCQUUKbBw2vAXTHQ3yinBQ== X-Google-Smtp-Source: AMsMyM5JSL7W12XrKKbKyXJP9wWmNB6SvooPIuwe6FpViFuujBLsvC9GOqJBnqlfQPAVFkWcVoC2Ng== X-Received: by 2002:ad4:5fc8:0:b0:4b7:6ca0:adc6 with SMTP id jq8-20020ad45fc8000000b004b76ca0adc6mr27173507qvb.120.1666745378233; Tue, 25 Oct 2022 17:49:38 -0700 (PDT) From: Gregory Price X-Google-Original-From: Gregory Price To: qemu-devel@nongnu.org Cc: jonathan.cameron@huawei.com, linux-cxl@vger.kernel.org, mst@redhat.com, marcel.apfelbaum@gmail.com, imammedo@redhat.com, ani@anisinha.ca, alison.schofield@intel.com, dave@stgolabs.net, a.manzanares@samsung.com, bwidawsk@kernel.org, gregory.price@memverge.com, hchkuo@avery-design.com.tw, cbrowy@avery-design.com, ira.weiny@intel.com Subject: [PATCH 4/4] hw/acpi/cxl.c: Fill in SRAT for vmem/pmem if NUMA node is assigned Date: Tue, 25 Oct 2022 20:47:37 -0400 Message-Id: <20221026004737.3646-5-gregory.price@memverge.com> X-Mailer: git-send-email 2.37.3 In-Reply-To: <20221026004737.3646-1-gregory.price@memverge.com> References: <20221026004737.3646-1-gregory.price@memverge.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=2607:f8b0:4864:20::f2b; envelope-from=gourry.memverge@gmail.com; helo=mail-qv1-xf2b.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "Qemu-devel" Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @gmail.com) X-ZM-MESSAGEID: 1666745445519100001 Content-Type: text/plain; charset="utf-8" This patch enables the direct assignment of a NUMA node to a volatile or persistent memory region on a CXL type-3 device. This is useful for testing static mapping for type-3 device memory regions as memory and leveraging them directly via its NUMA node. Co-developed-By: Davidlohr Bueso Signed-off-by: Gregory Price --- docs/system/devices/cxl.rst | 21 ++++++++++++ hw/acpi/cxl.c | 67 +++++++++++++++++++++++++++++++++++++ hw/i386/acpi-build.c | 4 +++ include/hw/acpi/cxl.h | 1 + 4 files changed, 93 insertions(+) diff --git a/docs/system/devices/cxl.rst b/docs/system/devices/cxl.rst index 9e165064c8..32bf84a97c 100644 --- a/docs/system/devices/cxl.rst +++ b/docs/system/devices/cxl.rst @@ -332,6 +332,27 @@ The same volatile setup may optionally include an LSA = region:: -device cxl-type3,bus=3Droot_port13,volatile-memdev=3Dvmem0,lsa=3Dcxl-ls= a0,id=3Dcxl-vmem0 \ -M cxl-fmw.0.targets.0=3Dcxl.1,cxl-fmw.0.size=3D4G =20 + +Volatile and Persistent Memory regions may also be assigned an SRAT entry = and statically +mapped into the system by manually assigning them a CPU-less NUMA node. Th= is is an example +of a CXL Type 3 Volatile Memory device being assigned an SRAT entry via a = NUMA node mapping:: + + qemu-system-aarch64 -M virt,gic-version=3D3,cxl=3Don -m 4g,maxmem=3D8G= ,slots=3D8 -cpu max \ + ... + -smp 4 \ + -enable-kvm \ + -nographic \ + -object memory-backend-ram,id=3Dmem0,size=3D2G,share=3Don \ + -object memory-backend-ram,id=3Dmem1,size=3D2G,share=3Don \ + -numa node,memdev=3Dmem0,cpus=3D0-3,nodeid=3D0 \ + -numa node,memdev=3Dmem1,nodeid=3D1, \ + -device pxb-cxl,id=3Dcxl.0,bus=3Dpcie.0,bus_nr=3D52 \ + -device cxl-rp,port=3D0,id=3Drp0,bus=3Dcxl.0,chassis=3D0,slot=3D0 \ + -device cxl-rp,port=3D1,id=3Drp1,bus=3Dcxl.0,chassis=3D0,slot=3D1 \ + -device cxl-type3,bus=3Drp0,volatile-memdev=3Dmem1,id=3Dcxl-mem0=20 + + + A setup suitable for 4 way interleave. Only one fixed window provided, to = enable 2 way interleave across 2 CXL host bridges. Each host bridge has 2 CXL Root Por= ts, with the CXL Type3 device directly attached (no switches).:: diff --git a/hw/acpi/cxl.c b/hw/acpi/cxl.c index 2bf8c07993..a8c6166b7f 100644 --- a/hw/acpi/cxl.c +++ b/hw/acpi/cxl.c @@ -254,3 +254,70 @@ void build_cxl_osc_method(Aml *dev) aml_append(dev, aml_name_decl("CTRC", aml_int(0))); aml_append(dev, __build_cxl_osc_method()); } + +static int cxl_device_list(Object *obj, void *opaque) +{ + GSList **list =3D opaque; + + if (object_dynamic_cast(obj, TYPE_CXL_TYPE3)) { + *list =3D g_slist_append(*list, DEVICE(obj)); + } + + object_child_foreach(obj, cxl_device_list, opaque); + return 0; +} + +static GSList *cxl_get_device_list(void) +{ + GSList *list =3D NULL; + + object_child_foreach(qdev_get_machine(), cxl_device_list, &list); + return list; +} + +void cxl_build_srat(GArray *table_data, NodeInfo* numa_info, int nb_numa_n= odes) +{ + GSList *device_list, *list =3D cxl_get_device_list(); + int node =3D 0; + + for (device_list =3D list; device_list; device_list =3D device_list->n= ext) { + DeviceState *dev =3D device_list->data; + CXLType3Dev *ct3d =3D CXL_TYPE3(dev); + MemoryRegion *mr =3D NULL; + + if (ct3d->hostvmem) { + /* Find the numa node associated with this memdev */ + for (node =3D 0; node < nb_numa_nodes; node++) { + if (numa_info[node].node_memdev =3D=3D ct3d->hostvmem) { + break; + } + } + if (node !=3D nb_numa_nodes) { + mr =3D host_memory_backend_get_memory(ct3d->hostvmem); + if (mr) { + build_srat_memory(table_data, mr->addr, mr->size, node, + (MEM_AFFINITY_ENABLED | MEM_AFFINITY_HOTPLUGGABL= E)); + } + } + } + + if (ct3d->hostpmem) { + /* Find the numa node associated with this memdev */ + for (node =3D 0; node < nb_numa_nodes; node++) { + if (numa_info[node].node_memdev =3D=3D ct3d->hostpmem) { + break; + } + } + if (node !=3D nb_numa_nodes) { + mr =3D host_memory_backend_get_memory(ct3d->hostpmem); + if (mr) { + build_srat_memory(table_data, mr->addr, mr->size, node, + (MEM_AFFINITY_ENABLED | MEM_AFFINITY_HOTPLUGGABL= E | + MEM_AFFINITY_NON_VOLATILE)); + } + } + } + } + + g_slist_free(list); +} diff --git a/hw/i386/acpi-build.c b/hw/i386/acpi-build.c index 4f54b61904..af62c888e5 100644 --- a/hw/i386/acpi-build.c +++ b/hw/i386/acpi-build.c @@ -2080,6 +2080,10 @@ build_srat(GArray *table_data, BIOSLinker *linker, M= achineState *machine) * Memory devices may override proximity set by this entry, * providing _PXM method if necessary. */ + if (pcms->cxl_devices_state.is_enabled) { + cxl_build_srat(table_data, numa_info, nb_numa_nodes); + } + if (hotpluggable_address_space_size) { build_srat_memory(table_data, machine->device_memory->base, hotpluggable_address_space_size, nb_numa_nodes -= 1, diff --git a/include/hw/acpi/cxl.h b/include/hw/acpi/cxl.h index acf4418886..b4974297db 100644 --- a/include/hw/acpi/cxl.h +++ b/include/hw/acpi/cxl.h @@ -25,5 +25,6 @@ void cxl_build_cedt(GArray *table_offsets, GArray *table_= data, BIOSLinker *linker, const char *oem_id, const char *oem_table_id, CXLState *cxl_state); void build_cxl_osc_method(Aml *dev); +void cxl_build_srat(GArray *table_data, NodeInfo* numa_info, int nb_numa_n= odes); =20 #endif --=20 2.37.3