From nobody Mon Nov 25 04:54:44 2024 Received: from NAM04-MW2-obe.outbound.protection.outlook.com (mail-mw2nam04on2065.outbound.protection.outlook.com [40.107.101.65]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id F2C5A213159 for ; Wed, 30 Oct 2024 15:52:15 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.101.65 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1730303542; cv=fail; b=gF17NzDk0SryafbJMGYwOAXCCyK54b9qw5h/lBUCDJmg6hqru1G7SN8hhAM44aPpDfs49NSQs25dJkoLyspB4Heib5MF0444sn1WpgTHv+MSXJ7g7r8TDy+uAmyP5dDt5HBd/KqEUNDOB3STQBEQkA8j0hctADc3nfLAlIKlawo= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1730303542; c=relaxed/simple; bh=0Taqd6CVHfGwRdzHYJXRlOdvfvGYlv3tZKOUH1UdZH0=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=pxFxtnobwaYRU1dBFGInPSGxE7qh9rGmLp+PGraM2vd+8OhkIa9/n4mP1f65YBrg3IY8wkZWh+Z2+/C9U4QwnF+tg8TnEHt8QcM17IG0Yn+9uin6LiqJDj8LwaKBcyB5VhMTcovMjCSp66Sx+S2udGi0wJok40QKIZXyQpvWCTw= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=Vq6IZvv0; arc=fail smtp.client-ip=40.107.101.65 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="Vq6IZvv0" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=UXBinUR+v1Gxtx+UhkL3YMlUYdq8PBL1rZH4iETl10EmLPQq72Uk3VB3eNx0x0U4XqmzyBBjBJFmX9lQom4iROiWZFrveYBezk4RJdIdPClTfN+VXyNAW4cmkWw24IX44mfTV5yJ4W0SAcgb5kHGbujC06eekiW7fo2ONYahoEvWt7IFBLKf6AZZ846bYHv/SexJX3+w57W/STr6ZAcpUyEOfbOob+FKasRMHTIndug+3UOUyM3MchOilsnUQVeV/9NnuUWlEhnjtzfTnGmyLWkK6xlW1zPhsaMDj3s6u9+FtQWVHr6QucFc8aPfOi974Sxe8nw3fax0lK0z44s25g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=5CzpcvSqz2ZMZ6PV7knKmbWEXcci27GTFBuQ4reWQlM=; b=SOwHNPF8CgxEFKz/F9Cf1vQ91A5kd0au9o7auLbJsTWVQS4+OQ/XtUhFPYokzs/xps9I5Eqk6ac1wTKYWfihD70CRoOctMgBxGVPOUXfbBAO3sEhi9H5JSxWsxoR2+XrX7ItR772HHNtnlN/MMZBkvnGAh0LHLOgxJPVG+tHkMC2h64lF7r422NFqS8jZwPrQ10yFAfGIo0h+JM0OaLa9TbwD/DnwrhIEBXXZqooqwPfp5beH0955w88MHDDLegFl1/StjOMLzf9PSvaqQ8D+Yw2R44A4uoGS9cSfD0Y+X0BhQHBCSwrstVUWu4DCXsf5biHsfExoZvt/UlmhU0kSQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=kernel.org smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=5CzpcvSqz2ZMZ6PV7knKmbWEXcci27GTFBuQ4reWQlM=; b=Vq6IZvv0zPMrXFqE2b5miFi/LJV8w2fv8ecrQ6vpPtH3SVPwvPd+i/3dGBOg9ud9y6SAZvk9GYQIVVAiKrRB1TNxygtx5Q5438s9SposaKlr3jgwAweDwRTWki5JAg0dDLoCqa0rTy5A2acjiUJ6bqnqXXMSMkWiu4k9MJQh0iM= Received: from MN2PR20CA0036.namprd20.prod.outlook.com (2603:10b6:208:e8::49) by DS7PR12MB8324.namprd12.prod.outlook.com (2603:10b6:8:ec::9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8114.20; Wed, 30 Oct 2024 15:52:12 +0000 Received: from BN3PEPF0000B073.namprd04.prod.outlook.com (2603:10b6:208:e8:cafe::18) by MN2PR20CA0036.outlook.office365.com (2603:10b6:208:e8::49) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8093.17 via Frontend Transport; Wed, 30 Oct 2024 15:52:12 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; pr=C Received: from SATLEXMB04.amd.com (165.204.84.17) by BN3PEPF0000B073.mail.protection.outlook.com (10.167.243.118) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.8114.16 via Frontend Transport; Wed, 30 Oct 2024 15:52:12 +0000 Received: from SATLEXMB06.amd.com (10.181.40.147) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Wed, 30 Oct 2024 10:52:07 -0500 Received: from SATLEXMB03.amd.com (10.181.40.144) by SATLEXMB06.amd.com (10.181.40.147) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Wed, 30 Oct 2024 10:52:07 -0500 Received: from xsjlizhih51.xilinx.com (10.180.168.240) by SATLEXMB03.amd.com (10.181.40.144) with Microsoft SMTP Server id 15.1.2507.39 via Frontend Transport; Wed, 30 Oct 2024 10:52:06 -0500 From: Lizhi Hou To: , , CC: Lizhi Hou , , , , , , George Yang Subject: [PATCH V6 03/10] accel/amdxdna: Support hardware mailbox Date: Wed, 30 Oct 2024 08:51:40 -0700 Message-ID: <20241030155147.103908-4-lizhi.hou@amd.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20241030155147.103908-1-lizhi.hou@amd.com> References: <20241030155147.103908-1-lizhi.hou@amd.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BN3PEPF0000B073:EE_|DS7PR12MB8324:EE_ X-MS-Office365-Filtering-Correlation-Id: 3a61fc48-bdb1-4514-3a46-08dcf8fad1c4 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|1800799024|36860700013|82310400026|376014; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?eTiU2dBiKaE/f3CEmJ1KKlO2BRI/VRyPpQU3FctElGq231T1uQx249RopLBc?= =?us-ascii?Q?3OZZGeiFRYbU0oEz2fQlawDlmifTz4ZXifkl5GIxqebQSx3ztLX/a1Y0eIWg?= =?us-ascii?Q?a15nWm5vE04s2RhWxMB7LS3etTjTmZpqksdw0goJNKXZy9HoE0Xvo7bIEn4Q?= =?us-ascii?Q?vhCTrBQi4ZPUnwZRujj5kIx6L+g7z+f+LameMCbrt1GhJsRIImG/wJxfLbYg?= =?us-ascii?Q?cKcPn0qXVCGXHCalTlMNueTzzEdign7zhSxRQutvtZNbtWuvowSrOxO9j4HC?= =?us-ascii?Q?mrLDLAXCSpIr6i8G1Q8WErHn4YyUz3l2GJ+5xYRqTqbRce2khfnFfDcG5m/Y?= =?us-ascii?Q?/N2amtCKIT4B1gQ7DGhAlEmFiUxb3SwcBwmFnAPeZZfGtDmv1CPq2miAbVmt?= =?us-ascii?Q?nQVAGqydxGnAQCr/HjBGUrj1cNwzLDDAWYJuCQLzy4W0Y/3LHB7U+aeYG1pY?= =?us-ascii?Q?pz2UlDey/RISIwHh8ik6h85WfFCx3rA5nvcgmctT0z2e6GH6NtSqRWBVhZ6w?= =?us-ascii?Q?gS+PlsgG4Y3gmzZPUlk7n8+gmptov2fg+5pyJnrI1Cf8HMJzkLlRBp4l+oRa?= =?us-ascii?Q?xP+B8W4HQgrj2Vy9Zo/A9dSnhJjBzF40SdireUW1zN3ue8oIe+N1oDq5LCD4?= =?us-ascii?Q?fg2RPSJCFggA8+jccj3awphvQi173T5v4mdBh2rUilhivvAOJN8GMd+sBS44?= =?us-ascii?Q?RdNHn/0ieSkXcXevYW8FCRM0anI1KrSWPGkZxijB923l15tuuXFkeq1xwFPZ?= =?us-ascii?Q?TOpMGElDyvKEbXGTgIXKBpJJLabbJqRaUCnAN8qcNiNdECozpYQHnp3ec/54?= =?us-ascii?Q?rGa+n/B+fXEV50ykIrN+nhhfZjV/9JKZyeRAkMTvvruJJcURU3Kb2PTV4FK0?= =?us-ascii?Q?n7abxKTs5k6P6baveLfiREvrT/fGavlTJ/SmAbeqIHVX5eGHae0LxzD7BH+T?= =?us-ascii?Q?spY3XFBZcSvCutj/kWiCFyekW7btT1oICwzblF4n/ciB8FNEqVx9oDDib6Fp?= =?us-ascii?Q?ptrwoqFoldCiQHIMMuFZkwP1MesY7NkGpdyj83RL/CoGfj4YXUIGKTJoT/cg?= =?us-ascii?Q?K6N98i4THyTSe3o2Pn483ndIIuHl5tU1mjSNY0DtTU5z3h+6xCN1BtM+Xz4c?= =?us-ascii?Q?pKFWxdX+G3dGucnc+MicpXyqvyuIv8KknVqaHW1lIRnCK8B6K6Pr6hnovB4E?= =?us-ascii?Q?xNCHUodOY0MTmCvI+LUPg+6LOiAcwUI1pGyZke/7lCF3Slb6p7Dze3JxhOgI?= =?us-ascii?Q?F8tb5cHGymybn/n7diwaDLiaVDuuDI+yIK7D94XKqZYDtaTmfkBLIrp8zKnS?= =?us-ascii?Q?i1bcXPfhCUJ5Rc5uYVl4gZIFdzJx90EnB9tal7RfA3bJyEBJBz9f7rqx9lWk?= =?us-ascii?Q?bRDPrH8=3D?= X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB04.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(1800799024)(36860700013)(82310400026)(376014);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 30 Oct 2024 15:52:12.3797 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 3a61fc48-bdb1-4514-3a46-08dcf8fad1c4 X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB04.amd.com] X-MS-Exchange-CrossTenant-AuthSource: BN3PEPF0000B073.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS7PR12MB8324 Content-Type: text/plain; charset="utf-8" The hardware mailboxes are used by the driver to submit requests to firmware and receive the completion notices from hardware. Initially, a management mailbox channel is up and running. The driver may request firmware to create/destroy more channels dynamically through management channel. Add driver internal mailbox interfaces. - create/destroy a mailbox channel instance - send a message to the firmware through a specific channel - wait for a notification from the specific channel Co-developed-by: George Yang Signed-off-by: George Yang Co-developed-by: Min Ma Signed-off-by: Min Ma Reviewed-by: Jeffrey Hugo Signed-off-by: Lizhi Hou --- MAINTAINERS | 1 + drivers/accel/amdxdna/Makefile | 3 + drivers/accel/amdxdna/aie2_message.c | 194 ++++++ drivers/accel/amdxdna/aie2_msg_priv.h | 370 +++++++++++ drivers/accel/amdxdna/aie2_pci.c | 256 +++++++- drivers/accel/amdxdna/aie2_pci.h | 62 ++ drivers/accel/amdxdna/aie2_psp.c | 2 + drivers/accel/amdxdna/amdxdna_mailbox.c | 576 ++++++++++++++++++ drivers/accel/amdxdna/amdxdna_mailbox.h | 124 ++++ .../accel/amdxdna/amdxdna_mailbox_helper.c | 56 ++ .../accel/amdxdna/amdxdna_mailbox_helper.h | 42 ++ drivers/accel/amdxdna/amdxdna_pci_drv.h | 8 + drivers/accel/amdxdna/amdxdna_sysfs.c | 11 + drivers/accel/amdxdna/npu1_regs.c | 1 + drivers/accel/amdxdna/npu2_regs.c | 1 + drivers/accel/amdxdna/npu4_regs.c | 1 + drivers/accel/amdxdna/npu5_regs.c | 1 + include/trace/events/amdxdna.h | 60 ++ 18 files changed, 1768 insertions(+), 1 deletion(-) create mode 100644 drivers/accel/amdxdna/aie2_message.c create mode 100644 drivers/accel/amdxdna/aie2_msg_priv.h create mode 100644 drivers/accel/amdxdna/amdxdna_mailbox.c create mode 100644 drivers/accel/amdxdna/amdxdna_mailbox.h create mode 100644 drivers/accel/amdxdna/amdxdna_mailbox_helper.c create mode 100644 drivers/accel/amdxdna/amdxdna_mailbox_helper.h create mode 100644 include/trace/events/amdxdna.h diff --git a/MAINTAINERS b/MAINTAINERS index 18f4feaed030..76054963fa3c 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -1186,6 +1186,7 @@ S: Supported T: git https://gitlab.freedesktop.org/drm/misc/kernel.git F: Documentation/accel/amdxdna/ F: drivers/accel/amdxdna/ +F: include/trace/events/amdxdna.h F: include/uapi/drm/amdxdna_accel.h =20 AMD XGBE DRIVER diff --git a/drivers/accel/amdxdna/Makefile b/drivers/accel/amdxdna/Makefile index 1dee0cba8390..1b4e78b43b44 100644 --- a/drivers/accel/amdxdna/Makefile +++ b/drivers/accel/amdxdna/Makefile @@ -1,9 +1,12 @@ # SPDX-License-Identifier: GPL-2.0-only =20 amdxdna-y :=3D \ + aie2_message.o \ aie2_pci.o \ aie2_psp.o \ aie2_smu.o \ + amdxdna_mailbox.o \ + amdxdna_mailbox_helper.o \ amdxdna_pci_drv.o \ amdxdna_sysfs.o \ npu1_regs.o \ diff --git a/drivers/accel/amdxdna/aie2_message.c b/drivers/accel/amdxdna/a= ie2_message.c new file mode 100644 index 000000000000..cbf8ee54c6c2 --- /dev/null +++ b/drivers/accel/amdxdna/aie2_message.c @@ -0,0 +1,194 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * Copyright (C) 2023-2024, Advanced Micro Devices, Inc. + */ + +#include +#include +#include +#include + +#include "aie2_msg_priv.h" +#include "aie2_pci.h" +#include "amdxdna_mailbox.h" +#include "amdxdna_mailbox_helper.h" +#include "amdxdna_pci_drv.h" + +#define DECLARE_AIE2_MSG(name, op) \ + DECLARE_XDNA_MSG_COMMON(name, op, MAX_AIE2_STATUS_CODE) + +static int aie2_send_mgmt_msg_wait(struct amdxdna_dev_hdl *ndev, + struct xdna_mailbox_msg *msg) +{ + struct amdxdna_dev *xdna =3D ndev->xdna; + struct xdna_notify *hdl =3D msg->handle; + int ret; + + if (!ndev->mgmt_chann) + return -ENODEV; + + drm_WARN_ON(&xdna->ddev, !mutex_is_locked(&xdna->dev_lock)); + ret =3D xdna_send_msg_wait(xdna, ndev->mgmt_chann, msg); + if (ret =3D=3D -ETIME) { + xdna_mailbox_stop_channel(ndev->mgmt_chann); + xdna_mailbox_destroy_channel(ndev->mgmt_chann); + ndev->mgmt_chann =3D NULL; + } + + if (!ret && *hdl->data !=3D AIE2_STATUS_SUCCESS) { + XDNA_ERR(xdna, "command opcode 0x%x failed, status 0x%x", + msg->opcode, *hdl->data); + ret =3D -EINVAL; + } + + return ret; +} + +int aie2_suspend_fw(struct amdxdna_dev_hdl *ndev) +{ + DECLARE_AIE2_MSG(suspend, MSG_OP_SUSPEND); + + return aie2_send_mgmt_msg_wait(ndev, &msg); +} + +int aie2_resume_fw(struct amdxdna_dev_hdl *ndev) +{ + DECLARE_AIE2_MSG(suspend, MSG_OP_RESUME); + + return aie2_send_mgmt_msg_wait(ndev, &msg); +} + +int aie2_set_runtime_cfg(struct amdxdna_dev_hdl *ndev, u32 type, u64 value) +{ + DECLARE_AIE2_MSG(set_runtime_cfg, MSG_OP_SET_RUNTIME_CONFIG); + + req.type =3D type; + req.value =3D value; + + return aie2_send_mgmt_msg_wait(ndev, &msg); +} + +int aie2_get_runtime_cfg(struct amdxdna_dev_hdl *ndev, u32 type, u64 *valu= e) +{ + DECLARE_AIE2_MSG(get_runtime_cfg, MSG_OP_GET_RUNTIME_CONFIG); + int ret; + + req.type =3D type; + ret =3D aie2_send_mgmt_msg_wait(ndev, &msg); + if (ret) { + XDNA_ERR(ndev->xdna, "Failed to get runtime config, ret %d", ret); + return ret; + } + + *value =3D resp.value; + return 0; +} + +int aie2_check_protocol_version(struct amdxdna_dev_hdl *ndev) +{ + DECLARE_AIE2_MSG(protocol_version, MSG_OP_GET_PROTOCOL_VERSION); + struct amdxdna_dev *xdna =3D ndev->xdna; + int ret; + + ret =3D aie2_send_mgmt_msg_wait(ndev, &msg); + if (ret) { + XDNA_ERR(xdna, "Failed to get protocol version, ret %d", ret); + return ret; + } + + if (resp.major !=3D ndev->priv->protocol_major) { + XDNA_ERR(xdna, "Incompatible firmware protocol version major %d minor %d= ", + resp.major, resp.minor); + return -EINVAL; + } + + if (resp.minor < ndev->priv->protocol_minor) { + XDNA_ERR(xdna, "Firmware minor version smaller than supported"); + return -EINVAL; + } + + return 0; +} + +int aie2_assign_mgmt_pasid(struct amdxdna_dev_hdl *ndev, u16 pasid) +{ + DECLARE_AIE2_MSG(assign_mgmt_pasid, MSG_OP_ASSIGN_MGMT_PASID); + + req.pasid =3D pasid; + + return aie2_send_mgmt_msg_wait(ndev, &msg); +} + +int aie2_query_aie_version(struct amdxdna_dev_hdl *ndev, struct aie_versio= n *version) +{ + DECLARE_AIE2_MSG(aie_version_info, MSG_OP_QUERY_AIE_VERSION); + struct amdxdna_dev *xdna =3D ndev->xdna; + int ret; + + ret =3D aie2_send_mgmt_msg_wait(ndev, &msg); + if (ret) + return ret; + + XDNA_DBG(xdna, "Query AIE version - major: %u minor: %u completed", + resp.major, resp.minor); + + version->major =3D resp.major; + version->minor =3D resp.minor; + + return 0; +} + +int aie2_query_aie_metadata(struct amdxdna_dev_hdl *ndev, struct aie_metad= ata *metadata) +{ + DECLARE_AIE2_MSG(aie_tile_info, MSG_OP_QUERY_AIE_TILE_INFO); + int ret; + + ret =3D aie2_send_mgmt_msg_wait(ndev, &msg); + if (ret) + return ret; + + metadata->size =3D resp.info.size; + metadata->cols =3D resp.info.cols; + metadata->rows =3D resp.info.rows; + + metadata->version.major =3D resp.info.major; + metadata->version.minor =3D resp.info.minor; + + metadata->core.row_count =3D resp.info.core_rows; + metadata->core.row_start =3D resp.info.core_row_start; + metadata->core.dma_channel_count =3D resp.info.core_dma_channels; + metadata->core.lock_count =3D resp.info.core_locks; + metadata->core.event_reg_count =3D resp.info.core_events; + + metadata->mem.row_count =3D resp.info.mem_rows; + metadata->mem.row_start =3D resp.info.mem_row_start; + metadata->mem.dma_channel_count =3D resp.info.mem_dma_channels; + metadata->mem.lock_count =3D resp.info.mem_locks; + metadata->mem.event_reg_count =3D resp.info.mem_events; + + metadata->shim.row_count =3D resp.info.shim_rows; + metadata->shim.row_start =3D resp.info.shim_row_start; + metadata->shim.dma_channel_count =3D resp.info.shim_dma_channels; + metadata->shim.lock_count =3D resp.info.shim_locks; + metadata->shim.event_reg_count =3D resp.info.shim_events; + + return 0; +} + +int aie2_query_firmware_version(struct amdxdna_dev_hdl *ndev, + struct amdxdna_fw_ver *fw_ver) +{ + DECLARE_AIE2_MSG(firmware_version, MSG_OP_GET_FIRMWARE_VERSION); + int ret; + + ret =3D aie2_send_mgmt_msg_wait(ndev, &msg); + if (ret) + return ret; + + fw_ver->major =3D resp.major; + fw_ver->minor =3D resp.minor; + fw_ver->sub =3D resp.sub; + fw_ver->build =3D resp.build; + + return 0; +} diff --git a/drivers/accel/amdxdna/aie2_msg_priv.h b/drivers/accel/amdxdna/= aie2_msg_priv.h new file mode 100644 index 000000000000..4e02e744b470 --- /dev/null +++ b/drivers/accel/amdxdna/aie2_msg_priv.h @@ -0,0 +1,370 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* + * Copyright (C) 2022-2024, Advanced Micro Devices, Inc. + */ + +#ifndef _AIE2_MSG_PRIV_H_ +#define _AIE2_MSG_PRIV_H_ + +enum aie2_msg_opcode { + MSG_OP_CREATE_CONTEXT =3D 0x2, + MSG_OP_DESTROY_CONTEXT =3D 0x3, + MSG_OP_SYNC_BO =3D 0x7, + MSG_OP_EXECUTE_BUFFER_CF =3D 0xC, + MSG_OP_QUERY_COL_STATUS =3D 0xD, + MSG_OP_QUERY_AIE_TILE_INFO =3D 0xE, + MSG_OP_QUERY_AIE_VERSION =3D 0xF, + MSG_OP_EXEC_DPU =3D 0x10, + MSG_OP_CONFIG_CU =3D 0x11, + MSG_OP_CHAIN_EXEC_BUFFER_CF =3D 0x12, + MSG_OP_CHAIN_EXEC_DPU =3D 0x13, + MSG_OP_MAX_XRT_OPCODE, + MSG_OP_SUSPEND =3D 0x101, + MSG_OP_RESUME =3D 0x102, + MSG_OP_ASSIGN_MGMT_PASID =3D 0x103, + MSG_OP_INVOKE_SELF_TEST =3D 0x104, + MSG_OP_MAP_HOST_BUFFER =3D 0x106, + MSG_OP_GET_FIRMWARE_VERSION =3D 0x108, + MSG_OP_SET_RUNTIME_CONFIG =3D 0x10A, + MSG_OP_GET_RUNTIME_CONFIG =3D 0x10B, + MSG_OP_REGISTER_ASYNC_EVENT_MSG =3D 0x10C, + MSG_OP_MAX_DRV_OPCODE, + MSG_OP_GET_PROTOCOL_VERSION =3D 0x301, + MSG_OP_MAX_OPCODE +}; + +enum aie2_msg_status { + AIE2_STATUS_SUCCESS =3D 0x0, + /* AIE Error codes */ + AIE2_STATUS_AIE_SATURATION_ERROR =3D 0x1000001, + AIE2_STATUS_AIE_FP_ERROR =3D 0x1000002, + AIE2_STATUS_AIE_STREAM_ERROR =3D 0x1000003, + AIE2_STATUS_AIE_ACCESS_ERROR =3D 0x1000004, + AIE2_STATUS_AIE_BUS_ERROR =3D 0x1000005, + AIE2_STATUS_AIE_INSTRUCTION_ERROR =3D 0x1000006, + AIE2_STATUS_AIE_ECC_ERROR =3D 0x1000007, + AIE2_STATUS_AIE_LOCK_ERROR =3D 0x1000008, + AIE2_STATUS_AIE_DMA_ERROR =3D 0x1000009, + AIE2_STATUS_AIE_MEM_PARITY_ERROR =3D 0x100000a, + AIE2_STATUS_AIE_PWR_CFG_ERROR =3D 0x100000b, + AIE2_STATUS_AIE_BACKTRACK_ERROR =3D 0x100000c, + AIE2_STATUS_MAX_AIE_STATUS_CODE, + /* MGMT ERT Error codes */ + AIE2_STATUS_MGMT_ERT_SELF_TEST_FAILURE =3D 0x2000001, + AIE2_STATUS_MGMT_ERT_HASH_MISMATCH, + AIE2_STATUS_MGMT_ERT_NOAVAIL, + AIE2_STATUS_MGMT_ERT_INVALID_PARAM, + AIE2_STATUS_MGMT_ERT_ENTER_SUSPEND_FAILURE, + AIE2_STATUS_MGMT_ERT_BUSY, + AIE2_STATUS_MGMT_ERT_APPLICATION_ACTIVE, + MAX_MGMT_ERT_STATUS_CODE, + /* APP ERT Error codes */ + AIE2_STATUS_APP_ERT_FIRST_ERROR =3D 0x3000001, + AIE2_STATUS_APP_INVALID_INSTR, + AIE2_STATUS_APP_LOAD_PDI_FAIL, + MAX_APP_ERT_STATUS_CODE, + /* NPU RTOS Error Codes */ + AIE2_STATUS_INVALID_INPUT_BUFFER =3D 0x4000001, + AIE2_STATUS_INVALID_COMMAND, + AIE2_STATUS_INVALID_PARAM, + AIE2_STATUS_INVALID_OPERATION =3D 0x4000006, + AIE2_STATUS_ASYNC_EVENT_MSGS_FULL, + AIE2_STATUS_MAX_RTOS_STATUS_CODE, + MAX_AIE2_STATUS_CODE +}; + +struct assign_mgmt_pasid_req { + __u16 pasid; + __u16 reserved; +} __packed; + +struct assign_mgmt_pasid_resp { + enum aie2_msg_status status; +} __packed; + +struct map_host_buffer_req { + __u32 context_id; + __u64 buf_addr; + __u64 buf_size; +} __packed; + +struct map_host_buffer_resp { + enum aie2_msg_status status; +} __packed; + +#define MAX_CQ_PAIRS 2 +struct cq_info { + __u32 head_addr; + __u32 tail_addr; + __u32 buf_addr; + __u32 buf_size; +}; + +struct cq_pair { + struct cq_info x2i_q; + struct cq_info i2x_q; +}; + +struct create_ctx_req { + __u32 aie_type; + __u8 start_col; + __u8 num_col; + __u16 reserved; + __u8 num_cq_pairs_requested; + __u8 reserved1; + __u16 pasid; + __u32 pad[2]; + __u32 sec_comm_target_type; + __u32 context_priority; +} __packed; + +struct create_ctx_resp { + enum aie2_msg_status status; + __u32 context_id; + __u16 msix_id; + __u8 num_cq_pairs_allocated; + __u8 reserved; + struct cq_pair cq_pair[MAX_CQ_PAIRS]; +} __packed; + +struct destroy_ctx_req { + __u32 context_id; +} __packed; + +struct destroy_ctx_resp { + enum aie2_msg_status status; +} __packed; + +struct execute_buffer_req { + __u32 cu_idx; + __u32 payload[19]; +} __packed; + +struct exec_dpu_req { + __u64 inst_buf_addr; + __u32 inst_size; + __u32 inst_prop_cnt; + __u32 cu_idx; + __u32 payload[35]; +} __packed; + +struct execute_buffer_resp { + enum aie2_msg_status status; +} __packed; + +struct aie_tile_info { + __u32 size; + __u16 major; + __u16 minor; + __u16 cols; + __u16 rows; + __u16 core_rows; + __u16 mem_rows; + __u16 shim_rows; + __u16 core_row_start; + __u16 mem_row_start; + __u16 shim_row_start; + __u16 core_dma_channels; + __u16 mem_dma_channels; + __u16 shim_dma_channels; + __u16 core_locks; + __u16 mem_locks; + __u16 shim_locks; + __u16 core_events; + __u16 mem_events; + __u16 shim_events; + __u16 reserved; +}; + +struct aie_tile_info_req { + __u32 reserved; +} __packed; + +struct aie_tile_info_resp { + enum aie2_msg_status status; + struct aie_tile_info info; +} __packed; + +struct aie_version_info_req { + __u32 reserved; +} __packed; + +struct aie_version_info_resp { + enum aie2_msg_status status; + __u16 major; + __u16 minor; +} __packed; + +struct aie_column_info_req { + __u64 dump_buff_addr; + __u32 dump_buff_size; + __u32 num_cols; + __u32 aie_bitmap; +} __packed; + +struct aie_column_info_resp { + enum aie2_msg_status status; + __u32 size; +} __packed; + +struct suspend_req { + __u32 place_holder; +} __packed; + +struct suspend_resp { + enum aie2_msg_status status; +} __packed; + +struct resume_req { + __u32 place_holder; +} __packed; + +struct resume_resp { + enum aie2_msg_status status; +} __packed; + +struct check_header_hash_req { + __u64 hash_high; + __u64 hash_low; +} __packed; + +struct check_header_hash_resp { + enum aie2_msg_status status; +} __packed; + +struct query_error_req { + __u64 buf_addr; + __u32 buf_size; + __u32 next_row; + __u32 next_column; + __u32 next_module; +} __packed; + +struct query_error_resp { + enum aie2_msg_status status; + __u32 num_err; + __u32 has_next_err; + __u32 next_row; + __u32 next_column; + __u32 next_module; +} __packed; + +struct protocol_version_req { + __u32 reserved; +} __packed; + +struct protocol_version_resp { + enum aie2_msg_status status; + __u32 major; + __u32 minor; +} __packed; + +struct firmware_version_req { + __u32 reserved; +} __packed; + +struct firmware_version_resp { + enum aie2_msg_status status; + __u32 major; + __u32 minor; + __u32 sub; + __u32 build; +} __packed; + +#define MAX_NUM_CUS 32 +#define AIE2_MSG_CFG_CU_PDI_ADDR GENMASK(16, 0) +#define AIE2_MSG_CFG_CU_FUNC GENMASK(24, 17) +struct config_cu_req { + __u32 num_cus; + __u32 cfgs[MAX_NUM_CUS]; +} __packed; + +struct config_cu_resp { + enum aie2_msg_status status; +} __packed; + +struct set_runtime_cfg_req { + __u32 type; + __u64 value; +} __packed; + +struct set_runtime_cfg_resp { + enum aie2_msg_status status; +} __packed; + +struct get_runtime_cfg_req { + __u32 type; +} __packed; + +struct get_runtime_cfg_resp { + enum aie2_msg_status status; + __u64 value; +} __packed; + +enum async_event_type { + ASYNC_EVENT_TYPE_AIE_ERROR, + ASYNC_EVENT_TYPE_EXCEPTION, + MAX_ASYNC_EVENT_TYPE +}; + +#define ASYNC_BUF_SIZE SZ_8K +struct async_event_msg_req { + __u64 buf_addr; + __u32 buf_size; +} __packed; + +struct async_event_msg_resp { + enum aie2_msg_status status; + enum async_event_type type; +} __packed; + +#define MAX_CHAIN_CMDBUF_SIZE SZ_4K +#define slot_cf_has_space(offset, payload_size) \ + (MAX_CHAIN_CMDBUF_SIZE - ((offset) + (payload_size)) > \ + offsetof(struct cmd_chain_slot_execbuf_cf, args[0])) +struct cmd_chain_slot_execbuf_cf { + __u32 cu_idx; + __u32 arg_cnt; + __u32 args[] __counted_by(arg_cnt); +}; + +#define slot_dpu_has_space(offset, payload_size) \ + (MAX_CHAIN_CMDBUF_SIZE - ((offset) + (payload_size)) > \ + offsetof(struct cmd_chain_slot_dpu, args[0])) +struct cmd_chain_slot_dpu { + __u64 inst_buf_addr; + __u32 inst_size; + __u32 inst_prop_cnt; + __u32 cu_idx; + __u32 arg_cnt; +#define MAX_DPU_ARGS_SIZE (34 * sizeof(__u32)) + __u32 args[] __counted_by(arg_cnt); +}; + +struct cmd_chain_req { + __u64 buf_addr; + __u32 buf_size; + __u32 count; +} __packed; + +struct cmd_chain_resp { + enum aie2_msg_status status; + __u32 fail_cmd_idx; + enum aie2_msg_status fail_cmd_status; +} __packed; + +#define AIE2_MSG_SYNC_BO_SRC_TYPE GENMASK(3, 0) +#define AIE2_MSG_SYNC_BO_DST_TYPE GENMASK(7, 4) +struct sync_bo_req { + __u64 src_addr; + __u64 dst_addr; + __u32 size; +#define SYNC_BO_DEV_MEM 0 +#define SYNC_BO_HOST_MEM 2 + __u32 type; +} __packed; + +struct sync_bo_resp { + enum aie2_msg_status status; +} __packed; +#endif /* _AIE2_MSG_PRIV_H_ */ diff --git a/drivers/accel/amdxdna/aie2_pci.c b/drivers/accel/amdxdna/aie2_= pci.c index bf1005af8b00..73dfcbae2ea7 100644 --- a/drivers/accel/amdxdna/aie2_pci.c +++ b/drivers/accel/amdxdna/aie2_pci.c @@ -9,16 +9,210 @@ #include #include #include +#include #include =20 +#include "aie2_msg_priv.h" #include "aie2_pci.h" +#include "amdxdna_mailbox.h" #include "amdxdna_pci_drv.h" =20 +/* + * The management mailbox channel is allocated by firmware. + * The related register and ring buffer information is on SRAM BAR. + * This struct is the register layout. + */ +struct mgmt_mbox_chann_info { + u32 x2i_tail; + u32 x2i_head; + u32 x2i_buf; + u32 x2i_buf_sz; + u32 i2x_tail; + u32 i2x_head; + u32 i2x_buf; + u32 i2x_buf_sz; +}; + +static void aie2_dump_chann_info_debug(struct amdxdna_dev_hdl *ndev) +{ + struct amdxdna_dev *xdna =3D ndev->xdna; + + XDNA_DBG(xdna, "i2x tail 0x%x", ndev->mgmt_i2x.mb_tail_ptr_reg); + XDNA_DBG(xdna, "i2x head 0x%x", ndev->mgmt_i2x.mb_head_ptr_reg); + XDNA_DBG(xdna, "i2x ringbuf 0x%x", ndev->mgmt_i2x.rb_start_addr); + XDNA_DBG(xdna, "i2x rsize 0x%x", ndev->mgmt_i2x.rb_size); + XDNA_DBG(xdna, "x2i tail 0x%x", ndev->mgmt_x2i.mb_tail_ptr_reg); + XDNA_DBG(xdna, "x2i head 0x%x", ndev->mgmt_x2i.mb_head_ptr_reg); + XDNA_DBG(xdna, "x2i ringbuf 0x%x", ndev->mgmt_x2i.rb_start_addr); + XDNA_DBG(xdna, "x2i rsize 0x%x", ndev->mgmt_x2i.rb_size); + XDNA_DBG(xdna, "x2i chann index 0x%x", ndev->mgmt_chan_idx); +} + +static int aie2_get_mgmt_chann_info(struct amdxdna_dev_hdl *ndev) +{ + struct mgmt_mbox_chann_info info_regs; + struct xdna_mailbox_chann_res *i2x; + struct xdna_mailbox_chann_res *x2i; + u32 addr, off; + u32 *reg; + int ret; + int i; + + /* + * Once firmware is alive, it will write management channel + * information in SRAM BAR and write the address of that information + * at FW_ALIVE_OFF offset in SRMA BAR. + * + * Read a non-zero value from FW_ALIVE_OFF implies that firmware + * is alive. + */ + ret =3D readx_poll_timeout(readl, SRAM_GET_ADDR(ndev, FW_ALIVE_OFF), + addr, addr, AIE2_INTERVAL, AIE2_TIMEOUT); + if (ret || !addr) + return -ETIME; + + off =3D AIE2_SRAM_OFF(ndev, addr); + reg =3D (u32 *)&info_regs; + for (i =3D 0; i < sizeof(info_regs) / sizeof(u32); i++) + reg[i] =3D readl(ndev->sram_base + off + i * sizeof(u32)); + + i2x =3D &ndev->mgmt_i2x; + x2i =3D &ndev->mgmt_x2i; + + i2x->mb_head_ptr_reg =3D AIE2_MBOX_OFF(ndev, info_regs.i2x_head); + i2x->mb_tail_ptr_reg =3D AIE2_MBOX_OFF(ndev, info_regs.i2x_tail); + i2x->rb_start_addr =3D AIE2_SRAM_OFF(ndev, info_regs.i2x_buf); + i2x->rb_size =3D info_regs.i2x_buf_sz; + + x2i->mb_head_ptr_reg =3D AIE2_MBOX_OFF(ndev, info_regs.x2i_head); + x2i->mb_tail_ptr_reg =3D AIE2_MBOX_OFF(ndev, info_regs.x2i_tail); + x2i->rb_start_addr =3D AIE2_SRAM_OFF(ndev, info_regs.x2i_buf); + x2i->rb_size =3D info_regs.x2i_buf_sz; + ndev->mgmt_chan_idx =3D CHANN_INDEX(ndev, x2i->rb_start_addr); + + aie2_dump_chann_info_debug(ndev); + + /* Must clear address at FW_ALIVE_OFF */ + writel(0, SRAM_GET_ADDR(ndev, FW_ALIVE_OFF)); + + return 0; +} + +static int aie2_runtime_cfg(struct amdxdna_dev_hdl *ndev) +{ + const struct rt_config *cfg =3D &ndev->priv->rt_config; + u64 value; + int ret; + + ret =3D aie2_set_runtime_cfg(ndev, cfg->type, cfg->value); + if (ret) { + XDNA_ERR(ndev->xdna, "Set runtime type %d value %d failed", + cfg->type, cfg->value); + return ret; + } + + ret =3D aie2_get_runtime_cfg(ndev, cfg->type, &value); + if (ret) { + XDNA_ERR(ndev->xdna, "Get runtime cfg failed"); + return ret; + } + + if (value !=3D cfg->value) + return -EINVAL; + + return 0; +} + +static int aie2_xdna_reset(struct amdxdna_dev_hdl *ndev) +{ + int ret; + + ret =3D aie2_suspend_fw(ndev); + if (ret) { + XDNA_ERR(ndev->xdna, "Suspend firmware failed"); + return ret; + } + + ret =3D aie2_resume_fw(ndev); + if (ret) { + XDNA_ERR(ndev->xdna, "Resume firmware failed"); + return ret; + } + + return 0; +} + +static int aie2_mgmt_fw_init(struct amdxdna_dev_hdl *ndev) +{ + int ret; + + ret =3D aie2_check_protocol_version(ndev); + if (ret) { + XDNA_ERR(ndev->xdna, "Check header hash failed"); + return ret; + } + + ret =3D aie2_runtime_cfg(ndev); + if (ret) { + XDNA_ERR(ndev->xdna, "Runtime config failed"); + return ret; + } + + ret =3D aie2_assign_mgmt_pasid(ndev, 0); + if (ret) { + XDNA_ERR(ndev->xdna, "Can not assign PASID"); + return ret; + } + + ret =3D aie2_xdna_reset(ndev); + if (ret) { + XDNA_ERR(ndev->xdna, "Reset firmware failed"); + return ret; + } + + return 0; +} + +static int aie2_mgmt_fw_query(struct amdxdna_dev_hdl *ndev) +{ + int ret; + + ret =3D aie2_query_firmware_version(ndev, &ndev->xdna->fw_ver); + if (ret) { + XDNA_ERR(ndev->xdna, "query firmware version failed"); + return ret; + } + + ret =3D aie2_query_aie_version(ndev, &ndev->version); + if (ret) { + XDNA_ERR(ndev->xdna, "Query AIE version failed"); + return ret; + } + + ret =3D aie2_query_aie_metadata(ndev, &ndev->metadata); + if (ret) { + XDNA_ERR(ndev->xdna, "Query AIE metadata failed"); + return ret; + } + + return 0; +} + +static void aie2_mgmt_fw_fini(struct amdxdna_dev_hdl *ndev) +{ + if (aie2_suspend_fw(ndev)) + XDNA_ERR(ndev->xdna, "Suspend_fw failed"); + XDNA_DBG(ndev->xdna, "Firmware suspended"); +} + static void aie2_hw_stop(struct amdxdna_dev *xdna) { struct pci_dev *pdev =3D to_pci_dev(xdna->ddev.dev); struct amdxdna_dev_hdl *ndev =3D xdna->dev_handle; =20 + aie2_mgmt_fw_fini(ndev); + xdna_mailbox_stop_channel(ndev->mgmt_chann); + xdna_mailbox_destroy_channel(ndev->mgmt_chann); aie2_psp_stop(ndev->psp_hdl); aie2_smu_fini(ndev); pci_disable_device(pdev); @@ -28,7 +222,9 @@ static int aie2_hw_start(struct amdxdna_dev *xdna) { struct pci_dev *pdev =3D to_pci_dev(xdna->ddev.dev); struct amdxdna_dev_hdl *ndev =3D xdna->dev_handle; - int ret; + struct xdna_mailbox_res mbox_res; + u32 xdna_mailbox_intr_reg; + int mgmt_mb_irq, ret; =20 ret =3D pci_enable_device(pdev); if (ret) { @@ -49,8 +245,56 @@ static int aie2_hw_start(struct amdxdna_dev *xdna) goto fini_smu; } =20 + ret =3D aie2_get_mgmt_chann_info(ndev); + if (ret) { + XDNA_ERR(xdna, "firmware is not alive"); + goto stop_psp; + } + + mbox_res.ringbuf_base =3D (u64)ndev->sram_base; + mbox_res.ringbuf_size =3D pci_resource_len(pdev, xdna->dev_info->sram_bar= ); + mbox_res.mbox_base =3D (u64)ndev->mbox_base; + mbox_res.mbox_size =3D MBOX_SIZE(ndev); + mbox_res.name =3D "xdna_mailbox"; + ndev->mbox =3D xdnam_mailbox_create(&xdna->ddev, &mbox_res); + if (!ndev->mbox) { + XDNA_ERR(xdna, "failed to create mailbox device"); + ret =3D -ENODEV; + goto stop_psp; + } + + mgmt_mb_irq =3D pci_irq_vector(pdev, ndev->mgmt_chan_idx); + if (mgmt_mb_irq < 0) { + ret =3D mgmt_mb_irq; + XDNA_ERR(xdna, "failed to alloc irq vector, ret %d", ret); + goto stop_psp; + } + + xdna_mailbox_intr_reg =3D ndev->mgmt_i2x.mb_head_ptr_reg + 4; + ndev->mgmt_chann =3D xdna_mailbox_create_channel(ndev->mbox, + &ndev->mgmt_x2i, + &ndev->mgmt_i2x, + xdna_mailbox_intr_reg, + mgmt_mb_irq); + if (!ndev->mgmt_chann) { + XDNA_ERR(xdna, "failed to create management mailbox channel"); + ret =3D -EINVAL; + goto stop_psp; + } + + ret =3D aie2_mgmt_fw_init(ndev); + if (ret) { + XDNA_ERR(xdna, "initial mgmt firmware failed, ret %d", ret); + goto destroy_mgmt_chann; + } + return 0; =20 +destroy_mgmt_chann: + xdna_mailbox_stop_channel(ndev->mgmt_chann); + xdna_mailbox_destroy_channel(ndev->mgmt_chann); +stop_psp: + aie2_psp_stop(ndev->psp_hdl); fini_smu: aie2_smu_fini(ndev); disable_dev: @@ -108,6 +352,7 @@ static int aie2_init(struct amdxdna_dev *xdna) =20 ndev->sram_base =3D tbl[xdna->dev_info->sram_bar]; ndev->smu_base =3D tbl[xdna->dev_info->smu_bar]; + ndev->mbox_base =3D tbl[xdna->dev_info->mbox_bar]; =20 ret =3D dma_set_mask_and_coherent(&pdev->dev, DMA_BIT_MASK(64)); if (ret) { @@ -152,9 +397,18 @@ static int aie2_init(struct amdxdna_dev *xdna) goto disable_sva; } =20 + ret =3D aie2_mgmt_fw_query(ndev); + if (ret) { + XDNA_ERR(xdna, "Query firmware failed, ret %d", ret); + goto stop_hw; + } + ndev->total_col =3D ndev->metadata.cols; + release_firmware(fw); return 0; =20 +stop_hw: + aie2_hw_stop(xdna); disable_sva: iommu_dev_disable_feature(&pdev->dev, IOMMU_DEV_FEAT_SVA); free_irq: diff --git a/drivers/accel/amdxdna/aie2_pci.h b/drivers/accel/amdxdna/aie2_= pci.h index 34f344b4b662..4c81d10a0998 100644 --- a/drivers/accel/amdxdna/aie2_pci.h +++ b/drivers/accel/amdxdna/aie2_pci.h @@ -6,6 +6,8 @@ #ifndef _AIE2_PCI_H_ #define _AIE2_PCI_H_ =20 +#include "amdxdna_mailbox.h" + #define AIE2_INTERVAL 20000 /* us */ #define AIE2_TIMEOUT 1000000 /* us */ =20 @@ -33,6 +35,17 @@ ((_ndev)->sram_base + SRAM_REG_OFF((_ndev), (idx))); \ }) =20 +#define CHAN_SLOT_SZ SZ_8K +#define CHANN_INDEX(ndev, rbuf_off) \ + (((rbuf_off) - SRAM_REG_OFF((ndev), MBOX_CHANN_OFF)) / CHAN_SLOT_SZ) + +#define MBOX_SIZE(ndev) \ +({ \ + typeof(ndev) _ndev =3D (ndev); \ + ((_ndev)->priv->mbox_size) ? (_ndev)->priv->mbox_size : \ + pci_resource_len(NDEV2PDEV(_ndev), (_ndev)->xdna->dev_info->mbox_bar); \ +}) + #define SMU_MPNPUCLK_FREQ_MAX(ndev) ((ndev)->priv->smu_mpnpuclk_freq_max) #define SMU_HCLK_FREQ_MAX(ndev) ((ndev)->priv->smu_hclk_freq_max) =20 @@ -63,12 +76,37 @@ enum psp_reg_idx { PSP_MAX_REGS /* Keep this at the end */ }; =20 +struct amdxdna_fw_ver; + struct psp_config { const void *fw_buf; u32 fw_size; void __iomem *psp_regs[PSP_MAX_REGS]; }; =20 +struct aie_version { + u16 major; + u16 minor; +}; + +struct aie_tile_metadata { + u16 row_count; + u16 row_start; + u16 dma_channel_count; + u16 lock_count; + u16 event_reg_count; +}; + +struct aie_metadata { + u32 size; + u16 cols; + u16 rows; + struct aie_version version; + struct aie_tile_metadata core; + struct aie_tile_metadata mem; + struct aie_tile_metadata shim; +}; + struct clock_entry { char name[16]; u32 freq_mhz; @@ -84,9 +122,22 @@ struct amdxdna_dev_hdl { const struct amdxdna_dev_priv *priv; void __iomem *sram_base; void __iomem *smu_base; + void __iomem *mbox_base; struct psp_device *psp_hdl; + + struct xdna_mailbox_chann_res mgmt_x2i; + struct xdna_mailbox_chann_res mgmt_i2x; + u32 mgmt_chan_idx; + + u32 total_col; + struct aie_version version; + struct aie_metadata metadata; struct clock_entry mp_npu_clock; struct clock_entry h_clock; + + /* Mailbox and the management channel */ + struct mailbox *mbox; + struct mailbox_channel *mgmt_chann; }; =20 #define DEFINE_BAR_OFFSET(reg_name, bar, reg_addr) \ @@ -127,4 +178,15 @@ struct psp_device *aie2m_psp_create(struct drm_device = *ddev, struct psp_config * int aie2_psp_start(struct psp_device *psp); void aie2_psp_stop(struct psp_device *psp); =20 +/* aie2_message.c */ +int aie2_suspend_fw(struct amdxdna_dev_hdl *ndev); +int aie2_resume_fw(struct amdxdna_dev_hdl *ndev); +int aie2_set_runtime_cfg(struct amdxdna_dev_hdl *ndev, u32 type, u64 value= ); +int aie2_get_runtime_cfg(struct amdxdna_dev_hdl *ndev, u32 type, u64 *valu= e); +int aie2_check_protocol_version(struct amdxdna_dev_hdl *ndev); +int aie2_assign_mgmt_pasid(struct amdxdna_dev_hdl *ndev, u16 pasid); +int aie2_query_aie_version(struct amdxdna_dev_hdl *ndev, struct aie_versio= n *version); +int aie2_query_aie_metadata(struct amdxdna_dev_hdl *ndev, struct aie_metad= ata *metadata); +int aie2_query_firmware_version(struct amdxdna_dev_hdl *ndev, + struct amdxdna_fw_ver *fw_ver); #endif /* _AIE2_PCI_H_ */ diff --git a/drivers/accel/amdxdna/aie2_psp.c b/drivers/accel/amdxdna/aie2_= psp.c index 610ed8aa824a..b03501e81065 100644 --- a/drivers/accel/amdxdna/aie2_psp.c +++ b/drivers/accel/amdxdna/aie2_psp.c @@ -10,6 +10,8 @@ #include =20 #include "aie2_pci.h" +#include "amdxdna_mailbox.h" +#include "amdxdna_pci_drv.h" =20 #define PSP_STATUS_READY BIT(31) =20 diff --git a/drivers/accel/amdxdna/amdxdna_mailbox.c b/drivers/accel/amdxdn= a/amdxdna_mailbox.c new file mode 100644 index 000000000000..415d99abaaa3 --- /dev/null +++ b/drivers/accel/amdxdna/amdxdna_mailbox.c @@ -0,0 +1,576 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * Copyright (C) 2022-2024, Advanced Micro Devices, Inc. + */ + +#include +#include +#include +#include + +#define CREATE_TRACE_POINTS +#include + +#include "amdxdna_mailbox.h" + +#define MB_ERR(chann, fmt, args...) \ +({ \ + typeof(chann) _chann =3D chann; \ + dev_err((_chann)->mb->dev, "xdna_mailbox.%d: "fmt, \ + (_chann)->msix_irq, ##args); \ +}) +#define MB_DBG(chann, fmt, args...) \ +({ \ + typeof(chann) _chann =3D chann; \ + dev_dbg((_chann)->mb->dev, "xdna_mailbox.%d: "fmt, \ + (_chann)->msix_irq, ##args); \ +}) +#define MB_WARN_ONCE(chann, fmt, args...) \ +({ \ + typeof(chann) _chann =3D chann; \ + dev_warn_once((_chann)->mb->dev, "xdna_mailbox.%d: "fmt, \ + (_chann)->msix_irq, ##args); \ +}) + +#define MAGIC_VAL 0x1D000000U +#define MAGIC_VAL_MASK 0xFF000000 +#define MAX_MSG_ID_ENTRIES 256 +#define MSG_RX_TIMER 200 /* milliseconds */ +#define MAILBOX_NAME "xdna_mailbox" + +enum channel_res_type { + CHAN_RES_X2I, + CHAN_RES_I2X, + CHAN_RES_NUM +}; + +struct mailbox { + struct device *dev; + struct xdna_mailbox_res res; +}; + +struct mailbox_channel { + struct mailbox *mb; + struct xdna_mailbox_chann_res res[CHAN_RES_NUM]; + int msix_irq; + u32 iohub_int_addr; + struct idr chan_idr; + spinlock_t chan_idr_lock; /* protect chan_idr */ + u32 x2i_tail; + + /* Received msg related fields */ + struct workqueue_struct *work_q; + struct work_struct rx_work; + u32 i2x_head; + bool bad_state; +}; + +#define MSG_BODY_SZ GENMASK(10, 0) +#define MSG_PROTO_VER GENMASK(23, 16) +struct xdna_msg_header { + __u32 total_size; + __u32 sz_ver; + __u32 id; + __u32 opcode; +} __packed; + +static_assert(sizeof(struct xdna_msg_header) =3D=3D 16); + +struct mailbox_pkg { + struct xdna_msg_header header; + __u32 payload[]; +}; + +/* The protocol version. */ +#define MSG_PROTOCOL_VERSION 0x1 +/* The tombstone value. */ +#define TOMBSTONE 0xDEADFACE + +struct mailbox_msg { + void *handle; + int (*notify_cb)(void *handle, const u32 *data, size_t size); + size_t pkg_size; /* package size in bytes */ + struct mailbox_pkg pkg; +}; + +static void mailbox_reg_write(struct mailbox_channel *mb_chann, u32 mbox_r= eg, u32 data) +{ + struct xdna_mailbox_res *mb_res =3D &mb_chann->mb->res; + u64 ringbuf_addr =3D mb_res->mbox_base + mbox_reg; + + writel(data, (void *)ringbuf_addr); +} + +static u32 mailbox_reg_read(struct mailbox_channel *mb_chann, u32 mbox_reg) +{ + struct xdna_mailbox_res *mb_res =3D &mb_chann->mb->res; + u64 ringbuf_addr =3D mb_res->mbox_base + mbox_reg; + + return readl((void *)ringbuf_addr); +} + +static int mailbox_reg_read_non_zero(struct mailbox_channel *mb_chann, u32= mbox_reg, u32 *val) +{ + struct xdna_mailbox_res *mb_res =3D &mb_chann->mb->res; + u64 ringbuf_addr =3D mb_res->mbox_base + mbox_reg; + int ret, value; + + /* Poll till value is not zero */ + ret =3D readx_poll_timeout(readl, (void *)ringbuf_addr, value, + value, 1 /* us */, 100); + if (ret < 0) + return ret; + + *val =3D value; + return 0; +} + +static inline void +mailbox_set_headptr(struct mailbox_channel *mb_chann, u32 headptr_val) +{ + mailbox_reg_write(mb_chann, mb_chann->res[CHAN_RES_I2X].mb_head_ptr_reg, = headptr_val); + mb_chann->i2x_head =3D headptr_val; +} + +static inline void +mailbox_set_tailptr(struct mailbox_channel *mb_chann, u32 tailptr_val) +{ + mailbox_reg_write(mb_chann, mb_chann->res[CHAN_RES_X2I].mb_tail_ptr_reg, = tailptr_val); + mb_chann->x2i_tail =3D tailptr_val; +} + +static inline u32 +mailbox_get_headptr(struct mailbox_channel *mb_chann, enum channel_res_typ= e type) +{ + return mailbox_reg_read(mb_chann, mb_chann->res[type].mb_head_ptr_reg); +} + +static inline u32 +mailbox_get_tailptr(struct mailbox_channel *mb_chann, enum channel_res_typ= e type) +{ + return mailbox_reg_read(mb_chann, mb_chann->res[type].mb_tail_ptr_reg); +} + +static inline u32 +mailbox_get_ringbuf_size(struct mailbox_channel *mb_chann, enum channel_re= s_type type) +{ + return mb_chann->res[type].rb_size; +} + +static inline int mailbox_validate_msgid(int msg_id) +{ + return (msg_id & MAGIC_VAL_MASK) =3D=3D MAGIC_VAL; +} + +static int mailbox_acquire_msgid(struct mailbox_channel *mb_chann, struct = mailbox_msg *mb_msg) +{ + unsigned long flags; + int msg_id; + + spin_lock_irqsave(&mb_chann->chan_idr_lock, flags); + msg_id =3D idr_alloc_cyclic(&mb_chann->chan_idr, mb_msg, 0, + MAX_MSG_ID_ENTRIES, GFP_NOWAIT); + spin_unlock_irqrestore(&mb_chann->chan_idr_lock, flags); + if (msg_id < 0) + return msg_id; + + /* + * The IDR becomes less efficient when dealing with larger IDs. + * Thus, add MAGIC_VAL to the higher bits. + */ + msg_id |=3D MAGIC_VAL; + return msg_id; +} + +static void mailbox_release_msgid(struct mailbox_channel *mb_chann, int ms= g_id) +{ + unsigned long flags; + + msg_id &=3D ~MAGIC_VAL_MASK; + spin_lock_irqsave(&mb_chann->chan_idr_lock, flags); + idr_remove(&mb_chann->chan_idr, msg_id); + spin_unlock_irqrestore(&mb_chann->chan_idr_lock, flags); +} + +static int mailbox_release_msg(int id, void *p, void *data) +{ + struct mailbox_channel *mb_chann =3D data; + struct mailbox_msg *mb_msg =3D p; + + MB_DBG(mb_chann, "msg_id 0x%x msg opcode 0x%x", + mb_msg->pkg.header.id, mb_msg->pkg.header.opcode); + mb_msg->notify_cb(mb_msg->handle, NULL, 0); + kfree(mb_msg); + + return 0; +} + +static int +mailbox_send_msg(struct mailbox_channel *mb_chann, struct mailbox_msg *mb_= msg) +{ + u32 ringbuf_size; + u32 head, tail; + u32 start_addr; + u64 write_addr; + u32 tmp_tail; + + head =3D mailbox_get_headptr(mb_chann, CHAN_RES_X2I); + tail =3D mb_chann->x2i_tail; + ringbuf_size =3D mailbox_get_ringbuf_size(mb_chann, CHAN_RES_X2I); + start_addr =3D mb_chann->res[CHAN_RES_X2I].rb_start_addr; + tmp_tail =3D tail + mb_msg->pkg_size; + + if (tail < head && tmp_tail >=3D head) + goto no_space; + + if (tail >=3D head && (tmp_tail > ringbuf_size - sizeof(u32) && + mb_msg->pkg_size >=3D head)) + goto no_space; + + if (tail >=3D head && tmp_tail > ringbuf_size - sizeof(u32)) { + write_addr =3D mb_chann->mb->res.ringbuf_base + start_addr + tail; + writel(TOMBSTONE, (void *)write_addr); + + /* tombstone is set. Write from the start of the ringbuf */ + tail =3D 0; + } + + write_addr =3D mb_chann->mb->res.ringbuf_base + start_addr + tail; + memcpy_toio((void *)write_addr, &mb_msg->pkg, mb_msg->pkg_size); + mailbox_set_tailptr(mb_chann, tail + mb_msg->pkg_size); + + trace_mbox_set_tail(MAILBOX_NAME, mb_chann->msix_irq, + mb_msg->pkg.header.opcode, + mb_msg->pkg.header.id); + + return 0; + +no_space: + return -ENOSPC; +} + +static int +mailbox_get_resp(struct mailbox_channel *mb_chann, struct xdna_msg_header = *header, + void *data) +{ + struct mailbox_msg *mb_msg; + unsigned long flags; + int msg_id; + int ret; + + msg_id =3D header->id; + if (!mailbox_validate_msgid(msg_id)) { + MB_ERR(mb_chann, "Bad message ID 0x%x", msg_id); + return -EINVAL; + } + + msg_id &=3D ~MAGIC_VAL_MASK; + spin_lock_irqsave(&mb_chann->chan_idr_lock, flags); + mb_msg =3D idr_find(&mb_chann->chan_idr, msg_id); + if (!mb_msg) { + MB_ERR(mb_chann, "Cannot find msg 0x%x", msg_id); + spin_unlock_irqrestore(&mb_chann->chan_idr_lock, flags); + return -EINVAL; + } + idr_remove(&mb_chann->chan_idr, msg_id); + spin_unlock_irqrestore(&mb_chann->chan_idr_lock, flags); + + MB_DBG(mb_chann, "opcode 0x%x size %d id 0x%x", + header->opcode, header->total_size, header->id); + ret =3D mb_msg->notify_cb(mb_msg->handle, data, header->total_size); + if (unlikely(ret)) + MB_ERR(mb_chann, "Message callback ret %d", ret); + + kfree(mb_msg); + return ret; +} + +static int mailbox_get_msg(struct mailbox_channel *mb_chann) +{ + struct xdna_msg_header header; + u32 msg_size, rest; + u32 ringbuf_size; + u32 head, tail; + u32 start_addr; + u64 read_addr; + int ret; + + if (mailbox_reg_read_non_zero(mb_chann, mb_chann->res[CHAN_RES_I2X].mb_ta= il_ptr_reg, &tail)) + return -EINVAL; + head =3D mb_chann->i2x_head; + ringbuf_size =3D mailbox_get_ringbuf_size(mb_chann, CHAN_RES_I2X); + start_addr =3D mb_chann->res[CHAN_RES_I2X].rb_start_addr; + + if (unlikely(tail > ringbuf_size || !IS_ALIGNED(tail, 4))) { + MB_WARN_ONCE(mb_chann, "Invalid tail 0x%x", tail); + return -EINVAL; + } + + /* ringbuf empty */ + if (head =3D=3D tail) + return -ENOENT; + + if (head =3D=3D ringbuf_size) + head =3D 0; + + /* Peek size of the message or TOMBSTONE */ + read_addr =3D mb_chann->mb->res.ringbuf_base + start_addr + head; + header.total_size =3D readl((void *)read_addr); + /* size is TOMBSTONE, set next read from 0 */ + if (header.total_size =3D=3D TOMBSTONE) { + if (head < tail) { + MB_WARN_ONCE(mb_chann, "Tombstone, head 0x%x tail 0x%x", + head, tail); + return -EINVAL; + } + mailbox_set_headptr(mb_chann, 0); + return 0; + } + + if (unlikely(!header.total_size || !IS_ALIGNED(header.total_size, 4))) { + MB_WARN_ONCE(mb_chann, "Invalid total size 0x%x", header.total_size); + return -EINVAL; + } + msg_size =3D sizeof(header) + header.total_size; + + if (msg_size > ringbuf_size - head || msg_size > tail - head) { + MB_WARN_ONCE(mb_chann, "Invalid message size %d, tail %d, head %d", + msg_size, tail, head); + return -EINVAL; + } + + rest =3D sizeof(header) - sizeof(u32); + read_addr +=3D sizeof(u32); + memcpy_fromio((u32 *)&header + 1, (void *)read_addr, rest); + read_addr +=3D rest; + + ret =3D mailbox_get_resp(mb_chann, &header, (u32 *)read_addr); + + mailbox_set_headptr(mb_chann, head + msg_size); + /* After update head, it can equal to ringbuf_size. This is expected. */ + trace_mbox_set_head(MAILBOX_NAME, mb_chann->msix_irq, + header.opcode, header.id); + + return ret; +} + +static irqreturn_t mailbox_irq_handler(int irq, void *p) +{ + struct mailbox_channel *mb_chann =3D p; + + trace_mbox_irq_handle(MAILBOX_NAME, irq); + /* Schedule a rx_work to call the callback functions */ + queue_work(mb_chann->work_q, &mb_chann->rx_work); + /* Clear IOHUB register */ + mailbox_reg_write(mb_chann, mb_chann->iohub_int_addr, 0); + + return IRQ_HANDLED; +} + +static void mailbox_rx_worker(struct work_struct *rx_work) +{ + struct mailbox_channel *mb_chann; + int ret; + + mb_chann =3D container_of(rx_work, struct mailbox_channel, rx_work); + + if (READ_ONCE(mb_chann->bad_state)) { + MB_ERR(mb_chann, "Channel in bad state, work aborted"); + return; + } + + while (1) { + /* + * If return is 0, keep consuming next message, until there is + * no messages or an error happened. + */ + ret =3D mailbox_get_msg(mb_chann); + if (ret =3D=3D -ENOENT) + break; + + /* Other error means device doesn't look good, disable irq. */ + if (unlikely(ret)) { + MB_ERR(mb_chann, "Unexpected ret %d, disable irq", ret); + WRITE_ONCE(mb_chann->bad_state, true); + disable_irq(mb_chann->msix_irq); + break; + } + } +} + +int xdna_mailbox_send_msg(struct mailbox_channel *mb_chann, + const struct xdna_mailbox_msg *msg, u64 tx_timeout) +{ + struct xdna_msg_header *header; + struct mailbox_msg *mb_msg; + size_t pkg_size; + int ret; + + pkg_size =3D sizeof(*header) + msg->send_size; + if (pkg_size > mailbox_get_ringbuf_size(mb_chann, CHAN_RES_X2I)) { + MB_ERR(mb_chann, "Message size larger than ringbuf size"); + return -EINVAL; + } + + if (unlikely(!IS_ALIGNED(msg->send_size, 4))) { + MB_ERR(mb_chann, "Message must be 4 bytes align"); + return -EINVAL; + } + + /* The fist word in payload can NOT be TOMBSTONE */ + if (unlikely(((u32 *)msg->send_data)[0] =3D=3D TOMBSTONE)) { + MB_ERR(mb_chann, "Tomb stone in data"); + return -EINVAL; + } + + if (READ_ONCE(mb_chann->bad_state)) { + MB_ERR(mb_chann, "Channel in bad state"); + return -EPIPE; + } + + mb_msg =3D kzalloc(sizeof(*mb_msg) + pkg_size, GFP_KERNEL); + if (!mb_msg) + return -ENOMEM; + + mb_msg->handle =3D msg->handle; + mb_msg->notify_cb =3D msg->notify_cb; + mb_msg->pkg_size =3D pkg_size; + + header =3D &mb_msg->pkg.header; + /* + * Hardware use total_size and size to split huge message. + * We do not support it here. Thus the values are the same. + */ + header->total_size =3D msg->send_size; + header->sz_ver =3D FIELD_PREP(MSG_BODY_SZ, msg->send_size) | + FIELD_PREP(MSG_PROTO_VER, MSG_PROTOCOL_VERSION); + header->opcode =3D msg->opcode; + memcpy(mb_msg->pkg.payload, msg->send_data, msg->send_size); + + ret =3D mailbox_acquire_msgid(mb_chann, mb_msg); + if (unlikely(ret < 0)) { + MB_ERR(mb_chann, "mailbox_acquire_msgid failed"); + goto msg_id_failed; + } + header->id =3D ret; + + MB_DBG(mb_chann, "opcode 0x%x size %d id 0x%x", + header->opcode, header->total_size, header->id); + + ret =3D mailbox_send_msg(mb_chann, mb_msg); + if (ret) { + MB_DBG(mb_chann, "Error in mailbox send msg, ret %d", ret); + goto release_id; + } + + return 0; + +release_id: + mailbox_release_msgid(mb_chann, header->id); +msg_id_failed: + kfree(mb_msg); + return ret; +} + +struct mailbox_channel * +xdna_mailbox_create_channel(struct mailbox *mb, + const struct xdna_mailbox_chann_res *x2i, + const struct xdna_mailbox_chann_res *i2x, + u32 iohub_int_addr, + int mb_irq) +{ + struct mailbox_channel *mb_chann; + int ret; + + if (!is_power_of_2(x2i->rb_size) || !is_power_of_2(i2x->rb_size)) { + pr_err("Ring buf size must be power of 2"); + return NULL; + } + + mb_chann =3D kzalloc(sizeof(*mb_chann), GFP_KERNEL); + if (!mb_chann) + return NULL; + + mb_chann->mb =3D mb; + mb_chann->msix_irq =3D mb_irq; + mb_chann->iohub_int_addr =3D iohub_int_addr; + memcpy(&mb_chann->res[CHAN_RES_X2I], x2i, sizeof(*x2i)); + memcpy(&mb_chann->res[CHAN_RES_I2X], i2x, sizeof(*i2x)); + + spin_lock_init(&mb_chann->chan_idr_lock); + idr_init(&mb_chann->chan_idr); + mb_chann->x2i_tail =3D mailbox_get_tailptr(mb_chann, CHAN_RES_X2I); + mb_chann->i2x_head =3D mailbox_get_headptr(mb_chann, CHAN_RES_I2X); + + INIT_WORK(&mb_chann->rx_work, mailbox_rx_worker); + mb_chann->work_q =3D create_singlethread_workqueue(MAILBOX_NAME); + if (!mb_chann->work_q) { + MB_ERR(mb_chann, "Create workqueue failed"); + goto free_and_out; + } + + /* Everything look good. Time to enable irq handler */ + ret =3D request_irq(mb_irq, mailbox_irq_handler, 0, MAILBOX_NAME, mb_chan= n); + if (ret) { + MB_ERR(mb_chann, "Failed to request irq %d ret %d", mb_irq, ret); + goto destroy_wq; + } + + mb_chann->bad_state =3D false; + + MB_DBG(mb_chann, "Mailbox channel created (irq: %d)", mb_chann->msix_irq); + return mb_chann; + +destroy_wq: + destroy_workqueue(mb_chann->work_q); +free_and_out: + kfree(mb_chann); + return NULL; +} + +int xdna_mailbox_destroy_channel(struct mailbox_channel *mb_chann) +{ + if (!mb_chann) + return 0; + + MB_DBG(mb_chann, "IRQ disabled and RX work cancelled"); + free_irq(mb_chann->msix_irq, mb_chann); + destroy_workqueue(mb_chann->work_q); + /* We can clean up and release resources */ + + idr_for_each(&mb_chann->chan_idr, mailbox_release_msg, mb_chann); + idr_destroy(&mb_chann->chan_idr); + + MB_DBG(mb_chann, "Mailbox channel destroyed, irq: %d", mb_chann->msix_irq= ); + kfree(mb_chann); + return 0; +} + +void xdna_mailbox_stop_channel(struct mailbox_channel *mb_chann) +{ + if (!mb_chann) + return; + + /* Disable an irq and wait. This might sleep. */ + disable_irq(mb_chann->msix_irq); + + /* Cancel RX work and wait for it to finish */ + cancel_work_sync(&mb_chann->rx_work); + MB_DBG(mb_chann, "IRQ disabled and RX work cancelled"); +} + +struct mailbox *xdnam_mailbox_create(struct drm_device *ddev, + const struct xdna_mailbox_res *res) +{ + struct mailbox *mb; + + mb =3D drmm_kzalloc(ddev, sizeof(*mb), GFP_KERNEL); + if (!mb) + return NULL; + mb->dev =3D ddev->dev; + + /* mailbox and ring buf base and size information */ + memcpy(&mb->res, res, sizeof(*res)); + + return mb; +} diff --git a/drivers/accel/amdxdna/amdxdna_mailbox.h b/drivers/accel/amdxdn= a/amdxdna_mailbox.h new file mode 100644 index 000000000000..6ab7f5424633 --- /dev/null +++ b/drivers/accel/amdxdna/amdxdna_mailbox.h @@ -0,0 +1,124 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* + * Copyright (C) 2022-2024, Advanced Micro Devices, Inc. + */ + +#ifndef _AIE2_MAILBOX_H_ +#define _AIE2_MAILBOX_H_ + +struct mailbox; +struct mailbox_channel; + +/* + * xdna_mailbox_msg - message struct + * + * @opcode: opcode for firmware + * @handle: handle used for the notify callback + * @notify_cb: callback function to notify the sender when there is respo= nse + * @send_data: pointing to sending data + * @send_size: size of the sending data + * + * The mailbox will split the sending data in to multiple firmware message= if + * the size of the data is too big. This is transparent to the sender. The + * sender will receive one notification. + */ +struct xdna_mailbox_msg { + u32 opcode; + void *handle; + int (*notify_cb)(void *handle, const u32 *data, size_t size); + u8 *send_data; + size_t send_size; +}; + +/* + * xdna_mailbox_res - mailbox hardware resource + * + * @ringbuf_base: ring buffer base address + * @ringbuf_size: ring buffer size + * @mbox_base: mailbox base address + * @mbox_size: mailbox size + */ +struct xdna_mailbox_res { + u64 ringbuf_base; + size_t ringbuf_size; + u64 mbox_base; + size_t mbox_size; + const char *name; +}; + +/* + * xdna_mailbox_chann_res - resources + * + * @rb_start_addr: ring buffer start address + * @rb_size: ring buffer size + * @mb_head_ptr_reg: mailbox head pointer register + * @mb_tail_ptr_reg: mailbox tail pointer register + */ +struct xdna_mailbox_chann_res { + u32 rb_start_addr; + u32 rb_size; + u32 mb_head_ptr_reg; + u32 mb_tail_ptr_reg; +}; + +/* + * xdna_mailbox_create() -- create mailbox subsystem and initialize + * + * @ddev: device pointer + * @res: SRAM and mailbox resources + * + * Return: If success, return a handle of mailbox subsystem. + * Otherwise, return NULL pointer. + */ +struct mailbox *xdnam_mailbox_create(struct drm_device *ddev, + const struct xdna_mailbox_res *res); + +/* + * xdna_mailbox_create_channel() -- Create a mailbox channel instance + * + * @mailbox: the handle return from xdna_mailbox_create() + * @x2i: host to firmware mailbox resources + * @i2x: firmware to host mailbox resources + * @xdna_mailbox_intr_reg: register addr of MSI-X interrupt + * @mb_irq: Linux IRQ number associated with mailbox MSI-X interrupt vecto= r index + * + * Return: If success, return a handle of mailbox channel. Otherwise, retu= rn NULL. + */ +struct mailbox_channel * +xdna_mailbox_create_channel(struct mailbox *mailbox, + const struct xdna_mailbox_chann_res *x2i, + const struct xdna_mailbox_chann_res *i2x, + u32 xdna_mailbox_intr_reg, + int mb_irq); + +/* + * xdna_mailbox_destroy_channel() -- destroy mailbox channel + * + * @mailbox_chann: the handle return from xdna_mailbox_create_channel() + * + * Return: if success, return 0. otherwise return error code + */ +int xdna_mailbox_destroy_channel(struct mailbox_channel *mailbox_chann); + +/* + * xdna_mailbox_stop_channel() -- stop mailbox channel + * + * @mailbox_chann: the handle return from xdna_mailbox_create_channel() + * + * Return: if success, return 0. otherwise return error code + */ +void xdna_mailbox_stop_channel(struct mailbox_channel *mailbox_chann); + +/* + * xdna_mailbox_send_msg() -- Send a message + * + * @mailbox_chann: Mailbox channel handle + * @msg: message struct for message information + * @tx_timeout: the timeout value for sending the message in ms. + * + * Return: If success return 0, otherwise, return error code + */ +int xdna_mailbox_send_msg(struct mailbox_channel *mailbox_chann, + const struct xdna_mailbox_msg *msg, u64 tx_timeout); + +#endif /* _AIE2_MAILBOX_ */ diff --git a/drivers/accel/amdxdna/amdxdna_mailbox_helper.c b/drivers/accel= /amdxdna/amdxdna_mailbox_helper.c new file mode 100644 index 000000000000..42b615394605 --- /dev/null +++ b/drivers/accel/amdxdna/amdxdna_mailbox_helper.c @@ -0,0 +1,56 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * Copyright (C) 2024, Advanced Micro Devices, Inc. + */ + +#include +#include +#include + +#include "amdxdna_mailbox.h" +#include "amdxdna_mailbox_helper.h" +#include "amdxdna_pci_drv.h" + +int xdna_msg_cb(void *handle, const u32 *data, size_t size) +{ + struct xdna_notify *cb_arg =3D handle; + int ret; + + if (unlikely(!data)) + goto out; + + if (unlikely(cb_arg->size !=3D size)) { + cb_arg->error =3D -EINVAL; + goto out; + } + + print_hex_dump_debug("resp data: ", DUMP_PREFIX_OFFSET, + 16, 4, data, cb_arg->size, true); + memcpy(cb_arg->data, data, cb_arg->size); +out: + ret =3D cb_arg->error; + complete(&cb_arg->comp); + return ret; +} + +int xdna_send_msg_wait(struct amdxdna_dev *xdna, struct mailbox_channel *c= hann, + struct xdna_mailbox_msg *msg) +{ + struct xdna_notify *hdl =3D msg->handle; + int ret; + + ret =3D xdna_mailbox_send_msg(chann, msg, TX_TIMEOUT); + if (ret) { + XDNA_ERR(xdna, "Send message failed, ret %d", ret); + return ret; + } + + ret =3D wait_for_completion_timeout(&hdl->comp, + msecs_to_jiffies(RX_TIMEOUT)); + if (!ret) { + XDNA_ERR(xdna, "Wait for completion timeout"); + return -ETIME; + } + + return hdl->error; +} diff --git a/drivers/accel/amdxdna/amdxdna_mailbox_helper.h b/drivers/accel= /amdxdna/amdxdna_mailbox_helper.h new file mode 100644 index 000000000000..23e1317b79fe --- /dev/null +++ b/drivers/accel/amdxdna/amdxdna_mailbox_helper.h @@ -0,0 +1,42 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* + * Copyright (C) 2023-2024, Advanced Micro Devices, Inc. + */ + +#ifndef _AMDXDNA_MAILBOX_HELPER_H +#define _AMDXDNA_MAILBOX_HELPER_H + +#define TX_TIMEOUT 2000 /* milliseconds */ +#define RX_TIMEOUT 5000 /* milliseconds */ + +struct amdxdna_dev; + +struct xdna_notify { + struct completion comp; + u32 *data; + size_t size; + int error; +}; + +#define DECLARE_XDNA_MSG_COMMON(name, op, status) \ + struct name##_req req =3D { 0 }; \ + struct name##_resp resp =3D { status }; \ + struct xdna_notify hdl =3D { \ + .error =3D 0, \ + .data =3D (u32 *)&resp, \ + .size =3D sizeof(resp), \ + .comp =3D COMPLETION_INITIALIZER_ONSTACK(hdl.comp), \ + }; \ + struct xdna_mailbox_msg msg =3D { \ + .send_data =3D (u8 *)&req, \ + .send_size =3D sizeof(req), \ + .handle =3D &hdl, \ + .opcode =3D op, \ + .notify_cb =3D xdna_msg_cb, \ + } + +int xdna_msg_cb(void *handle, const u32 *data, size_t size); +int xdna_send_msg_wait(struct amdxdna_dev *xdna, struct mailbox_channel *c= hann, + struct xdna_mailbox_msg *msg); + +#endif /* _AMDXDNA_MAILBOX_HELPER_H */ diff --git a/drivers/accel/amdxdna/amdxdna_pci_drv.h b/drivers/accel/amdxdn= a/amdxdna_pci_drv.h index 2f1a1c2441f9..64bce970514b 100644 --- a/drivers/accel/amdxdna/amdxdna_pci_drv.h +++ b/drivers/accel/amdxdna/amdxdna_pci_drv.h @@ -47,12 +47,20 @@ struct amdxdna_dev_info { const struct amdxdna_dev_ops *ops; }; =20 +struct amdxdna_fw_ver { + u32 major; + u32 minor; + u32 sub; + u32 build; +}; + struct amdxdna_dev { struct drm_device ddev; struct amdxdna_dev_hdl *dev_handle; const struct amdxdna_dev_info *dev_info; =20 struct mutex dev_lock; /* per device lock */ + struct amdxdna_fw_ver fw_ver; }; =20 /* diff --git a/drivers/accel/amdxdna/amdxdna_sysfs.c b/drivers/accel/amdxdna/= amdxdna_sysfs.c index 5dd652fcf9d4..668b94b92714 100644 --- a/drivers/accel/amdxdna/amdxdna_sysfs.c +++ b/drivers/accel/amdxdna/amdxdna_sysfs.c @@ -24,9 +24,20 @@ static ssize_t device_type_show(struct device *dev, stru= ct device_attribute *att } static DEVICE_ATTR_RO(device_type); =20 +static ssize_t fw_version_show(struct device *dev, struct device_attribute= *attr, char *buf) +{ + struct amdxdna_dev *xdna =3D dev_get_drvdata(dev); + + return sprintf(buf, "%d.%d.%d.%d\n", xdna->fw_ver.major, + xdna->fw_ver.minor, xdna->fw_ver.sub, + xdna->fw_ver.build); +} +static DEVICE_ATTR_RO(fw_version); + static struct attribute *amdxdna_attrs[] =3D { &dev_attr_device_type.attr, &dev_attr_vbnv.attr, + &dev_attr_fw_version.attr, NULL, }; =20 diff --git a/drivers/accel/amdxdna/npu1_regs.c b/drivers/accel/amdxdna/npu1= _regs.c index 858b31a82888..720aab0ed7c4 100644 --- a/drivers/accel/amdxdna/npu1_regs.c +++ b/drivers/accel/amdxdna/npu1_regs.c @@ -8,6 +8,7 @@ #include =20 #include "aie2_pci.h" +#include "amdxdna_mailbox.h" #include "amdxdna_pci_drv.h" =20 /* Address definition from NPU1 docs */ diff --git a/drivers/accel/amdxdna/npu2_regs.c b/drivers/accel/amdxdna/npu2= _regs.c index 02b0f22c9f14..f3ea18bcf294 100644 --- a/drivers/accel/amdxdna/npu2_regs.c +++ b/drivers/accel/amdxdna/npu2_regs.c @@ -8,6 +8,7 @@ #include =20 #include "aie2_pci.h" +#include "amdxdna_mailbox.h" #include "amdxdna_pci_drv.h" =20 /* NPU Public Registers on MpNPUAxiXbar (refer to Diag npu_registers.h) */ diff --git a/drivers/accel/amdxdna/npu4_regs.c b/drivers/accel/amdxdna/npu4= _regs.c index ca5ca5a6c751..db61142f0d4e 100644 --- a/drivers/accel/amdxdna/npu4_regs.c +++ b/drivers/accel/amdxdna/npu4_regs.c @@ -8,6 +8,7 @@ #include =20 #include "aie2_pci.h" +#include "amdxdna_mailbox.h" #include "amdxdna_pci_drv.h" =20 /* NPU Public Registers on MpNPUAxiXbar (refer to Diag npu_registers.h) */ diff --git a/drivers/accel/amdxdna/npu5_regs.c b/drivers/accel/amdxdna/npu5= _regs.c index 07fddcbb86ec..debf4e95b9bb 100644 --- a/drivers/accel/amdxdna/npu5_regs.c +++ b/drivers/accel/amdxdna/npu5_regs.c @@ -8,6 +8,7 @@ #include =20 #include "aie2_pci.h" +#include "amdxdna_mailbox.h" #include "amdxdna_pci_drv.h" =20 /* NPU Public Registers on MpNPUAxiXbar (refer to Diag npu_registers.h) */ diff --git a/include/trace/events/amdxdna.h b/include/trace/events/amdxdna.h new file mode 100644 index 000000000000..33343d8f0622 --- /dev/null +++ b/include/trace/events/amdxdna.h @@ -0,0 +1,60 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* + * Copyright (C) 2023-2024, Advanced Micro Devices, Inc. + */ + +#undef TRACE_SYSTEM +#define TRACE_SYSTEM amdxdna + +#if !defined(_TRACE_AMDXDNA_H) || defined(TRACE_HEADER_MULTI_READ) +#define _TRACE_AMDXDNA_H + +#include + +DECLARE_EVENT_CLASS(xdna_mbox_msg, + TP_PROTO(char *name, u8 chann_id, u32 opcode, u32 msg_id), + + TP_ARGS(name, chann_id, opcode, msg_id), + + TP_STRUCT__entry(__string(name, name) + __field(u32, chann_id) + __field(u32, opcode) + __field(u32, msg_id)), + + TP_fast_assign(__assign_str(name); + __entry->chann_id =3D chann_id; + __entry->opcode =3D opcode; + __entry->msg_id =3D msg_id;), + + TP_printk("%s.%d id 0x%x opcode 0x%x", __get_str(name), + __entry->chann_id, __entry->msg_id, __entry->opcode) +); + +DEFINE_EVENT(xdna_mbox_msg, mbox_set_tail, + TP_PROTO(char *name, u8 chann_id, u32 opcode, u32 id), + TP_ARGS(name, chann_id, opcode, id) +); + +DEFINE_EVENT(xdna_mbox_msg, mbox_set_head, + TP_PROTO(char *name, u8 chann_id, u32 opcode, u32 id), + TP_ARGS(name, chann_id, opcode, id) +); + +TRACE_EVENT(mbox_irq_handle, + TP_PROTO(char *name, int irq), + + TP_ARGS(name, irq), + + TP_STRUCT__entry(__string(name, name) + __field(int, irq)), + + TP_fast_assign(__assign_str(name); + __entry->irq =3D irq;), + + TP_printk("%s.%d", __get_str(name), __entry->irq) +); + +#endif /* !defined(_TRACE_AMDXDNA_H) || defined(TRACE_HEADER_MULTI_READ) */ + +/* This part must be outside protection */ +#include --=20 2.34.1