From nobody Mon Feb 9 04:14:33 2026 Received: from BL0PR03CU003.outbound.protection.outlook.com (mail-eastusazon11012009.outbound.protection.outlook.com [52.101.53.9]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 2DEA72FA0DF for ; Thu, 13 Nov 2025 16:12:05 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=52.101.53.9 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1763050328; cv=fail; b=dBWnLHQIvxTa3r7s9t+0YBKJg3Wc+PfpuLVTAly3ARYQxB0eNXwRmPMs0N1TNycRPqS0Ir/FiCefP3DieB56pvFo2Y18IdaJDCYBxRCsQkV5uX4oo0UEmPgxctFlGf43Kao4JK7O7Mqc7iCTSMgEqInKouBwlUjbaudRqJfkalU= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1763050328; c=relaxed/simple; bh=SeSPQrVDTvPyXbCEEel4kkrq3Hg+8sZvJHU6b8pNbEw=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=iN5HlG2QZhPgFGTA/GGTesTv7z9dAtABXCvKpR3DntmA5Yfo9tamr4HuZNM2jQ13Zeh5s7Bpu36F0x1TgwCjtHDfDFnnatfb2uYkaF8MMop2kaXdGMLtwb3gYLfnCwN9FeEAZD3/TQVcVnRv9jwe470uc5PPkRFz190XIVVHTlo= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=C1llwbQE; arc=fail smtp.client-ip=52.101.53.9 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="C1llwbQE" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=eTkzNId76t5T908Ne+VKPbDbJSJXfzmDALuunBxG6VGXr3DeoA0TzIxMBdqRcy/SS06kDsuJ+cLyztH1t3/QgqKMVwNxpLgnBvwkpVt6hdVhni3bqvWW5to4LFfbFpb5MAHTVhvK3DKETdSpr1FnNFl/Ajsc6yzvFR2pZT27Y+fiygMV1PkyPxfl6uuf57bQAJ6FGfZ0vMQBepx88JOqBOMq1C5Qpd31MYh7XMZBVT55PkMY9mKVIrCijk8JYd92TJawsIm5J4WGQbRozrwZBbXYEVOSJOWL8qFd8oR1VCEODky48BU+IEhLrsxVD2F1wIv+Q2wSpDQNRwXrPKTshg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=fFZ2eaw//3WCmkB7SjUIMEagBrNAQ4dospOtvl02RhY=; b=jtMEzDc3ItzdoIk+7y0BFf16pHi5rxPit6tHzazFgbjKEt/bzWTo9VPHS0Afs/t60QLeVr0VZ/bY/A03RsE/AGr9CuWxaWsY4ZxzO7r34TWRnLW0TNtb2PUxMZekABoxw9yJIFgJb1HZgVZEPeCH1EYrzTHK+6VnO7Td/nO69CNm8kr/TfVnRfWav1D6zgh66MxQKrbozy3wd9S37DmnUxCOn3OWLwlmaksKXDdxrZ0mQe++wRtbfSV4piIlIuFitZtJVqM+SMiUZAXLiKBLX7laVB78WX1XM6Q0d0RqRtyL+mOBom6LJsJ4GX8z5IsmA38lxe7EJNu2IgJP2TZ+SA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=gmail.com smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=fFZ2eaw//3WCmkB7SjUIMEagBrNAQ4dospOtvl02RhY=; b=C1llwbQEEoRBGk2xQZbTf3TknRid5JkNafTidkQe/enaDxrJ9JCtJLR8U7LT9Gslbts/XU5TJk6Lbzibs1jDP823WbNbc03CZB/4O5u9cycTGQYqRAK70Ud3io80yvG3mo4Dn/n5cTMIKE0Rso3fEX4kTNPg9ln/LKOvqCTrzok= Received: from MN2PR07CA0021.namprd07.prod.outlook.com (2603:10b6:208:1a0::31) by LV8PR12MB9359.namprd12.prod.outlook.com (2603:10b6:408:1fe::10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9320.17; Thu, 13 Nov 2025 16:12:02 +0000 Received: from BL02EPF0002992E.namprd02.prod.outlook.com (2603:10b6:208:1a0:cafe::11) by MN2PR07CA0021.outlook.office365.com (2603:10b6:208:1a0::31) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.9320.16 via Frontend Transport; Thu, 13 Nov 2025 16:11:58 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=satlexmb07.amd.com; pr=C Received: from satlexmb07.amd.com (165.204.84.17) by BL02EPF0002992E.mail.protection.outlook.com (10.167.249.59) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9320.13 via Frontend Transport; Thu, 13 Nov 2025 16:12:01 +0000 Received: from FRAPPELLOUX01-WSLPUB.amd.com (10.180.168.240) by satlexmb07.amd.com (10.181.42.216) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.17; Thu, 13 Nov 2025 08:11:55 -0800 From: Pierre-Eric Pelloux-Prayer To: Alex Deucher , =?UTF-8?q?Christian=20K=C3=B6nig?= , David Airlie , Simona Vetter , Felix Kuehling CC: Pierre-Eric Pelloux-Prayer , , , Subject: [PATCH v2 15/20] drm/amdgpu: pass all the sdma scheds to amdgpu_mman Date: Thu, 13 Nov 2025 17:05:56 +0100 Message-ID: <20251113160632.5889-16-pierre-eric.pelloux-prayer@amd.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20251113160632.5889-1-pierre-eric.pelloux-prayer@amd.com> References: <20251113160632.5889-1-pierre-eric.pelloux-prayer@amd.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: satlexmb07.amd.com (10.181.42.216) To satlexmb07.amd.com (10.181.42.216) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BL02EPF0002992E:EE_|LV8PR12MB9359:EE_ X-MS-Office365-Filtering-Correlation-Id: c3a5ca59-50fe-4746-0165-08de22cf60f7 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|1800799024|36860700013|82310400026|376014; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?vj4gtnBVfg9KYNtwNh80Fa9Pyzj5oYuNYyZEkJiRrUHWEQb/O4lx1ookZlUQ?= =?us-ascii?Q?zsB4wlFEkWxF3mNw2bingAr0aoVP0SCzI9+du/nfU1c8TcqyfJP/GlpWZSOV?= =?us-ascii?Q?1H7/StZm3haCuy6V+4PMrWHOUSAvOso2wk8Fvc0UA7gJm6O1TSfLCxij+hSC?= =?us-ascii?Q?nmer0bWlHP4cAT5vj3B2HAFc07XGLWI6TA/mI6WQ4TplQKKFH2pn4GYq2aQS?= =?us-ascii?Q?35nSSx0S7pW5+OketYwe0lZ5/ZBF6Q6JIloOezCMQ7tExa6ISL9oQKS7anEq?= =?us-ascii?Q?G3gFASXxAO8TDhAqdqc/MdduXmzBojshe1rFQ20YYNhIKcqZ4HeDjHGRLplb?= =?us-ascii?Q?gt2NWPobOqw4J3ixrRfso3I4C5owdj5/MyLPj8UWUuBCcia8ZbwFPwWIuS6Q?= =?us-ascii?Q?Lk7B/gTyi8PgpRmYefk1ZhHQGIEg6AjIXdVABkbSmTcsaHHg0z/uk+OU0BBB?= =?us-ascii?Q?Yo8j7bMtqoCmaNjhpriZHOiw/4VbF872uM/UR7eC07yEHZCm/Lp5vbUBEbni?= =?us-ascii?Q?6fwSWchzna+W2ygTWr81H+A+QxCuwFwiyg3cb4HOy/71tQl7qI3Bkt5fOnK4?= =?us-ascii?Q?AnGQLhiEomVajbBxh3f5fGO3o380udVr7RwcdecS9g5luEBJo8Cb1DauNDU0?= =?us-ascii?Q?CEog0sowTdCBRZV9tge11YsgJXrJO0g1FwG2MfmZFfvXCQhuWC5sSZ4qjHmw?= =?us-ascii?Q?hXmLtsBMOzN7+k1jF/NqXe1SeIbkPYwn2L+GeEyrLTTGi+WnZ2mr8wY+TxKY?= =?us-ascii?Q?+4cbe/4f4wR9ysDU3BI3XJOtJPDsi3y4S5l6Tbdl9N1ea2Exb6b1R5GgCDYw?= =?us-ascii?Q?iQsM+5FIQ+Fqaaw8BJIFgSdqfhtuMl4zw3vrHlBk/CobOMDsGLxo5fJjaYSm?= =?us-ascii?Q?0esKBeHp/KydLHWsv8n857U+4U5ArRFl8te/Iz15wkPPNA1cA2K0l/lQgVd2?= =?us-ascii?Q?QolNuFCavqnrUDAATM1n5guQNd5zSMGsb95fzMR5v01ccyempyZjxtjBAyIH?= =?us-ascii?Q?LX11XJGqO4QLXLq4sFihzPbXYXCOAM6ECH2XXn3+9clG4nNba4sMMuYWBOBi?= =?us-ascii?Q?tZIF46Yzil623/q63pvf/qln1aYiXIZlizQlGY3d2f1jwn3TUy/cn2CB9RjO?= =?us-ascii?Q?nJuvUuPutdNukfxXXwuD5ySyot4vGUVbxipnGKupCzXqSj3fEc5YxbcKqbVt?= =?us-ascii?Q?Kp8t5erBfyFqaDcT5tXnVk5X70GZf7ERNtoc8RLCN/Rxf/UFCHiwLFH4zzor?= =?us-ascii?Q?tP1FTLPreHnokX4N6m9ucxIzL1rbKm0r6t1GSdoBjQtwK6PdaMYsPXCu1R4K?= =?us-ascii?Q?I4htcaATEcB9JSZ9mJcENOBRBFUm0JCeUPfMsgACTGykM7W62KjZklRT3S8f?= =?us-ascii?Q?ZcaKIa1sNC69WXcyvqzgpGHNIhqrOReSLgobE20EwBYPr6HzFf3wENTQxNRp?= =?us-ascii?Q?RUSJ8C/GjBfC3Iw982tXIFL39AqZNmBn/0Grzl6htCJKq5X2n5wcjm+K/Qr1?= =?us-ascii?Q?l/C950rkCKTNnry4PNH9DUxUDMVRZVUM0Vfx31crf2RLWTcN/uEBvZ8Z7Ggn?= =?us-ascii?Q?ES+ddJW+5oi1TnwFjMY=3D?= X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:satlexmb07.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(1800799024)(36860700013)(82310400026)(376014);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 13 Nov 2025 16:12:01.3121 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: c3a5ca59-50fe-4746-0165-08de22cf60f7 X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[satlexmb07.amd.com] X-MS-Exchange-CrossTenant-AuthSource: BL02EPF0002992E.namprd02.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: LV8PR12MB9359 Content-Type: text/plain; charset="utf-8" This will allow the use of all of them for clear/fill buffer operations. Since drm_sched_entity_init requires a scheduler array, we store schedulers rather than rings. For the few places that need access to a ring, we can get it from the sched using container_of. Since the code is the same for all sdma versions, add a new helper amdgpu_sdma_set_buffer_funcs_scheds to set buffer_funcs_scheds based on the number of sdma instances. Note: the new sched array is identical to the amdgpu_vm_manager one. These 2 could be merged. Signed-off-by: Pierre-Eric Pelloux-Prayer Acked-by: Felix Kuehling --- drivers/gpu/drm/amd/amdgpu/amdgpu.h | 2 + drivers/gpu/drm/amd/amdgpu/amdgpu_benchmark.c | 4 +- drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 8 ++-- drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.c | 4 +- drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c | 41 +++++++++++++++---- drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.h | 3 +- drivers/gpu/drm/amd/amdgpu/cik_sdma.c | 3 +- drivers/gpu/drm/amd/amdgpu/sdma_v2_4.c | 3 +- drivers/gpu/drm/amd/amdgpu/sdma_v3_0.c | 3 +- drivers/gpu/drm/amd/amdgpu/sdma_v4_0.c | 6 +-- drivers/gpu/drm/amd/amdgpu/sdma_v4_4_2.c | 6 +-- drivers/gpu/drm/amd/amdgpu/sdma_v5_0.c | 6 +-- drivers/gpu/drm/amd/amdgpu/sdma_v5_2.c | 6 +-- drivers/gpu/drm/amd/amdgpu/sdma_v6_0.c | 3 +- drivers/gpu/drm/amd/amdgpu/sdma_v7_0.c | 3 +- drivers/gpu/drm/amd/amdgpu/si_dma.c | 3 +- drivers/gpu/drm/amd/amdkfd/kfd_migrate.c | 3 +- 17 files changed, 62 insertions(+), 45 deletions(-) diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu.h b/drivers/gpu/drm/amd/amdg= pu/amdgpu.h index 3fab3dc9f3e4..05c13fb0e6bf 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu.h +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu.h @@ -1615,6 +1615,8 @@ ssize_t amdgpu_get_soft_full_reset_mask(struct amdgpu= _ring *ring); ssize_t amdgpu_show_reset_mask(char *buf, uint32_t supported_reset); void amdgpu_sdma_set_vm_pte_scheds(struct amdgpu_device *adev, const struct amdgpu_vm_pte_funcs *vm_pte_funcs); +void amdgpu_sdma_set_buffer_funcs_scheds(struct amdgpu_device *adev, + const struct amdgpu_buffer_funcs *buffer_funcs); =20 /* atpx handler */ #if defined(CONFIG_VGA_SWITCHEROO) diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_benchmark.c b/drivers/gpu/dr= m/amd/amdgpu/amdgpu_benchmark.c index b59040a8771f..9ea927e07a77 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_benchmark.c +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_benchmark.c @@ -32,12 +32,14 @@ static int amdgpu_benchmark_do_move(struct amdgpu_devic= e *adev, unsigned size, uint64_t saddr, uint64_t daddr, int n, s64 *time_ms) { ktime_t stime, etime; + struct amdgpu_ring *ring; struct dma_fence *fence; int i, r; =20 + ring =3D container_of(adev->mman.buffer_funcs_scheds[0], struct amdgpu_ri= ng, sched); + stime =3D ktime_get(); for (i =3D 0; i < n; i++) { - struct amdgpu_ring *ring =3D adev->mman.buffer_funcs_ring; r =3D amdgpu_copy_buffer(ring, &adev->mman.default_entity.base, saddr, daddr, size, NULL, &fence, false, 0); diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c b/drivers/gpu/drm/a= md/amdgpu/amdgpu_device.c index b92234d63562..1927d940fbca 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c @@ -3303,7 +3303,7 @@ static int amdgpu_device_ip_init(struct amdgpu_device= *adev) if (r) goto init_failed; =20 - if (adev->mman.buffer_funcs_ring->sched.ready) + if (adev->mman.buffer_funcs_scheds[0]->ready) amdgpu_ttm_set_buffer_funcs_status(adev, true); =20 /* Don't init kfd if whole hive need to be reset during init */ @@ -4143,7 +4143,7 @@ static int amdgpu_device_ip_resume(struct amdgpu_devi= ce *adev) =20 r =3D amdgpu_device_ip_resume_phase2(adev); =20 - if (adev->mman.buffer_funcs_ring->sched.ready) + if (adev->mman.buffer_funcs_scheds[0]->ready) amdgpu_ttm_set_buffer_funcs_status(adev, true); =20 if (r) @@ -4493,7 +4493,7 @@ int amdgpu_device_init(struct amdgpu_device *adev, adev->num_rings =3D 0; RCU_INIT_POINTER(adev->gang_submit, dma_fence_get_stub()); adev->mman.buffer_funcs =3D NULL; - adev->mman.buffer_funcs_ring =3D NULL; + adev->mman.num_buffer_funcs_scheds =3D 0; adev->vm_manager.vm_pte_funcs =3D NULL; adev->vm_manager.vm_pte_num_scheds =3D 0; adev->gmc.gmc_funcs =3D NULL; @@ -5965,7 +5965,7 @@ int amdgpu_device_reinit_after_reset(struct amdgpu_re= set_context *reset_context) if (r) goto out; =20 - if (tmp_adev->mman.buffer_funcs_ring->sched.ready) + if (tmp_adev->mman.buffer_funcs_scheds[0]->ready) amdgpu_ttm_set_buffer_funcs_status(tmp_adev, true); =20 r =3D amdgpu_device_ip_resume_phase3(tmp_adev); diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.c b/drivers/gpu/drm/amd/= amdgpu/amdgpu_gmc.c index 2713dd51ab9a..4433d8620129 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.c +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.c @@ -651,12 +651,14 @@ int amdgpu_gmc_allocate_vm_inv_eng(struct amdgpu_devi= ce *adev) void amdgpu_gmc_flush_gpu_tlb(struct amdgpu_device *adev, uint32_t vmid, uint32_t vmhub, uint32_t flush_type) { - struct amdgpu_ring *ring =3D adev->mman.buffer_funcs_ring; + struct amdgpu_ring *ring; struct amdgpu_vmhub *hub =3D &adev->vmhub[vmhub]; struct dma_fence *fence; struct amdgpu_job *job; int r, i; =20 + ring =3D container_of(adev->mman.buffer_funcs_scheds[0], struct amdgpu_ri= ng, sched); + if (!hub->sdma_invalidation_workaround || vmid || !adev->mman.buffer_funcs_enabled || !adev->ib_pool_ready || !ring->sched.ready) { diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c b/drivers/gpu/drm/amd/= amdgpu/amdgpu_ttm.c index 6c333dba7a35..11fec0fa4c11 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c @@ -308,7 +308,7 @@ static int amdgpu_ttm_copy_mem_to_mem(struct amdgpu_dev= ice *adev, struct dma_resv *resv, struct dma_fence **f) { - struct amdgpu_ring *ring =3D adev->mman.buffer_funcs_ring; + struct amdgpu_ring *ring; struct amdgpu_res_cursor src_mm, dst_mm; struct dma_fence *fence =3D NULL; int r =3D 0; @@ -321,6 +321,8 @@ static int amdgpu_ttm_copy_mem_to_mem(struct amdgpu_dev= ice *adev, return -EINVAL; } =20 + ring =3D container_of(adev->mman.buffer_funcs_scheds[0], struct amdgpu_ri= ng, sched); + amdgpu_res_first(src->mem, src->offset, size, &src_mm); amdgpu_res_first(dst->mem, dst->offset, size, &dst_mm); =20 @@ -1493,6 +1495,7 @@ static int amdgpu_ttm_access_memory_sdma(struct ttm_b= uffer_object *bo, struct amdgpu_bo *abo =3D ttm_to_amdgpu_bo(bo); struct amdgpu_device *adev =3D amdgpu_ttm_adev(abo->tbo.bdev); struct amdgpu_res_cursor src_mm; + struct amdgpu_ring *ring; struct amdgpu_job *job; struct dma_fence *fence; uint64_t src_addr, dst_addr; @@ -1530,7 +1533,8 @@ static int amdgpu_ttm_access_memory_sdma(struct ttm_b= uffer_object *bo, amdgpu_emit_copy_buffer(adev, &job->ibs[0], src_addr, dst_addr, PAGE_SIZE, 0); =20 - amdgpu_ring_pad_ib(adev->mman.buffer_funcs_ring, &job->ibs[0]); + ring =3D container_of(adev->mman.buffer_funcs_scheds[0], struct amdgpu_ri= ng, sched); + amdgpu_ring_pad_ib(ring, &job->ibs[0]); WARN_ON(job->ibs[0].length_dw > num_dw); =20 fence =3D amdgpu_job_submit(job); @@ -2196,11 +2200,9 @@ u32 amdgpu_ttm_set_buffer_funcs_status(struct amdgpu= _device *adev, bool enable) return windows; =20 if (enable) { - struct amdgpu_ring *ring; struct drm_gpu_scheduler *sched; =20 - ring =3D adev->mman.buffer_funcs_ring; - sched =3D &ring->sched; + sched =3D adev->mman.buffer_funcs_scheds[0]; r =3D drm_sched_entity_init(&adev->mman.default_entity.base, DRM_SCHED_PRIORITY_KERNEL, &sched, 1, NULL); @@ -2432,7 +2434,7 @@ int amdgpu_ttm_clear_buffer(struct amdgpu_bo *bo, struct dma_fence **fence) { struct amdgpu_device *adev =3D amdgpu_ttm_adev(bo->tbo.bdev); - struct amdgpu_ring *ring =3D adev->mman.buffer_funcs_ring; + struct amdgpu_ring *ring; struct amdgpu_ttm_buffer_entity *entity; struct amdgpu_res_cursor cursor; u64 addr; @@ -2443,6 +2445,8 @@ int amdgpu_ttm_clear_buffer(struct amdgpu_bo *bo, =20 if (!fence) return -EINVAL; + + ring =3D container_of(adev->mman.buffer_funcs_scheds[0], struct amdgpu_ri= ng, sched); entity =3D &adev->mman.clear_entities[0]; *fence =3D dma_fence_get_stub(); =20 @@ -2494,9 +2498,9 @@ int amdgpu_fill_buffer(struct amdgpu_ttm_buffer_entit= y *entity, u64 k_job_id) { struct amdgpu_device *adev =3D amdgpu_ttm_adev(bo->tbo.bdev); - struct amdgpu_ring *ring =3D adev->mman.buffer_funcs_ring; struct dma_fence *fence =3D NULL; struct amdgpu_res_cursor dst; + struct amdgpu_ring *ring; int r, e; =20 if (!adev->mman.buffer_funcs_enabled) { @@ -2505,6 +2509,8 @@ int amdgpu_fill_buffer(struct amdgpu_ttm_buffer_entit= y *entity, return -EINVAL; } =20 + ring =3D container_of(adev->mman.buffer_funcs_scheds[0], struct amdgpu_ri= ng, sched); + if (entity =3D=3D NULL) { e =3D atomic_inc_return(&adev->mman.next_clear_entity) % adev->mman.num_clear_entities; @@ -2579,6 +2585,27 @@ int amdgpu_ttm_evict_resources(struct amdgpu_device = *adev, int mem_type) return ttm_resource_manager_evict_all(&adev->mman.bdev, man); } =20 +void amdgpu_sdma_set_buffer_funcs_scheds(struct amdgpu_device *adev, + const struct amdgpu_buffer_funcs *buffer_funcs) +{ + struct amdgpu_vmhub *hub =3D &adev->vmhub[AMDGPU_GFXHUB(0)]; + struct drm_gpu_scheduler *sched; + int i; + + adev->mman.buffer_funcs =3D buffer_funcs; + + for (i =3D 0; i < adev->sdma.num_instances; i++) { + if (adev->sdma.has_page_queue) + sched =3D &adev->sdma.instance[i].page.sched; + else + sched =3D &adev->sdma.instance[i].ring.sched; + adev->mman.buffer_funcs_scheds[i] =3D sched; + } + + adev->mman.num_buffer_funcs_scheds =3D hub->sdma_invalidation_workaround ? + 1 : adev->sdma.num_instances; +} + #if defined(CONFIG_DEBUG_FS) =20 static int amdgpu_ttm_page_pool_show(struct seq_file *m, void *unused) diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.h b/drivers/gpu/drm/amd/= amdgpu/amdgpu_ttm.h index 4844f001f590..63c3e2466708 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.h +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.h @@ -66,7 +66,8 @@ struct amdgpu_mman { =20 /* buffer handling */ const struct amdgpu_buffer_funcs *buffer_funcs; - struct amdgpu_ring *buffer_funcs_ring; + struct drm_gpu_scheduler *buffer_funcs_scheds[AMDGPU_MAX_RINGS]; + u32 num_buffer_funcs_scheds; bool buffer_funcs_enabled; =20 struct mutex gtt_window_lock; diff --git a/drivers/gpu/drm/amd/amdgpu/cik_sdma.c b/drivers/gpu/drm/amd/am= dgpu/cik_sdma.c index 5fe162f52c92..a36385ad8da8 100644 --- a/drivers/gpu/drm/amd/amdgpu/cik_sdma.c +++ b/drivers/gpu/drm/amd/amdgpu/cik_sdma.c @@ -1333,8 +1333,7 @@ static const struct amdgpu_buffer_funcs cik_sdma_buff= er_funcs =3D { =20 static void cik_sdma_set_buffer_funcs(struct amdgpu_device *adev) { - adev->mman.buffer_funcs =3D &cik_sdma_buffer_funcs; - adev->mman.buffer_funcs_ring =3D &adev->sdma.instance[0].ring; + amdgpu_sdma_set_buffer_funcs_scheds(adev, &cik_sdma_buffer_funcs); } =20 static const struct amdgpu_vm_pte_funcs cik_sdma_vm_pte_funcs =3D { diff --git a/drivers/gpu/drm/amd/amdgpu/sdma_v2_4.c b/drivers/gpu/drm/amd/a= mdgpu/sdma_v2_4.c index 63636643db3d..4a3ba136a36c 100644 --- a/drivers/gpu/drm/amd/amdgpu/sdma_v2_4.c +++ b/drivers/gpu/drm/amd/amdgpu/sdma_v2_4.c @@ -1228,8 +1228,7 @@ static const struct amdgpu_buffer_funcs sdma_v2_4_buf= fer_funcs =3D { =20 static void sdma_v2_4_set_buffer_funcs(struct amdgpu_device *adev) { - adev->mman.buffer_funcs =3D &sdma_v2_4_buffer_funcs; - adev->mman.buffer_funcs_ring =3D &adev->sdma.instance[0].ring; + amdgpu_sdma_set_buffer_funcs_scheds(adev, &sdma_v2_4_buffer_funcs); } =20 static const struct amdgpu_vm_pte_funcs sdma_v2_4_vm_pte_funcs =3D { diff --git a/drivers/gpu/drm/amd/amdgpu/sdma_v3_0.c b/drivers/gpu/drm/amd/a= mdgpu/sdma_v3_0.c index 0153626b5df2..3cf527bcadf6 100644 --- a/drivers/gpu/drm/amd/amdgpu/sdma_v3_0.c +++ b/drivers/gpu/drm/amd/amdgpu/sdma_v3_0.c @@ -1670,8 +1670,7 @@ static const struct amdgpu_buffer_funcs sdma_v3_0_buf= fer_funcs =3D { =20 static void sdma_v3_0_set_buffer_funcs(struct amdgpu_device *adev) { - adev->mman.buffer_funcs =3D &sdma_v3_0_buffer_funcs; - adev->mman.buffer_funcs_ring =3D &adev->sdma.instance[0].ring; + amdgpu_sdma_set_buffer_funcs_scheds(adev, &sdma_v3_0_buffer_funcs); } =20 static const struct amdgpu_vm_pte_funcs sdma_v3_0_vm_pte_funcs =3D { diff --git a/drivers/gpu/drm/amd/amdgpu/sdma_v4_0.c b/drivers/gpu/drm/amd/a= mdgpu/sdma_v4_0.c index 96a67b30854c..7e106baecad5 100644 --- a/drivers/gpu/drm/amd/amdgpu/sdma_v4_0.c +++ b/drivers/gpu/drm/amd/amdgpu/sdma_v4_0.c @@ -2608,11 +2608,7 @@ static const struct amdgpu_buffer_funcs sdma_v4_0_bu= ffer_funcs =3D { =20 static void sdma_v4_0_set_buffer_funcs(struct amdgpu_device *adev) { - adev->mman.buffer_funcs =3D &sdma_v4_0_buffer_funcs; - if (adev->sdma.has_page_queue) - adev->mman.buffer_funcs_ring =3D &adev->sdma.instance[0].page; - else - adev->mman.buffer_funcs_ring =3D &adev->sdma.instance[0].ring; + amdgpu_sdma_set_buffer_funcs_scheds(adev, &sdma_v4_0_buffer_funcs); } =20 static const struct amdgpu_vm_pte_funcs sdma_v4_0_vm_pte_funcs =3D { diff --git a/drivers/gpu/drm/amd/amdgpu/sdma_v4_4_2.c b/drivers/gpu/drm/amd= /amdgpu/sdma_v4_4_2.c index 04dc8a8f4d66..7cb0e213bab2 100644 --- a/drivers/gpu/drm/amd/amdgpu/sdma_v4_4_2.c +++ b/drivers/gpu/drm/amd/amdgpu/sdma_v4_4_2.c @@ -2309,11 +2309,7 @@ static const struct amdgpu_buffer_funcs sdma_v4_4_2_= buffer_funcs =3D { =20 static void sdma_v4_4_2_set_buffer_funcs(struct amdgpu_device *adev) { - adev->mman.buffer_funcs =3D &sdma_v4_4_2_buffer_funcs; - if (adev->sdma.has_page_queue) - adev->mman.buffer_funcs_ring =3D &adev->sdma.instance[0].page; - else - adev->mman.buffer_funcs_ring =3D &adev->sdma.instance[0].ring; + amdgpu_sdma_set_buffer_funcs_scheds(adev, &sdma_v4_4_2_buffer_funcs); } =20 static const struct amdgpu_vm_pte_funcs sdma_v4_4_2_vm_pte_funcs =3D { diff --git a/drivers/gpu/drm/amd/amdgpu/sdma_v5_0.c b/drivers/gpu/drm/amd/a= mdgpu/sdma_v5_0.c index 19c717f2c602..eab09c5fc762 100644 --- a/drivers/gpu/drm/amd/amdgpu/sdma_v5_0.c +++ b/drivers/gpu/drm/amd/amdgpu/sdma_v5_0.c @@ -2066,10 +2066,8 @@ static const struct amdgpu_buffer_funcs sdma_v5_0_bu= ffer_funcs =3D { =20 static void sdma_v5_0_set_buffer_funcs(struct amdgpu_device *adev) { - if (adev->mman.buffer_funcs =3D=3D NULL) { - adev->mman.buffer_funcs =3D &sdma_v5_0_buffer_funcs; - adev->mman.buffer_funcs_ring =3D &adev->sdma.instance[0].ring; - } + if (adev->mman.buffer_funcs =3D=3D NULL) + amdgpu_sdma_set_buffer_funcs_scheds(adev, &sdma_v5_0_buffer_funcs); } =20 static const struct amdgpu_vm_pte_funcs sdma_v5_0_vm_pte_funcs =3D { diff --git a/drivers/gpu/drm/amd/amdgpu/sdma_v5_2.c b/drivers/gpu/drm/amd/a= mdgpu/sdma_v5_2.c index 7a07b8f4e86d..e843da1dce59 100644 --- a/drivers/gpu/drm/amd/amdgpu/sdma_v5_2.c +++ b/drivers/gpu/drm/amd/amdgpu/sdma_v5_2.c @@ -2076,10 +2076,8 @@ static const struct amdgpu_buffer_funcs sdma_v5_2_bu= ffer_funcs =3D { =20 static void sdma_v5_2_set_buffer_funcs(struct amdgpu_device *adev) { - if (adev->mman.buffer_funcs =3D=3D NULL) { - adev->mman.buffer_funcs =3D &sdma_v5_2_buffer_funcs; - adev->mman.buffer_funcs_ring =3D &adev->sdma.instance[0].ring; - } + if (adev->mman.buffer_funcs =3D=3D NULL) + amdgpu_sdma_set_buffer_funcs_scheds(adev, &sdma_v5_2_buffer_funcs); } =20 static const struct amdgpu_vm_pte_funcs sdma_v5_2_vm_pte_funcs =3D { diff --git a/drivers/gpu/drm/amd/amdgpu/sdma_v6_0.c b/drivers/gpu/drm/amd/a= mdgpu/sdma_v6_0.c index 8f8228c7adee..d078bff42983 100644 --- a/drivers/gpu/drm/amd/amdgpu/sdma_v6_0.c +++ b/drivers/gpu/drm/amd/amdgpu/sdma_v6_0.c @@ -1884,8 +1884,7 @@ static const struct amdgpu_buffer_funcs sdma_v6_0_buf= fer_funcs =3D { =20 static void sdma_v6_0_set_buffer_funcs(struct amdgpu_device *adev) { - adev->mman.buffer_funcs =3D &sdma_v6_0_buffer_funcs; - adev->mman.buffer_funcs_ring =3D &adev->sdma.instance[0].ring; + amdgpu_sdma_set_buffer_funcs_scheds(adev, &sdma_v6_0_buffer_funcs); } =20 static const struct amdgpu_vm_pte_funcs sdma_v6_0_vm_pte_funcs =3D { diff --git a/drivers/gpu/drm/amd/amdgpu/sdma_v7_0.c b/drivers/gpu/drm/amd/a= mdgpu/sdma_v7_0.c index cf412d8fb0ed..77ad6f128e75 100644 --- a/drivers/gpu/drm/amd/amdgpu/sdma_v7_0.c +++ b/drivers/gpu/drm/amd/amdgpu/sdma_v7_0.c @@ -1826,8 +1826,7 @@ static const struct amdgpu_buffer_funcs sdma_v7_0_buf= fer_funcs =3D { =20 static void sdma_v7_0_set_buffer_funcs(struct amdgpu_device *adev) { - adev->mman.buffer_funcs =3D &sdma_v7_0_buffer_funcs; - adev->mman.buffer_funcs_ring =3D &adev->sdma.instance[0].ring; + amdgpu_sdma_set_buffer_funcs_scheds(adev, &sdma_v7_0_buffer_funcs); } =20 static const struct amdgpu_vm_pte_funcs sdma_v7_0_vm_pte_funcs =3D { diff --git a/drivers/gpu/drm/amd/amdgpu/si_dma.c b/drivers/gpu/drm/amd/amdg= pu/si_dma.c index 863e00086c30..4f6d7eeceb37 100644 --- a/drivers/gpu/drm/amd/amdgpu/si_dma.c +++ b/drivers/gpu/drm/amd/amdgpu/si_dma.c @@ -826,8 +826,7 @@ static const struct amdgpu_buffer_funcs si_dma_buffer_f= uncs =3D { =20 static void si_dma_set_buffer_funcs(struct amdgpu_device *adev) { - adev->mman.buffer_funcs =3D &si_dma_buffer_funcs; - adev->mman.buffer_funcs_ring =3D &adev->sdma.instance[0].ring; + amdgpu_sdma_set_buffer_funcs_scheds(adev, &si_dma_buffer_funcs); } =20 static const struct amdgpu_vm_pte_funcs si_dma_vm_pte_funcs =3D { diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_migrate.c b/drivers/gpu/drm/amd= /amdkfd/kfd_migrate.c index 943c3438c7ee..3f7b85aabb72 100644 --- a/drivers/gpu/drm/amd/amdkfd/kfd_migrate.c +++ b/drivers/gpu/drm/amd/amdkfd/kfd_migrate.c @@ -129,13 +129,14 @@ svm_migrate_copy_memory_gart(struct amdgpu_device *ad= ev, dma_addr_t *sys, struct dma_fence **mfence) { const u64 GTT_MAX_PAGES =3D AMDGPU_GTT_MAX_TRANSFER_SIZE; - struct amdgpu_ring *ring =3D adev->mman.buffer_funcs_ring; + struct amdgpu_ring *ring; struct amdgpu_ttm_buffer_entity *entity; u64 gart_s, gart_d; struct dma_fence *next; u64 size; int r; =20 + ring =3D container_of(adev->mman.buffer_funcs_scheds[0], struct amdgpu_ri= ng, sched); entity =3D &adev->mman.move_entities[0]; =20 mutex_lock(&entity->gart_window_lock); --=20 2.43.0