From nobody Sat Oct 4 22:37:10 2025 Received: from NAM11-BN8-obe.outbound.protection.outlook.com (mail-bn8nam11on2066.outbound.protection.outlook.com [40.107.236.66]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id EDC102080C0 for ; Tue, 12 Aug 2025 02:40:49 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.236.66 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1754966451; cv=fail; b=VCHgphwmuO8y4lr8h4RbzT9uGh9t5ycZ/HlPR9E4mZPaGpODBbKVQPV1V0dl/2WVs/NnJK4GzDD+DZQXRpHpOka9wrZVZY2YP/r5B3NBR2aVz0syPig9l4S+KcrAf8uySyjD/iITGuNHLYiTpBKcYOH4lLMH5/9IHaanTbFLPds= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1754966451; c=relaxed/simple; bh=PyrDHj9s2oNotafVlu9zzd0hptLifieEL79j77SUU24=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: Content-Type:MIME-Version; b=czhOibasJhMUOa4ABIAa5O7/DE2BxkIs17Bb+vD6K46cLaqEszJHx0Tf++pmUsRHRgCxMfXWTNxFeUH4UEy/PN/IsvgRLPMDvm3nMpUDQF5HyIpaOTJBOWrp6UuPYnmXiJ1f5Emkcfh7t6U1bJvBc2ij6AECbLEsUO1l9ZLs79U= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com; spf=fail smtp.mailfrom=nvidia.com; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b=VG3cboYs; arc=fail smtp.client-ip=40.107.236.66 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="VG3cboYs" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=qr+Rr8mPYW1KDgN9bznF99ET5kvzhBnd3rEDXC6HI/oN7hougXKyHfSn5SnDSAmj1XkUrTWE/yXV4lUp0jE+uvCA0F2/4x4eFFUjbhtImVaiKsDQbt3HOlrtYniwec560KPfLpNOIn7n3T48EEfzTG8aDrqjx4sk3XQ0STY7g9sdt9UhWcwv7y0xta7AwsFA7TrDOp1jBLsEUvTq7S1+7uzrGBZ+c1lqd1O/vD5htDw/N4JdzWVdReae1uagaIuaqO16LxwzkUxrZoH8quYkRhjsPVzbOpctLxQUlgPFL/NsVTbFX+sPltTzTkffdCild8drZlgW32EMBgcK1NuJ4Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=FLRnZ3g14MH+A0H5eZGV2PREorZoUiRXRp7IIDGKm18=; b=cD1qdbFfLCL7jSJh5jDO5BM65+dr7Af5hn99aAlO8S6XkJosewy96AC2gAxuM+uU9nrBgQDfb65WzUMwm8ZDVgg2vr1KQp6Rlq7/WbTS6NX4fPUXDZeQJlKrYiHiacCs7Up4NlnhakdGytzEUDo+z03HmK1HZs/Q2O33u+nQ2gsDtpuQfpjUg9x9bKYWX2+mqTHs08opbiyAwbMgoKBw7di8eVu1ztuNy3qKgZXeIx0JhZ3pAAVWGqS7VzUhgT4sPTXF/h5cmf3BFgqCV01N0OC3jta3nYeeXa3sgolJyJ+BwFYA1kbfhKKdKjMq3tHqrd7WZbd1hbaE63SCQyHyfQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=FLRnZ3g14MH+A0H5eZGV2PREorZoUiRXRp7IIDGKm18=; b=VG3cboYse2B0+8O2WYzp6eqBGIg6zr8orY6qAFzEC1nKYs0r/31iOYUX9/o8vXHQo/lQwbbKPw5yv2yugizaeWQRcq/urnusmN4QZF/yeW7Yei22d8jg1PN17JwEucljSG0eW4MhRWdhJ554myr+1cY2SU9wowvOdiry9zV0nRi7xIFczs4EV9+YfOrhpWcl2zYT2wZVfwgQoDjJ9xoomQsOcwEXgDckT/TnLtrskmDh79DsV2r/9nyjzCYE4G7cCMgho37V11IHbLoG6d45BW/NU0PTORXlfyxdk0YomU+K32QtEH6ysE/FkFVht1SImDindNFH37Nt4xy2NugMRA== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from PH8PR12MB7277.namprd12.prod.outlook.com (2603:10b6:510:223::13) by CY8PR12MB7172.namprd12.prod.outlook.com (2603:10b6:930:5b::7) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9009.18; Tue, 12 Aug 2025 02:40:45 +0000 Received: from PH8PR12MB7277.namprd12.prod.outlook.com ([fe80::3a4:70ea:ff05:1251]) by PH8PR12MB7277.namprd12.prod.outlook.com ([fe80::3a4:70ea:ff05:1251%5]) with mapi id 15.20.9009.018; Tue, 12 Aug 2025 02:40:45 +0000 From: Balbir Singh To: dri-devel@lists.freedesktop.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Balbir Singh , Andrew Morton , David Hildenbrand , Zi Yan , Joshua Hahn , Rakie Kim , Byungchul Park , Gregory Price , Ying Huang , Alistair Popple , Oscar Salvador , Lorenzo Stoakes , Baolin Wang , "Liam R. Howlett" , Nico Pache , Ryan Roberts , Dev Jain , Barry Song , Lyude Paul , Danilo Krummrich , David Airlie , Simona Vetter , Ralph Campbell , =?UTF-8?q?Mika=20Penttil=C3=A4?= , Matthew Brost , Francois Dugast Subject: [v3 01/11] mm/zone_device: support large zone device private folios Date: Tue, 12 Aug 2025 12:40:26 +1000 Message-ID: <20250812024036.690064-2-balbirs@nvidia.com> X-Mailer: git-send-email 2.50.1 In-Reply-To: <20250812024036.690064-1-balbirs@nvidia.com> References: <20250812024036.690064-1-balbirs@nvidia.com> Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: SY5P300CA0106.AUSP300.PROD.OUTLOOK.COM (2603:10c6:10:248::7) To PH8PR12MB7277.namprd12.prod.outlook.com (2603:10b6:510:223::13) Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: PH8PR12MB7277:EE_|CY8PR12MB7172:EE_ X-MS-Office365-Filtering-Correlation-Id: f015bfdf-0277-46c5-8ef0-08ddd949a304 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|366016|376014|7416014|1800799024; X-Microsoft-Antispam-Message-Info: =?utf-8?B?YUNDQWVQVFhQMzhwZDlVS2x6MkY4OTh5OU5ja043V01hcVpCenVuZG44TE4v?= =?utf-8?B?ZkkxSkd4bmJENDFEY0hOVWROWnl3TEYrZUNqWmpKaEtkc1ZzZFN5Zjl0cUVj?= =?utf-8?B?YTNOQTNCa2ZOVC95WWN1NzdPa1dtOHB1MDRSNnN6ZkJmdmJiMTBSSmhMU25o?= =?utf-8?B?Wm5XQm5waWFzcHJ0eWJPaTVadms0RHI4SGZlTU5tVVlUeW1ZNmxubnhNUWsy?= =?utf-8?B?U0JhR1hKdFlEYlF3NDFiVE1NVmV5TUVYdTFGamFKSmVTSWIrK0gvUlBuU1Y4?= =?utf-8?B?bGtuTkRyZDhHekRYOXlKVHR1TFFPZUlJQ21OcVNLNU10cUQrUDJwalZmYzFD?= =?utf-8?B?eklWcDZMRndieWpIc2svUXJHaVdTZjZISmlyQXdwZUN4UnhsNHpQQ3Jqa2Fm?= =?utf-8?B?RlFYNmcvbC9VT1R6azZwNlQrVkE0b3NCN1J3QkVNSkt6TUoxeTZrais3YkVY?= =?utf-8?B?V3I1WStoZDl0ZHBiekZWOXM4U0lMamZXdDJZZkhDLzNuSTlFaVJPK1p0UmZ0?= =?utf-8?B?VXNUcEVhSjZkU1Yvc3Z1MjV2TEcvL0xXVGdyRmVvNlJKQUsrK2ZuTGVWNGl0?= =?utf-8?B?VC9uTnhIL1dIMSsyQ2RtRmYzS1N1eVJRRGNjZm5KYWZ6b21YSUxEcVRwRTF0?= =?utf-8?B?QmxQTHJsbUFYdFFTRm5Zd0E4WkpvUDBrS1VtMTB4dUMwVkRJcUhOcjVqWmNN?= =?utf-8?B?UndLMXNvelRtaUxDK1FkV0d5SkVTakxNTE5wVVk4elFBODdpcjBJaGFOOFFa?= =?utf-8?B?NENQQVdPb2xUWVFTTmgyNkordGJYOEtVeStHazl1WDJNZ0ZlMEhydHlEMlVD?= =?utf-8?B?R3FMdmdjMDBuYjJaQVJTL1labFJRRmhGR1E1R2lJNHc0QlpqYm9DbW16aWk5?= =?utf-8?B?ekZqZGxGOGIxOEtwbkhBS2RrcVpZQmQ1bHkzZHp0SmlvYzdoU2x2M29QeER4?= =?utf-8?B?ZGpOdGcvU3owclp4MXpRVG40SXhOWTJrWVNhb0dxSFFySjh2dEtNR21Hbmpy?= =?utf-8?B?RGc0QU8rc1ZkbllBdnE1M2JuOXZGcUIraHloWW9tS2dxR2JNRGFWRXFSL0tM?= =?utf-8?B?MkFkZnhGbkw2aFk5SjdVa2krVzlYcGNNSUtWQ3BGbDNFREVRQ0J6cTByTFZu?= =?utf-8?B?UlMzd0NFZlVqU1JqajNGQWJjR0NwaE0rWWxVL2NiYmY1QTZKdHZSVThyOWxK?= =?utf-8?B?b1NLSC9hUmFINlF0ZE5vQ21rb2M4V2tZQkUwQ3lrQUtzdDNDcCtuK05oL3E1?= =?utf-8?B?MndDS3RTOTduK2RVVDBqUHNiYWlWVWg0MmgwMHkydnZpRnpuN044cVR0am40?= =?utf-8?B?VXdiaUJRdGFjQThWamVZZG5zR0sxcXFTRlRGK1A2ajdycjdoOGJJc0NIeWhL?= =?utf-8?B?VFRza1lCbkg2UnhKUWROMXo2d09LNEtFSDJhaVFucHBPdVFzMTJhMDEwVm52?= =?utf-8?B?WVVzbk1CRFIweFBhNnhoQVY5bDJGTnp6SDk2WUtObFNpUnNZdW9hNG9ibkV3?= =?utf-8?B?elY4elhWNTBiYjNxaVZkR0tyR3c3Kytjb3pMaldHeHZCdDRKTDNOSkVyWWU4?= =?utf-8?B?RGxXb3ZDdDFsQ1dseXV5bDJSYmd4R1E4cmJ1MFFhdnhuTVUza3U4VjVRNzAw?= =?utf-8?B?ZGxFU3l0cm1KT2kzYVN5VnR4Q2VCWnkydmlDeUpudGhqcldGYmZjc1loS1R4?= =?utf-8?B?VjVXTUJPYjlsZTdyQnhVSy8wQno5YkJQSjIzV3llRUhxbGZqT0cxYm5pcmt4?= =?utf-8?B?dUlBOFJXbW8veTEydjJ1MGYwWFQwQnRsTEhpRHk5c0dQYS96TlhlOFc4dnVp?= =?utf-8?B?R0RIOFFsZnNzYk1JRHB0Tm1VYjcvRmpKSnN5aUVSQktYVlVKWmFpRDYvVkdj?= =?utf-8?B?dE9QUnI3NW5SRUdLSFBIYUJSNVNPa0FwaXpOUGVCOFFMZURjNjVUSEJxU0pF?= =?utf-8?Q?0qZKDY26qIQ=3D?= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:PH8PR12MB7277.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(366016)(376014)(7416014)(1800799024);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?M0ZlOGlUN1lMQXlxSThGVHI5OW9OUkkzeVdqRTlCakZoelNmRWY2RmFMMU52?= =?utf-8?B?LzU4QWQwT0IyekNOWXJLbkZkSVMwSkdrMEMrMUJCWUNDNytrTlplUWJWQVlx?= =?utf-8?B?T2pSaXM2N3NDTWh4bkJoZXhIL2Z4bDJldG42cG9YRS9aeVJpR0E4akc2Qzgv?= =?utf-8?B?V3VIOG1YbFY0VStOMmpNSVk2alNtci90NkVLRFpXK3RENTN3cFhQN05OV0tQ?= =?utf-8?B?S1I5WlNocDRoNnd0UWh2cHRzR1JyQ21nR2dLRndBSFJNcTJnSkIyZU04a0Zn?= =?utf-8?B?UkdsdXFid0lJUUdCcGZONFExTGl4RWJqVjczOGdzK0dDWk03b3hSaWh1cG5X?= =?utf-8?B?di9vZHV6Y1hmLzVwMmt1TmxIbmtOVWI4WWlyZXR5aHBvQ1dxN0s4Zmx4YWxN?= =?utf-8?B?d0JnUFdkS0o3d2ZyclRqcFZ6aWJtNmtnOHhFc1djbnVUZzY4QUozSWRhSkRE?= =?utf-8?B?YWEyMkx6RFpnMStvcXNkYTFpMjI3c2hqNm00MmZ3YWR6KzVCWCtucU5Qak91?= =?utf-8?B?aEFCOHJWMVZOVHRGS0pzOGlXazhxUVRyRlU1TFJURHRxWXRJZksrNGp2enpu?= =?utf-8?B?eUQwWHVxek1DTU82alZSbjMvVUoyMlJXWlVmZ3pMcXhVTmlsOG1OYVk3OStB?= =?utf-8?B?RnZtYllzRFRQeUNVS011UXdEODZsR1UyUUpVcUQydnEvMjRMcGFDUEMyZmJQ?= =?utf-8?B?czlFTmNoUFFOQjlpUTJqRmVsNC9GZFNyZDg0dk5iR1FwS2E1b0F2SXkyU2J5?= =?utf-8?B?b2hHZ005cHdGQUYrUzZLUGE0dlJldll5clFFWG8zcFdSYmRReU5qa3RCZEtR?= =?utf-8?B?b1QzTXRMRk1SSWR6MmgwU2lOK0lZVDBpanZwVUJqMXFhMDRhZkhmd0dIdFdF?= =?utf-8?B?aU1LV1R5dE5WMlRCa2VsZmhBY1V2VUUwWjdPaU9hNkE2d1NMRnN3cmtrUzlE?= =?utf-8?B?UmFsSE1SZVZXZXlWaDVwNlVZWTBqNjNrNjFwenR5N2ZFRG82QjU3UEdYTjZu?= =?utf-8?B?TkVQc3ZVNEJMYWVFOE01dkMvREdFQ0xMNHVOalBYOTZaQnBSOHp5ZjMxU3kx?= =?utf-8?B?ME5jV2l4YkVWRUxZNTdhdDV0SGtqS1ZVUVBoWWJUb0VLNlFuM3N1b3cxVHlz?= =?utf-8?B?RTN4ZVpHNnR2aldabmdiN0UwSXZTd3dkZjJkWGgzc2ZWd2RTUEN4TE1OSWVP?= =?utf-8?B?L3FoVUVYcjF0SEI5aDJxWXdyTktpNW9Wa1Rtck9SOWl1SjFaRGFTdlhNY2ds?= =?utf-8?B?cnR0Yy96L2VvV1FLQjZJaXNnN3d1SzJLM3hHUVJIS0Vyc05PVGt3bW9PNjVG?= =?utf-8?B?VkNhRjQzZmRuaTZNWTdiR3dzVktVZkVvZWhONGxhV2NrUzFBR3FtazJpNWlt?= =?utf-8?B?OG53Zk5qV1BYRVNoOG5xc2EyZUpVUGtlckQrbWhGM2VrcU9ZN3pyUFh5Qmdz?= =?utf-8?B?TEZMeGNLVExQR3lmOHBOTU1kV20xeUVVRENweVJLcUVZNml0dkM3Y0JENUUy?= =?utf-8?B?YmQ1M2RSdkNUSnhVQ2ZzM1M5VDZ2Z3kxbms0NmEvMHJEL0IvSUhPMlF3QzR1?= =?utf-8?B?eW5RdEZqa1o2Yk9rQ3ZTeDBoR3BQY0p6eEduN0g4QkVnMmRYRHorVzdKQUFk?= =?utf-8?B?cTlVVUlOMVBndGM5aEpkbzZCcjBWNjhXWFdEVHRValA2N1pxd1EwN0NRdFNS?= =?utf-8?B?ZXlrdHpjKzJ6OGd6S0xsK3p2U2hqS2pBVlZMTUczTnVyUGhKQVFpYWZ2dHlr?= =?utf-8?B?QlhsNGVuMTVXeFlJQW9nclNvU0p4ZVRlSDJzTFpDY1hHeG5rbGE3ZlNxTTR0?= =?utf-8?B?ZzgxNVIvbm5xN041SFdlYTkxY3pXS0w3Snp2SUNDY2JnQS9hMWtybTB3REI5?= =?utf-8?B?TWVETEFZQ0VxK3JtVXQyTjNYNGZwUzUwWVNxeE1RSmNjeS9FZjk3WlJkeHhm?= =?utf-8?B?Q0NTUnA5MUtBeUlMTEh6UUhzeW5UZm5SSGJtNnA3MWhFVVA2VjcwZkkvSllL?= =?utf-8?B?Tll1bVF3V1RScHZpRG92M1VxeS9hZGZwZUl5MFdScldGK3JJc29MNXpIVURp?= =?utf-8?B?d3FZR1BVV3dES2ZzNHR2NXBxbHUzaWlXMkR2eURWZlZpWFAyRStzbDNUNjdN?= =?utf-8?Q?AVEB63sr931e4QR9Y35mbBkof?= X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: f015bfdf-0277-46c5-8ef0-08ddd949a304 X-MS-Exchange-CrossTenant-AuthSource: PH8PR12MB7277.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 12 Aug 2025 02:40:44.9996 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: rNLvql9T9Agr0CVSl8GWzCFbD7f8a/0XKEuLfSZPUWJIATpy1Y4mtp3rEqjhPkTJjVfgvBaKbtIM+hU978EprA== X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY8PR12MB7172 Add routines to support allocation of large order zone device folios and helper functions for zone device folios, to check if a folio is device private and helpers for setting zone device data. When large folios are used, the existing page_free() callback in pgmap is called when the folio is freed, this is true for both PAGE_SIZE and higher order pages. Zone device private large folios do not support deferred split and scan like normal THP folios. Cc: Andrew Morton Cc: David Hildenbrand Cc: Zi Yan Cc: Joshua Hahn Cc: Rakie Kim Cc: Byungchul Park Cc: Gregory Price Cc: Ying Huang Cc: Alistair Popple Cc: Oscar Salvador Cc: Lorenzo Stoakes Cc: Baolin Wang Cc: "Liam R. Howlett" Cc: Nico Pache Cc: Ryan Roberts Cc: Dev Jain Cc: Barry Song Cc: Lyude Paul Cc: Danilo Krummrich Cc: David Airlie Cc: Simona Vetter Cc: Ralph Campbell Cc: Mika Penttil=C3=A4 Cc: Matthew Brost Cc: Francois Dugast Signed-off-by: Balbir Singh --- include/linux/memremap.h | 10 +++++++++- mm/memremap.c | 38 +++++++++++++++++++++++++------------- mm/rmap.c | 6 +++++- 3 files changed, 39 insertions(+), 15 deletions(-) diff --git a/include/linux/memremap.h b/include/linux/memremap.h index 4aa151914eab..a0723b35eeaa 100644 --- a/include/linux/memremap.h +++ b/include/linux/memremap.h @@ -199,7 +199,7 @@ static inline bool folio_is_fsdax(const struct folio *f= olio) } =20 #ifdef CONFIG_ZONE_DEVICE -void zone_device_page_init(struct page *page); +void zone_device_folio_init(struct folio *folio, unsigned int order); void *memremap_pages(struct dev_pagemap *pgmap, int nid); void memunmap_pages(struct dev_pagemap *pgmap); void *devm_memremap_pages(struct device *dev, struct dev_pagemap *pgmap); @@ -209,6 +209,14 @@ struct dev_pagemap *get_dev_pagemap(unsigned long pfn, bool pgmap_pfn_valid(struct dev_pagemap *pgmap, unsigned long pfn); =20 unsigned long memremap_compat_align(void); + +static inline void zone_device_page_init(struct page *page) +{ + struct folio *folio =3D page_folio(page); + + zone_device_folio_init(folio, 0); +} + #else static inline void *devm_memremap_pages(struct device *dev, struct dev_pagemap *pgmap) diff --git a/mm/memremap.c b/mm/memremap.c index b0ce0d8254bd..13e87dd743ad 100644 --- a/mm/memremap.c +++ b/mm/memremap.c @@ -427,20 +427,19 @@ EXPORT_SYMBOL_GPL(get_dev_pagemap); void free_zone_device_folio(struct folio *folio) { struct dev_pagemap *pgmap =3D folio->pgmap; + unsigned long nr =3D folio_nr_pages(folio); + int i; =20 if (WARN_ON_ONCE(!pgmap)) return; =20 mem_cgroup_uncharge(folio); =20 - /* - * Note: we don't expect anonymous compound pages yet. Once supported - * and we could PTE-map them similar to THP, we'd have to clear - * PG_anon_exclusive on all tail pages. - */ if (folio_test_anon(folio)) { - VM_BUG_ON_FOLIO(folio_test_large(folio), folio); - __ClearPageAnonExclusive(folio_page(folio, 0)); + for (i =3D 0; i < nr; i++) + __ClearPageAnonExclusive(folio_page(folio, i)); + } else { + VM_WARN_ON_ONCE(folio_test_large(folio)); } =20 /* @@ -464,11 +463,15 @@ void free_zone_device_folio(struct folio *folio) =20 switch (pgmap->type) { case MEMORY_DEVICE_PRIVATE: + percpu_ref_put_many(&folio->pgmap->ref, nr); + pgmap->ops->page_free(&folio->page); + folio->page.mapping =3D NULL; + break; case MEMORY_DEVICE_COHERENT: if (WARN_ON_ONCE(!pgmap->ops || !pgmap->ops->page_free)) break; - pgmap->ops->page_free(folio_page(folio, 0)); - put_dev_pagemap(pgmap); + pgmap->ops->page_free(&folio->page); + percpu_ref_put(&folio->pgmap->ref); break; =20 case MEMORY_DEVICE_GENERIC: @@ -491,14 +494,23 @@ void free_zone_device_folio(struct folio *folio) } } =20 -void zone_device_page_init(struct page *page) +void zone_device_folio_init(struct folio *folio, unsigned int order) { + struct page *page =3D folio_page(folio, 0); + + VM_WARN_ON_ONCE(order > MAX_ORDER_NR_PAGES); + /* * Drivers shouldn't be allocating pages after calling * memunmap_pages(). */ - WARN_ON_ONCE(!percpu_ref_tryget_live(&page_pgmap(page)->ref)); - set_page_count(page, 1); + WARN_ON_ONCE(!percpu_ref_tryget_many(&page_pgmap(page)->ref, 1 << order)); + folio_set_count(folio, 1); lock_page(page); + + if (order > 1) { + prep_compound_page(page, order); + folio_set_large_rmappable(folio); + } } -EXPORT_SYMBOL_GPL(zone_device_page_init); +EXPORT_SYMBOL_GPL(zone_device_folio_init); diff --git a/mm/rmap.c b/mm/rmap.c index 568198e9efc2..b5837075b6e0 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -1769,9 +1769,13 @@ static __always_inline void __folio_remove_rmap(stru= ct folio *folio, * the folio is unmapped and at least one page is still mapped. * * Check partially_mapped first to ensure it is a large folio. + * + * Device private folios do not support deferred splitting and + * shrinker based scanning of the folios to free. */ if (partially_mapped && folio_test_anon(folio) && - !folio_test_partially_mapped(folio)) + !folio_test_partially_mapped(folio) && + !folio_is_device_private(folio)) deferred_split_folio(folio, true); =20 __folio_mod_stat(folio, -nr, -nr_pmdmapped); --=20 2.50.1 From nobody Sat Oct 4 22:37:10 2025 Received: from NAM11-DM6-obe.outbound.protection.outlook.com (mail-dm6nam11on2047.outbound.protection.outlook.com [40.107.223.47]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 896342E336C for ; Tue, 12 Aug 2025 02:40:54 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.223.47 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1754966457; cv=fail; b=qAZXZyvZXb+64qkX4goMXKznz3bmDDzo2gLSOw+R8xlNVE76+ArwR5ojUPImvymbEUFLj8z4D+cIvbt4s5Han+oPtMdefQ6mq6H8hLAEtZIKagOO+ajwEvXjMBijaaEp1xMP+WmPXH6IcAhciBvMWO4jgcEwjf2uPjIHFBlv+rI= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1754966457; c=relaxed/simple; bh=KpgEOLLr9gvzIv1dlwbr01WOfo0HMowl9fRWr+dKpcM=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: Content-Type:MIME-Version; b=YmEe9bkIekNtsrDlU/8MvOpsRVF1JJCcxNjq4mUpzFEMnGyvEkIY0DBhenrw/mgQbmETc+XZSdLAA9/ghz2BtT14048FnDMd7aBHSnZiYfQtYEqWmh5y8SGXKOlmQ6hwtFYzOBS8KidSQEtT+rmau+g8Hn7jmlcmTFzH39HWZXE= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com; spf=fail smtp.mailfrom=nvidia.com; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b=lMbXMniB; arc=fail smtp.client-ip=40.107.223.47 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="lMbXMniB" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=T6W8eZwVmtNoI+eaElKmHK+Oz1lAsU5PfNT0/wSZe/0sbJ6lhPnWUe9Ic+CgDayxCjZA4rzMKGs0DnrxJBMi/FqJmtWMYUc791OE3siHQAFPoVWhvD4ZNBfA6dHGP8B6YC3UDGtnPPChysCtSFRhBcIyGm4yL1hzeNBhAXEWY2/DxRZ6V11qbq5k1HanbliUXOCIL16mcTLzUzVSte6cYHIeIEo618aCUHYntcmXeimNInYwYp9JMoOKiYhWMPSa5QEpXh4NpxCZYRD2rrGN8k8I2KrDVvHAFey2IP6ycmM8FaNYxRmkeTHLwNb85JB94G0Ybgzv6LcputGzDxuFTw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=wepY4811lCD/cHhl4YfPbchgnSW4Dq67Ffvdh3Ykvx0=; b=LMCimuX9i0gOiODUEsgiPjrc1PcDYrRSWymPdEKTZ69ghquIBIil6FiYPVf7Q1lnzs9v7uKc5KHgeDB/Y5IoM3XEcz2W1Wl0HE+7GixQoGHQOht4OS4b0PoNHzo9ZdebFw8iU4lLgyD2VdP+HSmiv6GpPkPGLxnVV90+IXhyfGAtZPugb+9ekPjF/22fwWSkq4L3CV9HyBrkaYx0XzTB3tceaMaQ+037iz04bZpAshnqgs8uhhSSCOqZrJty70lWteFKXUhs5Jl+11QCTr+qcQdw8vdZg6lZdTGfEhTKD1FkECJRqW+3qmh2eUiZCYeCAy3OyC0AR+FoSNaplNZqww== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=wepY4811lCD/cHhl4YfPbchgnSW4Dq67Ffvdh3Ykvx0=; b=lMbXMniBtFEDe/ocwmAPPBJUuq0WFvUBzAszULztgnvBASwGhTXgm9mTDgm3jjnaWlTx4WIwt7UtOWTywbP3vLXAbkBc/ubLO72OMDGDALrKi3AqwRFc5zWpAX3o90VJwHtvOvKd2PKU3Q4tRf+qOvlcXAWSzt+EPng8nWgmb2B8hP9WFpyAtAH832W5Luy4rW0UXBUE8lQXaSQBk4M/jRmt83+uM6LyswHNj9yqDbXJbha/u91uVfWPwUMaU48K1akOuGyuOmI/qVVF2We9pUyR6kJRGUnG1heLLqys/HCdKT/bINtawe+y16zFfcUqbk3s61pnHq234ov4YoXG5A== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from PH8PR12MB7277.namprd12.prod.outlook.com (2603:10b6:510:223::13) by CY8PR12MB7172.namprd12.prod.outlook.com (2603:10b6:930:5b::7) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9009.18; Tue, 12 Aug 2025 02:40:48 +0000 Received: from PH8PR12MB7277.namprd12.prod.outlook.com ([fe80::3a4:70ea:ff05:1251]) by PH8PR12MB7277.namprd12.prod.outlook.com ([fe80::3a4:70ea:ff05:1251%5]) with mapi id 15.20.9009.018; Tue, 12 Aug 2025 02:40:48 +0000 From: Balbir Singh To: dri-devel@lists.freedesktop.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Balbir Singh , Andrew Morton , David Hildenbrand , Zi Yan , Joshua Hahn , Rakie Kim , Byungchul Park , Gregory Price , Ying Huang , Alistair Popple , Oscar Salvador , Lorenzo Stoakes , Baolin Wang , "Liam R. Howlett" , Nico Pache , Ryan Roberts , Dev Jain , Barry Song , Lyude Paul , Danilo Krummrich , David Airlie , Simona Vetter , Ralph Campbell , =?UTF-8?q?Mika=20Penttil=C3=A4?= , Matthew Brost , Francois Dugast Subject: [v3 02/11] mm/thp: zone_device awareness in THP handling code Date: Tue, 12 Aug 2025 12:40:27 +1000 Message-ID: <20250812024036.690064-3-balbirs@nvidia.com> X-Mailer: git-send-email 2.50.1 In-Reply-To: <20250812024036.690064-1-balbirs@nvidia.com> References: <20250812024036.690064-1-balbirs@nvidia.com> Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: BY5PR13CA0010.namprd13.prod.outlook.com (2603:10b6:a03:180::23) To PH8PR12MB7277.namprd12.prod.outlook.com (2603:10b6:510:223::13) Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: PH8PR12MB7277:EE_|CY8PR12MB7172:EE_ X-MS-Office365-Filtering-Correlation-Id: 9514fa28-5d3a-49d8-f660-08ddd949a4dd X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|366016|376014|7416014|1800799024; X-Microsoft-Antispam-Message-Info: =?utf-8?B?WklhS3Vpcmxuc2c3NTJzdS83bEs2Y3l2Z21VZE1sWTFlYUtERDg3QldCNVVk?= =?utf-8?B?ejUxQ0xkVGYwTUZ3V2RvZU5xU1VISWxONzlZY0dqaDRZZ2ZKaC9NdXNVK25N?= =?utf-8?B?Yjljb21BcDc5VVZsV2kzSUJxWnVsMUJuTHk2VHZkd1BvNXY2Q2NGcXdNOHZh?= =?utf-8?B?M3RWOHcyY3FyRm5PREhHMWhFL3VFbmE0VWlRZWJXUXJieUVid1NWSkg5V3ky?= =?utf-8?B?OU96QXZhdDBtSWhNNERpMzdoWWNHZXh1eEZ5cHZJVUttZ00wSzdBR2x0em9w?= =?utf-8?B?VmM2Qk42bEFpMVNxeHI1RTBGY0RpSDBKTEdkRUY2MzZZMDFUNURKTW5jYXc4?= =?utf-8?B?T01hNlp0S25Vck9Ga2ZHQXF6dXZkNzZGV2VrdkNobWZYUUQyeE5wTTg3RHBk?= =?utf-8?B?Y1ZTY21IWUFJWDdULzRrM1BLdkVzQnhvTWd6b2VkU2Rzb2ttVlBwTWd5ZWhv?= =?utf-8?B?N1Jqcnh0WmlGcDNYVWYvMDM2b0lRMURTRjFJWEUwYkQ5dVlEeGpvOVVLWEpr?= =?utf-8?B?QXdTZm9DN1lnWnBPbjFWMFZRUTYybFdnTVc2SDVsaE4rajlxQUFSdEZBeVY0?= =?utf-8?B?b1J6NGpYK3JwNCtpRkNwSDhSLy9EWWNaTDUyZndBS3ZhalhlNTlpdG4veTM5?= =?utf-8?B?M1JXcW9DOTVKTi96VDJPYUo3QzErdFVkdDhNNGhzQVA0NFlxbkQ5QVlzdk1k?= =?utf-8?B?dndlc3VvS1EzUTgvRHcwM0lZMkd4YXVaRHh1dmFHR1hDSzdsYjFWWmh2ZjNO?= =?utf-8?B?SldkS2NURCsxUGRUNENqNWhEajBVb0FuNFlMZndHZ3ozSGREei8zWUNjTVFL?= =?utf-8?B?YjU1YlpHcW9hNzliSWRFend0SGYyTTZLZFVXZFRBUzEyWmxpSlA5djFEbmt0?= =?utf-8?B?YTFkQ3RiWUtoS00xbzRwNzVlV0RZUEMzQUxmZEZ0VVFlR2pIbnREM3k2bVhy?= =?utf-8?B?MHBibURhZ2NmcnBsQ1A4ekdHaEVTMnYrVDIzT1BZZVlWMnBIQ0ZaTEptWTI2?= =?utf-8?B?ZFQrMHJUTFEybTNlbkJEYmVzUjV2WHdnaGZIN3EvZmJHR0UxU0VQWUJROEN1?= =?utf-8?B?VkNVNjl0RTdjTWlER3JoWDVSNXNveW4xeHBnd1ZCdVBzQW84Wkc2U0hKZlBR?= =?utf-8?B?cXNVZ09WWWdsVDFta1FwOFY1SDBpUlExSXRKdXcxRmw2T0Z4OEJnT3hBcklq?= =?utf-8?B?S3JYTHhhZG54c1Vud3MzWmlma0xCNG5CTHc0NHNnMVB6cEo4NWxxd3AwYkJv?= =?utf-8?B?dHcvWDNJVFpqTWg2bnp4Z2FTeEpLaXpFbU9ncStXWWhNR0ZzZGRGS1lXNjM1?= =?utf-8?B?TjhaUDJKVmViMEhFWTR5OUVuZWdqNkkrUUEzNkxsd1FXVUNPcHh0WmVWcnlE?= =?utf-8?B?TVJYdVIwaXdWWkhHMGdHNHFQbTZoc0x4VzJGNHl5MTNnWnRyWVdPUXp2RHo2?= =?utf-8?B?bGlwR2UvL1FzUmlkYXFEeVB3M0phQnJKd1AvWitTSmJSa2l4bHVrZXo2QndF?= =?utf-8?B?YTUyQ2h5a1RNdXZqYnBJY1pjZGZjbTVHalFPWjArQzZWaHRDbDZ6T0tUUW82?= =?utf-8?B?U2ZsSVlSYkxzN05WZERvZkptK2pDcUJ3Rkh0VW1DRGhtR05MSVRYVkJ0V0Jm?= =?utf-8?B?dDNTaGNpdjRGWnp3UW5XdDVGcFJPZkVYTVRnRmVLR244VGtBRTIwREx3R3hi?= =?utf-8?B?ejJONGhhWkIzN3R1b0VMTG5kVkRpR2Q3L2RFMXYzOHBPdERxN1YxYTNnZjZ6?= =?utf-8?B?REZjUGd3NXU5emdSZDVGa0lnaVo5d1U1NHVFdzNDSHhsSnYxU2dTb1kxdmJV?= =?utf-8?B?bnduNWxPVUJyV0gwK3J1NTQ5MXVuTzFhdWZXdENDMTVsOGc2dlNGVFJsUVF5?= =?utf-8?B?RjlyYnMvM29vVEtwNk52K2Jjam1kd0xaRVEydHRjUTZUdGZFcU80bUh4N0pN?= =?utf-8?Q?oN57+u7O/lw=3D?= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:PH8PR12MB7277.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(366016)(376014)(7416014)(1800799024);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?ZjNTTGxNaURqalFyUmpsNlNNZEUvQS80TGxOYjFiMDFmMTdMRzliUVQrQUxm?= =?utf-8?B?ZmJEYzhMSmtQdld6VW90MFpBRHJCTFRVeVo3WEdpOTFBU3NJQko4MGJBck85?= =?utf-8?B?dmJtY1RTeEtCQ3haSlJnQzB5L1RFUXgvcW9ld0VUeTgzRHdJY3hFc3V6U09w?= =?utf-8?B?UnI3T3luZFk1cFUzOWZ1N0l3OHZ1MFhNRk1hcHhnZTN6NWJNbTc1Q2tGaU96?= =?utf-8?B?RnIrQWcwYzhSTWUxeGVENGdFMzViSlpITmhFZi90QURxRGVVL1N2L3ZHZ3J6?= =?utf-8?B?VG9aaVdyejY3b25iejdZVUR1NXhHSkhFSDM5S3JPQnhoTXFranBMS1pjYTNu?= =?utf-8?B?bTFaNU1IMEM2MHlXYUZWQm5NWHRiN2k1ckwyWFBZTjdWdFlDMHFZL0dNaVNi?= =?utf-8?B?QXc3djNvbHk2MGJhdHdOMm5XS2pkRHRlQnZVdXo5dWxvZjcwbm52R3B2SXd1?= =?utf-8?B?WGQ0NVJJbXZRZEdrazNkVkNUZHhiNnBBeXZuNUJWVzMzcDArMVgrZVB1d3Zh?= =?utf-8?B?WW5tZ0paMHpjMnBGUjFDYWxjdHBDeHFKeFNHWE55d1JrVVFVaWo2MFI3cmdI?= =?utf-8?B?c0N4bThjN0JoNUdBWkxZbjAyTlljbjBOSkZIV2xCZnJtSElzTDhCSFRaN2gy?= =?utf-8?B?bXMwVW1ETG1wcWNVRm9tSFlHMG9TckVwR1JkWG55L1RQbGhVMXY4TVNSWS9j?= =?utf-8?B?UXJYVE1yZ2pUWUpocWxRclIxVXl5WnZoNW5FWXlPRGE1U0xlY0tYa3NsMlo5?= =?utf-8?B?Yi9RSUg2bUdZbk5xL2E4Rkt0MkN2MnlZTTZxWS9EVTJkYmp4RGJZU3Q3aVgv?= =?utf-8?B?dUZpSWpXdXNHZEZCYXhZOXFLc2w4alNMMlViY2haZEhSOXczVFRFVHVzRWow?= =?utf-8?B?bk1EdDNIMVN2VFdLT05YOHphTU40U3AwWGg1T0R1d1llQnVFRkliRXhjWmRB?= =?utf-8?B?YzZQdG1JZGtEczd3WndlUFNzZVAzc2NWU0JhZzErL01wMnZWSldKVzFmVDJr?= =?utf-8?B?OGNWNitmbGdqbnhkVHRPMUZlNkxlaDZXRFlkRFFFVzFkeDEvb0NSdEo4ZkJF?= =?utf-8?B?ejltRzVnYmpZSitmUmdkbmVGd0wway9yNDlibTFJa0NQdEl5ZkkvSkI1OEho?= =?utf-8?B?RFFNK1dTMTQzV0VMNGZZRDFvT1BlRE5CTUdLY1VtaG1pOW51MnBNQnFGaVpP?= =?utf-8?B?cVE4djBsS3lGNWdkYzZmaEVNL1lXY2ZmbEZBZk91WElzOHVOeVZLUUVWVE1S?= =?utf-8?B?Sjd6Zjh0WW1pbDMwS2tFZm1jcFdNM1NsSzA5NlJtRlpYNm1RZlZ5UTRLSFM2?= =?utf-8?B?NE1LeGh5VDdqZnBFRE9MNWJ2RkhnZ2t1N2JMdWR6YmFQZ1VOZkdHL2x4dEky?= =?utf-8?B?MlF3NUN2WFViakw2Mk42VHB3THgzUk51V2R5dS9lWEpGNTliUUlEaVBaQklv?= =?utf-8?B?TkhWdjhqcW5DQlhVZzdiKytENmZsN25SbTBTZkNiTTNGOGxTc29nZkgzVXFm?= =?utf-8?B?MStrMEF0YWVkNGljQ25xMGlFNFZNMDEwZ2Jtb2dsNFprdXFtdjRFYzkvQU13?= =?utf-8?B?U0JLd2RrMkZZdkpwcWhCWkVHTjBLTjRjdXNxSWZqY01WWHE0ZUVxc1lxK1NW?= =?utf-8?B?OXFOVFNIdEJNZVR5cng1TGViOFRaUGtVUE5zVk5Ta09vVXNnNDJJUFZXbUZJ?= =?utf-8?B?cXlERFptSmNhaXZZUU9oeVFJWmhtbXFtd2JzODdDRWt6b3JwSHl2NUFMU09s?= =?utf-8?B?ajB0QXQ3aVlPZFlORUJKNXdDWDhkVWRFbjk4VEw0T1l1UmE4WEZjZUJQNDZR?= =?utf-8?B?NUpISWtLLzI3ZG1Kc0N4bVRobHUycElRUzlJVEFVV2ZCOFU5ZTJiQUI2RHRS?= =?utf-8?B?ZElyZDUwTXljUEsvcDB6N3RYTjk4dzM2Mm9kd1NIYmJMcU5yb1ZOMTRVbmd5?= =?utf-8?B?YlUwdU5YTGNERlE4NS8wMzQ3UEdPK1kxTWIyR2tNRmxlREdOVmdwRm9weFlW?= =?utf-8?B?eStzUGRoVUsydkRLSFRHNFZxU2RWd3RjaWFROFpmSThXMHZDT2FleWhqOVd1?= =?utf-8?B?c3JFOVRqcThXSFhqK0haUGYzTm93UFVTWWg5aktUbVY2ekIxM3NDcFFDZ2VS?= =?utf-8?Q?2ZDgbwY2cVipgQ6y0STfxab54?= X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: 9514fa28-5d3a-49d8-f660-08ddd949a4dd X-MS-Exchange-CrossTenant-AuthSource: PH8PR12MB7277.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 12 Aug 2025 02:40:48.1385 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: XXfexiCx4U7mnJBoY8xFxNgXX6p+OQ9y7M42Jj9WYbrv0om+gxXXPLQvJ9aw1TMCMamTCkE7URrrzyPPApWkPA== X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY8PR12MB7172 Make THP handling code in the mm subsystem for THP pages aware of zone device pages. Although the code is designed to be generic when it comes to handling splitting of pages, the code is designed to work for THP page sizes corresponding to HPAGE_PMD_NR. Modify page_vma_mapped_walk() to return true when a zone device huge entry is present, enabling try_to_migrate() and other code migration paths to appropriately process the entry. page_vma_mapped_walk() will return true for zone device private large folios only when PVMW_THP_DEVICE_PRIVATE is passed. This is to prevent locations that are not zone device private pages from having to add awareness. The key callback that needs this flag is try_to_migrate_one(). The other callbacks page idle, damon use it for setting young/dirty bits, which is not significant when it comes to pmd level bit harvesting. pmd_pfn() does not work well with zone device entries, use pfn_pmd_entry_to_swap() for checking and comparison as for zone device entries. Support partial unmapping of zone device private entries, which happens via munmap(). munmap() causes the device private entry pmd to be split, but the corresponding folio is not split. Deferred split does not work for zone device private folios due to the need to split during fault handling. Get migrate_vma_collect_pmd() to handle this case by splitting partially unmapped device private folios. Cc: Andrew Morton Cc: David Hildenbrand Cc: Zi Yan Cc: Joshua Hahn Cc: Rakie Kim Cc: Byungchul Park Cc: Gregory Price Cc: Ying Huang Cc: Alistair Popple Cc: Oscar Salvador Cc: Lorenzo Stoakes Cc: Baolin Wang Cc: "Liam R. Howlett" Cc: Nico Pache Cc: Ryan Roberts Cc: Dev Jain Cc: Barry Song Cc: Lyude Paul Cc: Danilo Krummrich Cc: David Airlie Cc: Simona Vetter Cc: Ralph Campbell Cc: Mika Penttil=C3=A4 Cc: Matthew Brost Cc: Francois Dugast Signed-off-by: Matthew Brost Signed-off-by: Balbir Singh --- include/linux/rmap.h | 2 + include/linux/swapops.h | 17 ++++ lib/test_hmm.c | 2 +- mm/huge_memory.c | 214 +++++++++++++++++++++++++++++++--------- mm/migrate_device.c | 47 +++++++++ mm/page_vma_mapped.c | 13 ++- mm/pgtable-generic.c | 6 ++ mm/rmap.c | 24 ++++- 8 files changed, 272 insertions(+), 53 deletions(-) diff --git a/include/linux/rmap.h b/include/linux/rmap.h index 6cd020eea37a..dfb7aae3d77b 100644 --- a/include/linux/rmap.h +++ b/include/linux/rmap.h @@ -927,6 +927,8 @@ struct page *make_device_exclusive(struct mm_struct *mm= , unsigned long addr, #define PVMW_SYNC (1 << 0) /* Look for migration entries rather than present PTEs */ #define PVMW_MIGRATION (1 << 1) +/* Look for device private THP entries */ +#define PVMW_THP_DEVICE_PRIVATE (1 << 2) =20 struct page_vma_mapped_walk { unsigned long pfn; diff --git a/include/linux/swapops.h b/include/linux/swapops.h index 64ea151a7ae3..2641c01bd5d2 100644 --- a/include/linux/swapops.h +++ b/include/linux/swapops.h @@ -563,6 +563,7 @@ static inline int is_pmd_migration_entry(pmd_t pmd) { return is_swap_pmd(pmd) && is_migration_entry(pmd_to_swp_entry(pmd)); } + #else /* CONFIG_ARCH_ENABLE_THP_MIGRATION */ static inline int set_pmd_migration_entry(struct page_vma_mapped_walk *pvm= w, struct page *page) @@ -594,6 +595,22 @@ static inline int is_pmd_migration_entry(pmd_t pmd) } #endif /* CONFIG_ARCH_ENABLE_THP_MIGRATION */ =20 +#if defined(CONFIG_ZONE_DEVICE) && defined(CONFIG_ARCH_ENABLE_THP_MIGRATIO= N) + +static inline int is_pmd_device_private_entry(pmd_t pmd) +{ + return is_swap_pmd(pmd) && is_device_private_entry(pmd_to_swp_entry(pmd)); +} + +#else /* CONFIG_ZONE_DEVICE && CONFIG_ARCH_ENABLE_THP_MIGRATION */ + +static inline int is_pmd_device_private_entry(pmd_t pmd) +{ + return 0; +} + +#endif /* CONFIG_ZONE_DEVICE && CONFIG_ARCH_ENABLE_THP_MIGRATION */ + static inline int non_swap_entry(swp_entry_t entry) { return swp_type(entry) >=3D MAX_SWAPFILES; diff --git a/lib/test_hmm.c b/lib/test_hmm.c index 761725bc713c..297f1e034045 100644 --- a/lib/test_hmm.c +++ b/lib/test_hmm.c @@ -1408,7 +1408,7 @@ static vm_fault_t dmirror_devmem_fault(struct vm_faul= t *vmf) * the mirror but here we use it to hold the page for the simulated * device memory and that page holds the pointer to the mirror. */ - rpage =3D vmf->page->zone_device_data; + rpage =3D folio_page(page_folio(vmf->page), 0)->zone_device_data; dmirror =3D rpage->zone_device_data; =20 /* FIXME demonstrate how we can adjust migrate range */ diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 9c38a95e9f09..2495e3fdbfae 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -1711,8 +1711,11 @@ int copy_huge_pmd(struct mm_struct *dst_mm, struct m= m_struct *src_mm, if (unlikely(is_swap_pmd(pmd))) { swp_entry_t entry =3D pmd_to_swp_entry(pmd); =20 - VM_BUG_ON(!is_pmd_migration_entry(pmd)); - if (!is_readable_migration_entry(entry)) { + VM_WARN_ON(!is_pmd_migration_entry(pmd) && + !is_pmd_device_private_entry(pmd)); + + if (is_migration_entry(entry) && + is_writable_migration_entry(entry)) { entry =3D make_readable_migration_entry( swp_offset(entry)); pmd =3D swp_entry_to_pmd(entry); @@ -1722,6 +1725,32 @@ int copy_huge_pmd(struct mm_struct *dst_mm, struct m= m_struct *src_mm, pmd =3D pmd_swp_mkuffd_wp(pmd); set_pmd_at(src_mm, addr, src_pmd, pmd); } + + if (is_device_private_entry(entry)) { + if (is_writable_device_private_entry(entry)) { + entry =3D make_readable_device_private_entry( + swp_offset(entry)); + pmd =3D swp_entry_to_pmd(entry); + + if (pmd_swp_soft_dirty(*src_pmd)) + pmd =3D pmd_swp_mksoft_dirty(pmd); + if (pmd_swp_uffd_wp(*src_pmd)) + pmd =3D pmd_swp_mkuffd_wp(pmd); + set_pmd_at(src_mm, addr, src_pmd, pmd); + } + + src_folio =3D pfn_swap_entry_folio(entry); + VM_WARN_ON(!folio_test_large(src_folio)); + + folio_get(src_folio); + /* + * folio_try_dup_anon_rmap_pmd does not fail for + * device private entries. + */ + VM_WARN_ON(folio_try_dup_anon_rmap_pmd(src_folio, + &src_folio->page, dst_vma, src_vma)); + } + add_mm_counter(dst_mm, MM_ANONPAGES, HPAGE_PMD_NR); mm_inc_nr_ptes(dst_mm); pgtable_trans_huge_deposit(dst_mm, dst_pmd, pgtable); @@ -2219,15 +2248,22 @@ int zap_huge_pmd(struct mmu_gather *tlb, struct vm_= area_struct *vma, folio_remove_rmap_pmd(folio, page, vma); WARN_ON_ONCE(folio_mapcount(folio) < 0); VM_BUG_ON_PAGE(!PageHead(page), page); - } else if (thp_migration_supported()) { + } else if (is_pmd_migration_entry(orig_pmd) || + is_pmd_device_private_entry(orig_pmd)) { swp_entry_t entry; =20 - VM_BUG_ON(!is_pmd_migration_entry(orig_pmd)); entry =3D pmd_to_swp_entry(orig_pmd); folio =3D pfn_swap_entry_folio(entry); flush_needed =3D 0; - } else - WARN_ONCE(1, "Non present huge pmd without pmd migration enabled!"); + + if (!thp_migration_supported()) + WARN_ONCE(1, "Non present huge pmd without pmd migration enabled!"); + + if (is_pmd_device_private_entry(orig_pmd)) { + folio_remove_rmap_pmd(folio, &folio->page, vma); + WARN_ON_ONCE(folio_mapcount(folio) < 0); + } + } =20 if (folio_test_anon(folio)) { zap_deposited_table(tlb->mm, pmd); @@ -2247,6 +2283,15 @@ int zap_huge_pmd(struct mmu_gather *tlb, struct vm_a= rea_struct *vma, folio_mark_accessed(folio); } =20 + /* + * Do a folio put on zone device private pages after + * changes to mm_counter, because the folio_put() will + * clean folio->mapping and the folio_test_anon() check + * will not be usable. + */ + if (folio_is_device_private(folio)) + folio_put(folio); + spin_unlock(ptl); if (flush_needed) tlb_remove_page_size(tlb, &folio->page, HPAGE_PMD_SIZE); @@ -2375,7 +2420,8 @@ int change_huge_pmd(struct mmu_gather *tlb, struct vm= _area_struct *vma, struct folio *folio =3D pfn_swap_entry_folio(entry); pmd_t newpmd; =20 - VM_BUG_ON(!is_pmd_migration_entry(*pmd)); + VM_WARN_ON(!is_pmd_migration_entry(*pmd) && + !folio_is_device_private(folio)); if (is_writable_migration_entry(entry)) { /* * A protection check is difficult so @@ -2388,6 +2434,10 @@ int change_huge_pmd(struct mmu_gather *tlb, struct v= m_area_struct *vma, newpmd =3D swp_entry_to_pmd(entry); if (pmd_swp_soft_dirty(*pmd)) newpmd =3D pmd_swp_mksoft_dirty(newpmd); + } else if (is_writable_device_private_entry(entry)) { + entry =3D make_readable_device_private_entry( + swp_offset(entry)); + newpmd =3D swp_entry_to_pmd(entry); } else { newpmd =3D *pmd; } @@ -2842,16 +2892,19 @@ static void __split_huge_pmd_locked(struct vm_area_= struct *vma, pmd_t *pmd, struct page *page; pgtable_t pgtable; pmd_t old_pmd, _pmd; - bool young, write, soft_dirty, pmd_migration =3D false, uffd_wp =3D false; - bool anon_exclusive =3D false, dirty =3D false; + bool young, write, soft_dirty, uffd_wp =3D false; + bool anon_exclusive =3D false, dirty =3D false, present =3D false; unsigned long addr; pte_t *pte; int i; + swp_entry_t swp_entry; =20 VM_BUG_ON(haddr & ~HPAGE_PMD_MASK); VM_BUG_ON_VMA(vma->vm_start > haddr, vma); VM_BUG_ON_VMA(vma->vm_end < haddr + HPAGE_PMD_SIZE, vma); - VM_BUG_ON(!is_pmd_migration_entry(*pmd) && !pmd_trans_huge(*pmd)); + + VM_WARN_ON(!is_pmd_migration_entry(*pmd) && !pmd_trans_huge(*pmd) + && !(is_pmd_device_private_entry(*pmd))); =20 count_vm_event(THP_SPLIT_PMD); =20 @@ -2899,18 +2952,45 @@ static void __split_huge_pmd_locked(struct vm_area_= struct *vma, pmd_t *pmd, return __split_huge_zero_page_pmd(vma, haddr, pmd); } =20 - pmd_migration =3D is_pmd_migration_entry(*pmd); - if (unlikely(pmd_migration)) { - swp_entry_t entry; =20 + present =3D pmd_present(*pmd); + if (unlikely(!present)) { + swp_entry =3D pmd_to_swp_entry(*pmd); old_pmd =3D *pmd; - entry =3D pmd_to_swp_entry(old_pmd); - page =3D pfn_swap_entry_to_page(entry); - write =3D is_writable_migration_entry(entry); - if (PageAnon(page)) - anon_exclusive =3D is_readable_exclusive_migration_entry(entry); - young =3D is_migration_entry_young(entry); - dirty =3D is_migration_entry_dirty(entry); + + folio =3D pfn_swap_entry_folio(swp_entry); + VM_WARN_ON(!is_migration_entry(swp_entry) && + !is_device_private_entry(swp_entry)); + page =3D pfn_swap_entry_to_page(swp_entry); + + if (is_pmd_migration_entry(old_pmd)) { + write =3D is_writable_migration_entry(swp_entry); + if (PageAnon(page)) + anon_exclusive =3D + is_readable_exclusive_migration_entry( + swp_entry); + young =3D is_migration_entry_young(swp_entry); + dirty =3D is_migration_entry_dirty(swp_entry); + } else if (is_pmd_device_private_entry(old_pmd)) { + write =3D is_writable_device_private_entry(swp_entry); + anon_exclusive =3D PageAnonExclusive(page); + if (freeze && anon_exclusive && + folio_try_share_anon_rmap_pmd(folio, page)) + freeze =3D false; + if (!freeze) { + rmap_t rmap_flags =3D RMAP_NONE; + + if (anon_exclusive) + rmap_flags |=3D RMAP_EXCLUSIVE; + + folio_ref_add(folio, HPAGE_PMD_NR - 1); + if (anon_exclusive) + rmap_flags |=3D RMAP_EXCLUSIVE; + folio_add_anon_rmap_ptes(folio, page, HPAGE_PMD_NR, + vma, haddr, rmap_flags); + } + } + soft_dirty =3D pmd_swp_soft_dirty(old_pmd); uffd_wp =3D pmd_swp_uffd_wp(old_pmd); } else { @@ -2996,30 +3076,49 @@ static void __split_huge_pmd_locked(struct vm_area_= struct *vma, pmd_t *pmd, * Note that NUMA hinting access restrictions are not transferred to * avoid any possibility of altering permissions across VMAs. */ - if (freeze || pmd_migration) { + if (freeze || !present) { for (i =3D 0, addr =3D haddr; i < HPAGE_PMD_NR; i++, addr +=3D PAGE_SIZE= ) { pte_t entry; - swp_entry_t swp_entry; - - if (write) - swp_entry =3D make_writable_migration_entry( - page_to_pfn(page + i)); - else if (anon_exclusive) - swp_entry =3D make_readable_exclusive_migration_entry( - page_to_pfn(page + i)); - else - swp_entry =3D make_readable_migration_entry( - page_to_pfn(page + i)); - if (young) - swp_entry =3D make_migration_entry_young(swp_entry); - if (dirty) - swp_entry =3D make_migration_entry_dirty(swp_entry); - entry =3D swp_entry_to_pte(swp_entry); - if (soft_dirty) - entry =3D pte_swp_mksoft_dirty(entry); - if (uffd_wp) - entry =3D pte_swp_mkuffd_wp(entry); - + if (freeze || is_migration_entry(swp_entry)) { + if (write) + swp_entry =3D make_writable_migration_entry( + page_to_pfn(page + i)); + else if (anon_exclusive) + swp_entry =3D make_readable_exclusive_migration_entry( + page_to_pfn(page + i)); + else + swp_entry =3D make_readable_migration_entry( + page_to_pfn(page + i)); + if (young) + swp_entry =3D make_migration_entry_young(swp_entry); + if (dirty) + swp_entry =3D make_migration_entry_dirty(swp_entry); + entry =3D swp_entry_to_pte(swp_entry); + if (soft_dirty) + entry =3D pte_swp_mksoft_dirty(entry); + if (uffd_wp) + entry =3D pte_swp_mkuffd_wp(entry); + } else { + /* + * anon_exclusive was already propagated to the relevant + * pages corresponding to the pte entries when freeze + * is false. + */ + if (write) + swp_entry =3D make_writable_device_private_entry( + page_to_pfn(page + i)); + else + swp_entry =3D make_readable_device_private_entry( + page_to_pfn(page + i)); + /* + * Young and dirty bits are not progated via swp_entry + */ + entry =3D swp_entry_to_pte(swp_entry); + if (soft_dirty) + entry =3D pte_swp_mksoft_dirty(entry); + if (uffd_wp) + entry =3D pte_swp_mkuffd_wp(entry); + } VM_WARN_ON(!pte_none(ptep_get(pte + i))); set_pte_at(mm, addr, pte + i, entry); } @@ -3046,7 +3145,7 @@ static void __split_huge_pmd_locked(struct vm_area_st= ruct *vma, pmd_t *pmd, } pte_unmap(pte); =20 - if (!pmd_migration) + if (present) folio_remove_rmap_pmd(folio, page, vma); if (freeze) put_page(page); @@ -3058,8 +3157,10 @@ static void __split_huge_pmd_locked(struct vm_area_s= truct *vma, pmd_t *pmd, void split_huge_pmd_locked(struct vm_area_struct *vma, unsigned long addre= ss, pmd_t *pmd, bool freeze) { + VM_WARN_ON_ONCE(!IS_ALIGNED(address, HPAGE_PMD_SIZE)); - if (pmd_trans_huge(*pmd) || is_pmd_migration_entry(*pmd)) + if (pmd_trans_huge(*pmd) || is_pmd_migration_entry(*pmd) || + (is_pmd_device_private_entry(*pmd))) __split_huge_pmd_locked(vma, pmd, address, freeze); } =20 @@ -3238,6 +3339,9 @@ static void lru_add_split_folio(struct folio *folio, = struct folio *new_folio, VM_BUG_ON_FOLIO(folio_test_lru(new_folio), folio); lockdep_assert_held(&lruvec->lru_lock); =20 + if (folio_is_device_private(folio)) + return; + if (list) { /* page reclaim is reclaiming a huge page */ VM_WARN_ON(folio_test_lru(folio)); @@ -3252,6 +3356,7 @@ static void lru_add_split_folio(struct folio *folio, = struct folio *new_folio, list_add_tail(&new_folio->lru, &folio->lru); folio_set_lru(new_folio); } + } =20 /* Racy check whether the huge page can be split */ @@ -3727,7 +3832,7 @@ static int __folio_split(struct folio *folio, unsigne= d int new_order, =20 /* Prevent deferred_split_scan() touching ->_refcount */ spin_lock(&ds_queue->split_queue_lock); - if (folio_ref_freeze(folio, 1 + extra_pins)) { + if (folio_ref_freeze(folio, 1 + folio_expected_ref_count(folio))) { struct address_space *swap_cache =3D NULL; struct lruvec *lruvec; int expected_refs; @@ -3858,8 +3963,9 @@ static int __folio_split(struct folio *folio, unsigne= d int new_order, if (nr_shmem_dropped) shmem_uncharge(mapping->host, nr_shmem_dropped); =20 - if (!ret && is_anon) + if (!ret && is_anon && !folio_is_device_private(folio)) remap_flags =3D RMP_USE_SHARED_ZEROPAGE; + remap_page(folio, 1 << order, remap_flags); =20 /* @@ -4603,7 +4709,10 @@ int set_pmd_migration_entry(struct page_vma_mapped_w= alk *pvmw, return 0; =20 flush_cache_range(vma, address, address + HPAGE_PMD_SIZE); - pmdval =3D pmdp_invalidate(vma, address, pvmw->pmd); + if (unlikely(is_pmd_device_private_entry(*pvmw->pmd))) + pmdval =3D pmdp_huge_clear_flush(vma, address, pvmw->pmd); + else + pmdval =3D pmdp_invalidate(vma, address, pvmw->pmd); =20 /* See folio_try_share_anon_rmap_pmd(): invalidate PMD first. */ anon_exclusive =3D folio_test_anon(folio) && PageAnonExclusive(page); @@ -4653,6 +4762,17 @@ void remove_migration_pmd(struct page_vma_mapped_wal= k *pvmw, struct page *new) entry =3D pmd_to_swp_entry(*pvmw->pmd); folio_get(folio); pmde =3D folio_mk_pmd(folio, READ_ONCE(vma->vm_page_prot)); + + if (folio_is_device_private(folio)) { + if (pmd_write(pmde)) + entry =3D make_writable_device_private_entry( + page_to_pfn(new)); + else + entry =3D make_readable_device_private_entry( + page_to_pfn(new)); + pmde =3D swp_entry_to_pmd(entry); + } + if (pmd_swp_soft_dirty(*pvmw->pmd)) pmde =3D pmd_mksoft_dirty(pmde); if (is_writable_migration_entry(entry)) diff --git a/mm/migrate_device.c b/mm/migrate_device.c index e05e14d6eacd..0ed337f94fcd 100644 --- a/mm/migrate_device.c +++ b/mm/migrate_device.c @@ -136,6 +136,8 @@ static int migrate_vma_collect_pmd(pmd_t *pmdp, * page table entry. Other special swap entries are not * migratable, and we ignore regular swapped page. */ + struct folio *folio; + entry =3D pte_to_swp_entry(pte); if (!is_device_private_entry(entry)) goto next; @@ -147,6 +149,51 @@ static int migrate_vma_collect_pmd(pmd_t *pmdp, pgmap->owner !=3D migrate->pgmap_owner) goto next; =20 + folio =3D page_folio(page); + if (folio_test_large(folio)) { + struct folio *new_folio; + struct folio *new_fault_folio =3D NULL; + + /* + * The reason for finding pmd present with a + * device private pte and a large folio for the + * pte is partial unmaps. Split the folio now + * for the migration to be handled correctly + */ + pte_unmap_unlock(ptep, ptl); + + folio_get(folio); + if (folio !=3D fault_folio) + folio_lock(folio); + if (split_folio(folio)) { + if (folio !=3D fault_folio) + folio_unlock(folio); + ptep =3D pte_offset_map_lock(mm, pmdp, addr, &ptl); + goto next; + } + + new_folio =3D page_folio(page); + if (fault_folio) + new_fault_folio =3D page_folio(migrate->fault_page); + + /* + * Ensure the lock is held on the correct + * folio after the split + */ + if (!new_fault_folio) { + folio_unlock(folio); + folio_put(folio); + } else if (folio !=3D new_fault_folio) { + folio_get(new_fault_folio); + folio_lock(new_fault_folio); + folio_unlock(folio); + folio_put(folio); + } + + addr =3D start; + goto again; + } + mpfn =3D migrate_pfn(page_to_pfn(page)) | MIGRATE_PFN_MIGRATE; if (is_writable_device_private_entry(entry)) diff --git a/mm/page_vma_mapped.c b/mm/page_vma_mapped.c index e981a1a292d2..246e6c211f34 100644 --- a/mm/page_vma_mapped.c +++ b/mm/page_vma_mapped.c @@ -250,12 +250,11 @@ bool page_vma_mapped_walk(struct page_vma_mapped_walk= *pvmw) pvmw->ptl =3D pmd_lock(mm, pvmw->pmd); pmde =3D *pvmw->pmd; if (!pmd_present(pmde)) { - swp_entry_t entry; + swp_entry_t entry =3D pmd_to_swp_entry(pmde); =20 if (!thp_migration_supported() || !(pvmw->flags & PVMW_MIGRATION)) return not_found(pvmw); - entry =3D pmd_to_swp_entry(pmde); if (!is_migration_entry(entry) || !check_pmd(swp_offset_pfn(entry), pvmw)) return not_found(pvmw); @@ -277,6 +276,16 @@ bool page_vma_mapped_walk(struct page_vma_mapped_walk = *pvmw) * cannot return prematurely, while zap_huge_pmd() has * cleared *pmd but not decremented compound_mapcount(). */ + swp_entry_t entry; + + entry =3D pmd_to_swp_entry(pmde); + + if (is_device_private_entry(entry) && + (pvmw->flags & PVMW_THP_DEVICE_PRIVATE)) { + pvmw->ptl =3D pmd_lock(mm, pvmw->pmd); + return true; + } + if ((pvmw->flags & PVMW_SYNC) && thp_vma_suitable_order(vma, pvmw->address, PMD_ORDER) && diff --git a/mm/pgtable-generic.c b/mm/pgtable-generic.c index 567e2d084071..604e8206a2ec 100644 --- a/mm/pgtable-generic.c +++ b/mm/pgtable-generic.c @@ -292,6 +292,12 @@ pte_t *___pte_offset_map(pmd_t *pmd, unsigned long add= r, pmd_t *pmdvalp) *pmdvalp =3D pmdval; if (unlikely(pmd_none(pmdval) || is_pmd_migration_entry(pmdval))) goto nomap; + if (is_swap_pmd(pmdval)) { + swp_entry_t entry =3D pmd_to_swp_entry(pmdval); + + if (is_device_private_entry(entry)) + goto nomap; + } if (unlikely(pmd_trans_huge(pmdval))) goto nomap; if (unlikely(pmd_bad(pmdval))) { diff --git a/mm/rmap.c b/mm/rmap.c index b5837075b6e0..f40e45564295 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -2285,7 +2285,8 @@ static bool try_to_migrate_one(struct folio *folio, s= truct vm_area_struct *vma, unsigned long address, void *arg) { struct mm_struct *mm =3D vma->vm_mm; - DEFINE_FOLIO_VMA_WALK(pvmw, folio, vma, address, 0); + DEFINE_FOLIO_VMA_WALK(pvmw, folio, vma, address, + PVMW_THP_DEVICE_PRIVATE); bool anon_exclusive, writable, ret =3D true; pte_t pteval; struct page *subpage; @@ -2330,6 +2331,10 @@ static bool try_to_migrate_one(struct folio *folio, = struct vm_area_struct *vma, while (page_vma_mapped_walk(&pvmw)) { /* PMD-mapped THP migration entry */ if (!pvmw.pte) { +#ifdef CONFIG_ARCH_ENABLE_THP_MIGRATION + unsigned long pfn; +#endif + if (flags & TTU_SPLIT_HUGE_PMD) { split_huge_pmd_locked(vma, pvmw.address, pvmw.pmd, true); @@ -2338,8 +2343,21 @@ static bool try_to_migrate_one(struct folio *folio, = struct vm_area_struct *vma, break; } #ifdef CONFIG_ARCH_ENABLE_THP_MIGRATION - subpage =3D folio_page(folio, - pmd_pfn(*pvmw.pmd) - folio_pfn(folio)); + /* + * Zone device private folios do not work well with + * pmd_pfn() on some architectures due to pte + * inversion. + */ + if (is_pmd_device_private_entry(*pvmw.pmd)) { + swp_entry_t entry =3D pmd_to_swp_entry(*pvmw.pmd); + + pfn =3D swp_offset_pfn(entry); + } else { + pfn =3D pmd_pfn(*pvmw.pmd); + } + + subpage =3D folio_page(folio, pfn - folio_pfn(folio)); + VM_BUG_ON_FOLIO(folio_test_hugetlb(folio) || !folio_test_pmd_mappable(folio), folio); =20 --=20 2.50.1 From nobody Sat Oct 4 22:37:10 2025 Received: from NAM11-DM6-obe.outbound.protection.outlook.com (mail-dm6nam11on2047.outbound.protection.outlook.com [40.107.223.47]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 16E112E370E for ; Tue, 12 Aug 2025 02:40:58 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.223.47 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1754966462; cv=fail; b=j2Iql4toiP1HohGCGJ1bPwQe4WXf26/ZRNOS/b6vuwGJ1URBSRNlMPCpkEUdK7lGBChTkALQmBmy3JJmYCas2JCiHJByzP3m57oAuG7luD0lrxosHzU4LIN7ZlEPQR5kH94931BoOYZovR2Buu/4iqUMpz9alORtQ/xEIydDn1w= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1754966462; c=relaxed/simple; bh=RjqIEL6gG7epz32d0AfPwwU44+kjjz+F0r6Dy7KzanA=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: Content-Type:MIME-Version; b=uwEMoXpWm9iPSGyb2oOkI6Y8QZTQnSO3xmFSlvtRZsqG218231DWQEGuCDH/NqMpqGdlyixQ7y+cyLiROc+vXnUYwWWTq/yWO7jmK7SlUwaDx9AebAEbFshiA4oabTzofygzI1/zXBgJcgaTVpUvGcuyvEQcGS7p9efk4Z5GqQM= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com; spf=fail smtp.mailfrom=nvidia.com; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b=QjXM7gPH; arc=fail smtp.client-ip=40.107.223.47 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="QjXM7gPH" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=vQRTEtwOfvDLSPxgBJWNjeXdEYkKBneEfFfjoOs3S+5Xn1pjoJ5AnjIS7T3U67ApfPzSF8/7riTDt2rz1TUhGAbGKwoOpi3RP3KxpbXi5CTrhoVHL8S2XhwlzrtyFpqKY7KGQ24luLESZYoXcfwKhX8PN6FLkRfY9dnUmn/Ckk8cLqHjJd5Ls0gmPt0fiefeK235eizYVeY8gJ6pcE0IDyyLGwPNru6mW4tEuLhSBoNvWkAcsgE7VSUMDjuzZTN8BrXE/vwYF5oQXa2epkyFUeBp1GT8qla/UQ3bDRJoif0/a4GyvTrvfWJsBkmI6MmQgQU36Yvaf7pcFnhhkZhZSg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=JFkFwadZtXjcY5QZa6F74BJGKb3PmXhFv7ISIxUPz2g=; b=D5HSl0hS6VdWqPkXDj/lOF07SUK3kuXsuzvkw5o6L9R6mwGyxe0jX2CwXpq+gEQr6nG2miTTW02R1sIhW/aycsSrt4hkfoiDOlkxDiGxd/0Zp3vTgxNsQQDd3HL295TD0e39Ym9d6uIG5lbu7kiAtQ6Lfdt+WGBxue3La1ss7JkPsd6Cs2NcZ5HR15rTkOs//vV/NyM3+hqU/3nGjWdNYWZ+Ka2vGePhMGeKx3jnbxsYA+51t2HMQRactA8pgP/8dTjaSy8q2UTbK8b+uWFqprruhgwTpSd/7ZM0VkaqNFG9Sjl2pPptWjm2tAYkEmg5GsOTAGen+RepBVaoEEkiqg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=JFkFwadZtXjcY5QZa6F74BJGKb3PmXhFv7ISIxUPz2g=; b=QjXM7gPHrtJhmLmJgaajtPHozm7xyUYxsgkeNc2oCkfbslZema9cmHKB/Yp8bbbewzUippfxVv+U4PMSoaG+lMKrAoDY8shwvFZK8VwNihwJuVsvhyL0p/WQrDrIV478TKxCP4KkheJPlYw9i6X51k2gS2wGQtzi3GsvhRNm7WnXn9dbk3n+PHk5nicKKJ/XmNsmXlUzuVU4Jfk57buA4NLZMHRO8JWskw1zyQPDRnEBx2jBGdgkwXhnh1itW8cb4k+oWBVKH1ko+Zm5X9vHY87O4YOXq70z+fissVDxj29JirZKAM0qIOmbZr6sRW2UwsJxvIRhRyVNedh2c9cRMg== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from PH8PR12MB7277.namprd12.prod.outlook.com (2603:10b6:510:223::13) by CY8PR12MB7172.namprd12.prod.outlook.com (2603:10b6:930:5b::7) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9009.18; Tue, 12 Aug 2025 02:40:52 +0000 Received: from PH8PR12MB7277.namprd12.prod.outlook.com ([fe80::3a4:70ea:ff05:1251]) by PH8PR12MB7277.namprd12.prod.outlook.com ([fe80::3a4:70ea:ff05:1251%5]) with mapi id 15.20.9009.018; Tue, 12 Aug 2025 02:40:52 +0000 From: Balbir Singh To: dri-devel@lists.freedesktop.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Balbir Singh , Andrew Morton , David Hildenbrand , Zi Yan , Joshua Hahn , Rakie Kim , Byungchul Park , Gregory Price , Ying Huang , Alistair Popple , Oscar Salvador , Lorenzo Stoakes , Baolin Wang , "Liam R. Howlett" , Nico Pache , Ryan Roberts , Dev Jain , Barry Song , Lyude Paul , Danilo Krummrich , David Airlie , Simona Vetter , Ralph Campbell , =?UTF-8?q?Mika=20Penttil=C3=A4?= , Matthew Brost , Francois Dugast Subject: [v3 03/11] mm/migrate_device: THP migration of zone device pages Date: Tue, 12 Aug 2025 12:40:28 +1000 Message-ID: <20250812024036.690064-4-balbirs@nvidia.com> X-Mailer: git-send-email 2.50.1 In-Reply-To: <20250812024036.690064-1-balbirs@nvidia.com> References: <20250812024036.690064-1-balbirs@nvidia.com> Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: SY6PR01CA0158.ausprd01.prod.outlook.com (2603:10c6:10:1ba::6) To PH8PR12MB7277.namprd12.prod.outlook.com (2603:10b6:510:223::13) Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: PH8PR12MB7277:EE_|CY8PR12MB7172:EE_ X-MS-Office365-Filtering-Correlation-Id: c486f5a6-1cd9-49d1-54f1-08ddd949a70d X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|366016|376014|7416014|1800799024; X-Microsoft-Antispam-Message-Info: =?utf-8?B?clFuNDdqOHB1bG45alQwcDgwTlpPODQ2OGNJeU1sYnpQdHk4YlJzMzBaNTdZ?= =?utf-8?B?b3oyQVJpejRNQlVvTFpaZkQwWHdQMlpqUGE4NDVXME5OTk1EN2p4R0ZtbTdW?= =?utf-8?B?Zm5rSTJQY2gxQWlrc21tY3JjRkllb2VEOC9BTlZoWVdVM0NRdklZa1JrYjlZ?= =?utf-8?B?R2JyQnFOd2hSa0JaMm9QMzJpUXdGTUE4WlhxU3RCeVI1a1ZpWE5iVmdqQ2I0?= =?utf-8?B?SXA1NGVUYUNiVFFaci9kT2RwV2dKS05CTkVwcG50VXZJRlhDUDZhckNxOTNw?= =?utf-8?B?TkxSQm5mQkxnL3pJZlhOUTdJYnlub2Q4MFkwZkJVcHRTRlpTQ3FCcFZqQWVM?= =?utf-8?B?cys0bENTOUR0UXlpdDFFeUhsYUdtbStmdXpsTjFRQ2JuRHBUU1JQWUtaU29Z?= =?utf-8?B?eDhkRnA2dU4rSVdRSzdPbWZheHZOUmdHMlVsSmNycFhWRW5USnExcGNsdDY2?= =?utf-8?B?TG50NmJVNXRhZm5oK1RiQkJITnEyWUtKTzl0OW4xRElzSjhSNE9jUjFYNU43?= =?utf-8?B?UFNmeG5BZzYwWllpc3ZNb0FMLzJOeXNLMklESU05dXQ2NkxXSUtkb2p3THQ3?= =?utf-8?B?M01kdFFicWg2NkNVeE15aFdiTGxPazNvVVg3N1NGeHVjc3VqaXVLSTdybGFJ?= =?utf-8?B?Y3lscmpXSytwcHFNUUNNMUIwaWs0Y1dNL0JwQys1cWYrZ0VLaHVpY2s0QTJy?= =?utf-8?B?YkYwZWlTUDk4TkRNZWNmaUFJYkxuR2dRcGZUZ1RLc0dCN3Brdk5ETGxmcS9n?= =?utf-8?B?a0tXcWkxSTdGMEV6Y2J5NWprcy9JREIvczE1eG5sdFNONjBzcXl1S1h6dmVJ?= =?utf-8?B?OEF0VStqSGI0akxFWVFJbXFSbjI2YzZSSnJVR0RrN2ZwZFoyOEp5TURESkc2?= =?utf-8?B?TmcrWlByT0pNMHEwemUyN3JqTURwdGNCSGcydU5wZVZQSTd6cng0OVROV3gx?= =?utf-8?B?bHVsbFdBek0wemJXeDdJRmt4WGpXRXh2SXhLbUpMZ1Y5UlRSZnRhMmRxdmNQ?= =?utf-8?B?N0pxQlAyc0hYNkRlU2EwWjVtd3Vyd3E1YlF2d2xTblAwUVkxYXBadnQzMkFO?= =?utf-8?B?RzUzbWhTY1RTRnUzM2RtSHZrazhEL3ROWGxlN2J4WFJCNFM4OEV1b0pMNDVZ?= =?utf-8?B?SlE2clcvSEhDcEpHQ21YUXNtTXAxSkdPR3FWanpRK1VSckI1LzgyaDE5cFVH?= =?utf-8?B?MG10a0NTQUpjQjkyUVROVms0VzlkY051K2ppc0FpaXBCc2NQMUQ5N2VNaXlY?= =?utf-8?B?R1l5dVlIYmswWlQ3VGY0MHFLdEcrQjJ3OFZic29kOXlkQnVNb3ErYWk4cER0?= =?utf-8?B?VTZ6NU0xT1V1eXFhTnZ2TFdHbXhna1dPZGlUWmVBSFRoV1o2VytlQWNVeVNk?= =?utf-8?B?emFXUjB3NFNmb0l3d0tUSkJZMFpSR2d1ZWJ0Zk5sMzJMYm1LMEtJRk5tRG53?= =?utf-8?B?VXJtUy9vRGNvVW5lU25Qd0U3VXk1cmVEYzdzU3JKKzlQZjdRZ1Juem9MU2dY?= =?utf-8?B?NmhZeWo4ajlVUUJkdlFsWW5LeDAvZVhDaHlTYjFvSWlpTjZpbG1CRVJoTFNv?= =?utf-8?B?OXNTU2F0Q2l5S2l1eFIzbzJmMW5oVVI4c2lIaEQzV0Q4ZUJVRHJsMHVpcDc5?= =?utf-8?B?R0kwa2pKUk5iUENLamNMaGIybXBXd1pRU2g3Mmo4ZHh0cWtiZmFONlRCclNZ?= =?utf-8?B?NlpNcFEwYnoyekh2NVlMaWNpb1F0SjJsZVZ0Q3Q1V0ZNVVU3ZUpaWlRFWEJh?= =?utf-8?B?SVp5NHB1azFwZTRRMTBiZkZSS1d3V0tZcTVVbWNCVk9RcGlTZXBmQWFEdWd3?= =?utf-8?B?VmtRMnd5VUJpMlRRbmZFM0VDN0xmcFhscE93L2wzWVlDSERYS1BHcG5PenpE?= =?utf-8?B?L1FKcUtMYVA3czJDOXhES2VUZ2U1UFBIeUxYeWFpRWNPZ0RZamZ0YmowbThQ?= =?utf-8?Q?/lHEYPCdKME=3D?= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:PH8PR12MB7277.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(366016)(376014)(7416014)(1800799024);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?K09ReWlNdTF3dGJ0Tmk2bWh6SkR5aVBWUU03dWhkREVuU0s5QzVFMnN4cmpm?= =?utf-8?B?MmZnSEdhZXNIM3NaTmNvT3dhWFlXRGZ0R0dwZmhLYnVzRys1NjkyTDh5RmJl?= =?utf-8?B?aXowWTY1OHdGbXRuQW5hMWdDVitEUGRrM1g3SHdqYWF6ZWszcGFTVTVRU3Bk?= =?utf-8?B?MDZzbElYT2MzaG9hWXJKTk0vMVYwMjVPbGoyRFI0RUtjVnZDVmltYTJGbUh6?= =?utf-8?B?N3BFbzI0U2l0L3NDZnZKbGYxVVkwaGhqb25CMXU1WXRBOEVvamJaMG5DaXBO?= =?utf-8?B?OHM0SFQ4OWJaN2Frc2xDODhKczhjU3ZUZ3dTS0RaRGZMOVQ0Q3JjV256T1FE?= =?utf-8?B?WE1qdXVCRFd0V2QwVzk1R2ZuTHBoaUNsOXRQRFBkaG1maGpSdmxLUUg5amNS?= =?utf-8?B?cU5TSzRZZ0V1a1NvTlJBTTZKR0JzZENUUm0zaVBSSndoN1J1b3RlQnE1VEhM?= =?utf-8?B?VndkVHBGRkphMDZzODNZajFCdmQvSXo2MUVHS3F1em9vMFkyZ3NlNFd4cUg2?= =?utf-8?B?NlZuTnZoTVI4cmRsbHhQWWFsU2o5MWUxeWRsQ1dvTG4zb3BVNUJONzI2NGJC?= =?utf-8?B?bklrTzRJOGhza21tTHhGTVBlUUpFZ1ZQekNqMFp5WUVSdmpXSGJDdmVMdUlj?= =?utf-8?B?L05Kb2dvbGtvRlVUbnVyMHdobEF4L2cvU1ZGUS9zZFY3NktySmR1V0Mrd0Fm?= =?utf-8?B?L3NFcGswSUdlcTVRZUFLNE1qeHhVcnN2c0hsVTFsMDF2ZmtrS2hudnlrQjlT?= =?utf-8?B?Vm9vZWZyalVlQTRZdFFLWVRDNnNMMjVmQjdSdFB1STlBaGVLY2lxbW1mbWti?= =?utf-8?B?Vi93MkFUVFJiZW9TQ3czT0ZQNy95R01MbDVROXNTM3NRczJsWURqUUhSaFRF?= =?utf-8?B?TFg0bURWSHNoNHU4R2lvMEwzMkxkalo5SlQ3aVMzNnlsTXJHUFkzNC9vY3Yr?= =?utf-8?B?aFgwbHJCWG9Iayt0VzBUSlRXUlZTbUdkalhHczhJQTgrdDFObC90NllsOUNM?= =?utf-8?B?S0lKVzNLTmxqcHUzOXRxWkVzTjlBOWpnOHlmR3Y2eVloaTFDTzNFRjVCYXN0?= =?utf-8?B?RkZueENWTDNFSE1NMU1SUmpLKzRueGVXRHJGem9VbVpxNzdKZkRJRTVHbVlm?= =?utf-8?B?bWxZRm9pNWNtVVlVdEZXempRZVVJRHM2c3RYWk5jNUxLS0ZJWE1ndjk3aG1K?= =?utf-8?B?Ukh6d2xJV3B2WlpScXQwdlRzL3dFT1Rob3JreGFVc0FaSTJWeEc4aGhmU0NC?= =?utf-8?B?bld1R1FMQWNvK0VQYWNOVUxLVkdobllRdjIxTlJ4N2tKTlpnd3kyaWpTUVN4?= =?utf-8?B?UGppM3BCSlRPcERmVDdxUTZPTmdIKzEyRFA1ZVpHdTZtajdwUGhMWW9ranQz?= =?utf-8?B?RU1nVlY3YTdPUlFNQWRuZFcxVTc1NWhUbXRuUjFXV0M0VmZDUng5Y3MzOE9s?= =?utf-8?B?ZlZ5V0RCWThKendNSmZpbVJIVm5lQTk3MFFsTFArZHpkelA5K1l0Y1FxelIw?= =?utf-8?B?MTV4dXJ2RTFoMUVpR3FoY2hkRER0NytMOTFpSkduMnF4TmVydFQyWmk2UHRy?= =?utf-8?B?NzVjSkhwZ2ZxQklTcHFFL09KSUU0ZXZLZlUrb1JrdmxaeFVJRHlKaXhzQng0?= =?utf-8?B?T2RYUCthUkgyeHZaQW5UcUhvTmRDRk5pNnEva3FxVDhpckdjWmZjNDlobkFP?= =?utf-8?B?eG1mQWdnb1E0Vm9rbDZLeDFVOE5OS2lqbjFobk5sM01sUkhCMzdvenZMTlpU?= =?utf-8?B?VmhvVTBFWXRHTUNyWUZCNkNEOGNCVENCRG8xaUpmZytpZ2RLVWVGaFM0eWI4?= =?utf-8?B?U1puRkozZit2cHpLakdYNXZHQ3I1aGc3a0plY2czMTBSNUVyRjN6Mm0zU3VS?= =?utf-8?B?dWN5Y3IwK3NsSzQ4aXVkNWkwa3JiSGlaMGR5KzRWQjJoVytTMC93cTBxR1d5?= =?utf-8?B?cjM0ZzNjMGFyUmluYXJWMkQ0RWN4MGRIUGZ5aFhLQ3RSTmd1OXVZblExekky?= =?utf-8?B?b0tqWGRpNHFpc09aS0VHTVI0Y1V6SVdTZEZnVkRvdUxPeW84aEVQaXgxZHNu?= =?utf-8?B?VzZpY1JQU3Y5c1dJNTNPanZGeGxYWFZSQUVDTFFZYktMN2V0bGdxVTdEblhR?= =?utf-8?Q?ilA7kQjiGOtRWHTrVw3xxmPV8?= X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: c486f5a6-1cd9-49d1-54f1-08ddd949a70d X-MS-Exchange-CrossTenant-AuthSource: PH8PR12MB7277.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 12 Aug 2025 02:40:51.7734 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: QQsydig4LLe3+33Pb173tV/afxsdgnSgbCJHzWy27cLHwU0FF3CD+o/EpSvqnHVEyITUggTRx1o1cKwqOSDHpA== X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY8PR12MB7172 MIGRATE_VMA_SELECT_COMPOUND will be used to select THP pages during migrate_vma_setup() and MIGRATE_PFN_COMPOUND will make migrating device pages as compound pages during device pfn migration. migrate_device code paths go through the collect, setup and finalize phases of migration. The entries in src and dst arrays passed to these functions still remain at a PAGE_SIZE granularity. When a compound page is passed, the first entry has the PFN along with MIGRATE_PFN_COMPOUND and other flags set (MIGRATE_PFN_MIGRATE, MIGRATE_PFN_VALID), the remaining entries (HPAGE_PMD_NR - 1) are filled with 0's. This representation allows for the compound page to be split into smaller page sizes. migrate_vma_collect_hole(), migrate_vma_collect_pmd() are now THP page aware. Two new helper functions migrate_vma_collect_huge_pmd() and migrate_vma_insert_huge_pmd_page() have been added. migrate_vma_collect_huge_pmd() can collect THP pages, but if for some reason this fails, there is fallback support to split the folio and migrate it. migrate_vma_insert_huge_pmd_page() closely follows the logic of migrate_vma_insert_page() Support for splitting pages as needed for migration will follow in later patches in this series. Cc: Andrew Morton Cc: David Hildenbrand Cc: Zi Yan Cc: Joshua Hahn Cc: Rakie Kim Cc: Byungchul Park Cc: Gregory Price Cc: Ying Huang Cc: Alistair Popple Cc: Oscar Salvador Cc: Lorenzo Stoakes Cc: Baolin Wang Cc: "Liam R. Howlett" Cc: Nico Pache Cc: Ryan Roberts Cc: Dev Jain Cc: Barry Song Cc: Lyude Paul Cc: Danilo Krummrich Cc: David Airlie Cc: Simona Vetter Cc: Ralph Campbell Cc: Mika Penttil=C3=A4 Cc: Matthew Brost Cc: Francois Dugast Signed-off-by: Balbir Singh --- include/linux/migrate.h | 2 + mm/migrate_device.c | 457 ++++++++++++++++++++++++++++++++++------ 2 files changed, 396 insertions(+), 63 deletions(-) diff --git a/include/linux/migrate.h b/include/linux/migrate.h index acadd41e0b5c..d9cef0819f91 100644 --- a/include/linux/migrate.h +++ b/include/linux/migrate.h @@ -129,6 +129,7 @@ static inline int migrate_misplaced_folio(struct folio = *folio, int node) #define MIGRATE_PFN_VALID (1UL << 0) #define MIGRATE_PFN_MIGRATE (1UL << 1) #define MIGRATE_PFN_WRITE (1UL << 3) +#define MIGRATE_PFN_COMPOUND (1UL << 4) #define MIGRATE_PFN_SHIFT 6 =20 static inline struct page *migrate_pfn_to_page(unsigned long mpfn) @@ -147,6 +148,7 @@ enum migrate_vma_direction { MIGRATE_VMA_SELECT_SYSTEM =3D 1 << 0, MIGRATE_VMA_SELECT_DEVICE_PRIVATE =3D 1 << 1, MIGRATE_VMA_SELECT_DEVICE_COHERENT =3D 1 << 2, + MIGRATE_VMA_SELECT_COMPOUND =3D 1 << 3, }; =20 struct migrate_vma { diff --git a/mm/migrate_device.c b/mm/migrate_device.c index 0ed337f94fcd..6621bba62710 100644 --- a/mm/migrate_device.c +++ b/mm/migrate_device.c @@ -14,6 +14,7 @@ #include #include #include +#include #include #include "internal.h" =20 @@ -44,6 +45,23 @@ static int migrate_vma_collect_hole(unsigned long start, if (!vma_is_anonymous(walk->vma)) return migrate_vma_collect_skip(start, end, walk); =20 + if (thp_migration_supported() && + (migrate->flags & MIGRATE_VMA_SELECT_COMPOUND) && + (IS_ALIGNED(start, HPAGE_PMD_SIZE) && + IS_ALIGNED(end, HPAGE_PMD_SIZE))) { + migrate->src[migrate->npages] =3D MIGRATE_PFN_MIGRATE | + MIGRATE_PFN_COMPOUND; + migrate->dst[migrate->npages] =3D 0; + migrate->npages++; + migrate->cpages++; + + /* + * Collect the remaining entries as holes, in case we + * need to split later + */ + return migrate_vma_collect_skip(start + PAGE_SIZE, end, walk); + } + for (addr =3D start; addr < end; addr +=3D PAGE_SIZE) { migrate->src[migrate->npages] =3D MIGRATE_PFN_MIGRATE; migrate->dst[migrate->npages] =3D 0; @@ -54,57 +72,151 @@ static int migrate_vma_collect_hole(unsigned long star= t, return 0; } =20 -static int migrate_vma_collect_pmd(pmd_t *pmdp, - unsigned long start, - unsigned long end, - struct mm_walk *walk) +/** + * migrate_vma_collect_huge_pmd - collect THP pages without splitting the + * folio for device private pages. + * @pmdp: pointer to pmd entry + * @start: start address of the range for migration + * @end: end address of the range for migration + * @walk: mm_walk callback structure + * + * Collect the huge pmd entry at @pmdp for migration and set the + * MIGRATE_PFN_COMPOUND flag in the migrate src entry to indicate that + * migration will occur at HPAGE_PMD granularity + */ +static int migrate_vma_collect_huge_pmd(pmd_t *pmdp, unsigned long start, + unsigned long end, struct mm_walk *walk, + struct folio *fault_folio) { + struct mm_struct *mm =3D walk->mm; + struct folio *folio; struct migrate_vma *migrate =3D walk->private; - struct folio *fault_folio =3D migrate->fault_page ? - page_folio(migrate->fault_page) : NULL; - struct vm_area_struct *vma =3D walk->vma; - struct mm_struct *mm =3D vma->vm_mm; - unsigned long addr =3D start, unmapped =3D 0; spinlock_t *ptl; - pte_t *ptep; + swp_entry_t entry; + int ret; + unsigned long write =3D 0; =20 -again: - if (pmd_none(*pmdp)) + ptl =3D pmd_lock(mm, pmdp); + if (pmd_none(*pmdp)) { + spin_unlock(ptl); return migrate_vma_collect_hole(start, end, -1, walk); + } =20 if (pmd_trans_huge(*pmdp)) { - struct folio *folio; - - ptl =3D pmd_lock(mm, pmdp); - if (unlikely(!pmd_trans_huge(*pmdp))) { + if (!(migrate->flags & MIGRATE_VMA_SELECT_SYSTEM)) { spin_unlock(ptl); - goto again; + return migrate_vma_collect_skip(start, end, walk); } =20 folio =3D pmd_folio(*pmdp); if (is_huge_zero_folio(folio)) { spin_unlock(ptl); - split_huge_pmd(vma, pmdp, addr); - } else { - int ret; + return migrate_vma_collect_hole(start, end, -1, walk); + } + if (pmd_write(*pmdp)) + write =3D MIGRATE_PFN_WRITE; + } else if (!pmd_present(*pmdp)) { + entry =3D pmd_to_swp_entry(*pmdp); + folio =3D pfn_swap_entry_folio(entry); + + if (!is_device_private_entry(entry) || + !(migrate->flags & MIGRATE_VMA_SELECT_DEVICE_PRIVATE) || + (folio->pgmap->owner !=3D migrate->pgmap_owner)) { + spin_unlock(ptl); + return migrate_vma_collect_skip(start, end, walk); + } =20 - folio_get(folio); + if (is_migration_entry(entry)) { + migration_entry_wait_on_locked(entry, ptl); spin_unlock(ptl); - /* FIXME: we don't expect THP for fault_folio */ - if (WARN_ON_ONCE(fault_folio =3D=3D folio)) - return migrate_vma_collect_skip(start, end, - walk); - if (unlikely(!folio_trylock(folio))) - return migrate_vma_collect_skip(start, end, - walk); - ret =3D split_folio(folio); - if (fault_folio !=3D folio) - folio_unlock(folio); - folio_put(folio); - if (ret) - return migrate_vma_collect_skip(start, end, - walk); + return -EAGAIN; } + + if (is_writable_device_private_entry(entry)) + write =3D MIGRATE_PFN_WRITE; + } else { + spin_unlock(ptl); + return -EAGAIN; + } + + folio_get(folio); + if (folio !=3D fault_folio && unlikely(!folio_trylock(folio))) { + spin_unlock(ptl); + folio_put(folio); + return migrate_vma_collect_skip(start, end, walk); + } + + if (thp_migration_supported() && + (migrate->flags & MIGRATE_VMA_SELECT_COMPOUND) && + (IS_ALIGNED(start, HPAGE_PMD_SIZE) && + IS_ALIGNED(end, HPAGE_PMD_SIZE))) { + + struct page_vma_mapped_walk pvmw =3D { + .ptl =3D ptl, + .address =3D start, + .pmd =3D pmdp, + .vma =3D walk->vma, + }; + + unsigned long pfn =3D page_to_pfn(folio_page(folio, 0)); + + migrate->src[migrate->npages] =3D migrate_pfn(pfn) | write + | MIGRATE_PFN_MIGRATE + | MIGRATE_PFN_COMPOUND; + migrate->dst[migrate->npages++] =3D 0; + migrate->cpages++; + ret =3D set_pmd_migration_entry(&pvmw, folio_page(folio, 0)); + if (ret) { + migrate->npages--; + migrate->cpages--; + migrate->src[migrate->npages] =3D 0; + migrate->dst[migrate->npages] =3D 0; + goto fallback; + } + migrate_vma_collect_skip(start + PAGE_SIZE, end, walk); + spin_unlock(ptl); + return 0; + } + +fallback: + spin_unlock(ptl); + if (!folio_test_large(folio)) + goto done; + ret =3D split_folio(folio); + if (fault_folio !=3D folio) + folio_unlock(folio); + folio_put(folio); + if (ret) + return migrate_vma_collect_skip(start, end, walk); + if (pmd_none(pmdp_get_lockless(pmdp))) + return migrate_vma_collect_hole(start, end, -1, walk); + +done: + return -ENOENT; +} + +static int migrate_vma_collect_pmd(pmd_t *pmdp, + unsigned long start, + unsigned long end, + struct mm_walk *walk) +{ + struct migrate_vma *migrate =3D walk->private; + struct vm_area_struct *vma =3D walk->vma; + struct mm_struct *mm =3D vma->vm_mm; + unsigned long addr =3D start, unmapped =3D 0; + spinlock_t *ptl; + struct folio *fault_folio =3D migrate->fault_page ? + page_folio(migrate->fault_page) : NULL; + pte_t *ptep; + +again: + if (pmd_trans_huge(*pmdp) || !pmd_present(*pmdp)) { + int ret =3D migrate_vma_collect_huge_pmd(pmdp, start, end, walk, fault_f= olio); + + if (ret =3D=3D -EAGAIN) + goto again; + if (ret =3D=3D 0) + return 0; } =20 ptep =3D pte_offset_map_lock(mm, pmdp, addr, &ptl); @@ -222,8 +334,7 @@ static int migrate_vma_collect_pmd(pmd_t *pmdp, mpfn |=3D pte_write(pte) ? MIGRATE_PFN_WRITE : 0; } =20 - /* FIXME support THP */ - if (!page || !page->mapping || PageTransCompound(page)) { + if (!page || !page->mapping) { mpfn =3D 0; goto next; } @@ -394,14 +505,6 @@ static bool migrate_vma_check_page(struct page *page, = struct page *fault_page) */ int extra =3D 1 + (page =3D=3D fault_page); =20 - /* - * FIXME support THP (transparent huge page), it is bit more complex to - * check them than regular pages, because they can be mapped with a pmd - * or with a pte (split pte mapping). - */ - if (folio_test_large(folio)) - return false; - /* Page from ZONE_DEVICE have one extra reference */ if (folio_is_zone_device(folio)) extra++; @@ -432,17 +535,24 @@ static unsigned long migrate_device_unmap(unsigned lo= ng *src_pfns, =20 lru_add_drain(); =20 - for (i =3D 0; i < npages; i++) { + for (i =3D 0; i < npages; ) { struct page *page =3D migrate_pfn_to_page(src_pfns[i]); struct folio *folio; + unsigned int nr =3D 1; =20 if (!page) { if (src_pfns[i] & MIGRATE_PFN_MIGRATE) unmapped++; - continue; + goto next; } =20 folio =3D page_folio(page); + nr =3D folio_nr_pages(folio); + + if (nr > 1) + src_pfns[i] |=3D MIGRATE_PFN_COMPOUND; + + /* ZONE_DEVICE folios are not on LRU */ if (!folio_is_zone_device(folio)) { if (!folio_test_lru(folio) && allow_drain) { @@ -454,7 +564,7 @@ static unsigned long migrate_device_unmap(unsigned long= *src_pfns, if (!folio_isolate_lru(folio)) { src_pfns[i] &=3D ~MIGRATE_PFN_MIGRATE; restore++; - continue; + goto next; } =20 /* Drop the reference we took in collect */ @@ -473,10 +583,12 @@ static unsigned long migrate_device_unmap(unsigned lo= ng *src_pfns, =20 src_pfns[i] &=3D ~MIGRATE_PFN_MIGRATE; restore++; - continue; + goto next; } =20 unmapped++; +next: + i +=3D nr; } =20 for (i =3D 0; i < npages && restore; i++) { @@ -622,6 +734,147 @@ int migrate_vma_setup(struct migrate_vma *args) } EXPORT_SYMBOL(migrate_vma_setup); =20 +#ifdef CONFIG_ARCH_ENABLE_THP_MIGRATION +/** + * migrate_vma_insert_huge_pmd_page: Insert a huge folio into @migrate->vm= a->vm_mm + * at @addr. folio is already allocated as a part of the migration process= with + * large page. + * + * @folio needs to be initialized and setup after it's allocated. The code= bits + * here follow closely the code in __do_huge_pmd_anonymous_page(). This AP= I does + * not support THP zero pages. + * + * @migrate: migrate_vma arguments + * @addr: address where the folio will be inserted + * @folio: folio to be inserted at @addr + * @src: src pfn which is being migrated + * @pmdp: pointer to the pmd + */ +static int migrate_vma_insert_huge_pmd_page(struct migrate_vma *migrate, + unsigned long addr, + struct page *page, + unsigned long *src, + pmd_t *pmdp) +{ + struct vm_area_struct *vma =3D migrate->vma; + gfp_t gfp =3D vma_thp_gfp_mask(vma); + struct folio *folio =3D page_folio(page); + int ret; + vm_fault_t csa_ret; + spinlock_t *ptl; + pgtable_t pgtable; + pmd_t entry; + bool flush =3D false; + unsigned long i; + + VM_WARN_ON_FOLIO(!folio, folio); + VM_WARN_ON_ONCE(!pmd_none(*pmdp) && !is_huge_zero_pmd(*pmdp)); + + if (!thp_vma_suitable_order(vma, addr, HPAGE_PMD_ORDER)) + return -EINVAL; + + ret =3D anon_vma_prepare(vma); + if (ret) + return ret; + + folio_set_order(folio, HPAGE_PMD_ORDER); + folio_set_large_rmappable(folio); + + if (mem_cgroup_charge(folio, migrate->vma->vm_mm, gfp)) { + count_vm_event(THP_FAULT_FALLBACK); + count_mthp_stat(HPAGE_PMD_ORDER, MTHP_STAT_ANON_FAULT_FALLBACK_CHARGE); + ret =3D -ENOMEM; + goto abort; + } + + __folio_mark_uptodate(folio); + + pgtable =3D pte_alloc_one(vma->vm_mm); + if (unlikely(!pgtable)) + goto abort; + + if (folio_is_device_private(folio)) { + swp_entry_t swp_entry; + + if (vma->vm_flags & VM_WRITE) + swp_entry =3D make_writable_device_private_entry( + page_to_pfn(page)); + else + swp_entry =3D make_readable_device_private_entry( + page_to_pfn(page)); + entry =3D swp_entry_to_pmd(swp_entry); + } else { + if (folio_is_zone_device(folio) && + !folio_is_device_coherent(folio)) { + goto abort; + } + entry =3D folio_mk_pmd(folio, vma->vm_page_prot); + if (vma->vm_flags & VM_WRITE) + entry =3D pmd_mkwrite(pmd_mkdirty(entry), vma); + } + + ptl =3D pmd_lock(vma->vm_mm, pmdp); + csa_ret =3D check_stable_address_space(vma->vm_mm); + if (csa_ret) + goto abort; + + /* + * Check for userfaultfd but do not deliver the fault. Instead, + * just back off. + */ + if (userfaultfd_missing(vma)) + goto unlock_abort; + + if (!pmd_none(*pmdp)) { + if (!is_huge_zero_pmd(*pmdp)) + goto unlock_abort; + flush =3D true; + } else if (!pmd_none(*pmdp)) + goto unlock_abort; + + add_mm_counter(vma->vm_mm, MM_ANONPAGES, HPAGE_PMD_NR); + folio_add_new_anon_rmap(folio, vma, addr, RMAP_EXCLUSIVE); + if (!folio_is_zone_device(folio)) + folio_add_lru_vma(folio, vma); + folio_get(folio); + + if (flush) { + pte_free(vma->vm_mm, pgtable); + flush_cache_page(vma, addr, addr + HPAGE_PMD_SIZE); + pmdp_invalidate(vma, addr, pmdp); + } else { + pgtable_trans_huge_deposit(vma->vm_mm, pmdp, pgtable); + mm_inc_nr_ptes(vma->vm_mm); + } + set_pmd_at(vma->vm_mm, addr, pmdp, entry); + update_mmu_cache_pmd(vma, addr, pmdp); + + spin_unlock(ptl); + + count_vm_event(THP_FAULT_ALLOC); + count_mthp_stat(HPAGE_PMD_ORDER, MTHP_STAT_ANON_FAULT_ALLOC); + count_memcg_event_mm(vma->vm_mm, THP_FAULT_ALLOC); + + return 0; + +unlock_abort: + spin_unlock(ptl); +abort: + for (i =3D 0; i < HPAGE_PMD_NR; i++) + src[i] &=3D ~MIGRATE_PFN_MIGRATE; + return 0; +} +#else /* !CONFIG_ARCH_ENABLE_THP_MIGRATION */ +static int migrate_vma_insert_huge_pmd_page(struct migrate_vma *migrate, + unsigned long addr, + struct page *page, + unsigned long *src, + pmd_t *pmdp) +{ + return 0; +} +#endif + /* * This code closely matches the code in: * __handle_mm_fault() @@ -632,9 +885,10 @@ EXPORT_SYMBOL(migrate_vma_setup); */ static void migrate_vma_insert_page(struct migrate_vma *migrate, unsigned long addr, - struct page *page, + unsigned long *dst, unsigned long *src) { + struct page *page =3D migrate_pfn_to_page(*dst); struct folio *folio =3D page_folio(page); struct vm_area_struct *vma =3D migrate->vma; struct mm_struct *mm =3D vma->vm_mm; @@ -662,8 +916,25 @@ static void migrate_vma_insert_page(struct migrate_vma= *migrate, pmdp =3D pmd_alloc(mm, pudp, addr); if (!pmdp) goto abort; - if (pmd_trans_huge(*pmdp)) - goto abort; + + if (thp_migration_supported() && (*dst & MIGRATE_PFN_COMPOUND)) { + int ret =3D migrate_vma_insert_huge_pmd_page(migrate, addr, page, + src, pmdp); + if (ret) + goto abort; + return; + } + + if (!pmd_none(*pmdp)) { + if (pmd_trans_huge(*pmdp)) { + if (!is_huge_zero_pmd(*pmdp)) + goto abort; + folio_get(pmd_folio(*pmdp)); + split_huge_pmd(vma, pmdp, addr); + } else if (pmd_leaf(*pmdp)) + goto abort; + } + if (pte_alloc(mm, pmdp)) goto abort; if (unlikely(anon_vma_prepare(vma))) @@ -754,23 +1025,24 @@ static void __migrate_device_pages(unsigned long *sr= c_pfns, unsigned long i; bool notified =3D false; =20 - for (i =3D 0; i < npages; i++) { + for (i =3D 0; i < npages; ) { struct page *newpage =3D migrate_pfn_to_page(dst_pfns[i]); struct page *page =3D migrate_pfn_to_page(src_pfns[i]); struct address_space *mapping; struct folio *newfolio, *folio; int r, extra_cnt =3D 0; + unsigned long nr =3D 1; =20 if (!newpage) { src_pfns[i] &=3D ~MIGRATE_PFN_MIGRATE; - continue; + goto next; } =20 if (!page) { unsigned long addr; =20 if (!(src_pfns[i] & MIGRATE_PFN_MIGRATE)) - continue; + goto next; =20 /* * The only time there is no vma is when called from @@ -788,15 +1060,47 @@ static void __migrate_device_pages(unsigned long *sr= c_pfns, migrate->pgmap_owner); mmu_notifier_invalidate_range_start(&range); } - migrate_vma_insert_page(migrate, addr, newpage, + + if ((src_pfns[i] & MIGRATE_PFN_COMPOUND) && + (!(dst_pfns[i] & MIGRATE_PFN_COMPOUND))) { + nr =3D HPAGE_PMD_NR; + src_pfns[i] &=3D ~MIGRATE_PFN_COMPOUND; + src_pfns[i] &=3D ~MIGRATE_PFN_MIGRATE; + goto next; + } + + migrate_vma_insert_page(migrate, addr, &dst_pfns[i], &src_pfns[i]); - continue; + goto next; } =20 newfolio =3D page_folio(newpage); folio =3D page_folio(page); mapping =3D folio_mapping(folio); =20 + /* + * If THP migration is enabled, check if both src and dst + * can migrate large pages + */ + if (thp_migration_supported()) { + if ((src_pfns[i] & MIGRATE_PFN_MIGRATE) && + (src_pfns[i] & MIGRATE_PFN_COMPOUND) && + !(dst_pfns[i] & MIGRATE_PFN_COMPOUND)) { + + if (!migrate) { + src_pfns[i] &=3D ~(MIGRATE_PFN_MIGRATE | + MIGRATE_PFN_COMPOUND); + goto next; + } + src_pfns[i] &=3D ~MIGRATE_PFN_MIGRATE; + } else if ((src_pfns[i] & MIGRATE_PFN_MIGRATE) && + (dst_pfns[i] & MIGRATE_PFN_COMPOUND) && + !(src_pfns[i] & MIGRATE_PFN_COMPOUND)) { + src_pfns[i] &=3D ~MIGRATE_PFN_MIGRATE; + } + } + + if (folio_is_device_private(newfolio) || folio_is_device_coherent(newfolio)) { if (mapping) { @@ -809,7 +1113,7 @@ static void __migrate_device_pages(unsigned long *src_= pfns, if (!folio_test_anon(folio) || !folio_free_swap(folio)) { src_pfns[i] &=3D ~MIGRATE_PFN_MIGRATE; - continue; + goto next; } } } else if (folio_is_zone_device(newfolio)) { @@ -817,7 +1121,7 @@ static void __migrate_device_pages(unsigned long *src_= pfns, * Other types of ZONE_DEVICE page are not supported. */ src_pfns[i] &=3D ~MIGRATE_PFN_MIGRATE; - continue; + goto next; } =20 BUG_ON(folio_test_writeback(folio)); @@ -829,6 +1133,8 @@ static void __migrate_device_pages(unsigned long *src_= pfns, src_pfns[i] &=3D ~MIGRATE_PFN_MIGRATE; else folio_migrate_flags(newfolio, folio); +next: + i +=3D nr; } =20 if (notified) @@ -990,10 +1296,23 @@ static unsigned long migrate_device_pfn_lock(unsigne= d long pfn) int migrate_device_range(unsigned long *src_pfns, unsigned long start, unsigned long npages) { - unsigned long i, pfn; + unsigned long i, j, pfn; + + for (pfn =3D start, i =3D 0; i < npages; pfn++, i++) { + struct page *page =3D pfn_to_page(pfn); + struct folio *folio =3D page_folio(page); + unsigned int nr =3D 1; =20 - for (pfn =3D start, i =3D 0; i < npages; pfn++, i++) src_pfns[i] =3D migrate_device_pfn_lock(pfn); + nr =3D folio_nr_pages(folio); + if (nr > 1) { + src_pfns[i] |=3D MIGRATE_PFN_COMPOUND; + for (j =3D 1; j < nr; j++) + src_pfns[i+j] =3D 0; + i +=3D j - 1; + pfn +=3D j - 1; + } + } =20 migrate_device_unmap(src_pfns, npages, NULL); =20 @@ -1011,10 +1330,22 @@ EXPORT_SYMBOL(migrate_device_range); */ int migrate_device_pfns(unsigned long *src_pfns, unsigned long npages) { - unsigned long i; + unsigned long i, j; + + for (i =3D 0; i < npages; i++) { + struct page *page =3D pfn_to_page(src_pfns[i]); + struct folio *folio =3D page_folio(page); + unsigned int nr =3D 1; =20 - for (i =3D 0; i < npages; i++) src_pfns[i] =3D migrate_device_pfn_lock(src_pfns[i]); + nr =3D folio_nr_pages(folio); + if (nr > 1) { + src_pfns[i] |=3D MIGRATE_PFN_COMPOUND; + for (j =3D 1; j < nr; j++) + src_pfns[i+j] =3D 0; + i +=3D j - 1; + } + } =20 migrate_device_unmap(src_pfns, npages, NULL); =20 --=20 2.50.1 From nobody Sat Oct 4 22:37:10 2025 Received: from NAM11-DM6-obe.outbound.protection.outlook.com (mail-dm6nam11on2047.outbound.protection.outlook.com [40.107.223.47]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 2F39B2E36E9 for ; Tue, 12 Aug 2025 02:40:57 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.223.47 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1754966458; cv=fail; b=TxvcHrSbKPX8Zts8NoxctzHgIQfu6Y3CFDcU/TaUZ56xklTKk4W8muP7FlsODNPwobHyM70pVI9eH1fCpZWkRAndRHoA94y3N6bsZ/gP5iQS+ZTsh+PX4JmOIN+pWBckPDSdv3MixCfAhPnfHyzpO4nsnfXI5Cn6QJX6CUNOcS8= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1754966458; c=relaxed/simple; bh=phXEi6EgVoMihSCK4sO0808mmiMuIK20Oj3fvl5XnZg=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: Content-Type:MIME-Version; b=m5s6xhX/QIkZMCNgT3nBWVctfRi/vI/ww073cUsoyD+oMTWgOAsmn7VoZzKU9Iqa8NK7J+4HcPvObfz5XjvUNaluStrSc2hjN5w5OmNXS4sxKynGGxMZ93tb4QXod4DJUb2NnQiwAr7IArB/IPIMt0nujiUBv+gwbjYqVDuLU0I= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com; spf=fail smtp.mailfrom=nvidia.com; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b=JUzpHgsG; arc=fail smtp.client-ip=40.107.223.47 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="JUzpHgsG" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=y+QyMEOz+NDM+IXSjzQ7gpkPrcWbuMvl1U4U2r/a5yIVgbxlA7IRI5PmRP2+IzEhYV6FLtSyNnOqv7rtkJgVbUbw7OQBVZSoIuVF+L0d4qSZErFXms5GPiLK7u6GI8YgP3C1NfJK1bLZWtSlg0WZB9BwCJCu+W+Lm5WnpTma0OTchZDinRHbwW7onnhxsnTL08hpyLNE6veOCguc0/86oKR1M2pFo/WqmY3NEWRY6GDurtYV/nCQtpoNoWIGT+bpAexDzpPhT+NfCInu9gxxAdTbqAkx84E0jPOi/g2lrgt03vpXQtbC88RmnvZOQgGH53ceGFAyo7iCNNdzQaWqsQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=yY9WdugDnPOocmfuNJVlqZJtZk2WbFvx/um3nx8BpdM=; b=s1pOkoIR7/vmmBMI6sVP91IJMP5m/Tw+0oFxfK3c0KMRbnAWlqotgtj2kxeq09kQ5eHW8B5dU5Db8Hiv15zUmlzA1NuNqeYHXR+U6721dYnvlLuIsKFr6UbykmWrohcoc3GgYPnGnXw6AgSG+JTnPyCuRJkmq1KxeNVpy7bh3yp4bCfGsIt8wqfGSEjNy+HJ9zwwEmUTaKmx3VGBzUa2dcv5csNBc1y1AnaEuCFFecyv6usRxghlLfOkp2h6/nnUHhSKBK5uHglKR5qbugigbgCYU10HWSaOLBCJGIzlOnSIR7CPeRjLgQTbzfJFmPaf+rhxat85qc+q4pNhkCHHTA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=yY9WdugDnPOocmfuNJVlqZJtZk2WbFvx/um3nx8BpdM=; b=JUzpHgsGBOmGVGkw+TFOtKcDCm2rFz08oa2Y4uAUdhlDDwiAnQ2WPqa16pUXyu90S5f1HKSP/OTALA6MV94sQr3B1RHY8JdI3S0imgslKg5LA8/FRZVl4b2g1xVcxNU5+hrNwTvzTjV/UYo21zrwcMFVwEORLDvDHQEmcoLHbvCOPi5Z793n+X5mkFcvhgIKVs6snmMG6+pxIj+lqYGt3iA8eeMV1qwHuGot96PsXRWyXQ8pQEx9HRdmqLNZv/YlrI+/QCSI2mX0udaY7FHfPZ9vGXLMb9s4Fj65nZJ5jvCvO3AWj/3Qi2n+O5bLulO5gMCtSM+1a0LCRW3/0GyrnA== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from PH8PR12MB7277.namprd12.prod.outlook.com (2603:10b6:510:223::13) by CY8PR12MB7172.namprd12.prod.outlook.com (2603:10b6:930:5b::7) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9009.18; Tue, 12 Aug 2025 02:40:55 +0000 Received: from PH8PR12MB7277.namprd12.prod.outlook.com ([fe80::3a4:70ea:ff05:1251]) by PH8PR12MB7277.namprd12.prod.outlook.com ([fe80::3a4:70ea:ff05:1251%5]) with mapi id 15.20.9009.018; Tue, 12 Aug 2025 02:40:55 +0000 From: Balbir Singh To: dri-devel@lists.freedesktop.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Balbir Singh , Andrew Morton , David Hildenbrand , Zi Yan , Joshua Hahn , Rakie Kim , Byungchul Park , Gregory Price , Ying Huang , Alistair Popple , Oscar Salvador , Lorenzo Stoakes , Baolin Wang , "Liam R. Howlett" , Nico Pache , Ryan Roberts , Dev Jain , Barry Song , Lyude Paul , Danilo Krummrich , David Airlie , Simona Vetter , Ralph Campbell , =?UTF-8?q?Mika=20Penttil=C3=A4?= , Matthew Brost , Francois Dugast Subject: [v3 04/11] mm/memory/fault: add support for zone device THP fault handling Date: Tue, 12 Aug 2025 12:40:29 +1000 Message-ID: <20250812024036.690064-5-balbirs@nvidia.com> X-Mailer: git-send-email 2.50.1 In-Reply-To: <20250812024036.690064-1-balbirs@nvidia.com> References: <20250812024036.690064-1-balbirs@nvidia.com> Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: SY0PR01CA0011.ausprd01.prod.outlook.com (2603:10c6:10:1bb::8) To PH8PR12MB7277.namprd12.prod.outlook.com (2603:10b6:510:223::13) Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: PH8PR12MB7277:EE_|CY8PR12MB7172:EE_ X-MS-Office365-Filtering-Correlation-Id: cdc0e0a6-0b58-419c-acda-08ddd949a91c X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|366016|376014|7416014|1800799024; X-Microsoft-Antispam-Message-Info: =?utf-8?B?cmZGQTdwYUxTdXFpUHdhbUpBT1RNNTBhdU9ldWNjNDZBK0NrNjcrQi92QWRn?= =?utf-8?B?MWlVMnc3aFVObWpnSzZPVm5VUENER3BTYU96OU4rMEIrMlEybGtwQlBsczBC?= =?utf-8?B?QWlYa0RUK0tQVnVJbFBJdmVwRDQ1b0tRS2JrRXIrKzdUc29KSGFiTTM2R3E4?= =?utf-8?B?aGNyM0VyU0lMLzFsalROLzJuVjdIdm10cDQvb2lSZXFEaDhKcDhieWdKZ1hR?= =?utf-8?B?bzUxeHZaSnBmcVg1bHc3eS9icEhzRm0zeVA3UkZFRTl1ODJQVDM3LzlUOC9D?= =?utf-8?B?MEZMakV3bmxQZEJHSlZHYXBXSWFjYW01OGIvQWljYUVqOUlnREh5K3owdk8x?= =?utf-8?B?R2FMT0xYalc4c1VrVkdxeXpqNTVPTDBMZEJOSW9yMzNjNDl1blA0Ung5YjJB?= =?utf-8?B?WnVQZ0JOSzgzS0tDbXZmQ2RqbTh0ZVJZbEYwbGVyTnZFNkdlZzB5WkVtVUhV?= =?utf-8?B?T2NEalVoak1DMmhoSGJPSkM5UktkRzk0VXlKeWg2OGNNNU1zczNvRURKaG42?= =?utf-8?B?eVV2bWthVDYwYWtlOWQrVGxRKzV6N21FS3lIOXdFSXkzTmIxVjRzNEJ5Y1dR?= =?utf-8?B?ekl3MGxWU0lWYi9vZTlLbzJHTlVhTXhMMVkrMTlCMXhWdVNEeEpOMnI4ZGU3?= =?utf-8?B?KzUwdG9GU3Z1Q01lZTM5U1dSUlE3QUZPNzh4V3BXTlZqMERrNDdseVJxYmRS?= =?utf-8?B?NENIdGFEWnVOTGxIMVg2ZWZvV0xtQ1RlVHJIZWhUOWNQczNJZEhWU2NkMFRx?= =?utf-8?B?SlMrQ1RmQUdSaXFIaHhpbG5lSHdKUlVhbkE5Rnp2SU5DN2hwUzJYRU95cDBK?= =?utf-8?B?NEJLNmc5QTdLbVhTVG1COG5MaW96Lzd6MEdld1lOdVc3dWN5WGQ4cUxhc2Fm?= =?utf-8?B?dDNidWFCdXYzaHFDazg2VWxrRFZaQXl6MTlNeTZaUkxtYURrVmJCKzlSM2Y2?= =?utf-8?B?TENKbmRMK3RBMitZMGIzak11YTVmdm1NVzJBQ2JtMGhpajBiMlMxWjY1Mm93?= =?utf-8?B?em12SjRvd1hJVW1MMjJtT1hLVFpSbkdsSDdGU1BFUFR6R0xCNWNaTCt6VWpw?= =?utf-8?B?dGc4NFVjZjRERWJZby9pZVdFMGJ5emE3T0haUFlWOVlwdnMrRjR6L1h1eW5q?= =?utf-8?B?N2xIRFlpOEMxTnNOUDg3b0p4Y3Q5eFpOYWVkRUxTaExtK1R0cG1NeGp5YXZI?= =?utf-8?B?dkEvRFhQL1NINWowbHh1aUQwMzJrWFMyLzJzQ3lWRHYrVUErb0pwZU5na3dh?= =?utf-8?B?VVZuSlczdnNQNTQ5czJkL2VMWnhyTkJDeVNwNFNkeWRxSExFRFBGbnN4SGFT?= =?utf-8?B?VlBkYXQwT3oxYVdLdW0wNE9pTHYxcHl2RXZaOUNtdnE1azM2TmMzaEJvOG9R?= =?utf-8?B?VDRVcE9YNWYyRVZLa21ITmVjZjQ0Qm9mL0lpeEJFZFhrd1RRaVJJRHphRWxa?= =?utf-8?B?SEpjcy9PZ2xVU2xwN0V6NURuOTdPZVJrTVVFQXNiWnJHS2kydHBZRE9TY2RP?= =?utf-8?B?WlU5VmxoWEt6M3ZEZXRlZXM5S0I0Vm5TL1VXZ2diZERaNWdZbUhvb0tra00r?= =?utf-8?B?V2wwQWg2bGZyZTdJVVQyaHBlTmkveGtGWnNQcmtwZmtOQjZObHh1YUQ3ckxl?= =?utf-8?B?eXRzdWJ0RXJSbkVsMUVNMkdQRkFjWXE3c2dSQS9ac3NhV3FWZ21ybDBxKzND?= =?utf-8?B?NG9HUlgwZVhNU1Fkbmo5cVp3VE5XZjlCYjlEMEsxaW9rSEdaWnpoNGd2NUdw?= =?utf-8?B?b1hRazlvTVhCUUs0Z1FMQ1I4Q2NCYWtuYkZrUjBqZS94NFRrUWp1c3l5UHZO?= =?utf-8?B?a2ZOMm1vMVgrWWxyWFdFbmNGWDEvR3Q1QktJWStzdHlBd1JlbjMxeWlUSjVI?= =?utf-8?B?dml6QlpPTEwwMWJ2WXU0VFB3ZVNURW5sUzZ1cUp6Z1MwZDh2VUVMTG1Mc3VF?= =?utf-8?Q?rREEO7aJR7Y=3D?= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:PH8PR12MB7277.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(366016)(376014)(7416014)(1800799024);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?VGJCUTJvSmUvTWwxMkZ6UDNsQkc4ZFdkZjc3ZHFzWXBFWWJKNE4rcWFNNlF1?= =?utf-8?B?WWppemE0NUhHemtIaFdvWFQrTDk5ajBUYURtWEY3TTdpWm1tUEl0T3RqTFRX?= =?utf-8?B?V1VLS2NlNXFxb0kyL2NFcWhOa3hjOThLbnRKTzIrMVZmMVB0WEJkUjBDR2FW?= =?utf-8?B?Q2NWVFgvOGFuM3R0VHo1WWRyOVcveUdoa0xRQ29pdVJJUDJ1dnRWWXpMRElW?= =?utf-8?B?c0l3R09lV1QrbDMwT09Pc0xGMTdKa1JGSnEyRy8yamNmSUNLdUxDVWkyV1Ar?= =?utf-8?B?aER0SlFwT3RkaEpkendWREJxOEtWVmREY0hWUmFZaW5NUFZXa2FGMkwvVm11?= =?utf-8?B?U0dQdktiU2xlYXptc3NIVkd1bWNBelNwNm9scm9Xc0wvT3VWalQ5dC9uWkNW?= =?utf-8?B?c2lTT0cyWmNxbElJS0hySWZ3WlVuQVdCVVNEVUhmdGVIM3YzcTNPMWZrL1d2?= =?utf-8?B?UENOa3ZsN0dTN1J1dHl6d3ltVGdTQVhKNlQzbjhJaENnb2diRlZocnZUSjB0?= =?utf-8?B?SktBbzdRWUQzWmVRd3k0U1l1SUs2R3pTQlc4MzVRSHc3MXRHSFhIUkRGZith?= =?utf-8?B?MTdYWjF3bWhtZTVLUjhxZUMvYWEwcEZCMVlCVnh1VlliKzVSL2RlcG5uSVdl?= =?utf-8?B?VDJaTHovME8vNFpaQldCMHpOdXBUbGxCWnZSWVMzaXFDR0s2cUxpYkZ5bDk1?= =?utf-8?B?ZDJKUDd5eVlRS0JlQVA3UVlleFpBbERSekNqRVRaMlEwbzF0Mkcvemk2clZS?= =?utf-8?B?Y0ZFQTVuSStWaUlyWjFzbG55VlY5WHNURlZwL2hRUWxHblZacDVBOXdhMFov?= =?utf-8?B?UmtYVC96M0hBcU9JYkRqOVdWYU0vSGJDYVBKZUlhTVFLeFRwTm5Da21TYkFm?= =?utf-8?B?WkZjY1NXbkUyb0JIaldXRklpcVg2V2tTeFdBdkMybVNCU3N2bFhEUE5iQkNl?= =?utf-8?B?emJSbFgzYjR1YXJlZVJuZGxFdlpWaURPYnpoYUt1eWhrQk83WGkrYUpKOVZE?= =?utf-8?B?dnNxazdaNmM3TzNRc2FZRmVYNVBPNmd5eEo1cmpwd3BLSXRGUmp0ZjJJdW1M?= =?utf-8?B?K01sdnQrYkE1WnRBMzBvZTdnL3hsLzFZakhCYkt3eDdaeHgzaE1qMFVoVUhT?= =?utf-8?B?OHlDN2pQOXkwUWVnYVFBMGE5U3BBM1RTMHV3SXJKM2NKM2hvSWpXNm03b0RT?= =?utf-8?B?dTZiN3U2ckRYWktreEE3cUZ2bFhBRmoweXI3MnY3Z2E0cHZjRytDdmRwOWFQ?= =?utf-8?B?aFQ3Sno2TzhrZ01ZbldWZE5VZ3BWb1p0cW4vRFgzWnJabGZiYmhqOTNvazJV?= =?utf-8?B?SHdFYjR2RnZZTEllWUQvYzVCdWZ2VUxuWkEybjlSSHFlT0hFZXEyTVhiTFVZ?= =?utf-8?B?L0RmM09zYnArU2hDaER2T1hpRi8vTkNDblY5bHdhaVFJcUw4UUpqaVdyZWVG?= =?utf-8?B?NGFpcjQ2T1doZ3Zwc3l0T0o1aDRaLzZNZFRiWkRKbzZjakNHbXdzaVN1SnJ5?= =?utf-8?B?cWtPa3JMLzZVdjVBVHc0N3N6a0ZYYWsybVZXS29XTnkwRDhLNlFyZzBoOHB3?= =?utf-8?B?Nm8rV1ZGTm80MldPOXhINnVwbklKdEFwN0ZrOHNmU25vdmkvcFR0RkJsZjRo?= =?utf-8?B?MThzN3ZoSGduWnBUL2UrVm1NZFRydVlUSU9qL0UrS3NiZ3ZaUjAwais1RWww?= =?utf-8?B?NFQ4YXRrUHRmK0Nmd3hzUHMyekNmZW9PbkoxNGxJRjBrZEdvRUZ6WEVLNFJS?= =?utf-8?B?VlNaRnpMUWoycGdnZlltc0JwVVpJZzNnT0krKy9zZE8weVZaRWVIUVg2MFhm?= =?utf-8?B?UTk5WnQwNmc5S3laT0s5RjFsSis4Y2ZyeEErZ0ZMeEQyQTBUeFd0Ky93TnVh?= =?utf-8?B?Q1pidzBxUWxPMG1HR25jNjM5MVJCMGF1clFVR3ZiY1F1WjlQZnVPYldBVXlJ?= =?utf-8?B?TVB4UE1EMi9idUJMR0lwQkwxUzF1VWcxdTMrVTFmSjZvamxTMzFSWWFTM0ZP?= =?utf-8?B?Y0xkZ1JwUmVnQTJpeTNDSU5ZV2dBQnFIejhDektCY0lNYW0wdElHYUdjTXV4?= =?utf-8?B?WmFSQWFmbm92U0l4RzZnVXVVTXlPUEJiZnZNQjRDbkJ4K1ZOQnB0WHo5Y1Av?= =?utf-8?Q?4mpeISP6dLY3lQOVY6UkfDpxQ?= X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: cdc0e0a6-0b58-419c-acda-08ddd949a91c X-MS-Exchange-CrossTenant-AuthSource: PH8PR12MB7277.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 12 Aug 2025 02:40:55.2206 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: T5mHGOJCJ17ELfXVbWFVRM+tUK0RBJ2Y0Dxj90N7JRjhDr1DnbxtVrDr/1eZBz34/6jF3PxMffdOXE/IRYH7eQ== X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY8PR12MB7172 When the CPU touches a zone device THP entry, the data needs to be migrated back to the CPU, call migrate_to_ram() on these pages via do_huge_pmd_device_private() fault handling helper. Cc: Andrew Morton Cc: David Hildenbrand Cc: Zi Yan Cc: Joshua Hahn Cc: Rakie Kim Cc: Byungchul Park Cc: Gregory Price Cc: Ying Huang Cc: Alistair Popple Cc: Oscar Salvador Cc: Lorenzo Stoakes Cc: Baolin Wang Cc: "Liam R. Howlett" Cc: Nico Pache Cc: Ryan Roberts Cc: Dev Jain Cc: Barry Song Cc: Lyude Paul Cc: Danilo Krummrich Cc: David Airlie Cc: Simona Vetter Cc: Ralph Campbell Cc: Mika Penttil=C3=A4 Cc: Matthew Brost Cc: Francois Dugast Signed-off-by: Balbir Singh --- include/linux/huge_mm.h | 7 +++++++ mm/huge_memory.c | 36 ++++++++++++++++++++++++++++++++++++ mm/memory.c | 6 ++++-- 3 files changed, 47 insertions(+), 2 deletions(-) diff --git a/include/linux/huge_mm.h b/include/linux/huge_mm.h index 7748489fde1b..a4880fe98e46 100644 --- a/include/linux/huge_mm.h +++ b/include/linux/huge_mm.h @@ -474,6 +474,8 @@ static inline bool folio_test_pmd_mappable(struct folio= *folio) =20 vm_fault_t do_huge_pmd_numa_page(struct vm_fault *vmf); =20 +vm_fault_t do_huge_pmd_device_private(struct vm_fault *vmf); + extern struct folio *huge_zero_folio; extern unsigned long huge_zero_pfn; =20 @@ -632,6 +634,11 @@ static inline vm_fault_t do_huge_pmd_numa_page(struct = vm_fault *vmf) return 0; } =20 +static inline vm_fault_t do_huge_pmd_device_private(struct vm_fault *vmf) +{ + return 0; +} + static inline bool is_huge_zero_folio(const struct folio *folio) { return false; diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 2495e3fdbfae..8888140e57a3 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -1267,6 +1267,42 @@ static vm_fault_t __do_huge_pmd_anonymous_page(struc= t vm_fault *vmf) =20 } =20 +vm_fault_t do_huge_pmd_device_private(struct vm_fault *vmf) +{ + struct vm_area_struct *vma =3D vmf->vma; + vm_fault_t ret =3D 0; + spinlock_t *ptl; + swp_entry_t swp_entry; + struct page *page; + + if (vmf->flags & FAULT_FLAG_VMA_LOCK) { + vma_end_read(vma); + return VM_FAULT_RETRY; + } + + ptl =3D pmd_lock(vma->vm_mm, vmf->pmd); + if (unlikely(!pmd_same(*vmf->pmd, vmf->orig_pmd))) { + spin_unlock(ptl); + return 0; + } + + swp_entry =3D pmd_to_swp_entry(vmf->orig_pmd); + page =3D pfn_swap_entry_to_page(swp_entry); + vmf->page =3D page; + vmf->pte =3D NULL; + if (trylock_page(vmf->page)) { + get_page(page); + spin_unlock(ptl); + ret =3D page_pgmap(page)->ops->migrate_to_ram(vmf); + unlock_page(vmf->page); + put_page(page); + } else { + spin_unlock(ptl); + } + + return ret; +} + /* * always: directly stall for all thp allocations * defer: wake kswapd and fail if not immediately available diff --git a/mm/memory.c b/mm/memory.c index 92fd18a5d8d1..6c87f043eea1 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -6152,8 +6152,10 @@ static vm_fault_t __handle_mm_fault(struct vm_area_s= truct *vma, vmf.orig_pmd =3D pmdp_get_lockless(vmf.pmd); =20 if (unlikely(is_swap_pmd(vmf.orig_pmd))) { - VM_BUG_ON(thp_migration_supported() && - !is_pmd_migration_entry(vmf.orig_pmd)); + if (is_device_private_entry( + pmd_to_swp_entry(vmf.orig_pmd))) + return do_huge_pmd_device_private(&vmf); + if (is_pmd_migration_entry(vmf.orig_pmd)) pmd_migration_entry_wait(mm, vmf.pmd); return 0; --=20 2.50.1 From nobody Sat Oct 4 22:37:10 2025 Received: from NAM11-BN8-obe.outbound.protection.outlook.com (mail-bn8nam11on2044.outbound.protection.outlook.com [40.107.236.44]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id BCB6C2E540C for ; Tue, 12 Aug 2025 02:41:09 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.236.44 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1754966472; cv=fail; b=ogmZx6Hkx9AcV2s1zZTFi3ZS/z+zFBmL9sTTMwN947CeLSNouuOQ434VLdWHaeGp+x3r1F7xdeOSX3MSa059thB18WAd5eFvYsXp0H2tDdJDQGHBY9PUyBX+g0sBdNtd6NpFV7a7HvtNUkvJFJhZdh64ATbu6I5c7xfKmerddBQ= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1754966472; c=relaxed/simple; bh=2JAg2qDhkLxa1WqwVi6pYHreqqIkahKCdAj6v7bf99g=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: Content-Type:MIME-Version; b=WQQNHtlMlOCRdrWD1pGH3TUY/Xp1Ysj2bl4WgYsrWoRvV0cQiOziVSZQP0VjuSk2dX7ZDHbf2FcSM7qDRCX3yc6xdfC8oPNeSHVlamDajP8y1u7fGcEi99Ftmq/a3K3I6ovYicHKoWFSMdEn76jQDZ8TeKPw2teN/tTAj3trT38= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com; spf=fail smtp.mailfrom=nvidia.com; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b=gyMbyOpZ; arc=fail smtp.client-ip=40.107.236.44 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="gyMbyOpZ" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=HQ+uJwB8fxPpLpt7T2alc/R9qjcwPLwzSL96J6Ps50LVqfLwpYFtDq2zu44nEAFfOaQTTfzlLBBL7HSOOOOShCApDc6ZQOl3WtH+udtLkk7pYlPDZ4um/pdIWdube4MsR5zjCWv2ph6CWURfdtFBErH1dhaVQS8EFJh8au4fUGzbSax7SQNeL7SN86E4ETgurVRcU+OXjWYIekxCjyB6w+TlLuRHVpQ9MpUNzkpNNwOWbnRoSQUz+NtMbXx12pNfkNFav3VS7zEmfd+G7mpZnHEQQKlV03bDbmpSwUXqnYxD70FkqBtTEVLjGN9oYSFwNlPePQF6Rz2chFCQLyprEA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=7mPSJ6qvXlKqstThTE0pxKrOaVskUrNQDivi0wl0aXw=; b=MWdtL6sow80yfHtbVRQwo3l/dhPlAHtwHcrcis/ah/zToVgm4Gqi70yE2zP3LJ3TinKvb5wlSLEKIzNXuo75edG0zkD7JY/0YzohtUyuMAmSAttTKLXSgIojcrFxtswBwAjw8fth3tkHJsyIWLbrBTyuU0vP2++dHjpaUacxm4UhpGnSSmvIPXTE6RWfe/G5AFlCThz0A/WGOdBvNIdID+ORxB3J+vPjmdga++GCr+w4ZP9Z80Xb3jtKL9QmIRVfPqcnvW/2eh29k9115fa12M5ni7ewucHfAJKwygSuNe2Ty+/ip5WC5TYKXEHwZFg9z/yFPsJ+kfaf9MYLLaxq/w== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=7mPSJ6qvXlKqstThTE0pxKrOaVskUrNQDivi0wl0aXw=; b=gyMbyOpZyDfh+EL8E4wwWYOThXRBkNhHL+3hcZGx3NHnaQPS105aw1Dft2BgVxnF6Zz+jtVIfVSS65yij8xYTr2c+OtS8gOVnPE3e1LT+Gn/dHnY5qE6Un4f5jLfkS23tjGvcYs8ltzxnmalqmblZ6Qu+yWyUnl7ibx3M9/vyK59hixQKvjXIpg9gUiH3wgv8nbvZzowEwiVg0IMlPA2xCHPD8NHlh1jMioDMVXORpH1+GmsQEcUcWiQOOgAiTV8MBuy5YK+APBc/o3w79u8MvUUz9BPuW1hxFSJt1+/sn49IXJtC7OypURESCCWJtsLh4/Oxw/Bsl26EzufG+vOkQ== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from PH8PR12MB7277.namprd12.prod.outlook.com (2603:10b6:510:223::13) by CY8PR12MB7172.namprd12.prod.outlook.com (2603:10b6:930:5b::7) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9009.18; Tue, 12 Aug 2025 02:41:00 +0000 Received: from PH8PR12MB7277.namprd12.prod.outlook.com ([fe80::3a4:70ea:ff05:1251]) by PH8PR12MB7277.namprd12.prod.outlook.com ([fe80::3a4:70ea:ff05:1251%5]) with mapi id 15.20.9009.018; Tue, 12 Aug 2025 02:40:58 +0000 From: Balbir Singh To: dri-devel@lists.freedesktop.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Balbir Singh , Andrew Morton , David Hildenbrand , Zi Yan , Joshua Hahn , Rakie Kim , Byungchul Park , Gregory Price , Ying Huang , Alistair Popple , Oscar Salvador , Lorenzo Stoakes , Baolin Wang , "Liam R. Howlett" , Nico Pache , Ryan Roberts , Dev Jain , Barry Song , Lyude Paul , Danilo Krummrich , David Airlie , Simona Vetter , Ralph Campbell , =?UTF-8?q?Mika=20Penttil=C3=A4?= , Matthew Brost , Francois Dugast Subject: [v3 05/11] lib/test_hmm: test cases and support for zone device private THP Date: Tue, 12 Aug 2025 12:40:30 +1000 Message-ID: <20250812024036.690064-6-balbirs@nvidia.com> X-Mailer: git-send-email 2.50.1 In-Reply-To: <20250812024036.690064-1-balbirs@nvidia.com> References: <20250812024036.690064-1-balbirs@nvidia.com> Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: SY6PR01CA0072.ausprd01.prod.outlook.com (2603:10c6:10:ea::23) To PH8PR12MB7277.namprd12.prod.outlook.com (2603:10b6:510:223::13) Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: PH8PR12MB7277:EE_|CY8PR12MB7172:EE_ X-MS-Office365-Filtering-Correlation-Id: 0c30b656-80fa-43ae-3569-08ddd949ab42 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|366016|376014|7416014|1800799024; X-Microsoft-Antispam-Message-Info: =?utf-8?B?Rng2dllEaVZ4UEduUHU2ZGV6enZJN1hmTlh1NmV5NkdZMDgyL1ByUU92SGhl?= =?utf-8?B?a3ZBTUNYVDQxbEJTTldlaDBwN29Ya0ZGZzRzVm1lRjQrdWxnMXM5Tlh3Sm0y?= =?utf-8?B?c2dVWWdzVFRrblp6ZjhmZEU5dU5TTVhYTUZSV0lHYVg0WHhHdmNUb1M2ZHAx?= =?utf-8?B?ekJldEV3OXNhTzh5WWxubENtV3VRVDNUUEttR0ppU1owcGxQL05KaExZaisy?= =?utf-8?B?REJmSFRLbk1FUVVURUF0aTRnSHg1RGNKQ3gzWkV1Z056SDFvaVhocTZteUFM?= =?utf-8?B?VktEVXB3Q1FGZjB5NlRvSHNOUGdIcnFUZE85MnJva3lnZCtqSW9TR3k2L3Az?= =?utf-8?B?SHNtOW1TaFdocG55b0hISW14MjgrVUwwNzVBVlFILzlUNy9xbEhiVTZVcWE0?= =?utf-8?B?NUFGQmwxOUhyZWM3VlZncUd1VTN3cEhCR1FsSXk0RUlTN3pEOEZwMUR0TGdM?= =?utf-8?B?K3F1SXgreUx3djBkVjRraWdkZ2YwUVhKLzFrZHZRUUJHcGF2TUJzR0U3THBp?= =?utf-8?B?RHFHWGd2NTlrQlQzbjR6NGRKRnN0eU1wcy9xR3BRT2tGdm1QUU1DOGt2MkQx?= =?utf-8?B?cU4yU1k3QlEyODlsRHNpQWZCcWYrcC8vSmNYeW9xNnhYYmc1VEpvb3AzVTh5?= =?utf-8?B?aVZmaEROMUpmTk16NjIzRCtGL0NrQzRxVE9jRFBDVklUWko2ZWhKQ0JvV3JU?= =?utf-8?B?Zko3eStlaGQ3VHlxODJSaGdsWVpSbnVSMmFhejFlTmRBNS9aQ3Y4QXFGMFQ0?= =?utf-8?B?WWdNK0MzSldBL0MrdjNJaUVMZ2xkR0VmcTVGbTVvTUZWRy9qZTkrTDh2UVF2?= =?utf-8?B?bXpOc3BJajh1N2JabW4zMmtLdGxyOFJxOHZLeEpsdDhGU3RzdUZsbVpnYWZl?= =?utf-8?B?VkNsbVFIM0VHVnk1MDd1VnFkVjIySmZWUVdrYzdNbXYxU2ZWU01aTmpqSEtk?= =?utf-8?B?VVkwVjB6aHFVcWxheWk3KzRjTGt6TUk2Z2Z6bXZwclBRWjk4NFhzVWd5VTBO?= =?utf-8?B?UFg5U0ZyNnQwMFRjVkNFUEdKend3WllSeml3OXhpNFE3ZExmWWdpSjhrcUd5?= =?utf-8?B?TXgxbG9FeVpaVGJNZC9ibVMyYjlJNkVIeEF6dWpWMit1VnpxTFhiSWRtN1Ey?= =?utf-8?B?empqQkk5SWZmR0xiMW83SmdVZUplL3NXZmN5KyttemtoNmNmbkFkYW5hRmRB?= =?utf-8?B?TDBUbVZJYXhvMGVMbWUzUXN2MEdBZ0dhUFB6cHlGRFVaOC9MbjkrM2tPZEZh?= =?utf-8?B?Z0hiY1FKcVg0SEU0Q3VpWUFNSWNSc1JwcHNwdHdlSUNVc2ZsWnFNYnY0QmNM?= =?utf-8?B?dXlNT3hRcTVJQVZrZnNHeTZROGhyVWpHeGVyT3hJZ2doNVFDSW91SkdQZDBB?= =?utf-8?B?cm5DWDNsdkpXeEtLQWdLWHpmLzQ3OGlNOGVFYzJuWHpQMktxMENUb3NadVdk?= =?utf-8?B?Y0xsUUIzOUdXV3d6QVVrSDUvVkgzcml3L3VNOG9tTmxSb2VwWkFjZjAwRmhV?= =?utf-8?B?QnVjYWNiZU1iYWhuT0hTaFI2WVZCaWN2ZWtqcVJ4RFlWSEx6dkxiZHFGbGdj?= =?utf-8?B?TFcyd2FabTdKcDlkKzlqUEtzVGdvSFB2c1M3Q3BnOU5xTnBsVlAwVlBNeFNr?= =?utf-8?B?dmVnWDdHSThFNWlFY0VYRTgxdkhJU2dBR2hJRUVJQWludk9Cc1FtL2RTUEhE?= =?utf-8?B?SStpc1FXcnkzUVRFYmNtRU1UcmRFY1Rub20rVGVESE9MNFkwKzl3RHdWaWRw?= =?utf-8?B?em5TaTVnQzZQTHJZY05icWxYYUo5MUZyVUZVMHVNTDgxU1BLeVQ1dXJtNFNk?= =?utf-8?B?dmxnSHU0SWVMU0JWeDJjVHBud0xmQ1YxZklaR1EzeWtXczZkUDN2OHR2dXV3?= =?utf-8?B?V3B5UFViQTVrNVYvRzJqVEl0ZnEvaFZTcllGZmNmeDE3eVU3L0tyMTNkVkNj?= =?utf-8?Q?2AOucBL2bB4=3D?= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:PH8PR12MB7277.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(366016)(376014)(7416014)(1800799024);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?QlZXdW1NUndDemFjQTh3QmljT1E0SHhxWDBzMVRWUEdpWWxNTlQreGxTN25G?= =?utf-8?B?SUdLdTRqMG9lZnk0Rit5VHo0cEd4cGZuUWI5YnZzUWtHVko5a0NHaHl0YWhZ?= =?utf-8?B?SVAzRDlJSDdFY3crSXBzZXhGV05zUy9hSzNrVnFRVXlZNkp4QjlNd0ZjVzRJ?= =?utf-8?B?TkU3T3FmNXNWZDRlR1JWSnVPenkxUGVMaGxKNG9NL25HVkVJbEdmVzh1aDFn?= =?utf-8?B?YzJPN05YL09tYmhOQVl0TlExZFoydm5CZzY0NlpIUFU0aUNyMEt3UVdYeW9P?= =?utf-8?B?amgyUnVuR1NScGtNWWNwS2tkNThpME9KNmdzeE5ZNHJHSWJ2czc2N3lMU3U3?= =?utf-8?B?UlJoTUkvdjBhd05zZS9aZC9VSWpFcnNSRW5lQ2xsVnE1cmtqMTFRdzNhaTI5?= =?utf-8?B?SmUzUXF3d2kzOU1GNmtWZktlTk9YTzBuaEFobnVCNjQ2WXRSMnhCOHpsbHEr?= =?utf-8?B?Vjg3Q25ERVhJT1d6YUJjRmcwMmZ0bitvZHV4a1ZZRElscExRWDIzR1h1Mzdn?= =?utf-8?B?SlVlUEhLMGkxRDBVdVovbHRtTGIzM3pPeVgvdHQ1QnFOeXp3WUJldDR5Zi9n?= =?utf-8?B?L2ExQWRIV2VucC9sVWpEZGpYVGljVXBkZTRFbFhOOER1em5TSEV5dTRkMllw?= =?utf-8?B?WGtXdUt5bCt0c0VhNzRuY0RUL2ZmNkw2eHJOOWlvdFQ5bHZYMmRhN0txYnlv?= =?utf-8?B?Rm1WWmdZUEdjUWsyWG1UZFZ2Q29XNElKbnJmOFYvTzhaemJ1NCtvakd4djNq?= =?utf-8?B?U2kvTGVqSCt1SGpZK2dKdVl3TDdaNzRrOHVxMC9kT0IyMTBmTkpIVXZQT3FI?= =?utf-8?B?L0t6N1VuY1BFTDEvRXova1d6RGR2THVEL3k0anI0UDJXa2FYV2E5Ry9Yc3N1?= =?utf-8?B?RjFsYURtT2VNc0dWdEFvVGtoZS91VnlwS2kxMmYrR3lPUG5BQ0trdnZNU01q?= =?utf-8?B?YzJwcE5DRWtXQlhpL1dkc3ZJeGRtWnMra2UxWUhpYU5pMXRnZGN5bWJ6ZGla?= =?utf-8?B?dWQrN2xJcFpKRVZuem0yMVhEMURtdnZZSkVxNTRyWnlWZmNjeDBTSE45c3Nj?= =?utf-8?B?YkF5YytFU1JvWXR1N3I1aVlYbTJ2aHBKNFJxQng4SWYzZWVIY2Z3cys2WFdl?= =?utf-8?B?bWQ1bUFKOFg4eE9oTUc0M1hRY2t6T0tpZHJjb3IyM2tKYmR1TWdUUFpMM251?= =?utf-8?B?UWM4N2VrWVIyUW1CTWlHUDhTOVBQNDdoR2RXSlkyWlNOQ2ZoOGwxZDBuaWJ2?= =?utf-8?B?cW9ZU2JBcjhhOVBkVkJvVW5USENCeUVOVS80L1A3UWFDUHFQOHk1VDU1ZU9R?= =?utf-8?B?MFFtOHF3TDN3MUdVVGtrY2VIdlRtTWJLM1NGSU5UczVvdkd1aHhDTmpmVTY0?= =?utf-8?B?UlNndTNlUElKRU5lcGp5WjUxL3ZwWmZjWC9DTldZdmZTelIrSXlaQk10UjBK?= =?utf-8?B?VXBOaXFkMk8vTEhyUUR3L3BMdTRCMG1PRFhzNEV0N25ibUFOQ1ZQUXJHUlp1?= =?utf-8?B?aTVDdmFJaUl2UitBMjIxQVJSTFBvMHREZ0F1Y05XM3ZCVFI4MHN0YmVaVVNH?= =?utf-8?B?Nm5sN2FYYzdVM0w1Z1FnWU9nbitiRmExQ1BXQWg4MndNY2s2RG1IdzdGV01a?= =?utf-8?B?VFlZZWNPM2Q5dnFZRDBzcDRGc1BWaHBCcUF1NWZxa0t4OUx5Y2d5TllJOVJu?= =?utf-8?B?dUExLzd6QkhDZVpQQ3RvbkNZVHhDQThrVG85WEU4TDM0Z2phaUZ0S0ZuNnRL?= =?utf-8?B?azFEMkRYTklXYVpOL2xhRWVzSW1wNUtaNjVjWTJhNlY0VHVzN3FSZlR0SmM4?= =?utf-8?B?K2o2cUcwVVdoWkpGaU85cnk5MkcrYTVZaVZ1LzJEZ2h5OGU0Q0tvZXNqQzJm?= =?utf-8?B?eVVvejJya2xxM1g3MDB0c0xXeTh0Qm5IaWdxd2ZvWXg1R2JmWU9KWE1hYjYz?= =?utf-8?B?ZUxtTTgwSmhrdGxPTWIzNG90YjAvV1d5dDdMMHJ6V0xWTGJmYkVSV01CZW9p?= =?utf-8?B?Q2kzR2F5Vml1TjRaLzRPM0RTTVNPSzlHY2YzQkdBdWJYUnpGbG1kYklBY3NK?= =?utf-8?B?TkFkU043N1Z3OFNZdlUvRlNObDEvY09YTDNwSUE0emNTMzllQXppNFVMN2JI?= =?utf-8?Q?8fOebuk1X0pkxSnfBo+lL6mrH?= X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: 0c30b656-80fa-43ae-3569-08ddd949ab42 X-MS-Exchange-CrossTenant-AuthSource: PH8PR12MB7277.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 12 Aug 2025 02:40:58.8873 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: 1Vl4VAP+HZj9agdrt0qKCHLJavIXjsugrPpojpRHuxuHYW+Tj21Dt4C1i/EEG8zQIYIl2FOodTYTktnirzXG0g== X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY8PR12MB7172 Enhance the hmm test driver (lib/test_hmm) with support for THP pages. A new pool of free_folios() has now been added to the dmirror device, which can be allocated when a request for a THP zone device private page is made. Add compound page awareness to the allocation function during normal migration and fault based migration. These routines also copy folio_nr_pages() when moving data between system memory and device memory. args.src and args.dst used to hold migration entries are now dynamically allocated (as they need to hold HPAGE_PMD_NR entries or more). Split and migrate support will be added in future patches in this series. Cc: Andrew Morton Cc: David Hildenbrand Cc: Zi Yan Cc: Joshua Hahn Cc: Rakie Kim Cc: Byungchul Park Cc: Gregory Price Cc: Ying Huang Cc: Alistair Popple Cc: Oscar Salvador Cc: Lorenzo Stoakes Cc: Baolin Wang Cc: "Liam R. Howlett" Cc: Nico Pache Cc: Ryan Roberts Cc: Dev Jain Cc: Barry Song Cc: Lyude Paul Cc: Danilo Krummrich Cc: David Airlie Cc: Simona Vetter Cc: Ralph Campbell Cc: Mika Penttil=C3=A4 Cc: Matthew Brost Cc: Francois Dugast Signed-off-by: Balbir Singh --- include/linux/memremap.h | 12 ++ lib/test_hmm.c | 366 +++++++++++++++++++++++++++++++-------- 2 files changed, 303 insertions(+), 75 deletions(-) diff --git a/include/linux/memremap.h b/include/linux/memremap.h index a0723b35eeaa..0c5141a7d58c 100644 --- a/include/linux/memremap.h +++ b/include/linux/memremap.h @@ -169,6 +169,18 @@ static inline bool folio_is_device_private(const struc= t folio *folio) return is_device_private_page(&folio->page); } =20 +static inline void *folio_zone_device_data(const struct folio *folio) +{ + VM_WARN_ON_FOLIO(!folio_is_device_private(folio), folio); + return folio->page.zone_device_data; +} + +static inline void folio_set_zone_device_data(struct folio *folio, void *d= ata) +{ + VM_WARN_ON_FOLIO(!folio_is_device_private(folio), folio); + folio->page.zone_device_data =3D data; +} + static inline bool is_pci_p2pdma_page(const struct page *page) { return IS_ENABLED(CONFIG_PCI_P2PDMA) && diff --git a/lib/test_hmm.c b/lib/test_hmm.c index 297f1e034045..d814056151d0 100644 --- a/lib/test_hmm.c +++ b/lib/test_hmm.c @@ -119,6 +119,7 @@ struct dmirror_device { unsigned long calloc; unsigned long cfree; struct page *free_pages; + struct folio *free_folios; spinlock_t lock; /* protects the above */ }; =20 @@ -492,7 +493,7 @@ static int dmirror_write(struct dmirror *dmirror, struc= t hmm_dmirror_cmd *cmd) } =20 static int dmirror_allocate_chunk(struct dmirror_device *mdevice, - struct page **ppage) + struct page **ppage, bool is_large) { struct dmirror_chunk *devmem; struct resource *res =3D NULL; @@ -572,20 +573,45 @@ static int dmirror_allocate_chunk(struct dmirror_devi= ce *mdevice, pfn_first, pfn_last); =20 spin_lock(&mdevice->lock); - for (pfn =3D pfn_first; pfn < pfn_last; pfn++) { + for (pfn =3D pfn_first; pfn < pfn_last; ) { struct page *page =3D pfn_to_page(pfn); =20 + if (is_large && IS_ALIGNED(pfn, HPAGE_PMD_NR) + && (pfn + HPAGE_PMD_NR <=3D pfn_last)) { + page->zone_device_data =3D mdevice->free_folios; + mdevice->free_folios =3D page_folio(page); + pfn +=3D HPAGE_PMD_NR; + continue; + } + page->zone_device_data =3D mdevice->free_pages; mdevice->free_pages =3D page; + pfn++; } + + ret =3D 0; if (ppage) { - *ppage =3D mdevice->free_pages; - mdevice->free_pages =3D (*ppage)->zone_device_data; - mdevice->calloc++; + if (is_large) { + if (!mdevice->free_folios) { + ret =3D -ENOMEM; + goto err_unlock; + } + *ppage =3D folio_page(mdevice->free_folios, 0); + mdevice->free_folios =3D (*ppage)->zone_device_data; + mdevice->calloc +=3D HPAGE_PMD_NR; + } else if (mdevice->free_pages) { + *ppage =3D mdevice->free_pages; + mdevice->free_pages =3D (*ppage)->zone_device_data; + mdevice->calloc++; + } else { + ret =3D -ENOMEM; + goto err_unlock; + } } +err_unlock: spin_unlock(&mdevice->lock); =20 - return 0; + return ret; =20 err_release: mutex_unlock(&mdevice->devmem_lock); @@ -598,10 +624,13 @@ static int dmirror_allocate_chunk(struct dmirror_devi= ce *mdevice, return ret; } =20 -static struct page *dmirror_devmem_alloc_page(struct dmirror_device *mdevi= ce) +static struct page *dmirror_devmem_alloc_page(struct dmirror *dmirror, + bool is_large) { struct page *dpage =3D NULL; struct page *rpage =3D NULL; + unsigned int order =3D is_large ? HPAGE_PMD_ORDER : 0; + struct dmirror_device *mdevice =3D dmirror->mdevice; =20 /* * For ZONE_DEVICE private type, this is a fake device so we allocate @@ -610,49 +639,55 @@ static struct page *dmirror_devmem_alloc_page(struct = dmirror_device *mdevice) * data and ignore rpage. */ if (dmirror_is_private_zone(mdevice)) { - rpage =3D alloc_page(GFP_HIGHUSER); + rpage =3D folio_page(folio_alloc(GFP_HIGHUSER, order), 0); if (!rpage) return NULL; } spin_lock(&mdevice->lock); =20 - if (mdevice->free_pages) { + if (is_large && mdevice->free_folios) { + dpage =3D folio_page(mdevice->free_folios, 0); + mdevice->free_folios =3D dpage->zone_device_data; + mdevice->calloc +=3D 1 << order; + spin_unlock(&mdevice->lock); + } else if (!is_large && mdevice->free_pages) { dpage =3D mdevice->free_pages; mdevice->free_pages =3D dpage->zone_device_data; mdevice->calloc++; spin_unlock(&mdevice->lock); } else { spin_unlock(&mdevice->lock); - if (dmirror_allocate_chunk(mdevice, &dpage)) + if (dmirror_allocate_chunk(mdevice, &dpage, is_large)) goto error; } =20 - zone_device_page_init(dpage); + zone_device_folio_init(page_folio(dpage), order); dpage->zone_device_data =3D rpage; return dpage; =20 error: if (rpage) - __free_page(rpage); + __free_pages(rpage, order); return NULL; } =20 static void dmirror_migrate_alloc_and_copy(struct migrate_vma *args, struct dmirror *dmirror) { - struct dmirror_device *mdevice =3D dmirror->mdevice; const unsigned long *src =3D args->src; unsigned long *dst =3D args->dst; unsigned long addr; =20 - for (addr =3D args->start; addr < args->end; addr +=3D PAGE_SIZE, - src++, dst++) { + for (addr =3D args->start; addr < args->end; ) { struct page *spage; struct page *dpage; struct page *rpage; + bool is_large =3D *src & MIGRATE_PFN_COMPOUND; + int write =3D (*src & MIGRATE_PFN_WRITE) ? MIGRATE_PFN_WRITE : 0; + unsigned long nr =3D 1; =20 if (!(*src & MIGRATE_PFN_MIGRATE)) - continue; + goto next; =20 /* * Note that spage might be NULL which is OK since it is an @@ -662,17 +697,45 @@ static void dmirror_migrate_alloc_and_copy(struct mig= rate_vma *args, if (WARN(spage && is_zone_device_page(spage), "page already in device spage pfn: 0x%lx\n", page_to_pfn(spage))) + goto next; + + dpage =3D dmirror_devmem_alloc_page(dmirror, is_large); + if (!dpage) { + struct folio *folio; + unsigned long i; + unsigned long spfn =3D *src >> MIGRATE_PFN_SHIFT; + struct page *src_page; + + if (!is_large) + goto next; + + if (!spage && is_large) { + nr =3D HPAGE_PMD_NR; + } else { + folio =3D page_folio(spage); + nr =3D folio_nr_pages(folio); + } + + for (i =3D 0; i < nr && addr < args->end; i++) { + dpage =3D dmirror_devmem_alloc_page(dmirror, false); + rpage =3D BACKING_PAGE(dpage); + rpage->zone_device_data =3D dmirror; + + *dst =3D migrate_pfn(page_to_pfn(dpage)) | write; + src_page =3D pfn_to_page(spfn + i); + + if (spage) + copy_highpage(rpage, src_page); + else + clear_highpage(rpage); + src++; + dst++; + addr +=3D PAGE_SIZE; + } continue; - - dpage =3D dmirror_devmem_alloc_page(mdevice); - if (!dpage) - continue; + } =20 rpage =3D BACKING_PAGE(dpage); - if (spage) - copy_highpage(rpage, spage); - else - clear_highpage(rpage); =20 /* * Normally, a device would use the page->zone_device_data to @@ -684,10 +747,42 @@ static void dmirror_migrate_alloc_and_copy(struct mig= rate_vma *args, =20 pr_debug("migrating from sys to dev pfn src: 0x%lx pfn dst: 0x%lx\n", page_to_pfn(spage), page_to_pfn(dpage)); - *dst =3D migrate_pfn(page_to_pfn(dpage)); - if ((*src & MIGRATE_PFN_WRITE) || - (!spage && args->vma->vm_flags & VM_WRITE)) - *dst |=3D MIGRATE_PFN_WRITE; + + *dst =3D migrate_pfn(page_to_pfn(dpage)) | write; + + if (is_large) { + int i; + struct folio *folio =3D page_folio(dpage); + *dst |=3D MIGRATE_PFN_COMPOUND; + + if (folio_test_large(folio)) { + for (i =3D 0; i < folio_nr_pages(folio); i++) { + struct page *dst_page =3D + pfn_to_page(page_to_pfn(rpage) + i); + struct page *src_page =3D + pfn_to_page(page_to_pfn(spage) + i); + + if (spage) + copy_highpage(dst_page, src_page); + else + clear_highpage(dst_page); + src++; + dst++; + addr +=3D PAGE_SIZE; + } + continue; + } + } + + if (spage) + copy_highpage(rpage, spage); + else + clear_highpage(rpage); + +next: + src++; + dst++; + addr +=3D PAGE_SIZE; } } =20 @@ -734,14 +829,17 @@ static int dmirror_migrate_finalize_and_map(struct mi= grate_vma *args, const unsigned long *src =3D args->src; const unsigned long *dst =3D args->dst; unsigned long pfn; + const unsigned long start_pfn =3D start >> PAGE_SHIFT; + const unsigned long end_pfn =3D end >> PAGE_SHIFT; =20 /* Map the migrated pages into the device's page tables. */ mutex_lock(&dmirror->mutex); =20 - for (pfn =3D start >> PAGE_SHIFT; pfn < (end >> PAGE_SHIFT); pfn++, - src++, dst++) { + for (pfn =3D start_pfn; pfn < end_pfn; pfn++, src++, dst++) { struct page *dpage; void *entry; + int nr, i; + struct page *rpage; =20 if (!(*src & MIGRATE_PFN_MIGRATE)) continue; @@ -750,13 +848,25 @@ static int dmirror_migrate_finalize_and_map(struct mi= grate_vma *args, if (!dpage) continue; =20 - entry =3D BACKING_PAGE(dpage); - if (*dst & MIGRATE_PFN_WRITE) - entry =3D xa_tag_pointer(entry, DPT_XA_TAG_WRITE); - entry =3D xa_store(&dmirror->pt, pfn, entry, GFP_ATOMIC); - if (xa_is_err(entry)) { - mutex_unlock(&dmirror->mutex); - return xa_err(entry); + if (*dst & MIGRATE_PFN_COMPOUND) + nr =3D folio_nr_pages(page_folio(dpage)); + else + nr =3D 1; + + WARN_ON_ONCE(end_pfn < start_pfn + nr); + + rpage =3D BACKING_PAGE(dpage); + VM_WARN_ON(folio_nr_pages(page_folio(rpage)) !=3D nr); + + for (i =3D 0; i < nr; i++) { + entry =3D folio_page(page_folio(rpage), i); + if (*dst & MIGRATE_PFN_WRITE) + entry =3D xa_tag_pointer(entry, DPT_XA_TAG_WRITE); + entry =3D xa_store(&dmirror->pt, pfn + i, entry, GFP_ATOMIC); + if (xa_is_err(entry)) { + mutex_unlock(&dmirror->mutex); + return xa_err(entry); + } } } =20 @@ -829,31 +939,66 @@ static vm_fault_t dmirror_devmem_fault_alloc_and_copy= (struct migrate_vma *args, unsigned long start =3D args->start; unsigned long end =3D args->end; unsigned long addr; + unsigned int order =3D 0; + int i; =20 - for (addr =3D start; addr < end; addr +=3D PAGE_SIZE, - src++, dst++) { + for (addr =3D start; addr < end; ) { struct page *dpage, *spage; =20 spage =3D migrate_pfn_to_page(*src); - if (!spage || !(*src & MIGRATE_PFN_MIGRATE)) - continue; + if (!spage || !(*src & MIGRATE_PFN_MIGRATE)) { + addr +=3D PAGE_SIZE; + goto next; + } =20 if (WARN_ON(!is_device_private_page(spage) && - !is_device_coherent_page(spage))) - continue; + !is_device_coherent_page(spage))) { + addr +=3D PAGE_SIZE; + goto next; + } + spage =3D BACKING_PAGE(spage); - dpage =3D alloc_page_vma(GFP_HIGHUSER_MOVABLE, args->vma, addr); - if (!dpage) - continue; - pr_debug("migrating from dev to sys pfn src: 0x%lx pfn dst: 0x%lx\n", - page_to_pfn(spage), page_to_pfn(dpage)); + order =3D folio_order(page_folio(spage)); =20 + if (order) + dpage =3D folio_page(vma_alloc_folio(GFP_HIGHUSER_MOVABLE, + order, args->vma, addr), 0); + else + dpage =3D alloc_page_vma(GFP_HIGHUSER_MOVABLE, args->vma, addr); + + /* Try with smaller pages if large allocation fails */ + if (!dpage && order) { + dpage =3D alloc_page_vma(GFP_HIGHUSER_MOVABLE, args->vma, addr); + if (!dpage) + return VM_FAULT_OOM; + order =3D 0; + } + + pr_debug("migrating from sys to dev pfn src: 0x%lx pfn dst: 0x%lx\n", + page_to_pfn(spage), page_to_pfn(dpage)); lock_page(dpage); xa_erase(&dmirror->pt, addr >> PAGE_SHIFT); copy_highpage(dpage, spage); *dst =3D migrate_pfn(page_to_pfn(dpage)); if (*src & MIGRATE_PFN_WRITE) *dst |=3D MIGRATE_PFN_WRITE; + if (order) + *dst |=3D MIGRATE_PFN_COMPOUND; + + for (i =3D 0; i < (1 << order); i++) { + struct page *src_page; + struct page *dst_page; + + src_page =3D pfn_to_page(page_to_pfn(spage) + i); + dst_page =3D pfn_to_page(page_to_pfn(dpage) + i); + + xa_erase(&dmirror->pt, addr >> PAGE_SHIFT); + copy_highpage(dst_page, src_page); + } +next: + addr +=3D PAGE_SIZE << order; + src +=3D 1 << order; + dst +=3D 1 << order; } return 0; } @@ -879,11 +1024,14 @@ static int dmirror_migrate_to_system(struct dmirror = *dmirror, unsigned long size =3D cmd->npages << PAGE_SHIFT; struct mm_struct *mm =3D dmirror->notifier.mm; struct vm_area_struct *vma; - unsigned long src_pfns[32] =3D { 0 }; - unsigned long dst_pfns[32] =3D { 0 }; struct migrate_vma args =3D { 0 }; unsigned long next; int ret; + unsigned long *src_pfns; + unsigned long *dst_pfns; + + src_pfns =3D kvcalloc(PTRS_PER_PTE, sizeof(*src_pfns), GFP_KERNEL | __GFP= _NOFAIL); + dst_pfns =3D kvcalloc(PTRS_PER_PTE, sizeof(*dst_pfns), GFP_KERNEL | __GFP= _NOFAIL); =20 start =3D cmd->addr; end =3D start + size; @@ -902,7 +1050,7 @@ static int dmirror_migrate_to_system(struct dmirror *d= mirror, ret =3D -EINVAL; goto out; } - next =3D min(end, addr + (ARRAY_SIZE(src_pfns) << PAGE_SHIFT)); + next =3D min(end, addr + (PTRS_PER_PTE << PAGE_SHIFT)); if (next > vma->vm_end) next =3D vma->vm_end; =20 @@ -912,7 +1060,7 @@ static int dmirror_migrate_to_system(struct dmirror *d= mirror, args.start =3D addr; args.end =3D next; args.pgmap_owner =3D dmirror->mdevice; - args.flags =3D dmirror_select_device(dmirror); + args.flags =3D dmirror_select_device(dmirror) | MIGRATE_VMA_SELECT_COMPO= UND; =20 ret =3D migrate_vma_setup(&args); if (ret) @@ -928,6 +1076,8 @@ static int dmirror_migrate_to_system(struct dmirror *d= mirror, out: mmap_read_unlock(mm); mmput(mm); + kvfree(src_pfns); + kvfree(dst_pfns); =20 return ret; } @@ -939,12 +1089,12 @@ static int dmirror_migrate_to_device(struct dmirror = *dmirror, unsigned long size =3D cmd->npages << PAGE_SHIFT; struct mm_struct *mm =3D dmirror->notifier.mm; struct vm_area_struct *vma; - unsigned long src_pfns[32] =3D { 0 }; - unsigned long dst_pfns[32] =3D { 0 }; struct dmirror_bounce bounce; struct migrate_vma args =3D { 0 }; unsigned long next; int ret; + unsigned long *src_pfns =3D NULL; + unsigned long *dst_pfns =3D NULL; =20 start =3D cmd->addr; end =3D start + size; @@ -955,6 +1105,18 @@ static int dmirror_migrate_to_device(struct dmirror *= dmirror, if (!mmget_not_zero(mm)) return -EINVAL; =20 + ret =3D -ENOMEM; + src_pfns =3D kvcalloc(PTRS_PER_PTE, sizeof(*src_pfns), + GFP_KERNEL | __GFP_NOFAIL); + if (!src_pfns) + goto free_mem; + + dst_pfns =3D kvcalloc(PTRS_PER_PTE, sizeof(*dst_pfns), + GFP_KERNEL | __GFP_NOFAIL); + if (!dst_pfns) + goto free_mem; + + ret =3D 0; mmap_read_lock(mm); for (addr =3D start; addr < end; addr =3D next) { vma =3D vma_lookup(mm, addr); @@ -962,7 +1124,7 @@ static int dmirror_migrate_to_device(struct dmirror *d= mirror, ret =3D -EINVAL; goto out; } - next =3D min(end, addr + (ARRAY_SIZE(src_pfns) << PAGE_SHIFT)); + next =3D min(end, addr + (PTRS_PER_PTE << PAGE_SHIFT)); if (next > vma->vm_end) next =3D vma->vm_end; =20 @@ -972,7 +1134,8 @@ static int dmirror_migrate_to_device(struct dmirror *d= mirror, args.start =3D addr; args.end =3D next; args.pgmap_owner =3D dmirror->mdevice; - args.flags =3D MIGRATE_VMA_SELECT_SYSTEM; + args.flags =3D MIGRATE_VMA_SELECT_SYSTEM | + MIGRATE_VMA_SELECT_COMPOUND; ret =3D migrate_vma_setup(&args); if (ret) goto out; @@ -992,7 +1155,7 @@ static int dmirror_migrate_to_device(struct dmirror *d= mirror, */ ret =3D dmirror_bounce_init(&bounce, start, size); if (ret) - return ret; + goto free_mem; mutex_lock(&dmirror->mutex); ret =3D dmirror_do_read(dmirror, start, end, &bounce); mutex_unlock(&dmirror->mutex); @@ -1003,11 +1166,14 @@ static int dmirror_migrate_to_device(struct dmirror= *dmirror, } cmd->cpages =3D bounce.cpages; dmirror_bounce_fini(&bounce); - return ret; + goto free_mem; =20 out: mmap_read_unlock(mm); mmput(mm); +free_mem: + kfree(src_pfns); + kfree(dst_pfns); return ret; } =20 @@ -1200,6 +1366,7 @@ static void dmirror_device_evict_chunk(struct dmirror= _chunk *chunk) unsigned long i; unsigned long *src_pfns; unsigned long *dst_pfns; + unsigned int order =3D 0; =20 src_pfns =3D kvcalloc(npages, sizeof(*src_pfns), GFP_KERNEL | __GFP_NOFAI= L); dst_pfns =3D kvcalloc(npages, sizeof(*dst_pfns), GFP_KERNEL | __GFP_NOFAI= L); @@ -1215,13 +1382,25 @@ static void dmirror_device_evict_chunk(struct dmirr= or_chunk *chunk) if (WARN_ON(!is_device_private_page(spage) && !is_device_coherent_page(spage))) continue; + + order =3D folio_order(page_folio(spage)); spage =3D BACKING_PAGE(spage); - dpage =3D alloc_page(GFP_HIGHUSER_MOVABLE | __GFP_NOFAIL); + if (src_pfns[i] & MIGRATE_PFN_COMPOUND) { + dpage =3D folio_page(folio_alloc(GFP_HIGHUSER_MOVABLE, + order), 0); + } else { + dpage =3D alloc_page(GFP_HIGHUSER_MOVABLE | __GFP_NOFAIL); + order =3D 0; + } + + /* TODO Support splitting here */ lock_page(dpage); - copy_highpage(dpage, spage); dst_pfns[i] =3D migrate_pfn(page_to_pfn(dpage)); if (src_pfns[i] & MIGRATE_PFN_WRITE) dst_pfns[i] |=3D MIGRATE_PFN_WRITE; + if (order) + dst_pfns[i] |=3D MIGRATE_PFN_COMPOUND; + folio_copy(page_folio(dpage), page_folio(spage)); } migrate_device_pages(src_pfns, dst_pfns, npages); migrate_device_finalize(src_pfns, dst_pfns, npages); @@ -1234,7 +1413,12 @@ static void dmirror_remove_free_pages(struct dmirror= _chunk *devmem) { struct dmirror_device *mdevice =3D devmem->mdevice; struct page *page; + struct folio *folio; + =20 + for (folio =3D mdevice->free_folios; folio; folio =3D folio_zone_device_d= ata(folio)) + if (dmirror_page_to_chunk(folio_page(folio, 0)) =3D=3D devmem) + mdevice->free_folios =3D folio_zone_device_data(folio); for (page =3D mdevice->free_pages; page; page =3D page->zone_device_data) if (dmirror_page_to_chunk(page) =3D=3D devmem) mdevice->free_pages =3D page->zone_device_data; @@ -1265,6 +1449,7 @@ static void dmirror_device_remove_chunks(struct dmirr= or_device *mdevice) mdevice->devmem_count =3D 0; mdevice->devmem_capacity =3D 0; mdevice->free_pages =3D NULL; + mdevice->free_folios =3D NULL; kfree(mdevice->devmem_chunks); mdevice->devmem_chunks =3D NULL; } @@ -1378,18 +1563,30 @@ static void dmirror_devmem_free(struct page *page) { struct page *rpage =3D BACKING_PAGE(page); struct dmirror_device *mdevice; + struct folio *folio =3D page_folio(rpage); + unsigned int order =3D folio_order(folio); =20 - if (rpage !=3D page) - __free_page(rpage); + if (rpage !=3D page) { + if (order) + __free_pages(rpage, order); + else + __free_page(rpage); + rpage =3D NULL; + } =20 mdevice =3D dmirror_page_to_device(page); spin_lock(&mdevice->lock); =20 /* Return page to our allocator if not freeing the chunk */ if (!dmirror_page_to_chunk(page)->remove) { - mdevice->cfree++; - page->zone_device_data =3D mdevice->free_pages; - mdevice->free_pages =3D page; + mdevice->cfree +=3D 1 << order; + if (order) { + page->zone_device_data =3D mdevice->free_folios; + mdevice->free_folios =3D page_folio(page); + } else { + page->zone_device_data =3D mdevice->free_pages; + mdevice->free_pages =3D page; + } } spin_unlock(&mdevice->lock); } @@ -1397,11 +1594,10 @@ static void dmirror_devmem_free(struct page *page) static vm_fault_t dmirror_devmem_fault(struct vm_fault *vmf) { struct migrate_vma args =3D { 0 }; - unsigned long src_pfns =3D 0; - unsigned long dst_pfns =3D 0; struct page *rpage; struct dmirror *dmirror; - vm_fault_t ret; + vm_fault_t ret =3D 0; + unsigned int order, nr; =20 /* * Normally, a device would use the page->zone_device_data to point to @@ -1412,21 +1608,38 @@ static vm_fault_t dmirror_devmem_fault(struct vm_fa= ult *vmf) dmirror =3D rpage->zone_device_data; =20 /* FIXME demonstrate how we can adjust migrate range */ + order =3D folio_order(page_folio(vmf->page)); + nr =3D 1 << order; + + /* + * Consider a per-cpu cache of src and dst pfns, but with + * large number of cpus that might not scale well. + */ + args.start =3D ALIGN_DOWN(vmf->address, (PAGE_SIZE << order)); args.vma =3D vmf->vma; - args.start =3D vmf->address; - args.end =3D args.start + PAGE_SIZE; - args.src =3D &src_pfns; - args.dst =3D &dst_pfns; + args.end =3D args.start + (PAGE_SIZE << order); + + nr =3D (args.end - args.start) >> PAGE_SHIFT; + args.src =3D kcalloc(nr, sizeof(unsigned long), GFP_KERNEL); + args.dst =3D kcalloc(nr, sizeof(unsigned long), GFP_KERNEL); args.pgmap_owner =3D dmirror->mdevice; args.flags =3D dmirror_select_device(dmirror); args.fault_page =3D vmf->page; =20 + if (!args.src || !args.dst) { + ret =3D VM_FAULT_OOM; + goto err; + } + + if (order) + args.flags |=3D MIGRATE_VMA_SELECT_COMPOUND; + if (migrate_vma_setup(&args)) return VM_FAULT_SIGBUS; =20 ret =3D dmirror_devmem_fault_alloc_and_copy(&args, dmirror); if (ret) - return ret; + goto err; migrate_vma_pages(&args); /* * No device finalize step is needed since @@ -1434,7 +1647,10 @@ static vm_fault_t dmirror_devmem_fault(struct vm_fau= lt *vmf) * invalidated the device page table. */ migrate_vma_finalize(&args); - return 0; +err: + kfree(args.src); + kfree(args.dst); + return ret; } =20 static const struct dev_pagemap_ops dmirror_devmem_ops =3D { @@ -1465,7 +1681,7 @@ static int dmirror_device_init(struct dmirror_device = *mdevice, int id) return ret; =20 /* Build a list of free ZONE_DEVICE struct pages */ - return dmirror_allocate_chunk(mdevice, NULL); + return dmirror_allocate_chunk(mdevice, NULL, false); } =20 static void dmirror_device_remove(struct dmirror_device *mdevice) --=20 2.50.1 From nobody Sat Oct 4 22:37:10 2025 Received: from NAM11-BN8-obe.outbound.protection.outlook.com (mail-bn8nam11on2044.outbound.protection.outlook.com [40.107.236.44]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D4DBE2E4277 for ; Tue, 12 Aug 2025 02:41:05 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.236.44 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1754966468; cv=fail; b=BT48EB31hKK8aWoeEx8LZ84mYrYiGWI6HgXZVLxoMgGXDtcphVzPCMS8MLsiTzrybUw7MiLzTCwl24yyPEkssKwo2galrFy/AAIdLoTeWm0mrQHs6wovVlyA5zmogH8Nt7t409BpiLmLWzg0r1uOuI74l6x6hv5afHAdOW+fgA4= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1754966468; c=relaxed/simple; bh=zR1nixp391nyLINcCwvQWYLSHSWBgIPTfKWJxN5KhfM=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: Content-Type:MIME-Version; b=YngE28IsYmUUpad5sa4+pGOQ4uVqooNbvcMgXr79OQlvcorAMfT0HECAym27n4r6EldMfqXmTTsq8ZPjA+WleczpLbMM1VFMkkAqpYa1ctvhnH/mcI160SraI4pRN+xP849POC3CcUerMq4NHMKgCgiHTJovAV6av8hxBJT13QE= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com; spf=fail smtp.mailfrom=nvidia.com; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b=Kv0FXmGV; arc=fail smtp.client-ip=40.107.236.44 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="Kv0FXmGV" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=tgejnzdwvoCtjafmnDL54nn15pysBUx9wGyCe6pHAtQgiSVuCwaqV/MQbYip+SakaUQPDmuNj2GZLx7ejD7SAcDF2u/3RxfqkEWK7WXY/gEug7K7o04QWvMh3PWV07KQ+/u7B05z9RgH5bR+YdsDkGUFHnPr0YXJ/+/akTR8ky7QB3QxM/WG+i//3fXb1zF7W4h2Y7wK6vcqjsvIoADrrRblIxs/7n8u+bPucPCXeWuSFxt0aYh0qLJfluQucXivEVZq1fk9CAr3tvnYHOiKl+B9aLG5qWLI+GiNxUuZeJKZgrgHS+Phl0ye8PH7RRE325GgRa782YMpe1mCNpk75A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=apiVlSwc3q1Kek6WNczydp5lBFByNEaPx5lw0APHYQo=; b=WTmLCNvotaKxV9WhukDAlFeKMbkLc3805kfF/xnwfa8C4bHqj/aGkaxGJZ6MqWPSE5tu7joUQ/BelVC0cTmZHAfKBBYF18pk+rurQihd8R1w4ynd+2cNkiZUbAlFyTrAeoky/0xv40IbXRq+ioNcmx1MAyaXgGajlaKWDQmPFKpF5g1vWBRNFKLeK3BJChYsMYc+mcij8ijp3InBdOYG1WedzF+FlDyBBuOM56GSrI/Jpixifx7zb6FQSlULWS9kANq005NeRgt/pQB/UZgYmWS+oCf3oTkfmQ6z4UooWqNqjve+T5V7LBTqnZmkzuDddZBRi4QYwArnMgpnWdbjtg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=apiVlSwc3q1Kek6WNczydp5lBFByNEaPx5lw0APHYQo=; b=Kv0FXmGV9+FcFjmtYiB7vz/0QMP09XG95e+gLCmnCnT1ibxdw2vIaeAzUgRcHx3n1Hnip71OehzOL95/8hkMi+dEWiXznp/qqA6rgc3gZ1KHgVGoF+jD166hSRwVFtBPke5AUxYPjlh1lr0aI47B8plfJXAWJ+tU2gihnulBcBReSJNqb8RT0HGg/KQrcvMXgxvVnH++Us1QUDVNptVxrSIhEPOkpsOx9BGwuq2eEFnwUpwUdGrt5UmfAPeLiUcV7BrRWwiMbVxrtxjiigmrVghNQAKc31iyK0QZKt6BivOH7jUduNwEDcULYTIv62cge/pMcnUJDVufBs+5pFXW6A== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from PH8PR12MB7277.namprd12.prod.outlook.com (2603:10b6:510:223::13) by CY8PR12MB7172.namprd12.prod.outlook.com (2603:10b6:930:5b::7) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9009.18; Tue, 12 Aug 2025 02:41:03 +0000 Received: from PH8PR12MB7277.namprd12.prod.outlook.com ([fe80::3a4:70ea:ff05:1251]) by PH8PR12MB7277.namprd12.prod.outlook.com ([fe80::3a4:70ea:ff05:1251%5]) with mapi id 15.20.9009.018; Tue, 12 Aug 2025 02:41:02 +0000 From: Balbir Singh To: dri-devel@lists.freedesktop.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Balbir Singh , Andrew Morton , David Hildenbrand , Zi Yan , Joshua Hahn , Rakie Kim , Byungchul Park , Gregory Price , Ying Huang , Alistair Popple , Oscar Salvador , Lorenzo Stoakes , Baolin Wang , "Liam R. Howlett" , Nico Pache , Ryan Roberts , Dev Jain , Barry Song , Lyude Paul , Danilo Krummrich , David Airlie , Simona Vetter , Ralph Campbell , =?UTF-8?q?Mika=20Penttil=C3=A4?= , Matthew Brost , Francois Dugast Subject: [v3 06/11] mm/memremap: add folio_split support Date: Tue, 12 Aug 2025 12:40:31 +1000 Message-ID: <20250812024036.690064-7-balbirs@nvidia.com> X-Mailer: git-send-email 2.50.1 In-Reply-To: <20250812024036.690064-1-balbirs@nvidia.com> References: <20250812024036.690064-1-balbirs@nvidia.com> Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: SY5PR01CA0035.ausprd01.prod.outlook.com (2603:10c6:10:1f8::12) To PH8PR12MB7277.namprd12.prod.outlook.com (2603:10b6:510:223::13) Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: PH8PR12MB7277:EE_|CY8PR12MB7172:EE_ X-MS-Office365-Filtering-Correlation-Id: 65b46859-b08d-4b66-2276-08ddd949ada5 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|366016|376014|7416014|1800799024; X-Microsoft-Antispam-Message-Info: =?utf-8?B?ZkNwbUlzVzUyeXhoUEsvQ2VmZjVvNjE1N2NLbkVSOXpBVlB5TzV6VDR1VzZG?= =?utf-8?B?MTdvYkpCd21pSFN3RkFRVjErVWpMTHM4YmRvdkJaTHFZNzFXcEZtaW04YUlx?= =?utf-8?B?cFl2aXhkZzZRRlNKb0ZsK2J3MHpkNmxBVmphblAxOEVQOWxmc3pFYVFSdEFo?= =?utf-8?B?eXptNlF2SlQrZnRRTXN6ZGNnV01JQjMrSlNnL3VJelc3MjJyRmgxTUNCeTA0?= =?utf-8?B?ZERPSmdCOVhtVjdBYWFSaGJ4RmN1ZmtOTDUrUFU0QVZIWkZiYW1KRmZPcmJu?= =?utf-8?B?MEJTUXhQT3puMlVBNXMyaHkxeklqUVpGK01xK09QQVdHYTFsZCttMXJITWsx?= =?utf-8?B?Yk1BdGgxVjBXc1R3Tm9HYkhPbC90cENmZ1Y4M1RVQUZYMU1KRnFHL3ViRllZ?= =?utf-8?B?ZGV1ZVhVRGRqNytoZWNlKzNwYkF5QTFOL3Q4aGJBYVFFN2VQK2ZaU281MkZa?= =?utf-8?B?LytsRUlYOVJ4b09jRElOOTJVUldXUDdCZlZaYlFyTkZCK1IzcS9iMUxBRDBY?= =?utf-8?B?MW9WbExqKzgyUW1sbG9oaHM3aFpaSTJWTGtIZXdWMzEva2VvOXpmTGQ4ZEhT?= =?utf-8?B?akk1cUtrN2lyOFFsSU92UHF5Mnh2SVp5RXhVZEtnTmplWUxuUWdvTlhqRnpN?= =?utf-8?B?cHZxR3MwWm5pTUVLTEx3ckwzQ2R5VDV1VVpwZys4d2R2ZDUyOVN4czhFang5?= =?utf-8?B?VE9TdWNoYjFWWjBoR3JNNjgvVHRFaEVlL01jRlJPQ2tJMElzMiszV2dtTW5O?= =?utf-8?B?Y083MWVsTlNJZ0ZKaTlNbXhyV05YeWd3MjkxLzJMSHBqc1ROQWJsRnJJMDVj?= =?utf-8?B?emJtTlppRjRTenZpQlkxWjZTOWtENWRyTzZUUWVCSVFKb0RBaC9CZmNBMGly?= =?utf-8?B?UjhsZ3RUemd5dFNndjdZWGJjSmNRWlFTZytCOXV2dVhDUU5MeWdSV0xVRGNN?= =?utf-8?B?aGFjNkEzb0RJazF6d3hvMFRjaGlHaVNKNlVHK01HR0FFZzZPNUZwSjdTODgr?= =?utf-8?B?cWdwcmdyTFNiZ2ltOWlzb2J2WXUwUXpoYk9JZnhDc3pNVmJES3NYS3hMZDF6?= =?utf-8?B?L0x3ZzFsaGFxTkhVeWo4RlN3VHNXMllIWEZwR2pweUhIZFozUDA0SG1iQjN0?= =?utf-8?B?MGhEQmVUeWpkWDRsOGFsTWNhM0NyYlNPM1hMUjVzZFVUOXBhNlc2a1lGZHJT?= =?utf-8?B?Wi95UlBkdytNSXdWZFhWdHBWYjAvZWVySFQybFFkSHVNSWhuK014UDU2QUNW?= =?utf-8?B?VERlQXo5RVNvbU9wWTgvTXVsQit6aUNQckJvWStoY2hxWDVOZnh4NkNvcmxI?= =?utf-8?B?S0NWblowZDcrTFpTQlJCQzZsV25pbXZOUXA1M08xTTArQmlkTXVPcllDUUpE?= =?utf-8?B?Ny93OUtoYVlJUU5Qa3hpbENQenVsSnZWSCtJTzRPd2wzc3FpQ1pCTlRaM1g4?= =?utf-8?B?Tk9CcU5idmFXNWVtTVhmZ2xhQXYxTnJRcHI5THNhbkdzc1ZkMWdpWDU0aUxh?= =?utf-8?B?TkJzWGs4U1B2Yk1SbDMvZC83V09UYnllemVVZTdIT3FGUXB2Z244MmRJS1M4?= =?utf-8?B?RG5DMnpTL09IVGNQK29pOHpqeWxMYmlFQ1hhRFBqN0JPRkFpMFl5N2szOHI3?= =?utf-8?B?WS83RGlRK0FzUmtOK0VFd1lrSUhyeU5LSTBXTGlRNUhlL0xUOUxVempQcU95?= =?utf-8?B?YmExNVdDSytVcldvcFJmZ0VncXd1dnFVYktQTDNpaDROcGRMMDBzcmRRMGZF?= =?utf-8?B?U3hiSDVmbDZoS1g5NTBTRzhsNUpNMkVUbXA4ZUpTR2E1TGJuaTV0WEIvWjho?= =?utf-8?B?MHhKb292bkNsRWFDMTNhZ0hrOU1ZeWUyVERDMUJpM3BGeG5WeXludG1uMkZK?= =?utf-8?B?UFliekZKRmQxcEY0THFyVjZlbWN4TEN0UkhHNlJzMkJxYTlmbVlFT2NGdnQy?= =?utf-8?Q?oiRXvhaBLpM=3D?= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:PH8PR12MB7277.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(366016)(376014)(7416014)(1800799024);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?VlFZdHVRYTg0eFV0bWxCa3FWVUdqQUlZSjBzOXFVTWRTQTJjSzJyQW16NkFD?= =?utf-8?B?VmxsUFdlYitrcnRlaDd0L0s1YjhrMnNDTmhrOEx2cHEwcWZCVDdLbDlWQlpR?= =?utf-8?B?UVpSZndlbUNNSSsvK2MvNG9hTG1qS0Z3Um43aXdMdjZOdy8zV1gwL0dGQUEz?= =?utf-8?B?b21qbjducDZ4YisyY29NYlU2VkJpSHQvNnBJKzNUSXkyQldQMmErR0xCcHJJ?= =?utf-8?B?dFErcktxUXI1VEIrWkM2Sk9ON1k2YWRVb2NpQm1wd3ZhS3BrSFMrVjFSLzNr?= =?utf-8?B?OHZrMmdlNDBqSkVVWFRZTGMvcUM3UXV3Y2s2M3B5c3VzY25KQjlHSm9vMjJ2?= =?utf-8?B?bkx4bklZa2tpRjlQc2V3VVhiYVplTlV5L3diOUJQUysvRDk1YVhEbXFPYVJx?= =?utf-8?B?emhlcXF1eVpyV2x2RWZwSWF0SkptZGFKYitWVzdFbnl6YWZlalo5empBdlhI?= =?utf-8?B?TE50aVI0ditoMzZVeG9KUFF4T25acVVuaFFkZWg0WFIvbW0rN3ZGZ2Z5QzZR?= =?utf-8?B?Tk5DaVRGaFF3dDV1UUZLS0F3bFJja2g1RDdBbEV0ZllZR0xOUzBLNDJveU44?= =?utf-8?B?UUx4dGRXYUh1ZGY5UVRlVjZZc2xxZXdFM3lMcjFoMUU1aWRvZVlvRVFHTG5y?= =?utf-8?B?R2V4OEVCam9pK2plS3k2U0ExVGhxcG1uVThobUlJbWJnMWdPdGVMNkxoUWt6?= =?utf-8?B?RHRWQ3EyL0NTMWF1YUpGek16OGlIenZkZzRmOFBHd2ZQV3VEQzAxckJ6VUpN?= =?utf-8?B?bDZndCt3ODFWYk5zVGk0VnkvenpqRm01Tm9JSGp4eDV6SERDdHluMXJvVVlD?= =?utf-8?B?NUkvNnVnalN5SUZmVFpnM0lMVjVrRXBhMzgwM1FsdmdUZlBudms2d1VDaUZn?= =?utf-8?B?dDFxOXpEMTV4bFJTeldUakVTV2ZNOGN3RjBWV2Z3SHQ0bFR5MlRmTnJxbXMv?= =?utf-8?B?RFRKckVvN0tJcXRtQW92a2dkS1U5b1FUM2N6MW5jNGx5WHdOUGFPM0VBSXFh?= =?utf-8?B?bXFaQ25xZnA1ajMrMyt3NlZXY3ZId1o3bnRVMCtDaXNIalVoUUZJUHBxSjVk?= =?utf-8?B?S2NlMVM4ZzQ2N3ZJSmtVaHl1L3FRWjhBejJUa3E3azRPbjlYRjQxNnRxRzhh?= =?utf-8?B?elI5aWNxdnpJRTFqRk8rdkFPZm56WUk0T0ZpWVdMNnNyaGRqQVNnL1c4cXoy?= =?utf-8?B?Ui8va2VFejJuY2VZbktRMGxNaUFMOE9XbzJnb051T1YzMjU2RG1OK1V6UTFu?= =?utf-8?B?bTBjNHZ5SnpXU1RzN3dWUWM1ZC94eXEyNEI1YkJQUzh2cjM0VUlaU1RVTUlG?= =?utf-8?B?R1pXVFBBaXozVXNEOVdZMTNYVHVhclZQdDlyWW91Mks0eVk0RGYweHZpcHJv?= =?utf-8?B?TDJPOWVIMFNNVEJjaUI2eWdtVlZGbmpMY09raDdkWHc4Y0VzZytyYWdNNmlR?= =?utf-8?B?SjhqT08wZzh2WWFHWGk0WSszeU41cUtRYzYyTWJOUzZpVythWmY1RGcwOVZP?= =?utf-8?B?TFd2OXlqOWlOWHFaKzBZbUo1OFdtNGRmUmlFRnhmRXV0c1JndUhXckx1T2hU?= =?utf-8?B?Q0VyMlp5OTVvUnVkRVVaT2dvZGI2Vm5waXJwUGYyL2VDVFVHYW10QUk5L1lx?= =?utf-8?B?YzgzOWI3eGs2bHdvWDFHdHVIMDJlYlNWNTFHdDhHdVc1NGtRcW1FMFIxQkdG?= =?utf-8?B?TkJkZ3dYMTEreG82VDNUU3dVNldhdTMwc2g5NmQzS0MwbVlITlR3T2tUR3JF?= =?utf-8?B?N0hOLy9ucUkrb0RZRldETjZlckJRQlc2REo0czRWdHN2VmdKbkF4V1N4c0ZW?= =?utf-8?B?bFZGU3IyRHhHUVdTdkx3TzZQYW9JalNSZ2NQb3EySGZ3SGc3dnlHTDZSQXlB?= =?utf-8?B?b0NzT21SOTBxbTB3Y3VnZE11eE1IZmRMTmM4ZFhiNmhrcVRNbVF6TWFFVUJ0?= =?utf-8?B?Qmc1cXM4dHVGVm12SlRQemdVL1h5MTNrazdhZ01Dcm5XZU9lWnR2QkFaaWJw?= =?utf-8?B?THFhUHd4eEp2YXV5ZWNDR2dtMWg0RDNEZDVKRFd2Nm5YVEdHSXJEV0hubWs3?= =?utf-8?B?RVpGQ29sRUQ4OXZxWWZTbUFKb2EvVHorWUtWM1ZuWEt4R3ZxM2NxblRib0Jm?= =?utf-8?Q?R/il9pLIeh+CfUcPP7csgHt3R?= X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: 65b46859-b08d-4b66-2276-08ddd949ada5 X-MS-Exchange-CrossTenant-AuthSource: PH8PR12MB7277.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 12 Aug 2025 02:41:02.8908 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: Yubcm+FX3kT7uWGwzKK+vHD0Ar0toou9oVRdH/p6z4ZCTxFtxnsysbAUSWNOTP1QxBy55D5xsYoa7aje6bo/NA== X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY8PR12MB7172 When a zone device page is split (via huge pmd folio split). The driver callback for folio_split is invoked to let the device driver know that the folio size has been split into a smaller order. Provide a default implementation for drivers that do not provide this callback that copies the pgmap and mapping fields for the split folios. Update the HMM test driver to handle the split. Cc: Andrew Morton Cc: David Hildenbrand Cc: Zi Yan Cc: Joshua Hahn Cc: Rakie Kim Cc: Byungchul Park Cc: Gregory Price Cc: Ying Huang Cc: Alistair Popple Cc: Oscar Salvador Cc: Lorenzo Stoakes Cc: Baolin Wang Cc: "Liam R. Howlett" Cc: Nico Pache Cc: Ryan Roberts Cc: Dev Jain Cc: Barry Song Cc: Lyude Paul Cc: Danilo Krummrich Cc: David Airlie Cc: Simona Vetter Cc: Ralph Campbell Cc: Mika Penttil=C3=A4 Cc: Matthew Brost Cc: Francois Dugast Signed-off-by: Balbir Singh --- include/linux/memremap.h | 29 +++++++++++++++++++++++++++++ include/linux/mm.h | 1 + lib/test_hmm.c | 35 +++++++++++++++++++++++++++++++++++ mm/huge_memory.c | 2 +- 4 files changed, 66 insertions(+), 1 deletion(-) diff --git a/include/linux/memremap.h b/include/linux/memremap.h index 0c5141a7d58c..20f4b5ebbc93 100644 --- a/include/linux/memremap.h +++ b/include/linux/memremap.h @@ -100,6 +100,13 @@ struct dev_pagemap_ops { */ int (*memory_failure)(struct dev_pagemap *pgmap, unsigned long pfn, unsigned long nr_pages, int mf_flags); + + /* + * Used for private (un-addressable) device memory only. + * This callback is used when a folio is split into + * a smaller folio + */ + void (*folio_split)(struct folio *head, struct folio *tail); }; =20 #define PGMAP_ALTMAP_VALID (1 << 0) @@ -229,6 +236,23 @@ static inline void zone_device_page_init(struct page *= page) zone_device_folio_init(folio, 0); } =20 +static inline void zone_device_private_split_cb(struct folio *original_fol= io, + struct folio *new_folio) +{ + if (folio_is_device_private(original_folio)) { + if (!original_folio->pgmap->ops->folio_split) { + if (new_folio) { + new_folio->pgmap =3D original_folio->pgmap; + new_folio->page.mapping =3D + original_folio->page.mapping; + } + } else { + original_folio->pgmap->ops->folio_split(original_folio, + new_folio); + } + } +} + #else static inline void *devm_memremap_pages(struct device *dev, struct dev_pagemap *pgmap) @@ -263,6 +287,11 @@ static inline unsigned long memremap_compat_align(void) { return PAGE_SIZE; } + +static inline void zone_device_private_split_cb(struct folio *original_fol= io, + struct folio *new_folio) +{ +} #endif /* CONFIG_ZONE_DEVICE */ =20 static inline void put_dev_pagemap(struct dev_pagemap *pgmap) diff --git a/include/linux/mm.h b/include/linux/mm.h index ceaa780a703a..f755afe533e5 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -1185,6 +1185,7 @@ static inline struct folio *virt_to_folio(const void = *x) void __folio_put(struct folio *folio); =20 void split_page(struct page *page, unsigned int order); +void prep_compound_page(struct page *page, unsigned int order); void folio_copy(struct folio *dst, struct folio *src); int folio_mc_copy(struct folio *dst, struct folio *src); =20 diff --git a/lib/test_hmm.c b/lib/test_hmm.c index d814056151d0..14dbce719896 100644 --- a/lib/test_hmm.c +++ b/lib/test_hmm.c @@ -1653,9 +1653,44 @@ static vm_fault_t dmirror_devmem_fault(struct vm_fau= lt *vmf) return ret; } =20 +static void dmirror_devmem_folio_split(struct folio *head, struct folio *t= ail) +{ + struct page *rpage =3D BACKING_PAGE(folio_page(head, 0)); + struct page *rpage_tail; + struct folio *rfolio; + unsigned long offset =3D 0; + + if (!rpage) { + tail->page.zone_device_data =3D NULL; + return; + } + + rfolio =3D page_folio(rpage); + + if (tail =3D=3D NULL) { + folio_reset_order(rfolio); + rfolio->mapping =3D NULL; + folio_set_count(rfolio, 1); + return; + } + + offset =3D folio_pfn(tail) - folio_pfn(head); + + rpage_tail =3D folio_page(rfolio, offset); + tail->page.zone_device_data =3D rpage_tail; + rpage_tail->zone_device_data =3D rpage->zone_device_data; + clear_compound_head(rpage_tail); + rpage_tail->mapping =3D NULL; + + folio_page(tail, 0)->mapping =3D folio_page(head, 0)->mapping; + tail->pgmap =3D head->pgmap; + folio_set_count(page_folio(rpage_tail), 1); +} + static const struct dev_pagemap_ops dmirror_devmem_ops =3D { .page_free =3D dmirror_devmem_free, .migrate_to_ram =3D dmirror_devmem_fault, + .folio_split =3D dmirror_devmem_folio_split, }; =20 static int dmirror_device_init(struct dmirror_device *mdevice, int id) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 8888140e57a3..dc58081b661c 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -3922,7 +3922,6 @@ static int __folio_split(struct folio *folio, unsigne= d int new_order, =20 ret =3D __split_unmapped_folio(folio, new_order, split_at, &xas, mapping, uniform_split); - /* * Unfreeze after-split folios and put them back to the right * list. @folio should be kept frozon until page cache @@ -3973,6 +3972,7 @@ static int __folio_split(struct folio *folio, unsigne= d int new_order, __filemap_remove_folio(new_folio, NULL); folio_put_refs(new_folio, nr_pages); } + /* * Unfreeze @folio only after all page cache entries, which * used to point to it, have been updated with new folios. --=20 2.50.1 From nobody Sat Oct 4 22:37:10 2025 Received: from NAM02-SN1-obe.outbound.protection.outlook.com (mail-sn1nam02on2087.outbound.protection.outlook.com [40.107.96.87]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 72FF02E541B for ; Tue, 12 Aug 2025 02:41:11 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.96.87 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1754966473; cv=fail; b=A+nCjUlRnj804xXrfN99kBz4ujUkauHESCPmsfS38vmloAfQgG94rPi3MfCHQv7QYwVo/8iuBmbGHp1xcsiQBqMjzdRmS1IuwzG7u8Zhh7yf3JXuzXLoEIZwkv0UPTsadqZ2WVNB2/EyVWJ5QqrANeRFOXalmTIUjmzSuNlM0gM= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1754966473; c=relaxed/simple; bh=oG62AvrsBjZMKQtNe793ZunScOy/uYq58J/34N8Z/x0=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: Content-Type:MIME-Version; b=grNvh30AfetLdu13erREb7o1Jg8U+CbjA0CToPGwd/Q5mAFLgAr4F3zOnMIo7Yxa21JMeQdU59PRDKF1JTSDMq5cKPCQgjCG0U7BUGc79lyXIbdYUvE94P3H2DawrqQ/lv4BosGWLhvQiV68LCGZSlRh6/qiE6rZfg4sC2qqwsI= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com; spf=fail smtp.mailfrom=nvidia.com; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b=fuwfc53n; arc=fail smtp.client-ip=40.107.96.87 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="fuwfc53n" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=lN0qSHqkYkgIBmhtkmUqtiRA8NTgXkyZWy1Oi/KrBjHleflMMkug10qp39KsGvpGxZLrstEX+X/rcVveF5GRcSGZTwKHFFRDVfRYVF8eAxg7FIc9p4a/2BraFjs+tLWyTksKI1zFFasLbyRqEShVp2QBU1acLb/Fig3cFtcfDDqivse1wTC98VIG4cHCvh1xRBUFmswkCWORDlRVcdC8OlBC1WdGOsF/h9zpnGj/jzKSQrKIxT2AC+LHzog4FCibr9wQxom56QRxOzvd5aqHuL/NgJFlK2Oc2VQVEKjp6YFJoWvgNcG95xpGB9NrCKdh9DsjTN4MBZdkuccChf8n1Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=ar5uF/GQ6lEI7wxq78t4b6Aa3GN8D2TEWsnX3bw6rKI=; b=PgO+2P720vwsDdsWgNLu3n2dxdv+X3IlYLG20Wmqde7V+rYW7Q4YvcUE2lc5U4hjncvcK6w6cvOiG+vrUco7qq1bcB1FL6Ah7CRSvNUVqhTTTUOTlaBjOeMBKE+IFgGSuTJbmyL3FEJFoBo2LiT9Thds4/tWvzvNWa2ia2GUmb0GojUQmQrkk8bN4BjCjU5d5qOyUb3+mmRSI2zEofieRGjuBJBWQAnxRNT8iu1NyrIRcP/IQd+vhsDGBUfkR6sEYhtFiX87ghwjmXPmW5Sn5ZHEbHerBAYWlJPJrrlNroSnrjRFllXa8mXkJ7ibq2nk+wH03TE2yEwdMREHVypzYg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=ar5uF/GQ6lEI7wxq78t4b6Aa3GN8D2TEWsnX3bw6rKI=; b=fuwfc53nbgL13OlUoUzRk0alTJ6xN/mHsiTlpxOgZKjqr+IsSfdPMSrmtpMv4UlH+0YdQquoUaSwUOngBOqPERWM0YvyZy+Q0R1feYeeY8LoKyBd1bRLMK0Z3sGVucYoBGyC+m1t6JVqGo34pQaLjEEP61T2Dg8Saw3M2YguRYVqCkfDYAfyX+ZAgCrHs5Kst1gzmJR/6HI1mBOx+yEQA9FmXmuRPVdk8KMtkKQ8jaT2uNWrQWucg3u8M771m1MFeZNllPQCdGIdpBBLin0+DrZ63GOLrI36E4tONut2dl9srrf/JaGijVfk3teNL6zK3JGr1FXPKdQcrLtojUxuWg== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from PH8PR12MB7277.namprd12.prod.outlook.com (2603:10b6:510:223::13) by MN0PR12MB6317.namprd12.prod.outlook.com (2603:10b6:208:3c2::12) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9009.13; Tue, 12 Aug 2025 02:41:07 +0000 Received: from PH8PR12MB7277.namprd12.prod.outlook.com ([fe80::3a4:70ea:ff05:1251]) by PH8PR12MB7277.namprd12.prod.outlook.com ([fe80::3a4:70ea:ff05:1251%5]) with mapi id 15.20.9009.018; Tue, 12 Aug 2025 02:41:07 +0000 From: Balbir Singh To: dri-devel@lists.freedesktop.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Balbir Singh , Andrew Morton , David Hildenbrand , Zi Yan , Joshua Hahn , Rakie Kim , Byungchul Park , Gregory Price , Ying Huang , Alistair Popple , Oscar Salvador , Lorenzo Stoakes , Baolin Wang , "Liam R. Howlett" , Nico Pache , Ryan Roberts , Dev Jain , Barry Song , Lyude Paul , Danilo Krummrich , David Airlie , Simona Vetter , Ralph Campbell , =?UTF-8?q?Mika=20Penttil=C3=A4?= , Matthew Brost , Francois Dugast Subject: [v3 07/11] mm/thp: add split during migration support Date: Tue, 12 Aug 2025 12:40:32 +1000 Message-ID: <20250812024036.690064-8-balbirs@nvidia.com> X-Mailer: git-send-email 2.50.1 In-Reply-To: <20250812024036.690064-1-balbirs@nvidia.com> References: <20250812024036.690064-1-balbirs@nvidia.com> Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: SYBPR01CA0097.ausprd01.prod.outlook.com (2603:10c6:10:1::13) To PH8PR12MB7277.namprd12.prod.outlook.com (2603:10b6:510:223::13) Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: PH8PR12MB7277:EE_|MN0PR12MB6317:EE_ X-MS-Office365-Filtering-Correlation-Id: f584eb07-3938-413e-fdbb-08ddd949b041 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|376014|7416014|366016|1800799024; X-Microsoft-Antispam-Message-Info: =?utf-8?B?VUMrb3oyRWZENVFneVFFR25HQmkwcldHb2gvaHNWc1BydHN4UTkzL1Zud09O?= =?utf-8?B?TWZYU0dITkE0VW81RXV3SmpXQzhpak5COXNxRlFMVEk0V29HOFBwamFPM0d6?= =?utf-8?B?UkFUZ2VKTk5oMWpMdUFQZ3NMQ25JdGVkOG4zb3BMY2dwczRZNUhSVk93KzNo?= =?utf-8?B?ajdlL1N1aitWZ2RsT2tKWk1USUN6d2pSMEsxUU5FK2JaTmNNRFJyd05qeTlI?= =?utf-8?B?MlVTZVBNSUNuU29kaEM3cEJnUzRHWXZEOVhkUERUZXRlSFdvRHkwWXA1MTdp?= =?utf-8?B?c0ozOEZCSkszVUk3OFBKZXVxS3VQa0MyZ09EL2hmUnRXekx4dkIzOWFhQ1Ew?= =?utf-8?B?YjJjWXYrVXNKRnh2K1ZlWTk3OFMrSG5jWXRDSi9nRlFpQXNVcXBNN25ERXZn?= =?utf-8?B?K3p3WFBiNWwxQnJIaWdrMHhVeHh4aWtINStPT3NxZXJNWS81RXpobGpVT0xw?= =?utf-8?B?TUN1VW1nRmRlSWVWck4wSzZ0azVQVUQ4d2dGOGN2QmhMMHRBS09LR00wNmJu?= =?utf-8?B?ZDRHN29jK01NN3VwZWNSZnVJeTFLakV0aDBCSXl2S1FWb1E4cnMrVmt2N3JF?= =?utf-8?B?dFMyWWx4akIvbG1IbjVlV3IybURGK0NMM3BkU09HM2FSY1NzZ0V1N1FJSWNa?= =?utf-8?B?Qll3eUxkY2RFeVAxdFg0N0VCaG1ZSk0raG05NE94UTBlYXowMzZ5TWpaaHZT?= =?utf-8?B?ZkFsZnhyZkRHTjhONm41UXlpN0Y4aTN0YTg4cFlnNXlINGZ6RHpoS3NQQnlJ?= =?utf-8?B?bWlZMENCeFNKSzI4UEx3dlRNMjZOTjNIY09LelB6Sm9VS2ovL0laWnVvRVVQ?= =?utf-8?B?NkR0Tm9DRmlickZpRWR3NWFybVp6ZHBoR2FoS25sWlVCemxTclJIUUJzeWNt?= =?utf-8?B?WHFoM1UrL3hIT1NBMWNEQURTbm01eGFRVWZSSFRLZkcvOXdsUmxua2dCNVVV?= =?utf-8?B?NVBEY3VhRWZ6TmZKVUVBbWFOcVlNc3BRTU5QaWY1R2NOV29mT3BRb1VRekFD?= =?utf-8?B?OElFL3FrS1lITzJyZ2NsOFovTjZyQ3I4eFpTOGxUeWNqSzZtN1pSb3A3N2Ri?= =?utf-8?B?M0FRdVdYTnlDL3BHNjZnbHZYWEx1MFZnSFBjekxLaUovbURPNWFodHIxTElz?= =?utf-8?B?dDdkUFZ0Q0VPUVoxQVNEdmx2YmZpWVRYY1NzbWNORGNMWnEzQ1g5RjNOaWNQ?= =?utf-8?B?L0E2VXBjY1JvbXNLVDBkOS9hR2Z4UlpCWnl5eUExSGZyYnpYR3pucWQ2YnV0?= =?utf-8?B?ZFQ3NkJtdURrMy9HZjVxdEgrZTZ5WENmR2R5bVFVdmdqa3NOdGwrYWNWU0dt?= =?utf-8?B?cDV3bVFMK0gzN0dwRXVLZDlHaUhZU3h3QWxRS3Zxd2RjZHlUZHlyYXBneVd1?= =?utf-8?B?OG5yNVdKQTRhK0l3N2lWMFBoT0ZyRFBHenFrdUs4ZFV0ZDduMlRqdmJwaTMy?= =?utf-8?B?c2ZRZHJFVTlSY3Azd0U4ZjV4VC9JOWpjdDkrNmEwS1NjRkM3eXNpdFlNMmpN?= =?utf-8?B?bFFCQWlxNjdCZUlJeGRGWWhCVWliT2lXZ3RXMEprQ2VnU0Joc1M4dXAxVjVy?= =?utf-8?B?S2o0cytwM21Xd0pTcmQ2N3VTVUhCcGh0UUFyUDRNc2U2ZnpOdlpVaElrZXg2?= =?utf-8?B?VWVpRzliNlFHM3dBbDd6ck5PWC9STTJrWHNxd1B2dGkreWgyOG1sSkdFZWZt?= =?utf-8?B?UmZub2xYdHZzY3o2OVZ5VjdJaGtUVHhCc3IrZTdmd0x4RW9lQ05IZHErazNS?= =?utf-8?B?eTF4SVBKT0ZldnRFcS93MmsxODJqeHF3eHljZ1ZRN0Vxc2N4WXVwNWh2NmtJ?= =?utf-8?B?VjNCc1pQSVlaMmlDR3l1eVZJM2JiazlLUm9wQld6OStPZE9zVzQ3RGMvVFVu?= =?utf-8?B?U0c3K0hxck5ncXFDU2pWdUk4T2pxdEY5ZFJweW1ZN2IwMm9xc1FJWTdNWnRE?= =?utf-8?Q?kkTHehI6piA=3D?= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:PH8PR12MB7277.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(376014)(7416014)(366016)(1800799024);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?ZzFOVnlDOXNNOFFlMExmREJ1QWEwd1VadGozQmlETDk2bWZNTE9adlRqYXlo?= =?utf-8?B?Wnc2dWdKbk9kZ2xiTm5FVzQ1VGpkd2JqcVo3d215THlUSHJ4UDJiYVdjUHNR?= =?utf-8?B?ZU9aWnNRK1pJNGNFN1JWeDJPRThHemwwenhvT2NneGk5RE5WaWhPRFVzcXFL?= =?utf-8?B?Vkxtb1ZUclFBOXg2WUVYMVE0T09jV1cxUDV2QUpRS1dMdENSMFFlUHdvNFlW?= =?utf-8?B?eGVZUzM0OGZzcEk1Y0FjL2gxM29HQnVaeE1nVUdEMjZoRnRORU1sWE5oaGZx?= =?utf-8?B?ZDMxc0NPSExZYTdDbXJDN1N5SHpjRkZzZmUwYmRxQm9ZK2RXZXJ3SDQ2azVX?= =?utf-8?B?S2kwTnNjaFBXL2JkSDlFUU94TEppMzJSWGg4b0ZYaCtlbUd3SlhSelBqSlVl?= =?utf-8?B?TkN4Z1BZU2RKUHVVSHBIcCtmM0dha3VUM3NMaTFHL0lWVG1CSHU5RDg2b2VM?= =?utf-8?B?WU0rT3NlYXZkdVh0STRTdmorNzd3bkpjaTZZc3JTSEtoR2xoRG5ZM1FoeDJq?= =?utf-8?B?ZkQvdUdaYW1rbzFURHVta1ZFUFB0bmhleVlneXlUajVQNTAzYko5VFNMOWpj?= =?utf-8?B?bUtFb3JaaS9DTHRtWVhpVXVCWXM4eGdOUURpd1pBTkR6UlM0N3U0UHhmNjRB?= =?utf-8?B?U2hGTVdvbVlEVUM0QlZ3OUZUWGc2Qlg5N0xyZk82SkJpUFl1YVV6UUtxQjFR?= =?utf-8?B?NStGY1dZSjlOb21SUjRNUmdUem9FVU42RlI3M2NMbCtqZ0lwQVpGeTRTK0dY?= =?utf-8?B?WnFKSE03Z09SYWg2TnpNUFhvalRFZDFFaG8xRGhadzJyOEVaQm5lNGJtNlRY?= =?utf-8?B?Ni9uYjBjeCtzWlg4WGtnSStrU2lIMXdrU3dTcXJvK25xMXMrYTNqRVg2NXhQ?= =?utf-8?B?N1BPd3Q5K1dzSFB0clVUZmp3WHVnb1l4TkM4ZEx0ZTVhUFJxSU1MNVJ0SnlF?= =?utf-8?B?ZTdlU3BSQklLNU1aeXp2SFFSK1U4TWlFNHJZMG9RdGI4VkNxUk9kdXlYU1pP?= =?utf-8?B?a044YlpzOWFXSHppbFF0ZFVMRjRiSkxRclc5bzA0aFJHd3JSdHlPZFVTWW9q?= =?utf-8?B?Q2RaejE2b0twNEdvVlVxZWNPOXc0dnV3Y3E1SDdJNllyamQyRlREdWNvQ2JS?= =?utf-8?B?Y3lTM3RpdVloc2tLUFZJS1o0NEdBVlFzdERYeTE5Q2lGL2RNeWd1cm1XZW95?= =?utf-8?B?SE1NQXk3RlRKcnpvempPWGJ4S0w0VE9QbW1DSStib3NBQ3RVRDNQUlFHK3Jp?= =?utf-8?B?TU1KMkxDdkM3VXYyNFNjVkV4bldoYUZIbTlXd2N2QWs0WDBhcXd4NjFXWEtH?= =?utf-8?B?LzVFNjBVdVBnbUNWZ1RaWklqNU5zT1VWK1dGaTZUcXVpcHBUYjRKUVZjbDZI?= =?utf-8?B?T0hLbVViVHNqN1NKN3FOcEFQVDFjbERzYmtCTkFZWGsrQ2RmRmNwTXQ3TTB2?= =?utf-8?B?TDU4cGl5SmhUcnNTeXJoaEs5Vk4xUWJNZW84YnFFQjdDYTJpN0o1aDZVKzE5?= =?utf-8?B?eTN3SDZ0ak1VL3dIZytGTWdBUnI1Vngrb2VwVkFMdm1PNzh2Y3FJKzhSUVdn?= =?utf-8?B?ZjlZT2MrYnMrSVdCenVDL0dEQi9ldXkvVWNQRUtQek9wVjhLQU9DNkZIS2lp?= =?utf-8?B?ZDJqVmJGMytCWnlxenhIZUJLcmhlSXVsNlMrc20xRHpDZ1ZrMFd2bkVMYkZX?= =?utf-8?B?UUhkNUZsWnRGczlWZXV5RVcrZ0MrZTQ4NkxKbThoVlRhMEtXZ2pkUUJHZDJV?= =?utf-8?B?cE16NVVTZ2x4cDIrbFdnY2JRd3p1US9lWWxDOGpoWk1rYmkwOVFoNFl1RVZR?= =?utf-8?B?OGpLMUh4eGlOdWFBSkZSN1BReTBpVDUxaEovKzE5aDlPNU9lNlA5ZjM2cjlH?= =?utf-8?B?SDZxV2VnTGxjcWltUHQvWVU1WUxlM09VbVhWMlFmYUtORlBKdHNkMTlLL0NW?= =?utf-8?B?aHVpQmVUYk43RmlUTWwrTm9obXIwazZ4dGtralhlbGY3Q211NkhpeFJ1Mllw?= =?utf-8?B?VW10NE5tSktGZWhydVpEZHdRbzVSL3ZJTFpaUG5RSmljWHVTK1UvbnRPRHh4?= =?utf-8?B?T1Q0TEFPWGpTWGRQaC9obGMzRFVDeVJJbnhFZGRQejhLZEFOTXN1ekVNR0lr?= =?utf-8?Q?kW4zjupnhXEYnIUxVUgORlBLQ?= X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: f584eb07-3938-413e-fdbb-08ddd949b041 X-MS-Exchange-CrossTenant-AuthSource: PH8PR12MB7277.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 12 Aug 2025 02:41:07.1338 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: 1XNQS4+n1D1mZ2oixejFcSBwRxxQ5s4AercwY0f0LR4mvxi9FWk5U+uoJbtiJw5fDN9A7ao99JwyFdt4cC/GJw== X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN0PR12MB6317 Support splitting pages during THP zone device migration as needed. The common case that arises is that after setup, during migrate the destination might not be able to allocate MIGRATE_PFN_COMPOUND pages. Add a new routine migrate_vma_split_pages() to support the splitting of already isolated pages. The pages being migrated are already unmapped and marked for migration during setup (via unmap). folio_split() and __split_unmapped_folio() take additional isolated arguments, to avoid unmapping and remaping these pages and unlocking/putting the folio. Cc: Andrew Morton Cc: David Hildenbrand Cc: Zi Yan Cc: Joshua Hahn Cc: Rakie Kim Cc: Byungchul Park Cc: Gregory Price Cc: Ying Huang Cc: Alistair Popple Cc: Oscar Salvador Cc: Lorenzo Stoakes Cc: Baolin Wang Cc: "Liam R. Howlett" Cc: Nico Pache Cc: Ryan Roberts Cc: Dev Jain Cc: Barry Song Cc: Lyude Paul Cc: Danilo Krummrich Cc: David Airlie Cc: Simona Vetter Cc: Ralph Campbell Cc: Mika Penttil=C3=A4 Cc: Matthew Brost Cc: Francois Dugast Signed-off-by: Balbir Singh --- include/linux/huge_mm.h | 11 +++++-- lib/test_hmm.c | 9 ++++++ mm/huge_memory.c | 45 ++++++++++++++------------ mm/migrate_device.c | 71 ++++++++++++++++++++++++++++++++++------- 4 files changed, 101 insertions(+), 35 deletions(-) diff --git a/include/linux/huge_mm.h b/include/linux/huge_mm.h index a4880fe98e46..52d8b435950b 100644 --- a/include/linux/huge_mm.h +++ b/include/linux/huge_mm.h @@ -343,8 +343,8 @@ unsigned long thp_get_unmapped_area_vmflags(struct file= *filp, unsigned long add vm_flags_t vm_flags); =20 bool can_split_folio(struct folio *folio, int caller_pins, int *pextra_pin= s); -int split_huge_page_to_list_to_order(struct page *page, struct list_head *= list, - unsigned int new_order); +int __split_huge_page_to_list_to_order(struct page *page, struct list_head= *list, + unsigned int new_order, bool unmapped); int min_order_for_split(struct folio *folio); int split_folio_to_list(struct folio *folio, struct list_head *list); bool uniform_split_supported(struct folio *folio, unsigned int new_order, @@ -353,6 +353,13 @@ bool non_uniform_split_supported(struct folio *folio, = unsigned int new_order, bool warns); int folio_split(struct folio *folio, unsigned int new_order, struct page *= page, struct list_head *list); + +static inline int split_huge_page_to_list_to_order(struct page *page, stru= ct list_head *list, + unsigned int new_order) +{ + return __split_huge_page_to_list_to_order(page, list, new_order, false); +} + /* * try_folio_split - try to split a @folio at @page using non uniform spli= t. * @folio: folio to be split diff --git a/lib/test_hmm.c b/lib/test_hmm.c index 14dbce719896..dda87c34b440 100644 --- a/lib/test_hmm.c +++ b/lib/test_hmm.c @@ -1611,6 +1611,15 @@ static vm_fault_t dmirror_devmem_fault(struct vm_fau= lt *vmf) order =3D folio_order(page_folio(vmf->page)); nr =3D 1 << order; =20 + /* + * When folios are partially mapped, we can't rely on the folio + * order of vmf->page as the folio might not be fully split yet + */ + if (vmf->pte) { + order =3D 0; + nr =3D 1; + } + /* * Consider a per-cpu cache of src and dst pfns, but with * large number of cpus that might not scale well. diff --git a/mm/huge_memory.c b/mm/huge_memory.c index dc58081b661c..863393dec1f1 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -3474,15 +3474,6 @@ static void __split_folio_to_order(struct folio *fol= io, int old_order, new_folio->mapping =3D folio->mapping; new_folio->index =3D folio->index + i; =20 - /* - * page->private should not be set in tail pages. Fix up and warn once - * if private is unexpectedly set. - */ - if (unlikely(new_folio->private)) { - VM_WARN_ON_ONCE_PAGE(true, new_head); - new_folio->private =3D NULL; - } - if (folio_test_swapcache(folio)) new_folio->swap.val =3D folio->swap.val + i; =20 @@ -3711,6 +3702,7 @@ bool uniform_split_supported(struct folio *folio, uns= igned int new_order, * @lock_at: a page within @folio to be left locked to caller * @list: after-split folios will be put on it if non NULL * @uniform_split: perform uniform split or not (non-uniform split) + * @unmapped: The pages are already unmapped, they are migration entries. * * It calls __split_unmapped_folio() to perform uniform and non-uniform sp= lit. * It is in charge of checking whether the split is supported or not and @@ -3726,7 +3718,7 @@ bool uniform_split_supported(struct folio *folio, uns= igned int new_order, */ static int __folio_split(struct folio *folio, unsigned int new_order, struct page *split_at, struct page *lock_at, - struct list_head *list, bool uniform_split) + struct list_head *list, bool uniform_split, bool unmapped) { struct deferred_split *ds_queue =3D get_deferred_split_queue(folio); XA_STATE(xas, &folio->mapping->i_pages, folio->index); @@ -3776,13 +3768,15 @@ static int __folio_split(struct folio *folio, unsig= ned int new_order, * is taken to serialise against parallel split or collapse * operations. */ - anon_vma =3D folio_get_anon_vma(folio); - if (!anon_vma) { - ret =3D -EBUSY; - goto out; + if (!unmapped) { + anon_vma =3D folio_get_anon_vma(folio); + if (!anon_vma) { + ret =3D -EBUSY; + goto out; + } + anon_vma_lock_write(anon_vma); } mapping =3D NULL; - anon_vma_lock_write(anon_vma); } else { unsigned int min_order; gfp_t gfp; @@ -3849,7 +3843,8 @@ static int __folio_split(struct folio *folio, unsigne= d int new_order, goto out_unlock; } =20 - unmap_folio(folio); + if (!unmapped) + unmap_folio(folio); =20 /* block interrupt reentry in xa_lock and spinlock */ local_irq_disable(); @@ -3936,10 +3931,13 @@ static int __folio_split(struct folio *folio, unsig= ned int new_order, =20 next =3D folio_next(new_folio); =20 + zone_device_private_split_cb(folio, new_folio); + expected_refs =3D folio_expected_ref_count(new_folio) + 1; folio_ref_unfreeze(new_folio, expected_refs); =20 - lru_add_split_folio(folio, new_folio, lruvec, list); + if (!unmapped) + lru_add_split_folio(folio, new_folio, lruvec, list); =20 /* * Anonymous folio with swap cache. @@ -3973,6 +3971,7 @@ static int __folio_split(struct folio *folio, unsigne= d int new_order, folio_put_refs(new_folio, nr_pages); } =20 + zone_device_private_split_cb(folio, NULL); /* * Unfreeze @folio only after all page cache entries, which * used to point to it, have been updated with new folios. @@ -3996,6 +3995,9 @@ static int __folio_split(struct folio *folio, unsigne= d int new_order, =20 local_irq_enable(); =20 + if (unmapped) + return ret; + if (nr_shmem_dropped) shmem_uncharge(mapping->host, nr_shmem_dropped); =20 @@ -4086,12 +4088,13 @@ static int __folio_split(struct folio *folio, unsig= ned int new_order, * Returns -EINVAL when trying to split to an order that is incompatible * with the folio. Splitting to order 0 is compatible with all folios. */ -int split_huge_page_to_list_to_order(struct page *page, struct list_head *= list, - unsigned int new_order) +int __split_huge_page_to_list_to_order(struct page *page, struct list_head= *list, + unsigned int new_order, bool unmapped) { struct folio *folio =3D page_folio(page); =20 - return __folio_split(folio, new_order, &folio->page, page, list, true); + return __folio_split(folio, new_order, &folio->page, page, list, true, + unmapped); } =20 /* @@ -4120,7 +4123,7 @@ int folio_split(struct folio *folio, unsigned int new= _order, struct page *split_at, struct list_head *list) { return __folio_split(folio, new_order, split_at, &folio->page, list, - false); + false, false); } =20 int min_order_for_split(struct folio *folio) diff --git a/mm/migrate_device.c b/mm/migrate_device.c index 6621bba62710..9206a3d5c0d1 100644 --- a/mm/migrate_device.c +++ b/mm/migrate_device.c @@ -864,6 +864,29 @@ static int migrate_vma_insert_huge_pmd_page(struct mig= rate_vma *migrate, src[i] &=3D ~MIGRATE_PFN_MIGRATE; return 0; } + +static int migrate_vma_split_pages(struct migrate_vma *migrate, + unsigned long idx, unsigned long addr, + struct folio *folio) +{ + unsigned long i; + unsigned long pfn; + unsigned long flags; + int ret =3D 0; + + folio_get(folio); + split_huge_pmd_address(migrate->vma, addr, true); + ret =3D __split_huge_page_to_list_to_order(folio_page(folio, 0), NULL, + 0, true); + if (ret) + return ret; + migrate->src[idx] &=3D ~MIGRATE_PFN_COMPOUND; + flags =3D migrate->src[idx] & ((1UL << MIGRATE_PFN_SHIFT) - 1); + pfn =3D migrate->src[idx] >> MIGRATE_PFN_SHIFT; + for (i =3D 1; i < HPAGE_PMD_NR; i++) + migrate->src[i+idx] =3D migrate_pfn(pfn + i) | flags; + return ret; +} #else /* !CONFIG_ARCH_ENABLE_THP_MIGRATION */ static int migrate_vma_insert_huge_pmd_page(struct migrate_vma *migrate, unsigned long addr, @@ -873,6 +896,13 @@ static int migrate_vma_insert_huge_pmd_page(struct mig= rate_vma *migrate, { return 0; } + +static int migrate_vma_split_pages(struct migrate_vma *migrate, + unsigned long idx, unsigned long addr, + struct folio *folio) +{ + return 0; +} #endif =20 /* @@ -1022,8 +1052,9 @@ static void __migrate_device_pages(unsigned long *src= _pfns, struct migrate_vma *migrate) { struct mmu_notifier_range range; - unsigned long i; + unsigned long i, j; bool notified =3D false; + unsigned long addr; =20 for (i =3D 0; i < npages; ) { struct page *newpage =3D migrate_pfn_to_page(dst_pfns[i]); @@ -1065,12 +1096,16 @@ static void __migrate_device_pages(unsigned long *s= rc_pfns, (!(dst_pfns[i] & MIGRATE_PFN_COMPOUND))) { nr =3D HPAGE_PMD_NR; src_pfns[i] &=3D ~MIGRATE_PFN_COMPOUND; - src_pfns[i] &=3D ~MIGRATE_PFN_MIGRATE; - goto next; + } else { + nr =3D 1; } =20 - migrate_vma_insert_page(migrate, addr, &dst_pfns[i], - &src_pfns[i]); + for (j =3D 0; j < nr && i + j < npages; j++) { + src_pfns[i+j] |=3D MIGRATE_PFN_MIGRATE; + migrate_vma_insert_page(migrate, + addr + j * PAGE_SIZE, + &dst_pfns[i+j], &src_pfns[i+j]); + } goto next; } =20 @@ -1092,7 +1127,14 @@ static void __migrate_device_pages(unsigned long *sr= c_pfns, MIGRATE_PFN_COMPOUND); goto next; } - src_pfns[i] &=3D ~MIGRATE_PFN_MIGRATE; + nr =3D 1 << folio_order(folio); + addr =3D migrate->start + i * PAGE_SIZE; + if (migrate_vma_split_pages(migrate, i, addr, + folio)) { + src_pfns[i] &=3D ~(MIGRATE_PFN_MIGRATE | + MIGRATE_PFN_COMPOUND); + goto next; + } } else if ((src_pfns[i] & MIGRATE_PFN_MIGRATE) && (dst_pfns[i] & MIGRATE_PFN_COMPOUND) && !(src_pfns[i] & MIGRATE_PFN_COMPOUND)) { @@ -1127,12 +1169,17 @@ static void __migrate_device_pages(unsigned long *s= rc_pfns, BUG_ON(folio_test_writeback(folio)); =20 if (migrate && migrate->fault_page =3D=3D page) - extra_cnt =3D 1; - r =3D folio_migrate_mapping(mapping, newfolio, folio, extra_cnt); - if (r !=3D MIGRATEPAGE_SUCCESS) - src_pfns[i] &=3D ~MIGRATE_PFN_MIGRATE; - else - folio_migrate_flags(newfolio, folio); + extra_cnt++; + for (j =3D 0; j < nr && i + j < npages; j++) { + folio =3D page_folio(migrate_pfn_to_page(src_pfns[i+j])); + newfolio =3D page_folio(migrate_pfn_to_page(dst_pfns[i+j])); + + r =3D folio_migrate_mapping(mapping, newfolio, folio, extra_cnt); + if (r !=3D MIGRATEPAGE_SUCCESS) + src_pfns[i+j] &=3D ~MIGRATE_PFN_MIGRATE; + else + folio_migrate_flags(newfolio, folio); + } next: i +=3D nr; } --=20 2.50.1 From nobody Sat Oct 4 22:37:10 2025 Received: from NAM02-SN1-obe.outbound.protection.outlook.com (mail-sn1nam02on2087.outbound.protection.outlook.com [40.107.96.87]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id BC2B62E542B for ; Tue, 12 Aug 2025 02:41:13 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.96.87 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1754966475; cv=fail; b=JQodbDcgwk+Z0eyYOLvMbVevEKybsmCqosbz7csAtowPZdg340UqfVZH6g5q7qLk+ScEdocPN10B9pdlUAbW9V1Tm3h+Z5bRaWRRD70mIuc2qc4HGkErfwxQ0xSIq1JL83A1K5NuXQiJBux+537mrirr0hbzf90VWixTdo7OO1c= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1754966475; c=relaxed/simple; bh=YBtCeIRK8yGeO5nvPtfDLh3tX4jpYC5E0a0yzSeI3m4=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: Content-Type:MIME-Version; b=BJhj1xrpHvtN8VS2igU1sGtA7k4hT6phfIJ0Gw30QgdM7kvE4yUqZU6zs2SvzqIeE2yU6E9h61ZxdT3vNhCL/JC5Bszf/OdXCsbgXAsWUAj1HP4Fx6/LFR85uHX0Ov5PwbJSqo9P6SosNYRbGajnA3XW8jey5y3YS8HwbvJgauQ= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com; spf=fail smtp.mailfrom=nvidia.com; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b=MBboPy2L; arc=fail smtp.client-ip=40.107.96.87 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="MBboPy2L" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=popfRSY8UPAnGNzge05uRoFVIGsnWl/IFQ4CZhobK+zmseEg9dgwY1c9sGnxKwATEqgu6gTKXFWgHx/nC8BGAD+GihlIPrepGIqtlfLGyd8wHd8jm4XWESqe9AXKy/xucakLOYw1kfNRbV7vwrCPiNaDgYBnjXgFD3dVFVIrmOLGo0Az+/ztfLZD6sqd6NG6+um3i2faST/lp1+XClMP+2xqHWW3jc+BLe97/88BUlcr6Ln332EA+p7D0T3e1hZtbZoBI8IIAnXp3ZHY6JUf/A9ZJVcbsg1Ik8WPryJ+sq9OL8qd7xQua6xTtYZqFeufeRHOkvy+9eESe6S4RvuuDg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=wV6wQe/UiG2uRT1eVCn/mImD80c08zH/uViys4jM0M0=; b=DAszgh5Bf6BMQIRznJFb5Yg9eUwcP8KdzBhuN37rI3jbC0H8EbLOW8/MzoKsX5WG7lW12S2yEukPVqEa9sfNhc6L1R5IXHlKbOFIADT6vzE24e2tuNe4laoKW/s3xvKWakrLzehh5nTBjYZGJEc9GZpye0dfg/f++ny1ccIdlHQEF+dospb5RzgD11XihnxazTAUNM5QuSpRtOGoiuf37JjugmdNuGloYEKR3ffutMop2syUqn5vTdobDMNxTPrkvbluiBSlHubZV1+rPYr/XftKK91WLsyuAJA0VZ/ljFdn6b3zmO1p8svYnJK70fglQzFgSnomDfk1yRAMzIvjKw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=wV6wQe/UiG2uRT1eVCn/mImD80c08zH/uViys4jM0M0=; b=MBboPy2La6pCPG2bhwospGQFdBCVEGLRocBPFh4KFoSgLMkKJhhT4YFgMiQZXoKFHbH82ticfqyyA7nRaohM++HLMr1qv78Lm/AFpju8PYgjcD5fcQCwhav4lu4REz0B84g4Uwos6VWnfqyoP1r6YxOua7LiMg143+7oVa41SHdM7Nm7zYzfHOPCYXpI/c+lG4j5Mvh7/UFyfhQIGDr7FClVMH75Rp7mSwlUIYCtPtoFvFjCf4Vm3h01xvFd3z+b4wLwlb7yrKWa0PLfosT9C7ROSLXOmj7EGiVMqhUqHMjz2kjR0yYpXZMeVZyDqvwftpIEF7JTltu5vU+65+8Dtw== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from PH8PR12MB7277.namprd12.prod.outlook.com (2603:10b6:510:223::13) by MN0PR12MB6317.namprd12.prod.outlook.com (2603:10b6:208:3c2::12) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9009.13; Tue, 12 Aug 2025 02:41:10 +0000 Received: from PH8PR12MB7277.namprd12.prod.outlook.com ([fe80::3a4:70ea:ff05:1251]) by PH8PR12MB7277.namprd12.prod.outlook.com ([fe80::3a4:70ea:ff05:1251%5]) with mapi id 15.20.9009.018; Tue, 12 Aug 2025 02:41:10 +0000 From: Balbir Singh To: dri-devel@lists.freedesktop.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Balbir Singh , Andrew Morton , David Hildenbrand , Zi Yan , Joshua Hahn , Rakie Kim , Byungchul Park , Gregory Price , Ying Huang , Alistair Popple , Oscar Salvador , Lorenzo Stoakes , Baolin Wang , "Liam R. Howlett" , Nico Pache , Ryan Roberts , Dev Jain , Barry Song , Lyude Paul , Danilo Krummrich , David Airlie , Simona Vetter , Ralph Campbell , =?UTF-8?q?Mika=20Penttil=C3=A4?= , Matthew Brost , Francois Dugast Subject: [v3 08/11] lib/test_hmm: add test case for split pages Date: Tue, 12 Aug 2025 12:40:33 +1000 Message-ID: <20250812024036.690064-9-balbirs@nvidia.com> X-Mailer: git-send-email 2.50.1 In-Reply-To: <20250812024036.690064-1-balbirs@nvidia.com> References: <20250812024036.690064-1-balbirs@nvidia.com> Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: BYAPR07CA0019.namprd07.prod.outlook.com (2603:10b6:a02:bc::32) To PH8PR12MB7277.namprd12.prod.outlook.com (2603:10b6:510:223::13) Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: PH8PR12MB7277:EE_|MN0PR12MB6317:EE_ X-MS-Office365-Filtering-Correlation-Id: 46ec7676-f587-4a43-067f-08ddd949b1ed X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|376014|7416014|366016|1800799024; X-Microsoft-Antispam-Message-Info: =?utf-8?B?MEJzV0xiZWdQM01nWDhQOUtRSFE0aEgrKzdKL2Q2ZVNJTVV3cWR6TTdsOXJG?= =?utf-8?B?UXZjN3NvQVdIRmJON0RUL2o0SnZCMUVJT2xTN0NBWEhWTG16TFp1ckJtWjUw?= =?utf-8?B?TzUzQnkxdjBudVRqeHNtVFg1RVBuQ1FlV2t0R3pKakZucEtMTXA1YTJva05h?= =?utf-8?B?YTM5VVJFYmZDUFBPSE8yUkN5UmI0OXFpck1MMzR4YWp6eHVBOGFmR0FlSUhI?= =?utf-8?B?bXBOWU9TdmhWdHNZaVdNWmdxNXdVTythMFZla2IrV1RELzMwMTM3eXlnTHEx?= =?utf-8?B?Nzl0WGE4a3cvUVBSMUw0Uyt4Z0t6TnVCUzhIUUE3YnM3NGxkU0NiR2tuSlNm?= =?utf-8?B?TG9MS1lra001SHhPNEpuS3Rid0tNRWgyZFg3V0orMTlGTTAzRzJSK0hFb1dP?= =?utf-8?B?Qk5jY1pKZTIzSDBNbVVac2xybHIxeWNsbWtacjlXbkRNWXg5WXA5V3ZwaTl1?= =?utf-8?B?VENSeE5yMjdtM01hdGZ4STVzK1Q5QWpsVzIxMTdrU1lqeGdHdFd5cHpQcWFl?= =?utf-8?B?K1diOXM0cnE1WklCdStGMmtpM0NZaGlTZWNuRDV0RjZ4MzdhRUNHdHk1Zk5r?= =?utf-8?B?dkpCeHpjWFQxVnZRbnJtSnovOFkwazdpUUcxeCtWa1djb3l4ZUNkU0xQY3Vr?= =?utf-8?B?T0hXenBDdzBBVkJUck9qcFl6YkJ6WnZ2Tjlpam04TUlVOTVlZGgva01pWXJz?= =?utf-8?B?UkZVajNZYVZGNWlHNExWdlUxVFk1RTdBaDlpSXlJVXFKUWdvc3JWTEN1bGxB?= =?utf-8?B?OGFuUHNrNEVoVk9XeFkvT1RNMHdUMmE4RjJVTWtKWm9XZGhRbkl3OE52N0x5?= =?utf-8?B?bUlMeVgxckVJNCttVnZMK2I1WkRXRkFxaC9VVldUTzhIYVlKeEd1VU1qVXoz?= =?utf-8?B?Zk1KK3E1QWorYW5SOHc1YU93dm5TV3N2L3U4TjdaZGF3bVdoZWcveEU3cm1T?= =?utf-8?B?YXRUZkxhYWdpUXdrMW5LT3BpNXBGa3phanVwajJUVGhiek9vcVI1S29NbmVX?= =?utf-8?B?RGs2UTIzaU8rQnkxYmhQVG02dDRSQjlaTnBiK1NzKy85MjBlR3A5UEZxc25l?= =?utf-8?B?Slk3K2RtMk1zTlZsTTExS296ZTZVcG1YbnhNZGdKclNhK3U2WjFGNXM0Tngw?= =?utf-8?B?bHEyYTV3akEyeGhiZXdSWXN3c1B5RUV1R1BHRzAxbFlGejBZbThuQjAvWUt0?= =?utf-8?B?V2ZtOG1PVTZsMTJJU1hrVVl5dEJBRUx4UUZZUmc4VDdZSXpCQ09HZkxGc0ZD?= =?utf-8?B?Zk1rcTkrWC9mZUZ6WGRRNldEdnJ0bGVIZHFWN0lDZXBsamFBUHFFVm53czVD?= =?utf-8?B?dVZRd3lnL3lKYzcxMmREd1orZmF4WFNZUFZnMjBwanczdzFsTXJoYkI4UEtq?= =?utf-8?B?NGg2eFdJS3B5TmRhekxhWU55TDJ6OWRQQUIvMkp5UVE5bHYzYWlnbnZNdGVv?= =?utf-8?B?VFZzcXdrY3BlL3U0Vng4c3FhMThmRzkveitKRXdZa1NjZjgwdDg2aDU1ZVV5?= =?utf-8?B?KzJHVTRTYi9hOTNoTXNLV3FYZzBpT1gyN3FjKytrWVArU2k2aFAvL3hmYktM?= =?utf-8?B?aXdUUWMwY3BWYWR6Rm82bVhKc0JoUml6WEVDeGpIQTNVNXhDZUcxVXhUZHB4?= =?utf-8?B?OHFGd2lCUGhrY1J4bzdIaExMU3BkWnkwZG1TUDlMbUE0bXIrMUd1Sk5EVHZ1?= =?utf-8?B?eFBrdlNPK3JuMG8xK0RhaGJjbnYwNTk5TTRNcmowSDlwM0VLcGRGRFR4NUha?= =?utf-8?B?TTdONFQzWUExNkNQUEphSjdSbXlMZDIzclU5dklxbFhCNWNNYjNrazgxTk44?= =?utf-8?B?VVYrSkdicS94S1BHSUFBemp5V0tyZmpBanpLSXQxSDNIT3FTSzB4STdWYitI?= =?utf-8?B?ejBsWTliRmF2NzByT2tKYTAyVWtNTjVOc29reERjbURISnc9PQ==?= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:PH8PR12MB7277.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(376014)(7416014)(366016)(1800799024);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?ZkpwOEJGRS9vQzhrNHBhb2JiNTN1N0hIUDJqLytGZyt5eUxHREk0TXYveUhJ?= =?utf-8?B?cGs4V256MlNHcnZMa0ROTXMzaWQwRW94djc0WStLcFBIcXMwUVBSSEN6WExj?= =?utf-8?B?Kys5NmFOSkd0cjRKVHhJbVpDeDNhQmhmS3E1TTVoTXlRUWV2VXBQeTVQQTdX?= =?utf-8?B?SVdxbWVxZHpKd0NhWWVOWXEveGdDVERCaGRFSnhSWXFtQXVUUFRlU0x3NHFs?= =?utf-8?B?TzN4ZS9LZXVsTW5kZzRkcElPc21TQUlLcy9Od2RybnBlY2VXN0l5VWlwMjJv?= =?utf-8?B?Y0xrejlUSmU5c0FPK0VBSWFSZCtBWDZXSENMQ3BsVW9yVytIUTR5MDZkQkZa?= =?utf-8?B?RDFpWk9aYzhpTkdCaWdtelN1YWNJK09ZeEZsZUNtMDQ3akdvYVgvRWthc2R4?= =?utf-8?B?TGQwSUVrc1hzK0JxRnJVVWtPZUtheVRxaloyQU40L1RXYkd2QjNWazFXUlVj?= =?utf-8?B?M2RESGZZbURiNmpkcmhKWENkWlB4bG9vVUliVWIvT01qZnN0aHVYMkdFM0xM?= =?utf-8?B?QVdmU245a3lHSkxycllpZmpTRnljRGNlOUE2TlRRZTBlZXlnZVJKQjgvdkdV?= =?utf-8?B?bCttVHdFV1ZZYWE3QTMwWTBZL2FjSm15MENweU4zcXhEUE4ra25LL2ltalEy?= =?utf-8?B?bE9YYU5WT1JmdUZSRnQ4NTBvMnlCY3lVSnN2T1NHYWdyOVNDL2JQSEh0ZU4w?= =?utf-8?B?cDdQbVBNZ2pzQzQvdFh1dGZJemJYbU5TUDBDS0xPQmpuYlRqZVJUMENYRzFQ?= =?utf-8?B?UjArbFdGZlpDNEJwMU1reFc3TGlKdndzUnhqckp5dFh2OVZqQ2lPSDRwQXBN?= =?utf-8?B?a1pJck1FYzgzbXNPbzlyZ3VPeGtlZHloNUszKy9zeGdNMkpxYU1lbXhqSnR0?= =?utf-8?B?dm1Pb1N3UDdNK0FDVmc2M0JMaFhKdndlSVJ3WU9OdW42NU8yQW9xcGttcTlG?= =?utf-8?B?QVMwU3VIWXVwUkVaTXJCa2psS2VKTTFtWjNPeUlZY3F1ek9kcDZvRnVLMVBj?= =?utf-8?B?NWNjTGxQM3VQR0FOZEtUcUhwSlJ3enJpN2ZLYTFFUFBZWllxOFJlNVJRaHp4?= =?utf-8?B?K2JrYVVjNWFKcTJoaTQrblNvbE5UaG53V1lnYzAxd29VQ3NCZlV3RkdrOU5Y?= =?utf-8?B?dEZnNFpWZWRWSXNMQUE5b1NJVit4WWs4YldqQ2haOWdDdjIvV01oKzBkWUJt?= =?utf-8?B?Y1ZJNUFkZG1ERXhLOXFiYUVObWE3Y3R4b21WbEpaUm1LTUREY2gxcEtoL3RR?= =?utf-8?B?RHBGMDJlZ0l2RDU4dlZsTncwN29HbThWTTNpak5ocWd6Z3dQK1d1ZVZsV3Zx?= =?utf-8?B?UFVaMzFJaUUra0dXQ2pUR2thRHpzU21VR0RySTBGbUZmQkZTeVUweDNZZWNs?= =?utf-8?B?d1lyamZySndVTldXZE5BeUxXM0hiOUJvR3prZTJoaEMxcGI4d3lGeEh3TDR0?= =?utf-8?B?VCtlYlQ5eFhjOUI2SCtTUzVZMk95S0N0d25UOENWWUk3N2hjYlNzaFB1VGFO?= =?utf-8?B?YnJFT1RVZmoydzVHVmpkNXBPRVpoMEtoS2Q4VjdJT2x4cFdmcXE4R2hUbG1M?= =?utf-8?B?Vld0UllLYjZ5KzAvREh6WHhSeC9zY25SOUQwZzBveGlpQmZKOHBvN2R2Vi8v?= =?utf-8?B?ei9pR3RvR2poZjdCcG5VMFFKSU9MR0NqN0U1bzVPY0ZaeENOSXYySUdyMlZh?= =?utf-8?B?K0E2OUJRVDZ5aG5CN1BWNXdCaVFvMEh4THpRaWhvNjd3eHFoODlRUWJ5dnIv?= =?utf-8?B?OEprVndOTjlnY1NheTQ5c1VLcjBVOWo5STJCNE9jcndPbjFHbjNZZTVPbEda?= =?utf-8?B?NE9ybU5vanJQTnJFT29VUFppOUVEbHhyYVpBVHIzNU5vVkU0RjFGTlZoRVhw?= =?utf-8?B?TzI0dzQzR2dEMkFxdFVuVXBHSCtyZkV6T09tYVlRUUY4MnpxSytUMVVFbnlj?= =?utf-8?B?am9sMmxSZ0Z5emxYc2hzQ1ZUS2FSZDgyY0ZaVHVmUXJkUjVZQnJ5NkVrYmFR?= =?utf-8?B?VXRUSEVMR1YxQ005dHhrMkVCMVV2ZjJMZEtUNmJqK1dpWmc0TkRoTlE4VUpt?= =?utf-8?B?bjlEZStQclRjOWxxQjFFbUhnRGRnUmZmZnMvMnA5b2sxUXg1SWViNEk2WmVo?= =?utf-8?Q?SFWItsywIB+ARcxLrctMeqTW4?= X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: 46ec7676-f587-4a43-067f-08ddd949b1ed X-MS-Exchange-CrossTenant-AuthSource: PH8PR12MB7277.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 12 Aug 2025 02:41:10.0473 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: GDHcfCa7XDiNw1Lx3lY3vWkSs9qZjzom+dx4JPf1MbS6TX5cIcO4FjTjzHtPywxXW8HIs3IPxdFdaasx8+/Uqg== X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN0PR12MB6317 Add a new flag HMM_DMIRROR_FLAG_FAIL_ALLOC to emulate failure of allocating a large page. This tests the code paths involving split migration. Cc: Andrew Morton Cc: David Hildenbrand Cc: Zi Yan Cc: Joshua Hahn Cc: Rakie Kim Cc: Byungchul Park Cc: Gregory Price Cc: Ying Huang Cc: Alistair Popple Cc: Oscar Salvador Cc: Lorenzo Stoakes Cc: Baolin Wang Cc: "Liam R. Howlett" Cc: Nico Pache Cc: Ryan Roberts Cc: Dev Jain Cc: Barry Song Cc: Lyude Paul Cc: Danilo Krummrich Cc: David Airlie Cc: Simona Vetter Cc: Ralph Campbell Cc: Mika Penttil=C3=A4 Cc: Matthew Brost Cc: Francois Dugast Signed-off-by: Balbir Singh --- lib/test_hmm.c | 61 ++++++++++++++++++++++++++++++--------------- lib/test_hmm_uapi.h | 3 +++ 2 files changed, 44 insertions(+), 20 deletions(-) diff --git a/lib/test_hmm.c b/lib/test_hmm.c index dda87c34b440..5c5bfb48ec8a 100644 --- a/lib/test_hmm.c +++ b/lib/test_hmm.c @@ -92,6 +92,7 @@ struct dmirror { struct xarray pt; struct mmu_interval_notifier notifier; struct mutex mutex; + __u64 flags; }; =20 /* @@ -699,7 +700,12 @@ static void dmirror_migrate_alloc_and_copy(struct migr= ate_vma *args, page_to_pfn(spage))) goto next; =20 - dpage =3D dmirror_devmem_alloc_page(dmirror, is_large); + if (dmirror->flags & HMM_DMIRROR_FLAG_FAIL_ALLOC) { + dmirror->flags &=3D ~HMM_DMIRROR_FLAG_FAIL_ALLOC; + dpage =3D NULL; + } else + dpage =3D dmirror_devmem_alloc_page(dmirror, is_large); + if (!dpage) { struct folio *folio; unsigned long i; @@ -959,44 +965,55 @@ static vm_fault_t dmirror_devmem_fault_alloc_and_copy= (struct migrate_vma *args, =20 spage =3D BACKING_PAGE(spage); order =3D folio_order(page_folio(spage)); - if (order) + *dst =3D MIGRATE_PFN_COMPOUND; + if (*src & MIGRATE_PFN_WRITE) + *dst |=3D MIGRATE_PFN_WRITE; + + if (dmirror->flags & HMM_DMIRROR_FLAG_FAIL_ALLOC) { + dmirror->flags &=3D ~HMM_DMIRROR_FLAG_FAIL_ALLOC; + *dst &=3D ~MIGRATE_PFN_COMPOUND; + dpage =3D NULL; + } else if (order) { dpage =3D folio_page(vma_alloc_folio(GFP_HIGHUSER_MOVABLE, order, args->vma, addr), 0); - else - dpage =3D alloc_page_vma(GFP_HIGHUSER_MOVABLE, args->vma, addr); - - /* Try with smaller pages if large allocation fails */ - if (!dpage && order) { + } else { dpage =3D alloc_page_vma(GFP_HIGHUSER_MOVABLE, args->vma, addr); - if (!dpage) - return VM_FAULT_OOM; - order =3D 0; } =20 + if (!dpage && !order) + return VM_FAULT_OOM; + pr_debug("migrating from sys to dev pfn src: 0x%lx pfn dst: 0x%lx\n", page_to_pfn(spage), page_to_pfn(dpage)); - lock_page(dpage); - xa_erase(&dmirror->pt, addr >> PAGE_SHIFT); - copy_highpage(dpage, spage); - *dst =3D migrate_pfn(page_to_pfn(dpage)); - if (*src & MIGRATE_PFN_WRITE) - *dst |=3D MIGRATE_PFN_WRITE; - if (order) - *dst |=3D MIGRATE_PFN_COMPOUND; + + if (dpage) { + lock_page(dpage); + *dst |=3D migrate_pfn(page_to_pfn(dpage)); + } =20 for (i =3D 0; i < (1 << order); i++) { struct page *src_page; struct page *dst_page; =20 + /* Try with smaller pages if large allocation fails */ + if (!dpage && order) { + dpage =3D alloc_page_vma(GFP_HIGHUSER_MOVABLE, args->vma, addr); + lock_page(dpage); + dst[i] =3D migrate_pfn(page_to_pfn(dpage)); + dst_page =3D pfn_to_page(page_to_pfn(dpage)); + dpage =3D NULL; /* For the next iteration */ + } else { + dst_page =3D pfn_to_page(page_to_pfn(dpage) + i); + } + src_page =3D pfn_to_page(page_to_pfn(spage) + i); - dst_page =3D pfn_to_page(page_to_pfn(dpage) + i); =20 xa_erase(&dmirror->pt, addr >> PAGE_SHIFT); + addr +=3D PAGE_SIZE; copy_highpage(dst_page, src_page); } next: - addr +=3D PAGE_SIZE << order; src +=3D 1 << order; dst +=3D 1 << order; } @@ -1514,6 +1531,10 @@ static long dmirror_fops_unlocked_ioctl(struct file = *filp, dmirror_device_remove_chunks(dmirror->mdevice); ret =3D 0; break; + case HMM_DMIRROR_FLAGS: + dmirror->flags =3D cmd.npages; + ret =3D 0; + break; =20 default: return -EINVAL; diff --git a/lib/test_hmm_uapi.h b/lib/test_hmm_uapi.h index 8c818a2cf4f6..f94c6d457338 100644 --- a/lib/test_hmm_uapi.h +++ b/lib/test_hmm_uapi.h @@ -37,6 +37,9 @@ struct hmm_dmirror_cmd { #define HMM_DMIRROR_EXCLUSIVE _IOWR('H', 0x05, struct hmm_dmirror_cmd) #define HMM_DMIRROR_CHECK_EXCLUSIVE _IOWR('H', 0x06, struct hmm_dmirror_cm= d) #define HMM_DMIRROR_RELEASE _IOWR('H', 0x07, struct hmm_dmirror_cmd) +#define HMM_DMIRROR_FLAGS _IOWR('H', 0x08, struct hmm_dmirror_cmd) + +#define HMM_DMIRROR_FLAG_FAIL_ALLOC (1ULL << 0) =20 /* * Values returned in hmm_dmirror_cmd.ptr for HMM_DMIRROR_SNAPSHOT. --=20 2.50.1 From nobody Sat Oct 4 22:37:10 2025 Received: from NAM02-BN1-obe.outbound.protection.outlook.com (mail-bn1nam02on2076.outbound.protection.outlook.com [40.107.212.76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 651682E54B9 for ; Tue, 12 Aug 2025 02:41:17 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.212.76 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1754966480; cv=fail; b=X5XbDfi3wKxyEjVslJNb8q7631p9Go18gXLS990W24+yAaf28KwX5LUhKxVVGc3V33neQ7i6q1D2no2s9HBXwzJ9AZkq4+IjFFZf1UJ6BFtAJhT0HAUYokRi5jsGlTiNlT2It1aDABUsuG40LlU1u81KJ68m/GP2VAUpauaoGCw= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1754966480; c=relaxed/simple; bh=z9bF78a2yzZOuTltgWOkzU+Iyo0BlVwjLZW7wtuRD+4=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: Content-Type:MIME-Version; b=Ryd5AHBo2/fbYGHZadWZ+lqJnrb/zGxDjZsLWhcO2+7PScLMBg6+Y3JQcy6Mzwbmi4t8cGLlK74knMiwkCeLntKwH03dXeHQIFSkJg2CTNIjcgkP2YQXiNMUPdY4R4v+knnulN0vDgO+Uv4odOST1iQkQ3YZtyr0fokSk23dB2o= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com; spf=fail smtp.mailfrom=nvidia.com; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b=lO4f6V8l; arc=fail smtp.client-ip=40.107.212.76 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="lO4f6V8l" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=OfStJ4YiF62IIMK5w08xx9jIIyhAWAOwTKYRe1R+55DLkfsP//2nsI1P+z+0qMtzeET9pUx+iJuQiNhq8QdO+YHcJAmy1q1ZDyiCEBFBa4EHMQqpGhCuB6j84qHA1gIwdnV3G8jmcRmY52DgFVKtDFHsf4DUC4j2tgaY44frEiVZgOKeooszDH8Qr+4hAaJT9fN7+0X2jHUmKH7pVRWmO0YFvoaRhBlMMmZAn4XyPvlLn+CbwCR143hr+EY60+1jtplE74NPCIFbHLcNRUMdYOxg3oJUNGwdNUKR1Rqoe7Yu/OC2jaS1tLBKrYQMldgJ5XIA6F34O9Cu4o8ITz1CIA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=Kpj0DgtzxR5aJn4cHUzjRxXQ+GZQuw6pBgp0jLjQikM=; b=SSRttWqHHOy9phO8Wj4+kO/rcm9IAyvJDZwh5GQHSh9rWM8+A37VUBcsndddWKuEi9sCaMQtpFxaj8cHWFGqM+nz+URONQZNFQ8YLgh+BSQdCIABzdNOikEWI2ojWhBMGTVkgAjCju6LOm7tKBdt4XfRG5dZ8PkWKX5xt58HtZfKA+VnoDaxHumfu6HHf8jjghVGVuzVkcUyuSSPm61YofJb+/SASAPm2XqEgKGU1s/wwjqKZXnuHsQXebE4V0V4xIZLGBoLfuXy95sQa2YgtZwtsjRW9C37psqndi1kyt1DV971Hf/IoBG/98GC/GR3Qg1cWskfNidMkDmyVJNvXQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=Kpj0DgtzxR5aJn4cHUzjRxXQ+GZQuw6pBgp0jLjQikM=; b=lO4f6V8lfqFs9wE3j/p/InB5ZeAWPnPNRuNpyr3VgEXT+lP7ZEm2o90rIJO6mFqhxc9J/tcpUvt4XyreLuD1asibZiiPA9PU3vxmzkNEVsFLWJ4A57V3YwDwBnQjSC4JzVU/4zEX6523dP30dYoaiKXSLE7vO1WL+DvDqQtIPWzi6q416/XkDDJccd89uKlY6mpHZRqL3U7z0jP1+7MOWHf+Qfdi2tJzUgrgSrgbrlrG6OVA55Gv/E9o1rk7Bg1orqrN4AUMduUisU0Jzp7TNX6ENjk5NBYJ7BW+pV3MVGII4eeq0U19K+NkaUybGNiiw49nM5ENpHhlGm9rTdLllA== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from PH8PR12MB7277.namprd12.prod.outlook.com (2603:10b6:510:223::13) by MN0PR12MB6317.namprd12.prod.outlook.com (2603:10b6:208:3c2::12) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9009.13; Tue, 12 Aug 2025 02:41:14 +0000 Received: from PH8PR12MB7277.namprd12.prod.outlook.com ([fe80::3a4:70ea:ff05:1251]) by PH8PR12MB7277.namprd12.prod.outlook.com ([fe80::3a4:70ea:ff05:1251%5]) with mapi id 15.20.9009.018; Tue, 12 Aug 2025 02:41:13 +0000 From: Balbir Singh To: dri-devel@lists.freedesktop.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Balbir Singh , Andrew Morton , David Hildenbrand , Zi Yan , Joshua Hahn , Rakie Kim , Byungchul Park , Gregory Price , Ying Huang , Alistair Popple , Oscar Salvador , Lorenzo Stoakes , Baolin Wang , "Liam R. Howlett" , Nico Pache , Ryan Roberts , Dev Jain , Barry Song , Lyude Paul , Danilo Krummrich , David Airlie , Simona Vetter , Ralph Campbell , =?UTF-8?q?Mika=20Penttil=C3=A4?= , Matthew Brost , Francois Dugast Subject: [v3 09/11] selftests/mm/hmm-tests: new tests for zone device THP migration Date: Tue, 12 Aug 2025 12:40:34 +1000 Message-ID: <20250812024036.690064-10-balbirs@nvidia.com> X-Mailer: git-send-email 2.50.1 In-Reply-To: <20250812024036.690064-1-balbirs@nvidia.com> References: <20250812024036.690064-1-balbirs@nvidia.com> Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: SY5PR01CA0060.ausprd01.prod.outlook.com (2603:10c6:10:1fc::19) To PH8PR12MB7277.namprd12.prod.outlook.com (2603:10b6:510:223::13) Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: PH8PR12MB7277:EE_|MN0PR12MB6317:EE_ X-MS-Office365-Filtering-Correlation-Id: 01eff2f7-7e9f-4fc5-708c-08ddd949b43f X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|376014|7416014|366016|1800799024; X-Microsoft-Antispam-Message-Info: =?utf-8?B?ai8xL2JhUk9sYk1jRTFvaEZXWlZjMDNLdS9iUGF0VkR3dVVWTGJ0Nzd0S2dH?= =?utf-8?B?N2JuZlNSSkYyMW1mWWwrajNBQ084aXdQRE14WWVNTm1LYldBRXRmOHA3SUJE?= =?utf-8?B?WGc4dWVhc3RDZHI2WjVlQUtTcmZ0OVhlMVh3TjlvWGtPM3A2TmtlcGVldG1F?= =?utf-8?B?R1AybXRTS0xoeTNyeFNtb0ZjM25HRWpXN3JOSVBMdVBJWWMydFRiWjdjRUYv?= =?utf-8?B?MzQxbmJSYkZtWjJVYjJ6SFplZldhcFYvelJpamVzNFd4eEVWRTNWRzd2NDMv?= =?utf-8?B?SGg5OEd2L3FDVEFLMktCRW5LNERWNk5MR2toL0ZHekhzSkhQMW40NU94UkRS?= =?utf-8?B?TjR4QVQ5M1FBdExOekJCQnVNMmh1WnVuYm5KSjQ5NFdQa0JnQStRRWdmZ3JM?= =?utf-8?B?bk9MV1JzUTVhdFhxYWQvYlRvd1NqYmtOQ2NodkJNOFIvZDBYMThuL3NuK25C?= =?utf-8?B?Y3JGTVJLbTJaZk50RE50UjhCRXdOT2pNTjUxRGx3NVNrVFl1ZWRKc3VaQXNr?= =?utf-8?B?SVQ1SG9TcVJkZHk1Zkk5dDVpQ2VtVmZBZjdMK3Z6VVNlcVFwaE4xN3daSlZZ?= =?utf-8?B?L1pUNVJDV1lsanIvK3lON1hpNUxqTm1FWUFtMW16cktISnAzYXBNa1VpSmxX?= =?utf-8?B?dnhWaGxNbTlWQ2RTYWpqdVA3azBCTlM0SDJwOGRlVXVmTEgwaU1CMFA5cnl4?= =?utf-8?B?Q1NNTi9ZM2V6SGkyMnNzMk0zUG1zSTBBMUI5NC80MHFvMVFMaGF5emw4MTht?= =?utf-8?B?WUtVQ1FiWklrUjdXRUpyOHpHeEZ2Qy8wOHVUR09pUHhSd3JiYXg4MkhFekN2?= =?utf-8?B?a2NQQS91V2l2UlRIbjU2NzdxMmpJTHZlTlduRmlyaStiUUdjbHBsR3ZFZGpa?= =?utf-8?B?Ty9PQ3lpSW8rRGNRWnRLUThpL1VSTXlWMnpMRXFDOTltbXVheGhITmxFc1d5?= =?utf-8?B?YUhDNVlCd3g0dHZ3a3lrd0I3c3ArbkdCYmtteUYvNWhLbnZhMVBRQmlvWEgx?= =?utf-8?B?bEVMNUNBTCtjcGtLVUw4QW5mS2xjOERZUTBNeVd4RzBtZHE5eGtsZmdhTnFT?= =?utf-8?B?S215SlhBTDZyUXVab2ZBNUhkalZQTHp0eWoxTjF1RVZHQU1VcS95RmUvK2pm?= =?utf-8?B?bkFhZkt6Q05GTGRIRTRneXN3aTJzTThhS1czc29qSkNiQ0J3Zy8xVXpGcXMy?= =?utf-8?B?OG9SZUJjbEV4R3U1dksvdTNSWmhzVTFUTXYvRlVaaGRnMGRnUmtobHljMDJo?= =?utf-8?B?VVFQTXozUWJVTjJMQzRkSE50VWxuNWhyOHh6TEVOUXk5S2JTZnQzdCtyMTFt?= =?utf-8?B?dlRlU2ZpM25SZ0FtNXkwUXdhalRnbUpmUUprclEvVVNPNW5DTkpTOG41ZW9k?= =?utf-8?B?TDFUbkkzZE5QQzJSa2NqM2RrZHh6aWpDa1ZZL2JJVmxGU3JEV0FXZjE4dG8v?= =?utf-8?B?YWIvU1JRZWdPcVE5dXRHOHlWak5YWEMzbjB1RkhQaTEzQjU2TU5GaGlQR0ta?= =?utf-8?B?UGV5MVRubnE5S3BiU2ZlOWozWWJnRVJvejRZVFdtdjJ0UjBPa1FIQURndkpi?= =?utf-8?B?MHJuL01nazJQbzRDNWFhcGIvUC9UaTZ4NHFuMkI5RXU5TTFBcUtCR3dCMGU3?= =?utf-8?B?aVk2WkM0WDZaa1Rwam1WSDZROWhDc2JGSi9uNEVFZXBEZnN2dUFzaE9vMnky?= =?utf-8?B?NDh2NER0VFJDR1g5TDltcU1yNVAzWXdBVlp1WnZuSXpkQUJvcythLzJPdVh1?= =?utf-8?B?dyszSjk0RDhocWU1b3dqSERlWEgxYWpQV3ptWkVHam9oOFRTVTloTmloT2NN?= =?utf-8?B?T0tUNUFablZ4SWx4QlFNb0lXK1I4S2ZPbkJUWG0xanJPQ3BzRjU3cjNMTzBH?= =?utf-8?B?d0pzcmM2aHJtZzdYRXZsaldoTFhaTWZSRFNzYnN0M3o4UUE9PQ==?= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:PH8PR12MB7277.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(376014)(7416014)(366016)(1800799024);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?VUUrZ3p5RkUrTFlqZXJzQXZsZTNEd0VxTFJMZ3cvUlZlWDJtTkdyQzcrMkpj?= =?utf-8?B?RkdsUUlwUmd1bkZxTmxkOHpzVHlBS2xveE9ISWpad0NhSnVVOHBqSXhDL1ZF?= =?utf-8?B?dENUTlhML3NNU3F0NEJGWXZQZzRSTmJoblFJTi8wUGZDaWxwMm5BTXJxMDZq?= =?utf-8?B?OVBSSENXTDgvaVV5U2xhR3hORkNCVnh3YmVwNVlOQkw1WGpiTlVpc2QyMzhL?= =?utf-8?B?bEVBZWJhVVN1UlYxZEhaTWlBUzhZaythMWY1dTRVWml5eVBVU1oxWkh4RUZp?= =?utf-8?B?VG4rUmJ6c3IzanhyRnZUTzhYZVJySG1heTJWdHp1M2p0djlJSWNHbGp2dHdh?= =?utf-8?B?ZWZZc1l3WVZ2bkxlWGczNmpZSmJOSm5KK3diVFN0TkRycXpCNmJnc2FnajVI?= =?utf-8?B?cHFSOWVBeUdxSDVHclpva0NKaTMwSzNvSzhoNXBFTHhXTEVhZ0FULzk1Titz?= =?utf-8?B?aXhTTlBiTEtPYnlYWTYzbHFvU0NIV1R2cGNoY1lBT3A1RFUzUm0rbVBSRGk4?= =?utf-8?B?R3hDUmg4VVB4aFZ3VnAybFEzbWwzMi9vdERscXNkOTROSTFUb1FRL0JmK0Vy?= =?utf-8?B?eFhPUjlCZnZZWGVBT1czY0VzUGRJcE9mWDNBend4cGNWNE8xT1l4L2NGWGhR?= =?utf-8?B?cTZ6Y0FhYnJ5SVN5dElGSy83WmhHN25veEV0c3AwTXdvS0o0dXJ2ZEhQUDZO?= =?utf-8?B?QldPTTF5UlVpS2VpNitpMVJKejdFUU14MG10TVoyQTNyZjhqOWowOHlWYVg1?= =?utf-8?B?K2JYVE5KSkRrMXNrL0tocHp0SE94dzIxanQ4TGVlMU9BZDdWVnIva2ZXS3VI?= =?utf-8?B?alNJVmhtK3FRZFhzQ053czRqa2lheGZqOUtKMFBTeFI1azVyTzJTdUd3WDJk?= =?utf-8?B?ai9OQ09OVFlPZWk5NXJlS3VOR3BFMGNwS0xqRWsyelVzeHJScU9yakNIR1pJ?= =?utf-8?B?Nk5zcklKOGo2cldycnp1TjY4ZlVVVkx5UHFYbkx2YXJpckpTRHJVSjVPQjF5?= =?utf-8?B?OTI1cU9TWmNoTGJneEZhdjhYQzd3UjIwdW1RKy92eXNaVnBNenE1bFFoZUEy?= =?utf-8?B?MjNLQlJOWDhuM3dnaVJWcFdtSyt4TithL3JnaEFWcVFvRDRXQzRIQjB4czc1?= =?utf-8?B?UzgyWGRXY3BHeUQ1WTNvMHRQeU9KcDJrWlZpV1ZUQk1hSWpUUUxKWjBPc0Vr?= =?utf-8?B?WWhhdG9XTE55RnZrM2ZwV3JidEVyd2dlYmF3QjgzZGVvNzJsMy9tcWNZUjRY?= =?utf-8?B?S25JZzFIZUpwZ1ZjNzRaWjRuU1cveS80R3lUMFNFRkZvLzhsVGwvdTNUb2Jo?= =?utf-8?B?QWYreTJvUjFuRnIrTTd2Yzk3TEgwUkVjR0VUKzZYTVdoQnJwTy9YUGRGTnli?= =?utf-8?B?RkxiMHlXMkJOcXVuaUVFMjlBU2hBcW0wbklZRjhQayt4RUVHUXEwNFNqUmpl?= =?utf-8?B?amxqVVh3Vkc4RDU1cnNiMmZvMjlNRWNjcUpXWUhSRVVlaytNWlF5bi9vWlI5?= =?utf-8?B?NHM2WCtrcmRzYlZtLzMya3Uva1ZWWEVVOVcvM0k4RmEvWExTaUNFQUZhMkJJ?= =?utf-8?B?Vk4rTFpzWkQ5bjVnWmxFWi9xS0ROUFg1YUdmNDVQZVc0MjRsZU94SGpzaHR1?= =?utf-8?B?ckQ3MHRrWEVPYVdxQUxENzNaN0FrWklYSUx6bXYyL09EWG93UkZNcWRQZ2xJ?= =?utf-8?B?aW9UV1dWK2RaemFhVVpSMTJqTFVCNzZqS2E2amRNTVZZMklXY0lBT25SSVYw?= =?utf-8?B?OXNDTUNDUGpSWjdTMWdHTUVCMlhScXdCdkNuUEhpbnpaZkZmcTBlWnJtcGNB?= =?utf-8?B?ekxyYnAzcjliV0VnK3NjT0hwaVcwTVRldHA5Y0h5emZ2bytzQnJDbllVd0FX?= =?utf-8?B?Q1IranZvYytseGwzUkQzRnNCOVlIeWRsYVJtLzRVbkZJU2wwNjRnSVpQWkli?= =?utf-8?B?M2lHeHBncEZoY3ZObzA2UlN2SEtYekcrUVovK2ttZTFIVTJKN3o5cnFrL3FT?= =?utf-8?B?VGUwbUR4aDhRdzZNQzlQQ2VVbkZrVU1HYlhWTVJUUlJncURRSjBFYVBGOE4z?= =?utf-8?B?eEJJWXRvdEM4YnBWUXlyWHRYTDBvS09xanJBRCtGTWtvcTRPTjYzcXYrN0Rz?= =?utf-8?Q?o5p6FnqWfxa4NOCwyLIHTGX9S?= X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: 01eff2f7-7e9f-4fc5-708c-08ddd949b43f X-MS-Exchange-CrossTenant-AuthSource: PH8PR12MB7277.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 12 Aug 2025 02:41:13.8162 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: Pe/EIVhQMSn/sBYCMBMUOe3KxVOhe53T3cq+xUYQPOVg5Ew5htSSXzyY1mMXZp5pwcvMlUaxnvvGdogrbtoKVg== X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN0PR12MB6317 Add new tests for migrating anon THP pages, including anon_huge, anon_huge_zero and error cases involving forced splitting of pages during migration. Cc: Andrew Morton Cc: David Hildenbrand Cc: Zi Yan Cc: Joshua Hahn Cc: Rakie Kim Cc: Byungchul Park Cc: Gregory Price Cc: Ying Huang Cc: Alistair Popple Cc: Oscar Salvador Cc: Lorenzo Stoakes Cc: Baolin Wang Cc: "Liam R. Howlett" Cc: Nico Pache Cc: Ryan Roberts Cc: Dev Jain Cc: Barry Song Cc: Lyude Paul Cc: Danilo Krummrich Cc: David Airlie Cc: Simona Vetter Cc: Ralph Campbell Cc: Mika Penttil=C3=A4 Cc: Matthew Brost Cc: Francois Dugast Signed-off-by: Balbir Singh --- tools/testing/selftests/mm/hmm-tests.c | 410 +++++++++++++++++++++++++ 1 file changed, 410 insertions(+) diff --git a/tools/testing/selftests/mm/hmm-tests.c b/tools/testing/selftes= ts/mm/hmm-tests.c index 141bf63cbe05..da3322a1282c 100644 --- a/tools/testing/selftests/mm/hmm-tests.c +++ b/tools/testing/selftests/mm/hmm-tests.c @@ -2056,4 +2056,414 @@ TEST_F(hmm, hmm_cow_in_device) =20 hmm_buffer_free(buffer); } + +/* + * Migrate private anonymous huge empty page. + */ +TEST_F(hmm, migrate_anon_huge_empty) +{ + struct hmm_buffer *buffer; + unsigned long npages; + unsigned long size; + unsigned long i; + void *old_ptr; + void *map; + int *ptr; + int ret; + + size =3D TWOMEG; + + buffer =3D malloc(sizeof(*buffer)); + ASSERT_NE(buffer, NULL); + + buffer->fd =3D -1; + buffer->size =3D 2 * size; + buffer->mirror =3D malloc(size); + ASSERT_NE(buffer->mirror, NULL); + memset(buffer->mirror, 0xFF, size); + + buffer->ptr =3D mmap(NULL, 2 * size, + PROT_READ, + MAP_PRIVATE | MAP_ANONYMOUS, + buffer->fd, 0); + ASSERT_NE(buffer->ptr, MAP_FAILED); + + npages =3D size >> self->page_shift; + map =3D (void *)ALIGN((uintptr_t)buffer->ptr, size); + ret =3D madvise(map, size, MADV_HUGEPAGE); + ASSERT_EQ(ret, 0); + old_ptr =3D buffer->ptr; + buffer->ptr =3D map; + + /* Migrate memory to device. */ + ret =3D hmm_migrate_sys_to_dev(self->fd, buffer, npages); + ASSERT_EQ(ret, 0); + ASSERT_EQ(buffer->cpages, npages); + + /* Check what the device read. */ + for (i =3D 0, ptr =3D buffer->mirror; i < size / sizeof(*ptr); ++i) + ASSERT_EQ(ptr[i], 0); + + buffer->ptr =3D old_ptr; + hmm_buffer_free(buffer); +} + +/* + * Migrate private anonymous huge zero page. + */ +TEST_F(hmm, migrate_anon_huge_zero) +{ + struct hmm_buffer *buffer; + unsigned long npages; + unsigned long size; + unsigned long i; + void *old_ptr; + void *map; + int *ptr; + int ret; + int val; + + size =3D TWOMEG; + + buffer =3D malloc(sizeof(*buffer)); + ASSERT_NE(buffer, NULL); + + buffer->fd =3D -1; + buffer->size =3D 2 * size; + buffer->mirror =3D malloc(size); + ASSERT_NE(buffer->mirror, NULL); + memset(buffer->mirror, 0xFF, size); + + buffer->ptr =3D mmap(NULL, 2 * size, + PROT_READ, + MAP_PRIVATE | MAP_ANONYMOUS, + buffer->fd, 0); + ASSERT_NE(buffer->ptr, MAP_FAILED); + + npages =3D size >> self->page_shift; + map =3D (void *)ALIGN((uintptr_t)buffer->ptr, size); + ret =3D madvise(map, size, MADV_HUGEPAGE); + ASSERT_EQ(ret, 0); + old_ptr =3D buffer->ptr; + buffer->ptr =3D map; + + /* Initialize a read-only zero huge page. */ + val =3D *(int *)buffer->ptr; + ASSERT_EQ(val, 0); + + /* Migrate memory to device. */ + ret =3D hmm_migrate_sys_to_dev(self->fd, buffer, npages); + ASSERT_EQ(ret, 0); + ASSERT_EQ(buffer->cpages, npages); + + /* Check what the device read. */ + for (i =3D 0, ptr =3D buffer->mirror; i < size / sizeof(*ptr); ++i) + ASSERT_EQ(ptr[i], 0); + + /* Fault pages back to system memory and check them. */ + for (i =3D 0, ptr =3D buffer->ptr; i < size / sizeof(*ptr); ++i) { + ASSERT_EQ(ptr[i], 0); + /* If it asserts once, it probably will 500,000 times */ + if (ptr[i] !=3D 0) + break; + } + + buffer->ptr =3D old_ptr; + hmm_buffer_free(buffer); +} + +/* + * Migrate private anonymous huge page and free. + */ +TEST_F(hmm, migrate_anon_huge_free) +{ + struct hmm_buffer *buffer; + unsigned long npages; + unsigned long size; + unsigned long i; + void *old_ptr; + void *map; + int *ptr; + int ret; + + size =3D TWOMEG; + + buffer =3D malloc(sizeof(*buffer)); + ASSERT_NE(buffer, NULL); + + buffer->fd =3D -1; + buffer->size =3D 2 * size; + buffer->mirror =3D malloc(size); + ASSERT_NE(buffer->mirror, NULL); + memset(buffer->mirror, 0xFF, size); + + buffer->ptr =3D mmap(NULL, 2 * size, + PROT_READ | PROT_WRITE, + MAP_PRIVATE | MAP_ANONYMOUS, + buffer->fd, 0); + ASSERT_NE(buffer->ptr, MAP_FAILED); + + npages =3D size >> self->page_shift; + map =3D (void *)ALIGN((uintptr_t)buffer->ptr, size); + ret =3D madvise(map, size, MADV_HUGEPAGE); + ASSERT_EQ(ret, 0); + old_ptr =3D buffer->ptr; + buffer->ptr =3D map; + + /* Initialize buffer in system memory. */ + for (i =3D 0, ptr =3D buffer->ptr; i < size / sizeof(*ptr); ++i) + ptr[i] =3D i; + + /* Migrate memory to device. */ + ret =3D hmm_migrate_sys_to_dev(self->fd, buffer, npages); + ASSERT_EQ(ret, 0); + ASSERT_EQ(buffer->cpages, npages); + + /* Check what the device read. */ + for (i =3D 0, ptr =3D buffer->mirror; i < size / sizeof(*ptr); ++i) + ASSERT_EQ(ptr[i], i); + + /* Try freeing it. */ + ret =3D madvise(map, size, MADV_FREE); + ASSERT_EQ(ret, 0); + + buffer->ptr =3D old_ptr; + hmm_buffer_free(buffer); +} + +/* + * Migrate private anonymous huge page and fault back to sysmem. + */ +TEST_F(hmm, migrate_anon_huge_fault) +{ + struct hmm_buffer *buffer; + unsigned long npages; + unsigned long size; + unsigned long i; + void *old_ptr; + void *map; + int *ptr; + int ret; + + size =3D TWOMEG; + + buffer =3D malloc(sizeof(*buffer)); + ASSERT_NE(buffer, NULL); + + buffer->fd =3D -1; + buffer->size =3D 2 * size; + buffer->mirror =3D malloc(size); + ASSERT_NE(buffer->mirror, NULL); + memset(buffer->mirror, 0xFF, size); + + buffer->ptr =3D mmap(NULL, 2 * size, + PROT_READ | PROT_WRITE, + MAP_PRIVATE | MAP_ANONYMOUS, + buffer->fd, 0); + ASSERT_NE(buffer->ptr, MAP_FAILED); + + npages =3D size >> self->page_shift; + map =3D (void *)ALIGN((uintptr_t)buffer->ptr, size); + ret =3D madvise(map, size, MADV_HUGEPAGE); + ASSERT_EQ(ret, 0); + old_ptr =3D buffer->ptr; + buffer->ptr =3D map; + + /* Initialize buffer in system memory. */ + for (i =3D 0, ptr =3D buffer->ptr; i < size / sizeof(*ptr); ++i) + ptr[i] =3D i; + + /* Migrate memory to device. */ + ret =3D hmm_migrate_sys_to_dev(self->fd, buffer, npages); + ASSERT_EQ(ret, 0); + ASSERT_EQ(buffer->cpages, npages); + + /* Check what the device read. */ + for (i =3D 0, ptr =3D buffer->mirror; i < size / sizeof(*ptr); ++i) + ASSERT_EQ(ptr[i], i); + + /* Fault pages back to system memory and check them. */ + for (i =3D 0, ptr =3D buffer->ptr; i < size / sizeof(*ptr); ++i) + ASSERT_EQ(ptr[i], i); + + buffer->ptr =3D old_ptr; + hmm_buffer_free(buffer); +} + +/* + * Migrate private anonymous huge page with allocation errors. + */ +TEST_F(hmm, migrate_anon_huge_err) +{ + struct hmm_buffer *buffer; + unsigned long npages; + unsigned long size; + unsigned long i; + void *old_ptr; + void *map; + int *ptr; + int ret; + + size =3D TWOMEG; + + buffer =3D malloc(sizeof(*buffer)); + ASSERT_NE(buffer, NULL); + + buffer->fd =3D -1; + buffer->size =3D 2 * size; + buffer->mirror =3D malloc(2 * size); + ASSERT_NE(buffer->mirror, NULL); + memset(buffer->mirror, 0xFF, 2 * size); + + old_ptr =3D mmap(NULL, 2 * size, PROT_READ | PROT_WRITE, + MAP_PRIVATE | MAP_ANONYMOUS, buffer->fd, 0); + ASSERT_NE(old_ptr, MAP_FAILED); + + npages =3D size >> self->page_shift; + map =3D (void *)ALIGN((uintptr_t)old_ptr, size); + ret =3D madvise(map, size, MADV_HUGEPAGE); + ASSERT_EQ(ret, 0); + buffer->ptr =3D map; + + /* Initialize buffer in system memory. */ + for (i =3D 0, ptr =3D buffer->ptr; i < size / sizeof(*ptr); ++i) + ptr[i] =3D i; + + /* Migrate memory to device but force a THP allocation error. */ + ret =3D hmm_dmirror_cmd(self->fd, HMM_DMIRROR_FLAGS, buffer, + HMM_DMIRROR_FLAG_FAIL_ALLOC); + ASSERT_EQ(ret, 0); + ret =3D hmm_migrate_sys_to_dev(self->fd, buffer, npages); + ASSERT_EQ(ret, 0); + ASSERT_EQ(buffer->cpages, npages); + + /* Check what the device read. */ + for (i =3D 0, ptr =3D buffer->mirror; i < size / sizeof(*ptr); ++i) { + ASSERT_EQ(ptr[i], i); + if (ptr[i] !=3D i) + break; + } + + /* Try faulting back a single (PAGE_SIZE) page. */ + ptr =3D buffer->ptr; + ASSERT_EQ(ptr[2048], 2048); + + /* unmap and remap the region to reset things. */ + ret =3D munmap(old_ptr, 2 * size); + ASSERT_EQ(ret, 0); + old_ptr =3D mmap(NULL, 2 * size, PROT_READ | PROT_WRITE, + MAP_PRIVATE | MAP_ANONYMOUS, buffer->fd, 0); + ASSERT_NE(old_ptr, MAP_FAILED); + map =3D (void *)ALIGN((uintptr_t)old_ptr, size); + ret =3D madvise(map, size, MADV_HUGEPAGE); + ASSERT_EQ(ret, 0); + buffer->ptr =3D map; + + /* Initialize buffer in system memory. */ + for (i =3D 0, ptr =3D buffer->ptr; i < size / sizeof(*ptr); ++i) + ptr[i] =3D i; + + /* Migrate THP to device. */ + ret =3D hmm_migrate_sys_to_dev(self->fd, buffer, npages); + ASSERT_EQ(ret, 0); + ASSERT_EQ(buffer->cpages, npages); + + /* + * Force an allocation error when faulting back a THP resident in the + * device. + */ + ret =3D hmm_dmirror_cmd(self->fd, HMM_DMIRROR_FLAGS, buffer, + HMM_DMIRROR_FLAG_FAIL_ALLOC); + ASSERT_EQ(ret, 0); + + ret =3D hmm_migrate_dev_to_sys(self->fd, buffer, npages); + ASSERT_EQ(ret, 0); + ptr =3D buffer->ptr; + ASSERT_EQ(ptr[2048], 2048); + + buffer->ptr =3D old_ptr; + hmm_buffer_free(buffer); +} + +/* + * Migrate private anonymous huge zero page with allocation errors. + */ +TEST_F(hmm, migrate_anon_huge_zero_err) +{ + struct hmm_buffer *buffer; + unsigned long npages; + unsigned long size; + unsigned long i; + void *old_ptr; + void *map; + int *ptr; + int ret; + + size =3D TWOMEG; + + buffer =3D malloc(sizeof(*buffer)); + ASSERT_NE(buffer, NULL); + + buffer->fd =3D -1; + buffer->size =3D 2 * size; + buffer->mirror =3D malloc(2 * size); + ASSERT_NE(buffer->mirror, NULL); + memset(buffer->mirror, 0xFF, 2 * size); + + old_ptr =3D mmap(NULL, 2 * size, PROT_READ, + MAP_PRIVATE | MAP_ANONYMOUS, buffer->fd, 0); + ASSERT_NE(old_ptr, MAP_FAILED); + + npages =3D size >> self->page_shift; + map =3D (void *)ALIGN((uintptr_t)old_ptr, size); + ret =3D madvise(map, size, MADV_HUGEPAGE); + ASSERT_EQ(ret, 0); + buffer->ptr =3D map; + + /* Migrate memory to device but force a THP allocation error. */ + ret =3D hmm_dmirror_cmd(self->fd, HMM_DMIRROR_FLAGS, buffer, + HMM_DMIRROR_FLAG_FAIL_ALLOC); + ASSERT_EQ(ret, 0); + ret =3D hmm_migrate_sys_to_dev(self->fd, buffer, npages); + ASSERT_EQ(ret, 0); + ASSERT_EQ(buffer->cpages, npages); + + /* Check what the device read. */ + for (i =3D 0, ptr =3D buffer->mirror; i < size / sizeof(*ptr); ++i) + ASSERT_EQ(ptr[i], 0); + + /* Try faulting back a single (PAGE_SIZE) page. */ + ptr =3D buffer->ptr; + ASSERT_EQ(ptr[2048], 0); + + /* unmap and remap the region to reset things. */ + ret =3D munmap(old_ptr, 2 * size); + ASSERT_EQ(ret, 0); + old_ptr =3D mmap(NULL, 2 * size, PROT_READ, + MAP_PRIVATE | MAP_ANONYMOUS, buffer->fd, 0); + ASSERT_NE(old_ptr, MAP_FAILED); + map =3D (void *)ALIGN((uintptr_t)old_ptr, size); + ret =3D madvise(map, size, MADV_HUGEPAGE); + ASSERT_EQ(ret, 0); + buffer->ptr =3D map; + + /* Initialize buffer in system memory (zero THP page). */ + ret =3D ptr[0]; + ASSERT_EQ(ret, 0); + + /* Migrate memory to device but force a THP allocation error. */ + ret =3D hmm_dmirror_cmd(self->fd, HMM_DMIRROR_FLAGS, buffer, + HMM_DMIRROR_FLAG_FAIL_ALLOC); + ASSERT_EQ(ret, 0); + ret =3D hmm_migrate_sys_to_dev(self->fd, buffer, npages); + ASSERT_EQ(ret, 0); + ASSERT_EQ(buffer->cpages, npages); + + /* Fault the device memory back and check it. */ + for (i =3D 0, ptr =3D buffer->ptr; i < size / sizeof(*ptr); ++i) + ASSERT_EQ(ptr[i], 0); + + buffer->ptr =3D old_ptr; + hmm_buffer_free(buffer); +} TEST_HARNESS_MAIN --=20 2.50.1 From nobody Sat Oct 4 22:37:10 2025 Received: from NAM02-SN1-obe.outbound.protection.outlook.com (mail-sn1nam02on2089.outbound.protection.outlook.com [40.107.96.89]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 414EC2E5B0C for ; Tue, 12 Aug 2025 02:41:22 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.96.89 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1754966484; cv=fail; b=kfJIEO2p6K4K1gHk9KzYomGk/IOBfoB2v0ij3bE1uQby13g7vmpyMkDJQ216kbo/uXlDNj7xxEbc6kpyUdM3Gn69OVHkgiMTSKiwU6/9oIrzADu4Ep9E1j/a9vwNEgK5pE20fITbUs3MgAiFqXRzpr8XWhUcntV2SFpgaiGjLYg= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1754966484; c=relaxed/simple; bh=x1QkON2uZInQ8pqCNuXX17OV/c1IKdMmbuxJVi0v5AM=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: Content-Type:MIME-Version; b=E/GR9oWAaSZ9cEyp7Czt7wBFq6FPdTLU8W38GVUSe0+L0OJdCOd2wSaK7HP33W39vlnJIZ735ps3kVGR9pobm53mk0l/1zkvfrNAjiy+hkxhf07dgB1lZX0cAjSaoS71U/lK8xeRTbS3MNUSTrjohfuFOSnnUP5+KpZu+wfNdGw= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com; spf=fail smtp.mailfrom=nvidia.com; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b=WHrq9H5V; arc=fail smtp.client-ip=40.107.96.89 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="WHrq9H5V" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=gK2UsZCcBaif6ODZNgMID4AqwXvHT1izjr9zDDt+a5V5QwPng8msgUBP3qS4FxHv3sI4h+2t5Owo1u34E/o77LqyVzPX8HBmtt2aTBJv42OtS1LjY1zWrA6COnuQ63PA4/01zzWwcwjFRayaKGDfiBVEd+tIx92FONufvVMxhIWo0w1OiuU05pfQMlQvcIyoxLsvtkXZB9Fo68pDXmoTfZpU2lLO9Tx5HRkCO5YHy0Oppgsq85cstddKNdVoWxfcJ22nv6lRQ7/2KjODvTVwNuvu4QF3dPNOUhj6ki1YKn0uUB0HQbLYw5X+puICLbkZ6rRYo7Oil57XdpoYj2aYnw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=dcIeQ6mGEnh1gATpgNs0IJRozX0C5y8a8f0HsOJvhtk=; b=e0zUMklZr6M3vXj7ZtqnMpJYePP+aIgXHcejzMLNb6l0h/NHuK+PRQ0eEBcQMoOhwnPNhFo2HNdymeMzAE0VX3SIe2rk09KHhC9QHowr/5y+XryftLHwdXg9XyYN0tGG6pPqffE4IVaTsUL6IEJU0UCe67grKN/nAWFwfa4snhmXsRgD0LF8qPfRUC/8gjMtwve5qMKTMMRt8CIDT/tBMgDyZer1mE9HYGN5WuRvCc9As3wV9lqwAIVLpN9G7H+mq7/5CnU5pjBedZtGymlcV8pySZoP0LdhORmo3HMiGTK514ciGMiDCnyA3Gg6FGtcJr3oS/by3zzTRu0oYtyW8A== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=dcIeQ6mGEnh1gATpgNs0IJRozX0C5y8a8f0HsOJvhtk=; b=WHrq9H5VUXcU9x26xuVK5OvqiNvSAyJAfJqfZ0ATCm2TRE+28I/mw8K3CTj/ecuc6Q5Y31ruwHPjy2MSsPrFqvCsW91ovvu5HzqPZO8yE6iePouS2cFrIZBd3ZyWQ1hUgSTjH9+9uC8EOZWl3g7BHQfQCDYsjRTnHq6s/V6Gs8tMscasoSbJBV7kgbi1nnfKvcObM08WgTqyOJhatBgWjvJ27/fTm28kFcFqKqbSXSS3UGXJPAWTEKz6/IOPwdWVNqEpLLpnALHdkPRgy0jI6sp8IlCkh3fWz95FQjtHUIW9PJQDMn2hA+e6kKGpFkmGuYDUnIbUmp86Zy8KUoAFuw== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from PH8PR12MB7277.namprd12.prod.outlook.com (2603:10b6:510:223::13) by MN0PR12MB6317.namprd12.prod.outlook.com (2603:10b6:208:3c2::12) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9009.13; Tue, 12 Aug 2025 02:41:17 +0000 Received: from PH8PR12MB7277.namprd12.prod.outlook.com ([fe80::3a4:70ea:ff05:1251]) by PH8PR12MB7277.namprd12.prod.outlook.com ([fe80::3a4:70ea:ff05:1251%5]) with mapi id 15.20.9009.018; Tue, 12 Aug 2025 02:41:17 +0000 From: Balbir Singh To: dri-devel@lists.freedesktop.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Balbir Singh , Andrew Morton , David Hildenbrand , Zi Yan , Joshua Hahn , Rakie Kim , Byungchul Park , Gregory Price , Ying Huang , Alistair Popple , Oscar Salvador , Lorenzo Stoakes , Baolin Wang , "Liam R. Howlett" , Nico Pache , Ryan Roberts , Dev Jain , Barry Song , Lyude Paul , Danilo Krummrich , David Airlie , Simona Vetter , Ralph Campbell , =?UTF-8?q?Mika=20Penttil=C3=A4?= , Matthew Brost , Francois Dugast Subject: [v3 10/11] gpu/drm/nouveau: add THP migration support Date: Tue, 12 Aug 2025 12:40:35 +1000 Message-ID: <20250812024036.690064-11-balbirs@nvidia.com> X-Mailer: git-send-email 2.50.1 In-Reply-To: <20250812024036.690064-1-balbirs@nvidia.com> References: <20250812024036.690064-1-balbirs@nvidia.com> Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: SY6PR01CA0001.ausprd01.prod.outlook.com (2603:10c6:10:e8::6) To PH8PR12MB7277.namprd12.prod.outlook.com (2603:10b6:510:223::13) Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: PH8PR12MB7277:EE_|MN0PR12MB6317:EE_ X-MS-Office365-Filtering-Correlation-Id: 52293748-a871-44f0-b7c5-08ddd949b657 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|376014|7416014|366016|1800799024; X-Microsoft-Antispam-Message-Info: =?utf-8?B?cDNGZXVpMFpERkhuSDBDSG13TEE3aEJTTklsMnV2a2x4akNzdkVuYUtRU0xn?= =?utf-8?B?bUI2ZHF2YXYzWUx6OUlNZGZvNWxyL1hFZzJxcjZJNzdjUG1Dd1pvdExiYzV3?= =?utf-8?B?emF6R280NzBsaENEUGtiRzZuZUFyWm05dG1DRWpxOW83K1FMVFpDVm9LTDVo?= =?utf-8?B?NUpmTGFpZnhnSk1ieis4WHJ6WWtxUG0va0lZN0REdHorWXBJdzQ3SG1OakN3?= =?utf-8?B?aFZSWGNlLzdRdU1vRDhEMVVHL1Y3SG81cy9yNElDWnJxYzJxc3FvWFVWcFlS?= =?utf-8?B?YkdPRHlZZ0hEbEdmS3hnWWdxZVFIanJuV2FYQ0pLLytjVDNaa0ZWMU9hejBL?= =?utf-8?B?a2VYcDY4OTF6L2hjSUdLUi9UL2RCemNMVCtZOHNHOEp3ZFQ1M3JIZkpJN0Z4?= =?utf-8?B?SGc4WGdCN0xlcjliTjdWMzNldEhqZFp2d0pmYjFvdllIVUM4RGNOdDdGb2V3?= =?utf-8?B?Ykxwa2IwNnZOakVkbVZ5T2dDdThNRCtwdWhHd0o1UkpTb1hoVFFtUVdsTWlz?= =?utf-8?B?OHhRRHJjcDZDY2FaeE84QkU1T3FINUE4NEU0K2VNNmxSbFV0RzJmVUQwM3Nx?= =?utf-8?B?bDhTSE1VamlYcEJGdU1aSVBjRGxoWnVvZU5xby8rdjE2SERNV2ZjcEh0VHVI?= =?utf-8?B?MkxuWmIyMk0vWmJleTNEdGJwT1YreEk5OXhhOStXREV2WUl2MiszYTV1ODJ6?= =?utf-8?B?YVRKTTRPSnlqVWJjSlY5aXZyUm9BcE5ZM3dnWStYamtmRWtQWFRaQlp5M3N4?= =?utf-8?B?eGMzMWhGVnVkbzFDbEJaL3lIdTl2aGN5QXVZN0E1Y01KdWRQMndLdjJad3g4?= =?utf-8?B?ekdWWWFrTHNmcU1aK21WY0cwcTJZNzRnUUttaXJzTWlaV3A1c0RsYzliRGk5?= =?utf-8?B?T3ZTbjhIQ2tLYkx4K2diUXVWLzEzL0p0b0J2RktjaVJKUUFHaXREdE96aEdY?= =?utf-8?B?K2tCbDFOejMzL2l6b1YxYTM2ODVwenM3TTRDejAya0s0d0U0ejBab0dkTlU5?= =?utf-8?B?NnZEeTJTZWZETy92a0ZSOVVhWSsvWlEvT214VitMdHNpWG1zVVJPa2VhRFJO?= =?utf-8?B?OXZHbTVRMCtSN3Bhd01OYlVWeWlHa2hGdS8rbzFMS3R2VzV3L2YvN3ptc3da?= =?utf-8?B?dy9vMEZRNVlVVHl0aWJUNDNUalpRcW1kamJNU1lPdE8rT21adzZlUHRkSFZa?= =?utf-8?B?ZlVwRTVieEtmeUQ0Mm9MbGMxd3VZYStqZ1pGZTZhWVhhcUxCQXJTZDVyeXlk?= =?utf-8?B?U1RDRjlIK1Vmc1NoL2tIMlI0TjQzUC9SZFp1eXN6TUF0ZXZJN3lxWGY1TkZM?= =?utf-8?B?bW1PaFNUWHZ2Z0RJcmVPcUh5c2FFSnFKdWppczNIUHpFVFdlZmUvQ3gvd3N0?= =?utf-8?B?NWpnSHYzcktPSnZZWnZ0WmhSaW5zeTZ5WmkyblZFM1J3d2dYK1U2bm1EdG9Y?= =?utf-8?B?RGNBbWtQQ3ByemZhYms0Q2pmV3RoNjhydC84MEJ0RGhUblVudXV1N1p3clJB?= =?utf-8?B?cTgyT3lVcWZzNjVrV0FvTEdxblkwZVhGdVVqSm1ERGRGTEp4NmFGcklBeHcz?= =?utf-8?B?dXA1bGpOVmEzZnQzRlZxaVl0Rm5BUEU2am1aRUhXakVHZEwwaDBXRFh3Z0cw?= =?utf-8?B?N2JpNDh5VjBwOGdWQ0t6eUEva1dXbHRUaGErbFVQQmhTWDlqZ2NvcVpOTk9F?= =?utf-8?B?cWRsN2NJZjdQK2l2NElrYjJJZ1dDZ1orRGZMdVdSa2YwMDY4dHRZRmZZcWRm?= =?utf-8?B?VzBSSFFZRFQzVy9TZFVkdTFwRXQ4YS9wRmYza05jWUxDekNmYjl4b0xQTFRB?= =?utf-8?B?MERsZnR2UlExSHl5SkQ2QnJJdWdhUFowbFFuN096TVl4aGxKN3NKeFRVcWZW?= =?utf-8?B?YWQrdmtMWkVLOHRLYmxqOGY1K3BjQ09kMUl3M2ZvOFdRRlB1NTFrZnUzUTJ3?= =?utf-8?Q?58DOc5sGZB8=3D?= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:PH8PR12MB7277.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(376014)(7416014)(366016)(1800799024);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?clNlRWcvWHVFY3hmVnk3ZkMyMlVpbEJPdjJJMHdCdlZCNnhDcUFSYmd4SlF5?= =?utf-8?B?K3FMbUk4dmtITmJqUTViaGNrdEFGNWpUT2tIOGRuSSszenlFVnhid0w2VlJt?= =?utf-8?B?d1pEVHRWNURrSXBhejlFMDMzUUJldmdGYlpCRTFPdUhUVDl3UkJRSDRQWU4v?= =?utf-8?B?ajM2NHByWUdTZGhYU2lsZExGcHp6ZkVxNVNrcEFBSWxkYkhLUjd0eXdsTGFP?= =?utf-8?B?RWgralM0VXJ0RmJLRzJkdWdEbXYyVldtVUIySzRRcjJoM1libWxOMWMybDc4?= =?utf-8?B?Q3k5b3BjRGR0WUlrVmJIT29pN0VIa3dPUHRFVThWdi9sUWdTVjFHQlZ3Y0lK?= =?utf-8?B?bDZ1MFlrY0hCTXUvaWtqYkRlUk52bkpiWk8wQ3cvakVxbUY4THpwUXUyQ3l1?= =?utf-8?B?ZmovNVUrZkljU2hsTmxBRzJyd0huMitTbVVjcEFQWmI5K2ppdjlwLzJraTNp?= =?utf-8?B?SU9ablNPTUFSejhNVDh5a0VzVGRqUjBrMEo4L0k0Wk1iaGR0NWY0ajhReWVE?= =?utf-8?B?YVlrMEh6Q09iQUEydGRpeDRJNG52RlFOL2VqUDNmS0JaQUJ3MnFwRlNJeHht?= =?utf-8?B?K3N3ejd0WlExVklNZG5WL1BOMmNXbmN1cXQ3VzdrMEtTTkpKZTJlOHVBbHM1?= =?utf-8?B?cmtYRnpCUHRoamkyVHkxU25oaTY3bXovKzE0dFY5NTQyZDdpUmhSOERkbUpy?= =?utf-8?B?WGFSSkF0cDlWQlUwaERBVjdFTHZGaDFyektmaGNyMkxvMkZmK1FtLzRuSTF4?= =?utf-8?B?Y1JsMklFUkpuMlZoTHFzdEZGTWRUdFFJb2hrc256U214SXczd3ZWVWRFaUNS?= =?utf-8?B?b0tuUWpGckRad0pYNXBoMjN6SnRhVUZFZE5qdzN5UndtOGJ0SUFjUU5JY0FK?= =?utf-8?B?RWd2U1JFL3l0eHBoRDhhc3Z2M0ViallhVFk5dURzSEVzMGJ1VGtaM2tUZnB4?= =?utf-8?B?OUJ4Q1ZSMEYyNGJEcDV0Y05Pb2tlZG5XYi92YzRQRTJkTjZMaDJ5ZTQxbHd2?= =?utf-8?B?K0NteW9Sam9yb0lNN3NDOVd4d0ZrNGlXK1d5a01rODdiMTNXWGNkL0dkQlpS?= =?utf-8?B?TW9YUERVTlo5Um9mTGhlbmZBTXpkN3VaUDl3T3RVdndoUzRqM3ZjRHYzTmhm?= =?utf-8?B?VEVrWTZFN2pScm5yR0J2UnovbmZ5bFNueldudFFYQ05RUTBZZWkrY3Mra2FU?= =?utf-8?B?Wi9YUGJ4ajQ0S3JHaDN2L0NOOU9VejNSaVZOSnl0OWV6cml0cXhGYm01Tmpz?= =?utf-8?B?TEVYZGpVVklHSjJTcWZzYjdSa2cva1JDSi9vbHRYWUpPMFJlZWhNZ1JJRGxR?= =?utf-8?B?RE1WS3ZNZmsrd2o1NFhQK1VYeHlTS2hIZ1JXQ2FwZmpvaFo4QU5lK3lQd3Uv?= =?utf-8?B?azNUa1VQeHFCS3NVZmVIMjg3SWNBckxPWC8xczRQT0lpek1hVDdqaXNmTkhV?= =?utf-8?B?cnlmQmp3bmlmaUZZeC9DZlF1dkl4SnFwa3E4bHlEWTFHblphNFhYM0hwcktl?= =?utf-8?B?czVOUmhNMHRNaTRoNCtCL2pYdEtrMUZsV1JpcWg1UlRPWGdNbDZSRzRuM01o?= =?utf-8?B?RHVqVWpHVVJ3SGhBMGZhYVFZZjcvMVlqWCtPT28yR1VFcjFCaGZOT2lCc3NK?= =?utf-8?B?QVdEa1ZEU2NkWVl3ZzdJVWxtc3R5WGlpN2dNSXpHUmVVYnRhRC83RUlCR0F5?= =?utf-8?B?dWVjU1hKMzlwMCtYZ3VjQno0QWpFYmczUHZFTjhVMm52SWI3NHgrZ0ozWk1G?= =?utf-8?B?UmNDVTN3UnQrazM5dUZ1WFUydkN2YkFvLy9yUCtnMnRMYUFmaXExNmFwOG5M?= =?utf-8?B?eGNVSUNHc0tFemJXZ0pSMGpBek9FYkRDdlZZa0JDbmVlNWtXTW5RNVV1a3JX?= =?utf-8?B?U2NnT2luaVFRbjcvaUJjd0trUUQ5bnpWZm5raXpqeXZDc3NJRG5sNHZ4TFEv?= =?utf-8?B?d2VucFF2dVFZeFZDdkdqMEFtbDltNEVQM3BMckl0aFFCdDRFSUVNMWVVeTJi?= =?utf-8?B?RXBVWlI2VVI4TC8za3VnbjltMjZ6TTZFTHVWcFRoWVdaeXBYem1hMGdVZmlz?= =?utf-8?B?MXRBWVE5MUpmalBWWVV3bmpRMWZyaGZ5K0xBYzd1TG55WitXV29pakhPcVdV?= =?utf-8?Q?XX44gzXINmfkxVSyjdeIurWlC?= X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: 52293748-a871-44f0-b7c5-08ddd949b657 X-MS-Exchange-CrossTenant-AuthSource: PH8PR12MB7277.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 12 Aug 2025 02:41:17.3117 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: xKahMXWmA94MSg0kGLo4QDDPnyEhZ3Zka9CjDBjMtNtiL87f/oHrUgp5/dFL1HQuTxsWin5qf7XKGfhXnRlk1Q== X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN0PR12MB6317 Change the code to add support for MIGRATE_VMA_SELECT_COMPOUND and appropriately handling page sizes in the migrate/evict code paths. Cc: Andrew Morton Cc: David Hildenbrand Cc: Zi Yan Cc: Joshua Hahn Cc: Rakie Kim Cc: Byungchul Park Cc: Gregory Price Cc: Ying Huang Cc: Alistair Popple Cc: Oscar Salvador Cc: Lorenzo Stoakes Cc: Baolin Wang Cc: "Liam R. Howlett" Cc: Nico Pache Cc: Ryan Roberts Cc: Dev Jain Cc: Barry Song Cc: Lyude Paul Cc: Danilo Krummrich Cc: David Airlie Cc: Simona Vetter Cc: Ralph Campbell Cc: Mika Penttil=C3=A4 Cc: Matthew Brost Cc: Francois Dugast Signed-off-by: Balbir Singh --- drivers/gpu/drm/nouveau/nouveau_dmem.c | 253 ++++++++++++++++++------- drivers/gpu/drm/nouveau/nouveau_svm.c | 6 +- drivers/gpu/drm/nouveau/nouveau_svm.h | 3 +- 3 files changed, 186 insertions(+), 76 deletions(-) diff --git a/drivers/gpu/drm/nouveau/nouveau_dmem.c b/drivers/gpu/drm/nouve= au/nouveau_dmem.c index ca4932a150e3..408c1adf6f20 100644 --- a/drivers/gpu/drm/nouveau/nouveau_dmem.c +++ b/drivers/gpu/drm/nouveau/nouveau_dmem.c @@ -83,9 +83,15 @@ struct nouveau_dmem { struct list_head chunks; struct mutex mutex; struct page *free_pages; + struct folio *free_folios; spinlock_t lock; }; =20 +struct nouveau_dmem_dma_info { + dma_addr_t dma_addr; + size_t size; +}; + static struct nouveau_dmem_chunk *nouveau_page_to_chunk(struct page *page) { return container_of(page_pgmap(page), struct nouveau_dmem_chunk, @@ -112,10 +118,16 @@ static void nouveau_dmem_page_free(struct page *page) { struct nouveau_dmem_chunk *chunk =3D nouveau_page_to_chunk(page); struct nouveau_dmem *dmem =3D chunk->drm->dmem; + struct folio *folio =3D page_folio(page); =20 spin_lock(&dmem->lock); - page->zone_device_data =3D dmem->free_pages; - dmem->free_pages =3D page; + if (folio_order(folio)) { + folio_set_zone_device_data(folio, dmem->free_folios); + dmem->free_folios =3D folio; + } else { + page->zone_device_data =3D dmem->free_pages; + dmem->free_pages =3D page; + } =20 WARN_ON(!chunk->callocated); chunk->callocated--; @@ -139,20 +151,28 @@ static void nouveau_dmem_fence_done(struct nouveau_fe= nce **fence) } } =20 -static int nouveau_dmem_copy_one(struct nouveau_drm *drm, struct page *spa= ge, - struct page *dpage, dma_addr_t *dma_addr) +static int nouveau_dmem_copy_folio(struct nouveau_drm *drm, + struct folio *sfolio, struct folio *dfolio, + struct nouveau_dmem_dma_info *dma_info) { struct device *dev =3D drm->dev->dev; + struct page *dpage =3D folio_page(dfolio, 0); + struct page *spage =3D folio_page(sfolio, 0); =20 - lock_page(dpage); + folio_lock(dfolio); =20 - *dma_addr =3D dma_map_page(dev, dpage, 0, PAGE_SIZE, DMA_BIDIRECTIONAL); - if (dma_mapping_error(dev, *dma_addr)) + dma_info->dma_addr =3D dma_map_page(dev, dpage, 0, page_size(dpage), + DMA_BIDIRECTIONAL); + dma_info->size =3D page_size(dpage); + if (dma_mapping_error(dev, dma_info->dma_addr)) return -EIO; =20 - if (drm->dmem->migrate.copy_func(drm, 1, NOUVEAU_APER_HOST, *dma_addr, - NOUVEAU_APER_VRAM, nouveau_dmem_page_addr(spage))) { - dma_unmap_page(dev, *dma_addr, PAGE_SIZE, DMA_BIDIRECTIONAL); + if (drm->dmem->migrate.copy_func(drm, folio_nr_pages(sfolio), + NOUVEAU_APER_HOST, dma_info->dma_addr, + NOUVEAU_APER_VRAM, + nouveau_dmem_page_addr(spage))) { + dma_unmap_page(dev, dma_info->dma_addr, page_size(dpage), + DMA_BIDIRECTIONAL); return -EIO; } =20 @@ -165,21 +185,47 @@ static vm_fault_t nouveau_dmem_migrate_to_ram(struct = vm_fault *vmf) struct nouveau_dmem *dmem =3D drm->dmem; struct nouveau_fence *fence; struct nouveau_svmm *svmm; - struct page *spage, *dpage; - unsigned long src =3D 0, dst =3D 0; - dma_addr_t dma_addr =3D 0; + struct page *dpage; vm_fault_t ret =3D 0; struct migrate_vma args =3D { .vma =3D vmf->vma, - .start =3D vmf->address, - .end =3D vmf->address + PAGE_SIZE, - .src =3D &src, - .dst =3D &dst, .pgmap_owner =3D drm->dev, .fault_page =3D vmf->page, - .flags =3D MIGRATE_VMA_SELECT_DEVICE_PRIVATE, + .flags =3D MIGRATE_VMA_SELECT_DEVICE_PRIVATE | + MIGRATE_VMA_SELECT_COMPOUND, + .src =3D NULL, + .dst =3D NULL, }; + unsigned int order, nr; + struct folio *sfolio, *dfolio; + struct nouveau_dmem_dma_info dma_info; + + sfolio =3D page_folio(vmf->page); + order =3D folio_order(sfolio); + nr =3D 1 << order; =20 + /* + * Handle partial unmap faults, where the folio is large, but + * the pmd is split. + */ + if (vmf->pte) { + order =3D 0; + nr =3D 1; + } + + if (order) + args.flags |=3D MIGRATE_VMA_SELECT_COMPOUND; + + args.start =3D ALIGN_DOWN(vmf->address, (1 << (PAGE_SHIFT + order))); + args.vma =3D vmf->vma; + args.end =3D args.start + (PAGE_SIZE << order); + args.src =3D kcalloc(nr, sizeof(*args.src), GFP_KERNEL); + args.dst =3D kcalloc(nr, sizeof(*args.dst), GFP_KERNEL); + + if (!args.src || !args.dst) { + ret =3D VM_FAULT_OOM; + goto err; + } /* * FIXME what we really want is to find some heuristic to migrate more * than just one page on CPU fault. When such fault happens it is very @@ -190,20 +236,26 @@ static vm_fault_t nouveau_dmem_migrate_to_ram(struct = vm_fault *vmf) if (!args.cpages) return 0; =20 - spage =3D migrate_pfn_to_page(src); - if (!spage || !(src & MIGRATE_PFN_MIGRATE)) - goto done; - - dpage =3D alloc_page_vma(GFP_HIGHUSER | __GFP_ZERO, vmf->vma, vmf->addres= s); - if (!dpage) + if (order) + dpage =3D folio_page(vma_alloc_folio(GFP_HIGHUSER | __GFP_ZERO, + order, vmf->vma, vmf->address), 0); + else + dpage =3D alloc_page_vma(GFP_HIGHUSER | __GFP_ZERO, vmf->vma, + vmf->address); + if (!dpage) { + ret =3D VM_FAULT_OOM; goto done; + } =20 - dst =3D migrate_pfn(page_to_pfn(dpage)); + args.dst[0] =3D migrate_pfn(page_to_pfn(dpage)); + if (order) + args.dst[0] |=3D MIGRATE_PFN_COMPOUND; + dfolio =3D page_folio(dpage); =20 - svmm =3D spage->zone_device_data; + svmm =3D folio_zone_device_data(sfolio); mutex_lock(&svmm->mutex); nouveau_svmm_invalidate(svmm, args.start, args.end); - ret =3D nouveau_dmem_copy_one(drm, spage, dpage, &dma_addr); + ret =3D nouveau_dmem_copy_folio(drm, sfolio, dfolio, &dma_info); mutex_unlock(&svmm->mutex); if (ret) { ret =3D VM_FAULT_SIGBUS; @@ -213,19 +265,33 @@ static vm_fault_t nouveau_dmem_migrate_to_ram(struct = vm_fault *vmf) nouveau_fence_new(&fence, dmem->migrate.chan); migrate_vma_pages(&args); nouveau_dmem_fence_done(&fence); - dma_unmap_page(drm->dev->dev, dma_addr, PAGE_SIZE, DMA_BIDIRECTIONAL); + dma_unmap_page(drm->dev->dev, dma_info.dma_addr, PAGE_SIZE, + DMA_BIDIRECTIONAL); done: migrate_vma_finalize(&args); +err: + kfree(args.src); + kfree(args.dst); return ret; } =20 +static void nouveau_dmem_folio_split(struct folio *head, struct folio *tai= l) +{ + if (tail =3D=3D NULL) + return; + tail->pgmap =3D head->pgmap; + folio_set_zone_device_data(tail, folio_zone_device_data(head)); +} + static const struct dev_pagemap_ops nouveau_dmem_pagemap_ops =3D { .page_free =3D nouveau_dmem_page_free, .migrate_to_ram =3D nouveau_dmem_migrate_to_ram, + .folio_split =3D nouveau_dmem_folio_split, }; =20 static int -nouveau_dmem_chunk_alloc(struct nouveau_drm *drm, struct page **ppage) +nouveau_dmem_chunk_alloc(struct nouveau_drm *drm, struct page **ppage, + bool is_large) { struct nouveau_dmem_chunk *chunk; struct resource *res; @@ -274,16 +340,21 @@ nouveau_dmem_chunk_alloc(struct nouveau_drm *drm, str= uct page **ppage) pfn_first =3D chunk->pagemap.range.start >> PAGE_SHIFT; page =3D pfn_to_page(pfn_first); spin_lock(&drm->dmem->lock); - for (i =3D 0; i < DMEM_CHUNK_NPAGES - 1; ++i, ++page) { - page->zone_device_data =3D drm->dmem->free_pages; - drm->dmem->free_pages =3D page; + + if (!IS_ENABLED(CONFIG_TRANSPARENT_HUGEPAGE) || !is_large) { + for (i =3D 0; i < DMEM_CHUNK_NPAGES - 1; ++i, ++page) { + page->zone_device_data =3D drm->dmem->free_pages; + drm->dmem->free_pages =3D page; + } } + *ppage =3D page; chunk->callocated++; spin_unlock(&drm->dmem->lock); =20 - NV_INFO(drm, "DMEM: registered %ldMB of device memory\n", - DMEM_CHUNK_SIZE >> 20); + NV_INFO(drm, "DMEM: registered %ldMB of %sdevice memory %lx %lx\n", + DMEM_CHUNK_SIZE >> 20, is_large ? "THP " : "", pfn_first, + nouveau_dmem_page_addr(page)); =20 return 0; =20 @@ -298,27 +369,37 @@ nouveau_dmem_chunk_alloc(struct nouveau_drm *drm, str= uct page **ppage) } =20 static struct page * -nouveau_dmem_page_alloc_locked(struct nouveau_drm *drm) +nouveau_dmem_page_alloc_locked(struct nouveau_drm *drm, bool is_large) { struct nouveau_dmem_chunk *chunk; struct page *page =3D NULL; + struct folio *folio =3D NULL; int ret; + unsigned int order =3D 0; =20 spin_lock(&drm->dmem->lock); - if (drm->dmem->free_pages) { + if (is_large && drm->dmem->free_folios) { + folio =3D drm->dmem->free_folios; + drm->dmem->free_folios =3D folio_zone_device_data(folio); + chunk =3D nouveau_page_to_chunk(page); + chunk->callocated++; + spin_unlock(&drm->dmem->lock); + order =3D DMEM_CHUNK_NPAGES; + } else if (!is_large && drm->dmem->free_pages) { page =3D drm->dmem->free_pages; drm->dmem->free_pages =3D page->zone_device_data; chunk =3D nouveau_page_to_chunk(page); chunk->callocated++; spin_unlock(&drm->dmem->lock); + folio =3D page_folio(page); } else { spin_unlock(&drm->dmem->lock); - ret =3D nouveau_dmem_chunk_alloc(drm, &page); + ret =3D nouveau_dmem_chunk_alloc(drm, &page, is_large); if (ret) return NULL; } =20 - zone_device_page_init(page); + zone_device_folio_init(folio, order); return page; } =20 @@ -369,12 +450,12 @@ nouveau_dmem_evict_chunk(struct nouveau_dmem_chunk *c= hunk) { unsigned long i, npages =3D range_len(&chunk->pagemap.range) >> PAGE_SHIF= T; unsigned long *src_pfns, *dst_pfns; - dma_addr_t *dma_addrs; + struct nouveau_dmem_dma_info *dma_info; struct nouveau_fence *fence; =20 src_pfns =3D kvcalloc(npages, sizeof(*src_pfns), GFP_KERNEL | __GFP_NOFAI= L); dst_pfns =3D kvcalloc(npages, sizeof(*dst_pfns), GFP_KERNEL | __GFP_NOFAI= L); - dma_addrs =3D kvcalloc(npages, sizeof(*dma_addrs), GFP_KERNEL | __GFP_NOF= AIL); + dma_info =3D kvcalloc(npages, sizeof(*dma_info), GFP_KERNEL | __GFP_NOFAI= L); =20 migrate_device_range(src_pfns, chunk->pagemap.range.start >> PAGE_SHIFT, npages); @@ -382,17 +463,28 @@ nouveau_dmem_evict_chunk(struct nouveau_dmem_chunk *c= hunk) for (i =3D 0; i < npages; i++) { if (src_pfns[i] & MIGRATE_PFN_MIGRATE) { struct page *dpage; + struct folio *folio =3D page_folio( + migrate_pfn_to_page(src_pfns[i])); + unsigned int order =3D folio_order(folio); + + if (src_pfns[i] & MIGRATE_PFN_COMPOUND) { + dpage =3D folio_page( + folio_alloc( + GFP_HIGHUSER_MOVABLE, order), 0); + } else { + /* + * _GFP_NOFAIL because the GPU is going away and there + * is nothing sensible we can do if we can't copy the + * data back. + */ + dpage =3D alloc_page(GFP_HIGHUSER | __GFP_NOFAIL); + } =20 - /* - * _GFP_NOFAIL because the GPU is going away and there - * is nothing sensible we can do if we can't copy the - * data back. - */ - dpage =3D alloc_page(GFP_HIGHUSER | __GFP_NOFAIL); dst_pfns[i] =3D migrate_pfn(page_to_pfn(dpage)); - nouveau_dmem_copy_one(chunk->drm, - migrate_pfn_to_page(src_pfns[i]), dpage, - &dma_addrs[i]); + nouveau_dmem_copy_folio(chunk->drm, + page_folio(migrate_pfn_to_page(src_pfns[i])), + page_folio(dpage), + &dma_info[i]); } } =20 @@ -403,8 +495,9 @@ nouveau_dmem_evict_chunk(struct nouveau_dmem_chunk *chu= nk) kvfree(src_pfns); kvfree(dst_pfns); for (i =3D 0; i < npages; i++) - dma_unmap_page(chunk->drm->dev->dev, dma_addrs[i], PAGE_SIZE, DMA_BIDIRE= CTIONAL); - kvfree(dma_addrs); + dma_unmap_page(chunk->drm->dev->dev, dma_info[i].dma_addr, + dma_info[i].size, DMA_BIDIRECTIONAL); + kvfree(dma_info); } =20 void @@ -607,31 +700,35 @@ nouveau_dmem_init(struct nouveau_drm *drm) =20 static unsigned long nouveau_dmem_migrate_copy_one(struct nouveau_drm *drm, struct nouveau_svmm *svmm, unsigned long src, - dma_addr_t *dma_addr, u64 *pfn) + struct nouveau_dmem_dma_info *dma_info, u64 *pfn) { struct device *dev =3D drm->dev->dev; struct page *dpage, *spage; unsigned long paddr; + bool is_large =3D false; =20 spage =3D migrate_pfn_to_page(src); if (!(src & MIGRATE_PFN_MIGRATE)) goto out; =20 - dpage =3D nouveau_dmem_page_alloc_locked(drm); + is_large =3D src & MIGRATE_PFN_COMPOUND; + dpage =3D nouveau_dmem_page_alloc_locked(drm, is_large); if (!dpage) goto out; =20 paddr =3D nouveau_dmem_page_addr(dpage); if (spage) { - *dma_addr =3D dma_map_page(dev, spage, 0, page_size(spage), + dma_info->dma_addr =3D dma_map_page(dev, spage, 0, page_size(spage), DMA_BIDIRECTIONAL); - if (dma_mapping_error(dev, *dma_addr)) + dma_info->size =3D page_size(spage); + if (dma_mapping_error(dev, dma_info->dma_addr)) goto out_free_page; - if (drm->dmem->migrate.copy_func(drm, 1, - NOUVEAU_APER_VRAM, paddr, NOUVEAU_APER_HOST, *dma_addr)) + if (drm->dmem->migrate.copy_func(drm, folio_nr_pages(page_folio(spage)), + NOUVEAU_APER_VRAM, paddr, NOUVEAU_APER_HOST, + dma_info->dma_addr)) goto out_dma_unmap; } else { - *dma_addr =3D DMA_MAPPING_ERROR; + dma_info->dma_addr =3D DMA_MAPPING_ERROR; if (drm->dmem->migrate.clear_func(drm, page_size(dpage), NOUVEAU_APER_VRAM, paddr)) goto out_free_page; @@ -645,7 +742,7 @@ static unsigned long nouveau_dmem_migrate_copy_one(stru= ct nouveau_drm *drm, return migrate_pfn(page_to_pfn(dpage)); =20 out_dma_unmap: - dma_unmap_page(dev, *dma_addr, PAGE_SIZE, DMA_BIDIRECTIONAL); + dma_unmap_page(dev, dma_info->dma_addr, PAGE_SIZE, DMA_BIDIRECTIONAL); out_free_page: nouveau_dmem_page_free_locked(drm, dpage); out: @@ -655,27 +752,33 @@ static unsigned long nouveau_dmem_migrate_copy_one(st= ruct nouveau_drm *drm, =20 static void nouveau_dmem_migrate_chunk(struct nouveau_drm *drm, struct nouveau_svmm *svmm, struct migrate_vma *args, - dma_addr_t *dma_addrs, u64 *pfns) + struct nouveau_dmem_dma_info *dma_info, u64 *pfns) { struct nouveau_fence *fence; unsigned long addr =3D args->start, nr_dma =3D 0, i; + unsigned long order =3D 0; + + for (i =3D 0; addr < args->end; ) { + struct folio *folio; =20 - for (i =3D 0; addr < args->end; i++) { + folio =3D page_folio(migrate_pfn_to_page(args->dst[i])); + order =3D folio_order(folio); args->dst[i] =3D nouveau_dmem_migrate_copy_one(drm, svmm, - args->src[i], dma_addrs + nr_dma, pfns + i); - if (!dma_mapping_error(drm->dev->dev, dma_addrs[nr_dma])) + args->src[i], dma_info + nr_dma, pfns + i); + if (!dma_mapping_error(drm->dev->dev, dma_info[nr_dma].dma_addr)) nr_dma++; - addr +=3D PAGE_SIZE; + i +=3D 1 << order; + addr +=3D (1 << order) * PAGE_SIZE; } =20 nouveau_fence_new(&fence, drm->dmem->migrate.chan); migrate_vma_pages(args); nouveau_dmem_fence_done(&fence); - nouveau_pfns_map(svmm, args->vma->vm_mm, args->start, pfns, i); + nouveau_pfns_map(svmm, args->vma->vm_mm, args->start, pfns, i, order); =20 while (nr_dma--) { - dma_unmap_page(drm->dev->dev, dma_addrs[nr_dma], PAGE_SIZE, - DMA_BIDIRECTIONAL); + dma_unmap_page(drm->dev->dev, dma_info[nr_dma].dma_addr, + dma_info[nr_dma].size, DMA_BIDIRECTIONAL); } migrate_vma_finalize(args); } @@ -689,20 +792,24 @@ nouveau_dmem_migrate_vma(struct nouveau_drm *drm, { unsigned long npages =3D (end - start) >> PAGE_SHIFT; unsigned long max =3D min(SG_MAX_SINGLE_ALLOC, npages); - dma_addr_t *dma_addrs; struct migrate_vma args =3D { .vma =3D vma, .start =3D start, .pgmap_owner =3D drm->dev, - .flags =3D MIGRATE_VMA_SELECT_SYSTEM, + .flags =3D MIGRATE_VMA_SELECT_SYSTEM + | MIGRATE_VMA_SELECT_COMPOUND, }; unsigned long i; u64 *pfns; int ret =3D -ENOMEM; + struct nouveau_dmem_dma_info *dma_info; =20 if (drm->dmem =3D=3D NULL) return -ENODEV; =20 + if (IS_ENABLED(CONFIG_TRANSPARENT_HUGEPAGE)) + max =3D max(HPAGE_PMD_NR, max); + args.src =3D kcalloc(max, sizeof(*args.src), GFP_KERNEL); if (!args.src) goto out; @@ -710,8 +817,8 @@ nouveau_dmem_migrate_vma(struct nouveau_drm *drm, if (!args.dst) goto out_free_src; =20 - dma_addrs =3D kmalloc_array(max, sizeof(*dma_addrs), GFP_KERNEL); - if (!dma_addrs) + dma_info =3D kmalloc_array(max, sizeof(*dma_info), GFP_KERNEL); + if (!dma_info) goto out_free_dst; =20 pfns =3D nouveau_pfns_alloc(max); @@ -729,7 +836,7 @@ nouveau_dmem_migrate_vma(struct nouveau_drm *drm, goto out_free_pfns; =20 if (args.cpages) - nouveau_dmem_migrate_chunk(drm, svmm, &args, dma_addrs, + nouveau_dmem_migrate_chunk(drm, svmm, &args, dma_info, pfns); args.start =3D args.end; } @@ -738,7 +845,7 @@ nouveau_dmem_migrate_vma(struct nouveau_drm *drm, out_free_pfns: nouveau_pfns_free(pfns); out_free_dma: - kfree(dma_addrs); + kfree(dma_info); out_free_dst: kfree(args.dst); out_free_src: diff --git a/drivers/gpu/drm/nouveau/nouveau_svm.c b/drivers/gpu/drm/nouvea= u/nouveau_svm.c index 6fa387da0637..b8a3378154d5 100644 --- a/drivers/gpu/drm/nouveau/nouveau_svm.c +++ b/drivers/gpu/drm/nouveau/nouveau_svm.c @@ -921,12 +921,14 @@ nouveau_pfns_free(u64 *pfns) =20 void nouveau_pfns_map(struct nouveau_svmm *svmm, struct mm_struct *mm, - unsigned long addr, u64 *pfns, unsigned long npages) + unsigned long addr, u64 *pfns, unsigned long npages, + unsigned int page_shift) { struct nouveau_pfnmap_args *args =3D nouveau_pfns_to_args(pfns); =20 args->p.addr =3D addr; - args->p.size =3D npages << PAGE_SHIFT; + args->p.size =3D npages << page_shift; + args->p.page =3D page_shift; =20 mutex_lock(&svmm->mutex); =20 diff --git a/drivers/gpu/drm/nouveau/nouveau_svm.h b/drivers/gpu/drm/nouvea= u/nouveau_svm.h index e7d63d7f0c2d..3fd78662f17e 100644 --- a/drivers/gpu/drm/nouveau/nouveau_svm.h +++ b/drivers/gpu/drm/nouveau/nouveau_svm.h @@ -33,7 +33,8 @@ void nouveau_svmm_invalidate(struct nouveau_svmm *svmm, u= 64 start, u64 limit); u64 *nouveau_pfns_alloc(unsigned long npages); void nouveau_pfns_free(u64 *pfns); void nouveau_pfns_map(struct nouveau_svmm *svmm, struct mm_struct *mm, - unsigned long addr, u64 *pfns, unsigned long npages); + unsigned long addr, u64 *pfns, unsigned long npages, + unsigned int page_shift); #else /* IS_ENABLED(CONFIG_DRM_NOUVEAU_SVM) */ static inline void nouveau_svm_init(struct nouveau_drm *drm) {} static inline void nouveau_svm_fini(struct nouveau_drm *drm) {} --=20 2.50.1 From nobody Sat Oct 4 22:37:10 2025 Received: from NAM02-SN1-obe.outbound.protection.outlook.com (mail-sn1nam02on2089.outbound.protection.outlook.com [40.107.96.89]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A78E52E264F for ; Tue, 12 Aug 2025 02:41:24 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.96.89 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1754966486; cv=fail; b=g209SwLVTWFOS336LxwFEfyC5s/N4fVTyF274jH64gUS4Ga3o94V5YvT6GET5c6qznV33A2EfkcCjbsVE4FSDI5C2oUcN10kf4w3npGUE8dmUSKJ9WvR7zUDgl3nOUzwkVDM3M73ac/EYl8nHxZZlHZkjoSK29LoADddy/RQhKA= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1754966486; c=relaxed/simple; bh=XqjJuTV5smp/PPPK/lUulIB5P6VnIyUcdcaVaV8rqs4=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: Content-Type:MIME-Version; b=hjT5mdECemvCrfS352e+m0OY7ft79ghtNFpzRle8SK6Jkbu12G4Ux3JoA8ahaCbv3dW5WSG2+zvjoRTWtW2z4t7w5TsKMHhWKS0KZ8OldSsUsvEUvRmdZcPSVhLBp8IeSOUaVXv/QqhJwmDT1OeuqAieFEWkIOCY8fosW/fvLBE= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com; spf=fail smtp.mailfrom=nvidia.com; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b=Eitg4MIl; arc=fail smtp.client-ip=40.107.96.89 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="Eitg4MIl" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=FvVVPFxR0pYwQgIRUQMh46L6Lxu3mFeJIGKvGAgtp4BV0gj8wonCNiT8EmavdtQHeLPzChjg7UUt1FidLM3U+4PWUOva46eIgiEzupbxUrB53zr8G5MbYD1LUv7h5APIonhMPTL9AirgLEwBQa1yfHxYAMi5L6iTxHhipXNGY1Tctlf4aMNNrYBYgdTvAaBDzUW1ZkZpmNCZ5/gYa7mAxu29hhvPgAA6hfFEcvGkJuVdIp4nC9rwPafknAa0ozZ8vRCZlBCoba81rekRQSSW5x0936Pm9IGTNSLLI3vWsU1uwmuWXm3QYoI+GDzaD6zr30HsQTWyUX+GEPhjEaRo6w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=E2VJ5kH+tLdZwlATFAo1+hNpAmKX0dnctUob4DpO984=; b=NoeMukxo3cBL2JdaKlorCmfiKpwiZ2DFm5V6hZO1ehndu+LTHsaZyng3rWozot+OoaKCBjXLS8mvkoqLKZlVGp8KPZTYBPAFWczSM1HZ6nBjJ+exR0uw82rDW73+kcO2rKZwFXWkyJG6yelSJzwnc+sSjxvbhAeRolzMxRGG8wDvOYgOE+mZ+aXAMeaWMs59ZMPu59FC22lcDxHJi56ilGFkBmYwydfAARgLsXmZCXV5rXnprTihzIE1S6l8+hky5BKxtL58EoyTubm8GWPlJH+KxWjR7PQEkcMpBcRPtFlCOzGloxo9xawsnVv2wEH4EvYaFsalj29l3WvYHudPiw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=E2VJ5kH+tLdZwlATFAo1+hNpAmKX0dnctUob4DpO984=; b=Eitg4MIl4tiXvYc2SI4ekNABh65JB1U8z1RYePuFj2QS6SiU0RHYELw/7ZAFXzslWvHeMTaH0ufJrh9IgMm8iZ6gXdFpV5VLfxCVcDCg4Qn+5AeD5wiG1ZyY/3rW8oO3BaAOJhDnachAhjbWUhXCDaaAFvms5tao/D22Yv2E7WzORQSHfbyACjve65zUwlbtVkuCOob6zVc/BRJ9gUjk6WFAlfrDi+hKSWqB5cZp7ICdl9hHexCnk6PTlZ6IikaHaEmNrarImTK5wY5IBn3YB/7qoKKV4F7McaQ9f7LS+FFyZfDI9tRri43RgHZn/ofc48VCG84rm58st7AxeBOUlw== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from PH8PR12MB7277.namprd12.prod.outlook.com (2603:10b6:510:223::13) by MN0PR12MB6317.namprd12.prod.outlook.com (2603:10b6:208:3c2::12) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9009.13; Tue, 12 Aug 2025 02:41:21 +0000 Received: from PH8PR12MB7277.namprd12.prod.outlook.com ([fe80::3a4:70ea:ff05:1251]) by PH8PR12MB7277.namprd12.prod.outlook.com ([fe80::3a4:70ea:ff05:1251%5]) with mapi id 15.20.9009.018; Tue, 12 Aug 2025 02:41:21 +0000 From: Balbir Singh To: dri-devel@lists.freedesktop.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Balbir Singh , Andrew Morton , David Hildenbrand , Zi Yan , Joshua Hahn , Rakie Kim , Byungchul Park , Gregory Price , Ying Huang , Alistair Popple , Oscar Salvador , Lorenzo Stoakes , Baolin Wang , "Liam R. Howlett" , Nico Pache , Ryan Roberts , Dev Jain , Barry Song , Lyude Paul , Danilo Krummrich , David Airlie , Simona Vetter , Ralph Campbell , =?UTF-8?q?Mika=20Penttil=C3=A4?= , Matthew Brost , Francois Dugast Subject: [v3 11/11] selftests/mm/hmm-tests: new throughput tests including THP Date: Tue, 12 Aug 2025 12:40:36 +1000 Message-ID: <20250812024036.690064-12-balbirs@nvidia.com> X-Mailer: git-send-email 2.50.1 In-Reply-To: <20250812024036.690064-1-balbirs@nvidia.com> References: <20250812024036.690064-1-balbirs@nvidia.com> Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: SY5P300CA0024.AUSP300.PROD.OUTLOOK.COM (2603:10c6:10:1ff::14) To PH8PR12MB7277.namprd12.prod.outlook.com (2603:10b6:510:223::13) Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: PH8PR12MB7277:EE_|MN0PR12MB6317:EE_ X-MS-Office365-Filtering-Correlation-Id: 792e822d-75e2-4134-616d-08ddd949b893 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|376014|7416014|366016|1800799024; X-Microsoft-Antispam-Message-Info: =?utf-8?B?OHVjczFKTkFtbUNZUzVOUVJlZEZHUG9idDZSblNZU1RWNFlmTmF2T2FIaDJ4?= =?utf-8?B?ME5PeSsvSjVQTGUrWEJrNkJkMm4vOGsvNWFlQktTdlFYMXdob0ZkRXNiZnNX?= =?utf-8?B?bEpvT0dxZlFQUnQ1czdiZXFXUHlWSlRwdDg2NEI0eTZWbHZzWkd2cXVjY3dX?= =?utf-8?B?WjZjRnM1VXpIRjBOMGxMWXVUTmoyWWluRk91SkhVRXNqaVBVNnYrdTBuNlJh?= =?utf-8?B?b0Y4M0dTWkVYM1l3Rkhvc2IyVDE4cml0ak8yLzJIcThiaUc2N01zU0hwZnNE?= =?utf-8?B?eUtXQzdPaFNYTjZpa01QVzY2ZFlmdkJwNlh3S3lXYng2Sk1IVElNamlMTDVn?= =?utf-8?B?c0hETlk1aU56REUrOEY4N0dHMUQraEJHUnQwODg2MFY0ZjkzVytpSDhWU3FO?= =?utf-8?B?cjRXUWRpWVhhS3p1SVhOQkVuS3BTakZVaWZUWk9EZDQ0eEs1M1NsTXR6aHE3?= =?utf-8?B?Y2IzK250TnFuaDU2dFo5a0dhdk82cXNWcWYzdkZHdjdhbkJnbGMyemUwbmpK?= =?utf-8?B?Z3lzNGVQdHVEQk01Y2hUUzd0ZTFMZzBRamNwaDlydTdtdVFhQUVlbFlUcmdX?= =?utf-8?B?WUEyazRyUllhS3Q3elI1cDNkcmNjbTlSbXA5bCtBRW1tdnl5cXRrMWErNVRE?= =?utf-8?B?VHgrN1lIOXJxd1pJSElaRGF0Qyt4dXJrMU9XbWp2YmpyZm9DSlJFYWdLSS9L?= =?utf-8?B?NDJSTitjWUkya2hLMFdTRzEzdWllVGlZZ3FYeUUzeXdUa2RibFJCNGd0VDhR?= =?utf-8?B?ZTFFOW5OSmFieFBrbmRaV2pXTVpoR3hoNHhscUl4Y3ZUVy9hRklmN0xQRDVQ?= =?utf-8?B?TFhUZ04xWVRjUmc0VlZnN2RSNVpURElIdUxTTlhGVnA3Nm1jZUZrSmkvMlB2?= =?utf-8?B?dmhremZTa3VHemw5b1ZwemFtV2xKU0VUdWt6d09zSnRtRWFxcE5zai9rUGsr?= =?utf-8?B?SlZ1RXJ5T0d4WmI4TnIxU2xkM280TWdENVZueHFEb3JISGpGN3hTQmNSZm5l?= =?utf-8?B?Q1JiUDJMSkNMMDg0c1JmZVhzS1hKWkgwdE9DdDJZQnMwRW43d0tHVGQ3OGE5?= =?utf-8?B?TmRZYTNCa2pVSmRtOTBYeWEwSm9lVjdoTjR3ZjVNeHB6bE0zVjV4TDYzME1N?= =?utf-8?B?VkhESXJBZ1czUHYzRGZIaitNUTRzbVdTUVJIRXhNb3JWMm4xMW5nT3hqcWJJ?= =?utf-8?B?YUFiOWwxUURjdGNsZEhKZmpNdmhSUVBVRUVtUzRWUG1HNFJUeHJjZnlqYnI0?= =?utf-8?B?WlJlOG5MenVsRzl1S2xnMCtneEczcS9oaWUwQzJNSk4rWVFjRFcxaFA0a0xl?= =?utf-8?B?VldpeDRNdGlVWHNjN0RQUjZpYTdmRkZ5Mlptai9iMFBZdFZjZVd0QmsxelpB?= =?utf-8?B?NnN2MmhIaTI1Uk9VbUhHZDRSUHNJV29GeEFadkp6T2hMK2Rnd3ZHR3ZyN2Vk?= =?utf-8?B?ZEYyQ3IvNlBEY1ppUWxvQnN1NFRrNU1MMFFzZDR1Y0pZUldTYTVVbGlVZ09Q?= =?utf-8?B?VVJ6U3NKdTZmeDE2RDMvcnhkRTEwRVE1Y0dmYzNXWDh4ZlRYZmZJQUJTS0Jw?= =?utf-8?B?OU5PS21wR2gxdFFlOE5uWmVpbzQ0WW8vNUVVRzBUSUNPNUN5cG0vNHVwaEtZ?= =?utf-8?B?YzVPUHBnSzkxZFVUZ1ZQc3dxcTh1ajR6b0tuSnpJQnc0VkFBYStHajhkU0JM?= =?utf-8?B?dzRMeE01RCt3WldCTjJIWFVVNCtKL1lTYVV2ZFFFWW1mWFhtYjlJS0RDUVJT?= =?utf-8?B?U3JWT0JQMmxnemR6M3NBNHdCZFVJUEJNOEVyYjcvc25sN0FQb2o5NC9SM0w1?= =?utf-8?B?ejlBaVdBd1ZJL01qdVduZGlUbTg4RWtKQ3VIUDBjOUUxb0kwWDd5dHZNcm9X?= =?utf-8?B?NkVDS3ZLeDh1L3dBTUVpQnFOL284SFBlNWlEcWg5VXBwajZuLzVmSVA5SEEz?= =?utf-8?Q?x+cSTy1AtpE=3D?= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:PH8PR12MB7277.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(376014)(7416014)(366016)(1800799024);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?eTdMcll6L1l2cHBndlQzTGNjUDJST0tiVEVidmxlTjZaNFhYRUFWN3JOWlI0?= =?utf-8?B?SDFWT1hKOUxqYUhJVGEzbTRmTDgrZVdsdE9qdkVpUU1Oa2JFb0tmQ1duQjl4?= =?utf-8?B?aU85WXUxS0VVYVF1QzdpcFJ1TWQ4ck90LzcvM0xpdUpYNG1CRXlTeFRYSVhX?= =?utf-8?B?QzI3MUVTYVlVd1RTcmhWNHRzM0VxMjNTejNBL3JuMzZ4cVh4OE5CWXZzN0Jz?= =?utf-8?B?Y1R2QU5xQnBvSUJhNkszczg5YnN2ZTdvN2k3VnMvMkVtYkt4cjZNemIzU3Fh?= =?utf-8?B?MmdPM3U1Qm1wSzZZSm9HcWgzWEY5cEZobTRsTVpQSnR4VmNqZjkvbG5vOG0v?= =?utf-8?B?Z3RPNURvaDFFdUcyTmRCVHdLdnl3Z0hwZW9hN3pHNHdjcThhMzkxSytFRks4?= =?utf-8?B?K0RaeXBzaStGVjlTT2g4TjI5Z3g5WHl4dmFBeStoSzkrTjZnbnpWT1hQWENh?= =?utf-8?B?c1dSQ1N4dVROL0lEWWcvbWhkY3IzejRnemw2alYzakt2emc0ZWtvYlEwSUp5?= =?utf-8?B?TmZjQ0ptUWQ1UzNTWVJQR25xbFowQ2VIYThXV1BFQTVra0tHNy9JUkdPU3hQ?= =?utf-8?B?MlNYS082NC94Uk9qZUxnL2VwQmpmZXpTdWljMnpGZVZXdlRHTnRKRFZpV2hX?= =?utf-8?B?NldqcDBja3FsbDZHNGU1NE1xZVNPeGtKam43Sk0vOGo1WW4zc3pxd20wTnB3?= =?utf-8?B?MngreUFUWUwvTU00V01PbEZYb0dsUHN4U1lNT0xnOGJwRDhhblJWL2I1a3la?= =?utf-8?B?VHRrcXYwSkhFUll5TUFKSkcwUDV1MHJTYlI4Um1DL3MrZDg3K2NERWNTVmUr?= =?utf-8?B?eGxIcTc3RSs5b3U3U1VoRy81dzcxa21XZ0ZPSTZSMUtFZVM5THZRSXZOUWRk?= =?utf-8?B?aEw3d2Y5SG41amZwUGFDZ0hTU0EvVjR2NWdNYlNBa1U0ck1ZRWVHRW5PSGd3?= =?utf-8?B?VHJWVnlIc0w5WWdDcC9GNFJJWWF4b2ltOGttRXBvK0twVm1yMXAwb3FzcDdl?= =?utf-8?B?Z3AyR2JwSVRSd0hNeEN0em8va0FsbUJSWmRDdUhDSmR2TERjRWVEdlRyMHdF?= =?utf-8?B?QlIzbXlyMVJXUUdqOFo5YlZBVHdVUXdaUTRXbTZIY2dBTkRTU09Zbm5MNWpl?= =?utf-8?B?UG5zRlFnV1hsTUtmUlVXWGl2ZGNYa2twWEpBV3BpSWxjVzZYSUV1cFg0UXFN?= =?utf-8?B?Q3R6aStGNy9VdWJDMVN4N2F4V1lwbFhvZVZ1MzJQalF0RlQwTWdkcWF2QlFT?= =?utf-8?B?d0xYVS9pNVdQaWZ5L2ZOd0U0dXdabkZNbi9xTWtBbDdpNzExWUpMbWlEN1dz?= =?utf-8?B?eFl1MVQ0clJwRm1aMis4QUdPeE1hWFZQdTdnNzJ3ZXB0NmhBNzZybVRYTlAz?= =?utf-8?B?T0ZBNnNZbURscW8wMk5RN2ttdG1xczZhQ3RFOVJhSlhJajR5dU8ybkRBRXla?= =?utf-8?B?QWdjNWw1b3lmZmMzL0ZDMkswZGFDZWE3ZkhaQ1hlWnYybEZXREdLU2pNcUtN?= =?utf-8?B?KytPRDdZanFFTytDTkkyT3QrdmFzdFpOOEhLaVdFV1IwNGRhNnJvN0o4TkYv?= =?utf-8?B?dWdSdTdsb2M5OGtkMGZQQldoZlZSV2pOOXR3dDlqUTRUdlltdWwrVDBLb0NF?= =?utf-8?B?SEFjcUNSRGR2UC9IYjZmTnk0R2Y3a2NPQ3E3akZ5MGw5alk1M1JuVm1oYXhB?= =?utf-8?B?elUrMFcyK3drdklSMkpaZkwyOFk4WkVuTGhXK0ZjbkluNWg0OGNmY2JHM0Rs?= =?utf-8?B?YW9MVjlqWjdERkFtZE1vQjN0NVJYVVRTQWxDT3ZPOU5hNm5FbFNpTUtDRnd0?= =?utf-8?B?NnRJdVV6dUorNnorWEk3V3c1WHUwVzAxNml1aXgvT3BWaHYyU0lmelQyWFVG?= =?utf-8?B?MDNuUTJWOVJudUw2elVSajVCdnpDRGZ1b0tZRmpkQzRKRE95SERSd3VhM0NC?= =?utf-8?B?S0tKNHlJYXY3QTJOK2wzTEdLRVhoZGVMRGIwYldkcEx2eGVLTGtVM2UvemI2?= =?utf-8?B?QWgwU09TUkJad3YvRlFiMTZ2cy9rYVlsem1OWmt3MStFbzJjN2IyVWM5RjQ1?= =?utf-8?B?blBPdFNhZmVxOUQwOHZqVTc5NzdyUTZkd2V4Si9Nb1NrclIxSTh5R1ZBNUpI?= =?utf-8?Q?KyAYgdeJ055zO0HA5Z97e4o0L?= X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: 792e822d-75e2-4134-616d-08ddd949b893 X-MS-Exchange-CrossTenant-AuthSource: PH8PR12MB7277.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 12 Aug 2025 02:41:21.2144 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: MY57gPrWH4Gx+gD/bEZ5RNCrhxunz8VYU+sqlyfJTvtIsusvhIYMPGcMqHqgiAd54xTV6CMhzASTTHmhMvT7nw== X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN0PR12MB6317 Add new benchmark style support to test transfer bandwidth for zone device memory operations. Cc: Andrew Morton Cc: David Hildenbrand Cc: Zi Yan Cc: Joshua Hahn Cc: Rakie Kim Cc: Byungchul Park Cc: Gregory Price Cc: Ying Huang Cc: Alistair Popple Cc: Oscar Salvador Cc: Lorenzo Stoakes Cc: Baolin Wang Cc: "Liam R. Howlett" Cc: Nico Pache Cc: Ryan Roberts Cc: Dev Jain Cc: Barry Song Cc: Lyude Paul Cc: Danilo Krummrich Cc: David Airlie Cc: Simona Vetter Cc: Ralph Campbell Cc: Mika Penttil=C3=A4 Cc: Matthew Brost Cc: Francois Dugast Signed-off-by: Balbir Singh --- tools/testing/selftests/mm/hmm-tests.c | 197 ++++++++++++++++++++++++- 1 file changed, 196 insertions(+), 1 deletion(-) diff --git a/tools/testing/selftests/mm/hmm-tests.c b/tools/testing/selftes= ts/mm/hmm-tests.c index da3322a1282c..1325de70f44f 100644 --- a/tools/testing/selftests/mm/hmm-tests.c +++ b/tools/testing/selftests/mm/hmm-tests.c @@ -25,6 +25,7 @@ #include #include #include +#include =20 =20 /* @@ -207,8 +208,10 @@ static void hmm_buffer_free(struct hmm_buffer *buffer) if (buffer =3D=3D NULL) return; =20 - if (buffer->ptr) + if (buffer->ptr) { munmap(buffer->ptr, buffer->size); + buffer->ptr =3D NULL; + } free(buffer->mirror); free(buffer); } @@ -2466,4 +2469,196 @@ TEST_F(hmm, migrate_anon_huge_zero_err) buffer->ptr =3D old_ptr; hmm_buffer_free(buffer); } + +struct benchmark_results { + double sys_to_dev_time; + double dev_to_sys_time; + double throughput_s2d; + double throughput_d2s; +}; + +static double get_time_ms(void) +{ + struct timeval tv; + + gettimeofday(&tv, NULL); + return (tv.tv_sec * 1000.0) + (tv.tv_usec / 1000.0); +} + +static inline struct hmm_buffer *hmm_buffer_alloc(unsigned long size) +{ + struct hmm_buffer *buffer; + + buffer =3D malloc(sizeof(*buffer)); + + buffer->fd =3D -1; + buffer->size =3D size; + buffer->mirror =3D malloc(size); + memset(buffer->mirror, 0xFF, size); + return buffer; +} + +static void print_benchmark_results(const char *test_name, size_t buffer_s= ize, + struct benchmark_results *thp, + struct benchmark_results *regular) +{ + double s2d_improvement =3D ((regular->sys_to_dev_time - thp->sys_to_dev_t= ime) / + regular->sys_to_dev_time) * 100.0; + double d2s_improvement =3D ((regular->dev_to_sys_time - thp->dev_to_sys_t= ime) / + regular->dev_to_sys_time) * 100.0; + double throughput_s2d_improvement =3D ((thp->throughput_s2d - regular->th= roughput_s2d) / + regular->throughput_s2d) * 100.0; + double throughput_d2s_improvement =3D ((thp->throughput_d2s - regular->th= roughput_d2s) / + regular->throughput_d2s) * 100.0; + + printf("\n=3D=3D=3D %s (%.1f MB) =3D=3D=3D\n", test_name, buffer_size / (= 1024.0 * 1024.0)); + printf(" | With THP | Without THP | Improv= ement\n"); + printf("-----------------------------------------------------------------= ----\n"); + printf("Sys->Dev Migration | %.3f ms | %.3f ms | %.1f%%\n= ", + thp->sys_to_dev_time, regular->sys_to_dev_time, s2d_improvement); + printf("Dev->Sys Migration | %.3f ms | %.3f ms | %.1f%%\n= ", + thp->dev_to_sys_time, regular->dev_to_sys_time, d2s_improvement); + printf("S->D Throughput | %.2f GB/s | %.2f GB/s | %.1f%%\n= ", + thp->throughput_s2d, regular->throughput_s2d, throughput_s2d_impro= vement); + printf("D->S Throughput | %.2f GB/s | %.2f GB/s | %.1f%%\n= ", + thp->throughput_d2s, regular->throughput_d2s, throughput_d2s_impro= vement); +} + +/* + * Run a single migration benchmark + * fd: file descriptor for hmm device + * use_thp: whether to use THP + * buffer_size: size of buffer to allocate + * iterations: number of iterations + * results: where to store results + */ +static inline int run_migration_benchmark(int fd, int use_thp, size_t buff= er_size, + int iterations, struct benchmark_results *results) +{ + struct hmm_buffer *buffer; + unsigned long npages =3D buffer_size / sysconf(_SC_PAGESIZE); + double start, end; + double s2d_total =3D 0, d2s_total =3D 0; + int ret, i; + int *ptr; + + buffer =3D hmm_buffer_alloc(buffer_size); + + /* Map memory */ + buffer->ptr =3D mmap(NULL, buffer_size, PROT_READ | PROT_WRITE, + MAP_PRIVATE | MAP_ANONYMOUS, -1, 0); + + if (!buffer->ptr) + return -1; + + /* Apply THP hint if requested */ + if (use_thp) + ret =3D madvise(buffer->ptr, buffer_size, MADV_HUGEPAGE); + else + ret =3D madvise(buffer->ptr, buffer_size, MADV_NOHUGEPAGE); + + if (ret) + return ret; + + /* Initialize memory to make sure pages are allocated */ + ptr =3D (int *)buffer->ptr; + for (i =3D 0; i < buffer_size / sizeof(int); i++) + ptr[i] =3D i & 0xFF; + + /* Warmup iteration */ + ret =3D hmm_migrate_sys_to_dev(fd, buffer, npages); + if (ret) + return ret; + + ret =3D hmm_migrate_dev_to_sys(fd, buffer, npages); + if (ret) + return ret; + + /* Benchmark iterations */ + for (i =3D 0; i < iterations; i++) { + /* System to device migration */ + start =3D get_time_ms(); + + ret =3D hmm_migrate_sys_to_dev(fd, buffer, npages); + if (ret) + return ret; + + end =3D get_time_ms(); + s2d_total +=3D (end - start); + + /* Device to system migration */ + start =3D get_time_ms(); + + ret =3D hmm_migrate_dev_to_sys(fd, buffer, npages); + if (ret) + return ret; + + end =3D get_time_ms(); + d2s_total +=3D (end - start); + } + + /* Calculate average times and throughput */ + results->sys_to_dev_time =3D s2d_total / iterations; + results->dev_to_sys_time =3D d2s_total / iterations; + results->throughput_s2d =3D (buffer_size / (1024.0 * 1024.0 * 1024.0)) / + (results->sys_to_dev_time / 1000.0); + results->throughput_d2s =3D (buffer_size / (1024.0 * 1024.0 * 1024.0)) / + (results->dev_to_sys_time / 1000.0); + + /* Cleanup */ + hmm_buffer_free(buffer); + return 0; +} + +/* + * Benchmark THP migration with different buffer sizes + */ +TEST_F_TIMEOUT(hmm, benchmark_thp_migration, 120) +{ + struct benchmark_results thp_results, regular_results; + size_t thp_size =3D 2 * 1024 * 1024; /* 2MB - typical THP size */ + int iterations =3D 5; + + printf("\nHMM THP Migration Benchmark\n"); + printf("---------------------------\n"); + printf("System page size: %ld bytes\n", sysconf(_SC_PAGESIZE)); + + /* Test different buffer sizes */ + size_t test_sizes[] =3D { + thp_size / 4, /* 512KB - smaller than THP */ + thp_size / 2, /* 1MB - half THP */ + thp_size, /* 2MB - single THP */ + thp_size * 2, /* 4MB - two THPs */ + thp_size * 4, /* 8MB - four THPs */ + thp_size * 8, /* 16MB - eight THPs */ + thp_size * 128, /* 256MB - one twenty eight THPs */ + }; + + static const char *const test_names[] =3D { + "Small Buffer (512KB)", + "Half THP Size (1MB)", + "Single THP Size (2MB)", + "Two THP Size (4MB)", + "Four THP Size (8MB)", + "Eight THP Size (16MB)", + "One twenty eight THP Size (256MB)" + }; + + int num_tests =3D ARRAY_SIZE(test_sizes); + + /* Run all tests */ + for (int i =3D 0; i < num_tests; i++) { + /* Test with THP */ + ASSERT_EQ(run_migration_benchmark(self->fd, 1, test_sizes[i], + iterations, &thp_results), 0); + + /* Test without THP */ + ASSERT_EQ(run_migration_benchmark(self->fd, 0, test_sizes[i], + iterations, ®ular_results), 0); + + /* Print results */ + print_benchmark_results(test_names[i], test_sizes[i], + &thp_results, ®ular_results); + } +} TEST_HARNESS_MAIN --=20 2.50.1