From nobody Fri Dec 19 12:49:37 2025 Received: from mx0b-00069f02.pphosted.com (mx0b-00069f02.pphosted.com [205.220.177.32]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 970D2748D for ; Mon, 14 Apr 2025 03:46:53 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=205.220.177.32 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1744602415; cv=fail; b=cYfYFfyAfbMpT9ZzIvya9dvTKT+cbVVFNwAG5b8vX2TuYMVkzzwuwRcS99bdl9hOEv83HoQ0griICXA4UWNrQLOA8dcg4yJWVjJtxvzEFKe/NhNhrJAydxcY+mgq3BWZiv07Zz1zPTIEqXAHWBbD6D0d251v3pJ0fcl6JqprqdA= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1744602415; c=relaxed/simple; bh=AHJj92DKlvDLQn22ngtCnUDW/Cj1kJoxeMZhs3wnVHU=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: Content-Type:MIME-Version; b=V5JhDwusoFW5ltanRrMz+wI1dgNof8yxXPAfpvoMjFBOjp4iga9XS0QsRM7dQ5VilzJ0hjuRuTNOy4OvnWZFzGohpn2O0dtsE4yYKjgwuYBOl7DrUeuJVyPjj/cmjPP8MgTABIhIcV1ATKK1cn5P6YSo+kwF2r38DivBmJsDHyQ= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=oracle.com; spf=pass smtp.mailfrom=oracle.com; dkim=pass (2048-bit key) header.d=oracle.com header.i=@oracle.com header.b=dbtWDngw; dkim=pass (1024-bit key) header.d=oracle.onmicrosoft.com header.i=@oracle.onmicrosoft.com header.b=LO+ai67A; arc=fail smtp.client-ip=205.220.177.32 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=oracle.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=oracle.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=oracle.com header.i=@oracle.com header.b="dbtWDngw"; dkim=pass (1024-bit key) header.d=oracle.onmicrosoft.com header.i=@oracle.onmicrosoft.com header.b="LO+ai67A" Received: from pps.filterd (m0333520.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.18.1.2/8.18.1.2) with ESMTP id 53E3MKj4000707; Mon, 14 Apr 2025 03:46:12 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=cc :content-transfer-encoding:content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to; s= corp-2023-11-20; bh=IvLmEHZA0zPv9FTSfiq/rKt9GX75OUBIareZZvifBFE=; b= dbtWDngwTs+RsuXUaN9vILsT55WI9i8RdCISt8IqVmkpUsO6pXrYGiOF3QyUY/ti 45nzEfcaIGt96fu+aahnOsU6S0SRybrcG10ZaUcoNJeVsLAWOWyagbPw9FOR3jmN RFRBEojRlS3zYckyfk1rpQKDm6iQiJ7pKod7hgi5BHwxl886FCC3HyUNnaYoXr42 tn3SOjAUV8f8xZzNIi0HGqne7q7F+KavtXa2cYIzPwyEACZ52OiUuGKO5BO2PlNe dnmzkz5+NTiP5IxUuTR4YAJp23cG8mOASKHdUZ9HQ7XhEEz6MA55XMcrtpeWqxcB cqtgQ1fvPGEnTJ2kkTJa3g== Received: from phxpaimrmta03.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta03.appoci.oracle.com [138.1.37.129]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 460td100ka-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 14 Apr 2025 03:46:11 +0000 (GMT) Received: from pps.filterd (phxpaimrmta03.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by phxpaimrmta03.imrmtpd1.prodappphxaev1.oraclevcn.com (8.18.1.2/8.18.1.2) with ESMTP id 53E3UUVV024638; Mon, 14 Apr 2025 03:46:11 GMT Received: from byapr05cu005.outbound.protection.outlook.com (mail-westusazlp17010003.outbound.protection.outlook.com [40.93.1.3]) by phxpaimrmta03.imrmtpd1.prodappphxaev1.oraclevcn.com (PPS) with ESMTPS id 460d4x6e8h-2 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 14 Apr 2025 03:46:10 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=MWaa8ujdoblk6M+e5EKgIuqr9daqTVPeyH9xJFgV6btYWMsy8XG9tCyQmAmIA9jfNaPO/XMQyJU52wurOzjeSYT7O9Y7VQkRpbAW/j9ipbVsekP4rR9kxk/PFNA2J5ZiRa98Xa+BRF/a6RZKgnDVq2rj0Cw4Z0HiO46aRc2NLHKx446MTSXRZlohf0TgfoCAlzUFIyJ/QLF6nmPYatLn3Te09+0gdsBzaZM5WGDpxQe4LIW4X28Fb5AmZlpD44pnCWtxii4CNVLySh/jMkApEI9Ax+I3iJ+VDg6Zx1j4NNoupN4D2g6dDsb3TD7y3GWSi1p6WqMI7SSIuMOZDit/MQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=IvLmEHZA0zPv9FTSfiq/rKt9GX75OUBIareZZvifBFE=; b=HGnaD2eXzsqZH/5Cm3lB4P9DqplnnmkUbtWRlTpLxttkSiLSRbM2Fqf65VAPALSKn8QPTUAupcEnyjpl1xBawqdFUJrrtSNZ44Y42iw4wwmVU6XeDLxWNbHditRW/LrWVZqyT6nXhVgqRjsHqCKumb1LBMznFyvQiT58MFISR991WejXLEyVTuusynHrHO2DiWrgu7WRB2bbPEbCCFuQQ5xRRX7kJTUxP6r/iRcpiBKXEg6nvJxqvMsXPxZviagaHt0oGF2PQ8HSzC1sE4GpgxKpFETQMMgSioWDCF7X0CjhZ5vi/dw1yw9/XfiaNffIOysb05Hls9OVfcp00p8OJw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=IvLmEHZA0zPv9FTSfiq/rKt9GX75OUBIareZZvifBFE=; b=LO+ai67Au/iZi62Ae9P0yreDOdT2NQPy7g1sLX+kEPDgUIua9B/SvDm4CIWFhsEDfZmjmb9/Z4hQnlc3ss5B3i1h8WAcWcoeUtq8xKvkLVC90SUKHSEcX9B+yA7mP3V7Co1xT4ebx4Zg5jXNQ40xUBMMMymHdtme+M8Gz2LCtBk= Received: from CO6PR10MB5409.namprd10.prod.outlook.com (2603:10b6:5:357::14) by DS4PPFDB583D803.namprd10.prod.outlook.com (2603:10b6:f:fc00::d50) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8632.33; Mon, 14 Apr 2025 03:46:09 +0000 Received: from CO6PR10MB5409.namprd10.prod.outlook.com ([fe80::25a9:32c2:a7b0:de9e]) by CO6PR10MB5409.namprd10.prod.outlook.com ([fe80::25a9:32c2:a7b0:de9e%5]) with mapi id 15.20.8632.030; Mon, 14 Apr 2025 03:46:09 +0000 From: Ankur Arora To: linux-kernel@vger.kernel.org, linux-mm@kvack.org, x86@kernel.org Cc: torvalds@linux-foundation.org, akpm@linux-foundation.org, bp@alien8.de, dave.hansen@linux.intel.com, hpa@zytor.com, mingo@redhat.com, luto@kernel.org, peterz@infradead.org, paulmck@kernel.org, rostedt@goodmis.org, tglx@linutronix.de, willy@infradead.org, jon.grimm@amd.com, bharata@amd.com, raghavendra.kt@amd.com, boris.ostrovsky@oracle.com, konrad.wilk@oracle.com, ankur.a.arora@oracle.com Subject: [PATCH v3 1/4] x86/clear_page: extend clear_page*() for multi-page clearing Date: Sun, 13 Apr 2025 20:46:04 -0700 Message-Id: <20250414034607.762653-2-ankur.a.arora@oracle.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20250414034607.762653-1-ankur.a.arora@oracle.com> References: <20250414034607.762653-1-ankur.a.arora@oracle.com> Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: MW4PR04CA0340.namprd04.prod.outlook.com (2603:10b6:303:8a::15) To CO6PR10MB5409.namprd10.prod.outlook.com (2603:10b6:5:357::14) Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CO6PR10MB5409:EE_|DS4PPFDB583D803:EE_ X-MS-Office365-Filtering-Correlation-Id: 28f644c2-a573-4d97-876f-08dd7b06e4bf X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|1800799024|7416014|376014|366016; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?dYGOXcbeHkDqStIQjEDZ0jf7mSYRvBSWtSqYUFE0GmY/A6lt5f6EKMlniizO?= =?us-ascii?Q?tc4lmuy4Tj+ai72Sl7AwQZepiw3fuG2nl2A0DQH+uqx4xRLacPt49ThU+O8R?= =?us-ascii?Q?746MyiZYcfGTGEBTEemqBl5MykU9yxgTOxLIBAEPe+yxb2vgO5Whd6dfTvZ5?= =?us-ascii?Q?Voc0k0ousF/KynkHbAJ5vS7uwJtc/4bZwSeuIjdIkY6614m0nXyaI+DvCx3O?= =?us-ascii?Q?UYUUhFC96e58ieQP46bEkvn29sFFhZCZEUMRtIEez/a/TXZv8H0O5pV+fsw+?= =?us-ascii?Q?5W2gjUiP04Yv1ikUH025RApx5yXJLOkmPXJPQXeA82v55UgK2JAiA01lgnZT?= =?us-ascii?Q?ga9k/VvOLiLPv2OhUgQD4wU1j9o6PoU3aeT9GRcY0KI1MXNK0KjQmtAlaYMl?= =?us-ascii?Q?LmhseSJwiIdlKECBFOkXKw+KVbVAljSZoAY5JDvVc8VGjw4ApbVUFys9V0NT?= =?us-ascii?Q?Bk/7BMgIj5Z+DUMiaDcTK4/xjfvsXvAGlLQ5Tgk8WDrtrmj+2Kura3owqAtu?= =?us-ascii?Q?PJnFT7i5Zx3ncZuyxP7fSyJ+OlYGYi7eEybsP+ENZl9zovaqP4Q+HZVH+kC9?= =?us-ascii?Q?h4Pa7YKRRcunyO3uzz471K6Ma2NE7Ip4pf6BYYha7T5/HIiIKXtUchTp6Zms?= =?us-ascii?Q?B75VDq6NjoZlKGn9WbZapvMHMV62B+V0whWft4kkcmkC+kfLqr7MoK9Al0eh?= =?us-ascii?Q?yiRWNZrqrxdePi5/67x7uSN7KU60cqCAy2UDwhueIZSLkkL397Gf/GcG7Uir?= =?us-ascii?Q?Wt/y7ijcyCTSF7hVTHcLGKS2etDSsZDTMIqRVzmq4D1avKQJzDoVFDKbKnYW?= =?us-ascii?Q?BWE2119mxNR2sUrin0lMJy9HmP6RVTC9EHB5RdF4p1F7uPObEtDx1Y6JVtxQ?= =?us-ascii?Q?A4CENARfJtBVlQgOTZv6Mh0i+P3R+lmCnioTktFfV9zfpisyHbqryLRGXIzI?= =?us-ascii?Q?54aOefjbjob91yLNhcyq8KvWK1eXXfTIZ8KFhcOgwwzN0aCtc70K6VLLjbap?= =?us-ascii?Q?rsJMJI8tYw1yWQBjEqMKgY4b40FtvgMf2G2nktgYsKjaRDFvw84TKdlF4kFb?= =?us-ascii?Q?2nWOzQ+/WqdWbwv6eCAn/aauzaXyBNc70sTPh2TGdbC/EQiSCK30V7llpjRE?= =?us-ascii?Q?0ZJnLyAm64zq91n6udPSHhlpco4A9uKekTKQfHoam6CV2Jz58Nyj1tn2BP2/?= =?us-ascii?Q?udSGLS4eqIwBsqq2hK/4lZvi+hdiTygqa34ePFry8WgCQBopuy4011HDnyvH?= =?us-ascii?Q?oAQ4nHsU0Rw39mPQxuYyCTRVzo6LwH6+5TZGcCsXpATkZcnORiJrZIFwGzpw?= =?us-ascii?Q?PV4Gu0cnnlo1wlSterVKS5J7au08Pp2Gc9PoaB3vzO54ghQeuOw2x4c6Bf08?= =?us-ascii?Q?qWYtOG37a6lZrq7ho3RtIRlLGt8WtI2RSKHw50Eu7X1SCLvr/zTXhH5417WI?= =?us-ascii?Q?dp7LtqPZc8w=3D?= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:CO6PR10MB5409.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(1800799024)(7416014)(376014)(366016);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?LDYjfWQJTjlfGAHYdVbJk1/gHQsC//UHNTyjERnG3d6aa7HWLHEXS7oEDbjL?= =?us-ascii?Q?ML/tyjLYmvyzg64htSG8LpaxCyD+gAmSxc5yOr9bg7q/zTY+/mLxnp7Eb7YE?= =?us-ascii?Q?4qrAhA30b7dy/BTc5nU+KXK4/Vj6QuumIo4ZShe5Np+33c8R+kuY2OSYoJo9?= =?us-ascii?Q?xE0/ivxQyGgTzLvKzOP6G5c9JKgn66zYyoOwQF3Vvh/Rq5xK949RFCPY2jmE?= =?us-ascii?Q?afdWMWODZIYGZmmhJZu15q2kGrTMxkao2t1eTnLl6n3uBccE8zjTrgowxOP2?= =?us-ascii?Q?Hi8S2H8kOJa1utu0dtJDypaRk2qlQD6YWxqBwbvy6m7bMVr1AAUbFTS2rh9m?= =?us-ascii?Q?4oMPKYuWffCa9B0soXj+WhbA5MeeuqWR3pFYcx2ttUqgZvW9jgv0TwGrAksI?= =?us-ascii?Q?Cz6Fshkbpj+Cdi+jiSAoT7m8VwzgTXcDtZnh9R1Fn0NJ0egOV68yjMu046xV?= =?us-ascii?Q?qgZkTa2TPbR1vipFvnrfGeYCojn4N2lKkbpOT6+awH5SRssNy6Uvgrqli/Vx?= =?us-ascii?Q?+mIGs4/H+nCU21s9fMq7EmXW3pW+Qe1bhqTMW46D5/3R3uN93GzMUtRMRIyO?= =?us-ascii?Q?mTbop8zlcW+ead9SgTzuFduHNrPxQIgDv1v4pgW7GKWnwYcYxJughJj5Stti?= =?us-ascii?Q?uLQd/8cJen5TgVzX2QQCrqqEbJ5DD6NHvFVyLVIL4bWVvaoe+tCLh4pnljJ8?= =?us-ascii?Q?C9e3W8XD6ygeXOlrMPgICWh55GVdxu31I0X3+noc/bGIXg1twglc6cZRrXke?= =?us-ascii?Q?wegXE7QmCs1Py6FmGShSPRT1oqL5ocLooTVLz/pVbuBMAF/kQc7bXw0462P4?= =?us-ascii?Q?CyD9GLxx3vR9oNOC2vgr5unophu5KUi9JjOwjyT52S5NdSoDOsuvryT+6Nmp?= =?us-ascii?Q?Ay0sw10RNuWbfi7owUNcQrlNsXc6NG8jcZkduTazQsTnnz7PJslxfc0MQ4CY?= =?us-ascii?Q?5N8N9uC2EGxNRgJXDWqdtam9EQi/GXDeIZcBMxxU+bwmHeA8NKsvFZLzIBQv?= =?us-ascii?Q?++sOssmtRzd8W8JuTrKDCNUKUlH3PRC0abTqGQ5Xq55vIuf0oCPT4MSxq/hL?= =?us-ascii?Q?egLuSd0AVzl6eob8xsJUamwyYaE7Mlio99cXuXUrzNPe6TWkTpbQvrSV1YXo?= =?us-ascii?Q?VLkvQeBMSrYgZIREuWPPdRlpGLfDpnfOD9U9mg5ijWReO324oSjSy/ELQ2d2?= =?us-ascii?Q?Hw9IpiiEXnnqJHcBDJfpMBm4tx/CSdvWthiV+ThnDVhqf8kC2YupQzDOIVza?= =?us-ascii?Q?XazEcJa7rhubnDWas1niaSOCN5jh6Zp21ToglPuAnDxa4fCxbubgVVj1vwk7?= =?us-ascii?Q?NFOsHOjv34zR5e8kflf/qSD/recRGWa1vXgW7uS5YZkVyvLwGhIh+h1KQZxc?= =?us-ascii?Q?5ybidCbPMjVPMOmOy1tdx9G8qAZP/bR5hNtVLVRhOHScbq+Lf3CZ3g3/m8cD?= =?us-ascii?Q?M9q34SpRL+1o4kBJsrBNMOOKxSgdGakm0DWpRYUMxjt7HXx7BfFQ8eNGC6ve?= =?us-ascii?Q?UbDQz0y/dz5VrHKsAsRRN6cLzwZtEEYv1WcEkQMCqRMK5m4Y8RUBDF2zfDjs?= =?us-ascii?Q?OjAZALFayi4kAFTdr7dp5EE4kYToOM+FklY3hlqD8wZkeuvHaPqMT9PBDa87?= =?us-ascii?Q?bg=3D=3D?= X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: Afan5+0HLc2jaVUM/eBz+uxTKESeV84DMkXLqmXTp6Qk7tImQC7fojNnUiehIS0YZXput7Xu9PGJEh/6XluBCwYTAE4+qUkvHpn2+BvKb+Li5g/pBLPDEqHhvwwNKbXDmk2H1d+/QwWFZ+zwHHlRnK5Pxx07TgG5vPZ/TVx6V6VWpBnikrxNnsTOxzejx+wVYl2wZA8IvWZFdqi8p1HDiQLo+MJ+nzcVLRlkXOcoiK/N4TD9B2FjVTMjEEp0pFcK3MvjIP6JzwIQnSOUEPYItvYyRK6GbnVRtkj3Z3R2SkOUKyY/7PeYmvhHBFto7wTXyQpW/f4o978BWuUVDXy6OYqwxRa2J0BCk+WXkQoD/bKSJV3m/7aIhKZC2bwukAdree+UmfSHe6i1wTgazI4EEaEtmuMI6jp1AtBBXscGMlQ+n61a98g+64iz02XBF11rDxSSrq0h8+1gJqzs/bXE0XfodLHnx6qApCod/ysmge4C6SuI9jmKIgZMqCB4vfrZt2uHuRBwXbaEamiUK/33KSgQ541vZAoK9j2cZtwXX9D0OPSx20L9EPlcMSUjSQAFF6UcAS8HvH7hOMAnIkczG9bWWLky7ZmTZdr6lNUsE0g= X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 28f644c2-a573-4d97-876f-08dd7b06e4bf X-MS-Exchange-CrossTenant-AuthSource: CO6PR10MB5409.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 14 Apr 2025 03:46:09.5459 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: 3NRgNwNkOqwC0V1/AJVVknSxFhHFjy3O2O6Yf5Bh1GEtT9J+izpX5dLsK+6AJRYrZIPds/6T5kUgVR29oIxQsk4l8ehUqboI87073ZIdaX0= X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS4PPFDB583D803 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1095,Hydra:6.0.680,FMLib:17.12.68.34 definitions=2025-04-14_01,2025-04-10_01,2024-11-22_01 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 adultscore=0 phishscore=0 mlxlogscore=999 bulkscore=0 suspectscore=0 mlxscore=0 malwarescore=0 spamscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2502280000 definitions=main-2504140026 X-Proofpoint-ORIG-GUID: hqmA70WF9qnVQQHmjhfAeWjqmhgfLc2D X-Proofpoint-GUID: hqmA70WF9qnVQQHmjhfAeWjqmhgfLc2D Content-Type: text/plain; charset="utf-8" clear_page*() variants now take a page-aligned length parameter and clears the whole region. Rename to clear_pages*(). Signed-off-by: Ankur Arora --- arch/x86/include/asm/page_64.h | 20 +++++++------ arch/x86/lib/clear_page_64.S | 52 +++++++++++++++++++++++++--------- 2 files changed, 49 insertions(+), 23 deletions(-) diff --git a/arch/x86/include/asm/page_64.h b/arch/x86/include/asm/page_64.h index d3aab6f4e59a..45db74db9021 100644 --- a/arch/x86/include/asm/page_64.h +++ b/arch/x86/include/asm/page_64.h @@ -40,22 +40,24 @@ extern unsigned long __phys_addr_symbol(unsigned long); =20 #define __phys_reloc_hide(x) (x) =20 -void clear_page_orig(void *page); -void clear_page_rep(void *page); -void clear_page_erms(void *page); +void clear_pages_orig(void *page, unsigned int length); +void clear_pages_rep(void *page, unsigned int length); +void clear_pages_erms(void *page, unsigned int length); =20 static inline void clear_page(void *page) { + unsigned int length =3D PAGE_SIZE; /* - * Clean up KMSAN metadata for the page being cleared. The assembly call + * Clean up KMSAN metadata for the pages being cleared. The assembly call * below clobbers @page, so we perform unpoisoning before it. */ - kmsan_unpoison_memory(page, PAGE_SIZE); - alternative_call_2(clear_page_orig, - clear_page_rep, X86_FEATURE_REP_GOOD, - clear_page_erms, X86_FEATURE_ERMS, + kmsan_unpoison_memory(page, length); + + alternative_call_2(clear_pages_orig, + clear_pages_rep, X86_FEATURE_REP_GOOD, + clear_pages_erms, X86_FEATURE_ERMS, "=3DD" (page), - "D" (page), + ASM_INPUT("D" (page), "S" (length)), "cc", "memory", "rax", "rcx"); } =20 diff --git a/arch/x86/lib/clear_page_64.S b/arch/x86/lib/clear_page_64.S index a508e4a8c66a..bce516263b69 100644 --- a/arch/x86/lib/clear_page_64.S +++ b/arch/x86/lib/clear_page_64.S @@ -13,20 +13,35 @@ */ =20 /* - * Zero a page. - * %rdi - page + * Zero kernel page aligned region. + * + * Input: + * %rdi - destination + * %esi - length + * + * Clobbers: %rax, %rcx */ -SYM_TYPED_FUNC_START(clear_page_rep) - movl $4096/8,%ecx +SYM_TYPED_FUNC_START(clear_pages_rep) + movl %esi, %ecx xorl %eax,%eax + shrl $3,%ecx rep stosq RET -SYM_FUNC_END(clear_page_rep) -EXPORT_SYMBOL_GPL(clear_page_rep) +SYM_FUNC_END(clear_pages_rep) +EXPORT_SYMBOL_GPL(clear_pages_rep) =20 -SYM_TYPED_FUNC_START(clear_page_orig) +/* + * Original page zeroing loop. + * Input: + * %rdi - destination + * %esi - length + * + * Clobbers: %rax, %rcx, %rflags + */ +SYM_TYPED_FUNC_START(clear_pages_orig) + movl %esi, %ecx xorl %eax,%eax - movl $4096/64,%ecx + shrl $6,%ecx .p2align 4 .Lloop: decl %ecx @@ -43,16 +58,25 @@ SYM_TYPED_FUNC_START(clear_page_orig) jnz .Lloop nop RET -SYM_FUNC_END(clear_page_orig) -EXPORT_SYMBOL_GPL(clear_page_orig) +SYM_FUNC_END(clear_pages_orig) +EXPORT_SYMBOL_GPL(clear_pages_orig) =20 -SYM_TYPED_FUNC_START(clear_page_erms) - movl $4096,%ecx +/* + * Zero kernel page aligned region. + * + * Input: + * %rdi - destination + * %esi - length + * + * Clobbers: %rax, %rcx + */ +SYM_TYPED_FUNC_START(clear_pages_erms) + movl %esi, %ecx xorl %eax,%eax rep stosb RET -SYM_FUNC_END(clear_page_erms) -EXPORT_SYMBOL_GPL(clear_page_erms) +SYM_FUNC_END(clear_pages_erms) +EXPORT_SYMBOL_GPL(clear_pages_erms) =20 /* * Default clear user-space. --=20 2.31.1 From nobody Fri Dec 19 12:49:37 2025 Received: from mx0a-00069f02.pphosted.com (mx0a-00069f02.pphosted.com [205.220.165.32]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id BB53E155C83 for ; Mon, 14 Apr 2025 04:28:09 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=205.220.165.32 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1744604891; cv=fail; b=PlE9LOjJ8BDbLVeM/Dkc2vFW8/R6U3TLUMn634TnK9I/p4QoD7mOag0Qr06w+T7AetEnfYfaQ3vJwHq/iIqo/2jaOqAg3q3Vo5mHHa6eq+anjmqFqDAFiOkMwy0RgAk4BtzTZSzCKoztluHZrK3Wg6rkoUQFUu/7706VxRkuwKY= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1744604891; c=relaxed/simple; bh=qHUtjVOHzi07/EK7iNROhIvIcinkCW6IxC1JGyapAEI=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: Content-Type:MIME-Version; b=u4AT5XHnKbIzwPXVUMOJ4iZtAtg8oxyZZ0JaLf+S+1CiqGzuT7dfRqe3vb2ceBNkQJwykXj6S0DnxD4WsV1uD6HJ/XZZ2DTeKKMVWQJx9xnlntz0Mr8uK6KnX79hsStOzm7HQo7Oh3bqP32RUvOX8oovot4+/D4MrUpmCvlYqMY= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=oracle.com; spf=pass smtp.mailfrom=oracle.com; dkim=pass (2048-bit key) header.d=oracle.com header.i=@oracle.com header.b=Y87NErTq; dkim=pass (1024-bit key) header.d=oracle.onmicrosoft.com header.i=@oracle.onmicrosoft.com header.b=Dv4KFfMf; arc=fail smtp.client-ip=205.220.165.32 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=oracle.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=oracle.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=oracle.com header.i=@oracle.com header.b="Y87NErTq"; dkim=pass (1024-bit key) header.d=oracle.onmicrosoft.com header.i=@oracle.onmicrosoft.com header.b="Dv4KFfMf" Received: from pps.filterd (m0246629.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.18.1.2/8.18.1.2) with ESMTP id 53E3buF0007908; Mon, 14 Apr 2025 03:46:14 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=cc :content-transfer-encoding:content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to; s= corp-2023-11-20; bh=jvpnIooywk121/A9ba5yAbNWiYQIegMHpNkfFCXsNmw=; b= Y87NErTq8wZ5b2DQKAxhMI0zqI6c6JyGPX3FoGZXfhtFmmJv80ycOD57LOaQOdjs 4tp1UYHDd3ydhaodNJcrNgfbkCdRDRKtU0tPBROwqD4EbM81ZHZ2ukHCS+EyVAFG dvilYI2MSg4SP/lsmY6eJBLNXVo+NrruSOpYKsTTuYKcQKWS7/I/mQUJin3arhVK b5weJTYlhIqVGcwvaxvuF7NTZgJkxmhZcdPsyvxte4rN4xFNBr36ZRSgQl/kAblG +CikJooDvnsCGQyGUL2AoV/6IDL7BBXhwifpK+QhB+5DhDRH1GnpgDAMsXfXD69f /h6drV8O/LhNjaooxpQ2Jw== Received: from phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta01.appoci.oracle.com [138.1.114.2]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 460tm28076-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 14 Apr 2025 03:46:13 +0000 (GMT) Received: from pps.filterd (phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (8.18.1.2/8.18.1.2) with ESMTP id 53E3SnlE009213; Mon, 14 Apr 2025 03:46:12 GMT Received: from byapr05cu005.outbound.protection.outlook.com (mail-westusazlp17010001.outbound.protection.outlook.com [40.93.1.1]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (PPS) with ESMTPS id 460d3fxhkp-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 14 Apr 2025 03:46:12 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=gdeWhyktJQH66khcJm+Q0NEGsFc00RX/AZ09rutvmSzXnQ7g82IjWCQAc2A2YfknoJtMJ9hBUiSy5NHjZTlq0d8tLsXsYgO42sfapZL3KDaaXkU/WSm2+5qXcnwd0gGjHQ5tLPy5+9jstT+Oq5a2p3wfxShGldxrxIwhZA/FRg/+5F/oIrYgJWYFFyxJouj7LyjsVVt34aWGOMygr2d6lhYBC/FBbb57XKpC1HnxRRxo147XdgJfbO19fQcVs35jXxiUVgNz8nUsmxfNgwaYoD9mcajxKp4JUue6fW4GO3caxaPAjWuTfNRIxgrt/TOs8zn9Uc7RX54Srxg5zzYvjw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=jvpnIooywk121/A9ba5yAbNWiYQIegMHpNkfFCXsNmw=; b=mRbNjlkhanSiKnqVEyJn0eOl5Emw8OdhGipngJVLGfDdbNUwiDjf3taJVu8gX/JzThM+fDPJmZ0/Gx7dNXfL/fZH1dbGsrL46MGJIzhS0iVLn1QhzCOxG8j5KBnDDeEFB52CZy8t/TxAcyAPOcAKylxm5lgaDFrKT9ChXEGmaVZV9SNx33RXPJtU6C/nNdLiaiANvThERDAs8tSBRM3nzzkA0WJOSCtg+YGlEzu9z/9lZQmj1X370ZP2QBw5IkIKmf6hb1+h0x0uhqJ2Jdbsmo+lCah8CgrCANJwTqWDDS+qvJDJ09800KZCJ6zPKVRSvDgTnh/uQ8Buq/m16elIEQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=jvpnIooywk121/A9ba5yAbNWiYQIegMHpNkfFCXsNmw=; b=Dv4KFfMfCDduwfkY/iFjxj3gH4605U+hwLdpxvNbywcsW4m8mon8A+i3nWxDf57dN+YkUC9J3Nktl6LHXiJY4Hu3wamO132gSWZ/Y+iGrWdq+ZO2ZxCkxCXyl9oBCv5rAwaO20wdHad0xAbS3UCCRPRGyNmSyrWErohic4/M6jk= Received: from CO6PR10MB5409.namprd10.prod.outlook.com (2603:10b6:5:357::14) by DS4PPFDB583D803.namprd10.prod.outlook.com (2603:10b6:f:fc00::d50) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8632.33; Mon, 14 Apr 2025 03:46:10 +0000 Received: from CO6PR10MB5409.namprd10.prod.outlook.com ([fe80::25a9:32c2:a7b0:de9e]) by CO6PR10MB5409.namprd10.prod.outlook.com ([fe80::25a9:32c2:a7b0:de9e%5]) with mapi id 15.20.8632.030; Mon, 14 Apr 2025 03:46:10 +0000 From: Ankur Arora To: linux-kernel@vger.kernel.org, linux-mm@kvack.org, x86@kernel.org Cc: torvalds@linux-foundation.org, akpm@linux-foundation.org, bp@alien8.de, dave.hansen@linux.intel.com, hpa@zytor.com, mingo@redhat.com, luto@kernel.org, peterz@infradead.org, paulmck@kernel.org, rostedt@goodmis.org, tglx@linutronix.de, willy@infradead.org, jon.grimm@amd.com, bharata@amd.com, raghavendra.kt@amd.com, boris.ostrovsky@oracle.com, konrad.wilk@oracle.com, ankur.a.arora@oracle.com Subject: [PATCH v3 2/4] x86/clear_page: add clear_pages() Date: Sun, 13 Apr 2025 20:46:05 -0700 Message-Id: <20250414034607.762653-3-ankur.a.arora@oracle.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20250414034607.762653-1-ankur.a.arora@oracle.com> References: <20250414034607.762653-1-ankur.a.arora@oracle.com> Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: MW4PR04CA0343.namprd04.prod.outlook.com (2603:10b6:303:8a::18) To CO6PR10MB5409.namprd10.prod.outlook.com (2603:10b6:5:357::14) Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CO6PR10MB5409:EE_|DS4PPFDB583D803:EE_ X-MS-Office365-Filtering-Correlation-Id: 2fc85a70-6231-47ca-13eb-08dd7b06e57d X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|1800799024|7416014|376014|366016; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?TJ3d+Ipdo8geV6wal543AUJagHZoDD0TFyxPmOTSKlS6biNqeOKhoyd7HE6y?= =?us-ascii?Q?YvWzJRKiDGKH4cV591P0qIgnTU6eT+nr1NUJu35JOD7MvHzi9OABh84bmgYz?= =?us-ascii?Q?V2HNqsYADEADDWkdnRiR90nMzWn5vN3gX9/uEht4wr2FBDliIrGhq8gsB5QV?= =?us-ascii?Q?K3ZxGs1VU+c1ul/JcJwAhIousUbwvHkg79gqqE2N+3ZIXfR9+YOGnvSJPqOw?= =?us-ascii?Q?bid+CBbsZ6fxCSkhXGDiHzaGTqhz3AqzZd43tBtwGPSXvjcHZFUfEeWnCZTb?= =?us-ascii?Q?b5pq69jLdXZO6DTNnKfu8+w8KB0ZLRfctBhac/Zy6lOG4XSU6jOdFH4GC6d0?= =?us-ascii?Q?BLyNjHeWbkhGQZ7xAJ48Joa5cEAkp/OX91dI1J39pQ1KP4qu2+A5aC3Uwz8r?= =?us-ascii?Q?pkxHaWsR7sRJow3u8WdfeM7LdMtGzlFxqZXyCZ19u3FDIz/9ENrFerM0UVm/?= =?us-ascii?Q?Gk3+7FG/+YYQfDE7O9UV5So9fOkpsnhTpDV/HxfSSrLMUXgzwt65/58jynOH?= =?us-ascii?Q?9ryyQQHlQEkkJVHYjgkB5kkVPldHP6pzEJRY9QMuyBJ2x747R+2W08NBlBmV?= =?us-ascii?Q?E1q7UiYAkZdVoelUQxK7JXX2RP/ciy8NHC6b6I3HnSgK+QDF+PMZNvIYch+B?= =?us-ascii?Q?QN8lfJmkAyfM1bMEwj8tJ0DFYucXa2JNeQyqL5c90+AQQKJjIM64WiGzADO5?= =?us-ascii?Q?1VN6UtBcHKwAic5CCSbO0wuJC+idTZj/mPXsj4Za832GkJtDfTDjVxmtzHQk?= =?us-ascii?Q?NMfUlqrE+rrZevvYmwyANvlQ8iPMQANk9LY5HOT8SmPOujYsZ+aNr7+swtL9?= =?us-ascii?Q?9+7vN6CFbiXRlbnsMGFkD/x1yaK6sbRqTLKCwO4s4K15eicAly2FZ+YCTH9e?= =?us-ascii?Q?0K3A4gCb7Bc42AZ1XdqAJALQz1kAPFpbS4DvCLvIpP1WIb1wlqcz0FYxUN0o?= =?us-ascii?Q?6sxynNg371c1ohR9UWffVCUCqoj9E/9Cmtkd/E2vlzmm22YG4hMxkFtmEIem?= =?us-ascii?Q?Af3wFgSlyBpfUQj0E/ifZ6/C4Pa4cFyAdwf2Avj6sn12SCvnjXHtAwu3ESGc?= =?us-ascii?Q?sYspG2nsGTFVzj+MqEEa5fHQJToee28yEebzGytHk5VmslnPj7lWqAZ8dKX8?= =?us-ascii?Q?C9kEh3Zuc0eX91TFEbiIfAdxvy8GT78CWJK/FH4oEqRNKpK70MZYTEF2jVqA?= =?us-ascii?Q?O/ZhE/rk/nkkAxkPJ5E6PiXL2aT1XIDy1S7+kebuY4g/lz3JbGwREmPn0J8o?= =?us-ascii?Q?Es8TXcQdu4Z0OnY9Re5Shp/IJMnX/8wxvmfA3Ic1ck7KIKO826lap9IjIyXD?= =?us-ascii?Q?tZEaoE7H3l4jxxrWjZLDGMRSgyoTTskc0iD9ww78Asv9E9noxlbs0RknKVQN?= =?us-ascii?Q?saKrWZzh7OYgglNwsIRiWXwHGdiIO9/Q0XIqFuazL9qet345Zh46gDkm+1NY?= =?us-ascii?Q?a/E6PqHFwCA=3D?= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:CO6PR10MB5409.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(1800799024)(7416014)(376014)(366016);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?CT0ghD5wFdJm5m4aFMoqJcd9xBYjvFTuoW4mHltkecX+p48icsihDG1fsY/m?= =?us-ascii?Q?/eYX5n4QJy955onoPFSolmZPiSMaTPBfOfCxnUmV4v2q+F2IbhMOSaknaA32?= =?us-ascii?Q?PqD6N5jUGcn8vOyTdZLTS/X5I0OFtf7TmaJTDPe9bVTH6vellwvtRAvD6h/T?= =?us-ascii?Q?XLq6M1F3C95Sg9URieB7h5So2laevb3aWQvlKE+u7O0oyLPTJs6s471HrBA0?= =?us-ascii?Q?tfWrdFXvdo1c6e4LHt1iaLUQFp9uM6ombsvICdee7wl+1qdv+CpvXjA0Dpx8?= =?us-ascii?Q?nnrLlcI118Hn/bm3Q1AgATxf/xldUDEgH3hJegEF/xMetpdopXnwdLqa/aLR?= =?us-ascii?Q?LsiA9L9nGnmJif7oJnjjdhBuEPGRTK9u57hgpkfDlChYOYIWpkpzb3AeG/2c?= =?us-ascii?Q?rJ8/khZGc/8yz+iBBgWuNnZNluMvWfUGkXJjOH3rG5PN8N3Y5pzaQJtzSgKj?= =?us-ascii?Q?oOFhicrKHIKpa6Jofa+/e5O8UmIEJRgGO+US4xdWSrJsICf/TDTM9I+lpaAq?= =?us-ascii?Q?15h59Hnj0RmuLjiLMzLhQLcn+5BB9JHP8g8kp6t04uPyEvs6rgvhYtT7SU/a?= =?us-ascii?Q?PE+wuDhuoUS1dU3v+FcdDPYeAXrlkPLPYUGVcjOl2s8Z6NZbG9oBlhyQN7Zv?= =?us-ascii?Q?06dM26t3H+R+zLePZ7kQUBgXin9SHxPmckj3C2TBX3KS8dz0zVp8AyiDGIHc?= =?us-ascii?Q?d5BsYSfj8LefZpGhRhoYVqj+aG4nohGMgvZRTVdWBNAXhJooz+pw3vWBsgmi?= =?us-ascii?Q?EQG48M00L7VvK2yvm8lCc6eUzF/sgXuQJdUnekPxrjxh0p73C3oCpTg6ojg0?= =?us-ascii?Q?DYR8FLuTwDIzIXBd+TGoj17ZAm6F2nsoPxs64AUIGyea0ebI3N5bdKh0AVaO?= =?us-ascii?Q?s2xxq9JriJMDyFoIbDR9JrlpLPX+KtqJs9F/ga8u7HiW5oIM8h5KemJ3a7Iv?= =?us-ascii?Q?g3b4EZY8S7EWIYwMliQ1QZvEhn1nRegadZY21qXgOekyMJ4ZBAPTbiSUSyoX?= =?us-ascii?Q?HcOKHbG5WzgqUHNHcDWQbsIoGsA15wVoja+j8hxIcFi7JbXIjXI4IlXo48m/?= =?us-ascii?Q?2+DNssIZmTk1r/T4oFT1IfrvLD+nizk8Skn7FHaw4uk6sI+yU56nBKDwgpV/?= =?us-ascii?Q?1Tx58l62+fZGi5/tSxiQZeyCggd115S4dzfP3wJyy9pOumz3L+xUe+mO8H8b?= =?us-ascii?Q?n5g85GHVPFHe4aYCii+r2QUQO5QI7gjIpJ5cfKDhHLHE4BAIHVnb7wMR+OKb?= =?us-ascii?Q?YTBVz7zdAR8Ci5rrS05EIoRIqzyP/eiLNf33JuOCSLoojQnHpoXVzxMizPBS?= =?us-ascii?Q?2O65o3+i7oAUJ3pmw5nlI1U7KqRwkA+1+nIYRK4jYk87kLpaOBkDXAfEWJFB?= =?us-ascii?Q?bcrSOTTGuurRs+GzSlkMNba0mhvYpgLwZ9BZMio6ZB5YJgJSnYS1qhgpOnUM?= =?us-ascii?Q?FOqvGTJvaK1uFZdhqj7nlf+w4yUJdlGodiHOphYpqGeGwbeJ8OMAPCzqSVCO?= =?us-ascii?Q?XH4UC8X134n2vOg/GJYPbUrqWGSfgymYl06IDukZd9mq5GZ3MXvycB8SOS4a?= =?us-ascii?Q?QASts8FDGahY4wu2SpOt3EMraNy4mlfHfw0TzgQ/mZSWtTciFe/9KUa5s+22?= =?us-ascii?Q?Zw=3D=3D?= X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: YBaYVvS1ZPeMu4e0dx0wudRV3k2DH03oUMFCz5Z+sZRzOS8RvXmz3vUfs6yapvTYzoZouTliQvdaeqQ9m4J3EoNZWUn4xN1ofkBGkbbb8NT2ANuNTYgFkwjsZZb1O02aGe9hHImzdOk6XbN7V0GS5N8MY8AdbYamTgxUmhNBrA86s6bAw1NSyA28cyNv8iKxg0hReU9NIrbjdwK3WSwHUFOwDvXbWR46UAY8j9Z6YjNJkP5AUA/oQWgKxWQ05F6S9ouw0Dp/wOMh8ofuHxcqD0DxggLczipnXYcK9nvZbclsQiinnE/G73DMtJPkMva66tsMVTmF9T9NUj2s9yu3LwkA6Yuc4YB/6lE+bZSMOTgtkmBHnHaxYTER/nl3CnBQP28cxQkm33e/gGmc967/Q0V6+xXu2GYzro7ACRfFqSvf3i9gyMxhv1CWfHD9s/YkZ1WqvCz50EcpytCT73CoCDC2FB5Za2U2Wosam8d9cqbBlcGYRkdc69C6e/cu06cwgg9O9bNLwXOUt9feiBL3iFlGoOu1HHeUBjRJWwEe9TDvEv3uCG59mD1ZFgYVp5deVH3St2Xfw7wZgw/D8JsUWAB2r4KOsbGAM9toR/oLPqE= X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 2fc85a70-6231-47ca-13eb-08dd7b06e57d X-MS-Exchange-CrossTenant-AuthSource: CO6PR10MB5409.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 14 Apr 2025 03:46:10.7653 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: FukM796r1seXONLhmUE1D7qqNom6wgQjnd+yr2U/fsKc3f0QX+2BGeWDxtx9VlzdslMxuIeL4dEg7HXIt8gNegVE5rVb8KZ+QRAWs8pU1UE= X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS4PPFDB583D803 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1095,Hydra:6.0.680,FMLib:17.12.68.34 definitions=2025-04-14_01,2025-04-10_01,2024-11-22_01 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 mlxscore=0 bulkscore=0 malwarescore=0 mlxlogscore=999 phishscore=0 suspectscore=0 spamscore=0 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2502280000 definitions=main-2504140026 X-Proofpoint-ORIG-GUID: Q1WLOQTmTucUHw8OqWOwXPddsF9slvdC X-Proofpoint-GUID: Q1WLOQTmTucUHw8OqWOwXPddsF9slvdC Content-Type: text/plain; charset="utf-8" Expose multi-page clearing via clear_pages(). Signed-off-by: Ankur Arora --- arch/x86/include/asm/page_32.h | 6 ++++++ arch/x86/include/asm/page_64.h | 9 +++++++-- 2 files changed, 13 insertions(+), 2 deletions(-) diff --git a/arch/x86/include/asm/page_32.h b/arch/x86/include/asm/page_32.h index 0c623706cb7e..73c94c9e4eee 100644 --- a/arch/x86/include/asm/page_32.h +++ b/arch/x86/include/asm/page_32.h @@ -22,6 +22,12 @@ static inline void clear_page(void *page) memset(page, 0, PAGE_SIZE); } =20 +static inline void clear_pages(void *page, unsigned int nsubpages) +{ + for (int i =3D 0; i < nsubpages; i++) + clear_page(page + i * PAGE_SIZE); +} + static inline void copy_page(void *to, void *from) { memcpy(to, from, PAGE_SIZE); diff --git a/arch/x86/include/asm/page_64.h b/arch/x86/include/asm/page_64.h index 45db74db9021..6781cdee6541 100644 --- a/arch/x86/include/asm/page_64.h +++ b/arch/x86/include/asm/page_64.h @@ -44,9 +44,9 @@ void clear_pages_orig(void *page, unsigned int length); void clear_pages_rep(void *page, unsigned int length); void clear_pages_erms(void *page, unsigned int length); =20 -static inline void clear_page(void *page) +static inline void clear_pages(void *page, unsigned int nsubpages) { - unsigned int length =3D PAGE_SIZE; + unsigned int length =3D nsubpages * PAGE_SIZE; /* * Clean up KMSAN metadata for the pages being cleared. The assembly call * below clobbers @page, so we perform unpoisoning before it. @@ -61,6 +61,11 @@ static inline void clear_page(void *page) "cc", "memory", "rax", "rcx"); } =20 +static inline void clear_page(void *page) +{ + clear_pages(page, 1); +} + void copy_page(void *to, void *from); KCFI_REFERENCE(copy_page); =20 --=20 2.31.1 From nobody Fri Dec 19 12:49:37 2025 Received: from mx0a-00069f02.pphosted.com (mx0a-00069f02.pphosted.com [205.220.165.32]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E643514B945 for ; Mon, 14 Apr 2025 04:20:32 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=205.220.165.32 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1744604434; cv=fail; b=axINS7o7LgGX3+jNN3fpv41poKcy1ATy8ForyVM5jsjk3t1mrX3PNDJzz7RcEmx/OsRt7OLawdniqa0AWMt2vpmHqXHrAC8OmDoi12M0dg6QgfmNqvd5d00rG8zzJ14yhPqqzEH+NCngEQajxhRxju96wOjNIOAugBO5KywYPKI= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1744604434; c=relaxed/simple; bh=DTYYWXsBZgbyTRcxISdcfG84AvxH0MzRb65RgqF5tsM=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: Content-Type:MIME-Version; b=gpjIII+JsBX1mraBAkPvFpIHjnXANv+w61yRQUl6elvjJDw1FiNqo2Y5UcoNuA3HafgXHUNJ4av65BrlAsC5MeZbMG2HnHqf8mSO4SvrMMaZxcgdvyxB3Q975hh5bviIhwSB6qahcDPuR402yQ5K8Gpbn9kzEQEyuJdxbb+Rtzo= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=oracle.com; spf=pass smtp.mailfrom=oracle.com; dkim=pass (2048-bit key) header.d=oracle.com header.i=@oracle.com header.b=cJ+48OTb; dkim=pass (1024-bit key) header.d=oracle.onmicrosoft.com header.i=@oracle.onmicrosoft.com header.b=vZ9cUUFx; arc=fail smtp.client-ip=205.220.165.32 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=oracle.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=oracle.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=oracle.com header.i=@oracle.com header.b="cJ+48OTb"; dkim=pass (1024-bit key) header.d=oracle.onmicrosoft.com header.i=@oracle.onmicrosoft.com header.b="vZ9cUUFx" Received: from pps.filterd (m0246629.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.18.1.2/8.18.1.2) with ESMTP id 53E3bnSB007627; Mon, 14 Apr 2025 03:46:19 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=cc :content-transfer-encoding:content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to; s= corp-2023-11-20; bh=u9P3VpLIvdp+NSb70dIKhiVvV1dVwGDeX3t3RWLaGjs=; b= cJ+48OTbBWbtocjQ+mWKg2lWXX2UzDzZzSQi6a6Hqe9n3/doQPCEaBY5EHUBYwOR ++cT4bEj3UbXXdSWlBAmxOm2gp1P+HhqsifwSjMdnBsB2vnTq59zHwCxsbo+b7jE vaMu05dm1jyb56CXH1e67J8EjEAtt70yY1ndHdGm2++DymiXQWZiMU5OKbnjgXyI AlwPSSsv+VZCsQUttzx1m0cwWFkPFplGvX/ar85fmcZ1KgCHayA3kjBHKoB3iHiE Wv96N1EA+np6FR8+xf9TrsR2BzRzFBhSNqPLvqgmnPiuiihXwDCcf4ypucwqLUWL llDFu4CXNC23ZiJZJp+BMA== Received: from iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta03.appoci.oracle.com [130.35.103.27]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 460tm28079-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 14 Apr 2025 03:46:18 +0000 (GMT) Received: from pps.filterd (iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (8.18.1.2/8.18.1.2) with ESMTP id 53E3bADv038784; Mon, 14 Apr 2025 03:46:17 GMT Received: from byapr05cu005.outbound.protection.outlook.com (mail-westusazlp17010002.outbound.protection.outlook.com [40.93.1.2]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTPS id 460d4p6fw5-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 14 Apr 2025 03:46:17 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=E/hV5lD5nb9KfzWOkE1J4SuyOkwyYyG8zPvYgMeLA4JGFuw3nRcADsTL/wV7vCEwXYLFwN95LPcWqn+bhlJQlAKZInu6McmtI/VYAYsRtYUVujvv7BOQgNQ1xd0DWd9MNRYLefH++TriKQOSWe7IRFGzolGXZHwxZEOSv8GzG+eHb7UzSKKfXt42tVp1qop7buoowybXCzQ7AK3wAx3pYpCXIp0J/qKP/RMUUSm4qccgKPEpeAYOBWX/GsB7HeFIWA27wbnF7C6vcerPPlp7H84Gixqo4D3h9Kl3GceBR3oT73UboNqAbV1hZt4gfrp+tAmLpIThkBbwZFutZRZBaQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=u9P3VpLIvdp+NSb70dIKhiVvV1dVwGDeX3t3RWLaGjs=; b=tu9XXATBZyWtjb5uPKRjvIJweqiwXm+CrlEcoI8dcZggES9QG5AK8S+nTZ8lCEfxqoa+jiuiSofhsgrTsfN36wVivwtitlLd4lyEKbNF21jNK9R4L6XL7PxODMg/NayxmvZBXtMbtg0U1O/6xvdkMR6y+GViT+Wsvn9CGJ5cGt73swNYIAxBYdGnYBI7tSO4jC69QzV/kn0s1CE04sh8+e1mxAWKFbW++3/JRwAOTHJqCeSFrnt1N7RZvrqlC0PU6acNwZuWYVqrTzkJnuSig9aIlI8AZCapB8RN2Ld1utx/WW5X2wIEFQPABPiX+ORhrsqnCkv+zUl26U+qdk23GQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=u9P3VpLIvdp+NSb70dIKhiVvV1dVwGDeX3t3RWLaGjs=; b=vZ9cUUFxtoPR2vj4NvUhr/rtkc/UelCi1SlyormO3sKtGx2gR6z50hRI/vyw5ul1Shv0/jttlIUULWfv29OYPS5A1M6nGjRBE09VSS/8H2bIWHv6UyRmPvgfqD8Il8PwEKdo7nB+1bSSIz83X1//f+EJ7t8U8TBZ6E4JsXpB85g= Received: from CO6PR10MB5409.namprd10.prod.outlook.com (2603:10b6:5:357::14) by DS4PPFDB583D803.namprd10.prod.outlook.com (2603:10b6:f:fc00::d50) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8632.33; Mon, 14 Apr 2025 03:46:14 +0000 Received: from CO6PR10MB5409.namprd10.prod.outlook.com ([fe80::25a9:32c2:a7b0:de9e]) by CO6PR10MB5409.namprd10.prod.outlook.com ([fe80::25a9:32c2:a7b0:de9e%5]) with mapi id 15.20.8632.030; Mon, 14 Apr 2025 03:46:14 +0000 From: Ankur Arora To: linux-kernel@vger.kernel.org, linux-mm@kvack.org, x86@kernel.org Cc: torvalds@linux-foundation.org, akpm@linux-foundation.org, bp@alien8.de, dave.hansen@linux.intel.com, hpa@zytor.com, mingo@redhat.com, luto@kernel.org, peterz@infradead.org, paulmck@kernel.org, rostedt@goodmis.org, tglx@linutronix.de, willy@infradead.org, jon.grimm@amd.com, bharata@amd.com, raghavendra.kt@amd.com, boris.ostrovsky@oracle.com, konrad.wilk@oracle.com, ankur.a.arora@oracle.com Subject: [PATCH v3 3/4] huge_page: allow arch override for folio_zero_user() Date: Sun, 13 Apr 2025 20:46:06 -0700 Message-Id: <20250414034607.762653-4-ankur.a.arora@oracle.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20250414034607.762653-1-ankur.a.arora@oracle.com> References: <20250414034607.762653-1-ankur.a.arora@oracle.com> Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: MW4PR04CA0360.namprd04.prod.outlook.com (2603:10b6:303:8a::35) To CO6PR10MB5409.namprd10.prod.outlook.com (2603:10b6:5:357::14) Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CO6PR10MB5409:EE_|DS4PPFDB583D803:EE_ X-MS-Office365-Filtering-Correlation-Id: bf3fd167-6a40-4b0c-1e11-08dd7b06e7f0 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|1800799024|7416014|376014|366016; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?s5au/PX662NJWAMp2e9APYavoAzfzZB4foB5+U6xDCe1YCa3sngr6UgRNnhM?= =?us-ascii?Q?dqI0KzyegPNjvh5NbSU+i8nJZskF54O3tXmewJels5A4dFwHC2gy/JFbpqOA?= =?us-ascii?Q?zEOrgvBLL2i6GLwXBO7mj7XPGcCxsAsmPEV2BQtGrNEft7rMYIhAkhJPLxZq?= =?us-ascii?Q?AEXbBpL0I6Yg4ePvIBpTShVL9kXCEgWLclY88qaja9ER8GHa2MgZSjjAt+9Y?= =?us-ascii?Q?1WpheKsC3AKH3qz8FG+Jcb2A+6vwucQ9uczWg6G5uekgQPqYngs9WyNkDViL?= =?us-ascii?Q?2Q4d9ZNjnrvt/sI7ZtuGSqZz75xk+LXuL9NSvYrhN5DDPWvS7OGpUGBdvbfa?= =?us-ascii?Q?BBH4RX2IwV2fE1Etk+TLr5ac1wntMmEhAwsYyKAmkkFNwGpkDozqq8rrPink?= =?us-ascii?Q?EaITeUNETvEYd56kNIXe/ZuEImHH5Ndq9FSSsXJDjUJCyLAcU7rNc6J/JqbE?= =?us-ascii?Q?9hUbcRFyTaey11bnBAa0kJL2Ot3BPyRboAMneLZLptPY3EBxeTC3eeJCxxZo?= =?us-ascii?Q?mYMAdNyUO1+WaLDptNvjxa4elLv/wlIRNft0cYJSUzFpu1qYaeJ2AFaBmHcc?= =?us-ascii?Q?SK4YlKZklTATJpCw6kDEsqn2s+yj9ghUibD3AOZuLRp+vNC17VGBaZUuZ1XM?= =?us-ascii?Q?dT2deAHgBK2n8EJdwEwIIsHD+qSQEmgBpGWXjSJyRaGR2mf/O5qew3+Pp/3L?= =?us-ascii?Q?EMh3CEAmkmZDs+cGGnLVZBl3hBY92PS8upvlg9WHoEQxgPv7ZqQbJAa5J6KR?= =?us-ascii?Q?8SH9GLhAKY1Je4jLx+ehyMkDtk9BudQBCGAz78Lt0T4WdeM8E20WW2JGL6xW?= =?us-ascii?Q?dVZ+tYLKPKDJ5nA7xJClRyfy6bo5rrmWla6LxoT2WC8coxoqIGSkqy6yPGdA?= =?us-ascii?Q?PhiyeeMEX+CWjd4a6DN0RdqFx9sk5gFjGht4dBazuqb1LHI3XGUKOlANzLdb?= =?us-ascii?Q?28yRPxhqbkUDdkGoZKsbJijTMmfo68Yy4Ht0r5puHG5KrFutH8JihKDXrlKM?= =?us-ascii?Q?sepNh6zjVjWoK40njn268dxzPMQ6113FjQo/PB5lt72XzwV6Sb9NQuMUNkoO?= =?us-ascii?Q?lw92VoVT6J2feWSlBfG7CX8ZOSfrmEcPUrFHk3/RdRWsC/2GE93xQiXUVwoS?= =?us-ascii?Q?SrtHWKVG6buRfAUzhp0RAHCzfS2pndRn1gu0Ilmpd/j/E8z7b2V+mSiQcDpk?= =?us-ascii?Q?A17GCTx8J8wh7anxRqruowC/r4Zi3yew7CjqU+Z74BpxhQK4FHNI9GmtrYuH?= =?us-ascii?Q?BgNOSS7NVESDe6y5ji1r5AxoR28Wz2i6WPhY+81gMY7fdGswiuX44B9VmBJ5?= =?us-ascii?Q?wpzPY99iwR3YlS6qNC4tzaBJ8oeZqQYnnCgdD4+wNN39u6TLQLv67eCvIYR5?= =?us-ascii?Q?FDiBqzTjzV1raKNI3GxoOs+pST9cRCKoyRf7Ye/LX7ubWTSWDl+94KYWDPXy?= =?us-ascii?Q?zXKR67VOgDU=3D?= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:CO6PR10MB5409.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(1800799024)(7416014)(376014)(366016);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?QMUDcZdUwgCup14Ej7pnU36bgHMFUNZzdHpfOZpQlRiWD8Jvb0TRJoB3Ke0z?= =?us-ascii?Q?R/5tapaNHmZIrDYRqWuf94DmnX4T7m8wi4MShtMeoP07bWr0eEh/PjT07oWy?= =?us-ascii?Q?ZbctuS1ZSkmkNIRuAhOZRcpxI1nLaWwKZrq6ocQLEI9QeE9YlOyMqyd/8ffQ?= =?us-ascii?Q?NKaF9gjWd4kz8RtPK2PvjHxSw4YODoHE+H0hd3J69397dO3Vm5ymJK91H4l/?= =?us-ascii?Q?X6vvMZrIvBTIFG8jLRXF+cC4fmgice6YctCQf3QZHe61txGx462ukxQHQzSH?= =?us-ascii?Q?Om8p4eF+c+oVJhD8xA9eYSz6yjhlhkuItosX+EZwc6QF8WhUqhjrBwPlHMy4?= =?us-ascii?Q?0ngPA0bsDIwYJbGgmrXyK9p4JXTIOT1MQ7dyQM2UDOsFadQiF4hp5t4a9gRI?= =?us-ascii?Q?yMPZud5zWAmCc0NqpYD6i5GmxTqJdGrOiGvxIcPY5BMvmVdboUxO/mnQZkvq?= =?us-ascii?Q?uXr74d8C5txBDyUhIX/gbOjXr6SZtG/Dtl4mtWV6NGMH+Fai7n6wSq8fFjYi?= =?us-ascii?Q?rMGf0/7t5Am9pRq/jEn3BZlk+2wQqLwShtcWHsV95U7gXXBQnKJckvHFHfa5?= =?us-ascii?Q?BC3sZO3bBYVKePYmW/5fllxDMaAsbP7KALMULHwKlFPBbeciD72RmAGft1xa?= =?us-ascii?Q?mGyxqFB/CTPAXWywU+qmqaDgC7cz5Or5NSj+vVQgk/AQVfpu5nz9fFFiJO9F?= =?us-ascii?Q?YsWSckwDk0lZiB/zmTqDKWMZZ+G5sS4qWwFJYXrZPqYUW7fqSgTReFR/Zq7J?= =?us-ascii?Q?gg0h+9c1JVybD8aUbJ4gi+Sfe5BKowf1iI83TCuaQ1HRSststeKgLlTdco9b?= =?us-ascii?Q?awGK/uaKbVLkD3t2NLhZ4E86WDgPEBW7mi5zi09S/PzLqRa5e+GnGSbwzuUB?= =?us-ascii?Q?xOCkhe2GPbnk5CgrvRfWFQFp6z09TI9NJprYu6v+ZneitF3ixX/LaIl7h2Q2?= =?us-ascii?Q?xoZRPeGkGYMjyn0ovuDMkEeHyiBRTZZ/7pO7xd1jy0T8HqqHPy5bhDMifvyM?= =?us-ascii?Q?BB4IR+VXdcTpvu7lveW8AB68xIt8FmN97c1uPPgNg711p1HgqinUNfOOo6Uy?= =?us-ascii?Q?BABaJHgjCZHT60r1n5WmNmFvoWaXoG5ex97o2PWErlymsgEZB5JpZFdy8o34?= =?us-ascii?Q?6QICB+qwESu/Y+MAcvnZDIrvVoTN+FFwTNZY2v9Y1/45FEag4cbsXUhjAi9A?= =?us-ascii?Q?vYzlr1dngN8wsovVzZ/6Kq0vajolUqlVJEILdPTe8YSgCnjUQWo7kxJlj0lW?= =?us-ascii?Q?JLMc7J206f7rulx9T3kFpxdyYvyklH10qweS5GNdpsFMSYfF7J2+t8h9Oiiy?= =?us-ascii?Q?YKJKiQdNXMolGzHjpmjcI7xc4BAC08TcDGGmOBsDoMg2HmzuqnYclBzvqdcw?= =?us-ascii?Q?MLVOOzl9/EtdkeyOP/rdtKC8+CGIvt60WpQjDGJOOdNom+RmMuBZSu+L6MMK?= =?us-ascii?Q?YVAQIQxjZpj1jktuzfOBI/2hxVxXQO3NOr7WRydu0DNqAueSaDJs+BXzKSeM?= =?us-ascii?Q?2srU8c3Wws8LhoQo563bPPbQu/3YJzl3lneq2SvvZtWgFfdn3QjAORibDYgA?= =?us-ascii?Q?XdgdLdEqvwgsLX1Cc2WWrBLUCXQyEy/SPxnCUvhKbpVelCztilfyiK4BRNvF?= =?us-ascii?Q?ew=3D=3D?= X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: t+dRBiyBM8Q870hwtlLyjXueWRR7o8NK9Vuj3Hr6BQ2vyUojWLfxVAnUvXnugNIJUGNrEkga8prodaFdNtu9M/qKpweDqeY8M9VZ3bwdJ72WKMbf+hML04CXRY3BdoqItAxbTQqIuILLybFmcJrhWATkM7MZiOHvJkUAQoXyJHGJA/iNLRsign7ABzsMlXQtRoYWYe2iRsbDS3HXoXcm0W5lXGs5keWLOrkUkPF1QSYSvXzuuLCSM0jbMeFDCGifZ2Il/xa/+kbHg4tdctdZyj70xvxlgfsgELDSAxw5K6gE+4gnW6qzv0493gMUoxH7XXsPGEZpucHG1OINIiHFGJXEJ87yESDwpWkRHApf2WP7aD6D4xVNuWunCM3wbU3oMGsCEwuvYc7JCIdEla3lDkH0VHD8MdcP7+SH4CGc52/kOwssqWNexUFrrShqEMla8DD3OlJ+6SFjM00+4VsQt6PqROvbznND7z/OQe5/LEPizpU9ZGjxJ0/DRwYOhkCWrAy7qZURdCG5rjvFHvoVQHEF9bDpLSE64G3nNyB3DK5Y9QqUuNsG1pQCr8vRjKQPub1nmxJp8VCVJFvufHgjCCXnqOq/goJWZO04zirTcrQ= X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: bf3fd167-6a40-4b0c-1e11-08dd7b06e7f0 X-MS-Exchange-CrossTenant-AuthSource: CO6PR10MB5409.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 14 Apr 2025 03:46:14.8770 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: bU4gXNVzG/GowSFa7H1OFHyXbgeuIP9Eghq7rrdFPHYJS3fLdcakeuneCMGESPRQTDEpZjezCeSJyIRvGJNs9+yY2SR0w06Zz5Kp5zcg5d4= X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS4PPFDB583D803 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1095,Hydra:6.0.680,FMLib:17.12.68.34 definitions=2025-04-14_01,2025-04-10_01,2024-11-22_01 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 adultscore=0 suspectscore=0 spamscore=0 bulkscore=0 mlxscore=0 phishscore=0 malwarescore=0 mlxlogscore=999 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2502280000 definitions=main-2504140026 X-Proofpoint-ORIG-GUID: RudJO8jlJ9QWLlvv5h5ga7DrfUtBWKCO X-Proofpoint-GUID: RudJO8jlJ9QWLlvv5h5ga7DrfUtBWKCO Content-Type: text/plain; charset="utf-8" folio_zero_user() is constrained to operate in a page-at-a-time fashion because it needs to handle the CONFIG_HIGHMEM case. Additionally, cooperative preemption models (none, voluntary) need regular invocations of cond_resched() which limits the chunk size when zeroing. Move the page-at-a-time handling to __folio_zero_user(). And allow an architecture specific override. Note that when running under CONFIG_PREEMPT_DYNAMIC, we could switch between cooperative and preemptible models at runtime, falling back to __folio_zero_user() in the first case. Suggested-by: Linus Torvalds Signed-off-by: Ankur Arora --- mm/memory.c | 38 +++++++++++++++++++++++++++++++++----- 1 file changed, 33 insertions(+), 5 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index 2d8c265fc7d6..ac6a19d7bdf4 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -7235,6 +7235,32 @@ static int clear_subpage(unsigned long addr, int idx= , void *arg) return 0; } =20 +/* + * __folio_zero_user - page-at-a-time zeroing. + * + * Handle cases where we have nothing better available. This could be + * for a few reasons: + * + * - the architecture does not support multi-page zeroing (no override + * for folio_zero_user_preemptible()): because there might be no + * optimized zeroing primitive, or because CONFIG_HIGHMEM is supported. + * + * - !preempt_model_preemptible(): need to call cond_resched() + * periodically to provide reasonable latency. + */ +static void __folio_zero_user(struct folio *folio, unsigned long addr_hint) +{ + unsigned int nr_pages =3D folio_nr_pages(folio); + + if (unlikely(nr_pages > MAX_ORDER_NR_PAGES)) + clear_gigantic_page(folio, addr_hint, nr_pages); + else + process_huge_page(addr_hint, nr_pages, clear_subpage, folio); +} + +void __weak folio_zero_user_preemptible(struct folio *, unsigned long) + __alias(__folio_zero_user); + /** * folio_zero_user - Zero a folio which will be mapped to userspace. * @folio: The folio to zero. @@ -7242,12 +7268,14 @@ static int clear_subpage(unsigned long addr, int id= x, void *arg) */ void folio_zero_user(struct folio *folio, unsigned long addr_hint) { - unsigned int nr_pages =3D folio_nr_pages(folio); - - if (unlikely(nr_pages > MAX_ORDER_NR_PAGES)) - clear_gigantic_page(folio, addr_hint, nr_pages); + /* + * Use the arch optimized version if we are preemptible and can + * do zeroing of extended extents without worrying about latency. + */ + if (preempt_model_preemptible()) + folio_zero_user_preemptible(folio, addr_hint); else - process_huge_page(addr_hint, nr_pages, clear_subpage, folio); + __folio_zero_user(folio, addr_hint); } =20 static int copy_user_gigantic_page(struct folio *dst, struct folio *src, --=20 2.31.1 From nobody Fri Dec 19 12:49:37 2025 Received: from mx0a-00069f02.pphosted.com (mx0a-00069f02.pphosted.com [205.220.165.32]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 70C0A28F4 for ; Mon, 14 Apr 2025 06:08:13 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=205.220.165.32 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1744610895; cv=fail; b=IvoDVjXxA7Vw1yWnR4Hkx+SNc6LcUdfhwiTSg2pI7X7nseEOr4I3sWGOmo8V4RkYdp/RPDn6qB1Epk25mqojLGrzkr4DythDPcHlIbMkMddoI05Fva26dJ4FcMVJZUH8W4YzrDzNW0UqtLIfqNCbWQAulxBfg5KRz8d4KPtuxLo= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1744610895; c=relaxed/simple; bh=Z06ggt/wQhSguvD9qeHtQ0H6NUOfmJ6CRslppiKNoYI=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: Content-Type:MIME-Version; b=ll5JVf0hHMMTRtmdgE9EqPQyi7qjDCvt6PW0wVHk8eNiVx3uDqvxMPwa+kHcHOSEMxdWSxfP5hh+RHpEjBaSBUxTkJBXi2KhWMjaGESk0DAqjBua0Y5lvM0yLm663QVZxTMz8iv9kYuGVw9Hv7wON3ltMCFaaAF3dKPEZn3ZWxc= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=oracle.com; spf=pass smtp.mailfrom=oracle.com; dkim=pass (2048-bit key) header.d=oracle.com header.i=@oracle.com header.b=HuwvCD9X; dkim=pass (1024-bit key) header.d=oracle.onmicrosoft.com header.i=@oracle.onmicrosoft.com header.b=oPEXvWHB; arc=fail smtp.client-ip=205.220.165.32 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=oracle.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=oracle.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=oracle.com header.i=@oracle.com header.b="HuwvCD9X"; dkim=pass (1024-bit key) header.d=oracle.onmicrosoft.com header.i=@oracle.onmicrosoft.com header.b="oPEXvWHB" Received: from pps.filterd (m0333521.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.18.1.2/8.18.1.2) with ESMTP id 53E3NRmr023153; Mon, 14 Apr 2025 03:46:20 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=cc :content-transfer-encoding:content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to; s= corp-2023-11-20; bh=QS9jq8IqfRXva5KezIzWJiDSS2AnC/XV5bo2nTV5Tvo=; b= HuwvCD9XHblyct3eaf5GnHGIMgBvHJaDKfPk0TySlDakwYUlvkzwygOWjAh7tukC 8fXULszVaMjq4znh7QdePFkuW1m7Bd3spk7+/1pqbI6kV2RJt/MHKeYy/0T+D37N mQMX5eMQsMuqaaod96LMtCkHtrKK/ejVYk5gEw+jsLgrnyfnUWfRvg62nnv2oKmQ x8cF3VQq1OLNP2UhCVjBicwJCcfw5NWhIwF9sfue3QIK01AN4kCftUYd1ye3n09a mFwV1NBq5YGbwMv/QJGi0nu+GQne7Y4EudIQXJWQGmAderkQ/IZp/Jk0o3e2FerW eJ0spCeDaYrNpT1G0Qo+7g== Received: from iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta03.appoci.oracle.com [130.35.103.27]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 460td3r0hw-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 14 Apr 2025 03:46:19 +0000 (GMT) Received: from pps.filterd (iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (8.18.1.2/8.18.1.2) with ESMTP id 53E3bADw038784; Mon, 14 Apr 2025 03:46:18 GMT Received: from byapr05cu005.outbound.protection.outlook.com (mail-westusazlp17010002.outbound.protection.outlook.com [40.93.1.2]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTPS id 460d4p6fw5-2 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 14 Apr 2025 03:46:18 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=hj8Qw0PP9dUeGa7INdfx19Xwzwiq4ArwfxtSHXSyTT/yNJZjXH4hmxk74ERO/o5tkOiWZSVsB3ci95/kMWRW4oJxDkOmGpBwPB9Yu2YLSSEpOI3aqVJnmpsFDyLGMzRnaRo479fIUCHyPePT7grSo+xiOeOOIia9oyHLxc6ke0afx+ZGNx/jaZrLoR0bAE6Hg3T3L/WPdbsw3Plu1GEds7braFdBSClEFsTB18wkJte9Mli0CnvrVAofC1CWXgxmVbJjc3FSQhSFFzJlF6H6CGUs6BuGNGXGTn/49Q9cvpz9w0+gRk3oEa2F+KE24j8AHkNFXifwpzA9wK5eAao01g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=QS9jq8IqfRXva5KezIzWJiDSS2AnC/XV5bo2nTV5Tvo=; b=yCvBjA+Tf2SsZj4vSI1DVb6VfLUi3iKi6hBCeB+2xb5e/5+EhoEzfFx844PK5mlHumx97dFZqdUzjUpLWK3KNopae854nXFGTbgpvG1VDlRnhfPgidlnPseA2lF8UV8smu8xZCwIA9x/8+zjC0P/vpWPEhYxGtEQfLU9tcOMGyFdiUiTMitmE+/21/OttMjOQ/LudKE+cNskjFYZWmtz71BLQ10UwbKoHlbxz9DMRy5DM6V7wxReygxOmAWZqwH+21//WJfxErk1sYt9grAQJ3uk1bkaDptJqRlZ7eYQENPVwRVgZ0j2LkIcXZ1kGWN1iXrlt5k0H0/0jiRS9Wn6yw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=QS9jq8IqfRXva5KezIzWJiDSS2AnC/XV5bo2nTV5Tvo=; b=oPEXvWHBf7o4U1zznlQSyTnpb1r+Lfqci/Hr18sOaWouCIzLU6UxLHivcSBmtl9kuGhQuV7r689xJWYyK31on3Kt8sFle1FSya+fiJXoZlfFoqpAslZ4HQy825FQKIxg/HvoFnUjdamIeSrSf4uJuRJG9LKg+7ZL+2R7E8RRSwM= Received: from CO6PR10MB5409.namprd10.prod.outlook.com (2603:10b6:5:357::14) by DS4PPFDB583D803.namprd10.prod.outlook.com (2603:10b6:f:fc00::d50) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8632.33; Mon, 14 Apr 2025 03:46:15 +0000 Received: from CO6PR10MB5409.namprd10.prod.outlook.com ([fe80::25a9:32c2:a7b0:de9e]) by CO6PR10MB5409.namprd10.prod.outlook.com ([fe80::25a9:32c2:a7b0:de9e%5]) with mapi id 15.20.8632.030; Mon, 14 Apr 2025 03:46:15 +0000 From: Ankur Arora To: linux-kernel@vger.kernel.org, linux-mm@kvack.org, x86@kernel.org Cc: torvalds@linux-foundation.org, akpm@linux-foundation.org, bp@alien8.de, dave.hansen@linux.intel.com, hpa@zytor.com, mingo@redhat.com, luto@kernel.org, peterz@infradead.org, paulmck@kernel.org, rostedt@goodmis.org, tglx@linutronix.de, willy@infradead.org, jon.grimm@amd.com, bharata@amd.com, raghavendra.kt@amd.com, boris.ostrovsky@oracle.com, konrad.wilk@oracle.com, ankur.a.arora@oracle.com Subject: [PATCH v3 4/4] x86/folio_zero_user: multi-page clearing Date: Sun, 13 Apr 2025 20:46:07 -0700 Message-Id: <20250414034607.762653-5-ankur.a.arora@oracle.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20250414034607.762653-1-ankur.a.arora@oracle.com> References: <20250414034607.762653-1-ankur.a.arora@oracle.com> Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: MW4PR04CA0340.namprd04.prod.outlook.com (2603:10b6:303:8a::15) To CO6PR10MB5409.namprd10.prod.outlook.com (2603:10b6:5:357::14) Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CO6PR10MB5409:EE_|DS4PPFDB583D803:EE_ X-MS-Office365-Filtering-Correlation-Id: f7f12fac-2d80-43c8-e642-08dd7b06e878 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|1800799024|7416014|376014|366016; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?YP2SOdURxkuTdWp1K0h/3iuOT1h+7RjbPvBMeHVnh8hVO+OEU0tbdSUtDXnn?= =?us-ascii?Q?qmrsFq1qFdwsCH11mabru5+ui21Cr+CmIUGJ79mPacX9Rw5EfMrOQ5NDsKCU?= =?us-ascii?Q?nvp0jpueAxdUHKEFUaWlw6DdimqkPUqln/8J1OriJeCFZb9IbPhBaYt5RkiN?= =?us-ascii?Q?UKG5gzmhcNwKVsV12KF4HRTbglMpSGOW0xNj+Yr4z1s6MqZBPrsph5om0wC8?= =?us-ascii?Q?Sy4lYIqhrQPV1MhlQg4d3JViy/Zd0T6aHybB26l9CP1HH5VA+uWuS8EOGwrL?= =?us-ascii?Q?MFjJNDtMEaHMYGUyjseQu3lvcDjSquaeRBjU9THPE/m3APIU2kdIJvo1x7HW?= =?us-ascii?Q?FpGXiUn4aIoou7iXs3cJrTltgTYuwtcjjA9U5Y3qXGncc5qzoDl3Ys0A6YKr?= =?us-ascii?Q?7nteko3KQVgQyXbJ2tYjWMnubbwAFSTQOwXAAS1/+e2lNb0kIOvNjk051klF?= =?us-ascii?Q?znnSTUjgOc2yjqYvxHTsv+RvwaQqcZLmGmyJZoiKHxJNjnNjTUT5xRf0S4CN?= =?us-ascii?Q?tec17SpiP9tqDxPAacgTLuFSjldKxlQpK/jcVspoYnExzc1Armyr9xmcHZ8m?= =?us-ascii?Q?POIPaaEw4jTcq6PVGlrYvOtO8+1zme9vqhITNgk4xjhaGMSTeg8ZS3AXq3ao?= =?us-ascii?Q?KdqunEzwobNVi7k3229uvCQbo55tlII/f+HaE5hNEouI06+0MwB9msnR5PIA?= =?us-ascii?Q?IPA22tvV4oZx8lIDQVi8XczYXRi5Xnz0IynahBVusiyZCkS1U4OLbA4c/b3k?= =?us-ascii?Q?xlz8UhbVif0lLWmm9PfAxa08qQAN72NQJjWz0EGvdQdman4e7sq9v9QkDWjw?= =?us-ascii?Q?5zEViwvw6Cggs7j1icO2Ll3SfeWLufa/ExBx9Xirhj3WDflBnOQ0/WD2jsCJ?= =?us-ascii?Q?HQXSvISlDV1MHRn/0p5yLYbEquS9x4yWPZrUORmitYwkDyauQlJHT0cyzrfi?= =?us-ascii?Q?Gni9A8nBTW7Hz8P4hrs99Ve7i4aKydhR11hBa6FvYoUxF4X2aknecbIdE7To?= =?us-ascii?Q?6jbj5ufQ6yahCcrMhbf09K9QJ6Jyb56rUywIH5lLuNYT/ydu2NZHIexGWDsw?= =?us-ascii?Q?1kISc4f7lRMb0EMTMeCz9Du+hXxSI4iblUo2YMfQR9Gh8vgYGjJDvRunA1ok?= =?us-ascii?Q?KJaJ46Dy7euaCs9oMzh9iw4j0brcjayQXp/PG+/ZKSIewPV96oE9xVwFwu27?= =?us-ascii?Q?eq81PGkfjfWirnx+3PNvFUHVhCs6kEuE+yEZJUx56502Rdef5U/8T0hRy+g9?= =?us-ascii?Q?7U3LU49Zg9ZnEEZP/HhVc8BmlpQbtOBEoh7eQGFs8bBZKaQ6RoweQF4IZcdK?= =?us-ascii?Q?TFU7jWH1HjtKztqd3xma2Grguk9NRPO4CbPi0neDYzz1U+iXtyRdQCblFua/?= =?us-ascii?Q?tvu/hqdBzaRda/4gI7JRQcanZxNKPhKyzDR6s9NoPaeh4r+zu3x+SJjAtQVa?= =?us-ascii?Q?9HqVIJwcelM=3D?= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:CO6PR10MB5409.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(1800799024)(7416014)(376014)(366016);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?3/OkO2bHFRYajEHPHwI+KReLd85ayYYWDZwrGME4TGNwZVP0diVs+ZngEzbS?= =?us-ascii?Q?h+ee//yU+3BftitsHDkm5scT+TjXXSmo+bwzt3WypDrudLhqumx8HuhlYoem?= =?us-ascii?Q?dUchB/YEsDpxxrX2CIb/krOnI7LeqQRY16Rm4yccjfDapmY5aPhKLG9yToWb?= =?us-ascii?Q?4B+Qb2Yy2PkxbblzVav+3Sc9XqTzDwoH28etfZ8/wpuXQ2oofkYtwvzYnjnJ?= =?us-ascii?Q?9TLlf4OqieoN+Od7aoAZoVw+g0ogk7S3wC0iad1UJqMzDGsTf/I5cbBHmpR0?= =?us-ascii?Q?TA5zbrdMqBBfkqxpojqPaE/pKooLbSnZia06aiSZgwu8MUmTG5pR6fzwINfK?= =?us-ascii?Q?Uzdjy7ElFQ3tOxu5wG9uVTAouR3Iqqs23XBKxYnBLwHa4w4Yrv4QwiayMQ4Z?= =?us-ascii?Q?synB7RH9x8bvoD4vJY/ADj84aEeMGm1ajW2ywqt3bOSYtiJZzGk2H7pXYa3I?= =?us-ascii?Q?lnZ3EC/vXrm9HZLZBQCyGgz2kOm8uREddqTLeST0/XS8xiaS+c3qODc1Acz5?= =?us-ascii?Q?rlKMAms3mRHt/P1uWvGnQAOnPWsmL0xIL2v/1BOId9TCDOR5liiajhV8O1J8?= =?us-ascii?Q?ZgOMUDaxNnA3Hgcv9sFOg4OmzvlPXg16P8QeNECQv6wyBNMhLf/gr5Lu7wSw?= =?us-ascii?Q?J7TeiaNlLnxzicbCvchJpiUUu1/BTW+dnhjIi6LJyyYwBpQjX78PWD5n4b6r?= =?us-ascii?Q?u6Jr10yepRFM3wn0/Tl0K3xSshOHtW7RVfDEcB7R/gyj8p6teHkhVmWHrgr6?= =?us-ascii?Q?pW8OW5NWb3S/ciNr4qmRoj/W+C+vUx+OuMZ4rBdqCML3k/FxXSdyNmD4cENr?= =?us-ascii?Q?Sn/7yCuDItXpFZpOTXkyM5OLd/PyEJSNXJLuUdUsamXpbbq6AHAD81qzIny5?= =?us-ascii?Q?8g7aQitPYshIhJDfYwice3hom/4r3C1HJz/IvTLY7ZwYVoqSfODUn1T8QL83?= =?us-ascii?Q?AEHlzxxoJoHcGf+tmjCXm+rD/BLmH+y+7mJMMyFxJMMYIQQ0xmYGf7j6GGpa?= =?us-ascii?Q?w8Rw6BZjZUc6EZ2tzcdf9I0zoeiKW2wzMcjWbvAIbM4qiscOBH5x8eBhHfow?= =?us-ascii?Q?tPhH3ani3Fpe5s8zOVLuwdrMZZANfmw7OrHwPKE5TRNVOkOT807RPXBL7zSq?= =?us-ascii?Q?2nPMaH9aHdLMRaD2b75AeW6L1qql3EgX9wNir4vItxlserLovbkRZpS9g8xI?= =?us-ascii?Q?fzlcMDpH2XUfWYzibRSjMDJkVhWMHVMd5GtHB0RFFbmmRMSCJWaXrv+XeeIP?= =?us-ascii?Q?VyQyuupMhCrjutSzjmQRi0hgCchcy+KGhzcAEr+undoGge8U4h1759xFAHaQ?= =?us-ascii?Q?ICDZKFf4lz27CLbrm7rWeZrZRUZXmFZMszOt7Bg8D6Ke7NCMxYl1Nm/n9Cxw?= =?us-ascii?Q?2ConhBkS107vpkTvwzA55nfOj2oZXtTQMykDRon8X5xEUh2U3R27BBWyNv8q?= =?us-ascii?Q?6tJtcFTp8Kd5j+BkBW9pTQMmJI8KUchZ5OXOywut7mKUOipBTA8Qg3tcutnf?= =?us-ascii?Q?uWtaw0kAdE7B41cjvGrgCzE+SY2CTMp1lGpRJLMGsLa9uriYP94S6+OkgwQ+?= =?us-ascii?Q?qAX8uJ1o32bc1AgbjJTzQrai1OmUNW2y7D4RznESvQlLL8WK8DADO/rBynKr?= =?us-ascii?Q?MA=3D=3D?= X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: P5di3D2gqRr7aE8lBMVuoV+774d4vN4tw8XZUTXniuuPDfOdAP5xd897/+7Z+Yp2MLm1054J8+xHPLmVhn8YpayqfTuOYeOi+o7I2D5S4NpKtuoGuxkQKhleFNWXnN7Zzb86TnB0m/Y8R2i0spFWARGk6Nityo3yaoU/4bObEKWxTvPBuPovTm1qcGxi1Ff26pxjV79svpptAzpr3EIhnHxLd+QO+pYHNcCKPQoE6YZQmlKE/1Cm84YCMwV+HZ5EFH/2/uNv9M+U994fZ9RZd/czbRGrgc/Zjwwl4JO/06+0NbDGo2zOUb1cft4IBiMDNi0TsMcf4A1aGDDGhSIjzb8YoMQuGNMFDEBZ1qKS4N0mBFU2Jjw7UWR6H6q7EcRVBhqf3LxasV7MfOo59jiURnDVDLxB3XayXzVvkki9OYccxl6Sm1vBVYPfw4I8SViFAa32ZmS8srJFDJBJ2EeXYTgWEhphyt0NHHq38yaUmDFElkeXV+8c4p3h9+RcZcysymdf7fp4/fI6jfB9lAx6qcs5GJrxcMnC42kYZI8xxh+owZX2KHd5Y4cSy4PRcSqE9JIKgdr59F8vtpnZqY3hwpeIE5HfYMTgJf01JVODa9Q= X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: f7f12fac-2d80-43c8-e642-08dd7b06e878 X-MS-Exchange-CrossTenant-AuthSource: CO6PR10MB5409.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 14 Apr 2025 03:46:15.7971 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: ga1Ugbygf2JcPiBZu8+/sQ1s1BlYTsDqt2kOlONXXy7opZj4vr/QBsRSK8qofy1tXEynm+pk1JWCY8bq95ilgRDuLZSgrQGw9cqsOz3KfoA= X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS4PPFDB583D803 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1095,Hydra:6.0.680,FMLib:17.12.68.34 definitions=2025-04-14_01,2025-04-10_01,2024-11-22_01 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 adultscore=0 suspectscore=0 spamscore=0 bulkscore=0 mlxscore=0 phishscore=0 malwarescore=0 mlxlogscore=999 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2502280000 definitions=main-2504140026 X-Proofpoint-GUID: x2w-Hq3N11mBrFJuEoemsPm3_9WYSaHj X-Proofpoint-ORIG-GUID: x2w-Hq3N11mBrFJuEoemsPm3_9WYSaHj Content-Type: text/plain; charset="utf-8" clear_pages_rep(), clear_pages_erms() use string instructions to zero memory. When operating on more than a single page, we can use these more effectively by explicitly advertising the region-size to the processor, which can use that as a hint to optimize the clearing (ex. by eliding cacheline allocation.) As a secondary benefit, string instructions are typically microcoded, and working with larger regions helps amortize the cost of the decode. When zeroing the 2MB page, maximize spatial locality by clearing in=20 three sections: the faulting page and its immediate neighbourhood, the left and the right regions, with the local neighbourhood cleared last. Performance =3D=3D Use mmap(MAP_HUGETLB) to demand fault a 64GB region on the local NUMA node. Milan (EPYC 7J13, boost=3D0, preempt=3Dfull|lazy): mm/folio_zero_user x86/folio_zero_user change (GB/s +- stddev) (GB/s +- stddev) pg-sz=3D2MB 11.89 +- 0.78% 16.12 +- 0.12% + 35.5% pg-sz=3D1GB 16.51 +- 0.54% 42.80 +- 3.48% + 159.2% Milan uses a threshold of LLC-size (~32MB) for eliding cacheline allocation, so we see a dropoff in cacheline-allocations for pg-sz=3D1GB. pg-sz=3D1GB: - 9,250,034,512 cycles # 2.418 GHz = ( +- 0.43% ) (46.16%) - 544,878,976 instructions # 0.06 insn pe= r cycle - 2,331,332,516 L1-dcache-loads # 609.471 M/sec = ( +- 0.03% ) (46.16%) - 1,075,122,960 L1-dcache-load-misses # 46.12% of all = L1-dcache accesses ( +- 0.01% ) (46.15%) + 3,688,681,006 cycles # 2.420 GHz = ( +- 3.48% ) (46.01%) + 10,979,121 instructions # 0.00 insn pe= r cycle + 31,829,258 L1-dcache-loads # 20.881 M/sec = ( +- 4.92% ) (46.34%) + 13,677,295 L1-dcache-load-misses # 42.97% of all = L1-dcache accesses ( +- 6.15% ) (46.32%) That's not the case with pg-sz=3D2MB, where we also perform better but the number of cacheline allocations remain the same. It's not entirely clear why the performance for pg-sz=3D2MB improves. We decode fewer instructions and the hardware prefetcher can do a better job, but the perf stats for both of those aren't convincing enough to the extent of ~30%. pg-sz=3D2MB: - 13,110,306,584 cycles # 2.418 GHz = ( +- 0.48% ) (46.13%) - 607,589,360 instructions # 0.05 insn pe= r cycle - 2,416,130,434 L1-dcache-loads # 445.682 M/sec = ( +- 0.08% ) (46.19%) - 1,080,187,594 L1-dcache-load-misses # 44.71% of all = L1-dcache accesses ( +- 0.01% ) (46.18%) + 9,624,624,178 cycles # 2.418 GHz = ( +- 0.01% ) (46.13%) + 277,336,691 instructions # 0.03 insn pe= r cycle + 2,251,220,599 L1-dcache-loads # 565.624 M/sec = ( +- 0.01% ) (46.20%) + 1,092,386,130 L1-dcache-load-misses # 48.52% of all = L1-dcache accesses ( +- 0.02% ) (46.19%) Icelakex (Platinum 8358, no_turbo=3D1, preempt=3Dfull|lazy): mm/folio_zero_user x86/folio_zero_user change (GB/s +- stddev) (GB/s +- stddev) pg-sz=3D2MB 7.95 +- 0.30% 10.90 +- 0.26% + 37.10% pg-sz=3D1GB 8.01 +- 0.24% 11.26 +- 0.48% + 40.57% For both page-sizes, Icelakex, behaves similarly to Milan pg-sz=3D2MB: we see a drop in cycles but there's no drop in cacheline allocation. Performance for preempt=3Dnone|voluntary remains unchanged. Signed-off-by: Ankur Arora --- arch/x86/mm/Makefile | 1 + arch/x86/mm/memory.c | 60 ++++++++++++++++++++++++++++++++++++++++++++ include/linux/mm.h | 1 + 3 files changed, 62 insertions(+) create mode 100644 arch/x86/mm/memory.c diff --git a/arch/x86/mm/Makefile b/arch/x86/mm/Makefile index 32035d5be5a0..e61b4d331cdf 100644 --- a/arch/x86/mm/Makefile +++ b/arch/x86/mm/Makefile @@ -55,6 +55,7 @@ obj-$(CONFIG_MMIOTRACE_TEST) +=3D testmmiotrace.o obj-$(CONFIG_NUMA) +=3D numa.o numa_$(BITS).o obj-$(CONFIG_AMD_NUMA) +=3D amdtopology.o obj-$(CONFIG_ACPI_NUMA) +=3D srat.o +obj-$(CONFIG_PREEMPTION) +=3D memory.o =20 obj-$(CONFIG_X86_INTEL_MEMORY_PROTECTION_KEYS) +=3D pkeys.o obj-$(CONFIG_RANDOMIZE_MEMORY) +=3D kaslr.o diff --git a/arch/x86/mm/memory.c b/arch/x86/mm/memory.c new file mode 100644 index 000000000000..99851c246fcc --- /dev/null +++ b/arch/x86/mm/memory.c @@ -0,0 +1,60 @@ +// SPDX-License-Identifier: GPL-2.0-or-later +#include +#include +#include + +#ifndef CONFIG_HIGHMEM +/* + * folio_zero_user_preemptible(): multi-page clearing variant of folio_zer= o_user(). + * + * Taking inspiration from the common code variant, we split the zeroing in + * three parts: left of the fault, right of the fault, and up to 5 pages + * in the immediate neighbourhood of the target page. + * + * Cleared in that order to keep cache lines of the target region hot. + * + * For gigantic pages, there is no expectation of cache locality so just d= o a + * straight zero. + */ +void folio_zero_user_preemptible(struct folio *folio, unsigned long addr_h= int) +{ + unsigned long base_addr =3D ALIGN_DOWN(addr_hint, folio_size(folio)); + const long fault_idx =3D (addr_hint - base_addr) / PAGE_SIZE; + const struct range pg =3D DEFINE_RANGE(0, folio_nr_pages(folio) - 1); + int width =3D 2; /* pages cleared last on either side */ + struct range r[3]; + int i; + + if (folio_nr_pages(folio) > MAX_ORDER_NR_PAGES) { + clear_pages(page_address(folio_page(folio, 0)), folio_nr_pages(folio)); + goto out; + } + + /* + * Faulting page and its immediate neighbourhood. Cleared at the end to + * ensure it sticks around in the cache. + */ + r[2] =3D DEFINE_RANGE(clamp_t(s64, fault_idx - width, pg.start, pg.end), + clamp_t(s64, fault_idx + width, pg.start, pg.end)); + + /* Region to the left of the fault */ + r[1] =3D DEFINE_RANGE(pg.start, + clamp_t(s64, r[2].start-1, pg.start-1, r[2].start)); + + /* Region to the right of the fault: always valid for the common fault_id= x=3D0 case. */ + r[0] =3D DEFINE_RANGE(clamp_t(s64, r[2].end+1, r[2].end, pg.end+1), + pg.end); + + for (i =3D 0; i <=3D 2; i++) { + int len =3D range_len(&r[i]); + + if (len > 0) + clear_pages(page_address(folio_page(folio, r[i].start)), len); + } + +out: + /* Explicitly invoke cond_resched() to handle any live patching necessary= . */ + cond_resched(); +} + +#endif /* CONFIG_HIGHMEM */ diff --git a/include/linux/mm.h b/include/linux/mm.h index b7f13f087954..b57512da8173 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -4114,6 +4114,7 @@ enum mf_action_page_type { }; =20 #if defined(CONFIG_TRANSPARENT_HUGEPAGE) || defined(CONFIG_HUGETLBFS) +void folio_zero_user_preemptible(struct folio *fio, unsigned long addr_hin= t); void folio_zero_user(struct folio *folio, unsigned long addr_hint); int copy_user_large_folio(struct folio *dst, struct folio *src, unsigned long addr_hint, --=20 2.31.1