From nobody Sun Feb 8 12:14:42 2026 Received: from mx0a-00069f02.pphosted.com (mx0a-00069f02.pphosted.com [205.220.165.32]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C8F8A203706; Thu, 24 Apr 2025 08:18:07 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=205.220.165.32 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1745482689; cv=fail; b=gV2BlyUln7J1eEEIhMYvtOEkh3WLCk02+6dp0r0UckLajMQ6b5YYvt3K8euzbQpEPoWtjs2q7njBnPDIRhy552CgS0xhOmA6R/yKs2WIactveZkCxcQpBFnL6QKuaVFdIG1vl2ijWBH+B/bnm7MNggqpTjCNgsSF6wiEJQX+j00= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1745482689; c=relaxed/simple; bh=OwODhzF+asfaSqN2WiaXeROexOyz+VqoCrSQr4qcvzc=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: Content-Type:MIME-Version; b=MuYXtyGhoIWjMew5Enn9dY7CQoPDFWpS5N9VLkpKzgODSVSmRXeyJJimtSjM8KYnxO6TBu/bptX8276fJWkV/EDRNKbIrsPk+vJ9oSD3GbGXXeuyxt7gEkivDMIDzSjTUSsETrgf1sdKrqu8R+o7MAB2c8w0QZxOqvwLJqAtKmY= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=oracle.com; spf=pass smtp.mailfrom=oracle.com; dkim=pass (2048-bit key) header.d=oracle.com header.i=@oracle.com header.b=WfzEBDEq; dkim=pass (1024-bit key) header.d=oracle.onmicrosoft.com header.i=@oracle.onmicrosoft.com header.b=r4MJRWhK; arc=fail smtp.client-ip=205.220.165.32 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=oracle.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=oracle.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=oracle.com header.i=@oracle.com header.b="WfzEBDEq"; dkim=pass (1024-bit key) header.d=oracle.onmicrosoft.com header.i=@oracle.onmicrosoft.com header.b="r4MJRWhK" Received: from pps.filterd (m0246617.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.18.1.2/8.18.1.2) with ESMTP id 53O6upZ0009041; Thu, 24 Apr 2025 08:17:49 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=cc :content-transfer-encoding:content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to; s= corp-2023-11-20; bh=5Yrwh0GKU9wwDZWTrcd2hh/yg+nh7z44xkDhcXHLrzU=; b= WfzEBDEqhyh/QMWv/tSGrNS/VvQvxxW8WRvGlZtSeMM5EFu1BRP7h4CZLhnEX8bp xhJhxzEPRQGOxMMptfpyL2AYaN7DWA+jauWE1NuGWQKrFBd1WZaQZ3zSp3QW0VFs DeXelX0Uk0XfNoKyY4+cyP0twBN13FiZJCZrMabpJm1Tn5ZdGieph3HeKArA9Cav 2xQ0kGSSAjQw0Z1rBr6KUexB+V5A9J3nXqNz/V17GoHJ1cbMj5hMHsejgco+rLYR b41rTZqX+8Rljm+Neli3ebzHNpPGbm1HwmkOfX2bbhLXhvpEq2CiXfezkC6zKD5X R9LhuKJau41jJrDqMdZhdA== Received: from phxpaimrmta02.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta02.appoci.oracle.com [147.154.114.232]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 467fw2g65p-14 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 24 Apr 2025 08:17:48 +0000 (GMT) Received: from pps.filterd (phxpaimrmta02.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by phxpaimrmta02.imrmtpd1.prodappphxaev1.oraclevcn.com (8.18.1.2/8.18.1.2) with ESMTP id 53O8832c030957; Thu, 24 Apr 2025 08:08:34 GMT Received: from ch5pr02cu005.outbound.protection.outlook.com (mail-northcentralusazlp17012050.outbound.protection.outlook.com [40.93.20.50]) by phxpaimrmta02.imrmtpd1.prodappphxaev1.oraclevcn.com (PPS) with ESMTPS id 466k06xrde-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 24 Apr 2025 08:08:34 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=lVWYE8kMK6r38Z0Rqz/dIiiXPazxguVHP7p0ysd5QA3/w8FDrUmXhbkSzTBT97Mb0BCk8Pv2uydYYEoFcDonHlKDToYPBulhlugBa+VGeIJqqEGYklbXPuxBHz3/OoPIZK0k4KYFyeXSI7hQoyvZlPS1MbR7zWW4hGqkHgSYx5R5lnm4PX+kBoLa/eGpI+iZs5KW1stONABL/ThpOcVdIgW8t+Kv3Oy1BDhv8aEU/DsxXDvB9Z5Q3scRXiaS/m+X5O3xrMXWm/0pQOzcoMXPn4z1B0blhMWAPllKpLcs0s6oxTQbOfQXESL/T2LHMPRanyuO2F++qHUkA8HLRsHZAw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=5Yrwh0GKU9wwDZWTrcd2hh/yg+nh7z44xkDhcXHLrzU=; b=ZmiOuY9H87g8WZAXTVymC4oAhHrByvMrR0W1Gh7U0Wf9efK9/TspXHB+rv8H3VqwiMNpLji5kxKcWexEVeUYSvSb3KnZvzwswLAjjdl9iXyRncktTXEUgBYpDP+UOLu5QplN5wg9p73hMEFAdvuZLY1gwfP/LD2RY+RV/a72qzsamm5m9Qnt7jUBUvagDAHizfWtMph1xCW34MgTNVyit4BUe+rK9IbXIeZqfaUCurQjmcErjXFjezjY81i4nTRZ87nggLQBHD2XoL5w8xus+8xMgrkiC3BPqyBIVBkyYMRyGFOgtYF9PpSesDdhNKyPMd/T76JKOWBiMQxiWcu4Dw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=5Yrwh0GKU9wwDZWTrcd2hh/yg+nh7z44xkDhcXHLrzU=; b=r4MJRWhKMuO+mHeCNQtENJEzXu60uf+SpSBVJDpMjRWxcbzEMvVJAY5xknyLdMT2ubBZIg+Lu00SgsCun4KL1YYXZjmPAy9aCnAa1A30+qWLiG2jZv9WCdZyN56xqDXzOpVwqG++7OXtslvgxd5oJAliBP+3WaBYw5/Dl92xcWU= Received: from CH3PR10MB7329.namprd10.prod.outlook.com (2603:10b6:610:12c::16) by LV3PR10MB8156.namprd10.prod.outlook.com (2603:10b6:408:285::9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8678.23; Thu, 24 Apr 2025 08:08:10 +0000 Received: from CH3PR10MB7329.namprd10.prod.outlook.com ([fe80::f238:6143:104c:da23]) by CH3PR10MB7329.namprd10.prod.outlook.com ([fe80::f238:6143:104c:da23%7]) with mapi id 15.20.8678.021; Thu, 24 Apr 2025 08:08:10 +0000 From: Harry Yoo To: Vlastimil Babka , Christoph Lameter , David Rientjes , Andrew Morton , Dennis Zhou , Tejun Heo , Mateusz Guzik Cc: Jamal Hadi Salim , Cong Wang , Jiri Pirko , Vlad Buslov , Yevgeny Kliteynik , Jan Kara , Byungchul Park , linux-mm@kvack.org, netdev@vger.kernel.org, linux-kernel@vger.kernel.org, Harry Yoo Subject: [RFC PATCH 1/7] mm/slab: refactor freelist shuffle Date: Thu, 24 Apr 2025 17:07:49 +0900 Message-ID: <20250424080755.272925-2-harry.yoo@oracle.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20250424080755.272925-1-harry.yoo@oracle.com> References: <20250424080755.272925-1-harry.yoo@oracle.com> Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: SE2P216CA0032.KORP216.PROD.OUTLOOK.COM (2603:1096:101:116::9) To CH3PR10MB7329.namprd10.prod.outlook.com (2603:10b6:610:12c::16) Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CH3PR10MB7329:EE_|LV3PR10MB8156:EE_ X-MS-Office365-Filtering-Correlation-Id: dcac63c2-63c8-4319-b216-08dd83072767 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|376014|7416014|1800799024|366016; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?ZW85AqIFgf6pjP4eBZjAGEAxzdxP+I0WOTS7TQcFrR3rPf3YeOI7/7aAH4W7?= =?us-ascii?Q?aW9qiXgltU6ME1de5BVoKm5zbEYwVUf+CIP7C6z7KkziVPIZ5aCeKAy0oRdn?= =?us-ascii?Q?m0XSzQP7L3pidXux3eULoYMzLCQSZldbdIvh7USElGN6Suri33ZSbBLbcIu3?= =?us-ascii?Q?mgpjIU18w9B1I/CwPsT902AweCGqAPgTCua/rtvo/lXO1dqqfhtwTu07+jMI?= =?us-ascii?Q?FF5xbxHHQmr3Y7jdonAwMgJzOm+xAg1ZojgmrteFoFhlnmY/awbZkwTZVZtN?= =?us-ascii?Q?TitO5MTGqB0IqPpN5vJ9GFIR9EE8Twr2247GVrKx0fIWSQIc8LastU1hdLEX?= =?us-ascii?Q?x+V4GNytO+ZLC/HBZUzGnR3VCHvkeYsygLruhAixaTIx54jT20ACo8PxBPZl?= =?us-ascii?Q?/bRrEWQz0Zd9cuGC0l/gAv3wvR8hRUxAQWMHF6+AlJQ9J4YLIcy2OTXl862y?= =?us-ascii?Q?mWkNBoT2raY17PCdkn57n5o8Tgdwxun7NeqJL/6NT89eVo8cO6K7plt9xGzJ?= =?us-ascii?Q?0Cro68ycqnuQ1DHrBgpoyPek6/BZNC5LaiHCE/MNdMpmn/g4D+fTUZjxukq8?= =?us-ascii?Q?q1u7uQXrMlgjnHgLnvqLqstTkmA9gz0ZOt/t5D7941OE04Ijd9h4TIUP1POR?= =?us-ascii?Q?0fsFL/2X5y2g2F0ivKN1zJmU2PHEMnKoUOaE/dni5IrZn53zg2g9YqGwJOSG?= =?us-ascii?Q?aRCG9xDINvkbXo9Hb8aLwdZ/Qt+w+FGt0Rs7BFIlduX5lO83m9AeF8FZktfe?= =?us-ascii?Q?+TDzQCgNu/wMtI45kq7GUQzEFX3FvYRl8T47+m3HNgAv1vu5afHPVm6AGuIf?= =?us-ascii?Q?ipLHtW5H0c4itwZqamrYy+8pCuNV8o+4MV5k5ZYccichfjs+ZfCVfrC1ObOV?= =?us-ascii?Q?5t23Cg+prdhYIGcbJ+YpZE+SQYWwrsbp+3MhaiI9QZNw0UAeS/KeLE5eiWnX?= =?us-ascii?Q?oInRX98sXq7/xt31gWXUfq5UPC4vfiFbXSMtSoNkg0P+rNWoqvdmfoqBuD3A?= =?us-ascii?Q?eMTsSm/mHhERm/nEc/u7Gb/RoGo10PNiAaVILPMByHT8NVWAlq26ghMQZbKl?= =?us-ascii?Q?3WdpmUvc3DgFLSGHv97EN+wkXevQbvyfrsX2BaFDKTliig95+UyETz1Fv1yH?= =?us-ascii?Q?k0eZPjEA5C1B734MzNFFqB+MW0iZ+yas/rwt1l78l80ypO9bNiGpkAZvbxvO?= =?us-ascii?Q?lAdvf2oHgQ5jw43pJ2E3+6vTRJuP+vdmbe+LDnlqeLFWnpxOxdsIR2biGQah?= =?us-ascii?Q?rBjBRP2hNxkyfdt4VhDPAq5ib8a36ghkLt/PVHNgTnac7mlfSef+DlHl3cuk?= =?us-ascii?Q?LdPlo7i4IrNYXht8DdKcth7rCKJB8/Iz+7seLQ7qbt6UHoOsSqtxYXecDz55?= =?us-ascii?Q?5z380ewKB4l5tKJKriATWuyvtrNSz6m7NUJhfXyVtn8OrcyJ0Ljm/G5mrSI4?= =?us-ascii?Q?7iB4cnk5vPY=3D?= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:CH3PR10MB7329.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(376014)(7416014)(1800799024)(366016);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?ZbfMTB5WPGibyN5A5HniRlCyZ4AhUFG2HCbjvvpHRCp8mVhaif1EfHjvviFd?= =?us-ascii?Q?O2uM/AIr87lMAxsVzuImyQFFe2f/RxNDnIhdVeFFhECCyoZ/1Vi94FMKB84t?= =?us-ascii?Q?nXYgfRk4Ti9C5RI6e71Pvr1uhxBQvSbHBAdxua6wy33Dx7Qgmlmw3TzhdKv/?= =?us-ascii?Q?tX8Cbek24d4lCSMD+IA4WKAlVTCzTASfp20bxiRzdF50PZcXaIB32oI8K7Z9?= =?us-ascii?Q?3JAp05IfZyOkb68yGftfZF8nwEYQZny5Hf3+C4+UguvVQ70U5IhfEsr7X4SA?= =?us-ascii?Q?PC46tKXdUQP/uWAGZioCH5CiQy8kehrqt7dY/ZqNx032NnYDCEJvFtLfsDaL?= =?us-ascii?Q?Ure55sUWZha3sd0GsXxdNShl5F9fSa9x6zPhnGBjsggWHkOShx3H0AXrnKB1?= =?us-ascii?Q?/OmI4GM1kIt43WYymdPwCfRU0EWVxufEMwBt9vO4s/acnbqkjsZ0I1oQ7lv9?= =?us-ascii?Q?pQyVHfyXavjL7cxOrotoOiGH6Y3orSeLrgXXP/uN7Vuv2kQV6n3uOdCauhtr?= =?us-ascii?Q?EzH3r59AyM2cnIiLS0wFuUYlBS608I4MuObfq7VTab9kTm0pRjxYA8GS6zqf?= =?us-ascii?Q?cxjFZvqGyTzHsOuex9GsRB6XUMKsl7Qx+iYhidYTO9AKKYITBABRerEY1VWx?= =?us-ascii?Q?GYJ5NljxSkcdY1jzP91kFvRWJdHAq+eGA03hNwp2q9WViN9jfWmu/fgnIf8S?= =?us-ascii?Q?fWvG9TyYH21VtQFJ3t1TPUWrE7CUpVKDoIDszkF+1l3k7JMg8krgqMDty74u?= =?us-ascii?Q?Zp9Kd4SrqLnDuGC3NeYVxG6o58WYyKWUWg+cuMmtSzgtgDrnKbDSF63+Hi9c?= =?us-ascii?Q?iyuj/U0WLALyPJUI2CgtUnsoaYUtcnn9jzWOpqxxNkhhbe6TwDwTnsAFEjoa?= =?us-ascii?Q?pImJjtBEXYvLjyKGip0JqzcIXTnwLXCogYLmgIyspfmc6kVfKXmsd5QhnpIp?= =?us-ascii?Q?6Vlw0K4nqMtzGuHklLYMjar2VaL7BpN38J7R4QclNz2yyfyqaiGHFil14Y82?= =?us-ascii?Q?/eKxXzaaW4CmMd427Z/lZZhNi4c2bh+GjwiabvsbTTtrNzJIKF9w+6iZkjAt?= =?us-ascii?Q?JNis9CV9BHeqfv/FBoTLEwo5iwLOog6gkVT21Kn3e8HSdZrstB5P7sfxbZyH?= =?us-ascii?Q?Dp8My+q5fqf6yy62O6iRpCrytozRYQP2yyPAM5n9bud+xJYTUBcaB43PHc8/?= =?us-ascii?Q?JkXd/xX22wxwmiJXwCB4fp+JET8YI867aNICEHlcc/5CO/k6dwD7JSNNowHa?= =?us-ascii?Q?vmImrp63WyYTvO4D0MBVoJunpYDN38l4qon9cemgwQrxy1NBMO769JsJbDah?= =?us-ascii?Q?O9nzaYL8Nz/iY7uwpDWdOHHmICTqV1wknb+QqFrYblEB7u1VDGC48ggtMWaz?= =?us-ascii?Q?0y8OYGpCxEvxUpx40ikhlcUJwS65AYgiyEcQVGj6yZUCLNzThUgRNIoeT98d?= =?us-ascii?Q?aMoT/xv62omCgfKRWBsbcz2svPaYFbRlbvE3v2L8Iq0i59y3TNEU4HfcCKmd?= =?us-ascii?Q?K+SZ0fOH2xxVB/ze5kZAzztOkPZRtOnEoAM0GINNMheXlk9AqDgEOfXGoD9c?= =?us-ascii?Q?BlwRmHuFWO2OWCHSYF8nVvkq66KJU7YMlDFjpzO+?= X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: HzqiUNidfw/Wd91rCXleJoIbrOKBEPeYaApDuwBifmaXM3ntzZhky2P6eUS+GPRhLa5Q5vDnTLyatsvjbl5H/mgAIrH7GpMU1KDh+QgH2uPk+AYOpj9LW4/MfPcUmBgpT2aPk8CnZARJWqz9PyHw7NUYFLZv/8MtsPpTS5PzUY9d345Hx3V0laxChwkikHcBMg+7d3zvSWWIFNgylyXj6nZCKizl9Ml4FmSfogmllFZ+5UoEc6xbgUPjETQfIqjH5D3QrQueM3fDpTuDKi4HjJKNbZs49Z4bYGbVMbOtkvoE/C18sM8dcbDni/4Be+nA+08Yvk3G4y1pgJ5dVNJQp8dlnZldJVFHB3HGHrZ+YhDNvWRA+Frpw48VQILhZMuEb2nQbFW/TZyA2C2u/FLuE5epi7YXtxa62YpKbHOhQSF3wMqGSb1fYzD4DJG4978lyd8UICxN3oIaLlbCZYG+w7AXt642U9UH++fDoXiF4c2q2TiEqsuosZ29CBRr/LxTSoM7ozum+7c+8cOYfjI/q32+f+VknS+aM1v6Nqwww1MDVyV2R1H9quwS4RPLUofpIvjFlUOvd9pzv0tn3CCgoTbfB5g4Hfq47xwhRwTlFZ0= X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: dcac63c2-63c8-4319-b216-08dd83072767 X-MS-Exchange-CrossTenant-AuthSource: CH3PR10MB7329.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 24 Apr 2025 08:08:10.8184 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: ulBd3QA/UiMPlVIqCrgk9Vvm+FwksXAjhX3hZBg9+DJn7ubGVlH6V/ySCOs0waUwx2i5LZbm7wsjlTwVjy0EGg== X-MS-Exchange-Transport-CrossTenantHeadersStamped: LV3PR10MB8156 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1099,Hydra:6.0.680,FMLib:17.12.80.40 definitions=2025-04-24_04,2025-04-22_01,2025-02-21_01 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 phishscore=0 bulkscore=0 mlxscore=0 spamscore=0 malwarescore=0 mlxlogscore=999 suspectscore=0 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2504070000 definitions=main-2504240053 X-Proofpoint-GUID: sHpwQcPhtBb7vzqUiYIfLq6dI8xa-RFx X-Proofpoint-ORIG-GUID: sHpwQcPhtBb7vzqUiYIfLq6dI8xa-RFx X-Proofpoint-Spam-Details-Enc: AW1haW4tMjUwNDI0MDA1NCBTYWx0ZWRfX17C0lQO6SPPL Lc/TutA5z4h7okrIO9F/fzuNoin1//QiFc8Chzbox2u7TYWA5zVLdxGeihkDMaRdC10fyrT/oQb sK3ySiDBLCo07HklIbhQQSaGctgU/ZriAEC2HBFHRwJDFeV6vLIet8PHuXKSsssjn+aCrLCrj9+ 4wmxC3sW3klcL6kpnzRBsus7Rm+7xWplRshmF05PvASBoDfyCzHdzD3nhaJskX7D4I0NpATfV4n ze4VM+BOCR4Y+6WD02UMKjlV8Fc+PAivmBywMTWcLZWU5gesLujZ1MKYov0pJGznLptA8rOEga3 xc/ZfdvtP/8+QDiVAYXrfHUfvH5x0TmqQngx6NoukR5yAWKN1UoWC2Q1ax+CLNkFoHcXq2e8aYP DfAcC8WK Content-Type: text/plain; charset="utf-8" shuffle_freelist() function returns false when 1) it can't shuffle the freelist due to lack of a random sequence or, 2) when there is a single object. In a later patch, I'd like to return an error in shuffle_freelist() when setup_object() fails. But with current code, it'll be hard to determine whether it should initialize the objects anyway or let allocate_slab() return an error. To address this, decouple the shuffle eligibility checks into should_shuffle_freelist() function and call it before shuffle_freelist(). Change the return type of shuffle_freelist() to void for now. Signed-off-by: Harry Yoo --- mm/slub.c | 26 +++++++++++++++----------- 1 file changed, 15 insertions(+), 11 deletions(-) diff --git a/mm/slub.c b/mm/slub.c index dac149df1be1..95a9f04b5904 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -2534,17 +2534,21 @@ static void *next_freelist_entry(struct kmem_cache = *s, return (char *)start + idx; } =20 +static bool should_shuffle_freelist(struct kmem_cache *s, struct slab *sla= b) +{ + if (slab->objects < 2 || !s->random_seq) + return false; + return true; +} + /* Shuffle the single linked freelist based on a random pre-computed seque= nce */ -static bool shuffle_freelist(struct kmem_cache *s, struct slab *slab) +static void shuffle_freelist(struct kmem_cache *s, struct slab *slab) { void *start; void *cur; void *next; unsigned long idx, pos, page_limit, freelist_count; =20 - if (slab->objects < 2 || !s->random_seq) - return false; - freelist_count =3D oo_objects(s->oo); pos =3D get_random_u32_below(freelist_count); =20 @@ -2564,8 +2568,6 @@ static bool shuffle_freelist(struct kmem_cache *s, st= ruct slab *slab) cur =3D next; } set_freepointer(s, cur, NULL); - - return true; } #else static inline int init_cache_random_seq(struct kmem_cache *s) @@ -2573,10 +2575,13 @@ static inline int init_cache_random_seq(struct kmem= _cache *s) return 0; } static inline void init_freelist_randomization(void) { } -static inline bool shuffle_freelist(struct kmem_cache *s, struct slab *sla= b) +static inline bool should_shuffle_freelist(struct kmem_cache *s, + struct slab *slab) { return false; } +static inline void shuffle_freelist(struct kmem_cache *s, struct slab *sla= b) +{ } #endif /* CONFIG_SLAB_FREELIST_RANDOM */ =20 static __always_inline void account_slab(struct slab *slab, int order, @@ -2606,7 +2611,6 @@ static struct slab *allocate_slab(struct kmem_cache *= s, gfp_t flags, int node) gfp_t alloc_gfp; void *start, *p, *next; int idx; - bool shuffle; =20 flags &=3D gfp_allowed_mask; =20 @@ -2648,9 +2652,9 @@ static struct slab *allocate_slab(struct kmem_cache *= s, gfp_t flags, int node) =20 setup_slab_debug(s, slab, start); =20 - shuffle =3D shuffle_freelist(s, slab); - - if (!shuffle) { + if (should_shuffle_freelist(s, slab)) { + shuffle_freelist(s, slab); + } else { start =3D fixup_red_left(s, start); start =3D setup_object(s, start); slab->freelist =3D start; --=20 2.43.0 From nobody Sun Feb 8 12:14:42 2026 Received: from mx0b-00069f02.pphosted.com (mx0b-00069f02.pphosted.com [205.220.177.32]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0456D20ADD6; Thu, 24 Apr 2025 08:17:50 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=205.220.177.32 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1745482674; cv=fail; b=uHx4vb+rci7QV5EnCX1x2dQdZImF9rnZ3io4bgsskCJVTFbKLjHuu98RLRY1osW0qjTTCFj60bQ/QzGdvegqVcGXlgxim/jbWNWLH2WIeAneEoPZZOs1qS/b5xwweBN0d+OO/J1XSEa5m/b0pCX4qcLQKcxDb7f559fEGzM4iB4= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1745482674; c=relaxed/simple; bh=jISeOmo4VG4oEnl/prP0K9oZO06XbRY5I7WDfhjFmUA=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: Content-Type:MIME-Version; b=duvMJ2VNE1Gw/CtTWTTx/Xj5hhj4Ne7uY4EZd9bf7uFHqHtNgb1hfoKpO0hpJPzw2oC4HaguHzGTwnU4Rg7t9bFmsYqkiuoYbbwI+Y6OB74DpoEK3+fCcJAgYTGeHJ0RwgxGUYGGsKjawXdNRLkJkYIHMvar3VatiC6R32E5XJg= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=oracle.com; spf=pass smtp.mailfrom=oracle.com; dkim=pass (2048-bit key) header.d=oracle.com header.i=@oracle.com header.b=hYbU/Hhc; dkim=pass (1024-bit key) header.d=oracle.onmicrosoft.com header.i=@oracle.onmicrosoft.com header.b=O0SkN7mD; arc=fail smtp.client-ip=205.220.177.32 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=oracle.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=oracle.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=oracle.com header.i=@oracle.com header.b="hYbU/Hhc"; dkim=pass (1024-bit key) header.d=oracle.onmicrosoft.com header.i=@oracle.onmicrosoft.com header.b="O0SkN7mD" Received: from pps.filterd (m0333520.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.18.1.2/8.18.1.2) with ESMTP id 53O6ttQW005503; Thu, 24 Apr 2025 08:17:30 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=cc :content-transfer-encoding:content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to; s= corp-2023-11-20; bh=UHXVQICH9CKjOW2c0wTbDGV4qijQ2xd9nAtxR1Xijsk=; b= hYbU/HhcCFVDg6SCmQHmySoakvg2sz84XzDmIGqkYu0tWRCOBtyxDmWncoVznQKp WUwsbbTYAcMEQfxMJhi/JAaZX62l47RCftr7KvLQtvmICLcwivOvWQ8AvNTpUv8Y gQLcGQhDk1EPTFcmKeFfLmd/INfGZQJKp4CnM5RAUU8MeWWnTvQwQFdKNnRZ6qpF 65DnYufk5bmNhkoOAsIuTxfLthHrFsWjBRyZaMfIt+yXq6bhxNz7X5a8lKGR6djY KQVNKHAWgmvSU+f6sTRWK18wXp4DVBf308Q97PbuaznQ7U1/KDNEwZ49Q2jBG23X o6X73jztCLzyP7ZNS+4zrw== Received: from phxpaimrmta02.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta02.appoci.oracle.com [147.154.114.232]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 467fmk86pj-13 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 24 Apr 2025 08:17:29 +0000 (GMT) Received: from pps.filterd (phxpaimrmta02.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by phxpaimrmta02.imrmtpd1.prodappphxaev1.oraclevcn.com (8.18.1.2/8.18.1.2) with ESMTP id 53O8832e030957; Thu, 24 Apr 2025 08:08:35 GMT Received: from ch5pr02cu005.outbound.protection.outlook.com (mail-northcentralusazlp17012050.outbound.protection.outlook.com [40.93.20.50]) by phxpaimrmta02.imrmtpd1.prodappphxaev1.oraclevcn.com (PPS) with ESMTPS id 466k06xrde-2 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 24 Apr 2025 08:08:35 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=gQ477F5hLcVpGZwXr9vhizc1HypOKgDqNXtmsMzcQUF0t87YkGOGTTgD1r35PhLA7zOQV/7eJhu3hDdNWR+b7KR1jUHXtl/W0hCmXT3sUqq4JsKmoBqmXug0om7xFrCULk4UwK71MM89B/UDjOD5U26T7fq3IiKKGmxncUHvF0xSUQTvv41HpdaZOyQlYRqo5ZjupRI5FIYQTcWv3f5VdhFiF18DYLg+81sTBvppd+I6rIw4IW6pexmsUxQ+QXE48dVklGIFyjavEKMRL1WhnAmt+ZpgVttR4GI3BFg5MCkFjhMCWl05h/6siN/lE6p7wDayBtwY032BlCFU/ss3dQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=UHXVQICH9CKjOW2c0wTbDGV4qijQ2xd9nAtxR1Xijsk=; b=Nj4H42VMUK/1kWSeoNVW68MnylvmcuBy8rCFFgEmolOpnkbXWDqp3Z28s4hlkNLtgS7k0oBDBgjsnQXOQSDSVFFoqps9N1qN0A27imNh/7FKHRcZCIn8Uu/4Pfyca5b3RlnX4DQr1FR7TSOXnW7f67vk/jm9cl3dN8lhlifvq3pcl8mrOHozacmwYn3GMOJnbyAkYYOa3HMhDAT2X3PmwdgHP4fo1XyLWccULjOmcRyjCrvVmR8T1vCj9QELtGIprGZDIfUCZsykDaoYt/MxMme6lDBCdUpcAQT5gYScaWb0qBXNcUbTPwyzzV5X5JVDGcOHh/JrNRaSQjpgVMR+mg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=UHXVQICH9CKjOW2c0wTbDGV4qijQ2xd9nAtxR1Xijsk=; b=O0SkN7mDETOEZkKMQOMUzVbuSVT1vrUjZUW+2deAn2Cdxb6KoQvR1GheNZPALaRUhcnRZuFcw+QE291KwGaoqxl+ctKEl9Hi5G+L2ZYHsm2lq6r2Ars1YgTKEKG+wG6XMkKd1M8u6Ebdd0HEEmcxVjqPgjnqG6Z68TWpjtLhtuQ= Received: from CH3PR10MB7329.namprd10.prod.outlook.com (2603:10b6:610:12c::16) by LV3PR10MB8156.namprd10.prod.outlook.com (2603:10b6:408:285::9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8678.23; Thu, 24 Apr 2025 08:08:16 +0000 Received: from CH3PR10MB7329.namprd10.prod.outlook.com ([fe80::f238:6143:104c:da23]) by CH3PR10MB7329.namprd10.prod.outlook.com ([fe80::f238:6143:104c:da23%7]) with mapi id 15.20.8678.021; Thu, 24 Apr 2025 08:08:15 +0000 From: Harry Yoo To: Vlastimil Babka , Christoph Lameter , David Rientjes , Andrew Morton , Dennis Zhou , Tejun Heo , Mateusz Guzik Cc: Jamal Hadi Salim , Cong Wang , Jiri Pirko , Vlad Buslov , Yevgeny Kliteynik , Jan Kara , Byungchul Park , linux-mm@kvack.org, netdev@vger.kernel.org, linux-kernel@vger.kernel.org, Harry Yoo Subject: [RFC PATCH 2/7] treewide, slab: allow slab constructor to return an error Date: Thu, 24 Apr 2025 17:07:50 +0900 Message-ID: <20250424080755.272925-3-harry.yoo@oracle.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20250424080755.272925-1-harry.yoo@oracle.com> References: <20250424080755.272925-1-harry.yoo@oracle.com> Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: SE2P216CA0043.KORP216.PROD.OUTLOOK.COM (2603:1096:101:116::14) To CH3PR10MB7329.namprd10.prod.outlook.com (2603:10b6:610:12c::16) Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CH3PR10MB7329:EE_|LV3PR10MB8156:EE_ X-MS-Office365-Filtering-Correlation-Id: 2911013c-3806-44e4-8781-08dd83072a65 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|376014|7416014|1800799024|366016; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?jAwt8NVV6cLVjtazlqAqY1kOnUs7wjtMtDQgBEicwD9Kd9pxGUhbzjuk6Nk7?= =?us-ascii?Q?kEJVSQPHjC4unI5Xj1TZzvVhz5QEVP8lQaqo9euPnzlwC/Ht6q/dObWGHSMk?= =?us-ascii?Q?XjdCuPyidEKy3hLEElb90nftfREu6sf71AXq8Wf5vScZP68gJQ95lWJ/yKCJ?= =?us-ascii?Q?iS6JLeCTxd3bo7rMad363pIrd1fk9lYtPRbuSR9csALZiHeMNGzmAc0HnF9J?= =?us-ascii?Q?9wpukegY7+8FLqmU3kgaNQdjGGYQfDWq0FaNVDkRft60CicYSqMODuYYDU9R?= =?us-ascii?Q?bVn80E37Ir3H8smW1p7zvYf8uz6HV6pYzYRNQiSdL5s1uIUG0F5ni/GgJLCi?= =?us-ascii?Q?SjN8GeEyW94rYw80cnZ/NsTDk5l9UCivTG0RMcybGZTXACP7h9FVV6lymi60?= =?us-ascii?Q?ulOUYYRZQwsG2L2LGDbArUxb3/a/7qmH1YJl1sKKdxBuxAVEUJreRl/QOE0X?= =?us-ascii?Q?kl1oLpHYaO2ScVdEDvsrEPemRxaEqR0NZSYgsviMOxZP7zMBjHjMJCxL63rQ?= =?us-ascii?Q?wGE7DnFe5pfTa8lfIWQhpCwHJg/Fvf38CP14x6H1E9EfzE5yeJ0FcellivUW?= =?us-ascii?Q?AWpb96tmGuaE00/GWP/vTCxVNpATkS6gQgzkEDaOkKexyFyIOEyYSBXgDRvl?= =?us-ascii?Q?CAH6vMcPO1bGcQWMFxnfa6yTaaqWBb2XzkhG7IjLpGoJixybpuXusbe7wI9l?= =?us-ascii?Q?0b/H9HYQ6gKNHxc+OKGG3GwP7A3SbvILSXc0cimTF/nhnb6ZTcgwY+hiFFtK?= =?us-ascii?Q?0sRQn9OEZrJ5xizpM5eciz4B1YIhJ1LTvkZ8JIKTb9kSkJPrjIz4mvG3MGSr?= =?us-ascii?Q?F6cM20v3ENTZoFOQIOUNqBnLDVYl+vp2epbPdzAZPfbIcBMmu8WiQpLHVBlx?= =?us-ascii?Q?sdzzXVQC2hMBXk5aa4RzElihja4iYsSabM2du9MRFn0bM22esBYc5IAydaBU?= =?us-ascii?Q?xAkMCWzBFPekEiyHG5NXu8wlZW6/Qjiw4eyZlNpBv52o/jwO94+UWu0q8CAA?= =?us-ascii?Q?4JaKWhaGdz6J/dCn27vYmoz+1lvgUH5xRf8dR7B2k8etMxf46ctfJtyvaH61?= =?us-ascii?Q?ibHvf2KiY2yO6jzpSu5DdlIX+oWUw53b5Zwh9g7hLQOdfS0B3NhLVWtTB87s?= =?us-ascii?Q?Mw6v5SOsNsLSmGfV4JZ4VKcy3JrKiq31zXwoa4lfrrBB2pDiCwynEQELXRY7?= =?us-ascii?Q?ELk9PQ4mjqtnVtrDc548E5OM1jVaw/W4GAri6D6al8nAZVHmKN/gjOGvKxxQ?= =?us-ascii?Q?zZLnat8HOvELCb5NUUJWaVXSDDZu4cLxSkUa2dU5fcayW0ucdhGjzBGOiG3Q?= =?us-ascii?Q?DDX1c2FBu1I1zU1x0rAdq91ZthW4sIvXvSspXxFODgM1+YsLjv7XqaMj3CAE?= =?us-ascii?Q?mV3HHdXQDgtwD1SESwUxcd1TMUEXPPVfbe1hyuzD5l/GQOWxL3PIahu36JRy?= =?us-ascii?Q?gzL2nYHAGas=3D?= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:CH3PR10MB7329.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(376014)(7416014)(1800799024)(366016);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?a1nE71jcw8CneZIni8Xhk115i/Q/NiwjJsO815K4/MjsJj5gE1Cto7iScwf/?= =?us-ascii?Q?0E6+SBku/rpDmWi7Nk6fQOmo2MwduzxlmoOOXfQl10d+SYE+bSeF6RSROUEs?= =?us-ascii?Q?J4ctg3kVIzsF5+aEBhJBChx+GUgFg/g5/7AjU/JOJ1tQyUg4CFnEWwhwz9Ox?= =?us-ascii?Q?Ni2RvlJq5GF2OFrylXKBCnSgfIF5ES1jMDIzTJfjTtPYxqYoiEJiPtrliXWZ?= =?us-ascii?Q?ct1HMEsIstfLytYG1h7+GT8Q8aodBVHq35V4SHZ9aCHF2uu/Q/JwDJZngNcc?= =?us-ascii?Q?SHvUQhfHRL1oxS/zZcnb75fkOMHeVzBexhnHWJ/TkbzT3J7QNIz8gWLsESW5?= =?us-ascii?Q?N2SEVdyqNrGB7C17fVSOGRoalNz7NYHw4cMcZpCG6fwUuiq+97jiTmHuo6xQ?= =?us-ascii?Q?rqdvs7zHP8gmXYXccczA5S6R1atLeafTVetqAtAGdMRvdu5pjw7xmBOEQPC5?= =?us-ascii?Q?FV042PmCwjkG4Z15S3JG+Car0fX57iwwMQHqbW3tj5la40Fm2tIzULYVM6a1?= =?us-ascii?Q?Oa+hfH/DkVLeAZM+HaUbnAgjdtXx56irHNTaBihRqRvpr/DyYDGvDJP1GM5a?= =?us-ascii?Q?ph70hevpx4tm+O7+5C+l6I3dCQwCWokIGgQnrXiAEu+mjVje4Z/ku2RrsyvZ?= =?us-ascii?Q?n6AMFci6H43Hard/ZFpNHSwQhwmY7S2W8hTIDGPLGiV1BY0BpB3rOv4WxwQZ?= =?us-ascii?Q?6jVtc2Txbg+RuxQbvHG6BGUy4rsK/UzCdojFd6nr5QCaIhlapXyYHpSIPIOU?= =?us-ascii?Q?8yw0aiBYUpHfdFHlKLpr9bqniFXPMp0W8xHuo8T/Geuqfmu0NWnbc57Axuek?= =?us-ascii?Q?9ZuipxfClrVGIAfmvq1IZc8M/wf6q9I0Fsm7akG2jOH5V0e5Wjb4mczl2Wcd?= =?us-ascii?Q?00hRvpBPc80xOVu3Ptlu1cT9ZR1aM2KoHUrJDKAkda5E89DAMHeYrFFabmtD?= =?us-ascii?Q?pQ8ffYcIsu03h2zPf5DenFZ/8S9InpdGZ5pKhv3d/fWx5KYe0P1buNQc6XxV?= =?us-ascii?Q?pQdhqJfdeZaq80mBmq6DqgsI5NC11y5P3rNUJ+nVULB/QB+zws4O6AMyCtRn?= =?us-ascii?Q?bD2QpM37sWwbsP5iIyoGuDecdFVQi4nUyeoKOP7HAycGwBXJas0vwJu7s9PS?= =?us-ascii?Q?tTXdujMoXBQuGAfxCUz8Z36zOyXm1er/aHKPSevA65mDdOkvnN3S/UIAvimN?= =?us-ascii?Q?qK98EG6HVvDl2HsoTcvFG119d7LiTEwELUNMglo2gr9ZCXYto0qlSXLQtls8?= =?us-ascii?Q?D/sCqc3jr4y/pwvqDHfb2fTn4is5roqnoCb8o0TaWeFf4+JYS/Lygq2sxzJA?= =?us-ascii?Q?GtEqR3ZDznCmXOtfKKo9AX7qxL00As3Kk6WUmDsVJ6ausz2oVBdvbf230LhB?= =?us-ascii?Q?6Yxy0Ml4Itv3Aleqlrw7+KdV2KtHRX0rDiajd6LihLReszz3bpxBXq+pKQGs?= =?us-ascii?Q?21flOJgdplz86MwFnivy0PPauF1VHvECLpa/R5hc4/uJjOsJklyU7wQDzFxp?= =?us-ascii?Q?hh8Rksl4I8RlUCKyES2+sJfxu3UdwkzAx/PfBEsXsJyXADMo6IjWkyM9+wTm?= =?us-ascii?Q?FmpzCCKwLhgzV+MqXlg3bAIoAWmZX8vYdcfYKGH3?= X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: MOrZeyTrShSLKDCE5Xl0maLpDMf/tDqxRRjO+Gyhq6GFGi3+xOj3d4YIpbme8NiGfuMaf6WAYHiq8vV2KuAFJGLrOxJILYejYUzFBF0D/jYXZ/gopg+UC0DpXnASeczucFwgG2+Or3oZOfYBXeLsu78Fnm7sUnETTmNzRamTvgW9LDbzLnvfI2eT/o5hLTuElng8FmSS/ehVZNRxob52D0Jc21U67cGqNz1zuliOCa7OFnTMWwI4S7BRrYgp5reGfYuXSdSNkUl48UD7Ym3fESFp2iiWkxuNZRkloOeL79vGTHc89l3VH9U/MPFtU77PjPplyegPigHC0BMlJXY5HeQrDapB+gAInhuCWi2NoodRTJ8SKpNnPbarD0P76HOvN8jMjbYWV1nRj6GC+T3QTUasBPtZmI2926o1rQsdhYD00sRU89Xnl0cakaJ/OqoAQEr3bkH/DtInP+cD+rPwy6fZeKQHnWfZEaUmjMBDUaTGMnwhHdhlj8Oqljsx6RspvYXv8+OMc6M1YBsRLBqOSohXiF/rksvM429+aTi/94oE+fwoceYD7DW/JkSwpQFWjeiWpgFRf0gBQK74Y+2fjTfUmSWFHHjH1Is0ZArWJQg= X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 2911013c-3806-44e4-8781-08dd83072a65 X-MS-Exchange-CrossTenant-AuthSource: CH3PR10MB7329.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 24 Apr 2025 08:08:15.8928 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: 2Fx6ksrV9iIMhu3U6NmFGYmklF+xshzzCoR/x1HaKrPiivANE8hTLvJQBSovWV6HsuvUwQLaVbUb3/SdjnuPLw== X-MS-Exchange-Transport-CrossTenantHeadersStamped: LV3PR10MB8156 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1099,Hydra:6.0.680,FMLib:17.12.80.40 definitions=2025-04-24_04,2025-04-22_01,2025-02-21_01 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 phishscore=0 bulkscore=0 mlxscore=0 spamscore=0 malwarescore=0 mlxlogscore=999 suspectscore=0 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2504070000 definitions=main-2504240053 X-Proofpoint-GUID: OrV1V0ZkBouvSwZOt9AxxYegwrLuqCSH X-Proofpoint-ORIG-GUID: OrV1V0ZkBouvSwZOt9AxxYegwrLuqCSH X-Proofpoint-Spam-Details-Enc: AW1haW4tMjUwNDI0MDA1MyBTYWx0ZWRfXxmc7+8mB4iYY ic5HQm1eKDPVI35eO0MYgKZn5UfqyRd1jQsMFwyxPNvJLaQS/iPitXYc19+9JBkDkx8MWWBUxic Ag5TU6H9Ox3pJEZ1tdZSy9IXbdFG2p3G+fBpf21qzRQ0hPGf76qWUZPeuRf4pjqYVE7U3GXLC8h ExCshZsGlpwvSfT3DP7AE5Anj7yZDRJgkf6/CyM7hLStjaJ/Notfzdahkhdknj34WrPADXZd2qV tZtl3ZRyXe502cw7JHr+yJyxCifYAb78VgNmeWJJDYSg6HI2XRKQAJ23W/s1TJ5q2aTou3JyZsi PFny3EidpUqznqlQnQgzN9cU1+STIQGSNuXDeHn2EVNsgxVEr3TaL37MhVxRfwb9BNUm72Ud8YA tIVbj9Rc Content-Type: text/plain; charset="utf-8" From the beginning of slab and until now, slab allocator has not allowed constructors to fail. It made sense because operations like spinlock initialization, list initialization, and memset() do not fail. However, Mateusz Guzik explains [1] that allocating and freeing percpu memory for each slab object's lifetime suffers from the global serialization point of the percpu allocator. That said, allocating & freeing percpu memory in ctor/dtor pair can significantly reduce the contention. As a first step to that, allow constructors to fail and update all users of the constructor feature to return zero (no error). When a constructor fails, allocate_slab() returns an error. [1] https://lore.kernel.org/linux-mm/CAGudoHFc+Km-3usiy4Wdm1JkM+YjCgD9A8dDK= Q06pZP070f1ig@mail.gmail.com Signed-off-by: Harry Yoo --- arch/powerpc/include/asm/svm.h | 2 +- arch/powerpc/kvm/book3s_64_mmu_radix.c | 3 +- arch/powerpc/mm/init-common.c | 3 +- arch/powerpc/platforms/cell/spufs/inode.c | 3 +- arch/powerpc/platforms/pseries/setup.c | 2 +- arch/powerpc/platforms/pseries/svm.c | 4 +- arch/sh/mm/pgtable.c | 3 +- arch/sparc/mm/tsb.c | 8 ++- block/bdev.c | 3 +- drivers/dax/super.c | 3 +- drivers/gpu/drm/i915/i915_request.c | 3 +- drivers/misc/lkdtm/heap.c | 12 ++-- drivers/usb/mon/mon_text.c | 5 +- fs/9p/v9fs.c | 3 +- fs/adfs/super.c | 3 +- fs/affs/super.c | 3 +- fs/afs/super.c | 5 +- fs/befs/linuxvfs.c | 3 +- fs/bfs/inode.c | 3 +- fs/btrfs/inode.c | 3 +- fs/ceph/super.c | 3 +- fs/coda/inode.c | 3 +- fs/debugfs/inode.c | 3 +- fs/dlm/lowcomms.c | 3 +- fs/ecryptfs/main.c | 5 +- fs/efs/super.c | 3 +- fs/erofs/super.c | 3 +- fs/exfat/cache.c | 3 +- fs/exfat/super.c | 3 +- fs/ext2/super.c | 3 +- fs/ext4/super.c | 3 +- fs/fat/cache.c | 3 +- fs/fat/inode.c | 3 +- fs/fuse/inode.c | 3 +- fs/gfs2/main.c | 9 ++- fs/hfs/super.c | 3 +- fs/hfsplus/super.c | 3 +- fs/hpfs/super.c | 3 +- fs/hugetlbfs/inode.c | 3 +- fs/inode.c | 3 +- fs/isofs/inode.c | 3 +- fs/jffs2/super.c | 3 +- fs/jfs/super.c | 3 +- fs/minix/inode.c | 3 +- fs/nfs/inode.c | 3 +- fs/nfs/nfs42xattr.c | 3 +- fs/nilfs2/super.c | 6 +- fs/ntfs3/super.c | 3 +- fs/ocfs2/dlmfs/dlmfs.c | 3 +- fs/ocfs2/super.c | 3 +- fs/openpromfs/inode.c | 3 +- fs/orangefs/super.c | 3 +- fs/overlayfs/super.c | 3 +- fs/pidfs.c | 3 +- fs/proc/inode.c | 3 +- fs/qnx4/inode.c | 3 +- fs/qnx6/inode.c | 3 +- fs/romfs/super.c | 3 +- fs/smb/client/cifsfs.c | 3 +- fs/squashfs/super.c | 3 +- fs/tracefs/inode.c | 3 +- fs/ubifs/super.c | 3 +- fs/udf/super.c | 3 +- fs/ufs/super.c | 3 +- fs/userfaultfd.c | 3 +- fs/vboxsf/super.c | 3 +- fs/xfs/xfs_super.c | 3 +- include/linux/slab.h | 11 ++-- ipc/mqueue.c | 3 +- kernel/fork.c | 3 +- kernel/rcu/refscale.c | 3 +- lib/radix-tree.c | 3 +- lib/test_meminit.c | 3 +- mm/kfence/kfence_test.c | 5 +- mm/rmap.c | 3 +- mm/shmem.c | 3 +- mm/slab.h | 6 +- mm/slab_common.c | 4 +- mm/slub.c | 68 ++++++++++++++++------- net/socket.c | 3 +- net/sunrpc/rpc_pipe.c | 3 +- security/integrity/ima/ima_iint.c | 3 +- 82 files changed, 233 insertions(+), 120 deletions(-) diff --git a/arch/powerpc/include/asm/svm.h b/arch/powerpc/include/asm/svm.h index a02bd54b8948..0ebfbcd212cb 100644 --- a/arch/powerpc/include/asm/svm.h +++ b/arch/powerpc/include/asm/svm.h @@ -17,7 +17,7 @@ static inline bool is_secure_guest(void) return mfmsr() & MSR_S; } =20 -void dtl_cache_ctor(void *addr); +int dtl_cache_ctor(void *addr); #define get_dtl_cache_ctor() (is_secure_guest() ? dtl_cache_ctor : NULL) =20 #else /* CONFIG_PPC_SVM */ diff --git a/arch/powerpc/kvm/book3s_64_mmu_radix.c b/arch/powerpc/kvm/book= 3s_64_mmu_radix.c index b3e6e73d6a08..9d6171d6db65 100644 --- a/arch/powerpc/kvm/book3s_64_mmu_radix.c +++ b/arch/powerpc/kvm/book3s_64_mmu_radix.c @@ -1230,9 +1230,10 @@ int kvmppc_init_vm_radix(struct kvm *kvm) return 0; } =20 -static void pte_ctor(void *addr) +static int pte_ctor(void *addr) { memset(addr, 0, RADIX_PTE_TABLE_SIZE); + return 0; } =20 static void pmd_ctor(void *addr) diff --git a/arch/powerpc/mm/init-common.c b/arch/powerpc/mm/init-common.c index 745097554bea..4eb2cc44fa86 100644 --- a/arch/powerpc/mm/init-common.c +++ b/arch/powerpc/mm/init-common.c @@ -72,9 +72,10 @@ void setup_kup(void) setup_kuep(disable_kuep); } =20 -#define CTOR(shift) static void ctor_##shift(void *addr) \ +#define CTOR(shift) static int ctor_##shift(void *addr) \ { \ memset(addr, 0, sizeof(pgd_t) << (shift)); \ + return 0; \ } =20 CTOR(0); CTOR(1); CTOR(2); CTOR(3); CTOR(4); CTOR(5); CTOR(6); CTOR(7); diff --git a/arch/powerpc/platforms/cell/spufs/inode.c b/arch/powerpc/platf= orms/cell/spufs/inode.c index 9f9e4b871627..c654e95431fa 100644 --- a/arch/powerpc/platforms/cell/spufs/inode.c +++ b/arch/powerpc/platforms/cell/spufs/inode.c @@ -65,12 +65,13 @@ static void spufs_free_inode(struct inode *inode) kmem_cache_free(spufs_inode_cache, SPUFS_I(inode)); } =20 -static void +static int spufs_init_once(void *p) { struct spufs_inode_info *ei =3D p; =20 inode_init_once(&ei->vfs_inode); + return 0; } =20 static struct inode * diff --git a/arch/powerpc/platforms/pseries/setup.c b/arch/powerpc/platform= s/pseries/setup.c index b10a25325238..61f07df96f99 100644 --- a/arch/powerpc/platforms/pseries/setup.c +++ b/arch/powerpc/platforms/pseries/setup.c @@ -341,7 +341,7 @@ static inline int alloc_dispatch_logs(void) =20 static int alloc_dispatch_log_kmem_cache(void) { - void (*ctor)(void *) =3D get_dtl_cache_ctor(); + int (*ctor)(void *) =3D get_dtl_cache_ctor(); =20 dtl_cache =3D kmem_cache_create_usercopy("dtl", DISPATCH_LOG_BYTES, DISPATCH_LOG_BYTES, 0, 0, DISPATCH_LOG_BYTES, ctor); diff --git a/arch/powerpc/platforms/pseries/svm.c b/arch/powerpc/platforms/= pseries/svm.c index 384c9dc1899a..06af254bdec7 100644 --- a/arch/powerpc/platforms/pseries/svm.c +++ b/arch/powerpc/platforms/pseries/svm.c @@ -81,7 +81,7 @@ static bool is_dtl_page_shared(struct page *page) return false; } =20 -void dtl_cache_ctor(void *addr) +int dtl_cache_ctor(void *addr) { unsigned long pfn =3D PHYS_PFN(__pa(addr)); struct page *page =3D pfn_to_page(pfn); @@ -92,4 +92,6 @@ void dtl_cache_ctor(void *addr) WARN_ON(dtl_nr_pages >=3D NR_DTL_PAGE); uv_share_page(pfn, 1); } + + return 0; } diff --git a/arch/sh/mm/pgtable.c b/arch/sh/mm/pgtable.c index 3a4085ea0161..ec3ea909b3bd 100644 --- a/arch/sh/mm/pgtable.c +++ b/arch/sh/mm/pgtable.c @@ -9,7 +9,7 @@ static struct kmem_cache *pgd_cachep; static struct kmem_cache *pmd_cachep; #endif =20 -static void pgd_ctor(void *x) +static int pgd_ctor(void *x) { pgd_t *pgd =3D x; =20 @@ -17,6 +17,7 @@ static void pgd_ctor(void *x) memcpy(pgd + USER_PTRS_PER_PGD, swapper_pg_dir + USER_PTRS_PER_PGD, (PTRS_PER_PGD - USER_PTRS_PER_PGD) * sizeof(pgd_t)); + return 0; } =20 void pgtable_cache_init(void) diff --git a/arch/sparc/mm/tsb.c b/arch/sparc/mm/tsb.c index 5fe52a64c7e7..53b555ee9f7e 100644 --- a/arch/sparc/mm/tsb.c +++ b/arch/sparc/mm/tsb.c @@ -338,6 +338,12 @@ static const char *tsb_cache_names[8] =3D { "tsb_1MB", }; =20 +static inline int pgtable_ctor(void *objp) +{ + _clear_page(objp); + return 0; +} + void __init pgtable_cache_init(void) { unsigned long i; @@ -345,7 +351,7 @@ void __init pgtable_cache_init(void) pgtable_cache =3D kmem_cache_create("pgtable_cache", PAGE_SIZE, PAGE_SIZE, 0, - _clear_page); + pgtable_ctor); if (!pgtable_cache) { prom_printf("pgtable_cache_init(): Could not create!\n"); prom_halt(); diff --git a/block/bdev.c b/block/bdev.c index 4844d1e27b6f..de4c231fbb04 100644 --- a/block/bdev.c +++ b/block/bdev.c @@ -356,11 +356,12 @@ static void bdev_free_inode(struct inode *inode) kmem_cache_free(bdev_cachep, BDEV_I(inode)); } =20 -static void init_once(void *data) +static int init_once(void *data) { struct bdev_inode *ei =3D data; =20 inode_init_once(&ei->vfs_inode); + return 0; } =20 static void bdev_evict_inode(struct inode *inode) diff --git a/drivers/dax/super.c b/drivers/dax/super.c index e16d1d40d773..3b3bf03d10cf 100644 --- a/drivers/dax/super.c +++ b/drivers/dax/super.c @@ -532,13 +532,14 @@ void *dax_get_private(struct dax_device *dax_dev) } EXPORT_SYMBOL_GPL(dax_get_private); =20 -static void init_once(void *_dax_dev) +static int init_once(void *_dax_dev) { struct dax_device *dax_dev =3D _dax_dev; struct inode *inode =3D &dax_dev->inode; =20 memset(dax_dev, 0, sizeof(*dax_dev)); inode_init_once(inode); + return 0; } =20 static int dax_fs_init(void) diff --git a/drivers/gpu/drm/i915/i915_request.c b/drivers/gpu/drm/i915/i91= 5_request.c index c3d27eadc0a7..70fbd2b34240 100644 --- a/drivers/gpu/drm/i915/i915_request.c +++ b/drivers/gpu/drm/i915/i915_request.c @@ -869,7 +869,7 @@ request_alloc_slow(struct intel_timeline *tl, return kmem_cache_alloc(slab_requests, gfp); } =20 -static void __i915_request_ctor(void *arg) +static int __i915_request_ctor(void *arg) { struct i915_request *rq =3D arg; =20 @@ -882,6 +882,7 @@ static void __i915_request_ctor(void *arg) rq->batch_res =3D NULL; =20 init_llist_head(&rq->execute_cb); + return 0; } =20 #if IS_ENABLED(CONFIG_DRM_I915_SELFTEST) diff --git a/drivers/misc/lkdtm/heap.c b/drivers/misc/lkdtm/heap.c index b1b316f99703..0a821503bb31 100644 --- a/drivers/misc/lkdtm/heap.c +++ b/drivers/misc/lkdtm/heap.c @@ -359,12 +359,12 @@ static void lkdtm_SLAB_FREE_PAGE(void) * We have constructors to keep the caches distinctly separated without * needing to boot with "slab_nomerge". */ -static void ctor_double_free(void *region) -{ } -static void ctor_a(void *region) -{ } -static void ctor_b(void *region) -{ } +static int ctor_double_free(void *region) +{ return 0; } +static int ctor_a(void *region) +{ return 0; } +static int ctor_b(void *region) +{ return 0; } =20 void __init lkdtm_heap_init(void) { diff --git a/drivers/usb/mon/mon_text.c b/drivers/usb/mon/mon_text.c index 68b9b2b41189..ef754be94b18 100644 --- a/drivers/usb/mon/mon_text.c +++ b/drivers/usb/mon/mon_text.c @@ -95,7 +95,7 @@ struct mon_reader_text { =20 static struct dentry *mon_dir; /* Usually /sys/kernel/debug/usbmon */ =20 -static void mon_text_ctor(void *); +static int mon_text_ctor(void *); =20 struct mon_text_ptr { int cnt, limit; @@ -732,13 +732,14 @@ void mon_text_del(struct mon_bus *mbus) /* * Slab interface: constructor. */ -static void mon_text_ctor(void *mem) +static int mon_text_ctor(void *mem) { /* * Nothing to initialize. No, really! * So, we fill it with garbage to emulate a reused object. */ memset(mem, 0xe5, sizeof(struct mon_event_text)); + return 0; } =20 int __init mon_text_init(void) diff --git a/fs/9p/v9fs.c b/fs/9p/v9fs.c index 77e9c4387c1d..a11a39d369c5 100644 --- a/fs/9p/v9fs.c +++ b/fs/9p/v9fs.c @@ -620,12 +620,13 @@ static void v9fs_sysfs_cleanup(void) kobject_put(v9fs_kobj); } =20 -static void v9fs_inode_init_once(void *foo) +static int v9fs_inode_init_once(void *foo) { struct v9fs_inode *v9inode =3D (struct v9fs_inode *)foo; =20 memset(&v9inode->qid, 0, sizeof(v9inode->qid)); inode_init_once(&v9inode->netfs.inode); + return 0; } =20 /** diff --git a/fs/adfs/super.c b/fs/adfs/super.c index 017c48a80203..8b9df1dbfed5 100644 --- a/fs/adfs/super.c +++ b/fs/adfs/super.c @@ -212,11 +212,12 @@ static int adfs_drop_inode(struct inode *inode) return !IS_ENABLED(CONFIG_ADFS_FS_RW) || IS_RDONLY(inode); } =20 -static void init_once(void *foo) +static int init_once(void *foo) { struct adfs_inode_info *ei =3D (struct adfs_inode_info *) foo; =20 inode_init_once(&ei->vfs_inode); + return 0; } =20 static int __init init_inodecache(void) diff --git a/fs/affs/super.c b/fs/affs/super.c index 2fa40337776d..8cff0659d8f1 100644 --- a/fs/affs/super.c +++ b/fs/affs/super.c @@ -117,13 +117,14 @@ static void affs_free_inode(struct inode *inode) kmem_cache_free(affs_inode_cachep, AFFS_I(inode)); } =20 -static void init_once(void *foo) +static int init_once(void *foo) { struct affs_inode_info *ei =3D (struct affs_inode_info *) foo; =20 mutex_init(&ei->i_link_lock); mutex_init(&ei->i_ext_lock); inode_init_once(&ei->vfs_inode); + return 0; } =20 static int __init init_inodecache(void) diff --git a/fs/afs/super.c b/fs/afs/super.c index 25b306db6992..e6a4473cf113 100644 --- a/fs/afs/super.c +++ b/fs/afs/super.c @@ -29,7 +29,7 @@ #include #include "internal.h" =20 -static void afs_i_init_once(void *foo); +static int afs_i_init_once(void *foo); static void afs_kill_super(struct super_block *sb); static struct inode *afs_alloc_inode(struct super_block *sb); static void afs_destroy_inode(struct inode *inode); @@ -646,7 +646,7 @@ static int afs_init_fs_context(struct fs_context *fc) * afs_alloc_inode() *must* reset anything that could incorrectly leak fro= m one * inode to another. */ -static void afs_i_init_once(void *_vnode) +static int afs_i_init_once(void *_vnode) { struct afs_vnode *vnode =3D _vnode; =20 @@ -662,6 +662,7 @@ static void afs_i_init_once(void *_vnode) INIT_DELAYED_WORK(&vnode->lock_work, afs_lock_work); INIT_LIST_HEAD(&vnode->cb_mmap_link); seqlock_init(&vnode->cb_lock); + return 0; } =20 /* diff --git a/fs/befs/linuxvfs.c b/fs/befs/linuxvfs.c index 8f430ff8e445..bb94680b0ca8 100644 --- a/fs/befs/linuxvfs.c +++ b/fs/befs/linuxvfs.c @@ -287,11 +287,12 @@ static void befs_free_inode(struct inode *inode) kmem_cache_free(befs_inode_cachep, BEFS_I(inode)); } =20 -static void init_once(void *foo) +static int init_once(void *foo) { struct befs_inode_info *bi =3D (struct befs_inode_info *) foo; =20 inode_init_once(&bi->vfs_inode); + return 0; } =20 static struct inode *befs_iget(struct super_block *sb, unsigned long ino) diff --git a/fs/bfs/inode.c b/fs/bfs/inode.c index db81570c9637..c8eaf9d36507 100644 --- a/fs/bfs/inode.c +++ b/fs/bfs/inode.c @@ -247,11 +247,12 @@ static void bfs_free_inode(struct inode *inode) kmem_cache_free(bfs_inode_cachep, BFS_I(inode)); } =20 -static void init_once(void *foo) +static int init_once(void *foo) { struct bfs_inode_info *bi =3D foo; =20 inode_init_once(&bi->vfs_inode); + return 0; } =20 static int __init init_inodecache(void) diff --git a/fs/btrfs/inode.c b/fs/btrfs/inode.c index cc67d1a2d611..dc17d60ec78d 100644 --- a/fs/btrfs/inode.c +++ b/fs/btrfs/inode.c @@ -7871,11 +7871,12 @@ int btrfs_drop_inode(struct inode *inode) return generic_drop_inode(inode); } =20 -static void init_once(void *foo) +static int init_once(void *foo) { struct btrfs_inode *ei =3D foo; =20 inode_init_once(&ei->vfs_inode); + return 0; } =20 void __cold btrfs_destroy_cachep(void) diff --git a/fs/ceph/super.c b/fs/ceph/super.c index f3951253e393..e627c5f975c4 100644 --- a/fs/ceph/super.c +++ b/fs/ceph/super.c @@ -926,10 +926,11 @@ struct kmem_cache *ceph_dir_file_cachep; struct kmem_cache *ceph_mds_request_cachep; mempool_t *ceph_wb_pagevec_pool; =20 -static void ceph_inode_init_once(void *foo) +static int ceph_inode_init_once(void *foo) { struct ceph_inode_info *ci =3D foo; inode_init_once(&ci->netfs.inode); + return 0; } =20 static int __init init_caches(void) diff --git a/fs/coda/inode.c b/fs/coda/inode.c index 6896fce122e1..4580ea6ae053 100644 --- a/fs/coda/inode.c +++ b/fs/coda/inode.c @@ -61,11 +61,12 @@ static void coda_free_inode(struct inode *inode) kmem_cache_free(coda_inode_cachep, ITOC(inode)); } =20 -static void init_once(void *foo) +static int init_once(void *foo) { struct coda_inode_info *ei =3D (struct coda_inode_info *) foo; =20 inode_init_once(&ei->vfs_inode); + return 0; } =20 int __init coda_init_inodecache(void) diff --git a/fs/debugfs/inode.c b/fs/debugfs/inode.c index 75715d8877ee..b96b74b624ec 100644 --- a/fs/debugfs/inode.c +++ b/fs/debugfs/inode.c @@ -210,10 +210,11 @@ static int debugfs_show_options(struct seq_file *m, s= truct dentry *root) =20 static struct kmem_cache *debugfs_inode_cachep __ro_after_init; =20 -static void init_once(void *foo) +static int init_once(void *foo) { struct debugfs_inode_info *info =3D foo; inode_init_once(&info->vfs_inode); + return 0; } =20 static struct inode *debugfs_alloc_inode(struct super_block *sb) diff --git a/fs/dlm/lowcomms.c b/fs/dlm/lowcomms.c index 70abd4da17a6..6a4f7a68f34c 100644 --- a/fs/dlm/lowcomms.c +++ b/fs/dlm/lowcomms.c @@ -232,11 +232,12 @@ static void lowcomms_queue_rwork(struct connection *c= on) queue_work(io_workqueue, &con->rwork); } =20 -static void writequeue_entry_ctor(void *data) +static int writequeue_entry_ctor(void *data) { struct writequeue_entry *entry =3D data; =20 INIT_LIST_HEAD(&entry->msgs); + return 0; } =20 struct kmem_cache *dlm_lowcomms_writequeue_cache_create(void) diff --git a/fs/ecryptfs/main.c b/fs/ecryptfs/main.c index 8dd1d7189c3b..7afc008774e7 100644 --- a/fs/ecryptfs/main.c +++ b/fs/ecryptfs/main.c @@ -641,12 +641,13 @@ MODULE_ALIAS_FS("ecryptfs"); * * Initializes the ecryptfs_inode_info_cache when it is created */ -static void +static int inode_info_init_once(void *vptr) { struct ecryptfs_inode_info *ei =3D (struct ecryptfs_inode_info *)vptr; =20 inode_init_once(&ei->vfs_inode); + return 0; } =20 static struct ecryptfs_cache_info { @@ -654,7 +655,7 @@ static struct ecryptfs_cache_info { const char *name; size_t size; slab_flags_t flags; - void (*ctor)(void *obj); + int (*ctor)(void *obj); } ecryptfs_cache_infos[] =3D { { .cache =3D &ecryptfs_auth_tok_list_item_cache, diff --git a/fs/efs/super.c b/fs/efs/super.c index c59086b7eabf..09d4f5e0710b 100644 --- a/fs/efs/super.c +++ b/fs/efs/super.c @@ -76,11 +76,12 @@ static void efs_free_inode(struct inode *inode) kmem_cache_free(efs_inode_cachep, INODE_INFO(inode)); } =20 -static void init_once(void *foo) +static int init_once(void *foo) { struct efs_inode_info *ei =3D (struct efs_inode_info *) foo; =20 inode_init_once(&ei->vfs_inode); + return 0; } =20 static int __init init_inodecache(void) diff --git a/fs/erofs/super.c b/fs/erofs/super.c index cadec6b1b554..09b84a549c64 100644 --- a/fs/erofs/super.c +++ b/fs/erofs/super.c @@ -56,11 +56,12 @@ static int erofs_superblock_csum_verify(struct super_bl= ock *sb, void *sbdata) return -EBADMSG; } =20 -static void erofs_inode_init_once(void *ptr) +static int erofs_inode_init_once(void *ptr) { struct erofs_inode *vi =3D ptr; =20 inode_init_once(&vi->vfs_inode); + return 0; } =20 static struct inode *erofs_alloc_inode(struct super_block *sb) diff --git a/fs/exfat/cache.c b/fs/exfat/cache.c index d5ce0ae660ba..e8b2fffc60b4 100644 --- a/fs/exfat/cache.c +++ b/fs/exfat/cache.c @@ -35,11 +35,12 @@ struct exfat_cache_id { =20 static struct kmem_cache *exfat_cachep; =20 -static void exfat_cache_init_once(void *c) +static int exfat_cache_init_once(void *c) { struct exfat_cache *cache =3D (struct exfat_cache *)c; =20 INIT_LIST_HEAD(&cache->cache_list); + return 0; } =20 int exfat_cache_init(void) diff --git a/fs/exfat/super.c b/fs/exfat/super.c index 8465033a6cf0..946b50e72aea 100644 --- a/fs/exfat/super.c +++ b/fs/exfat/super.c @@ -818,7 +818,7 @@ static struct file_system_type exfat_fs_type =3D { .fs_flags =3D FS_REQUIRES_DEV | FS_ALLOW_IDMAP, }; =20 -static void exfat_inode_init_once(void *foo) +static int exfat_inode_init_once(void *foo) { struct exfat_inode_info *ei =3D (struct exfat_inode_info *)foo; =20 @@ -828,6 +828,7 @@ static void exfat_inode_init_once(void *foo) INIT_LIST_HEAD(&ei->cache_lru); INIT_HLIST_NODE(&ei->i_hash_fat); inode_init_once(&ei->vfs_inode); + return 0; } =20 static int __init init_exfat_fs(void) diff --git a/fs/ext2/super.c b/fs/ext2/super.c index 28ff47ec4be6..7a4a6d6c069b 100644 --- a/fs/ext2/super.c +++ b/fs/ext2/super.c @@ -224,7 +224,7 @@ static void ext2_free_in_core_inode(struct inode *inode) kmem_cache_free(ext2_inode_cachep, EXT2_I(inode)); } =20 -static void init_once(void *foo) +static int init_once(void *foo) { struct ext2_inode_info *ei =3D (struct ext2_inode_info *) foo; =20 @@ -234,6 +234,7 @@ static void init_once(void *foo) #endif mutex_init(&ei->truncate_mutex); inode_init_once(&ei->vfs_inode); + return 0; } =20 static int __init init_inodecache(void) diff --git a/fs/ext4/super.c b/fs/ext4/super.c index 181934499624..7616ab697cdf 100644 --- a/fs/ext4/super.c +++ b/fs/ext4/super.c @@ -1465,7 +1465,7 @@ static void ext4_shutdown(struct super_block *sb) ext4_force_shutdown(sb, EXT4_GOING_FLAGS_NOLOGFLUSH); } =20 -static void init_once(void *foo) +static int init_once(void *foo) { struct ext4_inode_info *ei =3D foo; =20 @@ -1474,6 +1474,7 @@ static void init_once(void *foo) init_rwsem(&ei->i_data_sem); inode_init_once(&ei->vfs_inode); ext4_fc_init_inode(&ei->vfs_inode); + return 0; } =20 static int __init init_inodecache(void) diff --git a/fs/fat/cache.c b/fs/fat/cache.c index 2af424e200b3..9478c8ca35e8 100644 --- a/fs/fat/cache.c +++ b/fs/fat/cache.c @@ -36,11 +36,12 @@ static inline int fat_max_cache(struct inode *inode) =20 static struct kmem_cache *fat_cache_cachep; =20 -static void init_once(void *foo) +static int init_once(void *foo) { struct fat_cache *cache =3D (struct fat_cache *)foo; =20 INIT_LIST_HEAD(&cache->cache_list); + return 0; } =20 int __init fat_cache_init(void) diff --git a/fs/fat/inode.c b/fs/fat/inode.c index 3852bb66358c..0576d7cda07d 100644 --- a/fs/fat/inode.c +++ b/fs/fat/inode.c @@ -767,7 +767,7 @@ static void fat_free_inode(struct inode *inode) kmem_cache_free(fat_inode_cachep, MSDOS_I(inode)); } =20 -static void init_once(void *foo) +static int init_once(void *foo) { struct msdos_inode_info *ei =3D (struct msdos_inode_info *)foo; =20 @@ -778,6 +778,7 @@ static void init_once(void *foo) INIT_HLIST_NODE(&ei->i_fat_hash); INIT_HLIST_NODE(&ei->i_dir_hash); inode_init_once(&ei->vfs_inode); + return 0; } =20 static int __init fat_init_inodecache(void) diff --git a/fs/fuse/inode.c b/fs/fuse/inode.c index fd48e8d37f2e..358f47f091c1 100644 --- a/fs/fuse/inode.c +++ b/fs/fuse/inode.c @@ -2131,11 +2131,12 @@ static inline void unregister_fuseblk(void) } #endif =20 -static void fuse_inode_init_once(void *foo) +static int fuse_inode_init_once(void *foo) { struct inode *inode =3D foo; =20 inode_init_once(inode); + return 0; } =20 static int __init fuse_fs_init(void) diff --git a/fs/gfs2/main.c b/fs/gfs2/main.c index 0727f60ad028..0923a091d926 100644 --- a/fs/gfs2/main.c +++ b/fs/gfs2/main.c @@ -31,7 +31,7 @@ =20 struct workqueue_struct *gfs2_control_wq; =20 -static void gfs2_init_inode_once(void *foo) +static int gfs2_init_inode_once(void *foo) { struct gfs2_inode *ip =3D foo; =20 @@ -45,9 +45,10 @@ static void gfs2_init_inode_once(void *foo) RB_CLEAR_NODE(&ip->i_res.rs_node); ip->i_hash_cache =3D NULL; gfs2_holder_mark_uninitialized(&ip->i_iopen_gh); + return 0; } =20 -static void gfs2_init_glock_once(void *foo) +static int gfs2_init_glock_once(void *foo) { struct gfs2_glock *gl =3D foo; =20 @@ -56,14 +57,16 @@ static void gfs2_init_glock_once(void *foo) INIT_LIST_HEAD(&gl->gl_ail_list); atomic_set(&gl->gl_ail_count, 0); atomic_set(&gl->gl_revokes, 0); + return 0; } =20 -static void gfs2_init_gl_aspace_once(void *foo) +static int gfs2_init_gl_aspace_once(void *foo) { struct gfs2_glock_aspace *gla =3D foo; =20 gfs2_init_glock_once(&gla->glock); address_space_init_once(&gla->mapping); + return 0; } =20 /** diff --git a/fs/hfs/super.c b/fs/hfs/super.c index fe09c2093a93..ab8bd6c895c1 100644 --- a/fs/hfs/super.c +++ b/fs/hfs/super.c @@ -436,11 +436,12 @@ static struct file_system_type hfs_fs_type =3D { }; MODULE_ALIAS_FS("hfs"); =20 -static void hfs_init_once(void *p) +static int hfs_init_once(void *p) { struct hfs_inode_info *i =3D p; =20 inode_init_once(&i->vfs_inode); + return 0; } =20 static int __init init_hfs_fs(void) diff --git a/fs/hfsplus/super.c b/fs/hfsplus/super.c index 948b8aaee33e..b390381e837f 100644 --- a/fs/hfsplus/super.c +++ b/fs/hfsplus/super.c @@ -678,11 +678,12 @@ static struct file_system_type hfsplus_fs_type =3D { }; MODULE_ALIAS_FS("hfsplus"); =20 -static void hfsplus_init_once(void *p) +static int hfsplus_init_once(void *p) { struct hfsplus_inode_info *i =3D p; =20 inode_init_once(&i->vfs_inode); + return 0; } =20 static int __init init_hfsplus_fs(void) diff --git a/fs/hpfs/super.c b/fs/hpfs/super.c index 27567920abe4..1af5a9f5f931 100644 --- a/fs/hpfs/super.c +++ b/fs/hpfs/super.c @@ -244,11 +244,12 @@ static void hpfs_free_inode(struct inode *inode) kmem_cache_free(hpfs_inode_cachep, hpfs_i(inode)); } =20 -static void init_once(void *foo) +static int init_once(void *foo) { struct hpfs_inode_info *ei =3D (struct hpfs_inode_info *) foo; =20 inode_init_once(&ei->vfs_inode); + return 0; } =20 static int init_inodecache(void) diff --git a/fs/hugetlbfs/inode.c b/fs/hugetlbfs/inode.c index e4de5425838d..c0513bb7ed88 100644 --- a/fs/hugetlbfs/inode.c +++ b/fs/hugetlbfs/inode.c @@ -1225,11 +1225,12 @@ static const struct address_space_operations hugetl= bfs_aops =3D { }; =20 =20 -static void init_once(void *foo) +static int init_once(void *foo) { struct hugetlbfs_inode_info *ei =3D foo; =20 inode_init_once(&ei->vfs_inode); + return 0; } =20 static const struct file_operations hugetlbfs_file_operations =3D { diff --git a/fs/inode.c b/fs/inode.c index 99318b157a9a..d1b73cbda4f5 100644 --- a/fs/inode.c +++ b/fs/inode.c @@ -514,11 +514,12 @@ void inode_init_once(struct inode *inode) } EXPORT_SYMBOL(inode_init_once); =20 -static void init_once(void *foo) +static int init_once(void *foo) { struct inode *inode =3D (struct inode *) foo; =20 inode_init_once(inode); + return 0; } =20 /* diff --git a/fs/isofs/inode.c b/fs/isofs/inode.c index 47038e660812..0251eefdb18f 100644 --- a/fs/isofs/inode.c +++ b/fs/isofs/inode.c @@ -82,11 +82,12 @@ static void isofs_free_inode(struct inode *inode) kmem_cache_free(isofs_inode_cachep, ISOFS_I(inode)); } =20 -static void init_once(void *foo) +static int init_once(void *foo) { struct iso_inode_info *ei =3D foo; =20 inode_init_once(&ei->vfs_inode); + return 0; } =20 static int __init init_inodecache(void) diff --git a/fs/jffs2/super.c b/fs/jffs2/super.c index 4545f885c41e..7249f6ad5b0a 100644 --- a/fs/jffs2/super.c +++ b/fs/jffs2/super.c @@ -53,13 +53,14 @@ static void jffs2_free_inode(struct inode *inode) kmem_cache_free(jffs2_inode_cachep, f); } =20 -static void jffs2_i_init_once(void *foo) +static int jffs2_i_init_once(void *foo) { struct jffs2_inode_info *f =3D foo; =20 mutex_init(&f->sem); f->target =3D NULL; inode_init_once(&f->vfs_inode); + return 0; } =20 static const char *jffs2_compr_name(unsigned int compr) diff --git a/fs/jfs/super.c b/fs/jfs/super.c index 10368c188c5e..f9bae8adfc18 100644 --- a/fs/jfs/super.c +++ b/fs/jfs/super.c @@ -934,7 +934,7 @@ static struct file_system_type jfs_fs_type =3D { }; MODULE_ALIAS_FS("jfs"); =20 -static void init_once(void *foo) +static int init_once(void *foo) { struct jfs_inode_info *jfs_ip =3D (struct jfs_inode_info *) foo; =20 @@ -946,6 +946,7 @@ static void init_once(void *foo) spin_lock_init(&jfs_ip->ag_lock); jfs_ip->active_ag =3D -1; inode_init_once(&jfs_ip->vfs_inode); + return 0; } =20 static int __init init_jfs_fs(void) diff --git a/fs/minix/inode.c b/fs/minix/inode.c index f007e389d5d2..dc13b652754b 100644 --- a/fs/minix/inode.c +++ b/fs/minix/inode.c @@ -75,11 +75,12 @@ static void minix_free_in_core_inode(struct inode *inod= e) kmem_cache_free(minix_inode_cachep, minix_i(inode)); } =20 -static void init_once(void *foo) +static int init_once(void *foo) { struct minix_inode_info *ei =3D (struct minix_inode_info *) foo; =20 inode_init_once(&ei->vfs_inode); + return 0; } =20 static int __init init_inodecache(void) diff --git a/fs/nfs/inode.c b/fs/nfs/inode.c index 119e447758b9..274cdad1c13e 100644 --- a/fs/nfs/inode.c +++ b/fs/nfs/inode.c @@ -2456,7 +2456,7 @@ static inline void nfs4_init_once(struct nfs_inode *n= fsi) #endif } =20 -static void init_once(void *foo) +static int init_once(void *foo) { struct nfs_inode *nfsi =3D foo; =20 @@ -2465,6 +2465,7 @@ static void init_once(void *foo) INIT_LIST_HEAD(&nfsi->access_cache_entry_lru); INIT_LIST_HEAD(&nfsi->access_cache_inode_lru); nfs4_init_once(nfsi); + return 0; } =20 static int __init nfs_init_inodecache(void) diff --git a/fs/nfs/nfs42xattr.c b/fs/nfs/nfs42xattr.c index 37d79400e5f4..22e5e9751656 100644 --- a/fs/nfs/nfs42xattr.c +++ b/fs/nfs/nfs42xattr.c @@ -960,7 +960,7 @@ nfs4_xattr_entry_count(struct shrinker *shrink, struct = shrink_control *sc) } =20 =20 -static void nfs4_xattr_cache_init_once(void *p) +static int nfs4_xattr_cache_init_once(void *p) { struct nfs4_xattr_cache *cache =3D p; =20 @@ -970,6 +970,7 @@ static void nfs4_xattr_cache_init_once(void *p) cache->listxattr =3D NULL; INIT_LIST_HEAD(&cache->lru); INIT_LIST_HEAD(&cache->dispose); + return 0; } =20 typedef unsigned long (*count_objects_cb)(struct shrinker *s, diff --git a/fs/nilfs2/super.c b/fs/nilfs2/super.c index badc2cbc895e..415381d9e5f9 100644 --- a/fs/nilfs2/super.c +++ b/fs/nilfs2/super.c @@ -1313,7 +1313,7 @@ struct file_system_type nilfs_fs_type =3D { }; MODULE_ALIAS_FS("nilfs2"); =20 -static void nilfs_inode_init_once(void *obj) +static int nilfs_inode_init_once(void *obj) { struct nilfs_inode_info *ii =3D obj; =20 @@ -1322,11 +1322,13 @@ static void nilfs_inode_init_once(void *obj) init_rwsem(&ii->xattr_sem); #endif inode_init_once(&ii->vfs_inode); + return 0; } =20 -static void nilfs_segbuf_init_once(void *obj) +static int nilfs_segbuf_init_once(void *obj) { memset(obj, 0, sizeof(struct nilfs_segment_buffer)); + return 0; } =20 static void nilfs_destroy_cachep(void) diff --git a/fs/ntfs3/super.c b/fs/ntfs3/super.c index 920a1ab47b63..8dcfe06ba996 100644 --- a/fs/ntfs3/super.c +++ b/fs/ntfs3/super.c @@ -628,11 +628,12 @@ static void ntfs_free_inode(struct inode *inode) kmem_cache_free(ntfs_inode_cachep, ni); } =20 -static void init_once(void *foo) +static int init_once(void *foo) { struct ntfs_inode *ni =3D foo; =20 inode_init_once(&ni->vfs_inode); + return 0; } =20 /* diff --git a/fs/ocfs2/dlmfs/dlmfs.c b/fs/ocfs2/dlmfs/dlmfs.c index 5130ec44e5e1..3c044f94970f 100644 --- a/fs/ocfs2/dlmfs/dlmfs.c +++ b/fs/ocfs2/dlmfs/dlmfs.c @@ -265,7 +265,7 @@ static ssize_t dlmfs_file_write(struct file *filp, return count; } =20 -static void dlmfs_init_once(void *foo) +static int dlmfs_init_once(void *foo) { struct dlmfs_inode_private *ip =3D (struct dlmfs_inode_private *) foo; @@ -274,6 +274,7 @@ static void dlmfs_init_once(void *foo) ip->ip_parent =3D NULL; =20 inode_init_once(&ip->ip_vfs_inode); + return 0; } =20 static struct inode *dlmfs_alloc_inode(struct super_block *sb) diff --git a/fs/ocfs2/super.c b/fs/ocfs2/super.c index 8bb5022f3082..b33185567db9 100644 --- a/fs/ocfs2/super.c +++ b/fs/ocfs2/super.c @@ -1614,7 +1614,7 @@ static int ocfs2_statfs(struct dentry *dentry, struct= kstatfs *buf) return status; } =20 -static void ocfs2_inode_init_once(void *data) +static int ocfs2_inode_init_once(void *data) { struct ocfs2_inode_info *oi =3D data; =20 @@ -1643,6 +1643,7 @@ static void ocfs2_inode_init_once(void *data) &ocfs2_inode_caching_ops); =20 inode_init_once(&oi->vfs_inode); + return 0; } =20 static int ocfs2_initialize_mem_caches(void) diff --git a/fs/openpromfs/inode.c b/fs/openpromfs/inode.c index 26ecda0e4d19..5f40a4e0dd55 100644 --- a/fs/openpromfs/inode.c +++ b/fs/openpromfs/inode.c @@ -431,11 +431,12 @@ static struct file_system_type openprom_fs_type =3D { }; MODULE_ALIAS_FS("openpromfs"); =20 -static void op_inode_init_once(void *data) +static int op_inode_init_once(void *data) { struct op_inode_info *oi =3D (struct op_inode_info *) data; =20 inode_init_once(&oi->vfs_inode); + return 0; } =20 static int __init init_openprom_fs(void) diff --git a/fs/orangefs/super.c b/fs/orangefs/super.c index eba3e357192e..9521452dea13 100644 --- a/fs/orangefs/super.c +++ b/fs/orangefs/super.c @@ -95,12 +95,13 @@ static int parse_mount_options(struct super_block *sb, = char *options, return -EINVAL; } =20 -static void orangefs_inode_cache_ctor(void *req) +static int orangefs_inode_cache_ctor(void *req) { struct orangefs_inode_s *orangefs_inode =3D req; =20 inode_init_once(&orangefs_inode->vfs_inode); init_rwsem(&orangefs_inode->xattr_sem); + return 0; } =20 static struct inode *orangefs_alloc_inode(struct super_block *sb) diff --git a/fs/overlayfs/super.c b/fs/overlayfs/super.c index b63474d1b064..d46d4ea30f66 100644 --- a/fs/overlayfs/super.c +++ b/fs/overlayfs/super.c @@ -1516,11 +1516,12 @@ struct file_system_type ovl_fs_type =3D { }; MODULE_ALIAS_FS("overlay"); =20 -static void ovl_inode_init_once(void *foo) +static int ovl_inode_init_once(void *foo) { struct ovl_inode *oi =3D foo; =20 inode_init_once(&oi->vfs_inode); + return 0; } =20 static int __init ovl_init(void) diff --git a/fs/pidfs.c b/fs/pidfs.c index d64a4cbeb0da..efc9aac05446 100644 --- a/fs/pidfs.c +++ b/fs/pidfs.c @@ -896,11 +896,12 @@ struct file *pidfs_alloc_file(struct pid *pid, unsign= ed int flags) return pidfd_file; } =20 -static void pidfs_inode_init_once(void *data) +static int pidfs_inode_init_once(void *data) { struct pidfs_inode *pi =3D data; =20 inode_init_once(&pi->vfs_inode); + return 0; } =20 void __init pidfs_init(void) diff --git a/fs/proc/inode.c b/fs/proc/inode.c index a3eb3b740f76..f1bf32900389 100644 --- a/fs/proc/inode.c +++ b/fs/proc/inode.c @@ -80,11 +80,12 @@ static void proc_free_inode(struct inode *inode) kmem_cache_free(proc_inode_cachep, PROC_I(inode)); } =20 -static void init_once(void *foo) +static int init_once(void *foo) { struct proc_inode *ei =3D (struct proc_inode *) foo; =20 inode_init_once(&ei->vfs_inode); + return 0; } =20 void __init proc_init_kmemcache(void) diff --git a/fs/qnx4/inode.c b/fs/qnx4/inode.c index e399e2dd3a12..624911ffc8a8 100644 --- a/fs/qnx4/inode.c +++ b/fs/qnx4/inode.c @@ -366,11 +366,12 @@ static void qnx4_free_inode(struct inode *inode) kmem_cache_free(qnx4_inode_cachep, qnx4_i(inode)); } =20 -static void init_once(void *foo) +static int init_once(void *foo) { struct qnx4_inode_info *ei =3D (struct qnx4_inode_info *) foo; =20 inode_init_once(&ei->vfs_inode); + return 0; } =20 static int init_inodecache(void) diff --git a/fs/qnx6/inode.c b/fs/qnx6/inode.c index 3310d1ad4d0e..196b4fafcc46 100644 --- a/fs/qnx6/inode.c +++ b/fs/qnx6/inode.c @@ -595,11 +595,12 @@ static void qnx6_free_inode(struct inode *inode) kmem_cache_free(qnx6_inode_cachep, QNX6_I(inode)); } =20 -static void init_once(void *foo) +static int init_once(void *foo) { struct qnx6_inode_info *ei =3D (struct qnx6_inode_info *) foo; =20 inode_init_once(&ei->vfs_inode); + return 0; } =20 static int init_inodecache(void) diff --git a/fs/romfs/super.c b/fs/romfs/super.c index 0addcc849ff2..622a388dfc8f 100644 --- a/fs/romfs/super.c +++ b/fs/romfs/super.c @@ -601,11 +601,12 @@ MODULE_ALIAS_FS("romfs"); /* * inode storage initialiser */ -static void romfs_i_init_once(void *_inode) +static int romfs_i_init_once(void *_inode) { struct romfs_inode_info *inode =3D _inode; =20 inode_init_once(&inode->vfs_inode); + return 0; } =20 /* diff --git a/fs/smb/client/cifsfs.c b/fs/smb/client/cifsfs.c index a08c42363ffc..4e16806552ba 100644 --- a/fs/smb/client/cifsfs.c +++ b/fs/smb/client/cifsfs.c @@ -1641,13 +1641,14 @@ const struct file_operations cifs_dir_ops =3D { .fsync =3D cifs_dir_fsync, }; =20 -static void +static int cifs_init_once(void *inode) { struct cifsInodeInfo *cifsi =3D inode; =20 inode_init_once(&cifsi->netfs.inode); init_rwsem(&cifsi->lock_sem); + return 0; } =20 static int __init diff --git a/fs/squashfs/super.c b/fs/squashfs/super.c index 67c55fe32ce8..0764d52ecc2c 100644 --- a/fs/squashfs/super.c +++ b/fs/squashfs/super.c @@ -606,11 +606,12 @@ static void squashfs_put_super(struct super_block *sb) static struct kmem_cache *squashfs_inode_cachep; =20 =20 -static void init_once(void *foo) +static int init_once(void *foo) { struct squashfs_inode_info *ei =3D foo; =20 inode_init_once(&ei->vfs_inode); + return 0; } =20 =20 diff --git a/fs/tracefs/inode.c b/fs/tracefs/inode.c index cb1af30b49f5..87cc889e47d8 100644 --- a/fs/tracefs/inode.c +++ b/fs/tracefs/inode.c @@ -792,7 +792,7 @@ bool tracefs_initialized(void) return tracefs_registered; } =20 -static void init_once(void *foo) +static int init_once(void *foo) { struct tracefs_inode *ti =3D (struct tracefs_inode *) foo; =20 @@ -801,6 +801,7 @@ static void init_once(void *foo) =20 /* Zero out the rest */ memset_after(ti, 0, vfs_inode); + return 0; } =20 static int __init tracefs_init(void) diff --git a/fs/ubifs/super.c b/fs/ubifs/super.c index f3e3b2068608..3591397d7df7 100644 --- a/fs/ubifs/super.c +++ b/fs/ubifs/super.c @@ -2376,10 +2376,11 @@ MODULE_ALIAS_FS("ubifs"); /* * Inode slab cache constructor. */ -static void inode_slab_ctor(void *obj) +static int inode_slab_ctor(void *obj) { struct ubifs_inode *ui =3D obj; inode_init_once(&ui->vfs_inode); + return 0; } =20 static int __init ubifs_init(void) diff --git a/fs/udf/super.c b/fs/udf/super.c index 1c8a736b3309..8fdf6efc7953 100644 --- a/fs/udf/super.c +++ b/fs/udf/super.c @@ -175,12 +175,13 @@ static void udf_free_in_core_inode(struct inode *inod= e) kmem_cache_free(udf_inode_cachep, UDF_I(inode)); } =20 -static void init_once(void *foo) +static int init_once(void *foo) { struct udf_inode_info *ei =3D foo; =20 ei->i_data =3D NULL; inode_init_once(&ei->vfs_inode); + return 0; } =20 static int __init init_inodecache(void) diff --git a/fs/ufs/super.c b/fs/ufs/super.c index 762699c1bcf6..b188ebe93939 100644 --- a/fs/ufs/super.c +++ b/fs/ufs/super.c @@ -1444,11 +1444,12 @@ static void ufs_free_in_core_inode(struct inode *in= ode) kmem_cache_free(ufs_inode_cachep, UFS_I(inode)); } =20 -static void init_once(void *foo) +static int init_once(void *foo) { struct ufs_inode_info *ei =3D (struct ufs_inode_info *) foo; =20 inode_init_once(&ei->vfs_inode); + return 0; } =20 static int __init init_inodecache(void) diff --git a/fs/userfaultfd.c b/fs/userfaultfd.c index d80f94346199..53d59a94983c 100644 --- a/fs/userfaultfd.c +++ b/fs/userfaultfd.c @@ -2073,7 +2073,7 @@ static const struct file_operations userfaultfd_fops = =3D { .llseek =3D noop_llseek, }; =20 -static void init_once_userfaultfd_ctx(void *mem) +static int init_once_userfaultfd_ctx(void *mem) { struct userfaultfd_ctx *ctx =3D (struct userfaultfd_ctx *) mem; =20 @@ -2082,6 +2082,7 @@ static void init_once_userfaultfd_ctx(void *mem) init_waitqueue_head(&ctx->event_wqh); init_waitqueue_head(&ctx->fd_wqh); seqcount_spinlock_init(&ctx->refile_seq, &ctx->fault_pending_wqh.lock); + return 0; } =20 static int new_userfaultfd(int flags) diff --git a/fs/vboxsf/super.c b/fs/vboxsf/super.c index 0bc96ab6580b..f4fc1cb03014 100644 --- a/fs/vboxsf/super.c +++ b/fs/vboxsf/super.c @@ -222,12 +222,13 @@ static int vboxsf_fill_super(struct super_block *sb, = struct fs_context *fc) return err; } =20 -static void vboxsf_inode_init_once(void *data) +static int vboxsf_inode_init_once(void *data) { struct vboxsf_inode *sf_i =3D data; =20 mutex_init(&sf_i->handle_list_mutex); inode_init_once(&sf_i->vfs_inode); + return 0; } =20 static struct inode *vboxsf_alloc_inode(struct super_block *sb) diff --git a/fs/xfs/xfs_super.c b/fs/xfs/xfs_super.c index b2dd0c0bf509..2eaab19c1fc5 100644 --- a/fs/xfs/xfs_super.c +++ b/fs/xfs/xfs_super.c @@ -725,7 +725,7 @@ xfs_fs_dirty_inode( * fields in the xfs inode that left in the initialise state * when freeing the inode. */ -STATIC void +STATIC int xfs_fs_inode_init_once( void *inode) { @@ -740,6 +740,7 @@ xfs_fs_inode_init_once( atomic_set(&ip->i_pincount, 0); spin_lock_init(&ip->i_flags_lock); init_rwsem(&ip->i_lock); + return 0; } =20 /* diff --git a/include/linux/slab.h b/include/linux/slab.h index d5a8ab98035c..1ef6d5384f0b 100644 --- a/include/linux/slab.h +++ b/include/linux/slab.h @@ -330,11 +330,12 @@ struct kmem_cache_args { * page. It is the cache user's responsibility to free object in the * same state as after calling the constructor, or deal appropriately * with any differences between a freshly constructed and a reallocated - * object. + * object. If ctor returns a nonzero value, indicating an error, slab + * allocation fails. * * %NULL means no constructor. */ - void (*ctor)(void *); + int (*ctor)(void *); }; =20 struct kmem_cache *__kmem_cache_create_args(const char *name, @@ -343,7 +344,7 @@ struct kmem_cache *__kmem_cache_create_args(const char = *name, slab_flags_t flags); static inline struct kmem_cache * __kmem_cache_create(const char *name, unsigned int size, unsigned int alig= n, - slab_flags_t flags, void (*ctor)(void *)) + slab_flags_t flags, int (*ctor)(void *)) { struct kmem_cache_args kmem_args =3D { .align =3D align, @@ -375,7 +376,7 @@ static inline struct kmem_cache * kmem_cache_create_usercopy(const char *name, unsigned int size, unsigned int align, slab_flags_t flags, unsigned int useroffset, unsigned int usersize, - void (*ctor)(void *)) + int (*ctor)(void *)) { struct kmem_cache_args kmem_args =3D { .align =3D align, @@ -775,7 +776,7 @@ void kmem_cache_free(struct kmem_cache *s, void *objp); =20 kmem_buckets *kmem_buckets_create(const char *name, slab_flags_t flags, unsigned int useroffset, unsigned int usersize, - void (*ctor)(void *)); + int (*ctor)(void *)); =20 /* * Bulk allocation and freeing operations. These are accelerated in an diff --git a/ipc/mqueue.c b/ipc/mqueue.c index 35b4f8659904..4e331e39f980 100644 --- a/ipc/mqueue.c +++ b/ipc/mqueue.c @@ -487,11 +487,12 @@ static struct vfsmount *mq_create_mount(struct ipc_na= mespace *ns) return mnt; } =20 -static void init_once(void *foo) +static int init_once(void *foo) { struct mqueue_inode_info *p =3D foo; =20 inode_init_once(&p->vfs_inode); + return 0; } =20 static struct inode *mqueue_alloc_inode(struct super_block *sb) diff --git a/kernel/fork.c b/kernel/fork.c index c4b26cd8998b..7966b0876dc3 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -3184,12 +3184,13 @@ void walk_process_tree(struct task_struct *top, pro= c_visitor visitor, void *data #define ARCH_MIN_MMSTRUCT_ALIGN 0 #endif =20 -static void sighand_ctor(void *data) +static int sighand_ctor(void *data) { struct sighand_struct *sighand =3D data; =20 spin_lock_init(&sighand->siglock); init_waitqueue_head(&sighand->signalfd_wqh); + return 0; } =20 void __init mm_cache_init(void) diff --git a/kernel/rcu/refscale.c b/kernel/rcu/refscale.c index f11a7c2af778..8ed7e3d4ca30 100644 --- a/kernel/rcu/refscale.c +++ b/kernel/rcu/refscale.c @@ -792,7 +792,7 @@ static struct refscale_typesafe *typesafe_alloc_one(voi= d) =20 // Slab-allocator constructor for refscale_typesafe structures created // out of a new slab of system memory. -static void refscale_typesafe_ctor(void *rtsp_in) +static int refscale_typesafe_ctor(void *rtsp_in) { struct refscale_typesafe *rtsp =3D rtsp_in; =20 @@ -801,6 +801,7 @@ static void refscale_typesafe_ctor(void *rtsp_in) preempt_disable(); rtsp->a =3D torture_random(this_cpu_ptr(&refscale_rand)); preempt_enable(); + return 0; } =20 static const struct ref_scale_ops typesafe_ref_ops; diff --git a/lib/radix-tree.c b/lib/radix-tree.c index 976b9bd02a1b..0e0f83ca9175 100644 --- a/lib/radix-tree.c +++ b/lib/radix-tree.c @@ -1566,13 +1566,14 @@ void idr_destroy(struct idr *idr) } EXPORT_SYMBOL(idr_destroy); =20 -static void +static int radix_tree_node_ctor(void *arg) { struct radix_tree_node *node =3D arg; =20 memset(node, 0, sizeof(*node)); INIT_LIST_HEAD(&node->private_list); + return 0; } =20 static int radix_tree_cpu_dead(unsigned int cpu) diff --git a/lib/test_meminit.c b/lib/test_meminit.c index 6298f66c964b..4697a2aedee2 100644 --- a/lib/test_meminit.c +++ b/lib/test_meminit.c @@ -169,9 +169,10 @@ static int __init test_kvmalloc(int *total_failures) #define CTOR_BYTES (sizeof(unsigned int)) #define CTOR_PATTERN (0x41414141) /* Initialize the first 4 bytes of the object. */ -static void test_ctor(void *obj) +static int test_ctor(void *obj) { *(unsigned int *)obj =3D CTOR_PATTERN; + return 0; } =20 /* diff --git a/mm/kfence/kfence_test.c b/mm/kfence/kfence_test.c index 00034e37bc9f..451760e8d1f4 100644 --- a/mm/kfence/kfence_test.c +++ b/mm/kfence/kfence_test.c @@ -184,7 +184,7 @@ static bool report_matches(const struct expect_report *= r) static struct kmem_cache *test_cache; =20 static size_t setup_test_cache(struct kunit *test, size_t size, slab_flags= _t flags, - void (*ctor)(void *)) + int (*ctor)(void *)) { if (test->priv !=3D TEST_PRIV_WANT_MEMCACHE) return size; @@ -539,10 +539,11 @@ static void test_shrink_memcache(struct kunit *test) KUNIT_EXPECT_FALSE(test, report_available()); } =20 -static void ctor_set_x(void *obj) +static int ctor_set_x(void *obj) { /* Every object has at least 8 bytes. */ memset(obj, 'x', 8); + return 0; } =20 /* Ensure that SL*B does not modify KFENCE objects on bulk free. */ diff --git a/mm/rmap.c b/mm/rmap.c index 67bb273dfb80..f2b45caa5acb 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -448,13 +448,14 @@ void unlink_anon_vmas(struct vm_area_struct *vma) } } =20 -static void anon_vma_ctor(void *data) +static int anon_vma_ctor(void *data) { struct anon_vma *anon_vma =3D data; =20 init_rwsem(&anon_vma->rwsem); atomic_set(&anon_vma->refcount, 0); anon_vma->rb_root =3D RB_ROOT_CACHED; + return 0; } =20 void __init anon_vma_init(void) diff --git a/mm/shmem.c b/mm/shmem.c index 99327c30507c..50ac15dfaff8 100644 --- a/mm/shmem.c +++ b/mm/shmem.c @@ -5165,10 +5165,11 @@ static void shmem_destroy_inode(struct inode *inode) simple_offset_destroy(shmem_get_offset_ctx(inode)); } =20 -static void shmem_init_inode(void *foo) +static int shmem_init_inode(void *foo) { struct shmem_inode_info *info =3D foo; inode_init_once(&info->vfs_inode); + return 0; } =20 static void __init shmem_init_inodecache(void) diff --git a/mm/slab.h b/mm/slab.h index 05a21dc796e0..30603907d936 100644 --- a/mm/slab.h +++ b/mm/slab.h @@ -278,7 +278,7 @@ struct kmem_cache { struct kmem_cache_order_objects min; gfp_t allocflags; /* gfp flags to use on each alloc */ int refcount; /* Refcount for slab cache destroy */ - void (*ctor)(void *object); /* Object constructor */ + int (*ctor)(void *object); /* Object constructor */ unsigned int inuse; /* Offset to metadata */ unsigned int align; /* Alignment */ unsigned int red_left_pad; /* Left redzone padding size */ @@ -438,10 +438,10 @@ extern void create_boot_cache(struct kmem_cache *, co= nst char *name, =20 int slab_unmergeable(struct kmem_cache *s); struct kmem_cache *find_mergeable(unsigned size, unsigned align, - slab_flags_t flags, const char *name, void (*ctor)(void *)); + slab_flags_t flags, const char *name, int (*ctor)(void *)); struct kmem_cache * __kmem_cache_alias(const char *name, unsigned int size, unsigned int align, - slab_flags_t flags, void (*ctor)(void *)); + slab_flags_t flags, int (*ctor)(void *)); =20 slab_flags_t kmem_cache_flags(slab_flags_t flags, const char *name); =20 diff --git a/mm/slab_common.c b/mm/slab_common.c index 5be257e03c7c..59938e44a8c2 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -173,7 +173,7 @@ int slab_unmergeable(struct kmem_cache *s) } =20 struct kmem_cache *find_mergeable(unsigned int size, unsigned int align, - slab_flags_t flags, const char *name, void (*ctor)(void *)) + slab_flags_t flags, const char *name, int (*ctor)(void *)) { struct kmem_cache *s; =20 @@ -382,7 +382,7 @@ static struct kmem_cache *kmem_buckets_cache __ro_after= _init; kmem_buckets *kmem_buckets_create(const char *name, slab_flags_t flags, unsigned int useroffset, unsigned int usersize, - void (*ctor)(void *)) + int (*ctor)(void *)) { unsigned long mask =3D 0; unsigned int idx; diff --git a/mm/slub.c b/mm/slub.c index 95a9f04b5904..10b9c87792b7 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -2439,7 +2439,8 @@ static void *setup_object(struct kmem_cache *s, void = *object) object =3D kasan_init_slab_obj(s, object); if (unlikely(s->ctor)) { kasan_unpoison_new_object(s, object); - s->ctor(object); + if (s->ctor(object)) + return NULL; kasan_poison_new_object(s, object); } return object; @@ -2542,7 +2543,7 @@ static bool should_shuffle_freelist(struct kmem_cache= *s, struct slab *slab) } =20 /* Shuffle the single linked freelist based on a random pre-computed seque= nce */ -static void shuffle_freelist(struct kmem_cache *s, struct slab *slab) +static bool shuffle_freelist(struct kmem_cache *s, struct slab *slab) { void *start; void *cur; @@ -2559,15 +2560,29 @@ static void shuffle_freelist(struct kmem_cache *s, = struct slab *slab) cur =3D next_freelist_entry(s, &pos, start, page_limit, freelist_count); cur =3D setup_object(s, cur); slab->freelist =3D cur; + if (!cur) { + return false; + } =20 for (idx =3D 1; idx < slab->objects; idx++) { next =3D next_freelist_entry(s, &pos, start, page_limit, freelist_count); next =3D setup_object(s, next); + if (!next) { + /* + * This is necessary because later we need to call + * the destructor for objects that the constructor + * successfully initialized. + */ + set_freepointer(s, cur, NULL); + return false; + } set_freepointer(s, cur, next); cur =3D next; } set_freepointer(s, cur, NULL); + + return true; } #else static inline int init_cache_random_seq(struct kmem_cache *s) @@ -2580,8 +2595,10 @@ static inline bool should_shuffle_freelist(struct km= em_cache *s, { return false; } -static inline void shuffle_freelist(struct kmem_cache *s, struct slab *sla= b) -{ } +static inline bool shuffle_freelist(struct kmem_cache *s, struct slab *sla= b) +{ + return false; +} #endif /* CONFIG_SLAB_FREELIST_RANDOM */ =20 static __always_inline void account_slab(struct slab *slab, int order, @@ -2604,6 +2621,20 @@ static __always_inline void unaccount_slab(struct sl= ab *slab, int order, -(PAGE_SIZE << order)); } =20 +static void __free_slab(struct kmem_cache *s, struct slab *slab) +{ + struct folio *folio =3D slab_folio(slab); + int order =3D folio_order(folio); + int pages =3D 1 << order; + + __slab_clear_pfmemalloc(slab); + folio->mapping =3D NULL; + __folio_clear_slab(folio); + mm_account_reclaimed_pages(pages); + unaccount_slab(slab, order, s); + free_frozen_pages(&folio->page, order); +} + static struct slab *allocate_slab(struct kmem_cache *s, gfp_t flags, int n= ode) { struct slab *slab; @@ -2653,14 +2684,22 @@ static struct slab *allocate_slab(struct kmem_cache= *s, gfp_t flags, int node) setup_slab_debug(s, slab, start); =20 if (should_shuffle_freelist(s, slab)) { - shuffle_freelist(s, slab); + if (!shuffle_freelist(s, slab)) + goto err; } else { start =3D fixup_red_left(s, start); start =3D setup_object(s, start); slab->freelist =3D start; + if (!start) + goto err; for (idx =3D 0, p =3D start; idx < slab->objects - 1; idx++) { next =3D p + s->size; next =3D setup_object(s, next); + if (!next) { + /* See comment in shuffle_freelist() */ + set_freepointer(s, p, NULL); + goto err; + } set_freepointer(s, p, next); p =3D next; } @@ -2668,6 +2707,9 @@ static struct slab *allocate_slab(struct kmem_cache *= s, gfp_t flags, int node) } =20 return slab; +err: + __free_slab(s, slab); + return NULL; } =20 static struct slab *new_slab(struct kmem_cache *s, gfp_t flags, int node) @@ -2681,20 +2723,6 @@ static struct slab *new_slab(struct kmem_cache *s, g= fp_t flags, int node) flags & (GFP_RECLAIM_MASK | GFP_CONSTRAINT_MASK), node); } =20 -static void __free_slab(struct kmem_cache *s, struct slab *slab) -{ - struct folio *folio =3D slab_folio(slab); - int order =3D folio_order(folio); - int pages =3D 1 << order; - - __slab_clear_pfmemalloc(slab); - folio->mapping =3D NULL; - __folio_clear_slab(folio); - mm_account_reclaimed_pages(pages); - unaccount_slab(slab, order, s); - free_frozen_pages(&folio->page, order); -} - static void rcu_free_slab(struct rcu_head *h) { struct slab *slab =3D container_of(h, struct slab, rcu_head); @@ -6377,7 +6405,7 @@ void __init kmem_cache_init_late(void) =20 struct kmem_cache * __kmem_cache_alias(const char *name, unsigned int size, unsigned int align, - slab_flags_t flags, void (*ctor)(void *)) + slab_flags_t flags, int (*ctor)(void *)) { struct kmem_cache *s; =20 diff --git a/net/socket.c b/net/socket.c index 9a0e720f0859..d5843d9c6ab4 100644 --- a/net/socket.c +++ b/net/socket.c @@ -330,11 +330,12 @@ static void sock_free_inode(struct inode *inode) kmem_cache_free(sock_inode_cachep, ei); } =20 -static void init_once(void *foo) +static int init_once(void *foo) { struct socket_alloc *ei =3D (struct socket_alloc *)foo; =20 inode_init_once(&ei->vfs_inode); + return 0; } =20 static void init_inodecache(void) diff --git a/net/sunrpc/rpc_pipe.c b/net/sunrpc/rpc_pipe.c index eadc00410ebc..d90d5f9b2fd7 100644 --- a/net/sunrpc/rpc_pipe.c +++ b/net/sunrpc/rpc_pipe.c @@ -1467,7 +1467,7 @@ static struct file_system_type rpc_pipe_fs_type =3D { MODULE_ALIAS_FS("rpc_pipefs"); MODULE_ALIAS("rpc_pipefs"); =20 -static void +static int init_once(void *foo) { struct rpc_inode *rpci =3D (struct rpc_inode *) foo; @@ -1476,6 +1476,7 @@ init_once(void *foo) rpci->private =3D NULL; rpci->pipe =3D NULL; init_waitqueue_head(&rpci->waitq); + return 0; } =20 int register_rpc_pipefs(void) diff --git a/security/integrity/ima/ima_iint.c b/security/integrity/ima/ima= _iint.c index 00b249101f98..f2af2ee91481 100644 --- a/security/integrity/ima/ima_iint.c +++ b/security/integrity/ima/ima_iint.c @@ -123,11 +123,12 @@ void ima_inode_free_rcu(void *inode_security) ima_iint_free(*iint_p); } =20 -static void ima_iint_init_once(void *foo) +static int ima_iint_init_once(void *foo) { struct ima_iint_cache *iint =3D (struct ima_iint_cache *)foo; =20 memset(iint, 0, sizeof(*iint)); + return 0; } =20 void __init ima_iintcache_init(void) --=20 2.43.0 From nobody Sun Feb 8 12:14:42 2026 Received: from mx0b-00069f02.pphosted.com (mx0b-00069f02.pphosted.com [205.220.177.32]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0692C20A5E1; Thu, 24 Apr 2025 08:20:51 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=205.220.177.32 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1745482854; cv=fail; b=WO1/dopd4VIItTS6e/RrcbndiZSlN18h3tta3Y9PQbN8hW25/rbhEpDT3ctreN37Uf2D1CGwSnErZrgJJLhnOJYCTVeGuZFJuvEvZKqpoRpclFH8b3qaQDfuUNO3FgPnb9iGQ0Pj9JdFyugx1iiX0KKw8lQEr+dRgYrv1ommG4M= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1745482854; c=relaxed/simple; bh=LrIVc7bfPdLmgfDoQdveyTvE54xhKRw7KWUbO/3EASg=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: Content-Type:MIME-Version; b=ESR4izMKmiSu5iusPBCzNBqZfLxsILNjnnYfZi3eoA9xe/GstiovBz6nsA+lWu736+cj4yYa2qT9pwbKSphIcvZhKopdniDpMfPD83Hcji39H5FIhI3AIISrgGxbHlz2ZDHq/Ote4Vl8Va63sAzEDEDO3+JAOeaQ5Fy9zU8MSgk= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=oracle.com; spf=pass smtp.mailfrom=oracle.com; dkim=pass (2048-bit key) header.d=oracle.com header.i=@oracle.com header.b=JC1Tguwe; dkim=pass (1024-bit key) header.d=oracle.onmicrosoft.com header.i=@oracle.onmicrosoft.com header.b=aZ6wJCED; arc=fail smtp.client-ip=205.220.177.32 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=oracle.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=oracle.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=oracle.com header.i=@oracle.com header.b="JC1Tguwe"; dkim=pass (1024-bit key) header.d=oracle.onmicrosoft.com header.i=@oracle.onmicrosoft.com header.b="aZ6wJCED" Received: from pps.filterd (m0333520.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.18.1.2/8.18.1.2) with ESMTP id 53O6ttR4005503; Thu, 24 Apr 2025 08:20:33 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=cc :content-transfer-encoding:content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to; s= corp-2023-11-20; bh=aog4JoMrhgq7+h5K2ucK6aO98+XcuAAgabvG8owueYA=; b= JC1Tguwef7cUYpy/BG4D/IN+Lnw3DDMNIkp7pd6LOXKBEvqtZ/Q0ow0trUTqKIaZ vVixCaaHrJAQlytiAw3IMGDSLi+ygrKUfr0ICq1AWgLFHsTMo4EJeAhEhBvz8ESR tvk70RHxpAAwerne0M+Vf2fUGRKGJdRSnneJX+VK+j2kK2jdDevqluJakUWol/aj loabtboDlMz5jedzGkBXEqoe2JczAbyMaSaUG4WGARGl6p/OZ6iMlTckz7zG1Ike PQhF1XtRL0qW2aK7R9n74YrnbaQrpGMTMyxDitP4NjSLNneLOi7ybtObNI4ZAUJQ u5NYJ0Xu0oJrrUY+c7dnFQ== Received: from phxpaimrmta02.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta02.appoci.oracle.com [147.154.114.232]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 467fmk86sf-10 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 24 Apr 2025 08:20:32 +0000 (GMT) Received: from pps.filterd (phxpaimrmta02.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by phxpaimrmta02.imrmtpd1.prodappphxaev1.oraclevcn.com (8.18.1.2/8.18.1.2) with ESMTP id 53O8832f030957; Thu, 24 Apr 2025 08:08:36 GMT Received: from ch5pr02cu005.outbound.protection.outlook.com (mail-northcentralusazlp17012050.outbound.protection.outlook.com [40.93.20.50]) by phxpaimrmta02.imrmtpd1.prodappphxaev1.oraclevcn.com (PPS) with ESMTPS id 466k06xrde-3 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 24 Apr 2025 08:08:36 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=dbAIuWoEk5+7SoqYkJvGpJ7mPNJhYt/OnoU7V8RNeiTMEL9E85FhBwqlVzIqXdLF2V/BDmDRcuex4ThO3CEtQdiwG0GZPcjJ8EcGFwiFe6nKu7mgt10YqpgABOmE06TcQOJldsAI+LmeehS5yTBRMD+lf5Wgc++3osBU8M9TXARscPi12OBd8EWlifxoy07Mj4mJjagtqORG14K//u9Ml7J4Vw6TI9HjnWI/EhvDVkPJRB+9IJZzNFNZ8PNd82sNZnbk5BR+grswwTJN3xd5TfW9sT91OvJckTRVGG39Vfj+4b0ZLM9blgC5jaIJeZR4UlWsxPvI+YyHyZn8E6YZKg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=aog4JoMrhgq7+h5K2ucK6aO98+XcuAAgabvG8owueYA=; b=DrYvIAGXAM+KwQ6Ssog4R5mEEUTb9weIt6V1KF6VjxDn1bqK01p9tzPyobrLckcO+6L3SqtjL8aJEk/cy1fjad6odDglS+xZUOsyAawPoh+vG9l2gsFjkzO23RDup4kRiKpb6fpJYajqWXd2L/4aC1yNo3dVZtzkag9Fw5Xy/SOXHAbyn8fNp3hpILIEUAxwBvUbHDrn1rtQBDib7Ehoo/bDmfBGYvd2X+Q7xKsGyeK4nUGwm00Z2q2fUO3YK3zvfS70viXCsAXxx78HMESYaeWiUx6Axi67DUaZxojH8G38SHYl36IN0JB4wv/C3Ig0w9YvEccj2E1G9Q+snFUt/w== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=aog4JoMrhgq7+h5K2ucK6aO98+XcuAAgabvG8owueYA=; b=aZ6wJCEDRefC2NYrLp7k5H4LldOnHuAb/WBXutrFEuq5uDREF3+UWT/v8YqwLuG8FqY9crpNb6vf/M2llivomRDH1AtM0973l4w8yvDqS090OWFvU8Iz3XmBPIQ4fjwE23tpw+cESHENQaGuKX2MEtZaO6rUQXILuiQ8vPE8FQc= Received: from CH3PR10MB7329.namprd10.prod.outlook.com (2603:10b6:610:12c::16) by LV3PR10MB8156.namprd10.prod.outlook.com (2603:10b6:408:285::9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8678.23; Thu, 24 Apr 2025 08:08:20 +0000 Received: from CH3PR10MB7329.namprd10.prod.outlook.com ([fe80::f238:6143:104c:da23]) by CH3PR10MB7329.namprd10.prod.outlook.com ([fe80::f238:6143:104c:da23%7]) with mapi id 15.20.8678.021; Thu, 24 Apr 2025 08:08:20 +0000 From: Harry Yoo To: Vlastimil Babka , Christoph Lameter , David Rientjes , Andrew Morton , Dennis Zhou , Tejun Heo , Mateusz Guzik Cc: Jamal Hadi Salim , Cong Wang , Jiri Pirko , Vlad Buslov , Yevgeny Kliteynik , Jan Kara , Byungchul Park , linux-mm@kvack.org, netdev@vger.kernel.org, linux-kernel@vger.kernel.org, Harry Yoo Subject: [RFC PATCH 3/7] mm/slab: revive the destructor feature in slab allocator Date: Thu, 24 Apr 2025 17:07:51 +0900 Message-ID: <20250424080755.272925-4-harry.yoo@oracle.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20250424080755.272925-1-harry.yoo@oracle.com> References: <20250424080755.272925-1-harry.yoo@oracle.com> Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: SL2PR03CA0006.apcprd03.prod.outlook.com (2603:1096:100:55::18) To CH3PR10MB7329.namprd10.prod.outlook.com (2603:10b6:610:12c::16) Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CH3PR10MB7329:EE_|LV3PR10MB8156:EE_ X-MS-Office365-Filtering-Correlation-Id: 0ea955fb-4da6-4949-1bde-08dd83072d32 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|376014|7416014|1800799024|366016; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?EyRiB0O6IHR2jArXDG46wI7V+LYfklYOfnA5/oQuX99ay/bMRcnCC+QDBP83?= =?us-ascii?Q?1rcJOHTfaPfbkeCboIKz3d1KIeA9OAOy6mokZgNN484+YT0iGYjwleyoBirS?= =?us-ascii?Q?xolzxZNayZzTLB6HjULHZ+5dWh0bO+0qaQ92Ha5eniPLgd2MWB0q+9erMutE?= =?us-ascii?Q?msVxp4/vkEyGE91Gp2nRYT13lnvjbdo76IrAY5gCdugzMBvQSx2gut6AZ+SJ?= =?us-ascii?Q?ENBCGcPCspQuWoYNNudNV3Iq/aNi/zmO445dpdnN/Hy/eVPGoU1pBGbnv+9n?= =?us-ascii?Q?4ahrjpxuiMrH22EB5YXuyKZakYdB+utxW9h4aNYA62CXtV1Fhd/krfxzoaCw?= =?us-ascii?Q?VrbV8SuphVUrEf+qPCcBvR2NgyGG9FmDx71tdCQt3QGYgl30jxSb33RfemgP?= =?us-ascii?Q?XBa1UOU/ABArsZb5fVsYGBwJ1Xrji5ctQzjb7UvK8S0jhl8W2q7rR5HGTnUb?= =?us-ascii?Q?zL0Ve3V+mu2FsVAtbYIIBmFvzxqj2QZ43WqwcK2gl77U1DEJhCDrBSss4pOR?= =?us-ascii?Q?fwFWmu8PX8E1IZVmuSHVU++7pUs0++JfOGZ1xV3J8iQU+CRfF8pzmsm6Ok67?= =?us-ascii?Q?Xv03I5m+4CXz9AoyGm3Nq03RaBeWEKwosop5kkFnG9tm20TxZU81zdNzM7un?= =?us-ascii?Q?MUZU4L9GqH/yqCAcHRSDeKq+QyQ1ibPFvZA356cs3hbce67Sz9St1tE+SCN8?= =?us-ascii?Q?tn2IGu4le1kWNs8nWDnnki6yPW6A6SXi2KR9m00pNL3AS4oVGrVeVcnNDW0o?= =?us-ascii?Q?TsTbLtvPM9flJmJ3hmH5hXXoSrh5CYeeUASrAHOxqcgxb+T3Hg6qu8X9zHGb?= =?us-ascii?Q?oEk4RXOqaDnLVI2wy4k9gUGvRb5g7BVDlhejqCCY2+tSFCv5dqtXzj1IUouJ?= =?us-ascii?Q?1mH0+juE/zGcynTbmx+a0USc+tOgSebB52VfyUI96LfOC6dmgc+LFS49Wrvt?= =?us-ascii?Q?K21siLP3ENYfPM72RVDsc/Gpw69snYfTKNSrtNOEwjX/GIRHX6QaH5wn62eL?= =?us-ascii?Q?f9WsItN5qhtDmMj2jZLn4HHnUyV3xy/T/zN70hmU47oVvvGitv7+7G+uFwtA?= =?us-ascii?Q?fjwrZD6ECCSNqxaH3vdVhMWShkcSbScXLrfOtx2rmPySa+OLhoQhiRQGcwGM?= =?us-ascii?Q?8jm/tLfT5qEp6jb05NQPd6uiv7Ok/GABF6Udp4wsYQqnZizTPZTIWaEYDhIX?= =?us-ascii?Q?KtwuRzWVLCcxfDMnhboTj9y1UlZ1mX4BksdV0gVhE0/SETUwT8B3oAnfkXHD?= =?us-ascii?Q?Av1xKs0j2hQ2TsSnWav70IOOggC7L3l+3HSebpfXr3Z401avg/qsuwOorDSy?= =?us-ascii?Q?6SKJ7kvfTSUlAn00IcDCVeUve2VR1kYTQ5Le7IX3jBLhGjr8sdd5gP5EdayX?= =?us-ascii?Q?HDKeir9aZP15uVeD9kUY/BY5riB4+C6yh3Vfvfy/JO2skssUPeDZ7dyc776N?= =?us-ascii?Q?AsuXgqh6ems=3D?= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:CH3PR10MB7329.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(376014)(7416014)(1800799024)(366016);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?CG+PMCbKLvt5YpOcPcUeeRYJlbc5xhgWsoSUAA+/Np16JXZSxv3JIgDRTy/j?= =?us-ascii?Q?pv8iPaHtbYVMscLy3ZPMum5DkZoTusMM2KBArwPpHrhZ4pTW/mzPX3+HdxEy?= =?us-ascii?Q?1MFLakLxJg8KSYmGbcqt65y744pbJqXRsh7ahFao8HgQ8THR0Rn30Qbb2m8y?= =?us-ascii?Q?427jgfWuPLsVZHzSzFl60F45SnloyHAMZLAI5Brk6GRx55BbqOiqHVHDgWXR?= =?us-ascii?Q?nVq2+yBFMzaAn+hphIxXQSmZaFl7pE9OdLlFgx3mgJcFHqua2/hhgQGLTvgZ?= =?us-ascii?Q?GEcPoEFDuaFU+0J+HlCBCNRfZ5UIswQr9Wb2qO6oLfULIGUMcw9t8tIcl0A8?= =?us-ascii?Q?eSt+c98UifjC2j/RcUfv/28KE/qaWdxhyv/F/BIv9F5jhLeJLRNJtt0ypkKa?= =?us-ascii?Q?lMvOpQoG8eYD0qEEc5P0J0vJLcyAJofZ6V9j30lCazYgqPgVO+7iK2ZE42GR?= =?us-ascii?Q?BHbzsjCIQ9kayWfxariEs3zl4Q30d4aNkkM9OsVGFoeUsKJ5M1nEjYHtBWOU?= =?us-ascii?Q?xTeHMvEnUUys6TuxfDBckiOlj4hpKrBjac4wBK6n7wr42XIyJXNjHfCpinWP?= =?us-ascii?Q?qlLcPz3JfwGen4d5sfRZGivwRR8caLZwAIw3iq1UGgDJxpj9lqwUUWDJYSRu?= =?us-ascii?Q?TTMXZSgrMDsHD5o5EsDvjG4UbSwh7yCnF8Nxwig/LQ7Gnen+kTJZBBJdOo0O?= =?us-ascii?Q?3qlYiboIhC69dRunncg+oSTzKcm8Nj56q8/3zl5qPYByRKUGZSsR0MO7SJkU?= =?us-ascii?Q?98oLbWdq8N72dj+ZNhnl+BeTqVQdneclJLc0kyktUK1H/eqmhaGgjhUbyCH1?= =?us-ascii?Q?IVi71ZMKUU1MO+t61GmGWEcFxmmY9tjPfQyN0hvh4+eOzNW2q+OZ/82c+psp?= =?us-ascii?Q?JjpACgVdTubiuClfKIApxaonmcnJMZX4p+aUJbjrrTVMuCeCn+6qTBNqJ3rO?= =?us-ascii?Q?p2CPLxTzmVJJInZ4hYuJmXT3dd3Ei2APnmn3GgemDyxuSBRjH4N2HAV7Ab5C?= =?us-ascii?Q?HSZGFhIF7uwS19ICjerjiyrcBZ0fVikxsQfB6NM6KxogvgvhyHJgZCsVT6DW?= =?us-ascii?Q?TQesGIr/fpPXRbCDZgsw1brbyEdlSfSxv877iiIWE61aH67UYng9nXCt+dBp?= =?us-ascii?Q?tVfsGg6o9EpbbysqxfRl0/o+39Y5Qmmalh26Zy9YJsXTff/Z1TZhVKmT5AaD?= =?us-ascii?Q?zxwGPimnB0j6cohUaUOjDM43gvkbjodX6zYgIQz++UiDTwZHxRDTKeVSD4Ok?= =?us-ascii?Q?iRGVQ11PaNmkcfkRPnrn3FVCBV/ef1tlpWqz/Q8knoXsLQi/xkjwTbSG5vdF?= =?us-ascii?Q?qf1HDm2ctVvDVVmrXd9fRkY/HoobytBgHU/raohilOZd4Enoq5StK0nxDw4p?= =?us-ascii?Q?3W95aKbnNYpo84srQOEp1/hXkNcgkZtYCM9bI9xuwDxp+VCbqfsKRz4nG9rS?= =?us-ascii?Q?0I7QbiWcYzX8Ob3QZjUDrlce+lQ9x1IHeqtGDxGC/uN7hhVmXLw1NlfCxCwh?= =?us-ascii?Q?T///fsP/tCcEtuNCGhZ9jpZl0vOLdCHOBZFQNOqAjGThnSmDsGGd8YgwNFBv?= =?us-ascii?Q?YKKEmdZxlWDQ5bwCgwceJ3FB+ebG1/lgnajNyzNE?= X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: Yt/gTetmstNudtAOjGqRgkevsxcjUiDpt48nk88vTWF0dOka8WFeQ05o5HUZQpVnGAMerUAoU8H/5cAFgSFxaVFK3xcX6Q+Fvdii9Np+cLEM+89ZiePpvZlrRNB8oPvzxp8g95cJpJZMsAF2okUzqcpCnnycn60oBnvfNifaHaKl97fRFoCdm9vrIBE7+nM9QKOXoWQgAMYbx2aZL7G8w6u/CtCQUkXPiYmnwtrHPegXel+iYSmPIhCDlUaDMv/uRCtMXus6+ksmYOhmteotuAPlkOHrsM9OKK7pyBtKC61sgQO0Szhfz9D/30BL9cMR8k12Vl3fw82NRu2PVvaG61E+OeemugiR0AogXJGlabiLlDwMvTjznfk9elwfpc4HVYcYZBymqOyvyI0WNJrGuSYXwfAySleyHjeMA0YuS5x34HLXlerIcx3qMAcHkq3urtf4hIWla04vexa9YeAxj+0BCbYq3gjcBTM++ERfMx7EyFYnO1mXQ3MfIJP4U3tKmcjBo2tkGt2sX80lWdl31+EYxWl9aL+XfYtWtvsN/4uu/x4mnFN9voBRJKrL5/9/yJwF3Ixp8s/vwivvDBv1YT4T2m0bSJCd9FG7cwmUlz0= X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 0ea955fb-4da6-4949-1bde-08dd83072d32 X-MS-Exchange-CrossTenant-AuthSource: CH3PR10MB7329.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 24 Apr 2025 08:08:20.4172 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: k0x24tZhiSF5/Q1sPEg4PyLt4AjdngpmlryWdU7pN1D1JoCidA3Ef8IAgrhn96TAlzh8ZYehoagGzTBaTkvLCQ== X-MS-Exchange-Transport-CrossTenantHeadersStamped: LV3PR10MB8156 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1099,Hydra:6.0.680,FMLib:17.12.80.40 definitions=2025-04-24_04,2025-04-22_01,2025-02-21_01 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 phishscore=0 bulkscore=0 mlxscore=0 spamscore=0 malwarescore=0 mlxlogscore=999 suspectscore=0 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2504070000 definitions=main-2504240053 X-Proofpoint-GUID: opq1G5kMpRkB9bMtpFthF1kXDRbuXE3r X-Proofpoint-ORIG-GUID: opq1G5kMpRkB9bMtpFthF1kXDRbuXE3r X-Proofpoint-Spam-Details-Enc: AW1haW4tMjUwNDI0MDA1MyBTYWx0ZWRfXynVW9g7MjHg0 rvxKbH/9h8q3gsTDOkr1GJ1zzKr94/FSJxDf06BoSWUvOF3KQlxFM3zeIHpySHj7oopa1ivComv 9e3uDq5sRVnKG8VjQcniy6imGHGlMlhXDtrGZ+DdxjjQQrklz7lL6SpMuSaRUkkORfdFeGYrExr JTDiDDX/bTrnqvxKeU4PbfU1wn5Nouw+RU8yFBXh+eZ8sdig1CH3nr8Ksz/YN4hePXdZFPl4+BZ cau33f2A5Awsqy+Byt7w42MTSh+rn8SpruaqdH8dqeg0Z1bvHr56jOZ2XXDapFMK5uiyYT69J+Y CM9AYvxYbS3hTdcDC31zV7ouLSQ3/rXfxHgWwpJVDxTDph/5vRzQd0rUgLtcXeNjtS+eGTRUdIU rVEh/7O3 Content-Type: text/plain; charset="utf-8" Commit c59def9f222d ("Slab allocators: Drop support for destructors") removed support for destructors, based on the belief that the feature had few users and its usefulness was questionable. Mateusz Guzik suggests [1] that using a constructor-destructor pair can be beneficial when object initialization and de-initialization require global serialization. For example, mm_struct allocates per-CPU variables that rely on a global lock for serialization. With the constructor-destructor pair, the serialization occurs only when a slab is allocated or freed, rather than each time an individual object is allocated or freed. Introduce the destructor feature. A user can enable it by specifying the 'dtor' field in kmem_cache_args. When a cache uses a destructor, cache merging is disabled and the destructor is called before freeing a sla= b. In case of SLAB_TYPESAFE_BY_RCU, it's invoked after RCU grace period. Unlike the constructor, the destructor does not fail and it mustn't. init_on_alloc, init_on_free, placing free pointer within the object, slab merging, __GFP_ZERO, object poisoning do not work for caches with destructor, for the same reason as constructor. Meanwhile, refactor __kmem_cache_alias() to remove the check for the the mergeability of the cache. Instead, these checks are performed before calling __kmem_cache_alias(). [1] https://lore.kernel.org/linux-mm/CAGudoHFc+Km-3usiy4Wdm1JkM+YjCgD9A8dDK= Q06pZP070f1ig@mail.gmail.com Suggested-by: Mateusz Guzik Signed-off-by: Harry Yoo --- include/linux/slab.h | 10 ++++++++++ mm/slab.h | 9 +++++---- mm/slab_common.c | 41 +++++++++++++++++++++++++++-------------- mm/slub.c | 29 ++++++++++++++++++++++++----- 4 files changed, 66 insertions(+), 23 deletions(-) diff --git a/include/linux/slab.h b/include/linux/slab.h index 1ef6d5384f0b..12a8a6b07050 100644 --- a/include/linux/slab.h +++ b/include/linux/slab.h @@ -336,6 +336,16 @@ struct kmem_cache_args { * %NULL means no constructor. */ int (*ctor)(void *); + /** + * @dtor: A destructor for the objects. + * + * The destructor is invoked for each object when a slab page is freed. + * In case of &SLAB_TYPESAFE_BY_RCU caches, dtor is called after RCU + * grace period. + * + * %NULL means no destructor. + */ + void (*dtor)(void *); }; =20 struct kmem_cache *__kmem_cache_create_args(const char *name, diff --git a/mm/slab.h b/mm/slab.h index 30603907d936..cffe960f2611 100644 --- a/mm/slab.h +++ b/mm/slab.h @@ -279,6 +279,7 @@ struct kmem_cache { gfp_t allocflags; /* gfp flags to use on each alloc */ int refcount; /* Refcount for slab cache destroy */ int (*ctor)(void *object); /* Object constructor */ + void (*dtor)(void *object); /* Object destructor */ unsigned int inuse; /* Offset to metadata */ unsigned int align; /* Alignment */ unsigned int red_left_pad; /* Left redzone padding size */ @@ -438,10 +439,10 @@ extern void create_boot_cache(struct kmem_cache *, co= nst char *name, =20 int slab_unmergeable(struct kmem_cache *s); struct kmem_cache *find_mergeable(unsigned size, unsigned align, - slab_flags_t flags, const char *name, int (*ctor)(void *)); + slab_flags_t flags, const char *name); struct kmem_cache * __kmem_cache_alias(const char *name, unsigned int size, unsigned int align, - slab_flags_t flags, int (*ctor)(void *)); + slab_flags_t flags); =20 slab_flags_t kmem_cache_flags(slab_flags_t flags, const char *name); =20 @@ -638,7 +639,7 @@ static inline bool slab_want_init_on_alloc(gfp_t flags,= struct kmem_cache *c) { if (static_branch_maybe(CONFIG_INIT_ON_ALLOC_DEFAULT_ON, &init_on_alloc)) { - if (c->ctor) + if (c->ctor || c->dtor) return false; if (c->flags & (SLAB_TYPESAFE_BY_RCU | SLAB_POISON)) return flags & __GFP_ZERO; @@ -651,7 +652,7 @@ static inline bool slab_want_init_on_free(struct kmem_c= ache *c) { if (static_branch_maybe(CONFIG_INIT_ON_FREE_DEFAULT_ON, &init_on_free)) - return !(c->ctor || + return !(c->ctor || c->dtor || (c->flags & (SLAB_TYPESAFE_BY_RCU | SLAB_POISON))); return false; } diff --git a/mm/slab_common.c b/mm/slab_common.c index 59938e44a8c2..f2f1f7bb7170 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -155,7 +155,7 @@ int slab_unmergeable(struct kmem_cache *s) if (slab_nomerge || (s->flags & SLAB_NEVER_MERGE)) return 1; =20 - if (s->ctor) + if (s->ctor || s->dtor) return 1; =20 #ifdef CONFIG_HARDENED_USERCOPY @@ -172,22 +172,36 @@ int slab_unmergeable(struct kmem_cache *s) return 0; } =20 -struct kmem_cache *find_mergeable(unsigned int size, unsigned int align, - slab_flags_t flags, const char *name, int (*ctor)(void *)) +/** + * Can this cache that's going to be created merged with others? + * We can't use struct kmem_cache here because it is not created yet. + */ +static bool is_mergeable(const char *name, slab_flags_t flags, + struct kmem_cache_args *args) { - struct kmem_cache *s; - if (slab_nomerge) - return NULL; - - if (ctor) - return NULL; + return false; =20 flags =3D kmem_cache_flags(flags, name); - if (flags & SLAB_NEVER_MERGE) - return NULL; + return false; =20 + if (args->ctor || args->dtor) + return false; + +#ifdef CONFIG_HARDENED_USERCOPY + if (args->usersize) + return false; +#endif + return true; +} + +struct kmem_cache *find_mergeable(unsigned int size, unsigned int align, + slab_flags_t flags, const char *name) +{ + struct kmem_cache *s; + + flags =3D kmem_cache_flags(flags, name); size =3D ALIGN(size, sizeof(void *)); align =3D calculate_alignment(flags, align, size); size =3D ALIGN(size, align); @@ -321,9 +335,8 @@ struct kmem_cache *__kmem_cache_create_args(const char = *name, object_size - args->usersize < args->useroffset)) args->usersize =3D args->useroffset =3D 0; =20 - if (!args->usersize) - s =3D __kmem_cache_alias(name, object_size, args->align, flags, - args->ctor); + if (is_mergeable(name, flags, args)) + s =3D __kmem_cache_alias(name, object_size, args->align, flags); if (s) goto out_unlock; =20 diff --git a/mm/slub.c b/mm/slub.c index 10b9c87792b7..b7e158da7ed3 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -2626,6 +2626,15 @@ static void __free_slab(struct kmem_cache *s, struct= slab *slab) struct folio *folio =3D slab_folio(slab); int order =3D folio_order(folio); int pages =3D 1 << order; + void *p; + + if (unlikely(s->dtor)) { + p =3D slab->freelist; + while (p !=3D NULL) { + s->dtor(p); + p =3D get_freepointer(s, p); + } + } =20 __slab_clear_pfmemalloc(slab); folio->mapping =3D NULL; @@ -2717,7 +2726,7 @@ static struct slab *new_slab(struct kmem_cache *s, gf= p_t flags, int node) if (unlikely(flags & GFP_SLAB_BUG_MASK)) flags =3D kmalloc_fix_flags(flags); =20 - WARN_ON_ONCE(s->ctor && (flags & __GFP_ZERO)); + WARN_ON_ONCE((s->ctor || s->dtor) && (flags & __GFP_ZERO)); =20 return allocate_slab(s, flags & (GFP_RECLAIM_MASK | GFP_CONSTRAINT_MASK), node); @@ -5735,7 +5744,7 @@ static int calculate_sizes(struct kmem_cache_args *ar= gs, struct kmem_cache *s) * then we should never poison the object itself. */ if ((flags & SLAB_POISON) && !(flags & SLAB_TYPESAFE_BY_RCU) && - !s->ctor) + !s->ctor && !s->dtor) s->flags |=3D __OBJECT_POISON; else s->flags &=3D ~__OBJECT_POISON; @@ -5757,7 +5766,7 @@ static int calculate_sizes(struct kmem_cache_args *ar= gs, struct kmem_cache *s) s->inuse =3D size; =20 if (((flags & SLAB_TYPESAFE_BY_RCU) && !args->use_freeptr_offset) || - (flags & SLAB_POISON) || s->ctor || + (flags & SLAB_POISON) || s->ctor || s->dtor || ((flags & SLAB_RED_ZONE) && (s->object_size < sizeof(void *) || slub_debug_orig_size(s)))) { /* @@ -6405,11 +6414,11 @@ void __init kmem_cache_init_late(void) =20 struct kmem_cache * __kmem_cache_alias(const char *name, unsigned int size, unsigned int align, - slab_flags_t flags, int (*ctor)(void *)) + slab_flags_t flags) { struct kmem_cache *s; =20 - s =3D find_mergeable(size, align, flags, name, ctor); + s =3D find_mergeable(size, align, flags, name); if (s) { if (sysfs_slab_alias(s, name)) pr_err("SLUB: Unable to add cache alias %s to sysfs\n", @@ -6443,6 +6452,7 @@ int do_kmem_cache_create(struct kmem_cache *s, const = char *name, #endif s->align =3D args->align; s->ctor =3D args->ctor; + s->dtor =3D args->dtor; #ifdef CONFIG_HARDENED_USERCOPY s->useroffset =3D args->useroffset; s->usersize =3D args->usersize; @@ -7003,6 +7013,14 @@ static ssize_t ctor_show(struct kmem_cache *s, char = *buf) } SLAB_ATTR_RO(ctor); =20 +static ssize_t dtor_show(struct kmem_cache *s, char *buf) +{ + if (!s->dtor) + return 0; + return sysfs_emit(buf, "%pS\n", s->dtor); +} +SLAB_ATTR_RO(dtor); + static ssize_t aliases_show(struct kmem_cache *s, char *buf) { return sysfs_emit(buf, "%d\n", s->refcount < 0 ? 0 : s->refcount - 1); @@ -7356,6 +7374,7 @@ static struct attribute *slab_attrs[] =3D { &partial_attr.attr, &cpu_slabs_attr.attr, &ctor_attr.attr, + &dtor_attr.attr, &aliases_attr.attr, &align_attr.attr, &hwcache_align_attr.attr, --=20 2.43.0 From nobody Sun Feb 8 12:14:42 2026 Received: from mx0b-00069f02.pphosted.com (mx0b-00069f02.pphosted.com [205.220.177.32]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8FC6A1F4622; Thu, 24 Apr 2025 08:17:48 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=205.220.177.32 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1745482670; cv=fail; b=SSoi7Sv78zy6L6UCNlcGjB3Tf96z/5o3rdsz3r7VKFGl+UDTvQIe6PFWxSnaII1G2LV2ZY774aopCm9aJPBu+MorGnQyUIfk810rZd9WEVsqrvtMru+RNW2Md+jBZE7t6kB8HZIjd3p5GogHQOZ/mtrMq7fsBlwpHJYmtuGRkqY= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1745482670; c=relaxed/simple; bh=z6uJNGzYJDb9vV37HfPCCuFkbkOoi6AuCaMlMQ5jdGM=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: Content-Type:MIME-Version; b=ksQnE8wjlnQLd+0L1D2u9+iVXujYGL2hwNfUEx2gwhVg5SFzNGzys3KrmfHU/RfmEdNF1Y7dJgrYshqP8jhZlTxcljvmNf/U9YZoIzvqaAJxye2uNYxeufru01lOREcxzM+vJb21x32qrmQVqS5cEY/rL5EZku500J2ZSlQpQ/g= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=oracle.com; spf=pass smtp.mailfrom=oracle.com; dkim=pass (2048-bit key) header.d=oracle.com header.i=@oracle.com header.b=eHgsFupC; dkim=pass (1024-bit key) header.d=oracle.onmicrosoft.com header.i=@oracle.onmicrosoft.com header.b=Kk31Kqwk; arc=fail smtp.client-ip=205.220.177.32 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=oracle.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=oracle.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=oracle.com header.i=@oracle.com header.b="eHgsFupC"; dkim=pass (1024-bit key) header.d=oracle.onmicrosoft.com header.i=@oracle.onmicrosoft.com header.b="Kk31Kqwk" Received: from pps.filterd (m0333520.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.18.1.2/8.18.1.2) with ESMTP id 53O6ttQX005503; Thu, 24 Apr 2025 08:17:31 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=cc :content-transfer-encoding:content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to; s= corp-2023-11-20; bh=Bc2L65wxnUxBeI3n4mrAHo69R4uH/SWMKuyIpMBn6fE=; b= eHgsFupCGkZeusebKgmm6bplK7btRdr6WXrZnCxhmUzlTLDZHk/mAsKywoLR+kPI N9NWi74oqBkZuzBa3FU0/y9Qh3WxeGUkCps9jcXhdCePxdjhOq5FDXbcdV6Hlhdn YSZNhSHDR25bsbz4yOYFJN+4NxugH4BrPlUgnKe+pBVGSGkSil+W9h5MN1xaHAUa cvxBD/yxPRV/UQWI7Fb3ob3JbRKyovl3szmxU4Rw82CwJql/Cl+CXi/NtSyS7hXt /HragOLoovGBUEpwYJyvMY8WTDJm9ZPuUVI7WOS9afExTGxtd4hbUjWlAliZZDgN LMvYkgq0m0nOtk/uJ02hJA== Received: from phxpaimrmta02.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta02.appoci.oracle.com [147.154.114.232]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 467fmk86pj-14 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 24 Apr 2025 08:17:30 +0000 (GMT) Received: from pps.filterd (phxpaimrmta02.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by phxpaimrmta02.imrmtpd1.prodappphxaev1.oraclevcn.com (8.18.1.2/8.18.1.2) with ESMTP id 53O8832h030957; Thu, 24 Apr 2025 08:08:38 GMT Received: from ch5pr02cu005.outbound.protection.outlook.com (mail-northcentralusazlp17012050.outbound.protection.outlook.com [40.93.20.50]) by phxpaimrmta02.imrmtpd1.prodappphxaev1.oraclevcn.com (PPS) with ESMTPS id 466k06xrde-4 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 24 Apr 2025 08:08:37 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=vOI80bKt3sYlb3Rc/ps9eNpvC9Q0gCmM82mCaSf9Uc2tPbfxVA2d1i4jWCdm/Y9y1XX4lyNXGkxzckU0XBHeaP7H01Bzuoi8OtupNJqKVUVsXn12Ipr5u4OHoY3x62poTJf+Df0a1e82YxSdv2D0xxBDOD/yF75AHzyHcvKF9g9vi6V13PS5uIKp+8nT/gor8JvkPczerePAx3SL1WbgZq24sMuYbPDCSOwK7QYgEoyfxdl9CTW3Rf79cDmwaMkNd6sb5upmJ+lUEFaUZp8/ORBH99UcnFBQnWewFp08kcJqbPuexTOMWQXm7PDoLzlwOqo2B1Nm5qrrVAtiQcyztA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=Bc2L65wxnUxBeI3n4mrAHo69R4uH/SWMKuyIpMBn6fE=; b=VgZBxC0DMd3J5FYbUtkUUqQL85dNv1Fw5NkzTxZxeI3Q9K1Z/5UxysUzV4DLTI2D91xUrcv7QgBsrtYe7CfCfc7Rl6bXA3nLWw5y7+dKGh53Q69NRuXNPlSduM6n8K2HlOaJ0AaCRgIhCcrrRf0CY3B68qCpJPLvJj6q18OQmmmCch7WlvlMphCMLiu3bvQ84yTm9pXg1Gfxc7R0Z4zdnjTdig/txIzIGQsjb2f12TPAuAUz58ugAWpiqbcBkgHWtCTt/xRfl/+MM54PzwFT/QA4/2Py1pAejK9xqF/MQMzANm5PBVMZ5URpj+PjXzrS7Qv6WEII1ibrZoCLMXEVxQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=Bc2L65wxnUxBeI3n4mrAHo69R4uH/SWMKuyIpMBn6fE=; b=Kk31KqwktAhrujmIqm6x+Vq5C+Xof+1SFMTqPJabnVaSC03b68TjXe9ibgqCeHSw6bqtPhYfNGhm4jafvEh4KzVVtCSgyTGJ6Svz8+Iz4naf8JwNdEODuxT7ZErxs2ZrQ+oQzYZQRE1knSJVCXmlRHTXDqR4B7Vz6tYnG7zpOYI= Received: from CH3PR10MB7329.namprd10.prod.outlook.com (2603:10b6:610:12c::16) by LV3PR10MB8156.namprd10.prod.outlook.com (2603:10b6:408:285::9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8678.23; Thu, 24 Apr 2025 08:08:28 +0000 Received: from CH3PR10MB7329.namprd10.prod.outlook.com ([fe80::f238:6143:104c:da23]) by CH3PR10MB7329.namprd10.prod.outlook.com ([fe80::f238:6143:104c:da23%7]) with mapi id 15.20.8678.021; Thu, 24 Apr 2025 08:08:28 +0000 From: Harry Yoo To: Vlastimil Babka , Christoph Lameter , David Rientjes , Andrew Morton , Dennis Zhou , Tejun Heo , Mateusz Guzik Cc: Jamal Hadi Salim , Cong Wang , Jiri Pirko , Vlad Buslov , Yevgeny Kliteynik , Jan Kara , Byungchul Park , linux-mm@kvack.org, netdev@vger.kernel.org, linux-kernel@vger.kernel.org, Harry Yoo Subject: [RFC PATCH 4/7] net/sched/act_api: use slab ctor/dtor to reduce contention on pcpu alloc Date: Thu, 24 Apr 2025 17:07:52 +0900 Message-ID: <20250424080755.272925-5-harry.yoo@oracle.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20250424080755.272925-1-harry.yoo@oracle.com> References: <20250424080755.272925-1-harry.yoo@oracle.com> Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: SE2P216CA0177.KORP216.PROD.OUTLOOK.COM (2603:1096:101:2ca::6) To CH3PR10MB7329.namprd10.prod.outlook.com (2603:10b6:610:12c::16) Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CH3PR10MB7329:EE_|LV3PR10MB8156:EE_ X-MS-Office365-Filtering-Correlation-Id: 6eb647c8-0ecf-4fa4-ed69-08dd830731b1 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|376014|7416014|1800799024|366016; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?JHtVAr2dMi1xjAyBhlert4eli+pdw0gki3Clu9g9A4ka2euzm7yCED5B+GjD?= =?us-ascii?Q?nG2KR2YDrexQWzofdRVVY6cg3EWCaxmIU/Zw/JFQrS61KzG/iVB4Otlhyvg8?= =?us-ascii?Q?F7lOFTooXLfskDB8i85cgamVjJXP1o1dLAreI64S6GQ7bjG4CcXV5lNoRCFB?= =?us-ascii?Q?Um8pa4g4aqa6L/RnE1u9bPLDRZeERa4/0GCmcwVcdIDguFD43s2B7SnumOsU?= =?us-ascii?Q?2hLjF+zTNu0v8I8RkhiXEQAnuCWvcecIVFlbjgIKcHl497o6wz4yDHVNBmis?= =?us-ascii?Q?SiPoYA7KPu2QAc3aNdO3H8px4rQ4EkUmQ/dyCxwIxFsbUO+tGvKzCL1XpyUJ?= =?us-ascii?Q?DjboY9K9+UJKl5UzaCO5NveeEqn34EWHaGgQ8sUEk5iCy7dycMSxprUrNwFG?= =?us-ascii?Q?LyHvs9lkpiQcWTP50ViIE1BaS2aVtYPGPSe4eTkPyKcT4xWlpqKpMl7vr03r?= =?us-ascii?Q?HEHU1Cyy/GFTYgerUh/2kKUN1+u/U0v/bNgImGskJN3b87XkYE8NQVLP7B9e?= =?us-ascii?Q?m/QLHdgLW07bZ9pE4JdYD1aCEeGLLh5yY+XYnr3RqcnxQHPyHr5PRwtJLY17?= =?us-ascii?Q?JQhvj0zijuFpz14CzkNI8dyLFEQ1/W+Bw6aL6vEcH/gGV5TYwirMU+tl1fgB?= =?us-ascii?Q?tPFWXD41YqnlZebfXWnRxV8Zy5P7YTFmu8mYG4KhKggFNMAYoY4yfdiINzNx?= =?us-ascii?Q?PSzITfgPexfXQlq89BPzjnxMTMXVF/3Ionst6R90kmVrQctcC8z2BxsnDRVc?= =?us-ascii?Q?RtMoytxQiPGA5eB60oV332e9fLv7q8XCHAxNz/AoMVkCZLltLAFEufMtTJqH?= =?us-ascii?Q?cL0jRC9DQctI0B7geizBCR0EHrEZ55T8LXDtFarSgHnTR9lu6OlT3lFU/Hq2?= =?us-ascii?Q?xEqlfPNaL6M+ALzAGL0LqRsPoVOEkMKIN/FlxxOVN6hB9G1eA/rITE8pZexc?= =?us-ascii?Q?q8RaAGo/CXGjLVN9n8UcytV9M5QkcabRc5U4ZONIcelraJepW4aBym5/AkSh?= =?us-ascii?Q?qs3o79XI8r5nOuTI6y984VryXYbjudmBFBVw8sWSgLJspyLyaq23QatMwcIp?= =?us-ascii?Q?fnpVE0ZNXgmnOe5OZHBxtbegpSa+NTuOkwGFcoNzVq+gg9hws1Z0hHvwfiJ5?= =?us-ascii?Q?0PQCKw3j9Zs6PwhZ0aA+iOcw7YZsOujBMxp4NHhd7ybkpz++VCCk8mkPX7EJ?= =?us-ascii?Q?loX2MSkfF2nAtQlUVl3SnbHi+VAtTLaFxOMv5ftxq4Gdu8W3U8VrNsxlzCww?= =?us-ascii?Q?le0x0/ZfnDszWBsI5jaTov9wWNp2s/7Viey4IwKmFcwFuctLszYb57mfNaJM?= =?us-ascii?Q?mWt52VwbtO2YwHNLDDskfxv1Zr69jbZZvr8cpRAcp+h1GYv9NuHRomsExuBI?= =?us-ascii?Q?lEvlAIqC6nYj/og4hI/smJFIdwlR1PSxaYHV8KmGX/jnjQJfhJ9JgahzrQlL?= =?us-ascii?Q?olXbX7kk2PU=3D?= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:CH3PR10MB7329.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(376014)(7416014)(1800799024)(366016);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?P+MNZxspAmm6g986bH/g9ByUZLDTtMQuAFMsY7kF8sQ5v1FlYJ8RS0ciKsZR?= =?us-ascii?Q?eQXJjdhQkcqDze7Ffk0pgqYhs7FlmuzbEhRRUJ60u3dnlOA/Q2vQ38UfbU4M?= =?us-ascii?Q?vO0EDrXbq5Ok+J0XN0mjz5qlOfAV0GOqS2QXv2Yet0bsdRqp30/vMD+CyLnL?= =?us-ascii?Q?yTIQ5O0cNSLp15DAgbjnmi6Y1ME3V7Qm/ISDeF1tOQS8/D7gZw55FkGALEWm?= =?us-ascii?Q?swJ+sEgx3ps8Jpe14SQCkI9/69Y9+O0p8usn/fJCeoTzCISIPdouYS8U67RJ?= =?us-ascii?Q?bmBQZCFESjbfDmgSty4+DUOvTpMucaHEB1vRIfpdg/AJhtFv+UEvD7lM7APZ?= =?us-ascii?Q?5F2knUeRRZtuNtGAsqm1NfNZmPRmreiR5L/8igkNQ0DSETAXUxLiAvBXFgjV?= =?us-ascii?Q?5ujXVpvrqCaqsQZ+QfYVzP8IW0WAfgOXJM8C2LWw96jG+I3z/dkl7U4Qif0b?= =?us-ascii?Q?6loEwplIBtBlur3izppkySdh3hZZbmBxIggc5F8G0b0dfz7zSz/mDGLqh6bQ?= =?us-ascii?Q?YmmqQ+4Uj06FhQ25ry2SaLsomfFIAUydqc7rpeAtcLUGs4JDx10PPVcdprfn?= =?us-ascii?Q?rE7M8dmTsUsoWNr1hgzWfquvsCjuJUAEDUPJWRd65ED34BV4TRhkFaXEuLRK?= =?us-ascii?Q?kxnJXMD+Y/lUmHqu0bhIcZnybGrsyzQ7omyP5eVsEZxb/cU75/dAjcNO/COf?= =?us-ascii?Q?ERrLm/g/d/f+ImsgE5ZqAKItADdk3btYrXpS+1uknsatNp+cT9Qv24EDi2PW?= =?us-ascii?Q?2qtOLygfAn8WWqstXUNXTfl6sJBPQCpfJ5oqrf6hWJ6eQgJtAZqz2Lr0LRiZ?= =?us-ascii?Q?jd6QJt92Z/fwbHKCxd2ccDC78FnGpKMJiWY7wsqxZH5NXQw7Xni60Dcge0Sv?= =?us-ascii?Q?HoCdEMB7E1AgfPJYE3wV6utAbPozj48I5vEMujaDxZW11t2RbQqpJs9XyNDc?= =?us-ascii?Q?R/NieU/c24YKMlbGyfr6df9uExsTsKgnvB81lh4BIIpQ7ElTeZe8+AjTt0q2?= =?us-ascii?Q?GpkAHMPNavrtIT9nQ27On3TEh+42E3nbTgxcUjLqKZA7mdwM2g7JQIJPuFrd?= =?us-ascii?Q?RYldPvnDYXELeZ8mvOvzYFxcu1vXc3kvrMMH+2uDUQQI2VjOh8JRhJjYpxEW?= =?us-ascii?Q?/oc/32yUsT87vHm5FddMpjlq4yJwtqJkCuhXmcTLipXgj8U5uYSeK2sIopbh?= =?us-ascii?Q?yGObKOvdONtHqN1gqgKWuayQmcWvwLk39RncLvDWVDaGPwvnKzzAPozHRApD?= =?us-ascii?Q?TP0XML4PvNaTLfXm55CyubwgZPEVlD8EzpQMZgj43bZ0+jj63qVv9seiwC3i?= =?us-ascii?Q?tl7FHu23A6+aNs5M27HC+dlypfAZMt4j2umwLv9enGUtlkWG+Mr3CJ52PFQK?= =?us-ascii?Q?D8ttXIkVqrVPta2kTDQPi7Khcv08pubdx3cmaoffq2FFS/NeoCtl+Il5YOPZ?= =?us-ascii?Q?9q5lhOfGdBi9LKnyiNEGG+QSbUU1n61EeEGLbdYMf586MBlIblFlry0KCGDi?= =?us-ascii?Q?lK+ubl0wQ1z2JpY1ujVBXWSY1UN5vnAoujuTuFgc+W9HOsci8xpsv6YTeaoQ?= =?us-ascii?Q?93p99ALGzUYpWATn36h2wUQhEy+l6mk8QbA/TiGT?= X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: kF4hGBiK5zV5ao+8OsHZVIk86/ovTh+8BOx1PDvMycneprz8G8LHrdlz+iqX2jLnWatYVkNZifdI1HdhyhUz9OVYYJBI8b6+O01istBuNN5X6hCFXlhAqKQcsO3uKJSvjJuSa1tfnFSD9qjIjIPRFSeVa2gykbmHEpq8nIZS6ALxeSaXCwQyIxUjzh2GIvI8vC1TWHDhvNzUcXl9WoR6+FbAYBT20mvGjF1TmAHdUWkHtLjFvm1NDNuPfLBujo5vW7UcshmX/+muAchbJMxVbp98qhOl96wn6XC6RrvNcHxG+nQzSL2sL0yChaY2P+LZILF7r9s6Ll8udOGRD1IGnBTOh5i+Jl0wXFREC6TTiJhtx+agiK+b8/4WF2hjxZu0ZCvOd1zFaQtFbNyglQxAz5MyTS5iuvRGFHqoAyCXiLN+BX00urW6f/i5rb6NO1p3oqvasYHm+x7cYZg2eFcmfitx/oUKZTSwfyyh+AMnBGAdCh0NSf1otqRj7wabAbaFcBdymQkksX7sj3q/dQh2tI6GVtdlLbw0XjXhF0KhP++uTJ3ylnt+aCOvy7NLi+1I33A9P1sQjegZKvwaWpzZ7/XAJ15fcdp2ak0wHDRkKfs= X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 6eb647c8-0ecf-4fa4-ed69-08dd830731b1 X-MS-Exchange-CrossTenant-AuthSource: CH3PR10MB7329.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 24 Apr 2025 08:08:28.1415 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: krdnLpQqIlLAPATYGOQeuqqDW9JlCRRq+Y6aqo7lN3e02I9dUv+wLLhhCWWmGByvjzApdBkQo82Fp+5Og+Vgrg== X-MS-Exchange-Transport-CrossTenantHeadersStamped: LV3PR10MB8156 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1099,Hydra:6.0.680,FMLib:17.12.80.40 definitions=2025-04-24_04,2025-04-22_01,2025-02-21_01 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 phishscore=0 bulkscore=0 mlxscore=0 spamscore=0 malwarescore=0 mlxlogscore=999 suspectscore=0 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2504070000 definitions=main-2504240053 X-Proofpoint-GUID: UtXUFSJ4Jl60hY1YNT5uPOBa5aCIleKw X-Proofpoint-ORIG-GUID: UtXUFSJ4Jl60hY1YNT5uPOBa5aCIleKw X-Proofpoint-Spam-Details-Enc: AW1haW4tMjUwNDI0MDA1MyBTYWx0ZWRfX+iqNPwB+QO2U m8v99tiFZVVeBhUNSqrZOs9qiEblWFgPMou5iMUSxNXxRibaZROpVCW8PDPypddzv21ScCmTDOf LAXAdt9CFjbS92zqxLoidHsj28bLTj7lV2CuWNHmiVHJHCHxWFs85G4P2tPfXhPQ1MHx6zFp/yC 8NlMA+qbvGCf7ZRolOCrDbTMhQHpBpz6zezij5yg3ZVAHL2rYdAIxQfTiKSumX0tHOTovxgf16q 8AagxwgQy7Lv+imjlx9JnGWK6UnXA0+DxDcDDKwqFlG5jIhSNLrfamwIr2u3gibSm6zzosRL4hk 7RYfaMT99d2gdagBc4rGlm+OE/axJJNlxDms70W61FGL50iGzA0/cphNUeMZfmRvEi30Kewxv/8 suPKEpnE Content-Type: text/plain; charset="utf-8" A tc_action object allocates three percpu memory regions and stores their pointers within the object. For each object's lifetime, this requires acquiring and releasing the globak lock, pcpu_alloc_mutex three times. In workloads that frequently create and destroy TC filters, this leads to severe lock contention due to the globl lock. By using the slab constructor/destructor pair, the contention on pcpu_alloc_mutex is shifted to the creation and destruction of slabs (which contains multiple objects), which occur far less frequently than allocating and freeing individual tc_action objects. When tested with the following command, a 26% reduction in system time was observed. $ cd tools/testing/selftests/tc-testing $ sudo python3 tdc.py -f ./tc-tests/filters/flower.json -d Lock contention as measured with `perf lock record/report`: Before: Name acquired contended avg wait total wait ma= x wait min wait 15042346 15042346 3.82 us 57.40 s 4= .00 ms 316 ns pcpu_alloc_mutex 10959650 10959650 7.10 us 1.30 m 3= .76 ms 313 ns After: Name acquired contended avg wait total wait ma= x wait min wait 15488031 15488031 5.16 us 1.33 m 512409= 5.50 h 316 ns pcpu_alloc_mutex 7695276 7695276 3.39 us 26.07 s 4= .03 ms 284 ns The contention has moved from pcpu_alloc_mutex to other locks (which are not symbolized and appear as blank in the output above). Signed-off-by: Harry Yoo --- net/sched/act_api.c | 82 +++++++++++++++++++++++++++++++-------------- 1 file changed, 56 insertions(+), 26 deletions(-) diff --git a/net/sched/act_api.c b/net/sched/act_api.c index 839790043256..60cde766135a 100644 --- a/net/sched/act_api.c +++ b/net/sched/act_api.c @@ -112,6 +112,8 @@ struct tcf_chain *tcf_action_set_ctrlact(struct tc_acti= on *a, int action, } EXPORT_SYMBOL(tcf_action_set_ctrlact); =20 +static struct kmem_cache *tc_action_cache; + /* XXX: For standalone actions, we don't need a RCU grace period either, b= ecause * actions are always connected to filters and filters are already destroy= ed in * RCU callbacks, so after a RCU grace period actions are already disconne= cted @@ -121,15 +123,15 @@ static void free_tcf(struct tc_action *p) { struct tcf_chain *chain =3D rcu_dereference_protected(p->goto_chain, 1); =20 - free_percpu(p->cpu_bstats); - free_percpu(p->cpu_bstats_hw); - free_percpu(p->cpu_qstats); =20 tcf_set_action_cookie(&p->user_cookie, NULL); if (chain) tcf_chain_put_by_act(chain); =20 - kfree(p); + if (p->cpu_bstats) + kmem_cache_free(tc_action_cache, p); + else + kfree(p); } =20 static void offload_action_hw_count_set(struct tc_action *act, @@ -778,27 +780,20 @@ int tcf_idr_create(struct tc_action_net *tn, u32 inde= x, struct nlattr *est, struct tc_action **a, const struct tc_action_ops *ops, int bind, bool cpustats, u32 flags) { - struct tc_action *p =3D kzalloc(ops->size, GFP_KERNEL); + struct tc_action *p; struct tcf_idrinfo *idrinfo =3D tn->idrinfo; int err =3D -ENOMEM; =20 + if (cpustats) + p =3D kmem_cache_alloc(tc_action_cache, GFP_KERNEL); + else + p =3D kzalloc(ops->size, GFP_KERNEL); + if (unlikely(!p)) return -ENOMEM; refcount_set(&p->tcfa_refcnt, 1); if (bind) atomic_set(&p->tcfa_bindcnt, 1); - - if (cpustats) { - p->cpu_bstats =3D netdev_alloc_pcpu_stats(struct gnet_stats_basic_sync); - if (!p->cpu_bstats) - goto err1; - p->cpu_bstats_hw =3D netdev_alloc_pcpu_stats(struct gnet_stats_basic_syn= c); - if (!p->cpu_bstats_hw) - goto err2; - p->cpu_qstats =3D alloc_percpu(struct gnet_stats_queue); - if (!p->cpu_qstats) - goto err3; - } gnet_stats_basic_sync_init(&p->tcfa_bstats); gnet_stats_basic_sync_init(&p->tcfa_bstats_hw); spin_lock_init(&p->tcfa_lock); @@ -812,7 +807,7 @@ int tcf_idr_create(struct tc_action_net *tn, u32 index,= struct nlattr *est, &p->tcfa_rate_est, &p->tcfa_lock, false, est); if (err) - goto err4; + goto err; } =20 p->idrinfo =3D idrinfo; @@ -820,14 +815,11 @@ int tcf_idr_create(struct tc_action_net *tn, u32 inde= x, struct nlattr *est, p->ops =3D ops; *a =3D p; return 0; -err4: - free_percpu(p->cpu_qstats); -err3: - free_percpu(p->cpu_bstats_hw); -err2: - free_percpu(p->cpu_bstats); -err1: - kfree(p); +err: + if (cpustats) + kmem_cache_free(tc_action_cache, p); + else + kfree(p); return err; } EXPORT_SYMBOL(tcf_idr_create); @@ -2270,8 +2262,46 @@ static const struct rtnl_msg_handler tc_action_rtnl_= msg_handlers[] __initconst =3D .dumpit =3D tc_dump_action}, }; =20 +static int tcf_action_ctor(void *object) { + struct tc_action *p =3D object; + + p->cpu_bstats =3D netdev_alloc_pcpu_stats(struct gnet_stats_basic_sync); + if (!p->cpu_bstats) + goto err1; + p->cpu_bstats_hw =3D netdev_alloc_pcpu_stats(struct gnet_stats_basic_sync= ); + if (!p->cpu_bstats_hw) + goto err2; + p->cpu_qstats =3D alloc_percpu(struct gnet_stats_queue); + if (!p->cpu_qstats) + goto err3; + return 0; + +err3: + free_percpu(p->cpu_bstats_hw); +err2: + free_percpu(p->cpu_bstats); +err1: + return -ENOMEM; +} + +static void tcf_action_dtor(void *object) { + struct tc_action *p =3D object; + + free_percpu(p->cpu_bstats); + free_percpu(p->cpu_bstats_hw); + free_percpu(p->cpu_qstats); +} + static int __init tc_action_init(void) { + struct kmem_cache_args kmem_args =3D { + .ctor =3D tcf_action_ctor, + .dtor =3D tcf_action_dtor, + }; + + tc_action_cache =3D kmem_cache_create("tc_action", + sizeof(struct tc_action), + &kmem_args, SLAB_PANIC); rtnl_register_many(tc_action_rtnl_msg_handlers); return 0; } --=20 2.43.0 From nobody Sun Feb 8 12:14:42 2026 Received: from mx0b-00069f02.pphosted.com (mx0b-00069f02.pphosted.com [205.220.177.32]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C3D4D1F6667; Thu, 24 Apr 2025 08:08:57 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=205.220.177.32 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1745482139; cv=fail; b=ANYbwFIs8THiV/LsSHT2Si/noqlaJZYm7gza0YzgMOhJPxNv0C7RfdJG8swkbFhcCHsnRYbVm7adjtnvVBObC2GizxHmJuEF0DXC3HsSM3sZk35kh5NyCbkkBNkbZicYEY1Tfiv06Wra4nA8LJE7/hjiZjKGcOnaHxP3VocCE9A= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1745482139; c=relaxed/simple; bh=lRGgiGyJ4vam8UCkGj+SfF6y2mEQVN0+yPhRUHgIFkA=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: Content-Type:MIME-Version; b=MioVw5SB4oTVB7WxA7ufzVzqfuL1U0QWAxtdNvQMuewdIqUp9Q0sfA7qMSymShG4Pu83j3HS7rZFcHq3JKJQgJhkpmalAtJ/vS626kVZSAZtSHygg7ExbpzttAVPgnmmAYjqFACdTj4Z7ee0FqlG5nqtTVaHPiSqALaUMRsm2jI= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=oracle.com; spf=pass smtp.mailfrom=oracle.com; dkim=pass (2048-bit key) header.d=oracle.com header.i=@oracle.com header.b=I+QKQGUw; dkim=pass (1024-bit key) header.d=oracle.onmicrosoft.com header.i=@oracle.onmicrosoft.com header.b=Z2sHzrU8; arc=fail smtp.client-ip=205.220.177.32 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=oracle.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=oracle.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=oracle.com header.i=@oracle.com header.b="I+QKQGUw"; dkim=pass (1024-bit key) header.d=oracle.onmicrosoft.com header.i=@oracle.onmicrosoft.com header.b="Z2sHzrU8" Received: from pps.filterd (m0246631.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.18.1.2/8.18.1.2) with ESMTP id 53O84swB003302; Thu, 24 Apr 2025 08:08:40 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=cc :content-transfer-encoding:content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to; s= corp-2023-11-20; bh=YUgNxv4cL/9Dxkiza7/asrzAre485Wsa4U55tpifUuU=; b= I+QKQGUwfbRL8szae7SkcSmHOGB6hPumojq4QA9OhQ7jEbOlIoF2V2mUvrrz5qTf uIFPk5x/dB8AWvhQsMpDiRsQOSn2LLeJKkoyBrgF+YjfyB4tVinoPs+/evMUAJ4t GyjaPon6MHzQ42FIGC0UU6D9dUpnF7U0vkoovXOfxud8YHyajSm5O0cfUNStLyUD 50jrYp22DT1YcNw4HoQf3KkMvZquzlYrM5mEbZbongbcfan7f2p6s7XVKEhTW73N vHK3bKmamYoQNBhwlYnKpjuSwG09DtfF4ePIlhwbfQeRfXOETnkZjTSIioASHSmM CYJ9lVpE7SRCWWJQ7dYjkA== Received: from iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta02.appoci.oracle.com [147.154.18.20]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 467hfm006v-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 24 Apr 2025 08:08:40 +0000 (GMT) Received: from pps.filterd (iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com (8.18.1.2/8.18.1.2) with ESMTP id 53O6ufhP031694; Thu, 24 Apr 2025 08:08:39 GMT Received: from dm5pr21cu001.outbound.protection.outlook.com (mail-centralusazlp17011028.outbound.protection.outlook.com [40.93.13.28]) by iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTPS id 467gfr25em-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 24 Apr 2025 08:08:39 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=HB/tyrT5/XVjY9Wv9YIf+m64fVyK31KfoHHujGRd5gThLQ2KohFD39UdVDhYcKwJYzhKNMCR17lDgiWz2D6XYPZaXq6A1sK1ihFJr4dmboH8+0NPAB8HG8FNGwWn6SD7EL/fK3eRdavXlnE9B2EhbuHn+g+Fsu17nJBULnWuyDPUUChHQhONxTLZ0hOyOTM4ohjsBzBykZT/kcX5lBv+0BxTp5y4EG5arn9PgA+uwhSlnZIoIBbhJw/TaOu9Bv94whkQLI3iuylLE632maKuSnEvYJrlPpMfJzSY52mJsSGg3ObA9JjJ4anW26IIYV6Yhx5jZ+A336GAv34hzM2moQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=YUgNxv4cL/9Dxkiza7/asrzAre485Wsa4U55tpifUuU=; b=iP1NdM516KUei+0O91Fmr65Kb49P0tfkD9R56CijARBwgL6ZzeabBLlS/sLtojZ/w3c3ZXAhzifyQBBRP4JNsH2nUYpSVWit1k6oBLfg3/BXCoQNYL2ufoIiegfo7Gvmp69UpLw8PXEnBhTws8jecR+t40RFpVU419lGlPvMDJhxGRTItmtFBkXjDu10WTrHBioSYYo4UHMNZ7Hgy9Dkrqo/FFaivUOBnS9l2EVoG+JGPwBiOs5ULDMN6XGeQjwDUoGT7FMgC/Q9xyjYCAk5bo5VBSrL4YUz+GkJGIvmC8T/aZlNPLUghZX6K7A1AUx1eVfBdlgdP7lpYhpZUtAVuQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=YUgNxv4cL/9Dxkiza7/asrzAre485Wsa4U55tpifUuU=; b=Z2sHzrU8GUATNFIzPjbCjjSVRKOJcFyHgaWzB87VG6xWdWP7QI7eygn97Qse2nr/qDmfR6y25PsmCQOKfm3gpE9IJpJB+yNZYRGo3A30wRxIF8Uy6mJ6O4c9xXVWemX/iRxsCy3OFXQ7VnzIJR7Q+USefQ8MnA30z5VOJhw4U4s= Received: from CH3PR10MB7329.namprd10.prod.outlook.com (2603:10b6:610:12c::16) by LV3PR10MB8156.namprd10.prod.outlook.com (2603:10b6:408:285::9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8678.23; Thu, 24 Apr 2025 08:08:33 +0000 Received: from CH3PR10MB7329.namprd10.prod.outlook.com ([fe80::f238:6143:104c:da23]) by CH3PR10MB7329.namprd10.prod.outlook.com ([fe80::f238:6143:104c:da23%7]) with mapi id 15.20.8678.021; Thu, 24 Apr 2025 08:08:33 +0000 From: Harry Yoo To: Vlastimil Babka , Christoph Lameter , David Rientjes , Andrew Morton , Dennis Zhou , Tejun Heo , Mateusz Guzik Cc: Jamal Hadi Salim , Cong Wang , Jiri Pirko , Vlad Buslov , Yevgeny Kliteynik , Jan Kara , Byungchul Park , linux-mm@kvack.org, netdev@vger.kernel.org, linux-kernel@vger.kernel.org, Harry Yoo Subject: [RFC PATCH 5/7] mm/percpu: allow (un)charging objects without alloc/free Date: Thu, 24 Apr 2025 17:07:53 +0900 Message-ID: <20250424080755.272925-6-harry.yoo@oracle.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20250424080755.272925-1-harry.yoo@oracle.com> References: <20250424080755.272925-1-harry.yoo@oracle.com> Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: SE2P216CA0005.KORP216.PROD.OUTLOOK.COM (2603:1096:101:117::16) To CH3PR10MB7329.namprd10.prod.outlook.com (2603:10b6:610:12c::16) Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CH3PR10MB7329:EE_|LV3PR10MB8156:EE_ X-MS-Office365-Filtering-Correlation-Id: 5ee4fbdb-c64d-4d9b-0b05-08dd830734a6 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|376014|7416014|1800799024|366016; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?OwNs6JJ06qIWEjRvl7QUsjGHln/I+xO8RZ5KPzSO3Mrx1XQUTfkH0eFZcpb0?= =?us-ascii?Q?fj7SKe1t2x2uGHzgCX/fvs91MjEhowHD/5BVJcwWQqroOXwjX7O59I3zwllO?= =?us-ascii?Q?J0dYgHKhnsGFNs7DLpDZyL4CsGsfkWYVDX1iZXZ3srozLTypdff1K3gbLB8j?= =?us-ascii?Q?TIJueRfEC9dY7tVoJevybykhLQDo1HGih9RMyNeQ7Aj86IILF5vyAyqmIRWK?= =?us-ascii?Q?tx3Ou/mISMrrOSvi09yfAXZCP5nH3I28UxrL4KJrWl71cka+veMg3zwgb7GX?= =?us-ascii?Q?+U3wVAM3lcqExjS5XIxWjg7blIcT6ilEdUasNdEwSC4mL8C/zAuvmhrmxscX?= =?us-ascii?Q?3uWN75Et+yAQCVUr1sruj6kIntE9WpzHyN8v8uP3/SNIGZlpqotQRKkIA65A?= =?us-ascii?Q?h5rBdcREY+Y2gcFvfTByCTxJbpujKdAPgGHK+KxkTGANXF2HIasPZSO8Ptis?= =?us-ascii?Q?OoQw6rErWSo+JFANB6Y6tZRXTZJ3AX+m0b+lMrMZsGpEzS3zvN+3Rb+0OHAj?= =?us-ascii?Q?eUEZVvbqOhyqV9apYeFhgAvVDA+Y/sS4pSi0SqML+NIqRYzNYKQqSec0tA8S?= =?us-ascii?Q?ewZIG6Jamuq+PBTajB9edLjsaXvFQ6BDdRdVqSO/qlq5UBXN5zib6rJRmknA?= =?us-ascii?Q?24klFJImbYdHLB8iFMYWWnerdvCOffoGknm9+by/0fKzEuyfeII2qOCko1hN?= =?us-ascii?Q?ji+Ql6W/IIkV1znBitCgQPeYsi+KbN1a+m3clDkVZkhgCsWMglOjxCs5/YHo?= =?us-ascii?Q?L8egIMp0oEZYwAvRVZz/K3hi+cVMtMDelQAVdhFq3dj04sVbG8XmCaaRVhZU?= =?us-ascii?Q?UPxyiYs94Vo+NS+wwxltzT9ndC4qbZYrNjr3cqtxqeMvMQdYjUTq71gIrhrS?= =?us-ascii?Q?e5SKQB6pVZxcqYNcAFDxCR08k0BfrBxfs1TrbwFryBI1lf+CEghVUrSs3ker?= =?us-ascii?Q?fXBZsI2JM0dQdXIyFg/Tg4JJXzMFuz5Vxr5BVGeMO4nCY8REEk2UGxp0+tUJ?= =?us-ascii?Q?4h+2do4InwC6jYczWPIcuZRiS2iua6JheK5cF4X0hf9Q/Eb4jPKTB0x+19kl?= =?us-ascii?Q?itMcemEW1/eSz3thj47N7vGRRXvlhBQB5hLLx8TrM9ypJdXEPukBMaBO00Aa?= =?us-ascii?Q?PiClkMcDHmbbmaAB/R5SldLNhoFESRwIEcOJwt79T51t4Dw3kMaD2K/c3O8e?= =?us-ascii?Q?bbJf4MvsI97HBsnmuUPNgU5Bpw18pyky49gjjNozL+kmZTHgwr3WJ4oq7CSN?= =?us-ascii?Q?oRWGwWu31v9UOXSzXVZ+s60y6nvl7JZhGUkkWFvC2Zplj015yIHZVLLG5hwt?= =?us-ascii?Q?FPVYcFecyrczUFxy4mDzBh8I9ru/3ZIgDxgL9+fhUvyhPuqsdHGsIyfco1hv?= =?us-ascii?Q?bQvQwvFVGyTSW+3XiWGAf5DKEeonnqfB/1TH8UxL0G2Ymi+KJA=3D=3D?= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:CH3PR10MB7329.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(376014)(7416014)(1800799024)(366016);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?tULQbJoQsiJmAYealKPif2pd+Of6+BEFYN/6pdWPd50e94012LRLEBO20kqi?= =?us-ascii?Q?EC70dwR3ClK8SpgFJg88aU3zvKuX+aqvId9XhSoRb5bwF/b6W+z4CUacpodM?= =?us-ascii?Q?2faIIJrcjf0JsX9GHUI/dXyHvuBvMQbjxdR2JIU0gb5WrbZyx0mHgVoG7OFF?= =?us-ascii?Q?wNt2mwgsvs9+UwZ0Wm0QnSmjzjbpahkvpr9xvSt2QISXXiAVJfMxstXFTt3s?= =?us-ascii?Q?4yKTmO8frdNXJB3qtNfrgTz3jG+eqMEiVMKJUCZ+xwvEwwg6ZIPnZ4Kmd3jd?= =?us-ascii?Q?aN0yFEd3kWwj0GtXRZ2JZz4YBdXq6EnFTdBvdjzbheB1qLLlLyNVzgc/hszH?= =?us-ascii?Q?RlEmssizmv5BwreZLwmYdU98yuM11BuoYNZc73i8+D+yWyPN8VE2R/KnJsvU?= =?us-ascii?Q?vhr2uH6TohKz5g1Eim9od57kdXLNEAx52LkhDB+u9DHABbrX613l9H9SOJmZ?= =?us-ascii?Q?vikzkBRI57QegoF1d2IpD/gVShBNsgE4S0bjAj2Hpk6+2m3aGO8bt3KfKcx8?= =?us-ascii?Q?BPGlX+l4Y4DTzxpM25J9C8v2xJuNlyGEvlMTCiDY5aHMviUBpjjc1dF65Qev?= =?us-ascii?Q?uMG++cdkx4xluypkLt/UmMq0f5S1j94i2kba3YY4CbgPH1A4P3Q9mNxkMtXL?= =?us-ascii?Q?V42fM3ckvbbQqWVioE8OtuPBVt3PwW3QuRxvytFACcVg09/xQFTTa4/F0NHY?= =?us-ascii?Q?C3FO4aywwG/T3CFG8atvgmuYG2UGnuGcXEZYfNRbkjzAw23S6GyQqHlVTQ57?= =?us-ascii?Q?9avgbllhcH/t/XQcBt97fp8fNXMwIHrYk3UrBxXcV2JR9qMrS8VT65+44dyp?= =?us-ascii?Q?C+1vOMgSFOUVWvXhR5mPdEisVNkuQYTK6T5M5WD7a/PtDnYYPt8Xm3Pr3DF2?= =?us-ascii?Q?9hgbhmrzm0qambHhtm4A5GEzAg9j7QWqLW74qZLCf6OO0SEpkTMxTNH6GIgC?= =?us-ascii?Q?2RbeC+DhQPRBB+E9eTMcVT1rqadydqZn+yMnNBoYltfsOYOca3C/zEbRtidX?= =?us-ascii?Q?GKrKTODxMIXr0as6OHg5+Ju0zptTYI+I1x4VvzL/hVQohAqmXmB5Didtnwe8?= =?us-ascii?Q?fudlgjDLzTC4ogBcFCLkfssCkUAmT95EUR8E1HpygZJA1JR8zysnFLSwuHt/?= =?us-ascii?Q?UIYinwS2J7LlsnvxBhjArAlwIoWYBhpiDxQncN9eDe13O4l4CEeS51KyIbnQ?= =?us-ascii?Q?sPaDapbYoyXB5urisbdD5ZvpP8p8UrHJhC2eyIpEwkqjhGeZzfaryOsIsVkO?= =?us-ascii?Q?vPE4rNRoT7mYPqM7HDmyKxg/mDi/+IvqQi83XlnEISQn5RlnPENjuNT9d1R+?= =?us-ascii?Q?7+epk+4jcpPTtKzW4G0EOkN4WBYZ0Xl/eK8uGLpfxeo0ufI000lT3+k2XjOU?= =?us-ascii?Q?tE/iTlQapoH3nd0N9Oc48h5JnoDYLLyghjRydjkhRz7VYEokUFWY9tRvOn9J?= =?us-ascii?Q?OOul4WHPEROf2mBifonajShcmEnpCZLAdRnotyYug7uX1W42Vef/9mc6TdMX?= =?us-ascii?Q?9l3G/WgYu8Y11mEDq0U6I9qwRihWvFJJP76GYKhBAow5O+tGRpewkpyuJ7xU?= =?us-ascii?Q?pX7sZELD/Mr7/n6oBdZMP+R2TvT2epq/js171CQj?= X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: HwvwQixPXvMjrB6whv0vqHiMgBOIXOXMiVPuBjwabYkPY7BxY+1jZo4HGJl/Qj+1kF4u5E34aXI4YCIEoQ1AfakoE2ZlfAcli652vyBugziaOaqIeygd03XJYUgc/Pw2tt2xGwjj/b8QgCwPFyHAtbEcZXUihqxQFyqKuaVa/MDxAmX4Wf1vTr8yjldd1GKkRhxUr3S7on0wxRSNAvpvz84yzaRFnsQQA2kylnAQmayWmStxObCf/9jVy+kUWA6SL6pomFx/il7c7Mc+ladSRrwAtc77PyqroWjlGb8hd9rLjtX77dEkFYTwAtwx/E2QHo0xIIS+fxsghag1+GmkyEnRVZVVGJJ7wmaEXJga0vGXqV20hvNwnbN73lKzxXWuTpmz7f8wfSKz03v66M199nX9PmcdW4JgTfV+7GSFEvF890agN4yxwSZvko2QCwSwxghunU7mefhkc5izw4BOe+L452m7eS7PxBZIRQC0MGvijy2ORM7L/91Sri90V+cFF5lq8O6izmWKz/KXHYykPny9vwfRV+SFRToNMC8skFYQKs8HBjmvXafoztRQlUC3xJYtFbKCGoy2EvwKA1GFxykiBhgaDRtqjFjQhiRAxVo= X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 5ee4fbdb-c64d-4d9b-0b05-08dd830734a6 X-MS-Exchange-CrossTenant-AuthSource: CH3PR10MB7329.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 24 Apr 2025 08:08:33.0827 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: +hCO/fq8qB/MuUTQzXPRUYK4YyuWpDDak6EDidM6Z2cOjVy6rSRCgjCGyxgvgt0q/mGAkO3G8M8d/filZSsbYQ== X-MS-Exchange-Transport-CrossTenantHeadersStamped: LV3PR10MB8156 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1099,Hydra:6.0.680,FMLib:17.12.80.40 definitions=2025-04-24_04,2025-04-22_01,2025-02-21_01 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 adultscore=0 phishscore=0 malwarescore=0 spamscore=0 mlxlogscore=999 mlxscore=0 bulkscore=0 suspectscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2504070000 definitions=main-2504240053 X-Proofpoint-GUID: --QRqZqRwSQrzYl9AdvUh0J3AExGv7Nd X-Proofpoint-Spam-Details-Enc: AW1haW4tMjUwNDI0MDA1MyBTYWx0ZWRfX3xFEBXD+2Z2T nlzsdMt3XI1fbNzIIEaykfy0RnG4wauJi7T1BXn7fNLq1/H/cs/Pq/8Nth5l8Wua7AiyvayIuYW q/Ce8tvaAZaDFlD2ptzL5AgpUkuKXwsAEXLrc5DQw2xoS0cAN2GJ0Y609mpcBaW5SkS/2Niped3 zoIsz96UtLgDJnStHeIuANWOqHq6XfliiR8mUAMpEmiR6bQKwEB5ftKRIJsqI2YX1qxPeIHrY4R Z+CWTm7eW17l952CDWlZTJsb9MxhITl/e7PThc5KiR1XGgiE5FYy5hB8iiQXgzpjG45tB9FZFmG 9cqtQ4tbTZ8sWsGI72+MsMc2VbQG+N0qbbV5aANcU8atFVgWW9msjnQ4PfLDivf01gTBoof5xCl Zc2dXHpE X-Proofpoint-ORIG-GUID: --QRqZqRwSQrzYl9AdvUh0J3AExGv7Nd Content-Type: text/plain; charset="utf-8" With a slab ctor/dtor pair, slab objects can retain a pointer to percpu memory that remains allocated until the slab destructor frees it. In such cases, the charging and uncharging of percpu memory should be invoked when slab objects are allocated and freed. Allow explicit (un)charging of percpu memory to ensure accurate memory accounting for the slab destructor users. Note that these APIs only (un)charge memory only for memory cgroups. They do not affect memory allocation profiling. Memory allocation profiling records percpu memory only when it is actually allocated or freed. Signed-off-by: Harry Yoo --- include/linux/percpu.h | 10 ++++++ mm/percpu.c | 79 +++++++++++++++++++++++++++++------------- 2 files changed, 64 insertions(+), 25 deletions(-) diff --git a/include/linux/percpu.h b/include/linux/percpu.h index 52b5ea663b9f..2d13ef0885d6 100644 --- a/include/linux/percpu.h +++ b/include/linux/percpu.h @@ -140,6 +140,16 @@ extern void __init setup_per_cpu_areas(void); extern void __percpu *pcpu_alloc_noprof(size_t size, size_t align, bool re= served, gfp_t gfp) __alloc_size(1); =20 +#ifdef CONFIG_MEMCG +extern bool pcpu_charge(void __percpu *__pdata, size_t size, gfp_t gfp); +extern void pcpu_uncharge(void __percpu *__pdata, size_t size); +#else +static inline bool pcpu_charge(void __percpu *__pdata, size_t size, gfp_t = gfp) +{ + return true; +} +static inline void pcpu_uncharge(void __percpu *__pdata, size_t size) { } +#endif #define __alloc_percpu_gfp(_size, _align, _gfp) \ alloc_hooks(pcpu_alloc_noprof(_size, _align, false, _gfp)) #define __alloc_percpu(_size, _align) \ diff --git a/mm/percpu.c b/mm/percpu.c index b35494c8ede2..069d8e593164 100644 --- a/mm/percpu.c +++ b/mm/percpu.c @@ -1606,6 +1606,32 @@ static struct pcpu_chunk *pcpu_chunk_addr_search(voi= d *addr) return pcpu_get_page_chunk(pcpu_addr_to_page(addr)); } =20 +#ifdef CONFIG_MEM_ALLOC_PROFILING +static void pcpu_alloc_tag_alloc_hook(struct pcpu_chunk *chunk, int off, + size_t size) +{ + if (mem_alloc_profiling_enabled() && likely(chunk->obj_exts)) { + alloc_tag_add(&chunk->obj_exts[off >> PCPU_MIN_ALLOC_SHIFT].tag, + current->alloc_tag, size); + } +} + +static void pcpu_alloc_tag_free_hook(struct pcpu_chunk *chunk, int off, si= ze_t size) +{ + if (mem_alloc_profiling_enabled() && likely(chunk->obj_exts)) + alloc_tag_sub(&chunk->obj_exts[off >> PCPU_MIN_ALLOC_SHIFT].tag, size); +} +#else +static void pcpu_alloc_tag_alloc_hook(struct pcpu_chunk *chunk, int off, + size_t size) +{ +} + +static void pcpu_alloc_tag_free_hook(struct pcpu_chunk *chunk, int off, si= ze_t size) +{ +} +#endif + #ifdef CONFIG_MEMCG static bool pcpu_memcg_pre_alloc_hook(size_t size, gfp_t gfp, struct obj_cgroup **objcgp) @@ -1667,7 +1693,35 @@ static void pcpu_memcg_free_hook(struct pcpu_chunk *= chunk, int off, size_t size) =20 obj_cgroup_put(objcg); } +bool pcpu_charge(void *ptr, size_t size, gfp_t gfp) +{ + struct obj_cgroup *objcg =3D NULL; + void *addr; + struct pcpu_chunk *chunk; + int off; + + addr =3D __pcpu_ptr_to_addr(ptr); + chunk =3D pcpu_chunk_addr_search(addr); + off =3D addr - chunk->base_addr; + + if (!pcpu_memcg_pre_alloc_hook(size, gfp, &objcg)) + return false; + pcpu_memcg_post_alloc_hook(objcg, chunk, off, size); + return true; +} + +void pcpu_uncharge(void *ptr, size_t size) +{ + void *addr; + struct pcpu_chunk *chunk; + int off; + + addr =3D __pcpu_ptr_to_addr(ptr); + chunk =3D pcpu_chunk_addr_search(addr); + off =3D addr - chunk->base_addr; =20 + pcpu_memcg_free_hook(chunk, off, size); +} #else /* CONFIG_MEMCG */ static bool pcpu_memcg_pre_alloc_hook(size_t size, gfp_t gfp, struct obj_cgroup **objc= gp) @@ -1686,31 +1740,6 @@ static void pcpu_memcg_free_hook(struct pcpu_chunk *= chunk, int off, size_t size) } #endif /* CONFIG_MEMCG */ =20 -#ifdef CONFIG_MEM_ALLOC_PROFILING -static void pcpu_alloc_tag_alloc_hook(struct pcpu_chunk *chunk, int off, - size_t size) -{ - if (mem_alloc_profiling_enabled() && likely(chunk->obj_exts)) { - alloc_tag_add(&chunk->obj_exts[off >> PCPU_MIN_ALLOC_SHIFT].tag, - current->alloc_tag, size); - } -} - -static void pcpu_alloc_tag_free_hook(struct pcpu_chunk *chunk, int off, si= ze_t size) -{ - if (mem_alloc_profiling_enabled() && likely(chunk->obj_exts)) - alloc_tag_sub(&chunk->obj_exts[off >> PCPU_MIN_ALLOC_SHIFT].tag, size); -} -#else -static void pcpu_alloc_tag_alloc_hook(struct pcpu_chunk *chunk, int off, - size_t size) -{ -} - -static void pcpu_alloc_tag_free_hook(struct pcpu_chunk *chunk, int off, si= ze_t size) -{ -} -#endif =20 /** * pcpu_alloc - the percpu allocator --=20 2.43.0 From nobody Sun Feb 8 12:14:42 2026 Received: from mx0b-00069f02.pphosted.com (mx0b-00069f02.pphosted.com [205.220.177.32]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9491920298A; Thu, 24 Apr 2025 08:09:01 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=205.220.177.32 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1745482144; cv=fail; b=k+ueojyNVNyhU6A5bkoGxirsqOp8ZfN3akx6VLc+kAqvugzATxrGqbKO16/edwqeXWxd8r+G0B0xbiytmHwPNvJuoFdYko4QVLN5zRuBjaewqE1j3nYA38+jcaZ+GbjDkF8f8IPfdI9HFP34GXFMhpNsfI8XwoWyCPsNj3bbI58= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1745482144; c=relaxed/simple; bh=Ax9QYXaawaCc99DWWOkCL+XMmVOMz0GJArci0LK4vag=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: Content-Type:MIME-Version; b=rP/roo09C996c025RjrmmbJ7tGzSUcwrm4nOWGbwp20FdDs3c7FN3gsNHEOTd60sGm5SVVo1c3Tch3o1Wek7KPo55x8r5F/nFq0ZEnmw50POxH5k4NsTUUvHX0jHhdAXXyH6ab3x4UZanOsqr/0HKPkGUUokG+RHIG14lrDD+nI= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=oracle.com; spf=pass smtp.mailfrom=oracle.com; dkim=pass (2048-bit key) header.d=oracle.com header.i=@oracle.com header.b=j9XI366b; dkim=pass (1024-bit key) header.d=oracle.onmicrosoft.com header.i=@oracle.onmicrosoft.com header.b=OzhEfmrq; arc=fail smtp.client-ip=205.220.177.32 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=oracle.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=oracle.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=oracle.com header.i=@oracle.com header.b="j9XI366b"; dkim=pass (1024-bit key) header.d=oracle.onmicrosoft.com header.i=@oracle.onmicrosoft.com header.b="OzhEfmrq" Received: from pps.filterd (m0246630.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.18.1.2/8.18.1.2) with ESMTP id 53O6tvo5013966; Thu, 24 Apr 2025 08:08:43 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=cc :content-transfer-encoding:content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to; s= corp-2023-11-20; bh=1dMUuYCwRnXUShlrqrBY53jr8XsAGAoUAqEqwagrNSk=; b= j9XI366b3UoFf3+PwVmQRzE4l4pquixAffPgzoWyLdhdqsKpcZruraXCDyh95dwy Ymzm5qHUlO6WtU/FH0e9neFKp+XslaJBWRKL9M+jLAg58a49XUtkuXSTquF8oMYL 4lk4r55dRM6P/RLBYhG+Gw4kum5TNPM870wwMS7M6+AEPbsUl0OoEmkBbRLrKN6P lSO14yyV0IXYbStsHBN7UDkCH8NlF+02vMXIycIc9h/Nhjj5tXc1Lc+hb8VOTB83 tT08e9CP2WsM5U2cWcEV9Hmn/IiFWdQJ70JjR33X+LN/QnA26eVINYWx6RCwjJZD VCJqtlQZ9jiY2Fu6Dqc9pg== Received: from iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta01.appoci.oracle.com [130.35.100.223]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 467eghr8tx-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 24 Apr 2025 08:08:43 +0000 (GMT) Received: from pps.filterd (iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (8.18.1.2/8.18.1.2) with ESMTP id 53O7806D025207; Thu, 24 Apr 2025 08:08:42 GMT Received: from sn4pr2101cu001.outbound.protection.outlook.com (mail-southcentralusazlp17012014.outbound.protection.outlook.com [40.93.14.14]) by iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTPS id 466jbrqxv5-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 24 Apr 2025 08:08:42 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=R+ZCO/SHmGhHWZjfV0SlB4U1okpr3zVQXrF5/un6ayqR4NegHcHSmZ1IruPDpzPLj1DSjcPP+SrlLWIzgdiDvnGML3fGo7t2HQT6qf5Xv3Ozq/KNqnBOEyEP25H7BfQO3loC2R2pS2KG9rxLZ/8WIV/ln5ExWRvpIEoN+91WsoIvJruDY/qeDCwmivj+gRi+0dLfuByLy7oekgcI+StH3ef8RGxmRqKhhlNmUznUBdINctC73ZJJy1fP0LFQUNsYCK+jLtyCeu6AqEDLNduypvlZUvz24WSLoiEGuVQn0bSl6gM3tGrwOYKHVlqHlgvKMf6Zd7Ql/0cYdmty+wXArw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=1dMUuYCwRnXUShlrqrBY53jr8XsAGAoUAqEqwagrNSk=; b=E0KUqfbRuiA66WLrqL25R0jSZgq8LUh8u/PGE2ANO4wHheutRXcmQ0z7RSRtlFfCiTJzv0umg+B2aHKVZmM3m5b9JUFyKm3XOHUJw3FjwBYS7U22//51/WWu5JEu6EBx7cEwsCVYvQFjTnvLkOsA4pEWa8PNKniF4viLB+YGrXkoUeedkkWDnc0qDGReB9B+NHmfW88hE1hgCdADi2nMrUFFPMnJ2LFwXlp6ERAT2J71er1+qR59hEFu0uq4JezZc8KUje3LwboVGSsP9MrcicFUozMCUNqvM5Q5Kak78W7Cm2k64G6wh0DEo+qa1VHGoYkNSeyauvyiVv8k8zEnqw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=1dMUuYCwRnXUShlrqrBY53jr8XsAGAoUAqEqwagrNSk=; b=OzhEfmrqJ6FQ7e8uH5KWEZRtVG5lke4lK31dEing8w1Olr+bRcsIYQAtjndy4Cl+46QQgfKXIEORe9WIK54D5oh/YP4QF86kaWGKrrqJ4Io6Oq4N6dy7tCVd9cr6beNf27yRDuHngfFXvf5kU5nEta+a+/gkUVxPofi09e2QjOw= Received: from CH3PR10MB7329.namprd10.prod.outlook.com (2603:10b6:610:12c::16) by LV3PR10MB8156.namprd10.prod.outlook.com (2603:10b6:408:285::9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8678.23; Thu, 24 Apr 2025 08:08:40 +0000 Received: from CH3PR10MB7329.namprd10.prod.outlook.com ([fe80::f238:6143:104c:da23]) by CH3PR10MB7329.namprd10.prod.outlook.com ([fe80::f238:6143:104c:da23%7]) with mapi id 15.20.8678.021; Thu, 24 Apr 2025 08:08:40 +0000 From: Harry Yoo To: Vlastimil Babka , Christoph Lameter , David Rientjes , Andrew Morton , Dennis Zhou , Tejun Heo , Mateusz Guzik Cc: Jamal Hadi Salim , Cong Wang , Jiri Pirko , Vlad Buslov , Yevgeny Kliteynik , Jan Kara , Byungchul Park , linux-mm@kvack.org, netdev@vger.kernel.org, linux-kernel@vger.kernel.org, Harry Yoo Subject: [RFC PATCH 6/7] lib/percpu_counter: allow (un)charging percpu counters without alloc/free Date: Thu, 24 Apr 2025 17:07:54 +0900 Message-ID: <20250424080755.272925-7-harry.yoo@oracle.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20250424080755.272925-1-harry.yoo@oracle.com> References: <20250424080755.272925-1-harry.yoo@oracle.com> Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: SE2P216CA0126.KORP216.PROD.OUTLOOK.COM (2603:1096:101:2c7::10) To CH3PR10MB7329.namprd10.prod.outlook.com (2603:10b6:610:12c::16) Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CH3PR10MB7329:EE_|LV3PR10MB8156:EE_ X-MS-Office365-Filtering-Correlation-Id: 62e3d334-9c42-4eb7-d3b7-08dd83073915 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|376014|7416014|1800799024|366016; X-Microsoft-Antispam-Message-Info: =?utf-8?B?SzlydnpMeGpaRVloVU9BRC9VcmRuak8xSTQ3SzFNU0tGdjFPOGtEQy9pd0Fv?= =?utf-8?B?NVQwQU5HOEJqbk5Ha2daYmh2ZzU2bW04bzdtYm9lVlNGSUdoSWV1K0Z1Zm9s?= =?utf-8?B?U2g4Y1g2NERiY01mWVYwSXlHazhtZDc5MnA5QklDMXFqdENlV25JdEtEWTRY?= =?utf-8?B?NDZvaGdUd1pnK2NEQ3VBb1F1KzFXT1g5TmVSWnNDMktFRmdJcHBSTFZva1VL?= =?utf-8?B?bFU5aGNURDljRXZBTGNRd1dBS2xJSzFVck8xWFV6OE5kQ3BYcnoydU9DREYy?= =?utf-8?B?cHpvSTJvZkdmajZzUXJCUkNVT3ZrTkRIS1hSN2xHM1FUMWpZN2JRT2xEeWlY?= =?utf-8?B?VVIvZG52VHd0eE9NQ1ZyU0RXVTZHb1pRNGlwczhCSFdad3Q3MGR5WUNzUGJ6?= =?utf-8?B?M3Q5MW8zaitaYXV4VVcvek40clQ3QjFDMTdYTnVRV2kyY3M1QjdMNEphVHAx?= =?utf-8?B?RlBMTjlSZzZDbktQSDJ6VnJkeDdBTTkxTHNUQ1pRNEMwTUxiZEhoUUQ5UytK?= =?utf-8?B?dzdDT0VlZitrd2xWSDhXSHRFTHExaW5vcE9SSHdNb3BhQUVLM0ZYYkU4cWRH?= =?utf-8?B?T2pRNjVyV05zeURrSUF0ZE16NXFtb0ZhcUpZalllTDJlOUM4L0hTQkw1RURS?= =?utf-8?B?eWlPeXhxdVB1WExiWDlVdnJ1cGViU29nZUxWUGxSd08velVmWWFaWDlzR3FJ?= =?utf-8?B?WStLMFFLNU02V3pzSjd5VGZ3aWJRbXRFZmRYTHpMUkVCWmNkRUNYanhYbFRz?= =?utf-8?B?VFNPaEE3bXY0SXFKbVdGMVhQcXFOT3VKajRNa1lnSU9ZbkMwRW5YSXVLRHhK?= =?utf-8?B?eWJKNVZhenJUWDZTSEw0QjR6NGo0eGF1cDRSY2ovQlByWlM3N0l0aXIrRjRy?= =?utf-8?B?aTg5RFlNaFY1OEtFUm56MTZoaUVaZWNtV3BCeWZhSUVWQjUwSW9RS003OWps?= =?utf-8?B?Y3NLdXdKMUhoWjliOUlHa1EreXVDU3NuNUFhdThrU2NQcCsySVM3WENtZGJH?= =?utf-8?B?VjFyL0VrbnZhZFVUcnNwcHBZWXZwZGJBdXBNYS9xbThGTXpvWWxVQUJUN2ht?= =?utf-8?B?aG9BT2tjRGFLUU1MRjhxOUMwR0NyZEg3cFprSHNKMi9ET2wxOW80WklVV3Bp?= =?utf-8?B?T1Q0UCsrNlVtV0hYdG9vYzB0YTdLZzlBclhzSzdoUktkUGt2VDF3aWZNai9G?= =?utf-8?B?UWR3dGhqU2dGRG03WnlEcm8rOCtPZkhTWHprNkl5ZTJ1M29oWXFwM2VsMjRk?= =?utf-8?B?VmpDNFVOVGt0MmF3SWt3SHBYbHBrYlFtZS9ZbGtRbkFhQ3p1UmpFd3l0QXYr?= =?utf-8?B?UXZHZjJZLzc2RlE5WTFxVGNkRmg2Um1YU01PUU9Uc01kbHkyMWpwdHo0VHNI?= =?utf-8?B?QkpzQTl6YThJbnRld2NGTWRtWjc5aXhicXpjZW1mVHp3bmVZRTEvNllYTHdL?= =?utf-8?B?TXIxbjM5K3dKd21UOFFpR2NxME9nYTg4SkxncU5lUFYweGo2R3RSYTYzWDlK?= =?utf-8?B?bldiSlJLSmxiVERVZXNuKy9KWWpwc2VtUm13dmxidEFVRXBteHk4Mjl0NUVQ?= =?utf-8?B?OXRtbXUydmNycUNaRGQwRzduTGVOb2o3SXNjdHlHeFUrNE4ycnJGbi9mejlS?= =?utf-8?B?YmpjL3prcUtsLzg1d3UwVFFtNDFDM2NMblZjZnhKZkJrQXphS1VCajZtM2NU?= =?utf-8?B?eUdsMmpOSTV5UmlCQW1EKzVJYjFGOE9nQzZzaHgvd1UxckhkelBYSEdwbkM2?= =?utf-8?B?M09RT3h6eHJDRGpDamFuZGw4V3dqeExuU3loTVpZWDJTT3RCNE5McEVpajY4?= =?utf-8?B?SjZSU1VJMkZGMHhXWnRWZlFNdG9XVXpvQVBvUkJzM2FNRy9uZXZIWXExVDNx?= =?utf-8?B?eGpaOU5TVHg4R0REUFJ0M3BMdThVTTk0SEVVbUQ0SmVCeE5teXU1R0NEdmhk?= =?utf-8?Q?wEBpajyjw5U=3D?= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:CH3PR10MB7329.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(376014)(7416014)(1800799024)(366016);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?WmhVYUw1VUZKSTljWkVWY28zWUI0L3VPM0ZtaVpEc05VT25YOHZLdTYvTnVr?= =?utf-8?B?OVU1VGd5MTJUZVg1Wk5tWlAzVjhtT3lXeWtFeUNQRWQ3OU12RVpQTTJQWGJ5?= =?utf-8?B?NlNFRUdFZlhSL2p1cVVHUE1DUE1ORm1PZW9NSkE5TmNpbFcvQVdBY1l3YlRL?= =?utf-8?B?MjVrRGE1R0RyZHFqSjBzaE5HQ21hZ1BVZ1JwZytPSVhDczNlM21mSmtFcmRw?= =?utf-8?B?N2JUVjc2L09CR3RDUzFaU3dJZGFVK1pITXhoTXVEY3YzMXpTK3R0R3dIS3dM?= =?utf-8?B?ZWtoZzYrZVNidENBMHF3QVNjYUlwb2tzUDJUZzBWcTdnTTN0TU56eVNmTmdz?= =?utf-8?B?dm41V0V2RVFsMHRZN2pWUS9KWUhNQTRtY090UGNDUUluSmlMd0Fkemh1anpn?= =?utf-8?B?b2xINExLdmNWNGs0ZTdib1MxWEdodnBoOVE4cFIvdk4ydzhLY1pYV0g2QWJT?= =?utf-8?B?c0hWMGdzUzZUemhPZkpwL3QvbjNmR0hKYkxidnlhcnE0QkE5cm9QVm1ublR6?= =?utf-8?B?NjJZb2NtMWRVOHgySEhTbnZtTGQ4WHdmaXpNTGFGYjg0eU1oMlBKb2NOMWl1?= =?utf-8?B?VmQ1S0JDbXRFam1YTElDdnFrM09hTksxbDFMb1BXR09EM2ZyUjdGQ2x5MXIz?= =?utf-8?B?RmdOajYrdWRRbVc4Q2RJMHZ0QS84REc2NVJUZkpLTWpZZlBHb1k4dlhvY0lH?= =?utf-8?B?QkhkVVJkaHBjYnQ1dEd5QjRsWHZmdXhLZDA2SVRpUmFockFhQWtXSUc4SDRP?= =?utf-8?B?RWlkOGRTODNXcEtta0MyQUQzMWo2U09jdDVWOXlJTmJuVjdYbzBvWGtQN3lQ?= =?utf-8?B?dTNrYUhtVC9sM1FESVNzVlRDMElrVVNJa2F2N2ZJU2ZOdVduYTRiMGxSQUc4?= =?utf-8?B?NE1mSzNGMzJJL1ZUQmxoMDRLT294U3h1OENQZVQ2UUFtc0FwL1BxcVF5b3hT?= =?utf-8?B?RFJPWnNZa3dJZm9PUUlmYXYzWjVORTlpSGFpakZvanZ5R1FsbVliWnVraTc2?= =?utf-8?B?S3Q4MmFMN25zNXdBMSsxWlVEZTRQa1hJTWFQa0F4NTl1VkFrVlYrM0tKM0pt?= =?utf-8?B?U3RoS0JMNXhNNkMzZnFZR1FOWTdXY25EVlFmTTMyMFNud3RsY2FLMVZkb2hp?= =?utf-8?B?cXFLV1NERC9mai9zc3dxWXZQSnBRakVKNzhJbVljbHRpOHBPQnFXcEVOV2hI?= =?utf-8?B?QWVXMTVXU0hQMGhyeDNxa1o1aW9rQzZDR2NYWXBtV0RhM3JRYldTV01rQ2tO?= =?utf-8?B?Zld1cDJHZHAxSnZRL0M0OGVxZ2R4SzF6ZTdGbkM0QmNiWGdXd1JnOEJIenRn?= =?utf-8?B?MXpmUGtWcENaZXFpbEhJQjA4NG5pbXI4WW5SVE9BSW8zUWZ6K09WQlNLUWpo?= =?utf-8?B?UnRYQ2d2WVBZcWJPek0rQURIQlI1SHhJcm4zOFN1YzgzdEsydks2STBDVC9i?= =?utf-8?B?QUsrVFJBRlhJbXRDNVZhc2dqVEdqRjg3N2t2S3dpRDllRFd6S3d3OHdSeXBw?= =?utf-8?B?akljaWlNcDk0ZnZubTdKOWM3ajdQSDRDNzFXdTRKb3NueUU0bDFybnlIaHQy?= =?utf-8?B?NmxnVzM3aUNGciswUzlRZVcvbXlCTHVXZUZtblBoZWdNZmp6bVBSOUwxazZ1?= =?utf-8?B?ZW40RUZmKzcxamZPRFNaYjhBREY0U0U3TEhhSzY4NzJqQ2ZGbFJJcEVsY1Nr?= =?utf-8?B?RGo3dWl4OGFIcVprTExLcHlYSE4rZ05Uc0ZkQlFkaEVST2JISlVYY3lPaEdz?= =?utf-8?B?SXV1dm93M3J5VGU3ZjRHMTBucU9pWjhndzhGRld2alR2aVprb0lMVkZndTVK?= =?utf-8?B?TTRTVU1LM1lGOEgzSjVlV0xnblI3b2pDUkJXNW95UVVPU3RSeU1KR3B5NjI4?= =?utf-8?B?eWpCVHJmdG1SdHlKanM2ZDFsUU5wRGF1SE9Vd1hrVEtMd1hlbDY3QTY0UkJp?= =?utf-8?B?RWVZOXF4WjdlQ2hFSm11bzVaYUlRZFhFVDVtZzVXcktvK2NXcWJyaHA0YW8x?= =?utf-8?B?TDRwSXY2SXBvRGFhMS8wMmRxd2c5aHpnTkt6ZWU1aUZrMUIwakVTRklWeFdN?= =?utf-8?B?dTZhTU04OElOUnhYMUhUVVZ4ZFp2eTBZOWNST3JZNDFMaUpxN0E2bTRRYUo0?= =?utf-8?Q?mYcesARIGk0hUYEuVhZLGl27N?= X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: EFx/pPF2XEfYrL+uCrGPjTwlr7PAZzBf7Qb0xqhL8fEAMxSiusfNBFmOLScJrU9mow+DHBxg3voxpbO10AZeYLe4nlyL2O8eVqIMcREqWXzLqFkuIpQD/5MJQWoCfpGS61oXbwo8pMGbsKWXsvE1BrKk524VFV4S01tkVN/yPZpbjl5iR+DznYUcV1EZ7TSHORguKL0ONQr/iFVtoDyG++UY2fwhC7Xmri44zCHaYulF8TzUCIYOO9SbLHHD9FkrtV1b5CUQpxCYKPfZ3AwEHzZD5tvE77CkzgZorAo7nixlntSsDBYMozFPu7NvEuBWGbzRBCpuH5zqjoPZiQTOFdmNCadTqxpFGt1D8UZRIWXeMTpnb77CwbsazE4iKizw/ZhrITSJ43L/PxI2XQTNcbNwH3lIR2t7+dhpcN60XnGCEiXgvzi0MP3gqpKPGQodRTzk0qLPSD4fR/LRu9VKERARgPPJj7T7pWK0emm7T2iNI1St7WbyQcoO0CqIFKuZvTHlaGqoLp4KPiligGHPeLo882WCi22Lz8mEXW/8jm2XyRtfO2ntusE3ETtIXwoOiuiB2OUugw4yRa7eDfmns36e6j3rrA3a6Lpu9MDv7bo= X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 62e3d334-9c42-4eb7-d3b7-08dd83073915 X-MS-Exchange-CrossTenant-AuthSource: CH3PR10MB7329.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 24 Apr 2025 08:08:40.3431 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: ANTpOCDP0W95J5amKB9oh9gxolyd37BlTw4I4MQZCDM4wSCBrp0of5P2VrvwEsZvggfIWJ7UfxTToFpjsfTjSA== X-MS-Exchange-Transport-CrossTenantHeadersStamped: LV3PR10MB8156 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1099,Hydra:6.0.680,FMLib:17.12.80.40 definitions=2025-04-24_04,2025-04-22_01,2025-02-21_01 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 mlxscore=0 mlxlogscore=999 suspectscore=0 malwarescore=0 bulkscore=0 phishscore=0 spamscore=0 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2504070000 definitions=main-2504240053 X-Proofpoint-GUID: buUtswGaxlDXrVhHq-wViwpgRJTz4I63 X-Proofpoint-Spam-Details-Enc: AW1haW4tMjUwNDI0MDA1MyBTYWx0ZWRfXxUeO4H1y6Jou JPNhagubbFnTGxRqYb45GgnXeCK7Yj/iiqnbbFTkS364g0mrGsz0KE9wvj6sKOnA6o3INekDP4y alAoTcyjn1JRKg9JStn1k9TlhJGSLJnyJNIuDGZ+T43Ipt35fHQIAEchNi6EvNFSRxIAUO8ByZH K3BNBbW5qHxqZUJ+JYPBGu2nwAdfcXTiYt1OA2Hu7uk4IREGJHkpEbTRY+YbG9mVJD2B+q6axj/ 0xZ7m1x5gJOfGOWVeBKqnQc8X9K3DtVuwp2rDvrsVSayeGciRYEi8gRGdLyCfYP8aB9/oQjpXre OLAo2x3juyzntlzdM9f68Tbr3znKgHnXOgQEpILNVVvCS3BPxT/jvGqlqQh1podOWNQpX9yUteX fb9olRrk X-Proofpoint-ORIG-GUID: buUtswGaxlDXrVhHq-wViwpgRJTz4I63 Introduce percpu_counter_{charge,uncharge}_many() to allow explicit charging and uncharging of percpu memory used by percpu_counter, without requiring allocation and deallocation of the counters just to (un)charge. This is useful in cases where percpu counters preallocated and reused multiple times=E2=80=94for example, when a slab constructor allocates percpu counters that are (un)charged multiple times over their lifetime, until they are finally freed by the destructor. Signed-off-by: Harry Yoo --- include/linux/percpu_counter.h | 2 ++ lib/percpu_counter.c | 25 +++++++++++++++++++++++++ 2 files changed, 27 insertions(+) diff --git a/include/linux/percpu_counter.h b/include/linux/percpu_counter.h index 3a44dd1e33d2..6e6b0752b1e4 100644 --- a/include/linux/percpu_counter.h +++ b/include/linux/percpu_counter.h @@ -46,6 +46,8 @@ int __percpu_counter_init_many(struct percpu_counter *fbc= , s64 amount, #define percpu_counter_init(fbc, value, gfp) \ percpu_counter_init_many(fbc, value, gfp, 1) =20 +bool percpu_counter_charge_many(struct percpu_counter *fbc, gfp_t gfp, u32= nr_counters); +void percpu_counter_uncharge_many(struct percpu_counter *fbc, u32 nr_count= ers); void percpu_counter_destroy_many(struct percpu_counter *fbc, u32 nr_counte= rs); static inline void percpu_counter_destroy(struct percpu_counter *fbc) { diff --git a/lib/percpu_counter.c b/lib/percpu_counter.c index 2891f94a11c6..a9fe96787725 100644 --- a/lib/percpu_counter.c +++ b/lib/percpu_counter.c @@ -224,6 +224,31 @@ int __percpu_counter_init_many(struct percpu_counter *= fbc, s64 amount, } EXPORT_SYMBOL(__percpu_counter_init_many); =20 +bool percpu_counter_charge_many(struct percpu_counter *fbc, gfp_t gfp, u32= nr_counters) +{ + s32 __percpu *counters; + size_t counter_size; + size_t charge_size; + + counter_size =3D ALIGN(sizeof(*counters), __alignof__(*counters)); + counters =3D fbc->counters; + charge_size =3D nr_counters * counter_size; + + return pcpu_charge(counters, charge_size, gfp); +} + +void percpu_counter_uncharge_many(struct percpu_counter *fbc, u32 nr_count= ers) +{ + s32 __percpu *counters; + size_t counter_size; + size_t uncharge_size; + + counter_size =3D ALIGN(sizeof(*counters), __alignof__(*counters)); + counters =3D fbc->counters; + uncharge_size =3D nr_counters * counter_size; + pcpu_uncharge(counters, uncharge_size); +} + void percpu_counter_destroy_many(struct percpu_counter *fbc, u32 nr_counte= rs) { unsigned long flags __maybe_unused; --=20 2.43.0 From nobody Sun Feb 8 12:14:42 2026 Received: from mx0b-00069f02.pphosted.com (mx0b-00069f02.pphosted.com [205.220.177.32]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 20015E552; Thu, 24 Apr 2025 08:09:12 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=205.220.177.32 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1745482155; cv=fail; b=gOJKMfQaKw512C68qAp7vOe8RfzlbnQjsRPLnEn7OAZwYYaEFHH1RcSnWOuPHYou7OZfRxVZbWi4Pp6Xnf5tx900uIBDpZ1STcIO2w+kLHMp+xkeO8U5gSM0ARbVMh22KCJv3ZHmwTV6EgJqzk45uTFJsm6pV4TvuiwsdbGtgcI= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1745482155; c=relaxed/simple; bh=dBAwtnAdflK0IWe6lp8h0TS3oHrP+Mk9jpYwENp1RXU=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: Content-Type:MIME-Version; b=FRcm7Qhan6I8fMUTsQfjnzKdJtk+hSZOOjHtMlPRuKrinzVxPTfsZYFBw14xFH+R6bw/naQYi+BFP2JgeOmUooWinVVZUmrkLsMHtmzExXQTZkScOuQ8gfABtbPqr3+lEvXRzeLTnVzZi1q+PNygT9cdtJTEkH0fUBydeMzWLYA= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=oracle.com; spf=pass smtp.mailfrom=oracle.com; dkim=pass (2048-bit key) header.d=oracle.com header.i=@oracle.com header.b=G9arMVap; dkim=pass (1024-bit key) header.d=oracle.onmicrosoft.com header.i=@oracle.onmicrosoft.com header.b=zuHrtalb; arc=fail smtp.client-ip=205.220.177.32 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=oracle.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=oracle.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=oracle.com header.i=@oracle.com header.b="G9arMVap"; dkim=pass (1024-bit key) header.d=oracle.onmicrosoft.com header.i=@oracle.onmicrosoft.com header.b="zuHrtalb" Received: from pps.filterd (m0246630.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.18.1.2/8.18.1.2) with ESMTP id 53O6twkk013986; Thu, 24 Apr 2025 08:08:55 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=cc :content-transfer-encoding:content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to; s= corp-2023-11-20; bh=umMv2cMz5y5y2+1ty2nqRMvnWPK7cWwZf91YYkODtsM=; b= G9arMVapbpLnx+Jc37ccC6vxqheYmATg4xuPJmqelXU7EWnjEmeZE26PqfiXCjcJ Ghy2pUs+nM1OFdTJfwDNGA1A5tSZ7IuWJXAhoQhDFKVxpKHtsz1Vc30pz6mxavKP WR3HjeXQ9EWPcrOHHMFyWi1vRjPMgVJaeBxfiYo+258wRPXgSxlGJZfmpKSa0EkK 6OykL7xp4rLIrLLX2GXCI9YAC67oyBRZ/2bpgadSKatf6hQq749Ll8POFCUwmv4L nvB3bofwXOZ9K6VmjUShYIWu0zUscfuslp4vOs9DpIaxEG0crXGxSuUntJM+qSCN hd6LYWPtGRPuJYp4K9RCOA== Received: from phxpaimrmta03.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta03.appoci.oracle.com [138.1.37.129]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 467eghr8ud-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 24 Apr 2025 08:08:55 +0000 (GMT) Received: from pps.filterd (phxpaimrmta03.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by phxpaimrmta03.imrmtpd1.prodappphxaev1.oraclevcn.com (8.18.1.2/8.18.1.2) with ESMTP id 53O60Aiv028507; Thu, 24 Apr 2025 08:08:54 GMT Received: from ch4pr04cu002.outbound.protection.outlook.com (mail-northcentralusazlp17013058.outbound.protection.outlook.com [40.93.20.58]) by phxpaimrmta03.imrmtpd1.prodappphxaev1.oraclevcn.com (PPS) with ESMTPS id 466jx76xaf-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 24 Apr 2025 08:08:53 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=HhydKng0h8F80lQN0sXVShcJJZil3gTN25Tor4FGWkOaiQ2jywuo7c4EOg9SadCkxfVrxwY7svQSvHCuzMyD+2h/Fe8RjeYhM7fDBDvH6h3t++QOcJrhSztBGNWt7tfR3CSwzrm2g7Z41HH+N1hTDJ3EI2JBk/Lvia3haj2/WTuQO24nxciHF231fKBgFPK5OYmEjcPyXnSRcGGw47aK3SxEocUStudasg7DKs0rgs22Kqgv90UQnMwAzBYvx46VttdKFoUQCnGTblnlyZSFwSSfgKvM8WUilQxKKVN+wLVtmTeSDLjH6uhCSwuRcMdINvmBmfIXo+mxdrImITDetA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=umMv2cMz5y5y2+1ty2nqRMvnWPK7cWwZf91YYkODtsM=; b=YHDAbJDYgjiLraH23QpXRJ3jkCh1GezFLzZ6VO9NiwxDX/j/MP9lMiCdHAsMXGb52X5fFtkO0q4XPf380FayNcPXx7lvKz7gtI36nXBhyCjkvn6RVeC0o3z1jljXcWKO0YlH86RPPayqbMhfd56/j1QkFXKOdauT5yveHRKbpq33kJoF690DHD/aiQZSkk8D3aRHNWBojzFKum3nKpPGWS8U632u/eExNw6XulpGKPKgmQ5YdaV1SRyljFOuHV96LysdlWjxRvBEr9p2EJArq4fsN+rdPhNYnf30Zxq6vj7OSC8lsCux4EZVOwNDW/rbYJpjwX1Ps4edhZ+O6eOKpQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=umMv2cMz5y5y2+1ty2nqRMvnWPK7cWwZf91YYkODtsM=; b=zuHrtalbdwfPgwTgkSqSyIwJ9tjKtOq5MS7D4IV8yzdAveY81pZRBwAnfxQYXa5o6Nu/8yRoYwvli3g5QS9Zm5f7npn82SV4mJvpYSqbed4Ujbxgx6Xf07qZ4/niaAqNouEt8F1lfy2Ivn4YtudNvUB8BMwoKBdLBoe5y/L8fxg= Received: from CH3PR10MB7329.namprd10.prod.outlook.com (2603:10b6:610:12c::16) by CH0PR10MB4905.namprd10.prod.outlook.com (2603:10b6:610:ca::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8655.35; Thu, 24 Apr 2025 08:08:45 +0000 Received: from CH3PR10MB7329.namprd10.prod.outlook.com ([fe80::f238:6143:104c:da23]) by CH3PR10MB7329.namprd10.prod.outlook.com ([fe80::f238:6143:104c:da23%7]) with mapi id 15.20.8678.021; Thu, 24 Apr 2025 08:08:45 +0000 From: Harry Yoo To: Vlastimil Babka , Christoph Lameter , David Rientjes , Andrew Morton , Dennis Zhou , Tejun Heo , Mateusz Guzik Cc: Jamal Hadi Salim , Cong Wang , Jiri Pirko , Vlad Buslov , Yevgeny Kliteynik , Jan Kara , Byungchul Park , linux-mm@kvack.org, netdev@vger.kernel.org, linux-kernel@vger.kernel.org, Harry Yoo Subject: [RFC PATCH 7/7] kernel/fork: improve exec() throughput with slab ctor/dtor pair Date: Thu, 24 Apr 2025 17:07:55 +0900 Message-ID: <20250424080755.272925-8-harry.yoo@oracle.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20250424080755.272925-1-harry.yoo@oracle.com> References: <20250424080755.272925-1-harry.yoo@oracle.com> Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: SEWP216CA0005.KORP216.PROD.OUTLOOK.COM (2603:1096:101:2b4::11) To CH3PR10MB7329.namprd10.prod.outlook.com (2603:10b6:610:12c::16) Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CH3PR10MB7329:EE_|CH0PR10MB4905:EE_ X-MS-Office365-Filtering-Correlation-Id: 03bcedd3-ecc0-4395-576a-08dd83073c1a X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|366016|1800799024|376014|7416014; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?3G96c8pHpCTXVXXoJIklQxwiYJHQ5YGiSndK6AkFmSHgjgG2R38ObFp59RzX?= =?us-ascii?Q?uSkAUNxbvzKrFUZpzCNgshh8ddyKnx4+DwHSrEkl9JtHJ2qVHG5gSRSggrfK?= =?us-ascii?Q?Uvwi5oGvmKoYVznECLB4s+9+v/M0O5YNxSEgJRDcUe8m+ZGszBrStZwXeQ3x?= =?us-ascii?Q?lS8XqzDJJx9B07zHDLdsSw8+E/Yc8Et8S6ntg+9vm0fPkjb1PiF/JaJRJM0c?= =?us-ascii?Q?vUYfyFSKc3ZuQcHkzRBFqOFyIovXjeQDMAh7Q/tS1et8U324NsKzru3Pe865?= =?us-ascii?Q?H1qikVezk+YUh1FYFp3UjYHWumvJZlShChPfH57NLlZ9e97BQ4Ogh/eABTwf?= =?us-ascii?Q?r0nbgo1jzJflY60VnUUsztqdc+leY6vHwXDfnPgDTSDPdG4sBHkpEjOslU5C?= =?us-ascii?Q?ojJicQQDQSTmne+HMFveJi7MkbGJanJueo37MKl8nqwfHBmFBE2T8HD/YcCm?= =?us-ascii?Q?WigB5ebCNdnZapWCrINBd+svECszBEZx28BExpyO5EWmG7LceMHKGDF/Yrzd?= =?us-ascii?Q?1ltUYcgnrJ3A2plGPcbnn4Pbw+u/w37l0an4pxxmrAswDTlsX8XNoUCFqi+8?= =?us-ascii?Q?Hf+Dgy2qSJF09Y15bHBD7K46GF8osx2RnvZNe67YecLYAH+HoZ30IuRO95aO?= =?us-ascii?Q?oICxDkbn7cmHrhZdv9Dok2Z5TvTj6m3e7n0vo3OKNFhHl8zQOnv5BFx8iEV2?= =?us-ascii?Q?Jruv3z5LXfh+hV/lWGjJLUtZgGKsLetSQC3uirCvCGo3LC1Bi2vDhqjgar7f?= =?us-ascii?Q?XZjO+UmUYYSk1CKl0HZLGfuNzyU9+t8iuwJIYaJAj847xLTO6a5qbqNQ8KFB?= =?us-ascii?Q?dN8Y3ClpQ7+jThk38utEiJlVzXMEgHRnzzOiS/PYOlm/21CCYwNThQxk1QjO?= =?us-ascii?Q?RjKixH8vPmujgcyKZOlxBv9q+ek7HM1/qBrwegD23MFVTUuhpk70wVFg26iB?= =?us-ascii?Q?+vd6sNxujwftHfDfz26INupPftZ6m+ziIj/vz7W0iUo3ggadSQj2imxODe+L?= =?us-ascii?Q?naVGzuB5rS/oSbnm1VaaznmGHcQnbIY4melKxBwcinHb7DyPTGRqWFqpxU/o?= =?us-ascii?Q?DeFnDM+FcgBtexu0j/9hw8ICsMwfCe34aENM6FSkLbR1poQ6kvoO2OKXcmCU?= =?us-ascii?Q?sjLwUIuKyPGe+IZt6OH3njTpxBENUW9eWhAiJOx5fJZDzNwFmTV8sozVPX5K?= =?us-ascii?Q?AsXTjD3kIV39FrvdeVkBIGvnc8/e+5TBrWRd/3JPTUz5kF+VeKqY6WFf80MK?= =?us-ascii?Q?NBDG2uIkaZURx7V8L2fkY5laGRxfNyN/9Z4PyWeEis24Ah6CnYPwAmHDYKzZ?= =?us-ascii?Q?nrR/1KktA58mwWCl50+1zEsGD0zCjl5Xk2Z3iPdm73k3zvZTf+QNDIBY36F/?= =?us-ascii?Q?aV3F61E=3D?= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:CH3PR10MB7329.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(366016)(1800799024)(376014)(7416014);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?43UBPa+lueESDSdVgECp11SH17Ci94RNEuAHDNJAW+H5KPd331+IHpYUWcoE?= =?us-ascii?Q?QEBaPsKNKXpsnS0qTJxSVOQV16tlwNO8Lu5pc3+fcygm5WaSILeo5mNwbisC?= =?us-ascii?Q?WW7JBjOvujj/F2TdPlsHe7a51BOma9zf4V6zNOuek8ZNdi6XL+jl7gveevG2?= =?us-ascii?Q?bp6lP3frEKIyQBDKjgO9NO+VMDZDHCO9n5LpKnI1bCkrKX38q/E4a0I5O9Fn?= =?us-ascii?Q?2Aqvy5GCPst365uIYWBBdzzVzpukOAnSLfvPdylaESIdiFqEfFiog6MyCfsL?= =?us-ascii?Q?bC3VRZiGZf3MUCoTY+R0Eavko/MxmAvSA7N/6ZQL3ubvjdloENxk5TJ/rbaN?= =?us-ascii?Q?STJ3uzbff4+5XlFMFq2tOfEMAukpWq8yyrpfq1GjMT64kgNWbG49CG/3Vm89?= =?us-ascii?Q?YTm8pbtEbxHTezGackJveTeIt0h7OmvMCbkHrjAVvfcVIKTbJw/PXCiyW4Xq?= =?us-ascii?Q?Tq9ozT9nmbdzfGpM83Mt2RO+tkZAH59fapeEvvr802kHdqDbT4Hc6N3lkUE8?= =?us-ascii?Q?X0z/xybj8imyeWJ2amKkE70lZCyTZ5sgYeYP+QZrYt1hN3PXTR/TshO6+Rep?= =?us-ascii?Q?jseM8aj0Z/LeUd2NVWg9x70dDQSt9wTxEjDNXZQmGhzSVTRkGUr5fdqvtfCi?= =?us-ascii?Q?DbVHxHVUecIBtSvImSa05EsuY7PxIBbZzCN9yZWjuF8C44gKeaGBagoabzEX?= =?us-ascii?Q?GoAmZSZJ2o/LPBAuhzQ2B3G34xZtx0Lf+NCDOVmjsvdkiG3fpVYBGANito/5?= =?us-ascii?Q?+aDJ/UV1Wi7c1NzR2aE/iZ/5bSLfuL5YQidCD7nOnOX22MAW1nF/Incnzbar?= =?us-ascii?Q?Z0Rr7YCNCBJRipRopMWhLH02mAoZmuSvJLsT+FflxPlCNtqiEX/hkUcxsHJU?= =?us-ascii?Q?4BjOEbCoQ0wDm9Pq9UTTBpLln8dZ+C7H0y7JMDsAibsOGxwKQ80pgG5DxSMU?= =?us-ascii?Q?55GawHdu7J+s9Mdakiv+UBcKMbWkkKm61j12NoK3a3wEb1CN8WHSpfKmyMyN?= =?us-ascii?Q?BKQHhissLqFH30VsLCdLEdL72EOzfkh1Ob6hbjIqr8f/4SeN/6l2vVCIokPj?= =?us-ascii?Q?hnoJ+byuPg3e7mOmjyyybib7QLOuHKIhn4ghcfCR+NSJrY3oOxh+nK1+0KKH?= =?us-ascii?Q?GZ0DZ2XQj7Cmsem5jTdZXCc0KjCtYxmdFaOH8Ja+M8WWEc/IzYUY+wp6KeJs?= =?us-ascii?Q?NYnl7//CJWHe4Wxr9YPuVSABDhMMPsLT6YzWqz5v/88d8eC+k6OiMv4R52t5?= =?us-ascii?Q?UyKjvUYf/JQ3N1elqL4sXkqiFSXMQKZCIxkCKkPcSTIElsGbsd07m/MmkVuV?= =?us-ascii?Q?3FRWpvYyqJ+EVrPm5dHqAVILf2xfPA2tji4Sjm7K5j87+5dPAL4vzXid+39H?= =?us-ascii?Q?CCGhCbiGXh/0EtM11Iu1kuzu85wPb5VxasqurRYlsM5cz6uwCsTM6RNPhylE?= =?us-ascii?Q?a9nh70MVc1eNMqpTL+goVdvglD4ReSfF08+7cHzF3/NdBb+b0PDlxAvbqg0w?= =?us-ascii?Q?m+qHLNGsjjzUFrWyuJBukcSPHkoAwgpLe9U4o5FBbipXCfi3WLI4oIwl9OgX?= =?us-ascii?Q?hp3saiB8C/BDDlomAoCXEjG1smaM+8AiHekBVzW0?= X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: AwMvr9tbnK2w2wtdCAFhRQ519u8/zRZh/X4H2Mrhb7Dxa7Z9SGW7h9nGNNFTPD1pKoroKrk7RsA7pQLXPO0qUd51e21SGgGn8rGr9xxpK7Ub8qEsNZzZrCiKjeQuh1j+bphvXcWyb8VbH+RS23Rd9ZJle3tZ8HThbNReg/Uwqe2s8SN2fivX5YuoBpLkgHD24TkIHBhUznVHXjplyJWwZDt4OQAVH2MKazJtmtvxAjGU0Pr+TNW2iR6WbF8YQr5bQ+XwsCryxRHBbJ2lYkkjcjvxqL1J4nIu7dBy+vf+wA1xtEBYluaIPto8qNbfuXLaCN2cKV58sEHsGLYk7Cggy8QJtpbhWwItHw0f0Htv/+eXrLMIOiafIV2Phpvu1CYlbjOvixMEOUZC/ngmM+/vTujBf+kQkMLHbxdh1VIQqU6AWBCrJX5qre6E2IyhlrIlm43C7HM+gax1PZKfYeIP/Hm54EYypfo0zdUvmDuq4a3YSmWW81ZE8b0hMsWtLKz70cA3EGMhX+nNZSND1ZBWlKASMme7bhOWQOGEtJjwj+ZWFbquADsY5dQsfV4yslGe7GvMIG5M9FePiOWiaAs/IEUlFZJ7cQvJB463dgo8xuY= X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 03bcedd3-ecc0-4395-576a-08dd83073c1a X-MS-Exchange-CrossTenant-AuthSource: CH3PR10MB7329.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 24 Apr 2025 08:08:45.4257 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: 2FGewwUUJFX70P3RGHG7febemAHooAv6+lGnoR3Eiey8mjoGXjWUSK0XSWM6m1rKR32ZL6cfRKvSTp4aF+T7Wg== X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH0PR10MB4905 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1099,Hydra:6.0.680,FMLib:17.12.80.40 definitions=2025-04-24_04,2025-04-22_01,2025-02-21_01 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 bulkscore=0 malwarescore=0 phishscore=0 suspectscore=0 mlxlogscore=999 adultscore=0 mlxscore=0 spamscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2504070000 definitions=main-2504240053 X-Proofpoint-GUID: Hn3Zr6g6TBTcvKJzhUWOXGm53UDAuChD X-Proofpoint-Spam-Details-Enc: AW1haW4tMjUwNDI0MDA1MyBTYWx0ZWRfXwf69TNGJ1ImM puS++nViVxfoyGD7zKj0dI5SA6nrPd61LDDLVMfh/zMuEtZRsLo57RoP3BS2p8KuCogQ8ONzEZv ZwsixvlWsWUFvV+QF0fAnZ3x28KrppwRvyivboyGNPm8oR1E4G4A9w4yyh4lNKeJ2SKCJHrj2eO UaKbNUw4zTftF3gpci3EbUYUAB2TtFmyIxlAew4xm8leAjBD+joTcSmHvX1X1ZC6Yj/Y6zqNUci 2wuzglzz4fr8Hs50/w0qmpqm7K2uJvUElJVbI1MMj/SOoPZm4D/JFvR7rDQmb6/Bur17EVVZ7GN gCbLbltwrbOKRn9unAI+I2uI7Zrrgld8jHETZGRICOVAV1Lx4nv2xNBJ4cwRXu43y5EQi0VxD49 I3uVKZmZ X-Proofpoint-ORIG-GUID: Hn3Zr6g6TBTcvKJzhUWOXGm53UDAuChD Content-Type: text/plain; charset="utf-8" When initializing newly allocated mm_struct, mm_init() allocate two chunks of percpu memory (pcpu_cid and rss_stat). Because percpu memory allocator uses a global mutex (pcpu_alloc_mutex), it becomes a global serialization point for exec(). Use slab ctor/dtor pair to allocate and free percpu memory (pcpu_cid, rss_stat) for mm_struct. mm_init_cid() is moved to mm_init(), and rss_stat percpu counter is charged in mm_init() and uncharged in __mmdrop(). As rss_stat and pcpu_cid fields should not be overwritten during memset() by mm_init() and memcy() by dup_mm(), move those fields to the end of mm_struct. Any field defined after 'ctor_fields_offset' won't be overwritten by these helpers. On the other hand, while cpu_bitmap[] is not initialized by the constructor, it should always be at the end of mm_struct. However, as cpu_bitmap[] is always initialized by mm_init(), not calling memset() and memcpy() for this field does not affect its current behavior. Note that check_mm() validates whether any rss counter is nonzero and reports an error if any nonzero value is found. In other words, under normal conditions, the counters should always be zero when an mm_struct is freed. Therefore is not necessary to reset the counters in mm_init() once they have been initialized by the constructor. To measure the performance impact, I ran the exec() benchmark [1], which launches one process per CPU and each proess repeatedly invokes exec(). On a desktop with 12 cores (24 hardware threads), it raises exec() throughput by an average of 4.56%. Even in a single-threaded run I see roughly a 4% gain, showing that the cost of acquiring and releasing pcpu_alloc_mutex is significant even when it is uncontended. On a dual-socket server with 192 cores the mutex becomes a contention hotpot; mitigating that contention boosts exec() throughput by 33%. Link: http://apollo.backplane.com/DFlyMisc/doexec.c [1] Link: https://lore.kernel.org/linux-mm/CAGudoHFc+Km-3usiy4Wdm1JkM+YjCgD9A8d= DKQ06pZP070f1ig@mail.gmail.com Suggested-by: Mateusz Guzik Signed-off-by: Harry Yoo --- include/linux/mm_types.h | 40 +++++++++++++++++--------- kernel/fork.c | 62 +++++++++++++++++++++++++++------------- 2 files changed, 69 insertions(+), 33 deletions(-) diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 56d07edd01f9..3000ca47b8ba 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -946,14 +946,6 @@ struct mm_struct { atomic_t mm_users; =20 #ifdef CONFIG_SCHED_MM_CID - /** - * @pcpu_cid: Per-cpu current cid. - * - * Keep track of the currently allocated mm_cid for each cpu. - * The per-cpu mm_cid values are serialized by their respective - * runqueue locks. - */ - struct mm_cid __percpu *pcpu_cid; /* * @mm_cid_next_scan: Next mm_cid scan (in jiffies). * @@ -982,6 +974,7 @@ struct mm_struct { * mm nr_cpus_allowed updates. */ raw_spinlock_t cpus_allowed_lock; + unsigned long _padding; /* for optimal offset of mmap_lock */ #endif #ifdef CONFIG_MMU atomic_long_t pgtables_bytes; /* size of all page tables */ @@ -1059,8 +1052,6 @@ struct mm_struct { =20 unsigned long saved_auxv[AT_VECTOR_SIZE]; /* for /proc/PID/auxv */ =20 - struct percpu_counter rss_stat[NR_MM_COUNTERS]; - struct linux_binfmt *binfmt; =20 /* Architecture-specific MM context */ @@ -1169,6 +1160,30 @@ struct mm_struct { #endif /* CONFIG_MM_ID */ } __randomize_layout; =20 + /* + * The fields below are not initialized by memset() or copied + * by memcpy(), in order to avoid overwriting values that are + * initialized by the slab constructor. + * + * The last field, cpu_bitmap, is an exception. This field is not + * initialized by the constructor and is always initialized by + * the mm_init() function. + */ + union { + unsigned long ctor_fields_offset; + struct percpu_counter rss_stat[NR_MM_COUNTERS]; + }; +#ifdef CONFIG_SCHED_MM_CID + /** + * @pcpu_cid: Per-cpu current cid. + * + * Keep track of the currently allocated mm_cid for each cpu. + * The per-cpu mm_cid values are serialized by their respective + * runqueue locks. + */ + struct mm_cid __percpu *pcpu_cid; +#endif + /* * The mm_cpumask needs to be at the end of mm_struct, because it * is dynamically sized based on nr_cpu_ids. @@ -1348,12 +1363,11 @@ static inline void mm_init_cid(struct mm_struct *mm= , struct task_struct *p) cpumask_clear(mm_cidmask(mm)); } =20 -static inline int mm_alloc_cid_noprof(struct mm_struct *mm, struct task_st= ruct *p) +static inline int mm_alloc_cid_noprof(struct mm_struct *mm) { mm->pcpu_cid =3D alloc_percpu_noprof(struct mm_cid); if (!mm->pcpu_cid) return -ENOMEM; - mm_init_cid(mm, p); return 0; } #define mm_alloc_cid(...) alloc_hooks(mm_alloc_cid_noprof(__VA_ARGS__)) @@ -1383,7 +1397,7 @@ static inline void mm_set_cpus_allowed(struct mm_stru= ct *mm, const struct cpumas } #else /* CONFIG_SCHED_MM_CID */ static inline void mm_init_cid(struct mm_struct *mm, struct task_struct *p= ) { } -static inline int mm_alloc_cid(struct mm_struct *mm, struct task_struct *p= ) { return 0; } +static inline int mm_alloc_cid(struct mm_struct *mm) { return 0; } static inline void mm_destroy_cid(struct mm_struct *mm) { } =20 static inline unsigned int mm_cid_size(void) diff --git a/kernel/fork.c b/kernel/fork.c index 7966b0876dc3..5940cf37379c 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -943,8 +943,7 @@ void __mmdrop(struct mm_struct *mm) check_mm(mm); put_user_ns(mm->user_ns); mm_pasid_drop(mm); - mm_destroy_cid(mm); - percpu_counter_destroy_many(mm->rss_stat, NR_MM_COUNTERS); + percpu_counter_uncharge_many(mm->rss_stat, NR_MM_COUNTERS); =20 free_mm(mm); } @@ -1295,7 +1294,6 @@ static struct mm_struct *mm_init(struct mm_struct *mm= , struct task_struct *p, mm->map_count =3D 0; mm->locked_vm =3D 0; atomic64_set(&mm->pinned_vm, 0); - memset(&mm->rss_stat, 0, sizeof(mm->rss_stat)); spin_lock_init(&mm->page_table_lock); spin_lock_init(&mm->arg_lock); mm_init_cpumask(mm); @@ -1328,21 +1326,17 @@ static struct mm_struct *mm_init(struct mm_struct *= mm, struct task_struct *p, if (init_new_context(p, mm)) goto fail_nocontext; =20 - if (mm_alloc_cid(mm, p)) - goto fail_cid; - - if (percpu_counter_init_many(mm->rss_stat, 0, GFP_KERNEL_ACCOUNT, - NR_MM_COUNTERS)) - goto fail_pcpu; + if (!percpu_counter_charge_many(mm->rss_stat, GFP_KERNEL_ACCOUNT, + NR_MM_COUNTERS)) + goto failed_charge; =20 + mm_init_cid(mm, p); mm->user_ns =3D get_user_ns(user_ns); lru_gen_init_mm(mm); return mm; =20 -fail_pcpu: +failed_charge: mm_destroy_cid(mm); -fail_cid: - destroy_context(mm); fail_nocontext: mm_free_id(mm); fail_noid: @@ -1363,7 +1357,7 @@ struct mm_struct *mm_alloc(void) if (!mm) return NULL; =20 - memset(mm, 0, sizeof(*mm)); + memset(mm, 0, offsetof(struct mm_struct, ctor_fields_offset)); return mm_init(mm, current, current_user_ns()); } EXPORT_SYMBOL_IF_KUNIT(mm_alloc); @@ -1725,7 +1719,7 @@ static struct mm_struct *dup_mm(struct task_struct *t= sk, if (!mm) goto fail_nomem; =20 - memcpy(mm, oldmm, sizeof(*mm)); + memcpy(mm, oldmm, offsetof(struct mm_struct, ctor_fields_offset)); =20 if (!mm_init(mm, tsk, mm->user_ns)) goto fail_nomem; @@ -3193,9 +3187,40 @@ static int sighand_ctor(void *data) return 0; } =20 +static int mm_struct_ctor(void *object) +{ + struct mm_struct *mm =3D object; + + if (mm_alloc_cid(mm)) + return -ENOMEM; + + if (percpu_counter_init_many(mm->rss_stat, 0, GFP_KERNEL, + NR_MM_COUNTERS)) { + mm_destroy_cid(mm); + return -ENOMEM; + } + + return 0; +} + +static void mm_struct_dtor(void *object) +{ + struct mm_struct *mm =3D object; + + mm_destroy_cid(mm); + percpu_counter_destroy_many(mm->rss_stat, NR_MM_COUNTERS); +} + void __init mm_cache_init(void) { unsigned int mm_size; + struct kmem_cache_args kmem_args =3D { + .align =3D ARCH_MIN_MMSTRUCT_ALIGN, + .useroffset =3D offsetof(struct mm_struct, saved_auxv), + .usersize =3D sizeof_field(struct mm_struct, saved_auxv), + .ctor =3D mm_struct_ctor, + .dtor =3D mm_struct_dtor, + }; =20 /* * The mm_cpumask is located at the end of mm_struct, and is @@ -3204,12 +3229,9 @@ void __init mm_cache_init(void) */ mm_size =3D sizeof(struct mm_struct) + cpumask_size() + mm_cid_size(); =20 - mm_cachep =3D kmem_cache_create_usercopy("mm_struct", - mm_size, ARCH_MIN_MMSTRUCT_ALIGN, - SLAB_HWCACHE_ALIGN|SLAB_PANIC|SLAB_ACCOUNT, - offsetof(struct mm_struct, saved_auxv), - sizeof_field(struct mm_struct, saved_auxv), - NULL); + mm_cachep =3D kmem_cache_create("mm_struct", mm_size, &kmem_args, + SLAB_HWCACHE_ALIGN|SLAB_PANIC| + SLAB_ACCOUNT); } =20 void __init proc_caches_init(void) --=20 2.43.0