From nobody Fri Dec 19 04:52:36 2025 Received: from NAM12-DM6-obe.outbound.protection.outlook.com (mail-dm6nam12on2057.outbound.protection.outlook.com [40.107.243.57]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 07AB522540B; Tue, 10 Dec 2024 09:49:07 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.243.57 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1733824149; cv=fail; b=b+fsHoxqDY2TCPve3XTt0+SzKWoqlyejxQcZdEqzFzKKWA4K0K82f0cBB78guq6e5ci+ZtZ5DJ18+zwSQ40uLh5GNiO7jp9RAeF7foMRmJh6z2+rE+XhToYqPDk0ML5PdYnqSsYvE4q/auxAfWU0cDaGuR0y3zaJAKBMnibGXnM= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1733824149; c=relaxed/simple; bh=Txpd8WfU6smANj+UIs6BwcUM03ycB9MDqRLazTQnhLI=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=GuJSXCgkPVtiLLTRGmkhf3OzHH2W+eleT3TXDR1LD/tdmlteRGTkYBTs2TbJwj3LBbDE+FFo60lFnA2xcmEbSwNX1xgSLuBG0DJXtwt+Oo0RQCOmRKIbulmLkQFLZ62iDSqyA7O+4xJKu8thx72LpseH85AlvbRvyTd7Bm30mjs= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=etz1KsKZ; arc=fail smtp.client-ip=40.107.243.57 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="etz1KsKZ" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=FPBHn5E5NLEDx2M5mVhmeFCgEP8qW2i//elzXjTCBw0gVZPGz1EdpK367IvPpeT4iesCW6cBclqreZQbxUPvabn+uYTHVd+KpUm2P8BJ47VBuircoubpUoZXWrj3tB/IMG5ycnfgBGj4hCiHO55rLgdl7725oy+GbRb3YYz579rAkPR2xc7Lb/9uOaYFiaeYZ3H0rI2qtgn7/iMj7LgCJ4jpqbkZUJoUXr/WDxyn59O8xYnhUwGT9hoCq3YszIgu42Wii6AG2FQeXFktmCxoO7cIOMhNUAzW/ehGb3A0CsDwpTF3Hc531RFMMnSWoSERzJFslOGRAn8OZsTtEiOPqg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=XaNMzxoG2eScPPvYOEWiCLCN/uTciIDbMbfl7KLstPQ=; b=ln0+1EdOrbhrtVpUTR39OHzF0FddqiF9liGI6ZtqxLhpp1i5+zy/8BNsvgqC0vWaRr926yz0klhguTOwtMI1oGxsFeLworDpqx+M4V3NO0LbEmH4aM919RTj2UPLaIoJpxoPfeuNZqwh2lVID8g+S3xnViVq+klwpS7d+lLUChQzDTx76/o85ljkwtFl1nvYCUcE/0ASRCNlhlxaHE1PFu7JUsPspvXb610Xn+8edyrAuMfU8p1Xb17CUWZRUu3Fyv4+WjprNAgrj8vzHMGK8lMPCdjyY0AG3hHmFs1+ry0M0SSfQAlfwschjNQ2YOeuI0eXLokiESY9C5knmZF68g== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=kernel.dk smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=XaNMzxoG2eScPPvYOEWiCLCN/uTciIDbMbfl7KLstPQ=; b=etz1KsKZ4C/8YXM94B0RAg78iPizwPZEMZRvjEqRHj70F95ZYjHY+TKqlMALYGwBPj4AoMymrNDvChaaOvk+cgU0ulkIirPp3MCyvN/JivalShwLyqRzprd08bYVdS+FBOBtaYivxqH/oEbP2cHBAHOTGRWjqHf5lZ4Pa5cWYXE= Received: from BL1PR13CA0259.namprd13.prod.outlook.com (2603:10b6:208:2ba::24) by SA1PR12MB9492.namprd12.prod.outlook.com (2603:10b6:806:459::7) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8230.18; Tue, 10 Dec 2024 09:49:02 +0000 Received: from BL6PEPF00020E64.namprd04.prod.outlook.com (2603:10b6:208:2ba:cafe::40) by BL1PR13CA0259.outlook.office365.com (2603:10b6:208:2ba::24) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.8251.12 via Frontend Transport; Tue, 10 Dec 2024 09:49:01 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; pr=C Received: from SATLEXMB04.amd.com (165.204.84.17) by BL6PEPF00020E64.mail.protection.outlook.com (10.167.249.25) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.8230.7 via Frontend Transport; Tue, 10 Dec 2024 09:49:01 +0000 Received: from BLR-L-BHARARAO.amd.com (10.180.168.240) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Tue, 10 Dec 2024 03:48:58 -0600 From: Bharata B Rao To: CC: , , , , , , , Subject: [PATCHSET v6 0/12] Uncached buffered IO Date: Tue, 10 Dec 2024 15:18:42 +0530 Message-ID: <20241210094842.204504-1-bharata@amd.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20241203153232.92224-2-axboe@kernel.dk> References: <20241203153232.92224-2-axboe@kernel.dk> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: SATLEXMB03.amd.com (10.181.40.144) To SATLEXMB04.amd.com (10.181.40.145) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BL6PEPF00020E64:EE_|SA1PR12MB9492:EE_ X-MS-Office365-Filtering-Correlation-Id: 64000f0b-e162-4ebb-f0d0-08dd18ffe05b X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|1800799024|36860700013|376014|82310400026; X-Microsoft-Antispam-Message-Info: =?utf-8?B?bndJK2sxV3crZURXTmJYWjRJYlZGL2U4bXNNM1pDUm5mWXg3VkJwR05QdFFl?= =?utf-8?B?QUdpT2RFV2RpcXhnd2FOY1U1a3lsbHpmamNodURtSnEyTE5BaGgwYVYyRDJG?= =?utf-8?B?SUdHVE1DT0RYbExCNk9NQ2t5NjU5YWw1ZVJCT0VqU1R3MGVpRUp6UkZEN01m?= =?utf-8?B?RmpTcGdZb3JnZ1F6N0pVSHRUaU1ZVW9ZTDFHQ3M1cTNOVEpDTWJ3NDY2Tyt3?= =?utf-8?B?SXNOamtiWWRKQmVGcXJtTVc0a0NBRStsRXVKQStsekZsTUFRd0NrVGRoSXdl?= =?utf-8?B?eEY4aFpCM1htZTdZWG4yUkwvM251aXZKY0JXam4zZVpnWXlUbUU2SnhBWU9l?= =?utf-8?B?RExBcmxuamhGME1YbU5adHlBVTVTWFEyQm84MDNSZ0k0dnhOdkU2TmFkTFZC?= =?utf-8?B?cEU2VFBLTXlITzBNakRTR1dEa0JTTzhDSjV4VE5yZ1F1N3FtTEVaYUZyeW9W?= =?utf-8?B?R2NQQjJVdEdSeUtTRjRkRmxyU0x3dTlac2dHYkZUMEphUVZ3ZlVyU1g3SUpt?= =?utf-8?B?dFB0SnFaYjNWeEx3amhvTUtoMTkzZkR6blNsTi81MklvWEg0NlBZODVEa2pR?= =?utf-8?B?d3BFQk1PclhoNzhZVzhjdExiSnZ0aENuOGZWbEhINkQ0V1lmYVc5RHNHd2lz?= =?utf-8?B?NXVwNkFuQnFSUlZmZElrV002UG9BOUJRUkRvRy9RVkJNdmY2aFY0TW5Oa044?= =?utf-8?B?UzFKTmlNMXNHQzRTUDdOdnRtSlhobjMralh2K3UvZDdHc1pxMFk2aHpndks5?= =?utf-8?B?YUNDWjBOalBtS0JUYlR1TTM1RnJRSkNhaDFzMmwxbVQwY2lpeWxFbHhJZnln?= =?utf-8?B?aS9udkQ0Nm9yN0lMSEU3WnZJODBaVWpzOXdpVXN6Vmt4U0htaGo3SDhDVFhk?= =?utf-8?B?UkVtYjhINEVlRzlydEVOOVR5WU9BVUN4cUwyVG15M015bVZjM1dMV3FocjlX?= =?utf-8?B?TXRpRG9iTlRmM1F4czY5cEZZVWx0Vzd6ZTFWSkk5dGZVbktyeWZMVWw2Q21y?= =?utf-8?B?bnpOSUdKMXBiQ2lQYkdqZVhkSG9zTlMwQ29Ta0w5SnJDOUpORHVuQTZXb3Q1?= =?utf-8?B?Z0VUVlBqQ0p6a3NrMEFtVHpiUCsyb0VHK1hLMHRVeWpaZ2RNQVRXR2R1Rzdu?= =?utf-8?B?KzRMbzNIcnBuS2dnMTZsZzJydmFLcmVaSkoySi9OOG52b2dKNDhVZVp0a25V?= =?utf-8?B?UnhHclEzN25Ick9ZaFpzK3Y4SU1qM2FRaHArYXN5cGczWC9vT21kOVFSQW8r?= =?utf-8?B?S2lpVUhENXAwRTBXa0Uzb0h3TnRSWVFXb0ZrWHRTVU9BdWtSVm10K3gyT0Vn?= =?utf-8?B?VTAzZ08rSjdrNFFzM2sxSmNsUCtWb0JIVk9RbFBXeFZHVUYzZm5hVGtwLytI?= =?utf-8?B?ZTdJM3pkL2xuSEdwcHJBaTZUSFJyYW9HQ0g1RThRQjdSQ09YdERGTERPQ2V4?= =?utf-8?B?WTczWEZ0cy8rN0N5VkZMVEdpMXdJc09hNkNIS0hjbzFaRE9wOTRCYjN1NjdX?= =?utf-8?B?SlB0R25FRW9ZMlU4ZFlha2lieFU4VFZsRnJVY090TUY3dVpCKzhyOGVhaVln?= =?utf-8?B?eVRKdnp0cnMxRGQ4MlpnekhMTFVzY2k2YW56UFBhVG5JR0RSK0ttRUp2QXZY?= =?utf-8?B?VXBKeGpIcGc0eDBpVzNyajlHQm5HZnE1cjVrUndtdlgwR3M3eXJZZDFlQVM2?= =?utf-8?B?VXRyTmZJaC9CYmlZTlBUaHhWY3VEWk5OWjQrRHZQVVFOdDYxakJDMUFkbEpy?= =?utf-8?B?VmhOSmNDVDFxc3VuTSs0RWFYcVFBbE5KNXd2UVNTT2FCSkpMODVaSHBsYXpJ?= =?utf-8?B?dWxhWVIwSlRBR2JlQzl1R1VQN3VEWUNjSzFmN1l1NlhiMGZCM25xNlozQVZz?= =?utf-8?B?Rlh3NEk2a3RSMFJHU2dSdmxCL3Z1bjVDRllhS01MeGd6Q1E9PQ==?= X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB04.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(1800799024)(36860700013)(376014)(82310400026);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 10 Dec 2024 09:49:01.5702 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 64000f0b-e162-4ebb-f0d0-08dd18ffe05b X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB04.amd.com] X-MS-Exchange-CrossTenant-AuthSource: BL6PEPF00020E64.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA1PR12MB9492 Hi Jens, I ran a couple of variants of FIO to check how this patchset affects the FIO numbers. My other motivation with this patchset is to check if it improves the scalability issues that were discussed in [1] and [2]. But for now, here are some initial numbers. To enabled uncached buffered IO, I have modified FIO pvsync2 engine to issue preadv2()/pwritev2() calls with RWF_UNCACHED set. The FIO change looks like below and I assume this is good enough to correctly use this patchset. diff --git a/engines/sync.c b/engines/sync.c index b8be4eb3..44e9da3d 100644 Acked-by: "Darrick J. Wong" --- a/engines/sync.c +++ b/engines/sync.c @@ -170,6 +170,8 @@ static enum fio_q_status fio_pvsyncio2_queue(struct thr= ead_data *td, if (o->nowait) flags |=3D RWF_NOWAIT; =20 + flags |=3D RWF_UNCACHED; + iov->iov_base =3D io_u->xfer_buf; iov->iov_len =3D io_u->xfer_buflen; =20 Also note that I am using your buffered-uncached.8 branch from https://git.kernel.dk/cgit/linux/log/?h=3Dbuffered-uncached.8 that has changes to enable uncached buffered IO for EXT4 and block devices. In the below reported numbers, 'base' means kernel from buffered-uncached.8 branch and 'patched' means kernel from buffered-uncached.8 branch + above shown FIO ch= ange FIO on EXT4 partitions =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D nvme1n1 259:12 0 3.5T 0 disk=20 =E2=94=9C=E2=94=80nvme1n1p1 259:13 0 894.3G 0 part /mnt1 =E2=94=9C=E2=94=80nvme1n1p2 259:14 0 894.3G 0 part /mnt2 =E2=94=9C=E2=94=80nvme1n1p3 259:15 0 894.3G 0 part /mnt3 =E2=94=94=E2=94=80nvme1n1p4 259:16 0 894.1G 0 part /mnt4 fio -directory=3D/mnt4/ -direct=3D0 -thread -size=3D3G -rw=3Drw -rwmixwrite= =3D30 --norandommap --randrepeat=3D0 -ioengine=3Dpvsync2 -bs=3D64k -numjobs= =3D252 -runtime=3D3600 --time_based -group_reporting -name=3Dmytest fio -directory=3D/mnt3/ -direct=3D0 -thread -size=3D3G -rw=3Drw -rwmixwrite= =3D30 --norandommap --randrepeat=3D0 -ioengine=3Dpvsync2 -bs=3D64k -numjobs= =3D252 -runtime=3D3600 --time_based -group_reporting -name=3Dmytest fio -directory=3D/mnt1/ -direct=3D0 -thread -size=3D3G -rw=3Drw -rwmixwrite= =3D30 --norandommap --randrepeat=3D0 -ioengine=3Dpvsync2 -bs=3D64k -numjobs= =3D252 -runtime=3D3600 --time_based -group_reporting -name=3Dmytest fio -directory=3D/mnt2/ -direct=3D0 -thread -size=3D3G -rw=3Drw -rwmixwrite= =3D30 --norandommap --randrepeat=3D0 -ioengine=3Dpvsync2 -bs=3D64k -numjobs= =3D252 -runtime=3D3600 --time_based -group_reporting -name=3Dmytest Four NVME devices are formatted with EXT4 and four parallel FIO instances are run on them with the options as shown above. FIO output looks like this: base: READ: bw=3D1233MiB/s (1293MB/s), 1233MiB/s-1233MiB/s (1293MB/s-1293MB/s)= , io=3D4335GiB (4654GB), run=3D3600097-3600097msec WRITE: bw=3D529MiB/s (554MB/s), 529MiB/s-529MiB/s (554MB/s-554MB/s), io= =3D1858GiB (1995GB), run=3D3600097-3600097msec READ: bw=3D1248MiB/s (1308MB/s), 1248MiB/s-1248MiB/s (1308MB/s-1308MB/s)= , io=3D4387GiB (4710GB), run=3D3600091-3600091msec WRITE: bw=3D535MiB/s (561MB/s), 535MiB/s-535MiB/s (561MB/s-561MB/s), io= =3D1880GiB (2019GB), run=3D3600091-3600091msec READ: bw=3D1235MiB/s (1294MB/s), 1235MiB/s-1235MiB/s (1294MB/s-1294MB/s)= , io=3D4340GiB (4660GB), run=3D3600094-3600094msec WRITE: bw=3D529MiB/s (555MB/s), 529MiB/s-529MiB/s (555MB/s-555MB/s), io= =3D1860GiB (1997GB), run=3D3600094-3600094msec READ: bw=3D1234MiB/s (1294MB/s), 1234MiB/s-1234MiB/s (1294MB/s-1294MB/s)= , io=3D4337GiB (4657GB), run=3D3600093-3600093msec WRITE: bw=3D529MiB/s (554MB/s), 529MiB/s-529MiB/s (554MB/s-554MB/s), io= =3D1859GiB (1996GB), run=3D3600093-3600093msec patched: READ: bw=3D1400MiB/s (1469MB/s), 1400MiB/s-1400MiB/s (1469MB/s-1469MB/s)= , io=3D4924GiB (5287GB), run=3D3600100-3600100msec WRITE: bw=3D600MiB/s (629MB/s), 600MiB/s-600MiB/s (629MB/s-629MB/s), io= =3D2110GiB (2266GB), run=3D3600100-3600100msec READ: bw=3D1395MiB/s (1463MB/s), 1395MiB/s-1395MiB/s (1463MB/s-1463MB/s)= , io=3D4904GiB (5266GB), run=3D3600148-3600148msec WRITE: bw=3D598MiB/s (627MB/s), 598MiB/s-598MiB/s (627MB/s-627MB/s), io= =3D2102GiB (2257GB), run=3D3600148-3600148msec READ: bw=3D1385MiB/s (1452MB/s), 1385MiB/s-1385MiB/s (1452MB/s-1452MB/s)= , io=3D4868GiB (5227GB), run=3D3600136-3600136msec WRITE: bw=3D594MiB/s (622MB/s), 594MiB/s-594MiB/s (622MB/s-622MB/s), io= =3D2087GiB (2241GB), run=3D3600136-3600136msec READ: bw=3D1376MiB/s (1443MB/s), 1376MiB/s-1376MiB/s (1443MB/s-1443MB/s)= , io=3D4837GiB (5194GB), run=3D3600145-3600145msec WRITE: bw=3D590MiB/s (618MB/s), 590MiB/s-590MiB/s (618MB/s-618MB/s), io= =3D2073GiB (2226GB), run=3D3600145-3600145msec FIO on block devices =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D nvme1n1 259:12 0 3.5T 0 disk=20 =E2=94=9C=E2=94=80nvme1n1p1 259:13 0 894.3G 0 part=20 =E2=94=9C=E2=94=80nvme1n1p2 259:14 0 894.3G 0 part=20 =E2=94=9C=E2=94=80nvme1n1p3 259:15 0 894.3G 0 part=20 =E2=94=94=E2=94=80nvme1n1p4 259:16 0 894.1G 0 part=20 fio -filename=3D/dev/nvme1n1p4 -direct=3D0 -thread -size=3D800G -rw=3Drw -r= wmixwrite=3D30 --norandommap --randrepeat=3D0 -ioengine=3Dpvsync2 -bs=3D64k= -numjobs=3D252 -runtime=3D3600 --time_based -group_reporting -name=3Dmytest fio -filename=3D/dev/nvme1n1p2 -direct=3D0 -thread -size=3D800G -rw=3Drw -r= wmixwrite=3D30 --norandommap --randrepeat=3D0 -ioengine=3Dpvsync2 -bs=3D64k= -numjobs=3D252 -runtime=3D3600 --time_based -group_reporting -name=3Dmytest fio -filename=3D/dev/nvme1n1p1 -direct=3D0 -thread -size=3D800G -rw=3Drw -r= wmixwrite=3D30 --norandommap --randrepeat=3D0 -ioengine=3Dpvsync2 -bs=3D64k= -numjobs=3D252 -runtime=3D3600 --time_based -group_reporting -name=3Dmytest fio -filename=3D/dev/nvme1n1p3 -direct=3D0 -thread -size=3D800G -rw=3Drw -r= wmixwrite=3D30 --norandommap --randrepeat=3D0 -ioengine=3Dpvsync2 -bs=3D64k= -numjobs=3D252 -runtime=3D3600 --time_based -group_reporting -name=3Dmytest Four instances of FIO are run on four different NVME block devices with the options as shown above. base: READ: bw=3D8712MiB/s (9135MB/s), 8712MiB/s-8712MiB/s (9135MB/s-9135MB/s)= , io=3D29.9TiB (32.9TB), run=3D3600011-3600011msec WRITE: bw=3D3734MiB/s (3915MB/s), 3734MiB/s-3734MiB/s (3915MB/s-3915MB/s)= , io=3D12.8TiB (14.1TB), run=3D3600011-3600011msec READ: bw=3D8727MiB/s (9151MB/s), 8727MiB/s-8727MiB/s (9151MB/s-9151MB/s)= , io=3D30.0TiB (32.9TB), run=3D3600005-3600005msec WRITE: bw=3D3740MiB/s (3922MB/s), 3740MiB/s-3740MiB/s (3922MB/s-3922MB/s)= , io=3D12.8TiB (14.1TB), run=3D3600005-3600005msec READ: bw=3D8701MiB/s (9123MB/s), 8701MiB/s-8701MiB/s (9123MB/s-9123MB/s)= , io=3D29.9TiB (32.8TB), run=3D3600004-3600004msec WRITE: bw=3D3729MiB/s (3910MB/s), 3729MiB/s-3729MiB/s (3910MB/s-3910MB/s)= , io=3D12.8TiB (14.1TB), run=3D3600004-3600004msec READ: bw=3D8706MiB/s (9128MB/s), 8706MiB/s-8706MiB/s (9128MB/s-9128MB/s)= , io=3D29.9TiB (32.9TB), run=3D3600005-3600005msec WRITE: bw=3D3731MiB/s (3913MB/s), 3731MiB/s-3731MiB/s (3913MB/s-3913MB/s)= , io=3D12.8TiB (14.1TB), run=3D3600005-3600005msec patched: READ: bw=3D1844MiB/s (1933MB/s), 1844MiB/s-1844MiB/s (1933MB/s-1933MB/s)= , io=3D6500GiB (6980GB), run=3D3610641-3610641msec WRITE: bw=3D790MiB/s (828MB/s), 790MiB/s-790MiB/s (828MB/s-828MB/s), io= =3D2786GiB (2991GB), run=3D3610642-3610642msec READ: bw=3D1753MiB/s (1838MB/s), 1753MiB/s-1753MiB/s (1838MB/s-1838MB/s)= , io=3D6235GiB (6695GB), run=3D3641973-3641973msec WRITE: bw=3D751MiB/s (788MB/s), 751MiB/s-751MiB/s (788MB/s-788MB/s), io= =3D2672GiB (2869GB), run=3D3641969-3641969msec READ: bw=3D1078MiB/s (1130MB/s), 1078MiB/s-1078MiB/s (1130MB/s-1130MB/s)= , io=3D3788GiB (4068GB), run=3D3600007-3600007msec WRITE: bw=3D462MiB/s (484MB/s), 462MiB/s-462MiB/s (484MB/s-484MB/s), io= =3D1624GiB (1743GB), run=3D3600007-3600007msec READ: bw=3D1752MiB/s (1838MB/s), 1752MiB/s-1752MiB/s (1838MB/s-1838MB/s)= , io=3D6234GiB (6694GB), run=3D3642657-3642657msec WRITE: bw=3D751MiB/s (788MB/s), 751MiB/s-751MiB/s (788MB/s-788MB/s), io= =3D2672GiB (2869GB), run=3D3642622-3642622msec While FIO on FS shows improvement, FIO on block shows numbers going down. Is this expected or am I missing enabling anything else for the block optio= n? Regards, Bharata. [1] https://lore.kernel.org/linux-mm/d2841226-e27b-4d3d-a578-63587a3aa4f3@a= md.com/ [2] https://lore.kernel.org/linux-fsdevel/20241127054737.33351-1-bharata@am= d.com/ From nobody Fri Dec 19 04:52:36 2025 Received: from mail-oi1-f176.google.com (mail-oi1-f176.google.com [209.85.167.176]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id AE7681F759C for ; Tue, 3 Dec 2024 15:32:47 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.167.176 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1733239971; cv=none; b=OJQqFFCUaUw0W1xN1sVmXprMTP8tJa88uJBoGLUa3AEuOSHOvxx1lXo8MYVyjgnsc2n0y6NJa+dbyO2KwQo/r3iQ6wHLVTF2aglGgz2XSBfinndb5P9sEhnsPrLAo9Nxv2fZGaLzCHl9VjBmQVYlMu7vRl/EA5N2ZtFLpdEUH+8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1733239971; c=relaxed/simple; bh=89Ccy3GI5ZUKxFiNR9lyKoxkgOUx6zEEiBrhRigykbc=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=SlxJFQM7kLhAZxhYrzLxpjP+a8PBNmYE8ROvVEDkfVMOvzjWXdev8ljl+H7dITkHW9hfw4FaiCEcXHi2xXbJDa6czUwOKDsjgjWB5dOs6RlqwX27TtK0oEnqHzdx+IujJXyjzfeakhw/dtCRDgLJBf9sSLYaRB1GxzAWtFsf/qw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=ssdLS1LP; arc=none smtp.client-ip=209.85.167.176 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="ssdLS1LP" Received: by mail-oi1-f176.google.com with SMTP id 5614622812f47-3ea5e405870so2463532b6e.3 for ; Tue, 03 Dec 2024 07:32:47 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1733239967; x=1733844767; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=ReI+Zr1x7wQ/SlfmJGhzAq3WrMPRqrYdy9B8AN1mepA=; b=ssdLS1LPBUHL1S31Q32Y/smkGXM0559dx9pYbkyDBWJ7eY6YZD+xY2uqJAfV12RuVW xwtblMjAy2gaeg7xJ3mkFST5v0G+JzIFu+gF7FWes+8Q7MPyarpDkxEWlN68cx39P79p qjZbtmWeQiUaON9nakCBeBVepmGECBc+cSpKmS1pPqYlRb+My0DohaccrNIzgMocBMs0 BERVtEJ35+2veCbgFwsQISDue53j3fIhlrIk9B843TKalWJPW9LaX+5PkTmNnFHdFkKf BSbv+W/kFJrSiXxrJY/NJvfPwSfbmL+9mK/81i1/r5v5p7J86z08tUOY9ObHCH7ng6e9 sroQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1733239967; x=1733844767; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ReI+Zr1x7wQ/SlfmJGhzAq3WrMPRqrYdy9B8AN1mepA=; b=LjjBc2HYgTmS4DxlDgPILGdtwwaKIEXsziXFnGu10yT+fYZwJCeuRMJsFj7/5+cFSc NJxxpccFlAC3j1t+V8X93JEgRT5+PUhFrBOLQ0xn78kurGqwHE8aexfCElZ9ZyrHKgE/ Rz7/E248BXnOByOPB7mHM9PcAod7ocDnZMzYjM8cJeUqbypTVEJIQrdIk4oS72HwDOND QOq07TCS0COOHc6WOY6CFOhlXRUB6nZfq5YejUnwYpFbOYsq7H8/+t9PdeZOKr4z0cue w9lFaf4THgL4ZzlNVou12frWdWNU7SPljQgolAXJoNoHpwB06VrR3ocBsRMZjntFbeWM kj4Q== X-Forwarded-Encrypted: i=1; AJvYcCWXGj4Orv6UEazHXok2SJObPJyufGNV3M5NsYhGqHIi/e5inYpWzjUZSY7UEjTxnDMcgu21hXIbb40rBjg=@vger.kernel.org X-Gm-Message-State: AOJu0YxI9vNHW6XqDZRwlDIDba29H7P8kfIwoT/K5ulmS3A4x3RPgZMN HTk/VZEZFpV/d6u0eAGQ2e3a8AZW75oS8tMNs6CMh1lrZHQqvlJ3JxOCOC7LDBNf2qHEqszBe6c e X-Gm-Gg: ASbGncsT2M5dFvsJe3OF6ldqFfBQ7rEN7U2SuyVbrfJjD64pUJbcGij0WuTF1t7yA0C fE9zHlpXEOGXhSsyBprCj5gtbRF5LaulckcZe/+1WhnEL2UaV+BiXwdhJ/aY2ott7WVkqVFbtnS NxbPzP9kuiYlLPSCWggGShfGBoOGq7rX0agPCO4QGG43xHUiVdKweFUKXP6ljCUHWOOo1Xemi0Z QO8JUv/pFPnfH8UbjEjiCOQYe4Yp3OA1UdtU3hq4cWam/6JVO0rVtXDIdE= X-Google-Smtp-Source: AGHT+IEjGwc7GFJjqyvYWDgJBqgGGBcMfQCEEJJRf9YUZZ1W6BgQFHNlpJWElLeXDmEpw+9XPHnUvQ== X-Received: by 2002:a05:6808:1406:b0:3e6:1ea5:6b30 with SMTP id 5614622812f47-3eae4f9162amr3792647b6e.24.1733239966815; Tue, 03 Dec 2024 07:32:46 -0800 (PST) Received: from localhost.localdomain ([130.250.255.163]) by smtp.gmail.com with ESMTPSA id 5614622812f47-3ea86036cbbsm2891878b6e.8.2024.12.03.07.32.45 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 03 Dec 2024 07:32:46 -0800 (PST) From: Jens Axboe To: linux-mm@kvack.org, linux-fsdevel@vger.kernel.org Cc: hannes@cmpxchg.org, clm@meta.com, linux-kernel@vger.kernel.org, willy@infradead.org, kirill@shutemov.name, bfoster@redhat.com, Jens Axboe Subject: [PATCH 01/12] mm/filemap: change filemap_create_folio() to take a struct kiocb Date: Tue, 3 Dec 2024 08:31:37 -0700 Message-ID: <20241203153232.92224-3-axboe@kernel.dk> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20241203153232.92224-2-axboe@kernel.dk> References: <20241203153232.92224-2-axboe@kernel.dk> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Rather than pass in both the file and position directly from the kiocb, just take a struct kiocb instead. While doing so, move the ki_flags checking into filemap_create_folio() as well. In preparation for actually needing the kiocb in the function. No functional changes in this patch. Signed-off-by: Jens Axboe Acked-by: "Darrick J. Wong" --- mm/filemap.c | 17 +++++++++-------- 1 file changed, 9 insertions(+), 8 deletions(-) diff --git a/mm/filemap.c b/mm/filemap.c index 7c76a123ba18..898e992039e8 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -2459,15 +2459,17 @@ static int filemap_update_page(struct kiocb *iocb, return error; } =20 -static int filemap_create_folio(struct file *file, - struct address_space *mapping, loff_t pos, - struct folio_batch *fbatch) +static int filemap_create_folio(struct kiocb *iocb, + struct address_space *mapping, struct folio_batch *fbatch) { struct folio *folio; int error; unsigned int min_order =3D mapping_min_folio_order(mapping); pgoff_t index; =20 + if (iocb->ki_flags & (IOCB_NOWAIT | IOCB_WAITQ)) + return -EAGAIN; + folio =3D filemap_alloc_folio(mapping_gfp_mask(mapping), min_order); if (!folio) return -ENOMEM; @@ -2486,7 +2488,7 @@ static int filemap_create_folio(struct file *file, * well to keep locking rules simple. */ filemap_invalidate_lock_shared(mapping); - index =3D (pos >> (PAGE_SHIFT + min_order)) << min_order; + index =3D (iocb->ki_pos >> (PAGE_SHIFT + min_order)) << min_order; error =3D filemap_add_folio(mapping, folio, index, mapping_gfp_constraint(mapping, GFP_KERNEL)); if (error =3D=3D -EEXIST) @@ -2494,7 +2496,8 @@ static int filemap_create_folio(struct file *file, if (error) goto error; =20 - error =3D filemap_read_folio(file, mapping->a_ops->read_folio, folio); + error =3D filemap_read_folio(iocb->ki_filp, mapping->a_ops->read_folio, + folio); if (error) goto error; =20 @@ -2550,9 +2553,7 @@ static int filemap_get_pages(struct kiocb *iocb, size= _t count, filemap_get_read_batch(mapping, index, last_index - 1, fbatch); } if (!folio_batch_count(fbatch)) { - if (iocb->ki_flags & (IOCB_NOWAIT | IOCB_WAITQ)) - return -EAGAIN; - err =3D filemap_create_folio(filp, mapping, iocb->ki_pos, fbatch); + err =3D filemap_create_folio(iocb, mapping, fbatch); if (err =3D=3D AOP_TRUNCATED_PAGE) goto retry; return err; --=20 2.45.2 From nobody Fri Dec 19 04:52:36 2025 Received: from mail-oi1-f181.google.com (mail-oi1-f181.google.com [209.85.167.181]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4A16B1F942C for ; Tue, 3 Dec 2024 15:32:49 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.167.181 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1733239971; cv=none; b=JdGdyOn8Mn05miQTX4xRyVhJvFgjYBukySt0uEbgZln65cMl1bKpiFvkHjFqJcJFgwcSLg6+/wiwrBDn+RQwsj4c5H0ktjqUhLH0M14as/9JLcBguSC2UshvkiON5eW301513IKILx2deZL9DXZNBEGLhyvJJ2VFuRqxQW3CvUc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1733239971; c=relaxed/simple; bh=8Bb0MfS/J32mJVu/hCpTfSTIXsQUeZKlcznJ/+p92u4=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=qV9bktfmXCAexAlJ91v8h2OPVXvYJNAkNgBBP4xlI+UsaDTIGUyIs16XNhcpHtSAUkX09YdU+Ejf4CPWkqQUdBCwndYq7YDrPIhgii9bp1LHLqKhP2H0UOUonvZwv3K8zabDQXsClVseREPpCL7nMVZfNia/onNjDR1n/x/Lpjg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=vI3+LtaN; arc=none smtp.client-ip=209.85.167.181 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="vI3+LtaN" Received: by mail-oi1-f181.google.com with SMTP id 5614622812f47-3ea6b011b27so3129180b6e.0 for ; Tue, 03 Dec 2024 07:32:48 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1733239968; x=1733844768; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=5mQz2vdNaN9LdJ9L4qyJBm7ibQRSfX5SzH0nqDyaIZA=; b=vI3+LtaNq/TnQ0w9S52h9H8vdWp5zjIWm4vmECGM4OLUaQkVfjYLWSZeBzwm8RkN+t QZBOBOs37SFN0AXY4lgL8QvsA+LIbBOvJy3jYzYBEf0DFjOW2Or/4MqeT4sUsqcdyKxH krxdvKTFDTUK4+uMvXVKGrP0QdooVKLGOdsjyPFAGiaROrSAHqUKPQqb2Esm7zmu755q L3snlWcVxQRYPSKR05GHn/anLu77M0zcXS+LauovAaCJGM/WjuKY9xah6ct2oTdUJ2vf 7xoXj9DChXxJKIhOnduYaKYh1Aiuwk/HeB70IdkRDyMFoj6lCcMhzdBpqWm16E2K/Wkb URNA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1733239968; x=1733844768; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=5mQz2vdNaN9LdJ9L4qyJBm7ibQRSfX5SzH0nqDyaIZA=; b=Q6pdBmAEhCYyjl2FdpNjZjQKf3o23+jUWv6ZJgUE6TOl8U8p417EZyNV0WLWd9ho17 0m2ZUUvjv1x6hQ4ndgS8MoZVq753vJKPNPELQBxji/lasJKBgpUMgYqujJswVWo5422E PmfFbNQ78dbol3fnilvtN58uX6fE2l8nXHNMbnKGFnA9mJfHO1Kqn+l/6JLWLAV91Tr/ j0Y+vLU3QSQ9rK0+qmXmxpVbrTDMmLsHkiIjMyN2P4alvK0SX3FkQoRkBOxu8cgeOmc5 6AMBAqtSBlseFKYtuPC9+86uBO0321dSr3X2m4Vhi1gloych5XjE7aISMf3GWcmfr5Il +08Q== X-Forwarded-Encrypted: i=1; AJvYcCVmgRxfcdI5LEF847663Wr7nW0TLtj7k672P0DFop0hafsxZ4sP9e04Y7KxEQ7An1YHKK3vA7imKDQez/w=@vger.kernel.org X-Gm-Message-State: AOJu0Yxnp4nB+Sd1O4qBc2a9aPvm0PJKscBU2P/LjO2SmlpYZI/nG0uX FmFm8KPu9ove1+i3jaY0hRd4CieyRe55gDDQY3OvMiHgDl6FxhfRNIr4sWyL3+M= X-Gm-Gg: ASbGnctZYfhqWYPpX9s45855nxi5eY9brefw6mOxudWHs66Oqasz76KGdT3e8KibIId 8xslhcuSeEnSP3H2zGM2i5XDwpBfinkSEMJS14el/VKAYSieMROvkDTVam/OLV1kmzvWA52i+Be lTIUuVaHx2Wj/ETTpARZh18rW4WQGnzzsYS8+YNlHuiVsBNkxfkyPunfHmsC4mcpq9AyPNyGQt+ U1umMfuyjDdrtXgeC88WE467I+aUIIpRAMyIrHpakU3s7cZYRCIXIdxjUQ= X-Google-Smtp-Source: AGHT+IFPC3QIjfGmFueYpArbFjHsCE/nJs1cTRfjgQkZc7hdX+t3EH5eMgGO4h5vvPNqW5LyAgQfEA== X-Received: by 2002:a05:6808:1454:b0:3ea:45d1:de13 with SMTP id 5614622812f47-3eae499b2b5mr1606169b6e.11.1733239968341; Tue, 03 Dec 2024 07:32:48 -0800 (PST) Received: from localhost.localdomain ([130.250.255.163]) by smtp.gmail.com with ESMTPSA id 5614622812f47-3ea86036cbbsm2891878b6e.8.2024.12.03.07.32.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 03 Dec 2024 07:32:47 -0800 (PST) From: Jens Axboe To: linux-mm@kvack.org, linux-fsdevel@vger.kernel.org Cc: hannes@cmpxchg.org, clm@meta.com, linux-kernel@vger.kernel.org, willy@infradead.org, kirill@shutemov.name, bfoster@redhat.com, Jens Axboe Subject: [PATCH 02/12] mm/readahead: add folio allocation helper Date: Tue, 3 Dec 2024 08:31:38 -0700 Message-ID: <20241203153232.92224-4-axboe@kernel.dk> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20241203153232.92224-2-axboe@kernel.dk> References: <20241203153232.92224-2-axboe@kernel.dk> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Just a wrapper around filemap_alloc_folio() for now, but add it in preparation for modifying the folio based on the 'ractl' being passed in. No functional changes in this patch. Signed-off-by: Jens Axboe Acked-by: "Darrick J. Wong" --- mm/readahead.c | 16 +++++++++++----- 1 file changed, 11 insertions(+), 5 deletions(-) diff --git a/mm/readahead.c b/mm/readahead.c index 8f1cf599b572..8424bf1b67e2 100644 --- a/mm/readahead.c +++ b/mm/readahead.c @@ -188,6 +188,12 @@ static void read_pages(struct readahead_control *rac) BUG_ON(readahead_count(rac)); } =20 +static struct folio *ractl_alloc_folio(struct readahead_control *ractl, + gfp_t gfp_mask, unsigned int order) +{ + return filemap_alloc_folio(gfp_mask, order); +} + /** * page_cache_ra_unbounded - Start unchecked readahead. * @ractl: Readahead control. @@ -265,8 +271,8 @@ void page_cache_ra_unbounded(struct readahead_control *= ractl, continue; } =20 - folio =3D filemap_alloc_folio(gfp_mask, - mapping_min_folio_order(mapping)); + folio =3D ractl_alloc_folio(ractl, gfp_mask, + mapping_min_folio_order(mapping)); if (!folio) break; =20 @@ -436,7 +442,7 @@ static inline int ra_alloc_folio(struct readahead_contr= ol *ractl, pgoff_t index, pgoff_t mark, unsigned int order, gfp_t gfp) { int err; - struct folio *folio =3D filemap_alloc_folio(gfp, order); + struct folio *folio =3D ractl_alloc_folio(ractl, gfp, order); =20 if (!folio) return -ENOMEM; @@ -751,7 +757,7 @@ void readahead_expand(struct readahead_control *ractl, if (folio && !xa_is_value(folio)) return; /* Folio apparently present */ =20 - folio =3D filemap_alloc_folio(gfp_mask, min_order); + folio =3D ractl_alloc_folio(ractl, gfp_mask, min_order); if (!folio) return; =20 @@ -780,7 +786,7 @@ void readahead_expand(struct readahead_control *ractl, if (folio && !xa_is_value(folio)) return; /* Folio apparently present */ =20 - folio =3D filemap_alloc_folio(gfp_mask, min_order); + folio =3D ractl_alloc_folio(ractl, gfp_mask, min_order); if (!folio) return; =20 --=20 2.45.2 From nobody Fri Dec 19 04:52:36 2025 Received: from mail-oi1-f174.google.com (mail-oi1-f174.google.com [209.85.167.174]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4F39F1F76C7 for ; Tue, 3 Dec 2024 15:32:50 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.167.174 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1733239972; cv=none; b=OAUuWw1lS74C8Np5aH8DAir+IKiRU2dzZVG7oeNTiZ6/d6nW3fNX6+jd0aP9fpgIB+GQqu8GTTBwck5ae0gzIQUAbnhq30xlE2/15Ef4tf33Zx5uDixnh/Z4EJE+kI1XrzFqj0t7D3zc7bkONH/FIGBzvKEMHz325aq0zRkAtz0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1733239972; c=relaxed/simple; bh=q5+uRl+k/fn3TorBhDOZ+b9X6VZ/V2geJNnascX9Tuw=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=QdcYAS80V3EluPddSj4dL2DczDCP3UxerEN+NzOCcSrzA/Kx+HpYfjSxZejW4sM5QV4fiRckSNA000TCiOl/rnYDGDK4gUVuDkZI9K8iYbjU5smnCIp1ua0xq8CUihwPg/fPgytzT3H7Ri04b4n61yKSnZSGBtiwazAbXhNtOcM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=VcxAfTER; arc=none smtp.client-ip=209.85.167.174 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="VcxAfTER" Received: by mail-oi1-f174.google.com with SMTP id 5614622812f47-3ea467fda32so2951972b6e.1 for ; Tue, 03 Dec 2024 07:32:50 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1733239970; x=1733844770; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=dSQf4jIfLBAkAFBsDD5CVUxWLID5QVKIZqR96ciKoMc=; b=VcxAfTERfrNCIjV4y56ah0/NIhjSnwZk+jEWqkNNz9Hga7fwuUrtk8VHECMIOKYbVL lP+1zgU68BxKo9y3CM0lkavVEVfY0mNlIKEMhg+OsemQxjokI3iXWRiamfutr66xSjgh MCQ1iJ+fWAqoKWCPNFG3jGqEgU3VLkSPeGFEjbRb2ATyMOrCZpztKXDftsKZiRzzVbDB 4PygPEUE0cYiNgP2yH/zNZI/JJi2VmviFY+rgGzyQaWZfbs1u3ukrWOdYkaASV6u0HZP JBAKdEucpfgW8VqPv2uAv8mKGuD7KR5bxdyoqP3S3xEnZMUHgCpLs5Lw5qE8R7UI41Rf zYGQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1733239970; x=1733844770; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=dSQf4jIfLBAkAFBsDD5CVUxWLID5QVKIZqR96ciKoMc=; b=deDidLTSxFa9tbfN1BP6SjS6otM3OCARDygV3HIZEiToVQcKBBpg+Jog9KsIgWTP49 A2lgmHa6rTNDmg1MC8CN7oZ7EsNVN/z0ClbreUG6Iy0swUcXo0WCHBLbXjKee/kMC+wg g3RrDBX9jpAL23y2y51hwu+Nr7CjUe9UnLhImQ1xG8/HaLXcq8XAsCeUpsvnXiUPTTO5 0XcHblbcIVRkeZbGJ+II2q2FFC5RUwPuknF9fzz1G+JZ/I2RnHTMBj/CC9PbdK3IWrqm knrqTlKnp6t4ETjaDE7/T7nYAee38hjW6TByqhIJ04QVaokVyJ/TsIs0IeU3Xy72MNke g7vg== X-Forwarded-Encrypted: i=1; AJvYcCXsuCASdNFh4zGwE2k9UF1IlHUDtC5ltsNWXrgt8Bl8njymcECeBoHd2e44cKy4TRl3sjRpPeRlhcyRIaA=@vger.kernel.org X-Gm-Message-State: AOJu0YxQpBODODZ9n+tsNBAZZrG/lpsQhS6es/h+u1KRn31k0Hxh7VgO Hu16WgXDoxsN9FfPOJQ1Dx4F+zUGCkNVB5yNpg3TnyeXEZyBFG611wER1PfhPho= X-Gm-Gg: ASbGnct2B7XyYARUzUM8HTqWIKLRguEzf0WCPUIe3NLC6LJ2B21Jv9yVxbYi+Z5IHlU kZe3idFKKIFyvwhsqSRalt2+bKwB3JDU8S6UhFF+BVikw+CSyyELFrdBkK9yfNlSigucY9Vx6Iq CNiNVUSLSSH7WZbm3bJS2OEPsmtI12GRQnsfarUESplkOpO7SNmxPHtvRTn/NlyBe3dVrdmcCaj JqgphwqBZ5ouX9X/OqhfdxunEgVDOdSRxpYI1pvO+wFLtPwLCDOHC6f3Dw= X-Google-Smtp-Source: AGHT+IF59+hbl1oxKnagTF4XdkSvu7+brogdJGlmHp3dV6C6q4drpkxZSCaanT7IxLge7RXlKXRl5g== X-Received: by 2002:a05:6808:1795:b0:3e7:9e07:432b with SMTP id 5614622812f47-3eae505e041mr2729493b6e.29.1733239969773; Tue, 03 Dec 2024 07:32:49 -0800 (PST) Received: from localhost.localdomain ([130.250.255.163]) by smtp.gmail.com with ESMTPSA id 5614622812f47-3ea86036cbbsm2891878b6e.8.2024.12.03.07.32.48 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 03 Dec 2024 07:32:49 -0800 (PST) From: Jens Axboe To: linux-mm@kvack.org, linux-fsdevel@vger.kernel.org Cc: hannes@cmpxchg.org, clm@meta.com, linux-kernel@vger.kernel.org, willy@infradead.org, kirill@shutemov.name, bfoster@redhat.com, Jens Axboe Subject: [PATCH 03/12] mm: add PG_uncached page flag Date: Tue, 3 Dec 2024 08:31:39 -0700 Message-ID: <20241203153232.92224-5-axboe@kernel.dk> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20241203153232.92224-2-axboe@kernel.dk> References: <20241203153232.92224-2-axboe@kernel.dk> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Add a page flag that file IO can use to indicate that the IO being done is uncached, as in it should not persist in the page cache after the IO has been completed. Signed-off-by: Jens Axboe Acked-by: "Darrick J. Wong" --- include/linux/page-flags.h | 5 +++++ include/trace/events/mmflags.h | 3 ++- 2 files changed, 7 insertions(+), 1 deletion(-) diff --git a/include/linux/page-flags.h b/include/linux/page-flags.h index 2220bfec278e..14346fa2470f 100644 --- a/include/linux/page-flags.h +++ b/include/linux/page-flags.h @@ -110,6 +110,7 @@ enum pageflags { PG_reclaim, /* To be reclaimed asap */ PG_swapbacked, /* Page is backed by RAM/swap */ PG_unevictable, /* Page is "unevictable" */ + PG_uncached, /* uncached read/write IO */ #ifdef CONFIG_MMU PG_mlocked, /* Page is vma mlocked */ #endif @@ -562,6 +563,10 @@ PAGEFLAG(Reclaim, reclaim, PF_NO_TAIL) FOLIO_FLAG(readahead, FOLIO_HEAD_PAGE) FOLIO_TEST_CLEAR_FLAG(readahead, FOLIO_HEAD_PAGE) =20 +FOLIO_FLAG(uncached, FOLIO_HEAD_PAGE) + FOLIO_TEST_CLEAR_FLAG(uncached, FOLIO_HEAD_PAGE) + __FOLIO_SET_FLAG(uncached, FOLIO_HEAD_PAGE) + #ifdef CONFIG_HIGHMEM /* * Must use a macro here due to header dependency issues. page_zone() is n= ot diff --git a/include/trace/events/mmflags.h b/include/trace/events/mmflags.h index bb8a59c6caa2..b60057284102 100644 --- a/include/trace/events/mmflags.h +++ b/include/trace/events/mmflags.h @@ -116,7 +116,8 @@ DEF_PAGEFLAG_NAME(head), \ DEF_PAGEFLAG_NAME(reclaim), \ DEF_PAGEFLAG_NAME(swapbacked), \ - DEF_PAGEFLAG_NAME(unevictable) \ + DEF_PAGEFLAG_NAME(unevictable), \ + DEF_PAGEFLAG_NAME(uncached) \ IF_HAVE_PG_MLOCK(mlocked) \ IF_HAVE_PG_HWPOISON(hwpoison) \ IF_HAVE_PG_IDLE(idle) \ --=20 2.45.2 From nobody Fri Dec 19 04:52:36 2025 Received: from mail-oi1-f178.google.com (mail-oi1-f178.google.com [209.85.167.178]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1F61A1FC0E0 for ; Tue, 3 Dec 2024 15:32:51 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.167.178 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1733239973; cv=none; b=bhyfUbwZ3dMVLnDRJp1lCu20aBsAeqHcayy4fSqTxQ5uRfPDgRiXbovmPToYYPorXZGsjlGqBx341ngl9Ab6G1YTKswbtrWD0zEszec9LJJblKdeZkF3J8v4HsSGvlIH7Ds6ZlKhIQfAyEhlfGXV4+wb22s/sADvx8gzSNfz5og= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1733239973; c=relaxed/simple; bh=ZgRCLG7Q7PSJRuEf4S9CorJsUog68v6zrN1kia1bNYk=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=miuAwY8KODoywbwaVM4P/C62NkSZ+XgBKJh6BKtecDMhVPkxlBsz0DWxP2MVHxWVpfx/EqdDJl94cGCqi5meMzD2FHdJxpwMnfpNxMf8BrRf7A76bi603qPx8rnfy2wYTtd6i4xx7QEdDEXS9jHUtieNu1FL07+nUtdHMjUwFoA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=HlbZFfOb; arc=none smtp.client-ip=209.85.167.178 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="HlbZFfOb" Received: by mail-oi1-f178.google.com with SMTP id 5614622812f47-3ea55c356caso3103891b6e.0 for ; Tue, 03 Dec 2024 07:32:51 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1733239971; x=1733844771; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=RwN531wgOLFM8EGeUWgrWwcMRYwBcN09BWbzS6SO3q8=; b=HlbZFfObU5xPz5v+qTCn8lR2qgozqdYGlOM6PCCDOyas9I5SUWycbDVa/ntlFyBsP7 Tacye40HugiN1g7Ivs1sfvOw6hcUV3WWcMJIa4YzOCvQSZOgiYlXKyqJMkv2bJ2jx6Mp rFeFWrfv8hwWjtYJDXSIazfJvDE3BP9wj7ZW08hxgjnTWTueZ8F3F7q2OURNwyjlcId6 +kyiwEtLqI7qmrkhimyRvSTNrFnjKXMkBKJMyFWkvPnoZZO2uMtWhL+VlgUSCNaNLTib N1p3geFG+eZWKu4UwIl2bfFNsZC8pmu4MC6DYRvW1kvVQZ5my8P3Xc64YrxHhdW/27Qe mUxQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1733239971; x=1733844771; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=RwN531wgOLFM8EGeUWgrWwcMRYwBcN09BWbzS6SO3q8=; b=XnKOJ221IbiQhfhFnGW3xC0gSDPrsW2ARYxZVioBB7LC5xo63g5mI7c9iHamdNEAbb 5oUex5YY+NPJNiQlfvm1xYyh//idIyz7DX7+OmUyj2WBwRSvFvYfmVEj2V4lSxilph8H XLcRCMu+CDb2/XeYdw8JZ0qt+zxmAfp5ST1cj4qzx8/atzSNFZo0n1l8y1+1mohfnoPV xoVlkX/88fYs6SWBUD4FvniulPMCocspq+QzfOOaBUEqysdOtr3U6yBY2BVSsRo9LyzL 2/tywHaaQmaqm5MVQjD9HYuIJQD9q9CTOjuiw3W8xaXIfi9Mu7zGQyJ1OFDoFntDRdqB umqA== X-Forwarded-Encrypted: i=1; AJvYcCWBfc0rKShosA5AcBHgt2GsQcr7DGh3aISsKaUCHBBXldEl65Y92O0lyVtlUWzR1v0LcT1XdGCsHScCvKo=@vger.kernel.org X-Gm-Message-State: AOJu0Yzx64fIZrf+ljkIWJTISlMfTTiV8czvGNvuIucW897S9FusfhIS YTGWqCEwFc2mgMg9elsasPQ2IdyRDrGzsFmsbow+t3kYRVfMkJ4PZ5A17eZ5ig8= X-Gm-Gg: ASbGncurv1eLZf2/GSuh7ziUs1ZRKrvIbrq/Kl2nTXHyzVcvzqfjDpF6tXmuiNIlCc/ VWCwdBoJ5MIQuvSGwhbnwJbMvyTVgGnoeVnnbdMBXuE9JF0dLHnxqVpRB7ywhfM93PawRtw0Rw3 dVnGmixCDL2VQZmK3U8K55A/sAETm/RKA4oJXZM2gheANk4cx52tn7a3cqkW8N4mTWSDm7HDANk K0Lbv7ZW98GSUN+se/fEN2WSLt6u/TajHQGfpZT43f35Wp9oJSl0GBNjmE= X-Google-Smtp-Source: AGHT+IF9UmPrglgGWAW3JFFN5BU2DOVhhB7roaz8QIV4CUJFyvDEFcQOUcjiZ4xJCcSYYgdgH9P94g== X-Received: by 2002:a05:6808:1405:b0:3e0:c13:9837 with SMTP id 5614622812f47-3eaf041efccmr453615b6e.37.1733239971292; Tue, 03 Dec 2024 07:32:51 -0800 (PST) Received: from localhost.localdomain ([130.250.255.163]) by smtp.gmail.com with ESMTPSA id 5614622812f47-3ea86036cbbsm2891878b6e.8.2024.12.03.07.32.49 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 03 Dec 2024 07:32:50 -0800 (PST) From: Jens Axboe To: linux-mm@kvack.org, linux-fsdevel@vger.kernel.org Cc: hannes@cmpxchg.org, clm@meta.com, linux-kernel@vger.kernel.org, willy@infradead.org, kirill@shutemov.name, bfoster@redhat.com, Jens Axboe Subject: [PATCH 04/12] mm/readahead: add readahead_control->uncached member Date: Tue, 3 Dec 2024 08:31:40 -0700 Message-ID: <20241203153232.92224-6-axboe@kernel.dk> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20241203153232.92224-2-axboe@kernel.dk> References: <20241203153232.92224-2-axboe@kernel.dk> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" If ractl->uncached is set to true, then folios created are marked as uncached as well. Signed-off-by: Jens Axboe Acked-by: "Darrick J. Wong" --- include/linux/pagemap.h | 1 + mm/readahead.c | 8 +++++++- 2 files changed, 8 insertions(+), 1 deletion(-) diff --git a/include/linux/pagemap.h b/include/linux/pagemap.h index bcf0865a38ae..72b03b37c265 100644 --- a/include/linux/pagemap.h +++ b/include/linux/pagemap.h @@ -1353,6 +1353,7 @@ struct readahead_control { pgoff_t _index; unsigned int _nr_pages; unsigned int _batch_count; + bool uncached; bool _workingset; unsigned long _pflags; }; diff --git a/mm/readahead.c b/mm/readahead.c index 8424bf1b67e2..33a2d0feae14 100644 --- a/mm/readahead.c +++ b/mm/readahead.c @@ -191,7 +191,13 @@ static void read_pages(struct readahead_control *rac) static struct folio *ractl_alloc_folio(struct readahead_control *ractl, gfp_t gfp_mask, unsigned int order) { - return filemap_alloc_folio(gfp_mask, order); + struct folio *folio; + + folio =3D filemap_alloc_folio(gfp_mask, order); + if (folio && ractl->uncached) + __folio_set_uncached(folio); + + return folio; } =20 /** --=20 2.45.2 From nobody Fri Dec 19 04:52:36 2025 Received: from mail-oi1-f174.google.com (mail-oi1-f174.google.com [209.85.167.174]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C41AF1F76DE for ; Tue, 3 Dec 2024 15:32:53 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.167.174 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1733239975; cv=none; b=quLIXFpZYgy0Wyme+/c2iwoaXvtSRMCFnI1cqTPR44rYMY10XIiu1XKdRYCBt9sErz1ICLlAhuhO8sP2Nj8R7W2ydrux+Jy/OyMS4QE8kpOlp5l/DStCYHv+FB8+chr2vtqhz5uUegzPMhPfwS7Ic+pjH0RT2Q7/N67WQ2DWxpc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1733239975; c=relaxed/simple; bh=SQI+ulnw+CRtDMqT4Q5a3RM/yVgI6AvOV11bMjB5isk=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=eU48wsTepPVK00X+mFMTjARA33zBy6LsruyjEV7tBsYfLK3wlE7eCiWiR5plTx/gnpyuk4ur7y8z/1Lj8Z7DgR0oNlikkftFPO5sdS5Dk6WOC0OJQJr4Jw4XKzxxxTcPZOyFy7dE+IwMM0vipjoqMDEc5gSMnSbMaq8ZOIX3LU8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=RuMwze1b; arc=none smtp.client-ip=209.85.167.174 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="RuMwze1b" Received: by mail-oi1-f174.google.com with SMTP id 5614622812f47-3ea55a2a38bso3014443b6e.1 for ; Tue, 03 Dec 2024 07:32:53 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1733239973; x=1733844773; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=elv86DXnTmRzX7BzNNN54cJv91/ZhMhRandVuyh+78M=; b=RuMwze1bubbkloeBcqqhIIKdhgXrelfY/tzbUVkN72jNmqL1Pu1EcLkjbX/221s3Ec gC2rfqKx9VyjuhWcbokwU5l6TMtPQ7D1KDHv31FBP0RgjchCEVvWU4qS9CUYNHcaScAm iAl6bJwhqxTCuPouyfBG9Yw44cwPJhpJjgEZ282vfGnD2YOlL83R1ZlgCiU2f+AhwtcY ViAk7w1NRl8Q+6m/mMvDaeDm5OGC8TyhI78JxVnjwC/4fwBcfY/FJNdBKtQONWx5YzTb t6mS9rkO5MPiEZghVzftLG2OUnEoZ78TmE80CO/jUbE8N/RArWNWGqmfE85hBajtwZjY W0DQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1733239973; x=1733844773; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=elv86DXnTmRzX7BzNNN54cJv91/ZhMhRandVuyh+78M=; b=uNYVfi7LvSyVlDgT1ynUYLt85IIlPA5SFk7HpMZBnvM1ufU+UJxws5jmJngohGTRK+ +xGZZFX0xGXlE8q1slQ/zO1Vrxyn2vKpyvo7eqHMJ78kwyfPqNAUuWUdNQBEQdTn3x/L YNEFb3+4EDXLpfTnm15agJ1mKbflB4WlgaXnhmVkCz8kdSGtp9dubYtPpowoXkzA3lgH nIEKyfFBf6KrbzPrVExOWj8lnL/vquzJbNHkLiP6clVTnCT+LjZ0VLgQQwrp/6BCsO7l DrltlPjfN1z6XdVMVWMu5SwU9n7c/FEi3uNFZjvpnQapmPxFkMVUagWCWcjfwNuPWjEN e11Q== X-Forwarded-Encrypted: i=1; AJvYcCUgnTn+CTsFBsyobiJ8oXXp/xGwUL3Qf6JIcr5V0/INM8Vo+jphWxBcbpEjlP67yaQO4Zzy+Bpwlzm+9GE=@vger.kernel.org X-Gm-Message-State: AOJu0YzR/D5DzUpZMK9kzlXcPrr5uldyktDIc5q9V4t87pBwz4q4FTG+ xg+fVpurTO80T00StVeydyKeHKDofqaaSluNzGnthDdXkyrCMuDkk6M+64uk+Zs= X-Gm-Gg: ASbGncvdCMlbsLi5uvwVOXJrhtQCabfTMV5mB1DBsT+QoRGd2vq8DuDUbeuJg6tEtTh tC7p0qO00rUa98MfXRXKLQSoAWvFfwZAhoINRo4kGtuK5IC41vHeUPgaQKe8W0QsY0v2JOgu65B BmAC2EG1q6yaW3dtb1nCyk7/dENm/Ahpv4N/8J78WF0YCOMbi3NEbPfAPUdZPAcSyJ9zdjfyT9D 3km7SMOMeLSj5ZS8JhpFLHTRjDqAwqpLLClA+vvrnOhAZt51dO9cZAJ2bE= X-Google-Smtp-Source: AGHT+IGEqY5mPPeclzVfJP1sqwpK2NEo6ZfmI2zg3SEWdjZlSMpJ+brY6rhQdeJiLlo5jFiEsUF9Rw== X-Received: by 2002:a05:6808:1795:b0:3ea:4c23:daf5 with SMTP id 5614622812f47-3eae4ecb2camr3172373b6e.8.1733239972824; Tue, 03 Dec 2024 07:32:52 -0800 (PST) Received: from localhost.localdomain ([130.250.255.163]) by smtp.gmail.com with ESMTPSA id 5614622812f47-3ea86036cbbsm2891878b6e.8.2024.12.03.07.32.51 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 03 Dec 2024 07:32:52 -0800 (PST) From: Jens Axboe To: linux-mm@kvack.org, linux-fsdevel@vger.kernel.org Cc: hannes@cmpxchg.org, clm@meta.com, linux-kernel@vger.kernel.org, willy@infradead.org, kirill@shutemov.name, bfoster@redhat.com, Jens Axboe Subject: [PATCH 05/12] mm/filemap: use page_cache_sync_ra() to kick off read-ahead Date: Tue, 3 Dec 2024 08:31:41 -0700 Message-ID: <20241203153232.92224-7-axboe@kernel.dk> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20241203153232.92224-2-axboe@kernel.dk> References: <20241203153232.92224-2-axboe@kernel.dk> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Rather than use the page_cache_sync_readahead() helper, define our own ractl and use page_cache_sync_ra() directly. In preparation for needing to modify ractl inside filemap_get_pages(). No functional changes in this patch. Signed-off-by: Jens Axboe Acked-by: "Darrick J. Wong" Reviewed-by: Christoph Hellwig --- mm/filemap.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/mm/filemap.c b/mm/filemap.c index 898e992039e8..dd3042de8038 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -2527,7 +2527,6 @@ static int filemap_get_pages(struct kiocb *iocb, size= _t count, { struct file *filp =3D iocb->ki_filp; struct address_space *mapping =3D filp->f_mapping; - struct file_ra_state *ra =3D &filp->f_ra; pgoff_t index =3D iocb->ki_pos >> PAGE_SHIFT; pgoff_t last_index; struct folio *folio; @@ -2542,12 +2541,13 @@ static int filemap_get_pages(struct kiocb *iocb, si= ze_t count, =20 filemap_get_read_batch(mapping, index, last_index - 1, fbatch); if (!folio_batch_count(fbatch)) { + DEFINE_READAHEAD(ractl, filp, &filp->f_ra, mapping, index); + if (iocb->ki_flags & IOCB_NOIO) return -EAGAIN; if (iocb->ki_flags & IOCB_NOWAIT) flags =3D memalloc_noio_save(); - page_cache_sync_readahead(mapping, ra, filp, index, - last_index - index); + page_cache_sync_ra(&ractl, last_index - index); if (iocb->ki_flags & IOCB_NOWAIT) memalloc_noio_restore(flags); filemap_get_read_batch(mapping, index, last_index - 1, fbatch); --=20 2.45.2 From nobody Fri Dec 19 04:52:36 2025 Received: from mail-oi1-f182.google.com (mail-oi1-f182.google.com [209.85.167.182]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 2886D1FECCC for ; Tue, 3 Dec 2024 15:32:55 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.167.182 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1733239976; cv=none; b=AGO4ON6uDRW6gs0Wbiv3uzu9mRtbjm4ez7zavz3NuQZH2vD4cp7qkMntQg+1qD9H7Fsayrinq9XYErB4RgzSppRgNJgay8bARFwrmZBWsKpxuQ3WVE0cCDZRl/V4UpgYBdfLsieu+cO1o1tVYSmQEmagAesDYA47w4gP85wHj9M= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1733239976; c=relaxed/simple; bh=YX3XCyHjYzyVsTXZV7qYbvuii/Uh6uyZoNv1uBKQubM=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=AteHWEhSfzFO7hH00JXM1zSmN+5s7S5n0KEzDq+Bxix/p2q9+3NgPADmooBkzPL0dbokUCXkF4Aij1X/zTREwjJSCKV1SXkB/mJfntyzotb9kW225zCkDQrHi6mFzXJoNCKpb8JBr5nE9twBCQDQsgDJCZLqjhlVGC+C38c5Qag= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=0jbp1Iuu; arc=none smtp.client-ip=209.85.167.182 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="0jbp1Iuu" Received: by mail-oi1-f182.google.com with SMTP id 5614622812f47-3ea369fdb0cso3320121b6e.3 for ; Tue, 03 Dec 2024 07:32:55 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1733239974; x=1733844774; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=DRb+N8QQBu8dVr+Sad3apJL8CLp+dfOfMlp0JKFijps=; b=0jbp1Iuur9naulsin3kmi+SybTDNMm5eTWmCVsvE0jKeqIrtgcbOc1lJt0ws5oRJuq Sjv3tz5r650N9b6YtmMmXTU/yhqB1dKl7xEj6mVG/KngfILqIJIOFzd9YlgGmSkU+gyK TeUfX5VOGwkJBc9Ycn3m5ZfdCG3bRYGBofsi0clxloal7RcEbypUuH3TDPa0pGqGOFC8 V86km6rLFWOz0gKPgtUnNMOnEcYsJ2UdA6endKWApYrCMpd21bSCCux3aOaEdnldy/vc MJRbv7eFoXmkwI/YMFMwW5fMiy+E8DwSYOvkaGYfcOfUO69SCaIXXsO1QEulBG4c9lpk mT4A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1733239974; x=1733844774; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=DRb+N8QQBu8dVr+Sad3apJL8CLp+dfOfMlp0JKFijps=; b=ERMtDuBNMJy7xtkdtv2kW/7uWN1YHosP6+Aa3YqBJUaJmSSO0dtHOiHxTEVihKugLZ 9eNx/K65wB053QH+Vp2P7xmhH+9+WAfXrGpm+P38DowmPFw889PqyEa6PxrtjTab5CdK Ua1lAk/zGXZ/x8knalRA2JB1Gk7k66JgsovXUFbKH9NENctqyVMkVuMViwFmCHbuMVr6 MAeOX6Igpb388ytNCgNR/DLuYFtXz1pHVwClV/yOqypbRJRiR37ST4hjrY7V9DmtnE8Q FQCTFelxjBxTU//HLOF7CnS325xqTOhoBunGW6laJTm/XLRrVxxo6VsI0HQUx5o574r8 VDKg== X-Forwarded-Encrypted: i=1; AJvYcCUPgtpp/77aOGDukmSaTp+S1Waa762FmUXXqHI/WcUc9GAbrFafU7ncOJApHhcfquGah8/7tyzjnn+4uCE=@vger.kernel.org X-Gm-Message-State: AOJu0Yx5aW9LvpQcW3Tr8v/jSYn3ipIfHIbCIF55JHbovtmHYVSrFYWL DBsN39lj+c1+2Tqj1fUk+ItOyOaJiQjUsbDwEf3Wgn3RrfylWCo5m6a9kdHTcvk= X-Gm-Gg: ASbGnctpItTSwEFRbXrRxb1dd1DSJu9BdowjOTq9VdPhhaxzIBPQmW+00eTG+5y+Mwg omsyByUabTtHWZWPv1UMu/8Y6sEjiXVzR7xTYYhhrr1i7hRwIezfVEVlkG9cux+ehEUiR2Lwlll zp0K2jT3WwNlf4R82hQU88YXBvCMcJsbTmKBfo7AAHOMudTK5O4R9pGz9Zh3ABcfqa+rB6Je5cc sEYGuli47xDPT9zUXiXgyUnE8g/qQBZPqiVN5GQare9u6cCoEZpWx11xEk= X-Google-Smtp-Source: AGHT+IGHTQ5LeGksRhzqMID0Kqh5YSoxN8Gylam5Ojf1AQcflOZOTzp2gKozYR7Jx08uJ7hl5NcXoA== X-Received: by 2002:a05:6808:3843:b0:3e8:1f5d:b7f8 with SMTP id 5614622812f47-3eae4edadc4mr3375768b6e.1.1733239974385; Tue, 03 Dec 2024 07:32:54 -0800 (PST) Received: from localhost.localdomain ([130.250.255.163]) by smtp.gmail.com with ESMTPSA id 5614622812f47-3ea86036cbbsm2891878b6e.8.2024.12.03.07.32.53 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 03 Dec 2024 07:32:53 -0800 (PST) From: Jens Axboe To: linux-mm@kvack.org, linux-fsdevel@vger.kernel.org Cc: hannes@cmpxchg.org, clm@meta.com, linux-kernel@vger.kernel.org, willy@infradead.org, kirill@shutemov.name, bfoster@redhat.com, Jens Axboe Subject: [PATCH 06/12] mm/truncate: add folio_unmap_invalidate() helper Date: Tue, 3 Dec 2024 08:31:42 -0700 Message-ID: <20241203153232.92224-8-axboe@kernel.dk> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20241203153232.92224-2-axboe@kernel.dk> References: <20241203153232.92224-2-axboe@kernel.dk> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Add a folio_unmap_invalidate() helper, which unmaps and invalidates a given folio. The caller must already have locked the folio. Use this new helper in invalidate_inode_pages2_range(), rather than duplicate the code there. In preparation for using this elsewhere as well, have it take a gfp_t mask rather than assume GFP_KERNEL is the right choice. This bubbles back to invalidate_complete_folio2() as well. Signed-off-by: Jens Axboe Acked-by: "Darrick J. Wong" --- include/linux/pagemap.h | 2 ++ mm/truncate.c | 35 ++++++++++++++++++++++------------- 2 files changed, 24 insertions(+), 13 deletions(-) diff --git a/include/linux/pagemap.h b/include/linux/pagemap.h index 72b03b37c265..f2d49dccb7c1 100644 --- a/include/linux/pagemap.h +++ b/include/linux/pagemap.h @@ -34,6 +34,8 @@ int kiocb_invalidate_pages(struct kiocb *iocb, size_t cou= nt); void kiocb_invalidate_post_direct_write(struct kiocb *iocb, size_t count); int filemap_invalidate_pages(struct address_space *mapping, loff_t pos, loff_t end, bool nowait); +int folio_unmap_invalidate(struct address_space *mapping, struct folio *fo= lio, + gfp_t gfp); =20 int write_inode_now(struct inode *, int sync); int filemap_fdatawrite(struct address_space *); diff --git a/mm/truncate.c b/mm/truncate.c index 7c304d2f0052..c1dfddb1122a 100644 --- a/mm/truncate.c +++ b/mm/truncate.c @@ -533,12 +533,12 @@ EXPORT_SYMBOL(invalidate_mapping_pages); * sitting in the folio_add_lru() caches. */ static int invalidate_complete_folio2(struct address_space *mapping, - struct folio *folio) + struct folio *folio, gfp_t gfp_mask) { if (folio->mapping !=3D mapping) return 0; =20 - if (!filemap_release_folio(folio, GFP_KERNEL)) + if (!filemap_release_folio(folio, gfp_mask)) return 0; =20 spin_lock(&mapping->host->i_lock); @@ -570,6 +570,25 @@ static int folio_launder(struct address_space *mapping= , struct folio *folio) return mapping->a_ops->launder_folio(folio); } =20 +int folio_unmap_invalidate(struct address_space *mapping, struct folio *fo= lio, + gfp_t gfp) +{ + int ret; + + VM_BUG_ON_FOLIO(!folio_test_locked(folio), folio); + + if (folio_test_dirty(folio)) + return 0; + if (folio_mapped(folio)) + unmap_mapping_folio(folio); + BUG_ON(folio_mapped(folio)); + + ret =3D folio_launder(mapping, folio); + if (!ret && !invalidate_complete_folio2(mapping, folio, gfp)) + return -EBUSY; + return ret; +} + /** * invalidate_inode_pages2_range - remove range of pages from an address_s= pace * @mapping: the address_space @@ -629,18 +648,8 @@ int invalidate_inode_pages2_range(struct address_space= *mapping, folio_unlock(folio); continue; } - VM_BUG_ON_FOLIO(!folio_contains(folio, indices[i]), folio); folio_wait_writeback(folio); - - if (folio_mapped(folio)) - unmap_mapping_folio(folio); - BUG_ON(folio_mapped(folio)); - - ret2 =3D folio_launder(mapping, folio); - if (ret2 =3D=3D 0) { - if (!invalidate_complete_folio2(mapping, folio)) - ret2 =3D -EBUSY; - } + ret2 =3D folio_unmap_invalidate(mapping, folio, GFP_KERNEL); if (ret2 < 0) ret =3D ret2; folio_unlock(folio); --=20 2.45.2 From nobody Fri Dec 19 04:52:36 2025 Received: from mail-oi1-f170.google.com (mail-oi1-f170.google.com [209.85.167.170]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id BA3E5203718 for ; Tue, 3 Dec 2024 15:32:56 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.167.170 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1733239978; cv=none; b=W2dJKZ9iNltIuhJOKX6YMd2cWDBbkZI7+86DySIX5zR3JJrseCHXBueKsGXOLH8hyCLNJl+yY5C/WWC8BCTAE3LExtsbG1EXuAuNNADQis2maT773wWggLs80UF4C6CgPv1n1A/dxzsxSFqZepAJ5ur8rWdvzKqsgFEBsVjsIwY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1733239978; c=relaxed/simple; bh=mhIaz7b2lyC1oOrpy7fcg+5AljdReMhMByTLtm2p4EM=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=qP/CODif2lgBncwAKDF7IaDiTILzlu5ZUA5sHYxrUmq1rZvSUdMA2JDhSX1Hy/KPHunXWd3JjSWooLpTQ87gd6IHGot8d7v16XROuB9cFwozgUIKlQnqY8ECN6QgDSYZnqLz7HpVqgdUjPQqHsx81SyYSrJtjWX3++FW0WX0t2A= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=fe79W8Nh; arc=none smtp.client-ip=209.85.167.170 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="fe79W8Nh" Received: by mail-oi1-f170.google.com with SMTP id 5614622812f47-3ea55a2a38bso3014473b6e.1 for ; Tue, 03 Dec 2024 07:32:56 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1733239976; x=1733844776; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=99ZtQ6Npm+o/kkpmwDDPCgpJncZgiN/IMlRLftnmHuc=; b=fe79W8Nht1SNccoLINB3gun47a42Nyg7qCWlAyUHn7wod5WZJVvfj5epG7rdRBk3EL IHI6Fy0gmAc/hoo/Rbj2tnqec0JQnK3TI8eslyBHs0GimM6Ve/bXBH8t1EsM0F/r4nfY H4Z6eo24b7YiLr6tn1N0IfuVI7ti0ebkN27qDrvk17HP0T/yfIErOWX8/joQS+PO8D1K XKVccScUJYUVpHgO01Ep1qUIMQLv5tSbIZAIOTweoUPn82mbpQd8wMNBlhJx/ax0jvx7 +36yZ/NEp7WyKD9QrNsMJu9mb7b1Ypg4OQD0Xftl71coTjK/xpubRFXvsgbVSogeKAJY L2kw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1733239976; x=1733844776; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=99ZtQ6Npm+o/kkpmwDDPCgpJncZgiN/IMlRLftnmHuc=; b=s+mM3D5aT+7g10EKCDUU2u+CD9sYb9Y8ju4fxwDQTyfygDc5X9F5WXnytuKSNfERP0 ahyPhDO1udoBibyJWfNtCI4A5tHoEHKgzK9nbat5Ee4tyAyD2tdyYbaSQIehp6WBkWT+ /03LyPAaOftbxeYMRFgZ3jPlHRFRC3vnQGcrb1s8972JnVRVxqVvn4tTwQLSXFz3mV5q UHLpk/qzIaEFOsbbMu9n8u1nZxlELi7luaxpdj2xjL8JpzGE3qma5oZ5bBR3zZwrjjdW xyRWensdbRJVcZ1U1ltqM4+tyalHt7yJds5ZYT0hUBLs1JDLXrLqemG9evv7yjhZftSX QiGg== X-Forwarded-Encrypted: i=1; AJvYcCXMkXW7H7nvRSNBuUorZ72CgBvqQ5OE27QKCXGARfSjI4n6gB4bujxOTwkmoom+ichLI0jAl6esLuejBvI=@vger.kernel.org X-Gm-Message-State: AOJu0YzkX2zVfbDXHjr9MjcheUu/rljhYkdhdhi2oy8nCj4MZpLRzVOb 3e1zHc5KYEEgelDxf6EwD1PAxb0+9sj1aaDgy+70VrmcCrRRS8VQfl4H5ooqwJs= X-Gm-Gg: ASbGncsW4+5EThuPcir51xgQHHGgNUw7hB8YY61ywxLAsbXV3TtT9h6GNL+GRhio01v pKepS3Qe7FueanGWR96n7KWL6p+yl9KjLkCq2QDUgfFG2XlEeV3lMXOWIkEDOmv+B/skavSrAwR hVJWS8XHCSK2zz34nPkd8Mk7pCU3vYnC18uNKeyQ8LH8E6CZbGp8B8CIg4VHQLcFrdaZ7UrEWDU 66+PUC5EBwGnvYmFEXnGFY7g0BYD/3QAVtZLu6m8Zg2kF3Gg9q/lHI4UAw= X-Google-Smtp-Source: AGHT+IHMJnWLWXZesvRG8H115i2+SnEyGT/XbMmAgLxKcYoNaa8g7OGxqlE3KOV7Yd59LM9TrdKc0w== X-Received: by 2002:a05:6808:152c:b0:3e6:14a6:4288 with SMTP id 5614622812f47-3eae4f366f8mr2926476b6e.11.1733239975822; Tue, 03 Dec 2024 07:32:55 -0800 (PST) Received: from localhost.localdomain ([130.250.255.163]) by smtp.gmail.com with ESMTPSA id 5614622812f47-3ea86036cbbsm2891878b6e.8.2024.12.03.07.32.54 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 03 Dec 2024 07:32:55 -0800 (PST) From: Jens Axboe To: linux-mm@kvack.org, linux-fsdevel@vger.kernel.org Cc: hannes@cmpxchg.org, clm@meta.com, linux-kernel@vger.kernel.org, willy@infradead.org, kirill@shutemov.name, bfoster@redhat.com, Jens Axboe Subject: [PATCH 07/12] fs: add RWF_UNCACHED iocb and FOP_UNCACHED file_operations flag Date: Tue, 3 Dec 2024 08:31:43 -0700 Message-ID: <20241203153232.92224-9-axboe@kernel.dk> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20241203153232.92224-2-axboe@kernel.dk> References: <20241203153232.92224-2-axboe@kernel.dk> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" If a file system supports uncached buffered IO, it may set FOP_UNCACHED and enable RWF_UNCACHED. If RWF_UNCACHED is attempted without the file system supporting it, it'll get errored with -EOPNOTSUPP. Signed-off-by: Jens Axboe Acked-by: "Darrick J. Wong" --- include/linux/fs.h | 14 +++++++++++++- include/uapi/linux/fs.h | 6 +++++- 2 files changed, 18 insertions(+), 2 deletions(-) diff --git a/include/linux/fs.h b/include/linux/fs.h index 7e29433c5ecc..b64a78582f06 100644 --- a/include/linux/fs.h +++ b/include/linux/fs.h @@ -322,6 +322,7 @@ struct readahead_control; #define IOCB_NOWAIT (__force int) RWF_NOWAIT #define IOCB_APPEND (__force int) RWF_APPEND #define IOCB_ATOMIC (__force int) RWF_ATOMIC +#define IOCB_UNCACHED (__force int) RWF_UNCACHED =20 /* non-RWF related bits - start at 16 */ #define IOCB_EVENTFD (1 << 16) @@ -356,7 +357,8 @@ struct readahead_control; { IOCB_SYNC, "SYNC" }, \ { IOCB_NOWAIT, "NOWAIT" }, \ { IOCB_APPEND, "APPEND" }, \ - { IOCB_ATOMIC, "ATOMIC"}, \ + { IOCB_ATOMIC, "ATOMIC" }, \ + { IOCB_UNCACHED, "UNCACHED" }, \ { IOCB_EVENTFD, "EVENTFD"}, \ { IOCB_DIRECT, "DIRECT" }, \ { IOCB_WRITE, "WRITE" }, \ @@ -2127,6 +2129,8 @@ struct file_operations { #define FOP_UNSIGNED_OFFSET ((__force fop_flags_t)(1 << 5)) /* Supports asynchronous lock callbacks */ #define FOP_ASYNC_LOCK ((__force fop_flags_t)(1 << 6)) +/* File system supports uncached read/write buffered IO */ +#define FOP_UNCACHED ((__force fop_flags_t)(1 << 7)) =20 /* Wrap a directory iterator that needs exclusive inode access */ int wrap_directory_iterator(struct file *, struct dir_context *, @@ -3614,6 +3618,14 @@ static inline int kiocb_set_rw_flags(struct kiocb *k= i, rwf_t flags, if (!(ki->ki_filp->f_mode & FMODE_CAN_ATOMIC_WRITE)) return -EOPNOTSUPP; } + if (flags & RWF_UNCACHED) { + /* file system must support it */ + if (!(ki->ki_filp->f_op->fop_flags & FOP_UNCACHED)) + return -EOPNOTSUPP; + /* DAX mappings not supported */ + if (IS_DAX(ki->ki_filp->f_mapping->host)) + return -EOPNOTSUPP; + } kiocb_flags |=3D (__force int) (flags & RWF_SUPPORTED); if (flags & RWF_SYNC) kiocb_flags |=3D IOCB_DSYNC; diff --git a/include/uapi/linux/fs.h b/include/uapi/linux/fs.h index 753971770733..dc77cd8ae1a3 100644 --- a/include/uapi/linux/fs.h +++ b/include/uapi/linux/fs.h @@ -332,9 +332,13 @@ typedef int __bitwise __kernel_rwf_t; /* Atomic Write */ #define RWF_ATOMIC ((__force __kernel_rwf_t)0x00000040) =20 +/* buffered IO that drops the cache after reading or writing data */ +#define RWF_UNCACHED ((__force __kernel_rwf_t)0x00000080) + /* mask of flags supported by the kernel */ #define RWF_SUPPORTED (RWF_HIPRI | RWF_DSYNC | RWF_SYNC | RWF_NOWAIT |\ - RWF_APPEND | RWF_NOAPPEND | RWF_ATOMIC) + RWF_APPEND | RWF_NOAPPEND | RWF_ATOMIC |\ + RWF_UNCACHED) =20 #define PROCFS_IOCTL_MAGIC 'f' =20 --=20 2.45.2 From nobody Fri Dec 19 04:52:36 2025 Received: from mail-oi1-f171.google.com (mail-oi1-f171.google.com [209.85.167.171]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 19E0F204F84 for ; Tue, 3 Dec 2024 15:32:57 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.167.171 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1733239979; cv=none; b=K0Mp7AkqkjtgrZnafwse8Rq7gPgvQdBTgEKLmqkG4AEWK7OtPy/p6ZAnP9CrdpCQkBa3j4GLeFwG0vJA6i1Mm5lztdSSZhWmKuQRL08V+x7M6QfXAQwoVPVpYrT4sXSLsRadnLVCQ/CDaE0MrQsisjwW0Mud9qSYtk4+EMpCLc4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1733239979; c=relaxed/simple; bh=LUYFIqzYiNC8iDjcYjzRM/NzCPA2BcvlCTGVyApeMyI=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=o08FJmCBs/4IfILxipcOgenysBz9Gh0DhKTOMNHPKx7HEjYHgyuFNaPEkSSQtSYz5LlK5stX2J8fmRSgu0lV5GP+F1GUz0+896s/cD4PV5IyeNxFcxouFcpCq5PI/z9jTzDoPjF36DBVqhMVMqc1mNX+0aLs7gnqq2Mdh94df7M= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=U5y+hE8A; arc=none smtp.client-ip=209.85.167.171 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="U5y+hE8A" Received: by mail-oi1-f171.google.com with SMTP id 5614622812f47-3ea68fc1a7cso2700767b6e.0 for ; Tue, 03 Dec 2024 07:32:57 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1733239977; x=1733844777; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=854g8zvMX205jFWe5CwR3kRcBo1TfCPiGvzDJnv2dMk=; b=U5y+hE8AlQ5RNTn1wvZKs/tYic8sjv4/RNNQaa6i4OIGS/hkyTlJsqcHVHm8gQKasG uQc1yxKexzBJUXxKM6+qwa7sl2c2c/eqF2yvp1x0Nfuz9seX1h1j4qsTluEBwxqdisVl S7KI2OpHt0VItCdaP1cnGq0lhnG2DxcXu9P8ClotpHeJcN5O4o9JUN9vgG6/hI9BDfza 2h+ylaZur13T8T8Jd+2SngaZkG0PWAYuAS7me3Xq4IomLG+cOlx1rM52I5pW/3AuauG0 bMfvd2DtFVfmc9lJazpc2CV9UeS2n3WzBba1I7DCHT9wNrfT9XBCYQZAZkRu51JmZSHh h9fg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1733239977; x=1733844777; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=854g8zvMX205jFWe5CwR3kRcBo1TfCPiGvzDJnv2dMk=; b=oYZ0L6xLoXBnVk2RiFZQOvq6rl+Vkmh4lWEhyGXgiF306xCQg5R4zWJARAJTLgDa/9 Ef2/YA3sIKKbc35AO+Q0fe2ws7+ToDA4ry5785t2Zn4isxW3gXdHzTue3gW1ecZf0FA4 M1WI0SfvMkgka2CzNgaZSgFdR9dg2mEbbfdVAN0Vkqns7mm4TE+xQk3/fxRqXLSdcFEG 0Acw/B9aDQFwC5Hxnf6i4lErts5awKoJ7webwAMUKTsECa2mbvdVDGqT3Y2B7siTKkwO 9T/bSbqx5bD9eoZP+YKvS6Kslbei4qUVvRzHX0MA20T0tp8TBn8GFr2LywBBTVjZOyhn D2oA== X-Forwarded-Encrypted: i=1; AJvYcCUC4VrLMSyt92OMWpORtSpw5lpGA9LDTiiln2OtikwGzK/p5M1xUOhndApADJ9cTi6vtuLIpED/u01GUiM=@vger.kernel.org X-Gm-Message-State: AOJu0Yz87jeFozaEjTGXIbuC4LMTYWKXfdTdqdEVit5RB+5bIu8vh5fb TR5fvXF1oJaW/qb3aJj8eEDe8LuhucbWF6LS1R2/bR4cdi6cC0Qcrw/on2aVQvM= X-Gm-Gg: ASbGnctqgibXzxoPdCI88luEkx0HYDxZPmr1RTnpbD+rsPHomoIS37MZlFqQBaJPuef xOzJJVa6ta2yUDfnhWuK/j1BohBNFsZLCaVWBwtsz4NJ5X0V6C6JsMZ7sOSTVol30y6UX2Aph07 IaHhNegYdhCeAJma2Gc44c78yTmWncRQAmUfVcdDscPTwEY+eZ+2HYD3iXiyItDnGPVjLQPncoh Nq3MQFgSr5j08WYn4/L5jSutUffnDlT8XWZjUclfkcmUtUBbqI6sYovQKo= X-Google-Smtp-Source: AGHT+IEn2a1k9EtjOpfVet1ysm6MWr9jkiUyE2X2SH8cUoAjtJnT3D5sVQSXdc6UVYtxwSGjKa8I4g== X-Received: by 2002:a05:6808:2e4e:b0:3ea:aa8a:c115 with SMTP id 5614622812f47-3eae4f8c2c0mr3705224b6e.21.1733239977325; Tue, 03 Dec 2024 07:32:57 -0800 (PST) Received: from localhost.localdomain ([130.250.255.163]) by smtp.gmail.com with ESMTPSA id 5614622812f47-3ea86036cbbsm2891878b6e.8.2024.12.03.07.32.56 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 03 Dec 2024 07:32:56 -0800 (PST) From: Jens Axboe To: linux-mm@kvack.org, linux-fsdevel@vger.kernel.org Cc: hannes@cmpxchg.org, clm@meta.com, linux-kernel@vger.kernel.org, willy@infradead.org, kirill@shutemov.name, bfoster@redhat.com, Jens Axboe Subject: [PATCH 08/12] mm/filemap: add read support for RWF_UNCACHED Date: Tue, 3 Dec 2024 08:31:44 -0700 Message-ID: <20241203153232.92224-10-axboe@kernel.dk> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20241203153232.92224-2-axboe@kernel.dk> References: <20241203153232.92224-2-axboe@kernel.dk> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Add RWF_UNCACHED as a read operation flag, which means that any data read wil be removed from the page cache upon completion. Uses the page cache to synchronize, and simply prunes folios that were instantiated when the operation completes. While it would be possible to use private pages for this, using the page cache as synchronization is handy for a variety of reasons: 1) No special truncate magic is needed 2) Async buffered reads need some place to serialize, using the page cache is a lot easier than writing extra code for this 3) The pruning cost is pretty reasonable and the code to support this is much simpler as a result. You can think of uncached buffered IO as being the much more attractive cousin of O_DIRECT - it has none of the restrictions of O_DIRECT. Yes, it will copy the data, but unlike regular buffered IO, it doesn't run into the unpredictability of the page cache in terms of reclaim. As an example, on a test box with 32 drives, reading them with buffered IO looks as follows: Reading bs 65536, uncached 0 1s: 145945MB/sec 2s: 158067MB/sec 3s: 157007MB/sec 4s: 148622MB/sec 5s: 118824MB/sec 6s: 70494MB/sec 7s: 41754MB/sec 8s: 90811MB/sec 9s: 92204MB/sec 10s: 95178MB/sec 11s: 95488MB/sec 12s: 95552MB/sec 13s: 96275MB/sec where it's quite easy to see where the page cache filled up, and performance went from good to erratic, and finally settles at a much lower rate. Looking at top while this is ongoing, we see: PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 7535 root 20 0 267004 0 0 S 3199 0.0 8:40.65 uncached 3326 root 20 0 0 0 0 R 100.0 0.0 0:16.40 kswapd4 3327 root 20 0 0 0 0 R 100.0 0.0 0:17.22 kswapd5 3328 root 20 0 0 0 0 R 100.0 0.0 0:13.29 kswapd6 3332 root 20 0 0 0 0 R 100.0 0.0 0:11.11 kswapd10 3339 root 20 0 0 0 0 R 100.0 0.0 0:16.25 kswapd17 3348 root 20 0 0 0 0 R 100.0 0.0 0:16.40 kswapd26 3343 root 20 0 0 0 0 R 100.0 0.0 0:16.30 kswapd21 3344 root 20 0 0 0 0 R 100.0 0.0 0:11.92 kswapd22 3349 root 20 0 0 0 0 R 100.0 0.0 0:16.28 kswapd27 3352 root 20 0 0 0 0 R 99.7 0.0 0:11.89 kswapd30 3353 root 20 0 0 0 0 R 96.7 0.0 0:16.04 kswapd31 3329 root 20 0 0 0 0 R 96.4 0.0 0:11.41 kswapd7 3345 root 20 0 0 0 0 R 96.4 0.0 0:13.40 kswapd23 3330 root 20 0 0 0 0 S 91.1 0.0 0:08.28 kswapd8 3350 root 20 0 0 0 0 S 86.8 0.0 0:11.13 kswapd28 3325 root 20 0 0 0 0 S 76.3 0.0 0:07.43 kswapd3 3341 root 20 0 0 0 0 S 74.7 0.0 0:08.85 kswapd19 3334 root 20 0 0 0 0 S 71.7 0.0 0:10.04 kswapd12 3351 root 20 0 0 0 0 R 60.5 0.0 0:09.59 kswapd29 3323 root 20 0 0 0 0 R 57.6 0.0 0:11.50 kswapd1 [...] which is just showing a partial list of the 32 kswapd threads that are running mostly full tilt, burning ~28 full CPU cores. If the same test case is run with RWF_UNCACHED set for the buffered read, the output looks as follows: Reading bs 65536, uncached 0 1s: 153144MB/sec 2s: 156760MB/sec 3s: 158110MB/sec 4s: 158009MB/sec 5s: 158043MB/sec 6s: 157638MB/sec 7s: 157999MB/sec 8s: 158024MB/sec 9s: 157764MB/sec 10s: 157477MB/sec 11s: 157417MB/sec 12s: 157455MB/sec 13s: 157233MB/sec 14s: 156692MB/sec which is just chugging along at ~155GB/sec of read performance. Looking at top, we see: PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 7961 root 20 0 267004 0 0 S 3180 0.0 5:37.95 uncached 8024 axboe 20 0 14292 4096 0 R 1.0 0.0 0:00.13 top where just the test app is using CPU, no reclaim is taking place outside of the main thread. Not only is performance 65% better, it's also using half the CPU to do it. Signed-off-by: Jens Axboe Acked-by: "Darrick J. Wong" --- mm/filemap.c | 28 ++++++++++++++++++++++++++-- mm/swap.c | 2 ++ 2 files changed, 28 insertions(+), 2 deletions(-) diff --git a/mm/filemap.c b/mm/filemap.c index dd3042de8038..139d1db79ff8 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -2473,6 +2473,8 @@ static int filemap_create_folio(struct kiocb *iocb, folio =3D filemap_alloc_folio(mapping_gfp_mask(mapping), min_order); if (!folio) return -ENOMEM; + if (iocb->ki_flags & IOCB_UNCACHED) + __folio_set_uncached(folio); =20 /* * Protect against truncate / hole punch. Grabbing invalidate_lock @@ -2518,6 +2520,8 @@ static int filemap_readahead(struct kiocb *iocb, stru= ct file *file, =20 if (iocb->ki_flags & IOCB_NOIO) return -EAGAIN; + if (iocb->ki_flags & IOCB_UNCACHED) + ractl.uncached =3D 1; page_cache_async_ra(&ractl, folio, last_index - folio->index); return 0; } @@ -2547,6 +2551,8 @@ static int filemap_get_pages(struct kiocb *iocb, size= _t count, return -EAGAIN; if (iocb->ki_flags & IOCB_NOWAIT) flags =3D memalloc_noio_save(); + if (iocb->ki_flags & IOCB_UNCACHED) + ractl.uncached =3D 1; page_cache_sync_ra(&ractl, last_index - index); if (iocb->ki_flags & IOCB_NOWAIT) memalloc_noio_restore(flags); @@ -2594,6 +2600,20 @@ static inline bool pos_same_folio(loff_t pos1, loff_= t pos2, struct folio *folio) return (pos1 >> shift =3D=3D pos2 >> shift); } =20 +static void filemap_uncached_read(struct address_space *mapping, + struct folio *folio) +{ + if (!folio_test_uncached(folio)) + return; + if (folio_test_writeback(folio) || folio_test_dirty(folio)) + return; + if (folio_trylock(folio)) { + if (folio_test_clear_uncached(folio)) + folio_unmap_invalidate(mapping, folio, 0); + folio_unlock(folio); + } +} + /** * filemap_read - Read data from the page cache. * @iocb: The iocb to read. @@ -2707,8 +2727,12 @@ ssize_t filemap_read(struct kiocb *iocb, struct iov_= iter *iter, } } put_folios: - for (i =3D 0; i < folio_batch_count(&fbatch); i++) - folio_put(fbatch.folios[i]); + for (i =3D 0; i < folio_batch_count(&fbatch); i++) { + struct folio *folio =3D fbatch.folios[i]; + + filemap_uncached_read(mapping, folio); + folio_put(folio); + } folio_batch_init(&fbatch); } while (iov_iter_count(iter) && iocb->ki_pos < isize && !error); =20 diff --git a/mm/swap.c b/mm/swap.c index 10decd9dffa1..4019ab371759 100644 --- a/mm/swap.c +++ b/mm/swap.c @@ -427,6 +427,8 @@ static void folio_inc_refs(struct folio *folio) */ void folio_mark_accessed(struct folio *folio) { + if (folio_test_uncached(folio)) + return; if (lru_gen_enabled()) { folio_inc_refs(folio); return; --=20 2.45.2 From nobody Fri Dec 19 04:52:36 2025 Received: from mail-oi1-f181.google.com (mail-oi1-f181.google.com [209.85.167.181]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6BFCA205AD0 for ; Tue, 3 Dec 2024 15:32:59 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.167.181 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1733239980; cv=none; b=DQImshbPwNMuwB+BXru6bur/65P9+xoZk5Eb9D9aO07iW+eHKnXOMzluZlURQfLD6ck4FtJYhWOm+XC2ucGoTPPGGXsS6rDe+7xiM5P0v/YITGJ++3e42m45DLYkkYKWEZhtffTww/JUMoBrm0Qe/GU5Ds5WCoMeczsztiMA+po= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1733239980; c=relaxed/simple; bh=vs/kjKbKvWVyON8cIhVh+OmQAXll4qsNA3yrIFPC0B8=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=D68yXB2eW1YCWtoSYrw3xIy/pvS5W3rgtwyatoKYS9+25Umbwday9RkdTJLumBKpkTRMCCARon7qI1aSKW+B37eou5c2tYua5xoXum3+gKXnJN9o3qng98ilfO5bm3DT2h/m/sNiAjd1eLCvW/5eIg3/XBiRbuKsk7fDTfJ45vQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=ZoXjuGJ6; arc=none smtp.client-ip=209.85.167.181 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="ZoXjuGJ6" Received: by mail-oi1-f181.google.com with SMTP id 5614622812f47-3ea60f074c3so3240308b6e.1 for ; Tue, 03 Dec 2024 07:32:59 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1733239978; x=1733844778; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=oKUI9o4nURcpztCkRpa2upslaug1l9SMNKWGbNjNG00=; b=ZoXjuGJ6LW812fJZgfsUYAW0o3hUhJ1ECO3V3B3mzS9aIwU9AjwvE85cw46HRAI5RZ epkxvG7nOJM1hP2wfxke+5kgfquRsWWR65Lk0pAEnMXQ2+gMnh4ysevXl+0+Vx0QEp1h cCYRdj3tlbvMIc6o2QzxKIA4awRVNLJiNLD3bcLulb1skvEHqI3ICwDZNgmnR+XJGlta PzOYkueg0mzh2ofTcsjh3eUtmS35rpWUUBzZIURqsN2uVxdy0dU5Be80xisMA3sZgCts XoQKMOK2tYHYoMpMmSRSQmG2SLD6J2Cx+284FNdBXzFv+ywcAQBF8/n9i4cgFz1lrk6i sjjA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1733239978; x=1733844778; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=oKUI9o4nURcpztCkRpa2upslaug1l9SMNKWGbNjNG00=; b=vb34TiNB636/bnvwD3D4E0soG08Jg/lwMi38z9+SiL+wU6QVPXl4t7eJpZSDg+p6Pt ww0we4sm1J88U7JgrCEYB3Iu6ksy3kR1/T+tj4iom+MYxnXkVSJS8MF9tkWx6+F22adD k7o1F4Valln3niPUVtB9y5wgBlJUh4PKmA/+VAw/02kBenBG95khsPDnJqxJCvg1SyrX PaHYZ2k0xhmWPh6zA3jvdBtFvbSvbZUKD265L1w/C1CR01cRP7x8A5pg36stzcdCl+ii 4hA5bcZlHXDIDiGhy522jnsCMfHAjFFdLVsKcWw33PjEC55BEDGNALE9s2/M4qhxnpHQ fI7g== X-Forwarded-Encrypted: i=1; AJvYcCW8EBkEEbrTYDWjwjST/06osCXMNyzQB6By1OJ4HykBo8ApWQ2aDH2ULNXj9A+pgd10rovYfRCrajWXh68=@vger.kernel.org X-Gm-Message-State: AOJu0Yz7Szke8StCKMgOBMzmiS+fmpAvEZZMeKBpX6KQUsvBb7Gs7mBh 2QKi1wxZ9O0/ZnfCMuxgE0yk0tsI9j/lOiSdBEspGZWzX8juq/P+XPpz6e6ix1g= X-Gm-Gg: ASbGncsCWyRcgYBuRieb3sD9idDFf57Lg/Ps+m4QPsOuKho+/S1N0qFW8k2kt86Z0F4 YsskjPxnRV34RykQAcJOSpqrn3c730qQcMmPNDaLRLe3tOegPJdjEXKdXxgb1gcqeLhDseDC2QT q56b72eOohNtOn1JYqmk7Q6D6TRlboRyD7g1YE04cGLtc2mLMhUrq252ejVxR6VzZJKd7gEBjCO rQmNaw+GBT6tAMkkfa7b08grej09EOsMwZtPaP2migAteBTdYqZz0BvjGk= X-Google-Smtp-Source: AGHT+IGmoU9sxAuWmjwWvkKLQ0cqG9D7TpsEEAnWsGzkgwInxbMdKTegKVNOgiZahNH2vDY40Qhwdw== X-Received: by 2002:a05:6808:218d:b0:3ea:4aad:a877 with SMTP id 5614622812f47-3eae50d7cd4mr2734954b6e.43.1733239978709; Tue, 03 Dec 2024 07:32:58 -0800 (PST) Received: from localhost.localdomain ([130.250.255.163]) by smtp.gmail.com with ESMTPSA id 5614622812f47-3ea86036cbbsm2891878b6e.8.2024.12.03.07.32.57 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 03 Dec 2024 07:32:58 -0800 (PST) From: Jens Axboe To: linux-mm@kvack.org, linux-fsdevel@vger.kernel.org Cc: hannes@cmpxchg.org, clm@meta.com, linux-kernel@vger.kernel.org, willy@infradead.org, kirill@shutemov.name, bfoster@redhat.com, Jens Axboe Subject: [PATCH 09/12] mm/filemap: drop uncached pages when writeback completes Date: Tue, 3 Dec 2024 08:31:45 -0700 Message-ID: <20241203153232.92224-11-axboe@kernel.dk> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20241203153232.92224-2-axboe@kernel.dk> References: <20241203153232.92224-2-axboe@kernel.dk> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" If the folio is marked as uncached, drop pages when writeback completes. Intended to be used with RWF_UNCACHED, to avoid needing sync writes for uncached IO. Signed-off-by: Jens Axboe Acked-by: "Darrick J. Wong" --- mm/filemap.c | 28 ++++++++++++++++++++++++++++ 1 file changed, 28 insertions(+) diff --git a/mm/filemap.c b/mm/filemap.c index 139d1db79ff8..eb6a8d39f9d0 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -1599,6 +1599,27 @@ int folio_wait_private_2_killable(struct folio *foli= o) } EXPORT_SYMBOL(folio_wait_private_2_killable); =20 +/* + * If folio was marked as uncached, then pages should be dropped when writ= eback + * completes. Do that now. If we fail, it's likely because of a big folio - + * just reset uncached for that case and latter completions should invalid= ate. + */ +static void folio_end_uncached(struct folio *folio) +{ + /* + * Hitting !in_task() should not happen off RWF_UNCACHED writeback, but + * can happen if normal writeback just happens to find dirty folios + * that were created as part of uncached writeback, and that writeback + * would otherwise not need non-IRQ handling. Just skip the + * invalidation in that case. + */ + if (in_task() && folio_trylock(folio)) { + if (folio->mapping) + folio_unmap_invalidate(folio->mapping, folio, 0); + folio_unlock(folio); + } +} + /** * folio_end_writeback - End writeback against a folio. * @folio: The folio. @@ -1609,6 +1630,8 @@ EXPORT_SYMBOL(folio_wait_private_2_killable); */ void folio_end_writeback(struct folio *folio) { + bool folio_uncached =3D false; + VM_BUG_ON_FOLIO(!folio_test_writeback(folio), folio); =20 /* @@ -1630,9 +1653,14 @@ void folio_end_writeback(struct folio *folio) * reused before the folio_wake_bit(). */ folio_get(folio); + if (!folio_test_dirty(folio)) + folio_uncached =3D folio_test_clear_uncached(folio); if (__folio_end_writeback(folio)) folio_wake_bit(folio, PG_writeback); acct_reclaim_writeback(folio); + + if (folio_uncached) + folio_end_uncached(folio); folio_put(folio); } EXPORT_SYMBOL(folio_end_writeback); --=20 2.45.2 From nobody Fri Dec 19 04:52:36 2025 Received: from mail-oi1-f182.google.com (mail-oi1-f182.google.com [209.85.167.182]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1DB56207A04 for ; Tue, 3 Dec 2024 15:33:00 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.167.182 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1733239982; cv=none; b=sY396jZp45Ioiv00ryIxUeuurgo1SqBk4oWwj8AHt32rJ1X5DWHEz2Evs/CzKzyhdLPXvDnYLL6p8mhE3nd3zETVtnfnGUX+UZTrUi/P+Q5M9IMyqqNJMyghYnUzKdz4/wj7I31ttT458yOIQ7ZJcc/KAwhaBW3qOSLU9+R+I1o= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1733239982; c=relaxed/simple; bh=+Kcl03qnXK2xysxVuDL9GwHkxgxUkvV623M1SHc2Ce8=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=dHQ29sIP+IzpbNjvDcPpjg7LW584XvZKOwvIlXBk/VN8LqusZ40Vu0YO+fYwbYHkRItbrkhVbY3qg3geb/ERgb1faoky+OMXop8tSrmPBaPAASRRGFEmiKfT4hWynHx2WZVT2cOg03PclzYH4X60EHsZRNrWRjG8zGqxBWmewKk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=zawtuFyk; arc=none smtp.client-ip=209.85.167.182 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="zawtuFyk" Received: by mail-oi1-f182.google.com with SMTP id 5614622812f47-3ea411ef5a9so2747596b6e.0 for ; Tue, 03 Dec 2024 07:33:00 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1733239980; x=1733844780; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=ckrp5/MogNB2HvhzJuNOPyu1ag5BruWn5XeKlERZse0=; b=zawtuFyksJZVwTNlOSWuLl0I7/NrJnGwM6osw35w7RF+X4LrOQBsIpdgqB21N2DG/L hs2QdLfiE//80U5o1IqKVnvGP/5rJciwzgJESHGtoiAGmd2A8D07PvYIoD/qq09Z7Hqd XvzlipIh94GKK1kJnwgRE3/19xA5Q9ihvdRUc1Slmyqe7J5l9co69/51Jva6CzI66LUW 9bMDGOXzlwMfzaK28tdx1giFwMEZS3GcXInjHWOOzqqbNyxG/kcGgC+tl1F3JdjMIycd kJ6BWTh3DFaDiIFxiFYz+iLsRlVvoYLhAzfZBlP49LBl3KnRdlxMazAvHXaCyDlSJdFv fJxA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1733239980; x=1733844780; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ckrp5/MogNB2HvhzJuNOPyu1ag5BruWn5XeKlERZse0=; b=qf72R8pQ5iaZLCsC6w73FHYLT71sgdlJZnn43m70LwW+AtmLrkPxIBIofUbtwr36Th wjSN0MlzUS2xSBOiYMiQZgdtVRetgGWs4M4ESCUm7lAxsQpEoKj08gGusKsDvklEAnS/ u3XBOi0+vaHNoNEg1jX1oGr88+X0dvj6yLmGhwkzJFUqbY5qkvcd9q7zgwM/BHpHdHm1 qY2SkhLmvTHf4uZx3Y8OHnbx6p4WnTaPYB6UduNK+qQorXf38rt1sthmjS7M6qBNPffg Xgb46r1LZ8EwUAbby5lfb8LkzMxnaEfpK5b9vcOn25m0mVhsMJqiEfI5IbfVUI/+GW6R Mufw== X-Forwarded-Encrypted: i=1; AJvYcCUgXUYFIUAlpIwE6HhS/swM+j3yF4sVs5QEy6qDvEkASh68FgfXL6JkXaBLx3IWvWnIdv/6+IDurGGrTjY=@vger.kernel.org X-Gm-Message-State: AOJu0YyM3KTaL88sWOwAPKRQJcxQsHZ9SH3AGIHcpnOZul5TFyZUn4a0 NCNtJE97mJ7NSdB5AVaIlzaqkM9B43Hc4hu/Y1bSQTb9qeshEejLMkYoBZSSkx4= X-Gm-Gg: ASbGncvIZUl0quSK27PW872eblCp+bOk8wtKrtWc3vOya4RPQhGcVFOyyma7Eal/mJT PFbHszwnRt+McFVIpIKNTYRH/PA66iVVeEYzMIyE8dryMqb0ymQc8/ha6A5FE+Nzel3iXIDxL9v BDK62m/POB4Jjsdd66E/2b9wLWWIHwnDisciMlcHbGCbJNorsjtz5YzfOS8SbBzvwjP9fgHdOGL ILRqjmWm8nbNGxo9g5Q2sczSdVAfp8L0GP2zCZes6ZyIH1A9KQ9aFuip9c= X-Google-Smtp-Source: AGHT+IG43FQHTFqL4PqM6F3qojzARV1JxsiCZ9D3Zgsdfa2TEHjYK26CmkPy2H7KTh+zkEYZWpQaaw== X-Received: by 2002:a05:6808:3085:b0:3ea:52b7:ebdd with SMTP id 5614622812f47-3eae50c51b8mr2904392b6e.42.1733239980115; Tue, 03 Dec 2024 07:33:00 -0800 (PST) Received: from localhost.localdomain ([130.250.255.163]) by smtp.gmail.com with ESMTPSA id 5614622812f47-3ea86036cbbsm2891878b6e.8.2024.12.03.07.32.58 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 03 Dec 2024 07:32:59 -0800 (PST) From: Jens Axboe To: linux-mm@kvack.org, linux-fsdevel@vger.kernel.org Cc: hannes@cmpxchg.org, clm@meta.com, linux-kernel@vger.kernel.org, willy@infradead.org, kirill@shutemov.name, bfoster@redhat.com, Jens Axboe Subject: [PATCH 10/12] mm/filemap: add filemap_fdatawrite_range_kick() helper Date: Tue, 3 Dec 2024 08:31:46 -0700 Message-ID: <20241203153232.92224-12-axboe@kernel.dk> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20241203153232.92224-2-axboe@kernel.dk> References: <20241203153232.92224-2-axboe@kernel.dk> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Works like filemap_fdatawrite_range(), except it's a non-integrity data writeback and hence only starts writeback on the specified range. Will help facilitate generically starting uncached writeback from generic_write_sync(), as header dependencies preclude doing this inline from fs.h. Signed-off-by: Jens Axboe Acked-by: "Darrick J. Wong" --- include/linux/fs.h | 2 ++ mm/filemap.c | 18 ++++++++++++++++++ 2 files changed, 20 insertions(+) diff --git a/include/linux/fs.h b/include/linux/fs.h index b64a78582f06..40383f5cc6a2 100644 --- a/include/linux/fs.h +++ b/include/linux/fs.h @@ -2878,6 +2878,8 @@ extern int __must_check file_fdatawait_range(struct f= ile *file, loff_t lstart, extern int __must_check file_check_and_advance_wb_err(struct file *file); extern int __must_check file_write_and_wait_range(struct file *file, loff_t start, loff_t end); +int filemap_fdatawrite_range_kick(struct address_space *mapping, loff_t st= art, + loff_t end); =20 static inline int file_write_and_wait(struct file *file) { diff --git a/mm/filemap.c b/mm/filemap.c index eb6a8d39f9d0..826df99e294f 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -449,6 +449,24 @@ int filemap_fdatawrite_range(struct address_space *map= ping, loff_t start, } EXPORT_SYMBOL(filemap_fdatawrite_range); =20 +/** + * filemap_fdatawrite_range_kick - start writeback on a range + * @mapping: target address_space + * @start: index to start writeback on + * @end: last (non-inclusive) index for writeback + * + * This is a non-integrity writeback helper, to start writing back folios + * for the indicated range. + * + * Return: %0 on success, negative error code otherwise. + */ +int filemap_fdatawrite_range_kick(struct address_space *mapping, loff_t st= art, + loff_t end) +{ + return __filemap_fdatawrite_range(mapping, start, end, WB_SYNC_NONE); +} +EXPORT_SYMBOL_GPL(filemap_fdatawrite_range_kick); + /** * filemap_flush - mostly a non-blocking flush * @mapping: target address_space --=20 2.45.2 From nobody Fri Dec 19 04:52:36 2025 Received: from mail-oi1-f172.google.com (mail-oi1-f172.google.com [209.85.167.172]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 75A47208966 for ; Tue, 3 Dec 2024 15:33:02 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.167.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1733239984; cv=none; b=oEWCf6n+w+b3GrGiVX1eTYnSQySW+2vlH4rv9EbdqonubX4jrbAtv/idZAX1DkQLf5iqKHQ6Ay2JdrJmwChg+MatWvD7FMnGzelPY47rblvjQmmpJoQHDSeirYZ18QPQQ5tQ/u7ZuLSjHCLhUIJ5qaM6TwUGndovQ5jYNvhSzOQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1733239984; c=relaxed/simple; bh=H+SJrhFgkMARuJzf6zQzkW6swewkOc9KT1deZMH8OhQ=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=PJLu5bUZouaxqAzhweO2G8KQLbIP9xGY/gRVLUKuhFtQndyT9LiF942FEe2zrxBEY4vo+r/9n8Dmc2lQbDNCyozwwYqr/L6X7Gz1a969VuVQqtsNe6dlq0+fKkFon5Jwab0tNiEJj8/q/HZVJCW9sBc5z+rQf6gqMve4TGFDB5Y= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=NvPvt5h7; arc=none smtp.client-ip=209.85.167.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="NvPvt5h7" Received: by mail-oi1-f172.google.com with SMTP id 5614622812f47-3ea5a7a5e48so2650371b6e.0 for ; Tue, 03 Dec 2024 07:33:02 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1733239981; x=1733844781; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=rNRkYctWaLbO70YbebeuOKFfbUg5iKQfEN1xFJYmHB8=; b=NvPvt5h7l8lngjBGNuxy1XfMurd28jtD71TE5JEBJN8m0imAbUo5S3bdcA1KRVni0K adBG9tVcZH3XkpLeOYkphqHUsl8KthkdbhxTVwzY/AYvhuXK8Y1+2DFvCv6V01nqbG2e 3oUcfMTQszBNztI4erRKjKFbzvlrRZNoM57g/od4rXo89X13JlAbyf7VExy+ZafmwNUN TmnqZ40rMEAHhACUFoJrPsE14XGVVZYsZnLmmGH3lDZfNy7cR9Ctj/KxowZFTmsInge0 Bvoc8xEaEU5E2ZG/+QlgV/0AKK+JojGNNqffpdz0vhhgRX0cutoayz9fj9wDcLTW1kct 3org== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1733239981; x=1733844781; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=rNRkYctWaLbO70YbebeuOKFfbUg5iKQfEN1xFJYmHB8=; b=FOh8SqmNtiAPCgEqdV5TRLXrTET/6Yg3blsgN9yMXuD3vfAPLTy8673nFscnIByH6Q dn/rmGsMi32wffIlZGzE/+80E9vy2Iok5ndOchF/FoAW+sV+K4RIwcX2vLsCVOEy+HIt LImJgLUHo0IvxDONOBp4/KlvQerYM8uz57kRPkC8hOuwpsHV011faMdFtfn2utoYVA9i kcM2Sm0Eno81RhMAhHaR1vFZ4zs18Dt5iMmqIvRoIHmmKxRB7/nWWIQ+1B6O14mv+Ab/ 8+gA5sj6PSdTVimwmTQ1iILFjh5iKP53lXcE5T45q3RqQDSrTTbbTCLvVoR58/36JWzB xhEA== X-Forwarded-Encrypted: i=1; AJvYcCWzvKUf3QGyJsMC+Rdk8RmJ1/tmb/ny+b4r5SX1HhkquD68sHv8GC+UDY3Lw2zUC65TUM68PHs9CkHueLk=@vger.kernel.org X-Gm-Message-State: AOJu0Yy0TGdG+5HJpLN1kk1bzGQM4lWnQODpmbtPNmwVRji9i3Hw5r6N 7Ga/yqFkSIQcKgzcqeZ6QBd5mN1nHN4i5T7u38lTYJCPd45+BxP3e9hsBU73flM= X-Gm-Gg: ASbGncuwcUqexw7o1bJONja0x/mfNQ/IT5bVrP0cXt/tA24CLAYxQ1rlar5GoeYHazU PUaGU3m44WxJZ6yhjnzPtrQNl+Pvh1vn6sXEr0+AhVF26JlqJq53hZVOhzXU3S9puzh2aIpB2nN asITXTyWWFpqSvL6zg0ltAYmfEbn5yl20VKNcUey2fcT+ZKFJnrnL3na91Ns1bGNNDndNzMPJMb T8L/XHZkIIHetJgnHQ29y17W3WPF0aUgQ07upTXuWkfjRMvTiY44BVlO9Q= X-Google-Smtp-Source: AGHT+IGnmoFBWMRz09BuY3oNVVI/1OH4qyI0X8Y48goC/ICz9U7NJeNOcoRPtjmNOo7azkeA+JUnjQ== X-Received: by 2002:a05:6808:200d:b0:3e6:6208:b737 with SMTP id 5614622812f47-3eae4f3bce5mr2757240b6e.17.1733239981545; Tue, 03 Dec 2024 07:33:01 -0800 (PST) Received: from localhost.localdomain ([130.250.255.163]) by smtp.gmail.com with ESMTPSA id 5614622812f47-3ea86036cbbsm2891878b6e.8.2024.12.03.07.33.00 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 03 Dec 2024 07:33:00 -0800 (PST) From: Jens Axboe To: linux-mm@kvack.org, linux-fsdevel@vger.kernel.org Cc: hannes@cmpxchg.org, clm@meta.com, linux-kernel@vger.kernel.org, willy@infradead.org, kirill@shutemov.name, bfoster@redhat.com, Jens Axboe Subject: [PATCH 11/12] mm/filemap: make buffered writes work with RWF_UNCACHED Date: Tue, 3 Dec 2024 08:31:47 -0700 Message-ID: <20241203153232.92224-13-axboe@kernel.dk> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20241203153232.92224-2-axboe@kernel.dk> References: <20241203153232.92224-2-axboe@kernel.dk> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" If RWF_UNCACHED is set for a write, mark new folios being written with uncached. This is done by passing in the fact that it's an uncached write through the folio pointer. We can only get there when IOCB_UNCACHED was allowed, which can only happen if the file system opts in. Opting in means they need to check for the LSB in the folio pointer to know if it's an uncached write or not. If it is, then FGP_UNCACHED should be used if creating new folios is necessary. Uncached writes will drop any folios they create upon writeback completion, but leave folios that may exist in that range alone. Since ->write_begin() doesn't currently take any flags, and to avoid needing to change the callback kernel wide, use the foliop being passed in to ->write_begin() to signal if this is an uncached write or not. File systems can then use that to mark newly created folios as uncached. This provides similar benefits to using RWF_UNCACHED with reads. Testing buffered writes on 32 files: writing bs 65536, uncached 0 1s: 196035MB/sec 2s: 132308MB/sec 3s: 132438MB/sec 4s: 116528MB/sec 5s: 103898MB/sec 6s: 108893MB/sec 7s: 99678MB/sec 8s: 106545MB/sec 9s: 106826MB/sec 10s: 101544MB/sec 11s: 111044MB/sec 12s: 124257MB/sec 13s: 116031MB/sec 14s: 114540MB/sec 15s: 115011MB/sec 16s: 115260MB/sec 17s: 116068MB/sec 18s: 116096MB/sec where it's quite obvious where the page cache filled, and performance dropped from to about half of where it started, settling in at around 115GB/sec. Meanwhile, 32 kswapds were running full steam trying to reclaim pages. Running the same test with uncached buffered writes: writing bs 65536, uncached 1 1s: 198974MB/sec 2s: 189618MB/sec 3s: 193601MB/sec 4s: 188582MB/sec 5s: 193487MB/sec 6s: 188341MB/sec 7s: 194325MB/sec 8s: 188114MB/sec 9s: 192740MB/sec 10s: 189206MB/sec 11s: 193442MB/sec 12s: 189659MB/sec 13s: 191732MB/sec 14s: 190701MB/sec 15s: 191789MB/sec 16s: 191259MB/sec 17s: 190613MB/sec 18s: 191951MB/sec and the behavior is fully predictable, performing the same throughout even after the page cache would otherwise have fully filled with dirty data. It's also about 65% faster, and using half the CPU of the system compared to the normal buffered write. Signed-off-by: Jens Axboe Acked-by: "Darrick J. Wong" --- include/linux/fs.h | 5 +++++ include/linux/pagemap.h | 9 +++++++++ mm/filemap.c | 12 +++++++++++- 3 files changed, 25 insertions(+), 1 deletion(-) diff --git a/include/linux/fs.h b/include/linux/fs.h index 40383f5cc6a2..32255473f79d 100644 --- a/include/linux/fs.h +++ b/include/linux/fs.h @@ -2912,6 +2912,11 @@ static inline ssize_t generic_write_sync(struct kioc= b *iocb, ssize_t count) (iocb->ki_flags & IOCB_SYNC) ? 0 : 1); if (ret) return ret; + } else if (iocb->ki_flags & IOCB_UNCACHED) { + struct address_space *mapping =3D iocb->ki_filp->f_mapping; + + filemap_fdatawrite_range_kick(mapping, iocb->ki_pos, + iocb->ki_pos + count); } =20 return count; diff --git a/include/linux/pagemap.h b/include/linux/pagemap.h index f2d49dccb7c1..e49587c40157 100644 --- a/include/linux/pagemap.h +++ b/include/linux/pagemap.h @@ -14,6 +14,7 @@ #include #include #include /* for in_interrupt() */ +#include #include =20 struct folio_batch; @@ -70,6 +71,14 @@ static inline int filemap_write_and_wait(struct address_= space *mapping) return filemap_write_and_wait_range(mapping, 0, LLONG_MAX); } =20 +/* + * Value passed in to ->write_begin() if IOCB_UNCACHED is set for the writ= e, + * and the ->write_begin() handler on a file system supporting FOP_UNCACHED + * must check for this and pass FGP_UNCACHED for folio creation. + */ +#define foliop_uncached ((struct folio *) 0xfee1c001) +#define foliop_is_uncached(foliop) (*(foliop) =3D=3D foliop_uncached) + /** * filemap_set_wb_err - set a writeback error on an address_space * @mapping: mapping in which to set writeback error diff --git a/mm/filemap.c b/mm/filemap.c index 826df99e294f..00f3c6c58629 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -4095,7 +4095,7 @@ ssize_t generic_perform_write(struct kiocb *iocb, str= uct iov_iter *i) ssize_t written =3D 0; =20 do { - struct folio *folio; + struct folio *folio =3D NULL; size_t offset; /* Offset into folio */ size_t bytes; /* Bytes to write to folio */ size_t copied; /* Bytes copied from user */ @@ -4123,6 +4123,16 @@ ssize_t generic_perform_write(struct kiocb *iocb, st= ruct iov_iter *i) break; } =20 + /* + * If IOCB_UNCACHED is set here, we now the file system + * supports it. And hence it'll know to check folip for being + * set to this magic value. If so, it's an uncached write. + * Whenever ->write_begin() changes prototypes again, this + * can go away and just pass iocb or iocb flags. + */ + if (iocb->ki_flags & IOCB_UNCACHED) + folio =3D foliop_uncached; + status =3D a_ops->write_begin(file, mapping, pos, bytes, &folio, &fsdata); if (unlikely(status < 0)) --=20 2.45.2 From nobody Fri Dec 19 04:52:36 2025 Received: from mail-oi1-f169.google.com (mail-oi1-f169.google.com [209.85.167.169]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D8647209692 for ; Tue, 3 Dec 2024 15:33:03 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.167.169 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1733239985; cv=none; b=VBSPylIKUYN9P0XBMf+jZA667wvVxbtcLQ9NsoQ0jGzIUOc85N2rsTaTbHIjZnx3jkR0c+cTHv587/W+WHsQFc8LOUvqM0mMI2pkRs+3cz27N8tK2h+Yv7oRAHAyHptm5B931ujlz3ZMw1dv1wz8W6nwUgrLpxWTxfj74+edw/I= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1733239985; c=relaxed/simple; bh=uuqM4dJdCE0QCImMtNfF8QapqPv5QHbWROfY8tXB1So=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=j2jAM7LNZAJlDVC0/kLWgvTBT1DKFgWrf9BVED7Jr0z6Eqq57MfUm40a+QUPZPGfAt+6u4/e8qy8EJBOfOkTIIEtlVgr3AMx04j6sdxaxGLOsU7JZAVjcoCXDZKNxoy/sXvd6vNXsvsSXEnZnUCWGd9b8ojVhOYR3u5GUL1pJUQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=QOc0N/NU; arc=none smtp.client-ip=209.85.167.169 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="QOc0N/NU" Received: by mail-oi1-f169.google.com with SMTP id 5614622812f47-3ea55a2a38bso3014532b6e.1 for ; Tue, 03 Dec 2024 07:33:03 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1733239983; x=1733844783; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=45ZT5fvdDImmdgAdnvc6z/+AzY3WZJ2Fl5NYIbZiSAo=; b=QOc0N/NUqzgyQBFL3YcsFrekttyd/1YnrV1H2WXWVqq8rRuDr/fMbYTz0t0nSvjZsZ PM0IJhaPyr1XFGkB+CgN5rJGrQo594jDOQh7cePyS+sg/JBAs1GtxQqcsovONgFLy5Op 1+ljoiONKHAsiMjXtSjgd+p1IUQBCGfFdjwWv68j6hNyma/YiG/+fSzPcVvx21A7TXKZ J8YTAo4MNYwj2TozO7JRy9v50QYghtb7lOuUhpepBkAbV48PxgCkaesFHij5OlaLIewn r6GeI3nXSYx2eMFHBv35VFSCGjE7XwwK8w5vv5wtuIXoUTRr94mCjgtrTWpg37lX2KUa Jc9w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1733239983; x=1733844783; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=45ZT5fvdDImmdgAdnvc6z/+AzY3WZJ2Fl5NYIbZiSAo=; b=lwV7JRiIjO+CNval9cFuhWjEb3gNnIr4nsAr4Pe982GwUTUNtZXWQpx7hYBarXOY+Q A4vooTEujwg+d6Fbl+J7VeHBfQdOzGmiXiRxb6d2JBTNu92otTWWiz9twnoclNCnGh/d 7l0qsnfidxyRoYdWCCqxowTo8DYG9azBOByUgQAq7eXRQynQaM8+Rcu55Be8ljWcpo1W 16i0Pi0Yg+VOKCmUV5KsnS7t8/CS3HqJCSnZN8H8lkO2cJQP5yuiz8KdS8roh27P114D zPjndMPwnwEqqSo4cCQ4Q/VE1bJxKJIu3f5x3M21PssSE9o4a4pUViGLxEiML+MmOMKL 9hsA== X-Forwarded-Encrypted: i=1; AJvYcCXUiq6Jdheg2KKd4hYH/Ta5pq46yD9rDpvO7i7NEdiM6dbc0E1D6fV/SAqBQIude15mUz0zcd3Lxd4nbMo=@vger.kernel.org X-Gm-Message-State: AOJu0YwAPyODMWOpYpQpjIpeyYAknyEDUIrHc4xugRGqAEicQyjOva8k LY91T53UpqIqQO2LsVeRWb7zZNiLk939Vh/ch5ZG8iD9aeSrSmah6joUZEbHx/Q= X-Gm-Gg: ASbGncvka8F1VOElDOkCmSxC8Tc1HfU7Yd/C9f9j5+or0zWe8JZUU//8uBwUIxdUe6M ugLOC/2yFqRkW/MMkDVknBzu+BYUi3jxYdIJZzGZY1AzWGvwavJ4WOr/aWXMowTiNmRrEc+qul8 8yTiY37NCQzRfnDqNmtOLcK+0K/yqheNx9tkwYPuHpWrcBB4DJx/iaJ+IzAVKxz9ulu9x/Bweh/ aNDyzE14yrQiQhM8hyF/AZl1/2zxwcMUEWYqGSdmYG4i80cLoi52C7t4MY= X-Google-Smtp-Source: AGHT+IFgNffB59kRtGSa279D+oxFDG6GGmZlBVzfslJYY485yS6Rp8valFBlgcp9p+EP83czXVtQIg== X-Received: by 2002:a05:6808:199e:b0:3ea:6533:f19d with SMTP id 5614622812f47-3eae505a0femr2960000b6e.30.1733239982974; Tue, 03 Dec 2024 07:33:02 -0800 (PST) Received: from localhost.localdomain ([130.250.255.163]) by smtp.gmail.com with ESMTPSA id 5614622812f47-3ea86036cbbsm2891878b6e.8.2024.12.03.07.33.01 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 03 Dec 2024 07:33:02 -0800 (PST) From: Jens Axboe To: linux-mm@kvack.org, linux-fsdevel@vger.kernel.org Cc: hannes@cmpxchg.org, clm@meta.com, linux-kernel@vger.kernel.org, willy@infradead.org, kirill@shutemov.name, bfoster@redhat.com, Jens Axboe Subject: [PATCH 12/12] mm: add FGP_UNCACHED folio creation flag Date: Tue, 3 Dec 2024 08:31:48 -0700 Message-ID: <20241203153232.92224-14-axboe@kernel.dk> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20241203153232.92224-2-axboe@kernel.dk> References: <20241203153232.92224-2-axboe@kernel.dk> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Callers can pass this in for uncached folio creation, in which case if a folio is newly created it gets marked as uncached. If a folio exists for this index and lookup succeeds, then it will not get marked as uncached. If an !uncached lookup finds a cached folio, clear the flag. For that case, there are competeting uncached and cached users of the folio, and it should not get pruned. Signed-off-by: Jens Axboe Acked-by: "Darrick J. Wong" --- include/linux/pagemap.h | 2 ++ mm/filemap.c | 5 +++++ 2 files changed, 7 insertions(+) diff --git a/include/linux/pagemap.h b/include/linux/pagemap.h index e49587c40157..374872acbe1d 100644 --- a/include/linux/pagemap.h +++ b/include/linux/pagemap.h @@ -721,6 +721,7 @@ pgoff_t page_cache_prev_miss(struct address_space *mapp= ing, * * %FGP_NOFS - __GFP_FS will get cleared in gfp. * * %FGP_NOWAIT - Don't block on the folio lock. * * %FGP_STABLE - Wait for the folio to be stable (finished writeback) + * * %FGP_UNCACHED - Uncached buffered IO * * %FGP_WRITEBEGIN - The flags to use in a filesystem write_begin() * implementation. */ @@ -734,6 +735,7 @@ typedef unsigned int __bitwise fgf_t; #define FGP_NOWAIT ((__force fgf_t)0x00000020) #define FGP_FOR_MMAP ((__force fgf_t)0x00000040) #define FGP_STABLE ((__force fgf_t)0x00000080) +#define FGP_UNCACHED ((__force fgf_t)0x00000100) #define FGF_GET_ORDER(fgf) (((__force unsigned)fgf) >> 26) /* top 6 bits */ =20 #define FGP_WRITEBEGIN (FGP_LOCK | FGP_WRITE | FGP_CREAT | FGP_STABLE) diff --git a/mm/filemap.c b/mm/filemap.c index 00f3c6c58629..a03a9b9127b8 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -2001,6 +2001,8 @@ struct folio *__filemap_get_folio(struct address_spac= e *mapping, pgoff_t index, /* Init accessed so avoid atomic mark_page_accessed later */ if (fgp_flags & FGP_ACCESSED) __folio_set_referenced(folio); + if (fgp_flags & FGP_UNCACHED) + __folio_set_uncached(folio); =20 err =3D filemap_add_folio(mapping, folio, index, gfp); if (!err) @@ -2023,6 +2025,9 @@ struct folio *__filemap_get_folio(struct address_spac= e *mapping, pgoff_t index, =20 if (!folio) return ERR_PTR(-ENOENT); + /* not an uncached lookup, clear uncached if set */ + if (folio_test_uncached(folio) && !(fgp_flags & FGP_UNCACHED)) + folio_clear_uncached(folio); return folio; } EXPORT_SYMBOL(__filemap_get_folio); --=20 2.45.2