From nobody Sat Feb 7 19:08:19 2026 Received: from mail-oi1-f181.google.com (mail-oi1-f181.google.com [209.85.167.181]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 092907D090 for ; Tue, 19 Mar 2024 09:41:42 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.167.181 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710841304; cv=none; b=dpxgE/P2qKYyjw4VLTCcqoW2jpscmIbLYv1FoPhJo/7pPAWmSEIc0jdbvPyxoGJEvw8MRvCdg2znyBmAy3zfHIHuCuM6WUi7vh+DOYjdOx9VofMgEQKzQUAzdF5IwnfE1aBMolPREsJiHwKYUbTTixAiOYLY4KoAbf6ZsrL4rXY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710841304; c=relaxed/simple; bh=5DGsGaRy1eaD1w4IlwHuhLUmJWP+gJa6fNqIfuItsZI=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=WDO0QqN+IUjtWxQQMyJnhstMPYhSfeJQUVRD9IZkJi+UMd/6roEN4bTl45t1stTcpsxZX4JPsSX7H5aiIT3+7HuA8V3/BT4057K3RnYA3lJZxgLDEWJnUz4AtDEP1o1PnK/AJaTYt3eL48kGYjgubTxzgGrcnut3UVIfBZ9c/M4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=U/mZW4cJ; arc=none smtp.client-ip=209.85.167.181 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="U/mZW4cJ" Received: by mail-oi1-f181.google.com with SMTP id 5614622812f47-3c38d4b670bso870388b6e.2 for ; Tue, 19 Mar 2024 02:41:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1710841302; x=1711446102; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:from:to:cc:subject :date:message-id:reply-to; bh=ZwvTELM6ByipvSR/wfDTLW1X7/KlLM1aOXjOWDL7DtM=; b=U/mZW4cJG/rtNBcBdJLoJ5y98UtuFxaHIoqpsaKXc5YwgW0ApgJZTOv6ofbbHis6Nr wc4c5J32lVqC1n0dib9V3hDRhpKCiqsSwjraBhWAbBOdFlP/x6qQDyIKOM3KD1KMblge bwiQt2H8kJPzRy0KxfoGVnNexKaGtVt2kLKeIZIzFm7I5g6kyPByhn/qFAEAQBxXoaXM J14OICgf9rRPEYd24OpBy8+yAqtgarKc7xTbCLzKw7pbpqnnSAl6zI3eo0U5vfJSYNs9 Feuh/U/+QIraMs9nASGlORjLUD6spo8LNXtf55JnXFlkwjm1TDDqtailm+TUdfqcwO8G Xpfw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1710841302; x=1711446102; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=ZwvTELM6ByipvSR/wfDTLW1X7/KlLM1aOXjOWDL7DtM=; b=Gx9wwofTo9MMp5WGpbZRrpb2KA9xqmUpaIE4QF28DlShvn965LEoe174dD/7sT1yp4 dQ76rsqFiMZxBwHhAyLTeW93+hRgJ0PzxTYPWMpopFySIYyK5O3HvTKkXp/CTmxKWp7u FUUdy7YEzfkcLndQpLAkBdVG74jOs2M9D/WRReDQ3j7/u1+aYIbahKWRdIwUEX/86f4N FgwTwL6L/Wy11bEEfq3TEIQQNiC0EkgqOWPgKJJiLulwi4pFzh0mu/1LapOn0i/+Bc1s 0wyG32Q63KZ4eBZx1Yqyn/RKtJQ1U9Bjapgn8iVusOZX0LkZqPIYDwNUzNIGxi7HQnrb Ol2w== X-Forwarded-Encrypted: i=1; AJvYcCXVTYVejlJrE8UxEymp5d9R4F4IW0EwA9SQDj9mHoD89Rz58h5eVzWxW2z8GyxEiKX5sStrqXiiK1I50R+uVDAUIHSvGHeGeL+8rDZ5 X-Gm-Message-State: AOJu0Yz4z7n0SnUaH6TkCu+69P6P2Vw9yH8hcLjCdx2KgXIR8ZxC7g5x 9yxfVFmwt/2hWAjYqw3bClqSmbQ8wxPWh+eebHuvnNxFQ5w7gxx7oU0hIdnx97unFMFp X-Google-Smtp-Source: AGHT+IGwOyvn+IjcJV9WoC0uqc9TVdMrZljGfip5NbPFBLB5b5fIZHNI4sP+MYpz3oKDJCbVHW9TSQ== X-Received: by 2002:a05:6808:1524:b0:3c3:88f3:96f2 with SMTP id u36-20020a056808152400b003c388f396f2mr6552697oiw.0.1710841302069; Tue, 19 Mar 2024 02:41:42 -0700 (PDT) Received: from KASONG-MB2.tencent.com ([43.132.141.21]) by smtp.gmail.com with ESMTPSA id i185-20020a62c1c2000000b006e583a649b4sm9708761pfg.210.2024.03.19.02.41.40 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Tue, 19 Mar 2024 02:41:41 -0700 (PDT) From: Kairui Song To: linux-mm@kvack.org Cc: Andrew Morton , Matthew Wilcox , linux-kernel@vger.kernel.org, Kairui Song Subject: [PATCH 1/4] mm/filemap: return early if failed to allocate memory for split Date: Tue, 19 Mar 2024 17:27:30 +0800 Message-ID: <20240319092733.4501-2-ryncsn@gmail.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240319092733.4501-1-ryncsn@gmail.com> References: <20240319092733.4501-1-ryncsn@gmail.com> Reply-To: Kairui Song Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Kairui Song xas_split_alloc could fail with NOMEM, and in such case, it should abort early instead of keep going and fail the xas_split below. Signed-off-by: Kairui Song Acked-by: Matthew Wilcox (Oracle) --- mm/filemap.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/mm/filemap.c b/mm/filemap.c index 7437b2bd75c1..f07ea0b97698 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -880,9 +880,12 @@ noinline int __filemap_add_folio(struct address_space = *mapping, unsigned int order =3D xa_get_order(xas.xa, xas.xa_index); void *entry, *old =3D NULL; =20 - if (order > folio_order(folio)) + if (order > folio_order(folio)) { xas_split_alloc(&xas, xa_load(xas.xa, xas.xa_index), order, gfp); + if (xas_error(&xas)) + goto error; + } xas_lock_irq(&xas); xas_for_each_conflict(&xas, entry) { old =3D entry; --=20 2.43.0 From nobody Sat Feb 7 19:08:19 2026 Received: from mail-oi1-f170.google.com (mail-oi1-f170.google.com [209.85.167.170]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1C5E47CF32 for ; Tue, 19 Mar 2024 09:41:44 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.167.170 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710841306; cv=none; b=V3wIWxAzC9P3lNcAhArYG3Szw9d/ijiu3P8bCQCG4EgsP1QMxhZScjQNw2z74tb7/rkehog0rx+Q515oFhc+3NYYbQa4Lem/dPBv3vP3yliGZ6Cqj1PKrh98jOt5q7PhCl8bvjICnJX51XEdFtWjqI5xM+4xx6/8nb02JwqUIWw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710841306; c=relaxed/simple; bh=DDs2rnx+E/mR/v1bFliS217+2gvtxPWAyB1Ek+cfMRY=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=eC1qB4dX/SiGY1HMT8ibBK8Diw+rfUg+JpjyvUKMi7h/KzrO9nEBsGOV4OUHuU7WntBOk5pHeXwFsGpkFEn/KpAStM9VhaMqgEF63GOB+cG8Yuw16kIkqtXzTbZjS2M3XG94/8IqwujbhQsj+xEtVBFHWsGe+n0L+2q1gOFr6/U= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=bgYzo7LQ; arc=none smtp.client-ip=209.85.167.170 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="bgYzo7LQ" Received: by mail-oi1-f170.google.com with SMTP id 5614622812f47-3c38d76384cso780487b6e.2 for ; Tue, 19 Mar 2024 02:41:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1710841304; x=1711446104; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:from:to:cc:subject :date:message-id:reply-to; bh=J34Qf2qKoII5Mgpe9QZtGvKa+BRn934GAZXO0p+CcPY=; b=bgYzo7LQDkBglL1zMQ3waOvR2ZJSL+EklGdid28ZDwkW5Ols6EusT2XIOYG9oznMAO RVrHVGg0jYyvsiGJdP6tJho1p3P6qlbQmjrrLvWCdy7Es6zMrwdbVTP/CP1CH8KBvTRF bZZrseAWf3BetFo1KCVmswMs0XXqqLgPw833sZy4hdyU91w3Dld6kkY/rvKQWMUjDd3c Jr6mM+J+6f37Mei9mw8tguJsITq7YYPMJIuuRW4ilNkXAX/+o+Zgyhy7GXwHRH3Q9u4h NqGqwJt4lbg+C3fgYV8clLDlbnXan6GTAWks3XUG0laU7t4+Is0KK4rNSvIUR4M5AGQl 70xA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1710841304; x=1711446104; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=J34Qf2qKoII5Mgpe9QZtGvKa+BRn934GAZXO0p+CcPY=; b=MbxE87wphr2dMDT2FaSLsohdvSwKYrSJTAn0aTbQMnV3RFgat7eF6yIgNW6TMx+qx3 YhYARsTsHD4/oFw+QXdgigzPtlzh9JSnW6uVbDAJ8BcigpisN3aFulaIJwNBJsQnX97s Mu970n9zo7F+LDNwA3L2MnonW866GjM2r591a8/vq984bDTWzfmKaAWK3Qa4gxvIVwP5 NqcZUa1rtMF2BxqeZ0HJVMuvwW8Xk60tzNDhzRKOD34Tco8HzQOpB9FHSkBXAl8T5Puw iuK5Lp5lYFem0TaKvpkxPzLEPmZMOx2UZqaa3Z3Yd1hYzPT+aSMbKpl+Ty5jvriVVJad 4lEw== X-Forwarded-Encrypted: i=1; AJvYcCXVj6sDCplRcC6/9Xeg4UwD/dbzupx0WDhnbje6VsdefMJfD7D7lc8f7md5KppL/Q2qvE0V8HrAhaEXPHU6RE914qfS2G4Ewno42sw2 X-Gm-Message-State: AOJu0YzjZ4Mkj9k/rrVXzH/F5vn+isnbnfuMyWYd7VmUwIUiGfttFpTL Vvle9yM+C3yYPpGNvQHpRqIrJd96aK3MFPv6y75FUyJzPEbz5+5s X-Google-Smtp-Source: AGHT+IEJHFnNbXQdn3Vy4EOryeIqwhnOxZinPVrn7nSGg3zBBB2/OolNOe72ReevQN6mJSPYvWaGpA== X-Received: by 2002:a05:6808:81:b0:3c2:616a:22 with SMTP id s1-20020a056808008100b003c2616a0022mr12406922oic.55.1710841304361; Tue, 19 Mar 2024 02:41:44 -0700 (PDT) Received: from KASONG-MB2.tencent.com ([43.132.141.21]) by smtp.gmail.com with ESMTPSA id i185-20020a62c1c2000000b006e583a649b4sm9708761pfg.210.2024.03.19.02.41.42 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Tue, 19 Mar 2024 02:41:43 -0700 (PDT) From: Kairui Song To: linux-mm@kvack.org Cc: Andrew Morton , Matthew Wilcox , linux-kernel@vger.kernel.org, Kairui Song Subject: [PATCH 2/4] mm/filemap: clean up hugetlb exclusion code Date: Tue, 19 Mar 2024 17:27:31 +0800 Message-ID: <20240319092733.4501-3-ryncsn@gmail.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240319092733.4501-1-ryncsn@gmail.com> References: <20240319092733.4501-1-ryncsn@gmail.com> Reply-To: Kairui Song Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Kairui Song __filemap_add_folio only has two callers, one never passes hugetlb folio and one always passes in hugetlb folio. So move the hugetlb related cgroup charging out of it to make the code cleaner. Signed-off-by: Kairui Song Acked-by: Matthew Wilcox (Oracle) --- mm/filemap.c | 21 ++++++++------------- 1 file changed, 8 insertions(+), 13 deletions(-) diff --git a/mm/filemap.c b/mm/filemap.c index f07ea0b97698..6bbec8783793 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -853,20 +853,12 @@ noinline int __filemap_add_folio(struct address_space= *mapping, { XA_STATE(xas, &mapping->i_pages, index); bool huge =3D folio_test_hugetlb(folio); - bool charged =3D false; - long nr =3D 1; + long nr; =20 VM_BUG_ON_FOLIO(!folio_test_locked(folio), folio); VM_BUG_ON_FOLIO(folio_test_swapbacked(folio), folio); mapping_set_update(&xas, mapping); =20 - if (!huge) { - int error =3D mem_cgroup_charge(folio, NULL, gfp); - if (error) - return error; - charged =3D true; - } - VM_BUG_ON_FOLIO(index & (folio_nr_pages(folio) - 1), folio); xas_set_order(&xas, index, folio_order(folio)); nr =3D folio_nr_pages(folio); @@ -931,8 +923,6 @@ noinline int __filemap_add_folio(struct address_space *= mapping, trace_mm_filemap_add_to_page_cache(folio); return 0; error: - if (charged) - mem_cgroup_uncharge(folio); folio->mapping =3D NULL; /* Leave page->index set: truncation relies upon it */ folio_put_refs(folio, nr); @@ -946,11 +936,16 @@ int filemap_add_folio(struct address_space *mapping, = struct folio *folio, void *shadow =3D NULL; int ret; =20 + ret =3D mem_cgroup_charge(folio, NULL, gfp); + if (ret) + return ret; + __folio_set_locked(folio); ret =3D __filemap_add_folio(mapping, folio, index, gfp, &shadow); - if (unlikely(ret)) + if (unlikely(ret)) { + mem_cgroup_uncharge(folio); __folio_clear_locked(folio); - else { + } else { /* * The folio might have been evicted from cache only * recently, in which case it should be activated like --=20 2.43.0 From nobody Sat Feb 7 19:08:19 2026 Received: from mail-oi1-f172.google.com (mail-oi1-f172.google.com [209.85.167.172]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5493A7E108 for ; Tue, 19 Mar 2024 09:41:47 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.167.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710841308; cv=none; b=T/55pWJz9TB8CBoSEKW9S9MQ60yFzlxFYmgySj/bgz0SCoIgwu9ET9GN/UGH/wq10bKmaQkxzfgfTyr+qZE0Rm6RELmX7DApuDRxkHMNeYo7nkZlo4TLJ2WKkzL68NwFz7dAYqfvd5dbnG6Svf6tHsurEkYF5ZXtIFirxj59C9k= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710841308; c=relaxed/simple; bh=mVMJQpmryZm4Ru99nD32KvjTOYfibZ/pE4Lu/SnCZ2o=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=CKZydMT0E1XkOIjBifwKIN1rI1V5xFfoLdGYEyQcUOLR9no5qckqY86KGCtHF+waGRq0UIzJLQROwZ/BmaIsHsOI/CgCCtxdiA+tdUvosRYNHrRPBbyLB9UCyvuTTC1Wnvwl2f+DS9mj5l8sOF+e2yNwHj1wLo9hSy+dMMQ0qss= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=klM9Doxj; arc=none smtp.client-ip=209.85.167.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="klM9Doxj" Received: by mail-oi1-f172.google.com with SMTP id 5614622812f47-3c38f4e18eeso768850b6e.1 for ; Tue, 19 Mar 2024 02:41:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1710841306; x=1711446106; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:from:to:cc:subject :date:message-id:reply-to; bh=XqzN+wWZGV0/ntTi34b/frY6opMs0aB0oKSPnzSGTOk=; b=klM9DoxjF2Ehc7z0FLxAU/zfbaQjuRtkTMcp55NoMe/z/GPD0ocTygxKmwM7wLQhxT rtyN87Mrpo7HVA3DKBbMYFvImpeBt92+RKo+T+3509S3OzsruVZgMfQmHCj5qJXq1igK tvFLQFr1XnvaDgOs19AMsfaJqs4t/3bom58CuDsDsWWnZWTxf2+AdZeqMYMsGlTVF+cM 5xspbkap/EGedOiGWsdsVqs9al5NXVsGfNQolgLkohanRsVdN59ylHRH9C+25T2RLOYX +GgbEw16cYJJRwKMFQQZMAh3rEk+3Bhjg4O5vxc1OcSoGg6zjmv9yqgx5Z3a6mcZb+gm HjoQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1710841306; x=1711446106; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=XqzN+wWZGV0/ntTi34b/frY6opMs0aB0oKSPnzSGTOk=; b=Z1Ofm+62QLdL8VRfTHCPKSLvjW/KuBlDRU55kGrRCDwVx2FEQaxRvJ6kwP//PpxZYu r4VFkE/MJEM/yjNRix+8eu/ju0FJA2bW4RPDyUSjGq4muRcDh5YErCOBCakKeH6nJhS+ Pe7U4IFioh2G7uEwRUzBhhhV0bOBQvCSo+3FIovwK7/dhD2sGgb7C1pyu1cZfVJbqpGw 2xw7FOb+vVGxHw1KA836QCktIMKXKwVkeb0omc3LaeA80Vg+r3RCLs3NEjuAEitRYM2Z X3TzVEzoU9xg5aQxPl/LxDJcZwbtgGAx8xG2XRpldy8XgIevz7/sqAEmYFSuI5WwLPVV x9ug== X-Forwarded-Encrypted: i=1; AJvYcCW3XhJAk3WqqmazW28y/hcXfXTipOq+BlwXK5+tJGyX3rNaJsTMpIxJ0RcExdQ0GYaj1QoIkx6t5QO7LHr2Pwz548+AlFPLdbcHU9Mx X-Gm-Message-State: AOJu0Yyc7BgR3NRPROVpcnGc55j6j4HUSV86EXRo6TABrxu9XHF7qa+0 cukYSYLAZHcldKq1MQwo5+n279sHPx5DnH/4c1l8GXjjDbVd12P1 X-Google-Smtp-Source: AGHT+IFpWzLUcgwPhIA7GrafY8so+K48aVsYG/cukRTuKwm2t4oZkaJoNlfObveyt98ye1pVmlIUsQ== X-Received: by 2002:a05:6808:180b:b0:3c3:82c3:ddcc with SMTP id bh11-20020a056808180b00b003c382c3ddccmr2678100oib.31.1710841306463; Tue, 19 Mar 2024 02:41:46 -0700 (PDT) Received: from KASONG-MB2.tencent.com ([43.132.141.21]) by smtp.gmail.com with ESMTPSA id i185-20020a62c1c2000000b006e583a649b4sm9708761pfg.210.2024.03.19.02.41.44 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Tue, 19 Mar 2024 02:41:46 -0700 (PDT) From: Kairui Song To: linux-mm@kvack.org Cc: Andrew Morton , Matthew Wilcox , linux-kernel@vger.kernel.org, Kairui Song Subject: [PATCH 3/4] lib/xarray: introduce a new helper xas_get_order Date: Tue, 19 Mar 2024 17:27:32 +0800 Message-ID: <20240319092733.4501-4-ryncsn@gmail.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240319092733.4501-1-ryncsn@gmail.com> References: <20240319092733.4501-1-ryncsn@gmail.com> Reply-To: Kairui Song Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Kairui Song It can be used after xas_load to check the order of loaded entries. Compared to xa_get_order, it saves an XA_STATE and avoid a rewalk. Signed-off-by: Kairui Song Acked-by: Matthew Wilcox (Oracle) --- include/linux/xarray.h | 6 ++++++ lib/xarray.c | 49 ++++++++++++++++++++++++++---------------- 2 files changed, 37 insertions(+), 18 deletions(-) diff --git a/include/linux/xarray.h b/include/linux/xarray.h index cb571dfcf4b1..d9d479334c9e 100644 --- a/include/linux/xarray.h +++ b/include/linux/xarray.h @@ -1548,6 +1548,7 @@ void xas_create_range(struct xa_state *); =20 #ifdef CONFIG_XARRAY_MULTI int xa_get_order(struct xarray *, unsigned long index); +int xas_get_order(struct xa_state *xas); void xas_split(struct xa_state *, void *entry, unsigned int order); void xas_split_alloc(struct xa_state *, void *entry, unsigned int order, g= fp_t); #else @@ -1556,6 +1557,11 @@ static inline int xa_get_order(struct xarray *xa, un= signed long index) return 0; } =20 +static inline int xas_get_order(struct xa_state *xas) +{ + return 0; +} + static inline void xas_split(struct xa_state *xas, void *entry, unsigned int order) { diff --git a/lib/xarray.c b/lib/xarray.c index 39f07bfc4dcc..88fbea481e1b 100644 --- a/lib/xarray.c +++ b/lib/xarray.c @@ -1750,39 +1750,52 @@ void *xa_store_range(struct xarray *xa, unsigned lo= ng first, EXPORT_SYMBOL(xa_store_range); =20 /** - * xa_get_order() - Get the order of an entry. - * @xa: XArray. - * @index: Index of the entry. + * xas_get_order() - Get the order of an loaded entry after xas_load. + * @xas: XArray operation state. + * + * Called after xas_load, the xas should not be in an error state. * * Return: A number between 0 and 63 indicating the order of the entry. */ -int xa_get_order(struct xarray *xa, unsigned long index) +int xas_get_order(struct xa_state *xas) { - XA_STATE(xas, xa, index); - void *entry; int order =3D 0; =20 - rcu_read_lock(); - entry =3D xas_load(&xas); - - if (!entry) - goto unlock; - - if (!xas.xa_node) - goto unlock; + if (!xas->xa_node) + return 0; =20 for (;;) { - unsigned int slot =3D xas.xa_offset + (1 << order); + unsigned int slot =3D xas->xa_offset + (1 << order); =20 if (slot >=3D XA_CHUNK_SIZE) break; - if (!xa_is_sibling(xas.xa_node->slots[slot])) + if (!xa_is_sibling(xas->xa_node->slots[slot])) break; order++; } =20 - order +=3D xas.xa_node->shift; -unlock: + order +=3D xas->xa_node->shift; + return order; +} +EXPORT_SYMBOL(xas_get_order); + +/** + * xa_get_order() - Get the order of an entry. + * @xa: XArray. + * @index: Index of the entry. + * + * Return: A number between 0 and 63 indicating the order of the entry. + */ +int xa_get_order(struct xarray *xa, unsigned long index) +{ + XA_STATE(xas, xa, index); + int order =3D 0; + void *entry; + + rcu_read_lock(); + entry =3D xas_load(&xas); + if (entry) + order =3D xas_get_order(&xas); rcu_read_unlock(); =20 return order; --=20 2.43.0 From nobody Sat Feb 7 19:08:19 2026 Received: from mail-pf1-f181.google.com (mail-pf1-f181.google.com [209.85.210.181]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4588A7E564 for ; Tue, 19 Mar 2024 09:41:49 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.181 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710841310; cv=none; b=LxxzAXvdRWx88Eh+X7xH4UfhtdTfLQTpbDccgghWcK4qcgxymDyauc8e0xh/a/pA6AmEv9BUzlTufY3JgJw8sRv7H+TZ/d2/Hf2RoZJkaGi4qQbPZoio5CNGymeMN45MoXhz0EP0YOAT3txIIFD/wWwtwUlYQk2wtmCcgTptHDU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710841310; c=relaxed/simple; bh=Q4MsXeGz7gjwSGkOMZ96Ds7ueS0MczucwPZ2yMN5ZlQ=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=WBqn9OqwA/QRZjcF1+4noe+ZE3hS2WMRzPipfiry8Y5WA270w3HzRalSgVdKGdkHehE335fiOPmL8ZYXQMQqMzuonNkw7Y9IlWoMRV8TSm+KWFGrfpOEkCaH7fnxAvgr0HLPWlVID1Z3n7T73vYEM1RymzyLcCxXEz1DnTxSyVU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=L6W15cES; arc=none smtp.client-ip=209.85.210.181 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="L6W15cES" Received: by mail-pf1-f181.google.com with SMTP id d2e1a72fcca58-6e73e8bdea2so890818b3a.0 for ; Tue, 19 Mar 2024 02:41:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1710841308; x=1711446108; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:from:to:cc:subject :date:message-id:reply-to; bh=+IPrpGzoHpDGCt5ZLHs8DWvQZcRMV24wjwQMoVjv/9U=; b=L6W15cESoM73hfcFlhUFEryUyBW6KiP8toxFpsdApH2pwoWtmiyb4NP3v91gsCmgip 3662Ds+zatXKoyIbhMdfIQ6QmwZsvhdwuNuDyeKaZVgZyNhXKZkOUoxneTaspgbtnAsX Nbus7JgBHGWSkZOt6uS+HpYeFJ93mQ/8BVdXm8IqWKoDn1ewp1CuZ/Y/zYnuhWD5TP8j Q36aOZadM+4eiKeA21lk5qYTinA9uuH2f59oiuuHO4gqTwR+FT3Hd1KkQT97GkYiUS49 EsGtT/hJKQrO++C4hc4xIIJAGVRrZCYBSsJVFbTYOKu4feVTgcwZwvIOAdQSkqEdW7s3 qudA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1710841308; x=1711446108; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=+IPrpGzoHpDGCt5ZLHs8DWvQZcRMV24wjwQMoVjv/9U=; b=Gx9ol9fMr40CKUefnhYyDqp8XwapQ3KkGbzHwkHCB+qIXaNzmpVSu6OAO0QS9tZu/B m3OmahA1aVQWQePNFRoaC/O3MB+ZRybx9ERKk9ZeLd8IZVSaKT1tNTxkAmm9fTeCeWwK xLhR0H8Qs3YpKZLNea6cAs/+SjkCaj11sFx/0tiEY7i4Vvvyf5XD6sk/YZaVVB+Z29gs 6ilmIIuj9QBNT84IT6Skl+TLGGUk0T6izLqs/1gaQAnOHkKVvaigHOsLiHAR6Z7QS5ND beDI6gM9HXMFGuYh1cPr7vYpswQ6G3L/mow1/c8vmqIndLW/thWdEZjm/+WpsPAUO/yr 6/xw== X-Forwarded-Encrypted: i=1; AJvYcCXD/Z2XiUMI5MrZBn3XKzTILWyJMRvgMYR6C/8up2AJw1Dy+yPVt/7JCplS2cw/SXFC2e6fmCLM/g2SbZXGMPeJ8qbgKBAAEPK4vN0r X-Gm-Message-State: AOJu0Yx4db1Gq6O22Z1N0uVkLRCwmNLSTUS7Q5mNWQThGOb3vkB6uA1a u+5oMYTn+6vqxs16nt56Gx8Sr387Wo9a4/ST2ip+QYq2NppEM0E6 X-Google-Smtp-Source: AGHT+IH9clmpXo+eDYRjLfrDjT4vum1cmIxLp7ORYVYCH84rdbd4eGTduRgvmmpEsgUtIio/VDV/8g== X-Received: by 2002:a05:6a00:9389:b0:6e6:a1ff:3661 with SMTP id ka9-20020a056a00938900b006e6a1ff3661mr19465566pfb.31.1710841308559; Tue, 19 Mar 2024 02:41:48 -0700 (PDT) Received: from KASONG-MB2.tencent.com ([43.132.141.21]) by smtp.gmail.com with ESMTPSA id i185-20020a62c1c2000000b006e583a649b4sm9708761pfg.210.2024.03.19.02.41.46 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Tue, 19 Mar 2024 02:41:48 -0700 (PDT) From: Kairui Song To: linux-mm@kvack.org Cc: Andrew Morton , Matthew Wilcox , linux-kernel@vger.kernel.org, Kairui Song Subject: [PATCH 4/4] mm/filemap: optimize filemap folio adding Date: Tue, 19 Mar 2024 17:27:33 +0800 Message-ID: <20240319092733.4501-5-ryncsn@gmail.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240319092733.4501-1-ryncsn@gmail.com> References: <20240319092733.4501-1-ryncsn@gmail.com> Reply-To: Kairui Song Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Kairui Song Instead of doing multiple tree walks, do one optimism range check with lock hold, and exit if raced with another insertion. If a shadow exists, check it with a new xas_get_order helper before releasing the lock to avoid redundant tree walks for getting its order. Drop the lock and do the allocation only if a split is needed. In the best case, it only need to walk the tree once. If it needs to alloc and split, 3 walks are issued (One for first ranced conflict check and order retrieving, one for the second check after allocation, one for the insert after split). Testing with 4k pages, in an 8G cgroup, with 20G brd as block device: fio -name=3Dcached --numjobs=3D16 --filename=3D/mnt/test.img \ --buffered=3D1 --ioengine=3Dmmap --rw=3Drandread --time_based \ --ramp_time=3D30s --runtime=3D5m --group_reporting Before: bw ( MiB/s): min=3D 790, max=3D 3665, per=3D100.00%, avg=3D2499.17, stdev= =3D20.64, samples=3D8698 iops : min=3D202295, max=3D938417, avg=3D639785.81, stdev=3D5284.08,= samples=3D8698 After (+4%): bw ( MiB/s): min=3D 451, max=3D 3868, per=3D100.00%, avg=3D2599.83, stdev= =3D23.39, samples=3D8653 iops : min=3D115596, max=3D990364, avg=3D665556.34, stdev=3D5988.20,= samples=3D8653 Test result with THP (do a THP randread then switch to 4K page in hope it issues a lot of splitting): fio -name=3Dcached --numjobs=3D16 --filename=3D/mnt/test.img \ --buffered=3D1 --ioengine mmap -thp=3D1 --readonly \ --rw=3Drandread --random_distribution=3Drandom \ --time_based --runtime=3D5m --group_reporting fio -name=3Dcached --numjobs=3D16 --filename=3D/mnt/test.img \ --buffered=3D1 --ioengine mmap --readonly \ --rw=3Drandread --random_distribution=3Drandom \ --time_based --runtime=3D5s --group_reporting Before: bw ( KiB/s): min=3D28071, max=3D62359, per=3D100.00%, avg=3D53542.44, stde= v=3D179.77, samples=3D9520 iops : min=3D 7012, max=3D15586, avg=3D13379.39, stdev=3D44.94, samp= les=3D9520 bw ( MiB/s): min=3D 2457, max=3D 6193, per=3D100.00%, avg=3D3923.21, stdev= =3D82.48, samples=3D144 iops : min=3D629220, max=3D1585642, avg=3D1004340.78, stdev=3D21116.= 07, samples=3D144 After (+-0.0%): bw ( KiB/s): min=3D30561, max=3D63064, per=3D100.00%, avg=3D53635.82, stde= v=3D177.21, samples=3D9520 iops : min=3D 7636, max=3D15762, avg=3D13402.82, stdev=3D44.29, samp= les=3D9520 bw ( MiB/s): min=3D 2449, max=3D 6145, per=3D100.00%, avg=3D3914.68, stdev= =3D81.15, samples=3D144 iops : min=3D627106, max=3D1573156, avg=3D1002158.11, stdev=3D20774.= 77, samples=3D144 The performance is better (+4%) for 4K cached read and unchanged for THP. Signed-off-by: Kairui Song --- mm/filemap.c | 127 ++++++++++++++++++++++++++++++--------------------- 1 file changed, 76 insertions(+), 51 deletions(-) diff --git a/mm/filemap.c b/mm/filemap.c index 6bbec8783793..c1484bcdbddb 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -848,12 +848,77 @@ void replace_page_cache_folio(struct folio *old, stru= ct folio *new) } EXPORT_SYMBOL_GPL(replace_page_cache_folio); =20 +static int __split_add_folio_locked(struct xa_state *xas, struct folio *fo= lio, + pgoff_t index, gfp_t gfp, void **shadowp) +{ + void *entry, *shadow, *alloced_shadow =3D NULL; + int order, alloced_order =3D 0; + + gfp &=3D GFP_RECLAIM_MASK; + for (;;) { + shadow =3D NULL; + order =3D 0; + + xas_for_each_conflict(xas, entry) { + if (!xa_is_value(entry)) + return -EEXIST; + shadow =3D entry; + } + + if (shadow) { + if (shadow =3D=3D xas_reload(xas)) { + order =3D xas_get_order(xas); + if (order && order > folio_order(folio)) { + /* entry may have been split before we acquired lock */ + if (shadow !=3D alloced_shadow || order !=3D alloced_order) + goto unlock; + xas_split(xas, shadow, order); + xas_reset(xas); + } + order =3D 0; + } + if (shadowp) + *shadowp =3D shadow; + } + + xas_store(xas, folio); + /* Success, return with mapping locked */ + if (!xas_error(xas)) + return 0; +unlock: + /* + * Unlock path, if errored, return unlocked. + * If allocation needed, alloc and retry. + */ + xas_unlock_irq(xas); + if (order) { + if (unlikely(alloced_order)) + xas_destroy(xas); + xas_split_alloc(xas, shadow, order, gfp); + if (!xas_error(xas)) { + alloced_shadow =3D shadow; + alloced_order =3D order; + } + goto next; + } + /* xas_nomem result checked by xas_error below */ + xas_nomem(xas, gfp); +next: + xas_lock_irq(xas); + if (xas_error(xas)) + return xas_error(xas); + + xas_reset(xas); + } +} + noinline int __filemap_add_folio(struct address_space *mapping, struct folio *folio, pgoff_t index, gfp_t gfp, void **shadowp) { XA_STATE(xas, &mapping->i_pages, index); bool huge =3D folio_test_hugetlb(folio); long nr; + int ret; =20 VM_BUG_ON_FOLIO(!folio_test_locked(folio), folio); VM_BUG_ON_FOLIO(folio_test_swapbacked(folio), folio); @@ -863,70 +928,30 @@ noinline int __filemap_add_folio(struct address_space= *mapping, xas_set_order(&xas, index, folio_order(folio)); nr =3D folio_nr_pages(folio); =20 - gfp &=3D GFP_RECLAIM_MASK; folio_ref_add(folio, nr); folio->mapping =3D mapping; folio->index =3D xas.xa_index; =20 - do { - unsigned int order =3D xa_get_order(xas.xa, xas.xa_index); - void *entry, *old =3D NULL; - - if (order > folio_order(folio)) { - xas_split_alloc(&xas, xa_load(xas.xa, xas.xa_index), - order, gfp); - if (xas_error(&xas)) - goto error; - } - xas_lock_irq(&xas); - xas_for_each_conflict(&xas, entry) { - old =3D entry; - if (!xa_is_value(entry)) { - xas_set_err(&xas, -EEXIST); - goto unlock; - } - } - - if (old) { - if (shadowp) - *shadowp =3D old; - /* entry may have been split before we acquired lock */ - order =3D xa_get_order(xas.xa, xas.xa_index); - if (order > folio_order(folio)) { - /* How to handle large swap entries? */ - BUG_ON(shmem_mapping(mapping)); - xas_split(&xas, old, order); - xas_reset(&xas); - } - } - - xas_store(&xas, folio); - if (xas_error(&xas)) - goto unlock; - + xas_lock_irq(&xas); + ret =3D __split_add_folio_locked(&xas, folio, index, gfp, shadowp); + if (likely(!ret)) { mapping->nrpages +=3D nr; - - /* hugetlb pages do not participate in page cache accounting */ if (!huge) { __lruvec_stat_mod_folio(folio, NR_FILE_PAGES, nr); if (folio_test_pmd_mappable(folio)) __lruvec_stat_mod_folio(folio, NR_FILE_THPS, nr); } -unlock: xas_unlock_irq(&xas); - } while (xas_nomem(&xas, gfp)); - - if (xas_error(&xas)) - goto error; + trace_mm_filemap_add_to_page_cache(folio); + } else { + xas_unlock_irq(&xas); + folio->mapping =3D NULL; + /* Leave page->index set: truncation relies upon it */ + folio_put_refs(folio, nr); + } =20 - trace_mm_filemap_add_to_page_cache(folio); - return 0; -error: - folio->mapping =3D NULL; - /* Leave page->index set: truncation relies upon it */ - folio_put_refs(folio, nr); - return xas_error(&xas); + return ret; } ALLOW_ERROR_INJECTION(__filemap_add_folio, ERRNO); =20 --=20 2.43.0