From nobody Fri Feb 13 09:49:17 2026 Received: from mail-pg1-f173.google.com (mail-pg1-f173.google.com [209.85.215.173]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A8575839E5 for ; Mon, 15 Apr 2024 17:19:14 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.215.173 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1713201555; cv=none; b=cZmZJNomMCfpAQ9MuThKXoRrQyVfXfGdy9neKjTCiC3j2fdut9HFh74XrghabgJB++/nSRcdwPfXMKrsR/EYtKE1U0JIr1JSWkhzAkNN6mrJZpKhqSn3HdnmiaAxXGt3EhNpVsuw/SnkUQ3ip5Z6zNUoU/KsS+tdqSwKg5O2hWE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1713201555; c=relaxed/simple; bh=lLIOvfVm18UTpicKdH4FOcN1juZAjvD4d0PBeYxJtIA=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=YOjA3jKwLK8Z7zqQXLKw+zOWB1jmZtvxc+2QeA409hqzyK6/vUbDOgg/Ayy+y9mJB7grTIlBQbp+DpJ/jPaV6PYB77GLcHrIbF8Y8U0u7nBvuJDfi40oo6kADxctGKfpAOY1YyiyVx6+fKsJ9p6c7qPUjpn5zRh0sAF9qQEOsFs= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=IEycy8+y; arc=none smtp.client-ip=209.85.215.173 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="IEycy8+y" Received: by mail-pg1-f173.google.com with SMTP id 41be03b00d2f7-5e4613f2b56so2209752a12.1 for ; Mon, 15 Apr 2024 10:19:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1713201554; x=1713806354; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:from:to:cc:subject :date:message-id:reply-to; bh=B+L8KNGbsq36FmjcGvQ6kYVRELiG3WmiEI53/0190Sw=; b=IEycy8+ydYC6Y4VMMbnJwWRyIxAOQyuRULZ+1PBRGcqNYa85G7iDk0WsdIbUEqxO6k cTtlHOmgya4T4BDoNaGgjfSXsCnaHtDtcsrPBIbWUjis4TG/sOcqHaul5G1He5EMTErh wWt6yJ79JEuUgbkQ/ErO+fPAkh7ACqSNaWCuIo6E9qKy284oqHOhJhQjTqW0mwo1f1k0 a1vwasgf86tOEz/fBTJdE1iC8nWOTCPUAanF751tuRb3VPwJgGVWn9fkVr0YnQ0RTrLT pmizCagyR+scK4h6tiyVfV2YKSejDNaEPq6rs1PkmY5tfFLW3Qi85PF1t51SQ/ST0kUH h7xA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1713201554; x=1713806354; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=B+L8KNGbsq36FmjcGvQ6kYVRELiG3WmiEI53/0190Sw=; b=ubQXXN6GBFPYMuoax4c4n394g/7r389GAPAnW45Vks8Oxh2cHVa2dPUtNeBm0MlpgC Ze7GJ5mNMX4c80rubLI7s/nfmBgFwfvppmKBfLp1McYdJ9uwmwyikFYc+ct+L9f9cVo4 c/NENsCyTqwfAbJI5Acv9/lTq51MeqqqSXHGJ7gwcE6AdWgCvz8ZrsQfq54lvr8FQM+3 9WFXk7ES2Qi942R//pXJ8MEuq9uhIgTHHaUy4f/VYkirCR8ND8BF0rOwPLG/jSqIiFie tgfsRL+AGLtDOyBeeW2YihuH/z3bJ3Cp0vrx85LUOFFJPCKT/UECXS5dxhqHTziWy/0u jcsw== X-Forwarded-Encrypted: i=1; AJvYcCWtnskAnWJIISPHaDpr7qwfYCzKgUTefVu1Kl2anRq55N6lVZ8A/6LwA9CJgHkx+AKA1QUvjBlTBiBzXN+XlLbdK1KrMV9/xnsb5fqs X-Gm-Message-State: AOJu0YykTiN6GVMY0YWGfbvUobmPJXB6reSw1Q8KuVQXr1i7DzYlCXHg /Mc0wKEZ4cTq/kACHdT+tDf4sLwRd49LW+Jl5I7Z86sZN+KmIOy8 X-Google-Smtp-Source: AGHT+IGZsDnsN/0G69F9r9pl4nCedE8as+ni4AOnw741cJ5yApDTX6VIYEFjdIjzcaV4wIkNMeB2jQ== X-Received: by 2002:a17:90a:d998:b0:2a2:b096:f1b2 with SMTP id d24-20020a17090ad99800b002a2b096f1b2mr7627869pjv.32.1713201553844; Mon, 15 Apr 2024 10:19:13 -0700 (PDT) Received: from localhost.localdomain ([101.32.222.185]) by smtp.gmail.com with ESMTPSA id o19-20020a17090aac1300b002a537abb536sm8751648pjq.57.2024.04.15.10.19.11 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Mon, 15 Apr 2024 10:19:13 -0700 (PDT) From: Kairui Song To: linux-mm@kvack.org Cc: Matthew Wilcox , Andrew Morton , linux-kernel@vger.kernel.org, Kairui Song Subject: [PATCH v3 1/4] mm/filemap: return early if failed to allocate memory for split Date: Tue, 16 Apr 2024 01:18:53 +0800 Message-ID: <20240415171857.19244-2-ryncsn@gmail.com> X-Mailer: git-send-email 2.44.0 In-Reply-To: <20240415171857.19244-1-ryncsn@gmail.com> References: <20240415171857.19244-1-ryncsn@gmail.com> Reply-To: Kairui Song Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Kairui Song xas_split_alloc could fail with NOMEM, and in such case, it should abort early instead of keep going and fail the xas_split below. Signed-off-by: Kairui Song Acked-by: Matthew Wilcox (Oracle) --- mm/filemap.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/mm/filemap.c b/mm/filemap.c index 480ae6589803..12089c24abfb 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -880,9 +880,12 @@ noinline int __filemap_add_folio(struct address_space = *mapping, unsigned int order =3D xa_get_order(xas.xa, xas.xa_index); void *entry, *old =3D NULL; =20 - if (order > folio_order(folio)) + if (order > folio_order(folio)) { xas_split_alloc(&xas, xa_load(xas.xa, xas.xa_index), order, gfp); + if (xas_error(&xas)) + goto error; + } xas_lock_irq(&xas); xas_for_each_conflict(&xas, entry) { old =3D entry; --=20 2.44.0 From nobody Fri Feb 13 09:49:17 2026 Received: from mail-pg1-f173.google.com (mail-pg1-f173.google.com [209.85.215.173]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B7ECD83CBD for ; Mon, 15 Apr 2024 17:19:16 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.215.173 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1713201558; cv=none; b=rtyLLWXngYwBK6HY8ngOYz4+fd7UFg2ggiB5+oql8pQf53uMgS0BjZrjGCgKTP0E32g8Ir8yH76NXr6drUqPR0jEGL0cgfRvwdJkaEseKLqda0UyHwY89kgckNWLoh1FILtua1yBWoqLPuoG5lcdF/G6rxRRC5SYlYIu/3cBCWo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1713201558; c=relaxed/simple; bh=bZhwZeYrlzAcy5RHy9jWTP4UjDiBbNQ8TWxMoaetbvQ=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=CzikxVQVvz5m1imYBNlUMS3XQAIhXTcadzSCHYN3/qru18sjDm+DEaJtZK1QV0QScAQkGX0QdgCgR8SYx8Vgv9Ty/mQp59/XOpz8MiHac9LaSXNDEsexdKraEBT5GaCJ+FGZw6F1jtwAeS1zKKO3vUQ5pUE4TkBWrayHqNcOc4E= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=RULvmBfJ; arc=none smtp.client-ip=209.85.215.173 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="RULvmBfJ" Received: by mail-pg1-f173.google.com with SMTP id 41be03b00d2f7-5e8470c1cb7so2281870a12.2 for ; Mon, 15 Apr 2024 10:19:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1713201556; x=1713806356; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:from:to:cc:subject :date:message-id:reply-to; bh=x5lu4xKhCQ/0haQPCj7eyNWwUk1BrbeMvC9dF21zPVk=; b=RULvmBfJ2xIeU7e3CPYXj3Awmy9vgLBTmnvE+od6mQwIa3NhetuhzaabRMK0LiNYXp Urdpx7McvjcbH4+LEfpGUd08bq9RAdFvFJJHANL0uWQdG7EsH/SENaAPXhsoxD9wwoa2 Cuk5bx2rBkcglnzIrt6i2ooFDQU1kl8mbipjBdPwpA9GYOoyLzp20g96ojvyWPNsv+7t 7PDIfCVZoqTxCLXsixo0jB4ee9UWtHBD2u+x/4XDdVwlctbyDjxzPa2erRyPM6z4Qm9v PWmpwnCQ3wjuBfHaI8kURVvCyBkzs91xK4aLZtVj7ZiB3FpsivYLcYoVe917ZIOMKrpz Lzlg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1713201556; x=1713806356; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=x5lu4xKhCQ/0haQPCj7eyNWwUk1BrbeMvC9dF21zPVk=; b=EXjIUU7nyKZC9nmqBIu8rBNBFDgTG4+qBLuOaJTnv2dLsJFxdXagXxVjJPrJwYEbsb G9rpv8nRlC1uCsqckYAegYxWM3Rh74UxG2VoqUDF2L1XXh7U1JGsusW79mG3f44EER7e VjvFHmdOYfDoqufFB+JbHNtxO7sp+JZ/jKvV9B6tnoBl89ttJPNmXXCiAtUzDHGFx3/e qcR+IwaC0Vl2CIJQmNpJ/MXxp7GyP4+vnq8hrRPmdU5xeJsfWILGzWX+AwymvzNwLA2M Zffriz7oNyP0q6qjVwB8VPKBzClMK+7YRwG11ObKR5F8pFBY3xX4xy/uBrurj/VPlEg9 4nRQ== X-Forwarded-Encrypted: i=1; AJvYcCVmkHUcQRO2HpOZSiFP+lO+a8yU+CNjP2DxQITpTTCgq6YHK/pvvS9DHb/GNN0eeBUxFmodERHLLosSIw4qLx65yC91H/kot3yVewcS X-Gm-Message-State: AOJu0YztTxDgaD5QZWm6pkU9AaHsTFUSatBrR6OKBUeZUOBfSGbec58q i0SqSmh0/etLtHJM83Jdb7VhODRuvo+926gYv/mf035GyYmTMCb/ X-Google-Smtp-Source: AGHT+IHCSW+MgtEmdHVUti9+WxD2hWCD0NmfCyU2sjka1LkmR7bv5TKrJwBuID2fpVaAEzlYPQwGNg== X-Received: by 2002:a17:90a:e50c:b0:2a2:7817:f591 with SMTP id t12-20020a17090ae50c00b002a27817f591mr7838981pjy.48.1713201555986; Mon, 15 Apr 2024 10:19:15 -0700 (PDT) Received: from localhost.localdomain ([101.32.222.185]) by smtp.gmail.com with ESMTPSA id o19-20020a17090aac1300b002a537abb536sm8751648pjq.57.2024.04.15.10.19.14 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Mon, 15 Apr 2024 10:19:15 -0700 (PDT) From: Kairui Song To: linux-mm@kvack.org Cc: Matthew Wilcox , Andrew Morton , linux-kernel@vger.kernel.org, Kairui Song Subject: [PATCH v3 2/4] mm/filemap: clean up hugetlb exclusion code Date: Tue, 16 Apr 2024 01:18:54 +0800 Message-ID: <20240415171857.19244-3-ryncsn@gmail.com> X-Mailer: git-send-email 2.44.0 In-Reply-To: <20240415171857.19244-1-ryncsn@gmail.com> References: <20240415171857.19244-1-ryncsn@gmail.com> Reply-To: Kairui Song Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Kairui Song __filemap_add_folio only has two callers, one never passes hugetlb folio and one always passes in hugetlb folio. So move the hugetlb related cgroup charging out of it to make the code cleaner. Signed-off-by: Kairui Song Acked-by: Matthew Wilcox (Oracle) --- mm/filemap.c | 21 ++++++++------------- 1 file changed, 8 insertions(+), 13 deletions(-) diff --git a/mm/filemap.c b/mm/filemap.c index 12089c24abfb..17a66ea544e7 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -853,20 +853,12 @@ noinline int __filemap_add_folio(struct address_space= *mapping, { XA_STATE(xas, &mapping->i_pages, index); bool huge =3D folio_test_hugetlb(folio); - bool charged =3D false; - long nr =3D 1; + long nr; =20 VM_BUG_ON_FOLIO(!folio_test_locked(folio), folio); VM_BUG_ON_FOLIO(folio_test_swapbacked(folio), folio); mapping_set_update(&xas, mapping); =20 - if (!huge) { - int error =3D mem_cgroup_charge(folio, NULL, gfp); - if (error) - return error; - charged =3D true; - } - VM_BUG_ON_FOLIO(index & (folio_nr_pages(folio) - 1), folio); xas_set_order(&xas, index, folio_order(folio)); nr =3D folio_nr_pages(folio); @@ -931,8 +923,6 @@ noinline int __filemap_add_folio(struct address_space *= mapping, trace_mm_filemap_add_to_page_cache(folio); return 0; error: - if (charged) - mem_cgroup_uncharge(folio); folio->mapping =3D NULL; /* Leave page->index set: truncation relies upon it */ folio_put_refs(folio, nr); @@ -946,11 +936,16 @@ int filemap_add_folio(struct address_space *mapping, = struct folio *folio, void *shadow =3D NULL; int ret; =20 + ret =3D mem_cgroup_charge(folio, NULL, gfp); + if (ret) + return ret; + __folio_set_locked(folio); ret =3D __filemap_add_folio(mapping, folio, index, gfp, &shadow); - if (unlikely(ret)) + if (unlikely(ret)) { + mem_cgroup_uncharge(folio); __folio_clear_locked(folio); - else { + } else { /* * The folio might have been evicted from cache only * recently, in which case it should be activated like --=20 2.44.0 From nobody Fri Feb 13 09:49:17 2026 Received: from mail-pj1-f52.google.com (mail-pj1-f52.google.com [209.85.216.52]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0647D84D0D for ; Mon, 15 Apr 2024 17:19:18 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.52 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1713201560; cv=none; b=nedkt5kOsyCFA3zTXw5K9lQGMW92eYupoq0FjgG0IXD1964FDhTO9MAMMEZXODYnXkAa9FtWLLMNFKxwlIdOGghzKV/KziQaOQmTFpTBUTpSYVcskE0kS33Ngc62QfLoWiBuLbyRwBZOjL8mV4i9Eo+DGXLm3R6m01IjDOdb2TM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1713201560; c=relaxed/simple; bh=1n8aWgPGqTpkEn1L9GhQ17dKsC5Kuq6hSgWoGawRKkk=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=H44i09csJK74+4kgS+DgJXao3+cD1CFDem5M/KzUoZfh/JTiwQuf7P5a6CqVZe312Kl4Vp4cklnQUwufl4Vz0WmsmcnoRcKNHq29nifCRbMrHengBXermRToBXLMZsb5VI9JwsmZcbXiWF5M9kI39Ym6Zvs8wF3O6kP2iISPW2E= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=JfvGqiVd; arc=none smtp.client-ip=209.85.216.52 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="JfvGqiVd" Received: by mail-pj1-f52.google.com with SMTP id 98e67ed59e1d1-2a484f772e2so1932858a91.3 for ; Mon, 15 Apr 2024 10:19:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1713201558; x=1713806358; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:from:to:cc:subject :date:message-id:reply-to; bh=cEjwwoQAn8m3grSTrvNR4sEkOR0nGA3RJ0TZjOfUHm4=; b=JfvGqiVdOdWhdDkOgUZR9JXwXkSX0Ud1g4N0Wqh7ehPrmbdjPBE8ovrxPrKBVy7Mvs HquYTJcxBS6ymJLexZv+oJHZHrGy4Ne+eghntY1eNVFtp7hJEYHaiMZLvfXAtAf3Wq3H Os9PWQdPy4rlFZ/WTvH4s3gCe35B/QVjGEMtbM6Lt8YpHbujKuWPuFNFu+bTQu/V46zx mQdK7Ffx51oTagm4m+Mdx1sjWVly0CVYb2E2K9yJhWNUfxlFSNVG/0hzEyY9BqdqN7F+ BE1vcpWwWvGnBVN7Rm3NoFWqDLtHbFbSOgNsKmv7fto12pwBAZcP1BK5FsaaR1D6CMl5 wC/A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1713201558; x=1713806358; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=cEjwwoQAn8m3grSTrvNR4sEkOR0nGA3RJ0TZjOfUHm4=; b=b1+AoQIvMRyOGaNJACOfvM+AG5CqbMrIQEbtyYL9uN4HsmQfyiitvjPlA27Njs9UGO iD4ziCZ4wrtDY0BS+8X1u2ChYf0D0UntS9seYvgTJZsY3/8RnVK3ImJgMZNIU3YAwSMq 3XL0r/c//vjmKJfNnZb+a8s7lBQftBYDYABrIxId9GA4ziN16JPwH6OjFo+E61dv6UQ9 5qpK878qHMqAmeaR95EwnpuJWyp08/0GQhGGwc0YOVgEZ1jDoJuXmcke1+JiCp+GSqby 8pkpc+jUYtkm5lwN6e3QaVa4RJixpa54HsNT9yDwpccwabcou7JzWbb5ysdy1fcrJcP9 fCeA== X-Forwarded-Encrypted: i=1; AJvYcCWEE3acCY4J8xVLNoyr/cHJvUG2C6ZnNu8cmbIZ0YAeIhAxPUbYOOCSMCcMdV2Mi7hiDVErpglCzfEvpaLtbl3sd49LWY6JHwAfihlp X-Gm-Message-State: AOJu0Yw+omAv/9WwHT6sojQ7Kbslz+tPocJmqlQtaFZY2hLDRg0JwOXg 1e5eI68PzP1kvTTJUR7R0bxsGLR7amO66V9tCezCH2t3I0yralGtE8Phc+Xt X-Google-Smtp-Source: AGHT+IFva3FT+6Qo1iTE2UIbKwGFak0JgfA5Q5ANtjD68Wt1gDflM7XbEgZEqtWRZbI/FtzxK97aFQ== X-Received: by 2002:a17:90b:3d3:b0:2a5:ba9d:a06b with SMTP id go19-20020a17090b03d300b002a5ba9da06bmr10332693pjb.5.1713201558299; Mon, 15 Apr 2024 10:19:18 -0700 (PDT) Received: from localhost.localdomain ([101.32.222.185]) by smtp.gmail.com with ESMTPSA id o19-20020a17090aac1300b002a537abb536sm8751648pjq.57.2024.04.15.10.19.16 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Mon, 15 Apr 2024 10:19:17 -0700 (PDT) From: Kairui Song To: linux-mm@kvack.org Cc: Matthew Wilcox , Andrew Morton , linux-kernel@vger.kernel.org, Kairui Song Subject: [PATCH v3 3/4] lib/xarray: introduce a new helper xas_get_order Date: Tue, 16 Apr 2024 01:18:55 +0800 Message-ID: <20240415171857.19244-4-ryncsn@gmail.com> X-Mailer: git-send-email 2.44.0 In-Reply-To: <20240415171857.19244-1-ryncsn@gmail.com> References: <20240415171857.19244-1-ryncsn@gmail.com> Reply-To: Kairui Song Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Kairui Song It can be used after xas_load to check the order of loaded entries. Compared to xa_get_order, it saves an XA_STATE and avoid a rewalk. Also add new test for xas_get_order, to make the test work we have to export xas_get_order with EXPORT_SYMBOL_GPL. Signed-off-by: Kairui Song --- include/linux/xarray.h | 6 ++++++ lib/test_xarray.c | 34 +++++++++++++++++++++++++++++ lib/xarray.c | 49 ++++++++++++++++++++++++++---------------- 3 files changed, 71 insertions(+), 18 deletions(-) diff --git a/include/linux/xarray.h b/include/linux/xarray.h index cb571dfcf4b1..d9d479334c9e 100644 --- a/include/linux/xarray.h +++ b/include/linux/xarray.h @@ -1548,6 +1548,7 @@ void xas_create_range(struct xa_state *); =20 #ifdef CONFIG_XARRAY_MULTI int xa_get_order(struct xarray *, unsigned long index); +int xas_get_order(struct xa_state *xas); void xas_split(struct xa_state *, void *entry, unsigned int order); void xas_split_alloc(struct xa_state *, void *entry, unsigned int order, g= fp_t); #else @@ -1556,6 +1557,11 @@ static inline int xa_get_order(struct xarray *xa, un= signed long index) return 0; } =20 +static inline int xas_get_order(struct xa_state *xas) +{ + return 0; +} + static inline void xas_split(struct xa_state *xas, void *entry, unsigned int order) { diff --git a/lib/test_xarray.c b/lib/test_xarray.c index ebe2af2e072d..0efde8f93490 100644 --- a/lib/test_xarray.c +++ b/lib/test_xarray.c @@ -1984,6 +1984,39 @@ static noinline void check_get_order(struct xarray *= xa) } } =20 +static noinline void check_xas_get_order(struct xarray *xa) +{ + XA_STATE(xas, xa, 0); + + unsigned int max_order =3D IS_ENABLED(CONFIG_XARRAY_MULTI) ? 20 : 1; + unsigned int order; + unsigned long i, j; + + for (order =3D 0; order < max_order; order++) { + for (i =3D 0; i < 10; i++) { + xas_set_order(&xas, i << order, order); + do { + xas_lock(&xas); + xas_store(&xas, xa_mk_value(i)); + xas_unlock(&xas); + } while (xas_nomem(&xas, GFP_KERNEL)); + + for (j =3D i << order; j < (i + 1) << order; j++) { + xas_set_order(&xas, j, 0); + rcu_read_lock(); + xas_load(&xas); + XA_BUG_ON(xa, xas_get_order(&xas) !=3D order); + rcu_read_unlock(); + } + + xas_lock(&xas); + xas_set_order(&xas, i << order, order); + xas_store(&xas, NULL); + xas_unlock(&xas); + } + } +} + static noinline void check_destroy(struct xarray *xa) { unsigned long index; @@ -2035,6 +2068,7 @@ static int xarray_checks(void) check_multi_store(&array); check_multi_store_advanced(&array); check_get_order(&array); + check_xas_get_order(&array); check_xa_alloc(); check_find(&array); check_find_entry(&array); diff --git a/lib/xarray.c b/lib/xarray.c index 39f07bfc4dcc..fbf1d1dd83bc 100644 --- a/lib/xarray.c +++ b/lib/xarray.c @@ -1750,39 +1750,52 @@ void *xa_store_range(struct xarray *xa, unsigned lo= ng first, EXPORT_SYMBOL(xa_store_range); =20 /** - * xa_get_order() - Get the order of an entry. - * @xa: XArray. - * @index: Index of the entry. + * xas_get_order() - Get the order of an loaded entry after xas_load. + * @xas: XArray operation state. + * + * Called after xas_load, the xas should not be in an error state. * * Return: A number between 0 and 63 indicating the order of the entry. */ -int xa_get_order(struct xarray *xa, unsigned long index) +int xas_get_order(struct xa_state *xas) { - XA_STATE(xas, xa, index); - void *entry; int order =3D 0; =20 - rcu_read_lock(); - entry =3D xas_load(&xas); - - if (!entry) - goto unlock; - - if (!xas.xa_node) - goto unlock; + if (!xas->xa_node) + return 0; =20 for (;;) { - unsigned int slot =3D xas.xa_offset + (1 << order); + unsigned int slot =3D xas->xa_offset + (1 << order); =20 if (slot >=3D XA_CHUNK_SIZE) break; - if (!xa_is_sibling(xas.xa_node->slots[slot])) + if (!xa_is_sibling(xas->xa_node->slots[slot])) break; order++; } =20 - order +=3D xas.xa_node->shift; -unlock: + order +=3D xas->xa_node->shift; + return order; +} +EXPORT_SYMBOL_GPL(xas_get_order); + +/** + * xa_get_order() - Get the order of an entry. + * @xa: XArray. + * @index: Index of the entry. + * + * Return: A number between 0 and 63 indicating the order of the entry. + */ +int xa_get_order(struct xarray *xa, unsigned long index) +{ + XA_STATE(xas, xa, index); + int order =3D 0; + void *entry; + + rcu_read_lock(); + entry =3D xas_load(&xas); + if (entry) + order =3D xas_get_order(&xas); rcu_read_unlock(); =20 return order; --=20 2.44.0 From nobody Fri Feb 13 09:49:17 2026 Received: from mail-pj1-f42.google.com (mail-pj1-f42.google.com [209.85.216.42]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8342585268 for ; Mon, 15 Apr 2024 17:19:21 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.42 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1713201563; cv=none; b=BtbxKhWmXsvIcauTrHEXMMABsQrJl4YHJrz5hDZI89zfzEvJUltpBAxYu2hmK7q2u35ukZ3gljOoTkn8lP6w4NaI7G1uzHO/at5C3FNj107TRjd1IU75LcqQY81gXhB42pOeq7SqX6QkfUN6TNN/k2sqpj/gTdgiwhBZjSA2vbA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1713201563; c=relaxed/simple; bh=pDqpPLj4qmpz6qIGdATZILDnKVbrqLD2jtZdMCM1byo=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=InU3vBYTAt+f3KvyuN1lg1VTRL3LzUoxR3Lj9CF1y8XL7w2CiAGBD89aT/xTWRvSNwrLk93XsGrnJ9Ks6pcbnzRlLJkbBSMkwBxOgR8FghEtwYilZE6siJd49wCfPAJz6Y86+bFIfroS6kxZtJQWylxi55ZZX4qhnrEC1D7ccqM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=JJLQUOs1; arc=none smtp.client-ip=209.85.216.42 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="JJLQUOs1" Received: by mail-pj1-f42.google.com with SMTP id 98e67ed59e1d1-2a87bd53dc3so997955a91.2 for ; Mon, 15 Apr 2024 10:19:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1713201561; x=1713806361; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:from:to:cc:subject :date:message-id:reply-to; bh=7ftt1wlC5vn+OAyWyZNr8D22lSmnA0oAAZ/IwpKgu3E=; b=JJLQUOs1Pdh+vNoK2d8VxpuRCZwf7Jf62rQkmwsoUWmHOUTPU6lYzhtCWia1/8Bm+Q 6ZKedxKVz8mOKLbfCxK4Ev6Tyh/VPAIHoL3XpAClgzmHrd8kr3w9VeY9YR7RCox7FYtI RPZ1SdUcDsQQA0diPoO5c+kNOPiUZ/QJro1cNI5IYNZvx3JH/j5CLa57DJxvzSXqbaId r//nqibSlsVZ2tPY9F2Y3REy2AIOkg9QNUFSVn2emWxpyRmZ7cNr5ikJ2hxk5fXnUNkY hrhOxuUIN6T/nRchcodQR8IRN786nb/88zRnDOznu8GMi2htkkuYJb8WRk3Eism8I2Pi NwjQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1713201561; x=1713806361; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=7ftt1wlC5vn+OAyWyZNr8D22lSmnA0oAAZ/IwpKgu3E=; b=PET+MXSvVF0F+8ajiKMZ0jWDqILlpqEebl844SScObWX6+OPS5VOMTPp6mpS2CArrg qBVdLvIo3zbsJWHZl3ENj+7HnXpn3Vrj5unK7sGK+jd6xgJ/evJVxsfrwaXi3WYLBITd w0bFA/DrXazOQwm8hMRKu9rXvTnPnXd7twigsfk1hln+oHfaIB5wZwsuplOG601JA3Zg K/kGfUAYSVuPMngytrp7E8y7g3VrV6vNCCAlYtmWpRvPCGM74QkHyGLwltaQcRpLkZxk bATlIBNMffm5AFt8xH52FWMaAf2m6UQi63SnIaPkoZf+NJjkGHA1plTlJw+CAvX3cgPI KHPA== X-Forwarded-Encrypted: i=1; AJvYcCUFEAdmoQNa0PoJpQXPDvJgFm00uVHSGwz42fBMY/pJf0tQDjMPT5iC/PwQThm7Bfr6DH7J+hOXUvp7/4geFy5qhWjQTfAqCWsxHnhy X-Gm-Message-State: AOJu0YxdrVs4oCFSKoB9Jry++WFthl+qhAV0cuoHuHRyzzFPbGcEcAwR MmfzzszzRzsgmHZY5QGa1bjWxEn7i9/ompCu5UUWJqlywp4b+9zL X-Google-Smtp-Source: AGHT+IF5I7CUmmaoFvwK3SVG9GhKVEy4AarbCQA8oQBcJAZLx+3bOZ1JGWosjLM5SoLJEKPUKgit5A== X-Received: by 2002:a17:90a:f2d7:b0:2a2:b9f8:e0b0 with SMTP id gt23-20020a17090af2d700b002a2b9f8e0b0mr6933990pjb.19.1713201560649; Mon, 15 Apr 2024 10:19:20 -0700 (PDT) Received: from localhost.localdomain ([101.32.222.185]) by smtp.gmail.com with ESMTPSA id o19-20020a17090aac1300b002a537abb536sm8751648pjq.57.2024.04.15.10.19.18 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Mon, 15 Apr 2024 10:19:20 -0700 (PDT) From: Kairui Song To: linux-mm@kvack.org Cc: Matthew Wilcox , Andrew Morton , linux-kernel@vger.kernel.org, Kairui Song Subject: [PATCH v3 4/4] mm/filemap: optimize filemap folio adding Date: Tue, 16 Apr 2024 01:18:56 +0800 Message-ID: <20240415171857.19244-5-ryncsn@gmail.com> X-Mailer: git-send-email 2.44.0 In-Reply-To: <20240415171857.19244-1-ryncsn@gmail.com> References: <20240415171857.19244-1-ryncsn@gmail.com> Reply-To: Kairui Song Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable From: Kairui Song Instead of doing multiple tree walks, do one optimism range check with lock hold, and exit if raced with another insertion. If a shadow exists, check it with a new xas_get_order helper before releasing the lock to avoid redundant tree walks for getting its order. Drop the lock and do the allocation only if a split is needed. In the best case, it only need to walk the tree once. If it needs to alloc and split, 3 walks are issued (One for first ranged conflict check and order retrieving, one for the second check after allocation, one for the insert after split). Testing with 4K pages, in an 8G cgroup, with 16G brd as block device: echo 3 > /proc/sys/vm/drop_caches fio -name=3Dcached --numjobs=3D16 --filename=3D/mnt/test.img \ --buffered=3D1 --ioengine=3Dmmap --rw=3Drandread --time_based \ --ramp_time=3D30s --runtime=3D5m --group_reporting Before: bw ( MiB/s): min=3D 1027, max=3D 3520, per=3D100.00%, avg=3D2445.02, stdev= =3D18.90, samples=3D8691 iops : min=3D263001, max=3D901288, avg=3D625924.36, stdev=3D4837.28,= samples=3D8691 After (+7.3%): bw ( MiB/s): min=3D 493, max=3D 3947, per=3D100.00%, avg=3D2625.56, stdev= =3D25.74, samples=3D8651 iops : min=3D126454, max=3D1010681, avg=3D672142.61, stdev=3D6590.48= , samples=3D8651 Test result with THP (do a THP randread then switch to 4K page in hope it issues a lot of splitting): echo 3 > /proc/sys/vm/drop_caches fio -name=3Dcached --numjobs=3D16 --filename=3D/mnt/test.img \ --buffered=3D1 --ioengine=3Dmmap -thp=3D1 --readonly \ --rw=3Drandread --time_based --ramp_time=3D30s --runtime=3D10m \ --group_reporting fio -name=3Dcached --numjobs=3D16 --filename=3D/mnt/test.img \ --buffered=3D1 --ioengine=3Dmmap \ --rw=3Drandread --time_based --runtime=3D5s --group_reporting Before: bw ( KiB/s): min=3D 4141, max=3D14202, per=3D100.00%, avg=3D7935.51, stdev= =3D96.85, samples=3D18976 iops : min=3D 1029, max=3D 3548, avg=3D1979.52, stdev=3D24.23, sampl= es=3D18976=C2=B7 READ: bw=3D4545B/s (4545B/s), 4545B/s-4545B/s (4545B/s-4545B/s), io=3D64.0K= iB (65.5kB), run=3D14419-14419msec After (+12.5%): bw ( KiB/s): min=3D 4611, max=3D15370, per=3D100.00%, avg=3D8928.74, stdev= =3D105.17, samples=3D19146 iops : min=3D 1151, max=3D 3842, avg=3D2231.27, stdev=3D26.29, sampl= es=3D19146 READ: bw=3D4635B/s (4635B/s), 4635B/s-4635B/s (4635B/s-4635B/s), io=3D64.0K= iB (65.5kB), run=3D14137-14137msec The performance is better for both 4K (+7.5%) and THP (+12.5%) cached read. Signed-off-by: Kairui Song --- lib/test_xarray.c | 59 +++++++++++++++++++++++++++++++++++++++++++++++ mm/filemap.c | 56 ++++++++++++++++++++++++++++++++------------ 2 files changed, 100 insertions(+), 15 deletions(-) diff --git a/lib/test_xarray.c b/lib/test_xarray.c index 0efde8f93490..8732a311f613 100644 --- a/lib/test_xarray.c +++ b/lib/test_xarray.c @@ -2017,6 +2017,64 @@ static noinline void check_xas_get_order(struct xarr= ay *xa) } } =20 +static noinline void check_xas_conflict_get_order(struct xarray *xa) +{ + XA_STATE(xas, xa, 0); + + void *entry; + int only_once; + unsigned int max_order =3D IS_ENABLED(CONFIG_XARRAY_MULTI) ? 20 : 1; + unsigned int order; + unsigned long i, j, k; + + for (order =3D 0; order < max_order; order++) { + for (i =3D 0; i < 10; i++) { + xas_set_order(&xas, i << order, order); + do { + xas_lock(&xas); + xas_store(&xas, xa_mk_value(i)); + xas_unlock(&xas); + } while (xas_nomem(&xas, GFP_KERNEL)); + + /* + * Ensure xas_get_order works with xas_for_each_conflict. + */ + j =3D i << order; + for (k =3D 0; k < order; k++) { + only_once =3D 0; + xas_set_order(&xas, j + (1 << k), k); + xas_lock(&xas); + xas_for_each_conflict(&xas, entry) { + XA_BUG_ON(xa, entry !=3D xa_mk_value(i)); + XA_BUG_ON(xa, xas_get_order(&xas) !=3D order); + only_once++; + } + XA_BUG_ON(xa, only_once !=3D 1); + xas_unlock(&xas); + } + + if (order < max_order - 1) { + only_once =3D 0; + xas_set_order(&xas, (i & ~1UL) << order, order + 1); + xas_lock(&xas); + xas_for_each_conflict(&xas, entry) { + XA_BUG_ON(xa, entry !=3D xa_mk_value(i)); + XA_BUG_ON(xa, xas_get_order(&xas) !=3D order); + only_once++; + } + XA_BUG_ON(xa, only_once !=3D 1); + xas_unlock(&xas); + } + + xas_set_order(&xas, i << order, order); + xas_lock(&xas); + xas_store(&xas, NULL); + xas_unlock(&xas); + } + } +} + + static noinline void check_destroy(struct xarray *xa) { unsigned long index; @@ -2069,6 +2127,7 @@ static int xarray_checks(void) check_multi_store_advanced(&array); check_get_order(&array); check_xas_get_order(&array); + check_xas_conflict_get_order(&array); check_xa_alloc(); check_find(&array); check_find_entry(&array); diff --git a/mm/filemap.c b/mm/filemap.c index 17a66ea544e7..7b0b2229d4ed 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -852,7 +852,9 @@ noinline int __filemap_add_folio(struct address_space *= mapping, struct folio *folio, pgoff_t index, gfp_t gfp, void **shadowp) { XA_STATE(xas, &mapping->i_pages, index); - bool huge =3D folio_test_hugetlb(folio); + void *alloced_shadow =3D NULL; + int alloced_order =3D 0; + bool huge; long nr; =20 VM_BUG_ON_FOLIO(!folio_test_locked(folio), folio); @@ -861,6 +863,7 @@ noinline int __filemap_add_folio(struct address_space *= mapping, =20 VM_BUG_ON_FOLIO(index & (folio_nr_pages(folio) - 1), folio); xas_set_order(&xas, index, folio_order(folio)); + huge =3D folio_test_hugetlb(folio); nr =3D folio_nr_pages(folio); =20 gfp &=3D GFP_RECLAIM_MASK; @@ -868,16 +871,10 @@ noinline int __filemap_add_folio(struct address_space= *mapping, folio->mapping =3D mapping; folio->index =3D xas.xa_index; =20 - do { - unsigned int order =3D xa_get_order(xas.xa, xas.xa_index); + for (;;) { + int order =3D -1, split_order =3D 0; void *entry, *old =3D NULL; =20 - if (order > folio_order(folio)) { - xas_split_alloc(&xas, xa_load(xas.xa, xas.xa_index), - order, gfp); - if (xas_error(&xas)) - goto error; - } xas_lock_irq(&xas); xas_for_each_conflict(&xas, entry) { old =3D entry; @@ -885,19 +882,33 @@ noinline int __filemap_add_folio(struct address_space= *mapping, xas_set_err(&xas, -EEXIST); goto unlock; } + /* + * If a larger entry exists, + * it will be the first and only entry iterated. + */ + if (order =3D=3D -1) + order =3D xas_get_order(&xas); + } + + /* entry may have changed before we re-acquire the lock */ + if (alloced_order && (old !=3D alloced_shadow || order !=3D alloced_orde= r)) { + xas_destroy(&xas); + alloced_order =3D 0; } =20 if (old) { - if (shadowp) - *shadowp =3D old; - /* entry may have been split before we acquired lock */ - order =3D xa_get_order(xas.xa, xas.xa_index); - if (order > folio_order(folio)) { + if (order > 0 && order > folio_order(folio)) { /* How to handle large swap entries? */ BUG_ON(shmem_mapping(mapping)); + if (!alloced_order) { + split_order =3D order; + goto unlock; + } xas_split(&xas, old, order); xas_reset(&xas); } + if (shadowp) + *shadowp =3D old; } =20 xas_store(&xas, folio); @@ -913,9 +924,24 @@ noinline int __filemap_add_folio(struct address_space = *mapping, __lruvec_stat_mod_folio(folio, NR_FILE_THPS, nr); } + unlock: xas_unlock_irq(&xas); - } while (xas_nomem(&xas, gfp)); + + /* split needed, alloc here and retry. */ + if (split_order) { + xas_split_alloc(&xas, old, split_order, gfp); + if (xas_error(&xas)) + goto error; + alloced_shadow =3D old; + alloced_order =3D split_order; + xas_reset(&xas); + continue; + } + + if (!xas_nomem(&xas, gfp)) + break; + } =20 if (xas_error(&xas)) goto error; --=20 2.44.0