From nobody Sat Feb 7 19:08:14 2026 Received: from mail-pf1-f171.google.com (mail-pf1-f171.google.com [209.85.210.171]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 08D5A84DF0 for ; Tue, 16 Apr 2024 07:17:48 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.171 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1713251870; cv=none; b=hiiVq5Befy91sDGBKbtx1V/aWQEmRUJQnZrR9FczM++XfESUSTdqa0OoK3J5qorcNTEup+sEDA1wILBMDkk7HPDCcljuBBjjzu4MVZlQM9XZk3SHuchdEbo0TGC4x3c5VrFqoa+1GTsikW2Jm4kvhIP9L+wLUGDrTczDWlCKjc4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1713251870; c=relaxed/simple; bh=1k7Mo6A53sEnM9AVmjmMFSd6Kr0X66nFRotuTggUvfA=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=oExlzZ+mx4JvOgTyTxyCo1YmIKuLPk2N6uAjUthKdAFZt9qJ9/JmJH1Fj5E4WVA2XX/LtYHH/zNEOxGV9LVDFiRGaczUY5+L7CkujMvOUBTCRhmjiE+YD8S00tnqvCZA/5nZLdmCoH5F9Zoni//F959paayQoM0SeSCfwzH1d3I= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=kFA+Xwgt; arc=none smtp.client-ip=209.85.210.171 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="kFA+Xwgt" Received: by mail-pf1-f171.google.com with SMTP id d2e1a72fcca58-6ed627829e6so4427498b3a.1 for ; Tue, 16 Apr 2024 00:17:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1713251868; x=1713856668; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:from:to:cc:subject :date:message-id:reply-to; bh=0LQlo9tV6jA7edPLPREtk8jPXGN28KNPzfvTbxW9Ygw=; b=kFA+XwgtFqWJNWQJMA5iLnf1SEnD3Bh+Hd8uF0+gWd/ODr8p5Lm76xrta4mWeeecqI hPB3qrP9e3e5hz42rb8wTlL/nuYwuO3HazcTm4iOo32zvzSGVFKiFgcpZKU0PhIadGoG N6ZB3pPjTKWemR1KduOoMM7HlLS9eYWAGxY59sdj1MlXrKWLAqBSkGWEfRVlewlc/KtA TOYLLY/iqpSwnV8N5es/iv1Aw9JVdwcLyJQxukbQ8rp9h6qKKttMAxYgJKeo96UYwqE0 KRkaRvaY00SEoisfXQNWAOqxlYADGW311Mjinb+39OKaLqwGulUrnnKDuPEtSmHRpBT/ P/Gg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1713251868; x=1713856668; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=0LQlo9tV6jA7edPLPREtk8jPXGN28KNPzfvTbxW9Ygw=; b=G3P+WChu4m5PRTQLKiyT02Uy9NrZoTdanfMGvqKi2PTX12wt3OqKZzi/J4FADu1XmR X0fDmDcWFLou5okWMfGkA+HMsyKsrAF1mPtIHKwavcIPiVozpf8ao+iM+C7+yaBAAeVO SpsC8u9yMoSvNHQcE/Jwj6K7o3Z3QAhn8NXr/5Nurg5xTzo1sdnGyfUAW/D1JEyQkMDk MgN0snjQtVvEcOL9Opmx3sxymGlJj8I346ytTAlzVl9N6nkrWXTQQqARTainam6bRS4m GGXoXt+b8kgBbmeKkojyQyEw1yrYk4xGnKRliR3x44F/bNqx0MyXS+zQfwOlcbYJZft1 CrMQ== X-Forwarded-Encrypted: i=1; AJvYcCXufIG0M83QKskk0xC/r9L507p5QO97kjryEc70/aH1DyyzJduDWklpP9lnQfKtvN1M09F26uD2gU1J0ynN9U6Xr9d5FdlkMiU6/A7O X-Gm-Message-State: AOJu0YyE+URnTWsLhH8Gg7gP8BruS138dsnmdzs6a7xIhdNWcnXwd9Il lLN4S4UiGshWwKQMDfQQpng3TcFrGa8d6tg73HIkhSUd5eEYTFrq X-Google-Smtp-Source: AGHT+IFA2zmy9+KvgrKxSYzQek06HbyAOJH+OqT9zi1O7kh6mqKJgpjWl+AmhEUdNRhj7ZdCRoeZAQ== X-Received: by 2002:a05:6a00:399d:b0:6ea:c2c7:5d6a with SMTP id fi29-20020a056a00399d00b006eac2c75d6amr15984911pfb.2.1713251868205; Tue, 16 Apr 2024 00:17:48 -0700 (PDT) Received: from KASONG-MB2.tencent.com ([43.132.141.21]) by smtp.gmail.com with ESMTPSA id lr48-20020a056a0073b000b006e724ccdc3esm8340029pfb.55.2024.04.16.00.17.46 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Tue, 16 Apr 2024 00:17:47 -0700 (PDT) From: Kairui Song To: linux-mm@kvack.org Cc: Andrew Morton , Matthew Wilcox , linux-kernel@vger.kernel.org, Kairui Song Subject: [PATCH v4 1/4] mm/filemap: return early if failed to allocate memory for split Date: Tue, 16 Apr 2024 15:17:19 +0800 Message-ID: <20240416071722.45997-2-ryncsn@gmail.com> X-Mailer: git-send-email 2.44.0 In-Reply-To: <20240416071722.45997-1-ryncsn@gmail.com> References: <20240416071722.45997-1-ryncsn@gmail.com> Reply-To: Kairui Song Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Kairui Song xas_split_alloc could fail with NOMEM, and in such case, it should abort early instead of keep going and fail the xas_split below. Signed-off-by: Kairui Song Acked-by: Matthew Wilcox (Oracle) --- mm/filemap.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/mm/filemap.c b/mm/filemap.c index 30de18c4fd28..d447b36ffd41 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -880,9 +880,12 @@ noinline int __filemap_add_folio(struct address_space = *mapping, unsigned int order =3D xa_get_order(xas.xa, xas.xa_index); void *entry, *old =3D NULL; =20 - if (order > folio_order(folio)) + if (order > folio_order(folio)) { xas_split_alloc(&xas, xa_load(xas.xa, xas.xa_index), order, gfp); + if (xas_error(&xas)) + goto error; + } xas_lock_irq(&xas); xas_for_each_conflict(&xas, entry) { old =3D entry; --=20 2.44.0 From nobody Sat Feb 7 19:08:14 2026 Received: from mail-oa1-f44.google.com (mail-oa1-f44.google.com [209.85.160.44]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3D03115921E for ; Tue, 16 Apr 2024 07:17:51 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.160.44 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1713251872; cv=none; b=bN/p15nYxCSlWzrImLe2x8frJm1Wq6gs3wrUJQvvucmScpTaN3ZToa9N3kQKZnjgET4sWxEfxah9LcDs3op3h7/i+R2bs6UVhTDISZ+gGMK/d/fyGQ2f53xlT4a9t0fccv6UJVGsWmoS6yH8W9Q/PMmuW4fanKtj+QsU0G+DgME= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1713251872; c=relaxed/simple; bh=kzdlZqYQWNrIXSnJqwtpoqwSYY1qBtvI2I8Lo4LwsCA=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=XOYu1V30DdzorjsvGTPgDZI9+zGsTNHEyrS4rZfG8zz119S4aGkD95IYK25zxfasym2F7jDqa+3C6BC65Vow0UIFcAagnWCSxlOnznYLv41AfowZlkqtIm7JVb/l7SfRVFoO1jkLJMtPhb9UnQYU4/008UHYqn8BmcoNlUF13EU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=PKHvZH05; arc=none smtp.client-ip=209.85.160.44 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="PKHvZH05" Received: by mail-oa1-f44.google.com with SMTP id 586e51a60fabf-22ed72b0082so2724213fac.3 for ; Tue, 16 Apr 2024 00:17:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1713251870; x=1713856670; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:from:to:cc:subject :date:message-id:reply-to; bh=F+Z2hNJP+z3T6EfHmF6/rzbCctrFBIWGBJWrf7iP/Ig=; b=PKHvZH05CSHHTCzGQYcpfewmlVPCHavBPJwc/OXAqABmm33u7/c9R4JBABsQYpms2J Pbl8rMEt+IeFkdTVVfaJ7ojwp0VIOQgoDg8awiXEQLa6lNzCDnBvijExqGgRUw5TpQEy 7jBtcaKUtK3NM/M/0R7JVFe6R/GFTixPV+L3j4XZkvaduuFDWiTDgcfXsbE3568c1wuu KGI7b8j6cQSBkT8vdC84wWzSFzfCIVUxxWgQKn2l+yBEreSFFu93nJa4h50jlNpsqtTm qy98gkus5GAaAuSbWvfSKA+9jze6OmrMozhbDgYo6tabb3W/e89kzll9KwLqPJUr6tHE QuCw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1713251870; x=1713856670; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=F+Z2hNJP+z3T6EfHmF6/rzbCctrFBIWGBJWrf7iP/Ig=; b=VcKsjignJmjyNcx5vEvTarynMCmMMT13AnVpjDJ8U7sBc0rnVfvjy6dLLc6uZU88ao 9JSDOF4mibWi3F3kMPcYl7aw/1aJmkN4SY9RCy5o6/HPR7QYHUr8oDKrvw41z7pj8nlv hrlR6gR8KyiF/Kq+aboWGhMTjG0ojgksMFXH3mjWo/Il5OOZhkarSb6e8ReJjNF5IZSd FgXFNJDlMnNA7sBFrzvxed/n2HGdCuZ6LxecByy5Atv0LHOHx3Lxc6zQvvoqZ5qBQk+W mHRZcsJodlxsSlmDUQbe5msxCTneODTa4KdpVBiHhoEQlnaMKoRu/LvA76mY69xfhfJE UCnA== X-Forwarded-Encrypted: i=1; AJvYcCW/PL8MTaWtIymfPf+/fMzHXuTpe9DvgWdlKmoD7hc99/hd/3b2knfNL1fwpAUvh6ZbuphWUELiREjoDqpl0bnIMtCHLKXcp78D2s3e X-Gm-Message-State: AOJu0YyGF8JVbNf67mdNHZjmw2KG9/bqmSuPFLmFYwqzNY+2i/WEskSm VlK8XRGM1BLjnUeRXh5doc9rLL9Mx1XJk3t4xQjd4I2ogirRlbJ4 X-Google-Smtp-Source: AGHT+IGTlCysBm/YOmaGrEGYXumxXfgrNAPvhr2uKFZUXitXWelGd6XrYs4GaN3Yt1Z5t4vLHkZtFQ== X-Received: by 2002:a05:6870:7312:b0:22e:e0c5:c729 with SMTP id q18-20020a056870731200b0022ee0c5c729mr15807231oal.3.1713251870398; Tue, 16 Apr 2024 00:17:50 -0700 (PDT) Received: from KASONG-MB2.tencent.com ([43.132.141.21]) by smtp.gmail.com with ESMTPSA id lr48-20020a056a0073b000b006e724ccdc3esm8340029pfb.55.2024.04.16.00.17.48 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Tue, 16 Apr 2024 00:17:50 -0700 (PDT) From: Kairui Song To: linux-mm@kvack.org Cc: Andrew Morton , Matthew Wilcox , linux-kernel@vger.kernel.org, Kairui Song Subject: [PATCH v4 2/4] mm/filemap: clean up hugetlb exclusion code Date: Tue, 16 Apr 2024 15:17:20 +0800 Message-ID: <20240416071722.45997-3-ryncsn@gmail.com> X-Mailer: git-send-email 2.44.0 In-Reply-To: <20240416071722.45997-1-ryncsn@gmail.com> References: <20240416071722.45997-1-ryncsn@gmail.com> Reply-To: Kairui Song Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Kairui Song __filemap_add_folio only has two callers, one never passes hugetlb folio and one always passes in hugetlb folio. So move the hugetlb related cgroup charging out of it to make the code cleaner. Signed-off-by: Kairui Song Acked-by: Matthew Wilcox (Oracle) --- mm/filemap.c | 21 ++++++++------------- 1 file changed, 8 insertions(+), 13 deletions(-) diff --git a/mm/filemap.c b/mm/filemap.c index d447b36ffd41..7f734b7ad273 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -853,20 +853,12 @@ noinline int __filemap_add_folio(struct address_space= *mapping, { XA_STATE(xas, &mapping->i_pages, index); bool huge =3D folio_test_hugetlb(folio); - bool charged =3D false; - long nr =3D 1; + long nr; =20 VM_BUG_ON_FOLIO(!folio_test_locked(folio), folio); VM_BUG_ON_FOLIO(folio_test_swapbacked(folio), folio); mapping_set_update(&xas, mapping); =20 - if (!huge) { - int error =3D mem_cgroup_charge(folio, NULL, gfp); - if (error) - return error; - charged =3D true; - } - VM_BUG_ON_FOLIO(index & (folio_nr_pages(folio) - 1), folio); xas_set_order(&xas, index, folio_order(folio)); nr =3D folio_nr_pages(folio); @@ -931,8 +923,6 @@ noinline int __filemap_add_folio(struct address_space *= mapping, trace_mm_filemap_add_to_page_cache(folio); return 0; error: - if (charged) - mem_cgroup_uncharge(folio); folio->mapping =3D NULL; /* Leave page->index set: truncation relies upon it */ folio_put_refs(folio, nr); @@ -946,11 +936,16 @@ int filemap_add_folio(struct address_space *mapping, = struct folio *folio, void *shadow =3D NULL; int ret; =20 + ret =3D mem_cgroup_charge(folio, NULL, gfp); + if (ret) + return ret; + __folio_set_locked(folio); ret =3D __filemap_add_folio(mapping, folio, index, gfp, &shadow); - if (unlikely(ret)) + if (unlikely(ret)) { + mem_cgroup_uncharge(folio); __folio_clear_locked(folio); - else { + } else { /* * The folio might have been evicted from cache only * recently, in which case it should be activated like --=20 2.44.0 From nobody Sat Feb 7 19:08:14 2026 Received: from mail-pf1-f174.google.com (mail-pf1-f174.google.com [209.85.210.174]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 61A5415A482 for ; Tue, 16 Apr 2024 07:17:53 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.174 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1713251874; cv=none; b=cFt7B/rXqfLx7deXOmBSPbIbxpZY39i6kI9zq0+T86kFRd6cW8ztuearTqJ5ciaV8lq8mn1gRNaBln7rb6Huj/S42FwF7eAPTAsb4uq/tD3Rtk5mCLbPb2sjvslmVJkhzLTdOq7DHmbbJl8uw0JkHggpofDUca2EL6h9mtxDuzY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1713251874; c=relaxed/simple; bh=55gbjDjFA8TyGhJfeRJbHTOAEYaKy6v7j3ItuuPVYIU=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=VxINrwdQ5DwsuYaDq9xscIJvvFZPMS2zhwQGqopZDtBxMLC+nyAS+4dVQubNzn+2OGwJOqBzZrzpz+ZNBBi9ZiVAj93uLMHr0Cbdh5z43FQmXP9B33THZN3ODvmFtuwWUFbsWTg7s+v0MiTvfBb3HLzqQ+j3jPJmtxSUr7WQZq8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=AMRSYHw7; arc=none smtp.client-ip=209.85.210.174 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="AMRSYHw7" Received: by mail-pf1-f174.google.com with SMTP id d2e1a72fcca58-6ee12766586so2674318b3a.0 for ; Tue, 16 Apr 2024 00:17:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1713251872; x=1713856672; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:from:to:cc:subject :date:message-id:reply-to; bh=MHV2+XE+cugEfsKZfQ4KNnLWAEqkFV7dIjX6To9XLTs=; b=AMRSYHw7IAAzIzWaFhOvbdPbF77WNy6uOW4Y4rSnOKBCqt7KohKjhvcOtcrkg0v1M2 BQS+E32cxaUy25h0+p3YfhldDCZXIskywT6G8iOsCXbN/sV1VmHzgGQrLjVPj3yCSmpK l5Ap1qzD6fqS1fDO5g4vcCedL4x2Z21GWEVlt9+YuLE4KvqdMY3ecc98zrNjZx6GLnrw Mx+iU7+GiqfQsf2YXZYljO8xQDC7HBjcyvgB2dK8m7NztiYquKaie7b9ZNnUWBYtwJWz BC8JgfwtX6iSsTdv0OwcSY7ThZT7k4dLLf0EFf4gA699oIlAso4c2CJtThzp10EXvazz M3eg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1713251872; x=1713856672; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=MHV2+XE+cugEfsKZfQ4KNnLWAEqkFV7dIjX6To9XLTs=; b=ir3oenqenIT8nQmnvfKiRv1OXyqOPU97cVrIhL09LaBc6F1VBRRdJbdnppiJZPQev4 +mmIyqfARh5zmtur+FMNHX1EqlRersNZU7XZvfBzqHLCnxOkHPvIMeaTmYrHHt54BWEY zfqlbKTuh71mt7U3mpttb5ndsxoK57u6Kjg/oW9xk42Q81qdYxSfd0m4FHdIElzAaN1S lbt8Akvp5OL1Fp2QoWsrV8s8ZE32AtA1kAIhGkncZ5cPSn7egF3+d/F/HtO5F/1vm9Zp Lth6pnSTVkKrEL79Wl3gBGjD82Vx71TMvNwmYIblqSqCIOJRIZLBIQK10ZU2Z03C5zr5 j7MA== X-Forwarded-Encrypted: i=1; AJvYcCUKuoC5gXj9tOrPo0K2hanp7R/SQu3BXmdmCsM++XVosh6gJFhSi7QNcv95lP8VkIbQK+mFiYcMidXf2XfmPCfVAjpxyizVIvEm8tS4 X-Gm-Message-State: AOJu0YwDwn/wKwdPi2e7mvYCm9TsG04Xnw2mNZOl6EzTK+NnPRwK9hzZ Ewj3mBMplNNjingkpQaCHaRvg2HBab/KzaOOabfTu2CXyURt1Xl8 X-Google-Smtp-Source: AGHT+IESJwVhtcMf3m99N3mRkzrlYgFLHCki4Z3/LDKtQDjTD4+lFVo8Wfkqwv+ogYZO1JOFNTHWzw== X-Received: by 2002:a05:6a00:39a6:b0:6ea:9b37:c288 with SMTP id fi38-20020a056a0039a600b006ea9b37c288mr1644456pfb.15.1713251872585; Tue, 16 Apr 2024 00:17:52 -0700 (PDT) Received: from KASONG-MB2.tencent.com ([43.132.141.21]) by smtp.gmail.com with ESMTPSA id lr48-20020a056a0073b000b006e724ccdc3esm8340029pfb.55.2024.04.16.00.17.50 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Tue, 16 Apr 2024 00:17:52 -0700 (PDT) From: Kairui Song To: linux-mm@kvack.org Cc: Andrew Morton , Matthew Wilcox , linux-kernel@vger.kernel.org, Kairui Song Subject: [PATCH v4 3/4] lib/xarray: introduce a new helper xas_get_order Date: Tue, 16 Apr 2024 15:17:21 +0800 Message-ID: <20240416071722.45997-4-ryncsn@gmail.com> X-Mailer: git-send-email 2.44.0 In-Reply-To: <20240416071722.45997-1-ryncsn@gmail.com> References: <20240416071722.45997-1-ryncsn@gmail.com> Reply-To: Kairui Song Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Kairui Song It can be used after xas_load to check the order of loaded entries. Compared to xa_get_order, it saves an XA_STATE and avoid a rewalk. Added new test for xas_get_order, to make the test work, we have to export xas_get_order with EXPORT_SYMBOL_GPL. Also fix a sparse warning by checking the slot value with xa_entry instead of accessing it directly, as suggested by Matthew Wilcox. Signed-off-by: Kairui Song --- include/linux/xarray.h | 6 ++++++ lib/test_xarray.c | 34 +++++++++++++++++++++++++++++ lib/xarray.c | 49 ++++++++++++++++++++++++++---------------- 3 files changed, 71 insertions(+), 18 deletions(-) diff --git a/include/linux/xarray.h b/include/linux/xarray.h index cb571dfcf4b1..d9d479334c9e 100644 --- a/include/linux/xarray.h +++ b/include/linux/xarray.h @@ -1548,6 +1548,7 @@ void xas_create_range(struct xa_state *); =20 #ifdef CONFIG_XARRAY_MULTI int xa_get_order(struct xarray *, unsigned long index); +int xas_get_order(struct xa_state *xas); void xas_split(struct xa_state *, void *entry, unsigned int order); void xas_split_alloc(struct xa_state *, void *entry, unsigned int order, g= fp_t); #else @@ -1556,6 +1557,11 @@ static inline int xa_get_order(struct xarray *xa, un= signed long index) return 0; } =20 +static inline int xas_get_order(struct xa_state *xas) +{ + return 0; +} + static inline void xas_split(struct xa_state *xas, void *entry, unsigned int order) { diff --git a/lib/test_xarray.c b/lib/test_xarray.c index ebe2af2e072d..0efde8f93490 100644 --- a/lib/test_xarray.c +++ b/lib/test_xarray.c @@ -1984,6 +1984,39 @@ static noinline void check_get_order(struct xarray *= xa) } } =20 +static noinline void check_xas_get_order(struct xarray *xa) +{ + XA_STATE(xas, xa, 0); + + unsigned int max_order =3D IS_ENABLED(CONFIG_XARRAY_MULTI) ? 20 : 1; + unsigned int order; + unsigned long i, j; + + for (order =3D 0; order < max_order; order++) { + for (i =3D 0; i < 10; i++) { + xas_set_order(&xas, i << order, order); + do { + xas_lock(&xas); + xas_store(&xas, xa_mk_value(i)); + xas_unlock(&xas); + } while (xas_nomem(&xas, GFP_KERNEL)); + + for (j =3D i << order; j < (i + 1) << order; j++) { + xas_set_order(&xas, j, 0); + rcu_read_lock(); + xas_load(&xas); + XA_BUG_ON(xa, xas_get_order(&xas) !=3D order); + rcu_read_unlock(); + } + + xas_lock(&xas); + xas_set_order(&xas, i << order, order); + xas_store(&xas, NULL); + xas_unlock(&xas); + } + } +} + static noinline void check_destroy(struct xarray *xa) { unsigned long index; @@ -2035,6 +2068,7 @@ static int xarray_checks(void) check_multi_store(&array); check_multi_store_advanced(&array); check_get_order(&array); + check_xas_get_order(&array); check_xa_alloc(); check_find(&array); check_find_entry(&array); diff --git a/lib/xarray.c b/lib/xarray.c index 39f07bfc4dcc..da79128ad754 100644 --- a/lib/xarray.c +++ b/lib/xarray.c @@ -1750,39 +1750,52 @@ void *xa_store_range(struct xarray *xa, unsigned lo= ng first, EXPORT_SYMBOL(xa_store_range); =20 /** - * xa_get_order() - Get the order of an entry. - * @xa: XArray. - * @index: Index of the entry. + * xas_get_order() - Get the order of an entry. + * @xas: XArray operation state. + * + * Called after xas_load, the xas should not be in an error state. * * Return: A number between 0 and 63 indicating the order of the entry. */ -int xa_get_order(struct xarray *xa, unsigned long index) +int xas_get_order(struct xa_state *xas) { - XA_STATE(xas, xa, index); - void *entry; int order =3D 0; =20 - rcu_read_lock(); - entry =3D xas_load(&xas); - - if (!entry) - goto unlock; - - if (!xas.xa_node) - goto unlock; + if (!xas->xa_node) + return 0; =20 for (;;) { - unsigned int slot =3D xas.xa_offset + (1 << order); + unsigned int slot =3D xas->xa_offset + (1 << order); =20 if (slot >=3D XA_CHUNK_SIZE) break; - if (!xa_is_sibling(xas.xa_node->slots[slot])) + if (!xa_is_sibling(xa_entry(xas->xa, xas->xa_node, slot))) break; order++; } =20 - order +=3D xas.xa_node->shift; -unlock: + order +=3D xas->xa_node->shift; + return order; +} +EXPORT_SYMBOL_GPL(xas_get_order); + +/** + * xa_get_order() - Get the order of an entry. + * @xa: XArray. + * @index: Index of the entry. + * + * Return: A number between 0 and 63 indicating the order of the entry. + */ +int xa_get_order(struct xarray *xa, unsigned long index) +{ + XA_STATE(xas, xa, index); + int order =3D 0; + void *entry; + + rcu_read_lock(); + entry =3D xas_load(&xas); + if (entry) + order =3D xas_get_order(&xas); rcu_read_unlock(); =20 return order; --=20 2.44.0 From nobody Sat Feb 7 19:08:14 2026 Received: from mail-pf1-f172.google.com (mail-pf1-f172.google.com [209.85.210.172]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 80D8815A4B0 for ; Tue, 16 Apr 2024 07:17:55 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1713251877; cv=none; b=oMHhHhYGwe9Nm1ehTl6bEAMZhmV76KBpSv/7Tr+b1IkDpelUfD+upriCpCE4e07/tYhrlq/d379B8crxfjdk+n+bB8uQKgMJJH0pKzrwhiOXiLGITfny+Y1QFwsB30KCE2WZ1Ick4EwIDjetpARRnDmQkIF9eV0I31UZk8W7UyE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1713251877; c=relaxed/simple; bh=xgAGSD8LlG8q7U5tdKc40OWhuOBcjOO9Q+duQbQ6tqY=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=Rk5LQZq/QBRiITQLWuPrC5ufaAuBfANkBA1HlthkXpnhGswU05ui0ZKitDAJ2yM7oeLeOuKF2Iu+jo0CqlPQCYA0Z2/VcwRWYLyFpIrlGXlsyhksyowUy+FJjE3S2k6TgQGNVsKvr0LRKmC12cZB17xhnjuuXBJvj1HMJ1n1kiU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=Jb+HNugn; arc=none smtp.client-ip=209.85.210.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="Jb+HNugn" Received: by mail-pf1-f172.google.com with SMTP id d2e1a72fcca58-6eaf9565e6bso2501089b3a.2 for ; Tue, 16 Apr 2024 00:17:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1713251875; x=1713856675; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:from:to:cc:subject :date:message-id:reply-to; bh=2PyPfFBA1Tc3D6QFaHrtVUXv3LRzBa54pk/bPz9W1MA=; b=Jb+HNugnIzXB84hWk3mKnB0KEjxg5Uz7p30eBAYV4Ce/BsKqZbQWVxr43BRG6Ij+2d FXkamiqHJTQKoqZubU0oUzdn0HMLGoa5MkL3djBVv4k4sPITeD+n6MVFuY28Gpg4zHLz NvByH+uyMNU+xOImVwlzOLh5LF/wRrouMTd00QQJwg33ewxVjw93XNvRu5q3/DueRXM5 R3zu48aNI+799N8boeY13fyp+d2tiHAfqb2z/yW7ZCoGC+X0lW8qQ4QHpMVF6e+PmGju Nr1ziimEhEiRBfM1leEiSzGniMyXHfd3XY3K1fPVvRDkNrvmSW0lWb6jCtuPwdqNpOqg us5w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1713251875; x=1713856675; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=2PyPfFBA1Tc3D6QFaHrtVUXv3LRzBa54pk/bPz9W1MA=; b=InU4VTwxDGe3BIQx+2FFHvhSTg0Bdo8MCV2fea9GjIwhTZ/JPS3XRvmebSlLxjg8lZ PDcQvQa/pCLjhAuJtgylNdCf7/RXldVfNVhYkT7+vfOmcKTjYo/SVl42/cQ7YVjMkPy/ 8H6EzlrauimA56FmQW2teWSiblG2x4vV3LrV39L37KHpi/VXdvW4jmv//XHGI4CgEDEd dKp9shgkvMOjWlZhhSHtf67ZjEe/t04NIc5iZS3trSibi1zghOYLef+LQTqyMJX1IXpX ymlzLsDZmGiMTKXOnmKEjvV+Gb7IaMXpIQRlj1Y/h4ZQKVOI5YoG8S+8tllyIsKNwXgY aXlQ== X-Forwarded-Encrypted: i=1; AJvYcCVyzP5WDh3j8cIG2lHJV/E9doumfMQwaoEAQi6PFiz/FURvKmbu8qGygOuWRg24G/zaUyP/13oQooFuymA3TXLq6DoICHVD7wgIhzIi X-Gm-Message-State: AOJu0Yw1XluQHAK7PTaQwA6L5ScEqBLxzEg1xoCiFdWqeUKtDZBLoANm oZoSmQfO+JIUb2YNxxRVxWrGcbGDD5v3/dhap7an9NDMhEFI6ny3 X-Google-Smtp-Source: AGHT+IHr7oN8taByNZNOFuS818KVFfhROjhg9COx9F5A8nzf64zhs0XWSwf3xe5Glor8jesGCLKr/A== X-Received: by 2002:a05:6a00:2353:b0:6eb:4:3f26 with SMTP id j19-20020a056a00235300b006eb00043f26mr13968297pfj.18.1713251874742; Tue, 16 Apr 2024 00:17:54 -0700 (PDT) Received: from KASONG-MB2.tencent.com ([43.132.141.21]) by smtp.gmail.com with ESMTPSA id lr48-20020a056a0073b000b006e724ccdc3esm8340029pfb.55.2024.04.16.00.17.52 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Tue, 16 Apr 2024 00:17:54 -0700 (PDT) From: Kairui Song To: linux-mm@kvack.org Cc: Andrew Morton , Matthew Wilcox , linux-kernel@vger.kernel.org, Kairui Song Subject: [PATCH v4 4/4] mm/filemap: optimize filemap folio adding Date: Tue, 16 Apr 2024 15:17:22 +0800 Message-ID: <20240416071722.45997-5-ryncsn@gmail.com> X-Mailer: git-send-email 2.44.0 In-Reply-To: <20240416071722.45997-1-ryncsn@gmail.com> References: <20240416071722.45997-1-ryncsn@gmail.com> Reply-To: Kairui Song Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable From: Kairui Song Instead of doing multiple tree walks, do one optimism range check with lock hold, and exit if raced with another insertion. If a shadow exists, check it with a new xas_get_order helper before releasing the lock to avoid redundant tree walks for getting its order. Drop the lock and do the allocation only if a split is needed. In the best case, it only need to walk the tree once. If it needs to alloc and split, 3 walks are issued (One for first ranged conflict check and order retrieving, one for the second check after allocation, one for the insert after split). Testing with 4K pages, in an 8G cgroup, with 16G brd as block device: echo 3 > /proc/sys/vm/drop_caches fio -name=3Dcached --numjobs=3D16 --filename=3D/mnt/test.img \ --buffered=3D1 --ioengine=3Dmmap --rw=3Drandread --time_based \ --ramp_time=3D30s --runtime=3D5m --group_reporting Before: bw ( MiB/s): min=3D 1027, max=3D 3520, per=3D100.00%, avg=3D2445.02, stdev= =3D18.90, samples=3D8691 iops : min=3D263001, max=3D901288, avg=3D625924.36, stdev=3D4837.28,= samples=3D8691 After (+7.3%): bw ( MiB/s): min=3D 493, max=3D 3947, per=3D100.00%, avg=3D2625.56, stdev= =3D25.74, samples=3D8651 iops : min=3D126454, max=3D1010681, avg=3D672142.61, stdev=3D6590.48= , samples=3D8651 Test result with THP (do a THP randread then switch to 4K page in hope it issues a lot of splitting): echo 3 > /proc/sys/vm/drop_caches fio -name=3Dcached --numjobs=3D16 --filename=3D/mnt/test.img \ --buffered=3D1 --ioengine=3Dmmap -thp=3D1 --readonly \ --rw=3Drandread --time_based --ramp_time=3D30s --runtime=3D10m \ --group_reporting fio -name=3Dcached --numjobs=3D16 --filename=3D/mnt/test.img \ --buffered=3D1 --ioengine=3Dmmap \ --rw=3Drandread --time_based --runtime=3D5s --group_reporting Before: bw ( KiB/s): min=3D 4141, max=3D14202, per=3D100.00%, avg=3D7935.51, stdev= =3D96.85, samples=3D18976 iops : min=3D 1029, max=3D 3548, avg=3D1979.52, stdev=3D24.23, sampl= es=3D18976=C2=B7 READ: bw=3D4545B/s (4545B/s), 4545B/s-4545B/s (4545B/s-4545B/s), io=3D64.0K= iB (65.5kB), run=3D14419-14419msec After (+12.5%): bw ( KiB/s): min=3D 4611, max=3D15370, per=3D100.00%, avg=3D8928.74, stdev= =3D105.17, samples=3D19146 iops : min=3D 1151, max=3D 3842, avg=3D2231.27, stdev=3D26.29, sampl= es=3D19146 READ: bw=3D4635B/s (4635B/s), 4635B/s-4635B/s (4635B/s-4635B/s), io=3D64.0K= iB (65.5kB), run=3D14137-14137msec The performance is better for both 4K (+7.5%) and THP (+12.5%) cached read. Signed-off-by: Kairui Song --- lib/test_xarray.c | 59 +++++++++++++++++++++++++++++++++++++++++++++++ mm/filemap.c | 56 ++++++++++++++++++++++++++++++++------------ 2 files changed, 100 insertions(+), 15 deletions(-) diff --git a/lib/test_xarray.c b/lib/test_xarray.c index 0efde8f93490..8732a311f613 100644 --- a/lib/test_xarray.c +++ b/lib/test_xarray.c @@ -2017,6 +2017,64 @@ static noinline void check_xas_get_order(struct xarr= ay *xa) } } =20 +static noinline void check_xas_conflict_get_order(struct xarray *xa) +{ + XA_STATE(xas, xa, 0); + + void *entry; + int only_once; + unsigned int max_order =3D IS_ENABLED(CONFIG_XARRAY_MULTI) ? 20 : 1; + unsigned int order; + unsigned long i, j, k; + + for (order =3D 0; order < max_order; order++) { + for (i =3D 0; i < 10; i++) { + xas_set_order(&xas, i << order, order); + do { + xas_lock(&xas); + xas_store(&xas, xa_mk_value(i)); + xas_unlock(&xas); + } while (xas_nomem(&xas, GFP_KERNEL)); + + /* + * Ensure xas_get_order works with xas_for_each_conflict. + */ + j =3D i << order; + for (k =3D 0; k < order; k++) { + only_once =3D 0; + xas_set_order(&xas, j + (1 << k), k); + xas_lock(&xas); + xas_for_each_conflict(&xas, entry) { + XA_BUG_ON(xa, entry !=3D xa_mk_value(i)); + XA_BUG_ON(xa, xas_get_order(&xas) !=3D order); + only_once++; + } + XA_BUG_ON(xa, only_once !=3D 1); + xas_unlock(&xas); + } + + if (order < max_order - 1) { + only_once =3D 0; + xas_set_order(&xas, (i & ~1UL) << order, order + 1); + xas_lock(&xas); + xas_for_each_conflict(&xas, entry) { + XA_BUG_ON(xa, entry !=3D xa_mk_value(i)); + XA_BUG_ON(xa, xas_get_order(&xas) !=3D order); + only_once++; + } + XA_BUG_ON(xa, only_once !=3D 1); + xas_unlock(&xas); + } + + xas_set_order(&xas, i << order, order); + xas_lock(&xas); + xas_store(&xas, NULL); + xas_unlock(&xas); + } + } +} + + static noinline void check_destroy(struct xarray *xa) { unsigned long index; @@ -2069,6 +2127,7 @@ static int xarray_checks(void) check_multi_store_advanced(&array); check_get_order(&array); check_xas_get_order(&array); + check_xas_conflict_get_order(&array); check_xa_alloc(); check_find(&array); check_find_entry(&array); diff --git a/mm/filemap.c b/mm/filemap.c index 7f734b7ad273..00177ecdc4a4 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -852,7 +852,9 @@ noinline int __filemap_add_folio(struct address_space *= mapping, struct folio *folio, pgoff_t index, gfp_t gfp, void **shadowp) { XA_STATE(xas, &mapping->i_pages, index); - bool huge =3D folio_test_hugetlb(folio); + void *alloced_shadow =3D NULL; + int alloced_order =3D 0; + bool huge; long nr; =20 VM_BUG_ON_FOLIO(!folio_test_locked(folio), folio); @@ -861,6 +863,7 @@ noinline int __filemap_add_folio(struct address_space *= mapping, =20 VM_BUG_ON_FOLIO(index & (folio_nr_pages(folio) - 1), folio); xas_set_order(&xas, index, folio_order(folio)); + huge =3D folio_test_hugetlb(folio); nr =3D folio_nr_pages(folio); =20 gfp &=3D GFP_RECLAIM_MASK; @@ -868,16 +871,10 @@ noinline int __filemap_add_folio(struct address_space= *mapping, folio->mapping =3D mapping; folio->index =3D xas.xa_index; =20 - do { - unsigned int order =3D xa_get_order(xas.xa, xas.xa_index); + for (;;) { + int order =3D -1, split_order =3D 0; void *entry, *old =3D NULL; =20 - if (order > folio_order(folio)) { - xas_split_alloc(&xas, xa_load(xas.xa, xas.xa_index), - order, gfp); - if (xas_error(&xas)) - goto error; - } xas_lock_irq(&xas); xas_for_each_conflict(&xas, entry) { old =3D entry; @@ -885,19 +882,33 @@ noinline int __filemap_add_folio(struct address_space= *mapping, xas_set_err(&xas, -EEXIST); goto unlock; } + /* + * If a larger entry exists, + * it will be the first and only entry iterated. + */ + if (order =3D=3D -1) + order =3D xas_get_order(&xas); + } + + /* entry may have changed before we re-acquire the lock */ + if (alloced_order && (old !=3D alloced_shadow || order !=3D alloced_orde= r)) { + xas_destroy(&xas); + alloced_order =3D 0; } =20 if (old) { - if (shadowp) - *shadowp =3D old; - /* entry may have been split before we acquired lock */ - order =3D xa_get_order(xas.xa, xas.xa_index); - if (order > folio_order(folio)) { + if (order > 0 && order > folio_order(folio)) { /* How to handle large swap entries? */ BUG_ON(shmem_mapping(mapping)); + if (!alloced_order) { + split_order =3D order; + goto unlock; + } xas_split(&xas, old, order); xas_reset(&xas); } + if (shadowp) + *shadowp =3D old; } =20 xas_store(&xas, folio); @@ -913,9 +924,24 @@ noinline int __filemap_add_folio(struct address_space = *mapping, __lruvec_stat_mod_folio(folio, NR_FILE_THPS, nr); } + unlock: xas_unlock_irq(&xas); - } while (xas_nomem(&xas, gfp)); + + /* split needed, alloc here and retry. */ + if (split_order) { + xas_split_alloc(&xas, old, split_order, gfp); + if (xas_error(&xas)) + goto error; + alloced_shadow =3D old; + alloced_order =3D split_order; + xas_reset(&xas); + continue; + } + + if (!xas_nomem(&xas, gfp)) + break; + } =20 if (xas_error(&xas)) goto error; --=20 2.44.0