From nobody Sun Feb 8 05:59:58 2026 Received: from mail-il1-f177.google.com (mail-il1-f177.google.com [209.85.166.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 36FFF13D527 for ; Mon, 25 Mar 2024 17:14:14 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.166.177 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711386855; cv=none; b=Dw3DPkCvzk0Yyr1upAd2oFuSv8h99/drC4MsGFhR6JNnC84IzIHaRmzh01/46cxGtSXQlXBMM/4gae/vFzZmkvB8Q0kGiLxsxMB95q5GRAYa9jcqZYG1OMdYltJwiBmaWqlabuB6Rf/niTupJvG4G+fvM74lmOn9Dl0Our6/Ek8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711386855; c=relaxed/simple; bh=bkeP5sDD48wsxQBkk56+ZqBCPj2agvXD72gbBDxPfKs=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=E5KRDdgo2SBmMSFoiRrw+nEl3yrRrKON6YYyJInfujVWosUh62DpONpo9PfyTtJJ3Ek/elz9gy02CqNzF9i9hG3Guwzyk0ppCIisNDrvQed/R9pf2YO2KIg77pS2yJRST3H1PUNgRv0t4CUfssqgYpVIeAbo+fkLgpjQ7ldm3yo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=fJhdmy0M; arc=none smtp.client-ip=209.85.166.177 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="fJhdmy0M" Received: by mail-il1-f177.google.com with SMTP id e9e14a558f8ab-366ac8bdd2cso21512035ab.3 for ; Mon, 25 Mar 2024 10:14:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1711386853; x=1711991653; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:from:to:cc:subject :date:message-id:reply-to; bh=rSbkdxxh6gTQxH+/m3hblAGdhy4V4YDv1roCho2fnaw=; b=fJhdmy0MIHr/OclmSr7AePrjg347BBFoYzU01KFiQFzpTHTQE6WJB1J5zBjxt/dmM1 KaSM23MjuQuJEFs5+mEU6mD1X5phcya84FNykeUrb9TgV15hTFDh3Im6So+xj3r9Jxa2 I9f4XmNaT/FmJ1TDlxhXOUrtok5R6WfFV/ecgsT51D6MWBJqq7PcPGjzhn9kw+ellw+p 7Afc32T7mB5Jd7GTXGRUmDXctGLTbhKrq8eGWCZ+h0ZeRlDklkrLeDQqh2d7k6NQ10JV cysQ1foF3Zcch917YLBS3XK1dHVTahRH/l+EZkNRpLgrTfLK2KbECIxqEUVeF+FMrqgL rGOA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711386853; x=1711991653; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=rSbkdxxh6gTQxH+/m3hblAGdhy4V4YDv1roCho2fnaw=; b=XvBJ7huOD9MkTO9G5uN2pj8sBWhADCi62sgaA5GPEZmG7Dxyon8XAaC9+mTVmk7YWl jgiUq3gNC1q9Gt2Myb5+CGK3uZmPlkc50sO3QR9oE4DU7jsWHxl2nNkTvNqqhGF/iF+L Z+NEXlLCAsJWnzHn9Ydbtjy6EK3k0KY4paS3T3iQmAxE8IajGwbY4sIdUB/wnlI8EUV8 afcC88NGiDWFH0YkCvG2C+8QwCtLOC82q7dcrLJBNvIH24vG74Ta/us25TUlnETYXbpd ehG4C9gXb4h4mz1aJxyOeawou68INcXip2+yoXX8wmB8B14fJkX1qQ48EYBrEN3OOpTV TEew== X-Forwarded-Encrypted: i=1; AJvYcCWlFZwVXBI9TWwBRV2nRthl1lJXUE+BWpiisygjUAvZLrO+uZ8Z6wwBS++YKfuIP5X/cLq5NVGN1FPB8ER/sdNS4HxowzdY1jHM/HQl X-Gm-Message-State: AOJu0Yz5oVhPEKRjSbVdEm0Ue2ys4zfrt7XO/Ot4m31c1vCO0q69+KDq LvKfvrxtnTOg5Rel+3vBME+26Hi+IUMrechoa1OgDE/mm84JJnhH X-Google-Smtp-Source: AGHT+IHSMojcSRK5YMTPHTkFJtuXwtvofR+6Do+4UqVrwzKlW3e5+hTuNcI2HZhXYkvzvs6sSeSj4Q== X-Received: by 2002:a92:c983:0:b0:368:9ba9:fcb5 with SMTP id y3-20020a92c983000000b003689ba9fcb5mr604940iln.4.1711386853393; Mon, 25 Mar 2024 10:14:13 -0700 (PDT) Received: from KASONG-MB2.tencent.com ([115.171.40.106]) by smtp.gmail.com with ESMTPSA id r16-20020a63d910000000b005dc5129ba9dsm6001812pgg.72.2024.03.25.10.14.11 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Mon, 25 Mar 2024 10:14:12 -0700 (PDT) From: Kairui Song To: linux-mm@kvack.org Cc: Matthew Wilcox , Andrew Morton , linux-kernel@vger.kernel.org, Kairui Song Subject: [PATCH v2 1/4] mm/filemap: return early if failed to allocate memory for split Date: Tue, 26 Mar 2024 01:14:02 +0800 Message-ID: <20240325171405.99971-2-ryncsn@gmail.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240325171405.99971-1-ryncsn@gmail.com> References: <20240325171405.99971-1-ryncsn@gmail.com> Reply-To: Kairui Song Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Kairui Song xas_split_alloc could fail with NOMEM, and in such case, it should abort early instead of keep going and fail the xas_split below. Signed-off-by: Kairui Song Acked-by: Matthew Wilcox (Oracle) --- mm/filemap.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/mm/filemap.c b/mm/filemap.c index 7437b2bd75c1..f07ea0b97698 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -880,9 +880,12 @@ noinline int __filemap_add_folio(struct address_space = *mapping, unsigned int order =3D xa_get_order(xas.xa, xas.xa_index); void *entry, *old =3D NULL; =20 - if (order > folio_order(folio)) + if (order > folio_order(folio)) { xas_split_alloc(&xas, xa_load(xas.xa, xas.xa_index), order, gfp); + if (xas_error(&xas)) + goto error; + } xas_lock_irq(&xas); xas_for_each_conflict(&xas, entry) { old =3D entry; --=20 2.43.0 From nobody Sun Feb 8 05:59:58 2026 Received: from mail-il1-f172.google.com (mail-il1-f172.google.com [209.85.166.172]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6380113D605 for ; Mon, 25 Mar 2024 17:14:16 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.166.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711386857; cv=none; b=HqhJNQDxJzNI5hVHoDuHkS1sor1oKS6qD+S36djQjBKAjh8F1a0qmIz+BYYHrLpwrSgKCQ9fe9iCYGUDNgLKZaYoD0XRjiVtMFI5bkgdRezVd4P+uOaR37d1vNpzMwMtWrDoEs1ArFcss0Ce2eDfxvWuq/kuUDABMU2tKhHcgjM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711386857; c=relaxed/simple; bh=LWlwlamGO0rfQ1EM8aLmO1k7D9FshXdChnWs/pBPSrU=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=OUgiJLmpOaSgX9lnXpU5OdMFQ9c6kIa01unmjDuFnforvWTtjWK15HB8f9guI0a5SirDW9at8bktMyFNvB89MAg5iG3cE8ZSLDy4xeh2uwv586bmdeXxnXOXAmL5nfFg31U51UgdS9yrYgDaJtXJiKnXa4rc26TTe82mZJaNAtM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=VBCS0I9R; arc=none smtp.client-ip=209.85.166.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="VBCS0I9R" Received: by mail-il1-f172.google.com with SMTP id e9e14a558f8ab-3688d20b28dso5512445ab.2 for ; Mon, 25 Mar 2024 10:14:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1711386855; x=1711991655; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:from:to:cc:subject :date:message-id:reply-to; bh=bN9a48YnOEXH7sPm0Ke4Oa5WKVmRd1RkDjnQJHDaF+o=; b=VBCS0I9RaAf56C8UqRa/PYanL+cxS52kVuGhIBQSlEb7aZh3l/xQoBJMsyhgcA7rwl h7rGkp2+L4x2/Wbt1FoRe3gtqJ7IC6Tm5CKko10CpCZzElnRGPC70tCQ3gAqh2JwvpR8 f1MwPzlC9/C44ffV7JHkvl24pd20V5vfMcf5Auk+zlEbKwCRfzoQWmaf/5TJwassnEu3 yihejF6TfMKUMsHEQil4u7u9HdFhkb5hlcVoocRcDkbkWeDRFRI+3ysjUEcCHjubWKDQ a9hFBkTwWm4WI/aypQLgiQbHpdUWPLq76yFPWPRpa2oBeNHxwhEDuGaL6j+oiIcg/Ozr SeeA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711386855; x=1711991655; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=bN9a48YnOEXH7sPm0Ke4Oa5WKVmRd1RkDjnQJHDaF+o=; b=X1UP9SDeMZIHyA1pZW6HO5IDEFbZuMkO20W7jp2A3AxwEEewLQZArD1FWjXhPkqbNb 5cNb1GOBkRcTzqREbHbtHrQOazq8qvkozibSfdYUaZ5ESRGHpZ5+jJ4pXqv1olNWK4JI i5Qc5DHwDnGTwEjqfKZxprQIbocm/0eCHIps72iEidzW1E/6bSVadSMwbjIrn8eIGz5i JHAH7rWiNSslrv1PMNnIgvjZ+iMc/cYJsKkKj8hhnB0aGEmahnUacqONyZ3ZeGBjUIZq w07TSVl0NPkLEtGiDLx3yxuku7tR3yTYvdNoF6BxQQLNappiHp9WOrTeGXLzvZtkhw9O L17g== X-Forwarded-Encrypted: i=1; AJvYcCUUrZVUV36eWN+SeJtXWy3mPQLd9uuCM9RDsjctdU7A5+TKgPjK5osC8zhRU7txKp0scZ4KFfX8vIhmMqUGpJOq8Lrd8R0jfvGKTmOD X-Gm-Message-State: AOJu0Yx2Y20Q/Isne93dxqQ4Fk5CTX9l+utzD3Rr+7o18cbQ+CvUoJRQ j9COdhink0gqQT1Kxs4xrlrZySl6LeruIsJjMrBqJw3fBAgmHdEZ X-Google-Smtp-Source: AGHT+IEV6m/IQGGduQkAd1EXsYeOBd+/asYh5Dr5GFoEqy3C5fYnTIRwAYULWlk6rk/10OQ+ts/pmA== X-Received: by 2002:a92:c883:0:b0:368:4a8f:85f5 with SMTP id w3-20020a92c883000000b003684a8f85f5mr7282293ilo.19.1711386855551; Mon, 25 Mar 2024 10:14:15 -0700 (PDT) Received: from KASONG-MB2.tencent.com ([115.171.40.106]) by smtp.gmail.com with ESMTPSA id r16-20020a63d910000000b005dc5129ba9dsm6001812pgg.72.2024.03.25.10.14.13 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Mon, 25 Mar 2024 10:14:15 -0700 (PDT) From: Kairui Song To: linux-mm@kvack.org Cc: Matthew Wilcox , Andrew Morton , linux-kernel@vger.kernel.org, Kairui Song Subject: [PATCH v2 2/4] mm/filemap: clean up hugetlb exclusion code Date: Tue, 26 Mar 2024 01:14:03 +0800 Message-ID: <20240325171405.99971-3-ryncsn@gmail.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240325171405.99971-1-ryncsn@gmail.com> References: <20240325171405.99971-1-ryncsn@gmail.com> Reply-To: Kairui Song Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Kairui Song __filemap_add_folio only has two callers, one never passes hugetlb folio and one always passes in hugetlb folio. So move the hugetlb related cgroup charging out of it to make the code cleaner. Signed-off-by: Kairui Song Acked-by: Matthew Wilcox (Oracle) --- mm/filemap.c | 21 ++++++++------------- 1 file changed, 8 insertions(+), 13 deletions(-) diff --git a/mm/filemap.c b/mm/filemap.c index f07ea0b97698..6bbec8783793 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -853,20 +853,12 @@ noinline int __filemap_add_folio(struct address_space= *mapping, { XA_STATE(xas, &mapping->i_pages, index); bool huge =3D folio_test_hugetlb(folio); - bool charged =3D false; - long nr =3D 1; + long nr; =20 VM_BUG_ON_FOLIO(!folio_test_locked(folio), folio); VM_BUG_ON_FOLIO(folio_test_swapbacked(folio), folio); mapping_set_update(&xas, mapping); =20 - if (!huge) { - int error =3D mem_cgroup_charge(folio, NULL, gfp); - if (error) - return error; - charged =3D true; - } - VM_BUG_ON_FOLIO(index & (folio_nr_pages(folio) - 1), folio); xas_set_order(&xas, index, folio_order(folio)); nr =3D folio_nr_pages(folio); @@ -931,8 +923,6 @@ noinline int __filemap_add_folio(struct address_space *= mapping, trace_mm_filemap_add_to_page_cache(folio); return 0; error: - if (charged) - mem_cgroup_uncharge(folio); folio->mapping =3D NULL; /* Leave page->index set: truncation relies upon it */ folio_put_refs(folio, nr); @@ -946,11 +936,16 @@ int filemap_add_folio(struct address_space *mapping, = struct folio *folio, void *shadow =3D NULL; int ret; =20 + ret =3D mem_cgroup_charge(folio, NULL, gfp); + if (ret) + return ret; + __folio_set_locked(folio); ret =3D __filemap_add_folio(mapping, folio, index, gfp, &shadow); - if (unlikely(ret)) + if (unlikely(ret)) { + mem_cgroup_uncharge(folio); __folio_clear_locked(folio); - else { + } else { /* * The folio might have been evicted from cache only * recently, in which case it should be activated like --=20 2.43.0 From nobody Sun Feb 8 05:59:58 2026 Received: from mail-oo1-f49.google.com (mail-oo1-f49.google.com [209.85.161.49]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D977613D527 for ; Mon, 25 Mar 2024 17:14:18 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.161.49 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711386860; cv=none; b=cFLy/BrcJL5tcsYMZYqvBrefWKlybpVmYivfBv4kzdV0dqiVD5SUmWmQnisC21W0ISNzi7XwJZyEzTNNHvEhbR77j3Hp1OgDwstAfAca6PM78a+PTllVA4lOubNPx2uq3ChRFqayYjbYQyt8Ch8N7aeI8+03Fc/Nka0yYLNhlK8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711386860; c=relaxed/simple; bh=4PYRdPCYyOAk+cgs0UXWoGo/KtFhnlDPC531UHMVdfM=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=JpaJkmxQZrfF0J9wWcMNUIeZ8HuVsJ8MfuJZjE2BjhMQc6U/4vGoVsq7jcEeMXAmHyrVGgj5cJIX2hDdKC/03iT8x9awcXBmXD+jO5TCm4Z2cCCBNSvoJhvnI4ADWyIkS9sGQPkJTUouAL1ofAWK+az/swRlbQD+q6mSsjMqLJM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=eNb+/ZxD; arc=none smtp.client-ip=209.85.161.49 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="eNb+/ZxD" Received: by mail-oo1-f49.google.com with SMTP id 006d021491bc7-5a480985625so2524211eaf.0 for ; Mon, 25 Mar 2024 10:14:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1711386858; x=1711991658; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:from:to:cc:subject :date:message-id:reply-to; bh=6sDmtk9HBtK4D63m2Orr58T2syAmM3af2Tjjpj9AWXM=; b=eNb+/ZxDid90Kj5mUDd2MqFSpSG7NDIb3ggBlmRFc7An4iwlgeZn84SgQKbeqtaZdJ hNM+rUOnvXsgxsGtmNJnY7iiog2dpkGy2oo9qgzdHYGJefLxW1RGROeeMUAeT0J8/9SU 3xiKbejEliAHcgbo8+E2xJyBrmkSKOv0qGdyAUj5DdkcW2JU/V9VWzf3vzQ3EsM6paoE xV8vNIGyXjmLR9koJeaE7qJktHl2pgqnMw1O9idAci1ERb83F798iB+sgo739exH6oPx LsPDZxIjLl/h/8vccJnLxGsKnVuzxdu2w7YMgwc7g+BIkQU3NttYo8ImvNt2viQo2Dyg dFBA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711386858; x=1711991658; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=6sDmtk9HBtK4D63m2Orr58T2syAmM3af2Tjjpj9AWXM=; b=oAcxEZd9XKxBbFAH9LEchfqnuVf/j8rqMhpvKzbrMoJ1/ZUTs4ssiC27tUoxNBHIC5 YqqKfVRTI/O1Y9ulorwJUg4UAp8Ly7EvZkTeYaZtdRHn/U+cnxfQrCN2i9+zzRyxE/IH qh2UbQyA75Owo2GXAvuZhgu8YHSUk8wGMGeWzHXCWq0rYAzwdXFobhraTCqxADAO3Ka5 7LELDR0NiEP4HoCQ3rzxKwpuGigaSkxhaHU0hoP1wqAhFeYmka7aXNPni4H8KuEPbq7C D6CA6MwPKdKsyMGMHcVXdGzgETqaaqcCgpTQKKJQNTm5VRhvxl3OHr1ThIOW0Yu7O/WT qF+Q== X-Forwarded-Encrypted: i=1; AJvYcCXWmm08FFmoJxJi8YkBpFhGk9erACDzQYJIOTT0noxKF40T4/b+Y6hE30F4mdaYwlSCyEkp4+egdWklfXM9BUJz36bQMJOfJHAGo/HA X-Gm-Message-State: AOJu0YwBhvizw34F516WsUcK6pqX5hLkGF1ZwLgnep41Xf+b9j+8OG9f h92bxTUDxRO6+NLbt1tsKImNrw9bH1+KzX7lgIy1yk2dYK8fUqjKuqmSPaFJ6apK7oSb X-Google-Smtp-Source: AGHT+IH812g8lEhOIH3UUXbXCSKLPkSjpe82QohMIQkOyOH031CL/t5qi83Q7yr/40ax5hVVYS2Ijg== X-Received: by 2002:a05:6358:3996:b0:17e:9ca7:c3a2 with SMTP id b22-20020a056358399600b0017e9ca7c3a2mr9245069rwe.21.1711386857956; Mon, 25 Mar 2024 10:14:17 -0700 (PDT) Received: from KASONG-MB2.tencent.com ([115.171.40.106]) by smtp.gmail.com with ESMTPSA id r16-20020a63d910000000b005dc5129ba9dsm6001812pgg.72.2024.03.25.10.14.15 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Mon, 25 Mar 2024 10:14:17 -0700 (PDT) From: Kairui Song To: linux-mm@kvack.org Cc: Matthew Wilcox , Andrew Morton , linux-kernel@vger.kernel.org, Kairui Song Subject: [PATCH v2 3/4] lib/xarray: introduce a new helper xas_get_order Date: Tue, 26 Mar 2024 01:14:04 +0800 Message-ID: <20240325171405.99971-4-ryncsn@gmail.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240325171405.99971-1-ryncsn@gmail.com> References: <20240325171405.99971-1-ryncsn@gmail.com> Reply-To: Kairui Song Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Kairui Song It can be used after xas_load to check the order of loaded entries. Compared to xa_get_order, it saves an XA_STATE and avoid a rewalk. Also add new test for xas_get_order, to make the test work we have to export xas_get_order with EXPORT_SYMBOL_GPL. Signed-off-by: Kairui Song --- include/linux/xarray.h | 6 ++++++ lib/test_xarray.c | 32 +++++++++++++++++++++++++++ lib/xarray.c | 49 ++++++++++++++++++++++++++---------------- 3 files changed, 69 insertions(+), 18 deletions(-) diff --git a/include/linux/xarray.h b/include/linux/xarray.h index cb571dfcf4b1..d9d479334c9e 100644 --- a/include/linux/xarray.h +++ b/include/linux/xarray.h @@ -1548,6 +1548,7 @@ void xas_create_range(struct xa_state *); =20 #ifdef CONFIG_XARRAY_MULTI int xa_get_order(struct xarray *, unsigned long index); +int xas_get_order(struct xa_state *xas); void xas_split(struct xa_state *, void *entry, unsigned int order); void xas_split_alloc(struct xa_state *, void *entry, unsigned int order, g= fp_t); #else @@ -1556,6 +1557,11 @@ static inline int xa_get_order(struct xarray *xa, un= signed long index) return 0; } =20 +static inline int xas_get_order(struct xa_state *xas) +{ + return 0; +} + static inline void xas_split(struct xa_state *xas, void *entry, unsigned int order) { diff --git a/lib/test_xarray.c b/lib/test_xarray.c index ebe2af2e072d..26e28b65d60a 100644 --- a/lib/test_xarray.c +++ b/lib/test_xarray.c @@ -1984,6 +1984,37 @@ static noinline void check_get_order(struct xarray *= xa) } } =20 +static noinline void check_xas_get_order(struct xarray *xa) +{ + XA_STATE(xas, xa, 0); + + unsigned int max_order =3D IS_ENABLED(CONFIG_XARRAY_MULTI) ? 20 : 1; + unsigned int order; + unsigned long i, j; + + for (order =3D 0; order < max_order; order++) { + for (i =3D 0; i < 10; i++) { + xas_set_order(&xas, i << order, order); + do { + xas_lock(&xas); + xas_store(&xas, xa_mk_value(i)); + xas_unlock(&xas); + } while (xas_nomem(&xas, GFP_KERNEL)); + + xas_lock(&xas); + for (j =3D i << order; j < (i + 1) << order; j++) { + xas_set_order(&xas, j, 0); + xas_load(&xas); + XA_BUG_ON(xa, xas_get_order(&xas) !=3D order); + } + xas_unlock(&xas); + + xas_set_order(&xas, i << order, order); + xas_store(&xas, NULL); + } + } +} + static noinline void check_destroy(struct xarray *xa) { unsigned long index; @@ -2035,6 +2066,7 @@ static int xarray_checks(void) check_multi_store(&array); check_multi_store_advanced(&array); check_get_order(&array); + check_xas_get_order(&array); check_xa_alloc(); check_find(&array); check_find_entry(&array); diff --git a/lib/xarray.c b/lib/xarray.c index 39f07bfc4dcc..fbf1d1dd83bc 100644 --- a/lib/xarray.c +++ b/lib/xarray.c @@ -1750,39 +1750,52 @@ void *xa_store_range(struct xarray *xa, unsigned lo= ng first, EXPORT_SYMBOL(xa_store_range); =20 /** - * xa_get_order() - Get the order of an entry. - * @xa: XArray. - * @index: Index of the entry. + * xas_get_order() - Get the order of an loaded entry after xas_load. + * @xas: XArray operation state. + * + * Called after xas_load, the xas should not be in an error state. * * Return: A number between 0 and 63 indicating the order of the entry. */ -int xa_get_order(struct xarray *xa, unsigned long index) +int xas_get_order(struct xa_state *xas) { - XA_STATE(xas, xa, index); - void *entry; int order =3D 0; =20 - rcu_read_lock(); - entry =3D xas_load(&xas); - - if (!entry) - goto unlock; - - if (!xas.xa_node) - goto unlock; + if (!xas->xa_node) + return 0; =20 for (;;) { - unsigned int slot =3D xas.xa_offset + (1 << order); + unsigned int slot =3D xas->xa_offset + (1 << order); =20 if (slot >=3D XA_CHUNK_SIZE) break; - if (!xa_is_sibling(xas.xa_node->slots[slot])) + if (!xa_is_sibling(xas->xa_node->slots[slot])) break; order++; } =20 - order +=3D xas.xa_node->shift; -unlock: + order +=3D xas->xa_node->shift; + return order; +} +EXPORT_SYMBOL_GPL(xas_get_order); + +/** + * xa_get_order() - Get the order of an entry. + * @xa: XArray. + * @index: Index of the entry. + * + * Return: A number between 0 and 63 indicating the order of the entry. + */ +int xa_get_order(struct xarray *xa, unsigned long index) +{ + XA_STATE(xas, xa, index); + int order =3D 0; + void *entry; + + rcu_read_lock(); + entry =3D xas_load(&xas); + if (entry) + order =3D xas_get_order(&xas); rcu_read_unlock(); =20 return order; --=20 2.43.0 From nobody Sun Feb 8 05:59:58 2026 Received: from mail-pf1-f171.google.com (mail-pf1-f171.google.com [209.85.210.171]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 199C713E040 for ; Mon, 25 Mar 2024 17:14:21 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.171 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711386863; cv=none; b=TEQDtR5g2Ti2xmWFYLRsGPUaEEkdpl0Mspf0OK4MhSvr8fdU0hKZsBkJmUftjhoCEhWIlV+umxThBnPCPUJ0K5/NWdgkC3DGRBgNy1cWywZCBZkhKGCd5SCJmxvj0PZ3fxUiSH9S11dQZkDXRJLiCSXCLh9nL8UxTiS9GMZMexw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711386863; c=relaxed/simple; bh=iPw0XrxbGujTBmrqCR3womr127ESmPQ9iYgG7xlX2ek=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=nQi3PcM0/GXN9aHOeKTMQdQn2zI7IGbYW56Tqf71vFnbHG82fQtVHV2F1xeePjbRcAVA0itSMWuJuwpiPH2rQnT3JVaav62KRjlAyl1DPrr+qjNnc24H3ULCgE5sC4ZQ2BYv7zrYCpgAD5yXzIeQRjmAdg13OMww394U9j+Exc8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=L9PMTRzo; arc=none smtp.client-ip=209.85.210.171 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="L9PMTRzo" Received: by mail-pf1-f171.google.com with SMTP id d2e1a72fcca58-6e6c0098328so3054255b3a.3 for ; Mon, 25 Mar 2024 10:14:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1711386861; x=1711991661; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:from:to:cc:subject :date:message-id:reply-to; bh=bxkVFzSJVoLggCSIdplAlTRqteX3zHBKMrMrFYGST98=; b=L9PMTRzoDJZN3HSwEq+F8FTMEIFQ8bq011jjck3UWbdMWvbKZ1bwEnR+LKRcuRxvcn nRu60PfgSgtwU8iYf4Bk6jYt8ZI/JAQTR0hQERuEzgtb2U+wQY6L90XagPRjx/OM70VX MhSjLt4gb8NRbbqyRVqxOMGKakqxbHDGIvHW8XjOA2wTeC3fpcrJNl7trZNWZLTjiOwD uYx+AmgPsut+lUHc2fuijfPCyIssJrjEn1pk/KxdsqOSFeYik1OTqlGZQvgyCFHGrWUG NzPfJQSUoMJBpSuHcskn3aHmUEJ7FnV9XuKUg+H/a3W4aWZ4oh/HXPpOyqyY//GqDdMM kruQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711386861; x=1711991661; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=bxkVFzSJVoLggCSIdplAlTRqteX3zHBKMrMrFYGST98=; b=k5ANQDIAI7APOaABOKfWuG7jTuPD8CiT6UKgcexaHKDg0OIKVCMUGLpw8WbL9tjun+ 7hpv3bCyZuzrWPhiMk0wiRp5FL9Q1tPyoioJx7SBZmHS/yXVcYrkQt5TmAF0n7/MV7Mu lu/SeMnamZf1mbBgabZIsoAineliu71kCp7FcQ4csUrRn/3iKylLhDrP47Su3aE/1Nt/ KPtsusm6flT2maq8c3bcx3CZLj7tP0IFkb9V3s9moQ6FzHtQCvZ7CuAflBg0EUiS96WE NQxWYnOLJ/PKEqOevog7igDIcEjAtvILZrJAsdpqM83OYn/8sxkCPs4clbRRehj3sKw3 NOOg== X-Forwarded-Encrypted: i=1; AJvYcCUk7YKmDzLleu82jRSi3h/+ug6ZevKyNnMF2HKxnE+s+PM9z5QQR5QYaOK6TddoAhc2lI02e146GeLVjFsbtHHY1/Xbc9ytgTp0Lqam X-Gm-Message-State: AOJu0Yzr6NOXQehdZfFakvL8kuEnu1wrd5JjYfWQ9BGd/wGjqIp50cIz z1bkEPUuVoaGMVgLGqa6ZdNnChOvYqs1KhHAN7blKdHq2EY+6y3P X-Google-Smtp-Source: AGHT+IHqt40rBnvxQlAYOk3f3IH9W03fIAB5lQLk0ER2F//3YE3Sl2ab/cKHdSL/dz1j/62/GlTTNw== X-Received: by 2002:a05:6a00:399f:b0:6e6:8b59:1bad with SMTP id fi31-20020a056a00399f00b006e68b591badmr8163470pfb.25.1711386860921; Mon, 25 Mar 2024 10:14:20 -0700 (PDT) Received: from KASONG-MB2.tencent.com ([115.171.40.106]) by smtp.gmail.com with ESMTPSA id r16-20020a63d910000000b005dc5129ba9dsm6001812pgg.72.2024.03.25.10.14.18 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Mon, 25 Mar 2024 10:14:20 -0700 (PDT) From: Kairui Song To: linux-mm@kvack.org Cc: Matthew Wilcox , Andrew Morton , linux-kernel@vger.kernel.org, Kairui Song Subject: [PATCH v2 4/4] mm/filemap: optimize filemap folio adding Date: Tue, 26 Mar 2024 01:14:05 +0800 Message-ID: <20240325171405.99971-5-ryncsn@gmail.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240325171405.99971-1-ryncsn@gmail.com> References: <20240325171405.99971-1-ryncsn@gmail.com> Reply-To: Kairui Song Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable From: Kairui Song Instead of doing multiple tree walks, do one optimism range check with lock hold, and exit if raced with another insertion. If a shadow exists, check it with a new xas_get_order helper before releasing the lock to avoid redundant tree walks for getting its order. Drop the lock and do the allocation only if a split is needed. In the best case, it only need to walk the tree once. If it needs to alloc and split, 3 walks are issued (One for first ranged conflict check and order retrieving, one for the second check after allocation, one for the insert after split). Testing with 4K pages, in an 8G cgroup, with 16G brd as block device: echo 3 > /proc/sys/vm/drop_caches fio -name=3Dcached --numjobs=3D16 --filename=3D/mnt/test.img \ --buffered=3D1 --ioengine=3Dmmap --rw=3Drandread --time_based \ --ramp_time=3D30s --runtime=3D5m --group_reporting Before: bw ( MiB/s): min=3D 1027, max=3D 3520, per=3D100.00%, avg=3D2445.02, stdev= =3D18.90, samples=3D8691 iops : min=3D263001, max=3D901288, avg=3D625924.36, stdev=3D4837.28,= samples=3D8691 After (+7.3%): bw ( MiB/s): min=3D 493, max=3D 3947, per=3D100.00%, avg=3D2625.56, stdev= =3D25.74, samples=3D8651 iops : min=3D126454, max=3D1010681, avg=3D672142.61, stdev=3D6590.48= , samples=3D8651 Test result with THP (do a THP randread then switch to 4K page in hope it issues a lot of splitting): echo 3 > /proc/sys/vm/drop_caches fio -name=3Dcached --numjobs=3D16 --filename=3D/mnt/test.img \ --buffered=3D1 --ioengine=3Dmmap -thp=3D1 --readonly \ --rw=3Drandread --time_based --ramp_time=3D30s --runtime=3D10m \ --group_reporting fio -name=3Dcached --numjobs=3D16 --filename=3D/mnt/test.img \ --buffered=3D1 --ioengine=3Dmmap \ --rw=3Drandread --time_based --runtime=3D5s --group_reporting Before: bw ( KiB/s): min=3D 4141, max=3D14202, per=3D100.00%, avg=3D7935.51, stdev= =3D96.85, samples=3D18976 iops : min=3D 1029, max=3D 3548, avg=3D1979.52, stdev=3D24.23, sampl= es=3D18976=C2=B7 READ: bw=3D4545B/s (4545B/s), 4545B/s-4545B/s (4545B/s-4545B/s), io=3D64.0K= iB (65.5kB), run=3D14419-14419msec After (+12.5%): bw ( KiB/s): min=3D 4611, max=3D15370, per=3D100.00%, avg=3D8928.74, stdev= =3D105.17, samples=3D19146 iops : min=3D 1151, max=3D 3842, avg=3D2231.27, stdev=3D26.29, sampl= es=3D19146 READ: bw=3D4635B/s (4635B/s), 4635B/s-4635B/s (4635B/s-4635B/s), io=3D64.0K= iB (65.5kB), run=3D14137-14137msec The performance is better for both 4K (+7.5%) and THP (+12.5%) cached read. Signed-off-by: Kairui Song --- lib/test_xarray.c | 57 +++++++++++++++++++++++++++++++++++++++++++++++ mm/filemap.c | 56 +++++++++++++++++++++++++++++++++------------- 2 files changed, 98 insertions(+), 15 deletions(-) diff --git a/lib/test_xarray.c b/lib/test_xarray.c index 26e28b65d60a..f0e02a1ee3d5 100644 --- a/lib/test_xarray.c +++ b/lib/test_xarray.c @@ -2015,6 +2015,62 @@ static noinline void check_xas_get_order(struct xarr= ay *xa) } } =20 +static noinline void check_xas_conflict_get_order(struct xarray *xa) +{ + XA_STATE(xas, xa, 0); + + void *entry; + int only_once; + unsigned int max_order =3D IS_ENABLED(CONFIG_XARRAY_MULTI) ? 20 : 1; + unsigned int order; + unsigned long i, j, k; + + for (order =3D 0; order < max_order; order++) { + for (i =3D 0; i < 10; i++) { + xas_set_order(&xas, i << order, order); + do { + xas_lock(&xas); + xas_store(&xas, xa_mk_value(i)); + xas_unlock(&xas); + } while (xas_nomem(&xas, GFP_KERNEL)); + + /* + * Ensure xas_get_order works with xas_for_each_conflict. + */ + j =3D i << order; + for (k =3D 0; k < order; k++) { + only_once =3D 0; + xas_lock(&xas); + xas_set_order(&xas, j + (1 << k), k); + xas_for_each_conflict(&xas, entry) { + XA_BUG_ON(xa, entry !=3D xa_mk_value(i)); + XA_BUG_ON(xa, xas_get_order(&xas) !=3D order); + only_once++; + } + XA_BUG_ON(xa, only_once !=3D 1); + xas_unlock(&xas); + } + + if (order < max_order - 1) { + only_once =3D 0; + xas_lock(&xas); + xas_set_order(&xas, (i & ~1UL) << order, order + 1); + xas_for_each_conflict(&xas, entry) { + XA_BUG_ON(xa, entry !=3D xa_mk_value(i)); + XA_BUG_ON(xa, xas_get_order(&xas) !=3D order); + only_once++; + } + XA_BUG_ON(xa, only_once !=3D 1); + xas_unlock(&xas); + } + + xas_set_order(&xas, i << order, order); + xas_store(&xas, NULL); + } + } +} + + static noinline void check_destroy(struct xarray *xa) { unsigned long index; @@ -2067,6 +2123,7 @@ static int xarray_checks(void) check_multi_store_advanced(&array); check_get_order(&array); check_xas_get_order(&array); + check_xas_conflict_get_order(&array); check_xa_alloc(); check_find(&array); check_find_entry(&array); diff --git a/mm/filemap.c b/mm/filemap.c index 6bbec8783793..90b86f22a9df 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -852,7 +852,9 @@ noinline int __filemap_add_folio(struct address_space *= mapping, struct folio *folio, pgoff_t index, gfp_t gfp, void **shadowp) { XA_STATE(xas, &mapping->i_pages, index); - bool huge =3D folio_test_hugetlb(folio); + void *alloced_shadow =3D NULL; + int alloced_order =3D 0; + bool huge; long nr; =20 VM_BUG_ON_FOLIO(!folio_test_locked(folio), folio); @@ -861,6 +863,7 @@ noinline int __filemap_add_folio(struct address_space *= mapping, =20 VM_BUG_ON_FOLIO(index & (folio_nr_pages(folio) - 1), folio); xas_set_order(&xas, index, folio_order(folio)); + huge =3D folio_test_hugetlb(folio); nr =3D folio_nr_pages(folio); =20 gfp &=3D GFP_RECLAIM_MASK; @@ -868,16 +871,10 @@ noinline int __filemap_add_folio(struct address_space= *mapping, folio->mapping =3D mapping; folio->index =3D xas.xa_index; =20 - do { - unsigned int order =3D xa_get_order(xas.xa, xas.xa_index); + for (;;) { + int order =3D -1, split_order =3D 0; void *entry, *old =3D NULL; =20 - if (order > folio_order(folio)) { - xas_split_alloc(&xas, xa_load(xas.xa, xas.xa_index), - order, gfp); - if (xas_error(&xas)) - goto error; - } xas_lock_irq(&xas); xas_for_each_conflict(&xas, entry) { old =3D entry; @@ -885,19 +882,33 @@ noinline int __filemap_add_folio(struct address_space= *mapping, xas_set_err(&xas, -EEXIST); goto unlock; } + /* + * If a larger entry exists, + * it will be the first and only entry iterated. + */ + if (order =3D=3D -1) + order =3D xas_get_order(&xas); + } + + /* entry may have changed before we re-acquire the lock */ + if (alloced_order && (old !=3D alloced_shadow || order !=3D alloced_orde= r)) { + xas_destroy(&xas); + alloced_order =3D 0; } =20 if (old) { - if (shadowp) - *shadowp =3D old; - /* entry may have been split before we acquired lock */ - order =3D xa_get_order(xas.xa, xas.xa_index); - if (order > folio_order(folio)) { + if (order > 0 && order > folio_order(folio)) { /* How to handle large swap entries? */ BUG_ON(shmem_mapping(mapping)); + if (!alloced_order) { + split_order =3D order; + goto unlock; + } xas_split(&xas, old, order); xas_reset(&xas); } + if (shadowp) + *shadowp =3D old; } =20 xas_store(&xas, folio); @@ -913,9 +924,24 @@ noinline int __filemap_add_folio(struct address_space = *mapping, __lruvec_stat_mod_folio(folio, NR_FILE_THPS, nr); } + unlock: xas_unlock_irq(&xas); - } while (xas_nomem(&xas, gfp)); + + /* split needed, alloc here and retry. */ + if (split_order) { + xas_split_alloc(&xas, old, split_order, gfp); + if (xas_error(&xas)) + goto error; + alloced_shadow =3D old; + alloced_order =3D split_order; + xas_reset(&xas); + continue; + } + + if (!xas_nomem(&xas, gfp)) + break; + } =20 if (xas_error(&xas)) goto error; --=20 2.43.0