From nobody Tue Sep 16 07:20:14 2025 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 782D4C3DA7D for ; Thu, 5 Jan 2023 05:35:44 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230054AbjAEFfn (ORCPT ); Thu, 5 Jan 2023 00:35:43 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59086 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230073AbjAEFfa (ORCPT ); Thu, 5 Jan 2023 00:35:30 -0500 Received: from mail-pf1-x435.google.com (mail-pf1-x435.google.com [IPv6:2607:f8b0:4864:20::435]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id ABACC37265 for ; Wed, 4 Jan 2023 21:35:29 -0800 (PST) Received: by mail-pf1-x435.google.com with SMTP id w203so7936723pfc.12 for ; Wed, 04 Jan 2023 21:35:29 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=OHS7GwecdRuxXk3+Zc4WVXvP35qWQPtvOh3gevBpM+A=; b=b6rJqUBUXZ6Q+Bgt7gJ5/kTxgMlrEP1hR1HsAaFEIJOVqcYGEqzmgqaayIu44OklLY aKr7mCHk9NuWDdeZ/sLDjprbzjTysci4voRXkScpJ9J7O3UQJDwMi88MSEXKhAd31MRX IkgpaAnfci6cSUtZQFzoqH74BOPlvcbzaXSNQ= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=OHS7GwecdRuxXk3+Zc4WVXvP35qWQPtvOh3gevBpM+A=; b=ySD4Hu3Nk5iTpym+vHItdty2npWhr6iYSH5qYbhvINfg2Kan7JKa+7YijKRrvxEiy0 MjkbvDjnYiCoom+iGcF8ENxfrNECJNYPQ6q4SNsbymwXLgcCOz+6al+Z/AHT8eIxTh9i fEfh9DOMMe3h0T6FA9c/cmVrcq4rgvn6DKUL5vdEe0HJyXP6Pe7Vf9V2AWyuvTLP8D2w CWrZ4RbObOizS64NRRTxJiXujeeY2nV2BFhuANwJ/SubS4zpeGHyrnCSU6OSIMan+Kj3 8bMQSUrBiJ2WY9xZZCHfvMSmpLLVU5J5PB9+f0mLhol2YmW/ow+kycEBK96WGmbVI7PT YngQ== X-Gm-Message-State: AFqh2kqfJndxu3nhN5uUSxSaHrXo5x0IbWrTK+e8K5ioDdsN0qn8O4+p 5rfV7TRrn4D3Q+dbMsAcc470Bet2QRvev+z0 X-Google-Smtp-Source: AMrXdXvCZQaKMjk2imZtfP2WyTYli30IFBbctSlFI2dnkGh9PE8CQrY3FcAiS28OhT3LjTERiI6OGw== X-Received: by 2002:a05:6a00:84c:b0:581:1ee0:75a with SMTP id q12-20020a056a00084c00b005811ee0075amr43961009pfk.32.1672896929190; Wed, 04 Jan 2023 21:35:29 -0800 (PST) Received: from tigerii.tok.corp.google.com ([2401:fa00:8f:203:da84:4bce:bb29:7dea]) by smtp.gmail.com with ESMTPSA id v27-20020aa799db000000b00577c5915138sm2036855pfi.128.2023.01.04.21.35.27 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 04 Jan 2023 21:35:28 -0800 (PST) From: Sergey Senozhatsky To: Minchan Kim , Andrew Morton Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, Sergey Senozhatsky Subject: [PATCH 1/4] zsmalloc: rework zspage chain size selection Date: Thu, 5 Jan 2023 14:35:07 +0900 Message-Id: <20230105053510.1819862-2-senozhatsky@chromium.org> X-Mailer: git-send-email 2.39.0.314.g84b9a713c41-goog In-Reply-To: <20230105053510.1819862-1-senozhatsky@chromium.org> References: <20230105053510.1819862-1-senozhatsky@chromium.org> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Computers are bad at division. We currently decide the best zspage chain size (max number of physical pages per-zspage) by looking at a `used percentage` value. This is not enough as we lose precision during usage percentage calculations For example, let's look at size class 208: pages per zspage wasted bytes used% 1 144 96 2 80 99 3 16 99 4 160 99 Current algorithm will select 2 page per zspage configuration, as it's the first one to reach 99%. However, 3 pages per zspage waste less memory. Change algorithm and select zspage configuration that has lowest wasted value. Signed-off-by: Sergey Senozhatsky --- mm/zsmalloc.c | 56 +++++++++++++++++---------------------------------- 1 file changed, 19 insertions(+), 37 deletions(-) diff --git a/mm/zsmalloc.c b/mm/zsmalloc.c index 9445bee6b014..959126e708a3 100644 --- a/mm/zsmalloc.c +++ b/mm/zsmalloc.c @@ -802,42 +802,6 @@ static enum fullness_group fix_fullness_group(struct s= ize_class *class, return newfg; } =20 -/* - * We have to decide on how many pages to link together - * to form a zspage for each size class. This is important - * to reduce wastage due to unusable space left at end of - * each zspage which is given as: - * wastage =3D Zp % class_size - * usage =3D Zp - wastage - * where Zp =3D zspage size =3D k * PAGE_SIZE where k =3D 1, 2, ... - * - * For example, for size class of 3/8 * PAGE_SIZE, we should - * link together 3 PAGE_SIZE sized pages to form a zspage - * since then we can perfectly fit in 8 such objects. - */ -static int get_pages_per_zspage(int class_size) -{ - int i, max_usedpc =3D 0; - /* zspage order which gives maximum used size per KB */ - int max_usedpc_order =3D 1; - - for (i =3D 1; i <=3D ZS_MAX_PAGES_PER_ZSPAGE; i++) { - int zspage_size; - int waste, usedpc; - - zspage_size =3D i * PAGE_SIZE; - waste =3D zspage_size % class_size; - usedpc =3D (zspage_size - waste) * 100 / zspage_size; - - if (usedpc > max_usedpc) { - max_usedpc =3D usedpc; - max_usedpc_order =3D i; - } - } - - return max_usedpc_order; -} - static struct zspage *get_zspage(struct page *page) { struct zspage *zspage =3D (struct zspage *)page_private(page); @@ -2321,6 +2285,24 @@ static int zs_register_shrinker(struct zs_pool *pool) pool->name); } =20 +static int calculate_zspage_chain_size(int class_size) +{ + int i, min_waste =3D INT_MAX; + int chain_size =3D 1; + + for (i =3D 1; i <=3D ZS_MAX_PAGES_PER_ZSPAGE; i++) { + int waste; + + waste =3D (i * PAGE_SIZE) % class_size; + if (waste < min_waste) { + min_waste =3D waste; + chain_size =3D i; + } + } + + return chain_size; +} + /** * zs_create_pool - Creates an allocation pool to work from. * @name: pool name to be created @@ -2365,7 +2347,7 @@ struct zs_pool *zs_create_pool(const char *name) size =3D ZS_MIN_ALLOC_SIZE + i * ZS_SIZE_CLASS_DELTA; if (size > ZS_MAX_ALLOC_SIZE) size =3D ZS_MAX_ALLOC_SIZE; - pages_per_zspage =3D get_pages_per_zspage(size); + pages_per_zspage =3D calculate_zspage_chain_size(size); objs_per_zspage =3D pages_per_zspage * PAGE_SIZE / size; =20 /* --=20 2.39.0.314.g84b9a713c41-goog From nobody Tue Sep 16 07:20:14 2025 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id EA41FC3DA7D for ; Thu, 5 Jan 2023 05:35:49 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229755AbjAEFfr (ORCPT ); Thu, 5 Jan 2023 00:35:47 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59086 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229786AbjAEFfi (ORCPT ); Thu, 5 Jan 2023 00:35:38 -0500 Received: from mail-pf1-x433.google.com (mail-pf1-x433.google.com [IPv6:2607:f8b0:4864:20::433]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0EA6C5132D for ; Wed, 4 Jan 2023 21:35:38 -0800 (PST) Received: by mail-pf1-x433.google.com with SMTP id a184so13786447pfa.9 for ; Wed, 04 Jan 2023 21:35:38 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=EvvS87mfgDK3AanjNOiSdWY479pGlUeMdWmydMONAbM=; b=ZOigF9gP3ox2wQ1zSpV804cSF8sMjwowDtxNQg8uNjyVvXHeLqggLSqwx1tRbjrl9B bnJrcbil85V6vAirx2XZPoC0C8R/EU2ZbD+54woGiRCo9GR8JP4hAMJkbFPpQyAk6EhY U7FslFAbJPenKL019W92H94OSDxTBVtGcc3wM= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=EvvS87mfgDK3AanjNOiSdWY479pGlUeMdWmydMONAbM=; b=P4HT3Ih5sLVbh0c104uhJe1Ohv4oQ7SZ9gzC2SlYdmiWe4EsM56d+W4zKWLXG3fdNj dmFWe8/EDKSr/Tw6sIghq99nK6SX1YqUfxe7rMPvNUTsR/cY6cS9wOx1yx8OIwcy+omh AwVeyYKjtNhxB98iYZAgP6cqcYBLph68w7eI7uVgteDn9OZCyFUork8ZGa4ZV+98y+ir qkvVaw8EAO4gLwWkzCUs5bycsN+ChCk5gL3a4Vn+oXhMMuHp0M27JDO061mvMFODLbEJ ERWbazj1BybsXHh1cVB5GUIP7j51TpO4ZUNn4tPHGxL3rM0LfrGPyQJaZjQx0cuLo9H0 RBCQ== X-Gm-Message-State: AFqh2krySk0jj7j/JzFOp2O6GoqDSSwN9r1CEt62acEprbOQqAXjzutE 4Yb/iG4ZCyFC4Ke3ZkQF5imlRQ== X-Google-Smtp-Source: AMrXdXs7ikuwn0bVOye9xIcYyunW3Of68phhtfVO0eCIT+czMyzUJEhFuS9IQZf97Bw1n6CWls1t3g== X-Received: by 2002:a62:d45e:0:b0:56e:dca8:ba71 with SMTP id u30-20020a62d45e000000b0056edca8ba71mr43482025pfl.32.1672896937609; Wed, 04 Jan 2023 21:35:37 -0800 (PST) Received: from tigerii.tok.corp.google.com ([2401:fa00:8f:203:da84:4bce:bb29:7dea]) by smtp.gmail.com with ESMTPSA id v27-20020aa799db000000b00577c5915138sm2036855pfi.128.2023.01.04.21.35.35 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 04 Jan 2023 21:35:37 -0800 (PST) From: Sergey Senozhatsky To: Minchan Kim , Andrew Morton Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, Sergey Senozhatsky Subject: [PATCH 2/4] zsmalloc: skip chain size calculation for pow_of_2 classes Date: Thu, 5 Jan 2023 14:35:08 +0900 Message-Id: <20230105053510.1819862-3-senozhatsky@chromium.org> X-Mailer: git-send-email 2.39.0.314.g84b9a713c41-goog In-Reply-To: <20230105053510.1819862-1-senozhatsky@chromium.org> References: <20230105053510.1819862-1-senozhatsky@chromium.org> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" If a class size is power of 2 then it wastes no memory and the best configuration is 1 physical page per-zspage. Signed-off-by: Sergey Senozhatsky --- mm/zsmalloc.c | 3 +++ 1 file changed, 3 insertions(+) diff --git a/mm/zsmalloc.c b/mm/zsmalloc.c index 959126e708a3..9a0f1963b803 100644 --- a/mm/zsmalloc.c +++ b/mm/zsmalloc.c @@ -2290,6 +2290,9 @@ static int calculate_zspage_chain_size(int class_size) int i, min_waste =3D INT_MAX; int chain_size =3D 1; =20 + if (is_power_of_2(class_size)) + return chain_size; + for (i =3D 1; i <=3D ZS_MAX_PAGES_PER_ZSPAGE; i++) { int waste; =20 --=20 2.39.0.314.g84b9a713c41-goog From nobody Tue Sep 16 07:20:14 2025 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 81525C3DA7D for ; Thu, 5 Jan 2023 05:35:58 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230207AbjAEFfy (ORCPT ); Thu, 5 Jan 2023 00:35:54 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59128 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230159AbjAEFfm (ORCPT ); Thu, 5 Jan 2023 00:35:42 -0500 Received: from mail-pg1-x52c.google.com (mail-pg1-x52c.google.com [IPv6:2607:f8b0:4864:20::52c]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id EAC5D4858B for ; Wed, 4 Jan 2023 21:35:40 -0800 (PST) Received: by mail-pg1-x52c.google.com with SMTP id r18so23819837pgr.12 for ; Wed, 04 Jan 2023 21:35:40 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=jNDbEU6GJSl4dMMRDfb+E6NT+uZQaENbR2FZEs35hTc=; b=UlX/lnwlf6fIQFIsZcBBD4u/IPlewAyXweh0bMynwmBpANcJ7jnMIod48XA9in383A 1UK4jVAaY2SfCjbsggvlu9ABgeEPohET6f7Sd1wbMcEVoU+rDpvh1LLGoGnsjZnmxfXt OztWNCYcuYgxz/E1e+NlICj1UXsrLnCP3mY68= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=jNDbEU6GJSl4dMMRDfb+E6NT+uZQaENbR2FZEs35hTc=; b=lrEJZ2/TP1UhioZ6HqrllQ9I5Z1VhWOhBYtdboeAQKwujPluYOMoRqEJ6CV1WkfSWr 9R/GSjJfoFr7O0JN6lIAUr2eRb72zRsW+ldggFYmKFXV87esSYUNY6YE/jbTV4kRM4jK UDpdyzegYPScVbISVrxnI4umtJPDa/HF5yMctpaYk0bX0nY2ACEbU7cZghPvW+sFEY86 XFGbcIIi6Tjd6Ra98NAf9HH7eFcd7bMnYcXFLLa9XR79eMRulogna6ejT3gW2g32egHW WLhiJ6RnDXuL299CxGeb8eV/PGckd+6H4guYyUf4cwwAg/aoyCQ4FxSAn3HxS9EXqh1L lfbA== X-Gm-Message-State: AFqh2krNdHmDyl7eaVEwYVHVoWVhgYgF3VbqmgSan3ZZQcXBjXMhemWi oERPagLf2zrU8ObVRgT6pf1nVg== X-Google-Smtp-Source: AMrXdXs8xtoHQAu80/AXDIApYwbIuB3LIkfP6hr+4B2fJbDkMV6pCTZqhjjdA4EnSCvzd9059skuwg== X-Received: by 2002:a05:6a00:2281:b0:581:a8dc:8f95 with SMTP id f1-20020a056a00228100b00581a8dc8f95mr34967574pfe.12.1672896940075; Wed, 04 Jan 2023 21:35:40 -0800 (PST) Received: from tigerii.tok.corp.google.com ([2401:fa00:8f:203:da84:4bce:bb29:7dea]) by smtp.gmail.com with ESMTPSA id v27-20020aa799db000000b00577c5915138sm2036855pfi.128.2023.01.04.21.35.38 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 04 Jan 2023 21:35:39 -0800 (PST) From: Sergey Senozhatsky To: Minchan Kim , Andrew Morton Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, Sergey Senozhatsky Subject: [PATCH 3/4] zsmalloc: make zspage chain size configurable Date: Thu, 5 Jan 2023 14:35:09 +0900 Message-Id: <20230105053510.1819862-4-senozhatsky@chromium.org> X-Mailer: git-send-email 2.39.0.314.g84b9a713c41-goog In-Reply-To: <20230105053510.1819862-1-senozhatsky@chromium.org> References: <20230105053510.1819862-1-senozhatsky@chromium.org> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Remove hard coded limit on the maximum number of physical pages per-zspage. This will allow tuning of zsmalloc pool as zspage chain size changes `pages per-zspage` and `objects per-zspage` characteristics of size classes which also affects size classes clustering (the way size classes are merged). Signed-off-by: Sergey Senozhatsky --- .../admin-guide/blockdev/zsmalloc.rst | 157 ++++++++++++++++++ mm/Kconfig | 19 +++ mm/zsmalloc.c | 15 +- 3 files changed, 180 insertions(+), 11 deletions(-) create mode 100644 Documentation/admin-guide/blockdev/zsmalloc.rst diff --git a/Documentation/admin-guide/blockdev/zsmalloc.rst b/Documentatio= n/admin-guide/blockdev/zsmalloc.rst new file mode 100644 index 000000000000..2e238afb1b4b --- /dev/null +++ b/Documentation/admin-guide/blockdev/zsmalloc.rst @@ -0,0 +1,157 @@ +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D +zsmalloc allocator +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + +Internals +--------- + +zsmalloc has 255 size classes. Size classes hold a number of zspages, each +zspage can consist of up to ZSMALLOC_CHAIN_SIZE physical (0 order) pages. +The exact (most optimal) zspage chain size is calculated for each size cla= ss +during zsmalloc pool creation (see calculate_zspage_chain_size()). + +As a reasonable optimization, zsmalloc merges size classes that have +similar characteristics: number of pages per zspage and number of +objects zspage can store. + +For example, let's look at the following size classes::: + +class size almost_full almost_empty obj_allocated obj_used pages_used p= ages_per_zspage freeable +.. + 94 1536 0 0 0 0 0 = 3 0 + 100 1632 0 0 0 0 0 = 2 0 +.. + +Size classes #95-99 are merged with size class #100. That is, each time +we store an object of size, say, 1568 bytes instead of using class #96 +we end up storing it in size class #100. Class #100 is for objects of +1632 bytes in size, hence every 1568 bytes object wastes 1632-1568 bytes. +Class #100 zspages consist of 2 physical pages and can hold 5 objects. +When we need to store, say, 13 objects of size 1568 we end up allocating +three zspages; in other words, 6 physical pages. + +However, if we'll look closer at size class #96 (which should hold objects +of size 1568 bytes) and trace calculate_zspage_chain_size()::: + + pages per zspage wasted bytes used% + 1 960 76 + 2 352 95 + 3 1312 89 + 4 704 95 + 5 96 99 + +We'd notice that the most optimal zspage configuration for this class is +when it consists of 5 physical pages. A 5 page class #96 configuration +would store 13 objects of size 1568 in a single zspage, allocating 5 physi= cal +pages, as opposed to 6 physical pages that class #100 would allocate other= wise. + +A larger zspage chain size for class #96 also changes its key characterist= ics: +pages per-zspage and objects per-zspage. As a result we merge less classes= . In +other words classes are grouped in a more compact way, which decreases mem= ory +wastage. + +Let's take a closer look at the bottom of /sys/kernel/debug/zsmalloc/zramX= /classes::: + +class size almost_full almost_empty obj_allocated obj_used pages_used p= ages_per_zspage freeable +.. + 202 3264 0 0 0 0 0 = 4 0 + 254 4096 0 0 0 0 0 = 1 0 +.. + +For exactly same reason - maximum 4 pages per zspage - the last non-huge +size class is #202, which stores objects of size 3264 bytes. Any object +larger than 3264 bytes, hence, is considered to be huge and lands in size +class #254, which uses a whole physical page to store every object (objects +in huge classes don't share physical pages). + +Another consequence of larger zspages chain sizes is that we move the huge +size class watermark up and as a result have less huge classes and store +large objects in a more compact way. + +For zspage chain size of 8, huge class watermark becomes 3632 bytes::: + +class size almost_full almost_empty obj_allocated obj_used pages_used p= ages_per_zspage freeable +.. + 202 3264 0 0 0 0 0 = 4 0 + 211 3408 0 0 0 0 0 = 5 0 + 217 3504 0 0 0 0 0 = 6 0 + 222 3584 0 0 0 0 0 = 7 0 + 225 3632 0 0 0 0 0 = 8 0 + 254 4096 0 0 0 0 0 = 1 0 +.. + +For zspage chain size of 16, huge class watermark becomes 3840 bytes::: + +class size almost_full almost_empty obj_allocated obj_used pages_used p= ages_per_zspage freeable +.. + 202 3264 0 0 0 0 0 = 4 0 + 206 3328 0 0 0 0 0 = 13 0 + 207 3344 0 0 0 0 0 = 9 0 + 208 3360 0 0 0 0 0 = 14 0 + 211 3408 0 0 0 0 0 = 5 0 + 212 3424 0 0 0 0 0 = 16 0 + 214 3456 0 0 0 0 0 = 11 0 + 217 3504 0 0 0 0 0 = 6 0 + 219 3536 0 0 0 0 0 = 13 0 + 222 3584 0 0 0 0 0 = 7 0 + 223 3600 0 0 0 0 0 = 15 0 + 225 3632 0 0 0 0 0 = 8 0 + 228 3680 0 0 0 0 0 = 9 0 + 230 3712 0 0 0 0 0 = 10 0 + 232 3744 0 0 0 0 0 = 11 0 + 234 3776 0 0 0 0 0 = 12 0 + 235 3792 0 0 0 0 0 = 13 0 + 236 3808 0 0 0 0 0 = 14 0 + 238 3840 0 0 0 0 0 = 15 0 + 254 4096 0 0 0 0 0 = 1 0 +.. + +Overall the combined zspage chain size effect on zsmalloc pool configurati= on::: + +pages per zspage number of size classes (clusters) huge size class wat= ermark + 4 69 3264 + 5 86 3408 + 6 93 3504 + 7 112 3584 + 8 123 3632 + 9 140 3680 + 10 143 3712 + 11 159 3744 + 12 164 3776 + 13 180 3792 + 14 183 3808 + 15 188 3840 + 16 191 3840 + +A synthetic test::: + +CONFIG_ZSMALLOC_CHAIN_SIZE=3D4 + +zsmalloc classes stats + class size almost_full almost_empty obj_allocated obj_used pages_used = pages_per_zspage freeable + .. + Total 13 51 413836 412973 159955 = 3 + +zram mm_stat +1691783168 628083717 655175680 0 655175680 60 0 340= 48 34049 + +CONFIG_ZSMALLOC_CHAIN_SIZE=3D8 + +zsmalloc classes stats + class size almost_full almost_empty obj_allocated obj_used pages_used = pages_per_zspage freeable + .. + Total 18 87 414852 412978 156666 = 0 + +zram mm_stat +1691803648 627793930 641703936 0 641703936 60 0 335= 91 33591 + +Note that for the same amount of data zsmalloc uses less physical pages: d= own +to 156666 from 159955, and maximum zsmalloc pool memory usage also went do= wn +from 655175680 to 641703936 bytes. + +The obvious downside of larger zspage chains is that some zspages require +more physical pages, which can, in theory, increase system memory pressure +in cases when zspool suffers from heavy internal fragmentation and zspool +compaction cannot relocate objects and release some zspages. In such cases +users are advised to lower zspage chain size limit (CONFIG_ZSMALLOC_CHAIN_= SIZE +option). diff --git a/mm/Kconfig b/mm/Kconfig index ff7b209dec05..995a7c4083c2 100644 --- a/mm/Kconfig +++ b/mm/Kconfig @@ -191,6 +191,25 @@ config ZSMALLOC_STAT information to userspace via debugfs. If unsure, say N. =20 +config ZSMALLOC_CHAIN_SIZE + int "Maximum number of physical pages per-zspage" + default 4 + range 1 16 + depends on ZSMALLOC + help + Each zmalloc page (zspage) can consist of 1 or more physical + (0 order) non contiguous pages. This option sets the upper + (hard) limit on that number. + + The exact zspage chain size is calculated for each size class + individually during pool initialisation. Changing this results + in different size classes characteristics (pages per-zspage, + objects per-zspage) which in turn results in different pool + configurations: zsmalloc merges size classes that share key + characteristics. + + Please read zsmalloc documentation for more details. + menu "SLAB allocator options" =20 choice diff --git a/mm/zsmalloc.c b/mm/zsmalloc.c index 9a0f1963b803..34ba97d1175f 100644 --- a/mm/zsmalloc.c +++ b/mm/zsmalloc.c @@ -73,13 +73,6 @@ */ #define ZS_ALIGN 8 =20 -/* - * A single 'zspage' is composed of up to 2^N discontiguous 0-order (singl= e) - * pages. ZS_MAX_ZSPAGE_ORDER defines upper limit on N. - */ -#define ZS_MAX_ZSPAGE_ORDER 2 -#define ZS_MAX_PAGES_PER_ZSPAGE (_AC(1, UL) << ZS_MAX_ZSPAGE_ORDER) - #define ZS_HANDLE_SIZE (sizeof(unsigned long)) =20 /* @@ -126,7 +119,7 @@ #define MAX(a, b) ((a) >=3D (b) ? (a) : (b)) /* ZS_MIN_ALLOC_SIZE must be multiple of ZS_ALIGN */ #define ZS_MIN_ALLOC_SIZE \ - MAX(32, (ZS_MAX_PAGES_PER_ZSPAGE << PAGE_SHIFT >> OBJ_INDEX_BITS)) + MAX(32, (CONFIG_ZSMALLOC_CHAIN_SIZE << PAGE_SHIFT >> OBJ_INDEX_BITS)) /* each chunk includes extra space to keep handle */ #define ZS_MAX_ALLOC_SIZE PAGE_SIZE =20 @@ -1078,7 +1071,7 @@ static struct zspage *alloc_zspage(struct zs_pool *po= ol, gfp_t gfp) { int i; - struct page *pages[ZS_MAX_PAGES_PER_ZSPAGE]; + struct page *pages[CONFIG_ZSMALLOC_CHAIN_SIZE]; struct zspage *zspage =3D cache_alloc_zspage(pool, gfp); =20 if (!zspage) @@ -1910,7 +1903,7 @@ static void replace_sub_page(struct size_class *class= , struct zspage *zspage, struct page *newpage, struct page *oldpage) { struct page *page; - struct page *pages[ZS_MAX_PAGES_PER_ZSPAGE] =3D {NULL, }; + struct page *pages[CONFIG_ZSMALLOC_CHAIN_SIZE] =3D {NULL, }; int idx =3D 0; =20 page =3D get_first_page(zspage); @@ -2293,7 +2286,7 @@ static int calculate_zspage_chain_size(int class_size) if (is_power_of_2(class_size)) return chain_size; =20 - for (i =3D 1; i <=3D ZS_MAX_PAGES_PER_ZSPAGE; i++) { + for (i =3D 1; i <=3D CONFIG_ZSMALLOC_CHAIN_SIZE; i++) { int waste; =20 waste =3D (i * PAGE_SIZE) % class_size; --=20 2.39.0.314.g84b9a713c41-goog From nobody Tue Sep 16 07:20:14 2025 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4C3EBC3DA7D for ; Thu, 5 Jan 2023 05:36:06 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230255AbjAEFgC (ORCPT ); Thu, 5 Jan 2023 00:36:02 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59270 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230073AbjAEFfo (ORCPT ); Thu, 5 Jan 2023 00:35:44 -0500 Received: from mail-pf1-x432.google.com (mail-pf1-x432.google.com [IPv6:2607:f8b0:4864:20::432]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A278637240 for ; Wed, 4 Jan 2023 21:35:43 -0800 (PST) Received: by mail-pf1-x432.google.com with SMTP id x26so18331564pfq.10 for ; Wed, 04 Jan 2023 21:35:43 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=u3RcO+ukEEU2tRMGKkuSA42+1jjoCDn3FvnF70ujYWI=; b=fU3ZxfwS35G2sd1ofLS0XH6rgua1/Y1Q81HU7Kz9UP7+pJN4hKAikLBBHWCqGEhwNJ 6sKba61QIGyVO9mZeRIBc6WKZ39X1PvO+i5DuWQQI9UZWlRkhNIcN9zzZhUCawyqxwCS Yasvmb2ntqX0rHqa/Jzqhid2eBOHoCUxCpZgY= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=u3RcO+ukEEU2tRMGKkuSA42+1jjoCDn3FvnF70ujYWI=; b=CTkQHKPxgqShuxOKklAgKpU84mKpVEeU3xMG6njgR1X9gC+nabWmETzcRtIpC3cc2g lvfZunghVtICY75WXaCD0krsrmvBTYu1YdR+S9tkJe8fCtgZyF8xL+GX2aY5uRrDiLW2 kzj2Xyr3LbHRFDi2Z8am92/E0OoZWe88n+zh7I60o+r+I2cO0MnDyN4duC1e52fV5c/8 Cp4JJuXEcyP/XuNQltsTBaLGO3mKMPdVdFrENBcjN4yERochmr21jl/3GlMVZ+jJjHHs NNqHVebQX2Fa8Dou/3EjVOeDLKpPBus5T/CWwbz/RC0WW4EEqX3Qi0UgLg1SJ9P34pER Lv6w== X-Gm-Message-State: AFqh2ko7hZNYFJ0tV55YNYXG8AjHIdaHXThu0hZ6zmKJK5LM5C4j9TV5 uMXGfUXBB1nUld0AfBc4uHRkUR36389ekfPM X-Google-Smtp-Source: AMrXdXsLJCXHNR4z3LOOHq/EWKiZHaAIInb3wQ5+cuJXqsQVsZ3H0KlsmcXXMWSqiEVtXBS5uFTQMA== X-Received: by 2002:a05:6a00:450a:b0:56e:664f:a5f5 with SMTP id cw10-20020a056a00450a00b0056e664fa5f5mr61013213pfb.8.1672896943236; Wed, 04 Jan 2023 21:35:43 -0800 (PST) Received: from tigerii.tok.corp.google.com ([2401:fa00:8f:203:da84:4bce:bb29:7dea]) by smtp.gmail.com with ESMTPSA id v27-20020aa799db000000b00577c5915138sm2036855pfi.128.2023.01.04.21.35.41 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 04 Jan 2023 21:35:42 -0800 (PST) From: Sergey Senozhatsky To: Minchan Kim , Andrew Morton Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, Sergey Senozhatsky Subject: [PATCH 4/4] zsmalloc: set default zspage chain size to 8 Date: Thu, 5 Jan 2023 14:35:10 +0900 Message-Id: <20230105053510.1819862-5-senozhatsky@chromium.org> X-Mailer: git-send-email 2.39.0.314.g84b9a713c41-goog In-Reply-To: <20230105053510.1819862-1-senozhatsky@chromium.org> References: <20230105053510.1819862-1-senozhatsky@chromium.org> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" This changes key characteristics (pages per-zspage and objects per-zspage) of a number of size classes which in results in different pool configuration. With zspage chain size of 8 we have more size clases clusters (123) and higher huge size class watermark (3632 bytes). Please read zsmalloc documentation for more details. Signed-off-by: Sergey Senozhatsky --- mm/Kconfig | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mm/Kconfig b/mm/Kconfig index 995a7c4083c2..9ab5c1996a04 100644 --- a/mm/Kconfig +++ b/mm/Kconfig @@ -193,7 +193,7 @@ config ZSMALLOC_STAT =20 config ZSMALLOC_CHAIN_SIZE int "Maximum number of physical pages per-zspage" - default 4 + default 8 range 1 16 depends on ZSMALLOC help --=20 2.39.0.314.g84b9a713c41-goog