From nobody Sun Feb 8 02:21:39 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3B903C77B73 for ; Mon, 22 May 2023 11:14:39 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233210AbjEVLOh (ORCPT ); Mon, 22 May 2023 07:14:37 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58030 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232524AbjEVLN4 (ORCPT ); Mon, 22 May 2023 07:13:56 -0400 Received: from mail-ej1-x630.google.com (mail-ej1-x630.google.com [IPv6:2a00:1450:4864:20::630]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id BA8FA19BF for ; Mon, 22 May 2023 04:08:54 -0700 (PDT) Received: by mail-ej1-x630.google.com with SMTP id a640c23a62f3a-96fffe11714so96048966b.0 for ; Mon, 22 May 2023 04:08:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1684753733; x=1687345733; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=C3POfPMkD0T4i+gkKz2qkPU4t7/WZTiyT2qCdNLU280=; b=VBUTLfKbApHu6QZhrVrOCB3BjALZb8HbF0H20xKFhc6s2ddiN7suwTGMU0bL+g+sYn RmbQ1cww5mUZenVFKfKM0obeU2G6DDRfd8SBMnH9fmFpOyakPo9ae6ksAd2XakoCBv0E 7Ov6b3cng1MnPemi03HJHLAXu9L7D+fD1mqdF1IW9NYQd6lcGKgDypewZcwaRLFcrhmr yjR9yAKjvtwtrmWZi3nCGabZYIW/EQNSzPSvJbp0e3+m5rd5yon1lw8lX3KGm/ik7TPO pykNacp5zTZge80vp2rYbryAhbquVFUZS46oKag6NjrQQcMLsIE3M8wIwP5IFy1Gt0Ul IcQQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1684753733; x=1687345733; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=C3POfPMkD0T4i+gkKz2qkPU4t7/WZTiyT2qCdNLU280=; b=DWtQWnPxDRa1HKXMy1KSj82cHj8PG0ivfj7kHpBRqlKmAq1c9cwWNx7JuqjB/uyLhd 724tTd3H7bcTKZKiaqW/lsoucF1kp4pp6jRKBGJ49vQ9IWvKvThWQRjPtam1hSyMh5VW KHYlnSUu5y0BnF2MtjX7LRPnLOfrNayGgIgw3g9pBsBv/2GPt5kpS/6Z6HTxKvzbc+Ll kA9gihmshuOG0bMac15MuCSINPSwO0Ulw0XNX0aXjrvgHML0lz3PKclmA/PTkEXX9Hw0 uRQLvshb0HDNwJWdDjCBESsR6legxTOnMkWOYVRdw+ZimhAqrmBfhf/7i3+5G/ZlrDnK 28rQ== X-Gm-Message-State: AC+VfDwcV5lPdrlOBNl+U6j9o7Tbtx+Gp8YlQE3vEnVamPEefOLFT3sL AASTmX4gz81/Qzk/w2IkyPE= X-Google-Smtp-Source: ACHHUZ7sSj0PR0+CyXEW2CQQzfPJq5rycOMctGBSYZ5gIV+Lpig9VckuJmCX23ieD4b+A7HcD+ROww== X-Received: by 2002:a17:907:1c8b:b0:94e:48ac:9a51 with SMTP id nb11-20020a1709071c8b00b0094e48ac9a51mr11815922ejc.4.1684753732807; Mon, 22 May 2023 04:08:52 -0700 (PDT) Received: from pc638.lan ([155.137.26.201]) by smtp.gmail.com with ESMTPSA id lx19-20020a170906af1300b0094f07545d40sm2949453ejb.220.2023.05.22.04.08.52 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 22 May 2023 04:08:52 -0700 (PDT) From: "Uladzislau Rezki (Sony)" To: linux-mm@kvack.org, Andrew Morton Cc: LKML , Baoquan He , Lorenzo Stoakes , Christoph Hellwig , Matthew Wilcox , "Liam R . Howlett" , Dave Chinner , "Paul E . McKenney" , Joel Fernandes , Uladzislau Rezki , Oleksiy Avramchenko Subject: [PATCH 1/9] mm: vmalloc: Add va_alloc() helper Date: Mon, 22 May 2023 13:08:41 +0200 Message-Id: <20230522110849.2921-2-urezki@gmail.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20230522110849.2921-1-urezki@gmail.com> References: <20230522110849.2921-1-urezki@gmail.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Currently __alloc_vmap_area() function contains an open codded logic that finds and adjusts a VA based on allocation request. Introduce a va_alloc() helper that adjusts found VA only. It will be used later at least in two places. There is no a functional change as a result of this patch. Signed-off-by: Uladzislau Rezki (Sony) Reviewed-by: Christoph Hellwig Reviewed-by: Lorenzo Stoakes --- mm/vmalloc.c | 41 ++++++++++++++++++++++++++++------------- 1 file changed, 28 insertions(+), 13 deletions(-) diff --git a/mm/vmalloc.c b/mm/vmalloc.c index 31ff782d368b..409285b68a67 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -1482,6 +1482,32 @@ adjust_va_to_fit_type(struct rb_root *root, struct l= ist_head *head, return 0; } =20 +static unsigned long +va_alloc(struct vmap_area *va, + struct rb_root *root, struct list_head *head, + unsigned long size, unsigned long align, + unsigned long vstart, unsigned long vend) +{ + unsigned long nva_start_addr; + int ret; + + if (va->va_start > vstart) + nva_start_addr =3D ALIGN(va->va_start, align); + else + nva_start_addr =3D ALIGN(vstart, align); + + /* Check the "vend" restriction. */ + if (nva_start_addr + size > vend) + return vend; + + /* Update the free vmap_area. */ + ret =3D adjust_va_to_fit_type(root, head, va, nva_start_addr, size); + if (WARN_ON_ONCE(ret)) + return vend; + + return nva_start_addr; +} + /* * Returns a start address of the newly allocated area, if success. * Otherwise a vend is returned that indicates failure. @@ -1494,7 +1520,6 @@ __alloc_vmap_area(struct rb_root *root, struct list_h= ead *head, bool adjust_search_size =3D true; unsigned long nva_start_addr; struct vmap_area *va; - int ret; =20 /* * Do not adjust when: @@ -1512,18 +1537,8 @@ __alloc_vmap_area(struct rb_root *root, struct list_= head *head, if (unlikely(!va)) return vend; =20 - if (va->va_start > vstart) - nva_start_addr =3D ALIGN(va->va_start, align); - else - nva_start_addr =3D ALIGN(vstart, align); - - /* Check the "vend" restriction. */ - if (nva_start_addr + size > vend) - return vend; - - /* Update the free vmap_area. */ - ret =3D adjust_va_to_fit_type(root, head, va, nva_start_addr, size); - if (WARN_ON_ONCE(ret)) + nva_start_addr =3D va_alloc(va, root, head, size, align, vstart, vend); + if (nva_start_addr =3D=3D vend) return vend; =20 #if DEBUG_AUGMENT_LOWEST_MATCH_CHECK --=20 2.30.2 From nobody Sun Feb 8 02:21:39 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id B439CC77B75 for ; Mon, 22 May 2023 11:14:41 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233223AbjEVLOk (ORCPT ); Mon, 22 May 2023 07:14:40 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58266 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232442AbjEVLN4 (ORCPT ); Mon, 22 May 2023 07:13:56 -0400 Received: from mail-ej1-x632.google.com (mail-ej1-x632.google.com [IPv6:2a00:1450:4864:20::632]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 49A3340E8 for ; Mon, 22 May 2023 04:08:55 -0700 (PDT) Received: by mail-ej1-x632.google.com with SMTP id a640c23a62f3a-96f7377c86aso590975466b.1 for ; Mon, 22 May 2023 04:08:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1684753733; x=1687345733; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=AqE7n0km0cT3SoKgXn9+9ffSGk68u7ZexKpVeyjeuP0=; b=UI0Yt0lWDXh9ljMMkPAajAGV57h0505ib1ttO6nC+S6/ox2EoI9VGZd7+i0KyUT3Iz UrOeQv6HeCqlYMLmrO9xLceNit+eAAhp+ayorSX9GWMqWetCfy65fNLgsBYIBMUhhJ2L U75zeNwLejw8wEtPfmFV0nYxaTwgfXdcMVWerdFeHWF2BfssxPxEikaABdcefX8aVmFB OzcafrJAiZhR3U1i8vQurAWGDU7VgmaPUGDQ2nyXctKGB+RLSHtg6lFIAG/yBViPwShe HnTgtWcZBRbZXXW3R63z1IHpNcsR0E7wzcXRYrATiECZ/KAKp4PAjxmjbmz2ChABim6V aqqg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1684753733; x=1687345733; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=AqE7n0km0cT3SoKgXn9+9ffSGk68u7ZexKpVeyjeuP0=; b=FmKUQSv/MjcYtCNoiHTuepGu3R8lTdL7NCEDNARzsccux8cUpsUKUO0gGbw2xAM2VG EnJskQzntNvpghe1aFBIUnPymWi9pM44dTNPg3qRJqMxPLrOdIuUm+oPaa++Rfrj7DK2 6Bsqw9C6dX4tOdt7oFJfL+pZZKgzsYdKtL7m+fBh4JjHu94i/lDyiLOGrW/Uc3bmXgaD sLp1bUga3EHITkbwgtfiiPLvtmBbPbIL00dYMj3yhAgQM/0sXLzItYxt70GHzADwKZSk 95JLoi8AO3NYYMt66t3oxRO/lnruNWNvv36e9uxfs0Z6J4sDnEhw37qFs7RMvbAGQC/o cHtQ== X-Gm-Message-State: AC+VfDz/REzDP87GihVnDMULruGiFuQLcEbc+obyOMGVJhnMQHMlsXD/ C+dYwoiSwCahUgCCTEebcbc= X-Google-Smtp-Source: ACHHUZ6msUCypNMoSNhIQMunC9pTKnBPX5E3jFgX9C4tJ4LV2wmznUsQq48JjiudHWKxzuBo6hrzCQ== X-Received: by 2002:a17:907:2d23:b0:96a:928c:d382 with SMTP id gs35-20020a1709072d2300b0096a928cd382mr10346599ejc.48.1684753733572; Mon, 22 May 2023 04:08:53 -0700 (PDT) Received: from pc638.lan ([155.137.26.201]) by smtp.gmail.com with ESMTPSA id lx19-20020a170906af1300b0094f07545d40sm2949453ejb.220.2023.05.22.04.08.52 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 22 May 2023 04:08:53 -0700 (PDT) From: "Uladzislau Rezki (Sony)" To: linux-mm@kvack.org, Andrew Morton Cc: LKML , Baoquan He , Lorenzo Stoakes , Christoph Hellwig , Matthew Wilcox , "Liam R . Howlett" , Dave Chinner , "Paul E . McKenney" , Joel Fernandes , Uladzislau Rezki , Oleksiy Avramchenko Subject: [PATCH 2/9] mm: vmalloc: Rename adjust_va_to_fit_type() function Date: Mon, 22 May 2023 13:08:42 +0200 Message-Id: <20230522110849.2921-3-urezki@gmail.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20230522110849.2921-1-urezki@gmail.com> References: <20230522110849.2921-1-urezki@gmail.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" This patch renames the adjust_va_to_fit_type() function to a shorter variant and more expressive. A new name is va_clip(). There is no a functional change as a result of this patch. Signed-off-by: Uladzislau Rezki (Sony) Reviewed-by: Christoph Hellwig Reviewed-by: Lorenzo Stoakes --- mm/vmalloc.c | 13 ++++++------- 1 file changed, 6 insertions(+), 7 deletions(-) diff --git a/mm/vmalloc.c b/mm/vmalloc.c index 409285b68a67..5f900efec6a9 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -1383,9 +1383,9 @@ classify_va_fit_type(struct vmap_area *va, } =20 static __always_inline int -adjust_va_to_fit_type(struct rb_root *root, struct list_head *head, - struct vmap_area *va, unsigned long nva_start_addr, - unsigned long size) +va_clip(struct rb_root *root, struct list_head *head, + struct vmap_area *va, unsigned long nva_start_addr, + unsigned long size) { struct vmap_area *lva =3D NULL; enum fit_type type =3D classify_va_fit_type(va, nva_start_addr, size); @@ -1501,7 +1501,7 @@ va_alloc(struct vmap_area *va, return vend; =20 /* Update the free vmap_area. */ - ret =3D adjust_va_to_fit_type(root, head, va, nva_start_addr, size); + ret =3D va_clip(root, head, va, nva_start_addr, size); if (WARN_ON_ONCE(ret)) return vend; =20 @@ -3979,9 +3979,8 @@ struct vm_struct **pcpu_get_vm_areas(const unsigned l= ong *offsets, /* It is a BUG(), but trigger recovery instead. */ goto recovery; =20 - ret =3D adjust_va_to_fit_type(&free_vmap_area_root, - &free_vmap_area_list, - va, start, size); + ret =3D va_clip(&free_vmap_area_root, + &free_vmap_area_list, va, start, size); if (WARN_ON_ONCE(unlikely(ret))) /* It is a BUG(), but trigger recovery instead. */ goto recovery; --=20 2.30.2 From nobody Sun Feb 8 02:21:39 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id E65FAC77B75 for ; Mon, 22 May 2023 11:14:44 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233228AbjEVLOn (ORCPT ); Mon, 22 May 2023 07:14:43 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58036 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232979AbjEVLN5 (ORCPT ); Mon, 22 May 2023 07:13:57 -0400 Received: from mail-ej1-x636.google.com (mail-ej1-x636.google.com [IPv6:2a00:1450:4864:20::636]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2186F40ED for ; Mon, 22 May 2023 04:08:56 -0700 (PDT) Received: by mail-ej1-x636.google.com with SMTP id a640c23a62f3a-96f683e8855so488847866b.2 for ; Mon, 22 May 2023 04:08:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1684753734; x=1687345734; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=sNtfrgi/quI4TC521AyXgwFWeKCoK/ixdzMUo1Hjyag=; b=GkR4/dm/njRpBQa+KVraZ+7JOlNk7shw4BEphYd/T4MuYuRj9UeHx7STRIkE9qZ3ZV 2oXVDGHFmqIOsQbp8tUn9Y00B46Baxi7Y8dneLOn6DV42d85pL1sEsY68+QnjQ1f2vmJ aDPoq7XcD9rLqsdzZWq3ANly5HjVSjJ29W0p5d8YhCo15pKjujxSAlIMECHBsZDAyuMN tLgaKTuxmHTfb/eGGN0rAdS+/lKXxYAVJyynl/B47Xw/ZEaCyR2lt1lpmnS2FpLh3BXG MgpEFyKwNZW3Q2xr0xt2mb+dk5JSHw7+52EgYFgJevxmwbWAu2YUPoOjuE6sjvnpUmYm ZEog== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1684753734; x=1687345734; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=sNtfrgi/quI4TC521AyXgwFWeKCoK/ixdzMUo1Hjyag=; b=FTZ3yfk6jS6vKVXbW+/KmL42W+/A/0LU6hHyovEVYmhf1EB2QBqDn19pmtG8/mQ4Ei hWnhTRwkh55HHMi+wxdu1FsFpjA2IMpVzKW4uEMJpuTBOfdz0qoMzFsnbA5tS90o7Mdm xS2KE+rY+MPoRCrl8f+74hO79VwqCNVdMSfze9cbbqkR2dW8h6Egn78MKFoidwQxY6fB ld/s2KdjEBesyC13rGxOYctu4CZrQVwStEu3Rrcubc3z/MAgkWNUf95rOdTWRelFOQyK mJWrQtwbRt+gzwre07P2ubdUVjVSQ1vlPEot5vhEub/qi/0oJB2am4yKleZXnXwoNjmi xztQ== X-Gm-Message-State: AC+VfDwI+MLuZ2nmwbQF63GjwJOKw9BOOGtMsx7ULvMErwYslkD6CjuK FjXKwbA2Ue0DCk3DX1KRldA= X-Google-Smtp-Source: ACHHUZ6CkjutlmxyUmi96KfbjXVfsWOUvXlEtbViniM+ZZDlWRxo42QLeXCAxHA9y4X8Xg8XpIQzUA== X-Received: by 2002:a17:907:d21:b0:965:4b43:11f1 with SMTP id gn33-20020a1709070d2100b009654b4311f1mr9711913ejc.3.1684753734338; Mon, 22 May 2023 04:08:54 -0700 (PDT) Received: from pc638.lan ([155.137.26.201]) by smtp.gmail.com with ESMTPSA id lx19-20020a170906af1300b0094f07545d40sm2949453ejb.220.2023.05.22.04.08.53 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 22 May 2023 04:08:54 -0700 (PDT) From: "Uladzislau Rezki (Sony)" To: linux-mm@kvack.org, Andrew Morton Cc: LKML , Baoquan He , Lorenzo Stoakes , Christoph Hellwig , Matthew Wilcox , "Liam R . Howlett" , Dave Chinner , "Paul E . McKenney" , Joel Fernandes , Uladzislau Rezki , Oleksiy Avramchenko Subject: [PATCH 3/9] mm: vmalloc: Move vmap_init_free_space() down in vmalloc.c Date: Mon, 22 May 2023 13:08:43 +0200 Message-Id: <20230522110849.2921-4-urezki@gmail.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20230522110849.2921-1-urezki@gmail.com> References: <20230522110849.2921-1-urezki@gmail.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" A vmap_init_free_space() is a function that setups a vmap space and is considered as part of initialization phase. Since a main entry which is vmalloc_init(), has been moved down in vmalloc.c it makes sense to follow the pattern. There is no a functional change as a result of this patch. Signed-off-by: Uladzislau Rezki (Sony) Reviewed-by: Christoph Hellwig Reviewed-by: Lorenzo Stoakes --- mm/vmalloc.c | 82 ++++++++++++++++++++++++++-------------------------- 1 file changed, 41 insertions(+), 41 deletions(-) diff --git a/mm/vmalloc.c b/mm/vmalloc.c index 5f900efec6a9..19dcffb0d563 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -2416,47 +2416,6 @@ void __init vm_area_register_early(struct vm_struct = *vm, size_t align) kasan_populate_early_vm_area_shadow(vm->addr, vm->size); } =20 -static void vmap_init_free_space(void) -{ - unsigned long vmap_start =3D 1; - const unsigned long vmap_end =3D ULONG_MAX; - struct vmap_area *busy, *free; - - /* - * B F B B B F - * -|-----|.....|-----|-----|-----|.....|- - * | The KVA space | - * |<--------------------------------->| - */ - list_for_each_entry(busy, &vmap_area_list, list) { - if (busy->va_start - vmap_start > 0) { - free =3D kmem_cache_zalloc(vmap_area_cachep, GFP_NOWAIT); - if (!WARN_ON_ONCE(!free)) { - free->va_start =3D vmap_start; - free->va_end =3D busy->va_start; - - insert_vmap_area_augment(free, NULL, - &free_vmap_area_root, - &free_vmap_area_list); - } - } - - vmap_start =3D busy->va_end; - } - - if (vmap_end - vmap_start > 0) { - free =3D kmem_cache_zalloc(vmap_area_cachep, GFP_NOWAIT); - if (!WARN_ON_ONCE(!free)) { - free->va_start =3D vmap_start; - free->va_end =3D vmap_end; - - insert_vmap_area_augment(free, NULL, - &free_vmap_area_root, - &free_vmap_area_list); - } - } -} - static inline void setup_vmalloc_vm_locked(struct vm_struct *vm, struct vmap_area *va, unsigned long flags, const void *caller) { @@ -4271,6 +4230,47 @@ module_init(proc_vmalloc_init); =20 #endif =20 +static void vmap_init_free_space(void) +{ + unsigned long vmap_start =3D 1; + const unsigned long vmap_end =3D ULONG_MAX; + struct vmap_area *busy, *free; + + /* + * B F B B B F + * -|-----|.....|-----|-----|-----|.....|- + * | The KVA space | + * |<--------------------------------->| + */ + list_for_each_entry(busy, &vmap_area_list, list) { + if (busy->va_start - vmap_start > 0) { + free =3D kmem_cache_zalloc(vmap_area_cachep, GFP_NOWAIT); + if (!WARN_ON_ONCE(!free)) { + free->va_start =3D vmap_start; + free->va_end =3D busy->va_start; + + insert_vmap_area_augment(free, NULL, + &free_vmap_area_root, + &free_vmap_area_list); + } + } + + vmap_start =3D busy->va_end; + } + + if (vmap_end - vmap_start > 0) { + free =3D kmem_cache_zalloc(vmap_area_cachep, GFP_NOWAIT); + if (!WARN_ON_ONCE(!free)) { + free->va_start =3D vmap_start; + free->va_end =3D vmap_end; + + insert_vmap_area_augment(free, NULL, + &free_vmap_area_root, + &free_vmap_area_list); + } + } +} + void __init vmalloc_init(void) { struct vmap_area *va; --=20 2.30.2 From nobody Sun Feb 8 02:21:39 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id EEFA8C77B73 for ; Mon, 22 May 2023 11:14:48 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233197AbjEVLOr (ORCPT ); Mon, 22 May 2023 07:14:47 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58068 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233096AbjEVLN7 (ORCPT ); Mon, 22 May 2023 07:13:59 -0400 Received: from mail-ej1-x62e.google.com (mail-ej1-x62e.google.com [IPv6:2a00:1450:4864:20::62e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7C8DE1BCF for ; Mon, 22 May 2023 04:08:57 -0700 (PDT) Received: by mail-ej1-x62e.google.com with SMTP id a640c23a62f3a-970028cfb6cso94840866b.1 for ; Mon, 22 May 2023 04:08:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1684753735; x=1687345735; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=WvsDwavnfIYng0FXX5ciX4/rx3a5CWQd4rOb3eMOJOg=; b=iDb9tK4TOgaJNMq61xrRvzxoVUaPP0VvNEnz6NrY+zacf0FfJnwnuypFPmwH5SLNBS GyEOPDWFsssngIu4YWW0FrNovP9ZjqfzkUf/51MOpeMHMX80L0GnQUdqQZEtrUUDgAN5 CqpASsjwkoQ7U63fy74gThmTbQvI9Ie3XaHwC17zsSXEk9lHt40AkqJDIcGVhr55gFza Ab5qRCRkiZfB7QM1qkbc7/2z6/RGesk6zCBFvjSfwFLXPP4gxqH+BsOk1j0UanBK1aWA e+0Ct1vWALCAGVP46sp1nfM2tzxvrxxrGXz0I5hY5eIN0kmUd5nEpcG0c3MTS3rmdd8s WEcQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1684753735; x=1687345735; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=WvsDwavnfIYng0FXX5ciX4/rx3a5CWQd4rOb3eMOJOg=; b=EkECDYmO1cmq7RrmSSdvaPq3iu2YuN+IjQuaqMIMMVcYC+lSqik1QlC5sPR38AcuO6 DQrzdJd4a54m3R0mgdtdVUPfn+jNCGU9NycSiqrvRhOBYLNx7qlkQx+9VXvGpCIx4B98 ty0jwlBP/GlTafI5uWnwUnccsiRH/UEifmAGfFoRjQ2ber3v67U4UZH3o5cU/O1LKF5l paS7n7986s07d/iY+z+BZvnz3DhvtsxJFK4nZMtKOh8QuFxKX2EXeh7i+/tWkQfBOtyO 9k6YWlDDaw8Raycf4+6tpptpcCsTaQa3q4Daxrax/BlNrr6i4BDvd7n5QzNgNb/IxubQ opzA== X-Gm-Message-State: AC+VfDzSUYRIkfJHKrTOwqVlv9l8HE/NZIF+e41P6gNgUnwYhknaeaew Bp1wP6ZvVIm6Xf2jQ1uJMX8= X-Google-Smtp-Source: ACHHUZ5RSuVJjDArE0w69xzADXBpvX8mfpX9Veg7DqUbJpCMdvLgHaYw03z8mFT9WRUtE0g9JS1nlg== X-Received: by 2002:a17:907:2d0e:b0:96f:cde5:5f5e with SMTP id gs14-20020a1709072d0e00b0096fcde55f5emr3930140ejc.29.1684753735155; Mon, 22 May 2023 04:08:55 -0700 (PDT) Received: from pc638.lan ([155.137.26.201]) by smtp.gmail.com with ESMTPSA id lx19-20020a170906af1300b0094f07545d40sm2949453ejb.220.2023.05.22.04.08.54 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 22 May 2023 04:08:54 -0700 (PDT) From: "Uladzislau Rezki (Sony)" To: linux-mm@kvack.org, Andrew Morton Cc: LKML , Baoquan He , Lorenzo Stoakes , Christoph Hellwig , Matthew Wilcox , "Liam R . Howlett" , Dave Chinner , "Paul E . McKenney" , Joel Fernandes , Uladzislau Rezki , Oleksiy Avramchenko Subject: [PATCH 4/9] mm: vmalloc: Add a per-CPU-zone infrastructure Date: Mon, 22 May 2023 13:08:44 +0200 Message-Id: <20230522110849.2921-5-urezki@gmail.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20230522110849.2921-1-urezki@gmail.com> References: <20230522110849.2921-1-urezki@gmail.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Introduce a basic skeleton that defines a cache entity for a CPU. It consist of bookkeeping data which belong to a CPU zone. A CPU-zone defines a virtual address range with its fixed size. This approach gives a possibility to map any address to a particular CPU-zone it belongs to. Below is a high-level description how it is used in alloc and free paths: alloc-path: in-front-pre-fetch occurs if a cache is empty. A chunk is placed to a free-tree of this CPU. Finally this CPU cache is clipped in order to accomplish a request. A fresh VA resides in a busy-tree of specific CPU-zone it is bound to. free-path: va->va_start is converted to a zone, so a lookup occurs in a bookkeeping zone it belongs to. Once it gets removed a VA is moved to a lazy-tree. It is freed after a TLB flushing. A "cvz_size", that reflects a zone size, is set to ULONG_MAX by this patch. The reason is we would like to follow an old behavior until all dependent changes are in place. There is no a functional change as a result of this patch. Signed-off-by: Uladzislau Rezki (Sony) --- mm/vmalloc.c | 55 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 55 insertions(+) diff --git a/mm/vmalloc.c b/mm/vmalloc.c index 19dcffb0d563..f6da2590b4de 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -773,6 +773,61 @@ static struct rb_root free_vmap_area_root =3D RB_ROOT; */ static DEFINE_PER_CPU(struct vmap_area *, ne_fit_preload_node); =20 +/* + * A per-cpu effective vmap cache logic. It allows to pre-fetch + * a memory vmap chunk with a specified size. A bookkeeping data + * is stored and accessed per-CPU. + */ +enum { FREE =3D 0, BUSY, LAZY, NFBL }; + +#define fbl(z, i, m) z->fbl[i].m +#define fbl_root(z, i) fbl(z, i, root) +#define fbl_head(z, i) fbl(z, i, head) + +#define fbl_lock(z, i) spin_lock(&fbl(z, i, lock)) +#define fbl_unlock(z, i) spin_unlock(&fbl(z, i, lock)) + +struct cpu_vmap_zone { + /* + * FREE, BUSY, LAZY bookkeeping data of this CPU zone. + */ + struct { + struct rb_root root; + struct list_head head; + spinlock_t lock; + } fbl[NFBL]; + + /* + * A list of outstanding lazily-freed areas, ready to + * be drained, which belong(address space) to this CPU + * zone. + */ + struct list_head purge_list; + + /* + * It controls that only one CPU can pre-fetch this + * CPU zone. + */ + atomic_t fill_in_progress; +}; + +static DEFINE_PER_CPU(struct cpu_vmap_zone, cpu_vmap_zone); + +/* Disable a per-cpu caching. */ +static __read_mostly unsigned long cvz_size =3D ULONG_MAX; + +static inline unsigned int +addr_to_cpu(unsigned long addr) +{ + return (addr / cvz_size) % num_possible_cpus(); +} + +static inline struct cpu_vmap_zone * +__maybe_unused addr_to_cvz(unsigned long addr) +{ + return &per_cpu(cpu_vmap_zone, addr_to_cpu(addr)); +} + static __always_inline unsigned long va_size(struct vmap_area *va) { --=20 2.30.2 From nobody Sun Feb 8 02:21:39 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3E5B8C77B75 for ; Mon, 22 May 2023 11:15:10 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233237AbjEVLOu (ORCPT ); Mon, 22 May 2023 07:14:50 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58070 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233117AbjEVLOA (ORCPT ); Mon, 22 May 2023 07:14:00 -0400 Received: from mail-ej1-x62d.google.com (mail-ej1-x62d.google.com [IPv6:2a00:1450:4864:20::62d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B8DEC1BD4 for ; Mon, 22 May 2023 04:08:57 -0700 (PDT) Received: by mail-ej1-x62d.google.com with SMTP id a640c23a62f3a-96fab30d1e1so374667766b.0 for ; Mon, 22 May 2023 04:08:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1684753736; x=1687345736; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=DrvLnO25dwGFFlUS7GxG9ba70dEBhOOiZuR4m0pcTUM=; b=flXewQ2yArAVYlv2skHOfRQSd1S7Cqlwt9MNZZJrqFsSrWWwyouqmQjJ95oq8wCC6C ad1/P8xGdXwlSwJMGJC8rDQai6cooJJwK0WkPCEKTPeV1R4JhMhzlsAxID+3OAx7hyz8 qGIYAx4Fe+Vqa5htZtPnUE8LV6kn4rgy+MrpLn2fFSbuz9hRvORwAIU7W24PqNv/v/A+ Vvb0nbx4byHbsHsQfe2U6VreF5LdyXSZ3hV+iYjamkAgVT/lQoTYVkBdU0CzbUpSU/0c 71/85NUEE602yGx29yMZLdgzUhf6phivNIaZUU/ts0HMXqTu7rh81+XTKlpL8mliOzmO yxdA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1684753736; x=1687345736; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=DrvLnO25dwGFFlUS7GxG9ba70dEBhOOiZuR4m0pcTUM=; b=h2Kwnm8vDDqbKunIIATkwt0b+UcxSqEo01h6/45STBs0711TjZwFQEnrawbLF9ANQQ k954fTwA1a0MEBa/y8YLZv0hjgOwbB5ZlQYKKhWh5uk/F7bSuPGMOa99TcSo+9XBjS9I q/amZY1cYcfmpEWQqQqgCAmPkh6WuGXNVyC3MK15e/07qo7Q+lfCKFC3NcQBUlLdzVSL FTlGpe+51SFVH1YOgP0P+uip2uO8XnyoOAgjduYDkj5kqnGvxXkhOwCvJZZa7N8a9tbo uPPxZT3df8jMbXCZ5zJvK2CBgERYqRVNcNnuCY0PDSLcrFmqyX0dw6ZVc5C5ogX6tLHM YXMg== X-Gm-Message-State: AC+VfDzHgwMa8o5V+1pRfW3uKWEUpKPGwIPtvnBnK/7t80kX9aV7tJUZ Vrx1HNQxRVpEw/Ngbl0SVpwsYZ2RnWm2Ew== X-Google-Smtp-Source: ACHHUZ7/gXy6/G0+Tbl7lFR3/PBdDHxq7cTgGtR+aeCBks9BrXjxTCNVbHQV45jJ3L/ks2X1AHOsiw== X-Received: by 2002:a17:907:961e:b0:96f:c545:6fd with SMTP id gb30-20020a170907961e00b0096fc54506fdmr4438428ejc.10.1684753736024; Mon, 22 May 2023 04:08:56 -0700 (PDT) Received: from pc638.lan ([155.137.26.201]) by smtp.gmail.com with ESMTPSA id lx19-20020a170906af1300b0094f07545d40sm2949453ejb.220.2023.05.22.04.08.55 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 22 May 2023 04:08:55 -0700 (PDT) From: "Uladzislau Rezki (Sony)" To: linux-mm@kvack.org, Andrew Morton Cc: LKML , Baoquan He , Lorenzo Stoakes , Christoph Hellwig , Matthew Wilcox , "Liam R . Howlett" , Dave Chinner , "Paul E . McKenney" , Joel Fernandes , Uladzislau Rezki , Oleksiy Avramchenko Subject: [PATCH 5/9] mm: vmalloc: Insert busy-VA per-cpu zone Date: Mon, 22 May 2023 13:08:45 +0200 Message-Id: <20230522110849.2921-6-urezki@gmail.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20230522110849.2921-1-urezki@gmail.com> References: <20230522110849.2921-1-urezki@gmail.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Store busy-VA objects per a CPU zone. A va->va_start address is converted into a correct zone where it is placed and resides. An addr_to_cvz() function is used to do a proper address conversion. Such approach balances VAs across CPUs. That is why an access becomes scalable to number of CPUs in a system. Please note: Since a zone size is set to ULONG_MAX, i.e. everything is bound thus accessed to the CPU_0 so far, this patch does not give any difference comparing with a current behavior. The global vmap_area_lock, vmap_area_root are removed as there is no need in it anymore. The vmap_area_list is still kept and is _empty_. It is exported for a kexec only. The vmallocinfo and vread() have to be reworked to be able to handle multiple zones. As a result of this patch it can handle only one zone, i.e. when cache is disabled. Signed-off-by: Uladzislau Rezki (Sony) --- mm/vmalloc.c | 127 ++++++++++++++++++++++++++++++++++----------------- 1 file changed, 84 insertions(+), 43 deletions(-) diff --git a/mm/vmalloc.c b/mm/vmalloc.c index f6da2590b4de..a9170fe19909 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -729,11 +729,9 @@ EXPORT_SYMBOL(vmalloc_to_pfn); #define DEBUG_AUGMENT_LOWEST_MATCH_CHECK 0 =20 =20 -static DEFINE_SPINLOCK(vmap_area_lock); static DEFINE_SPINLOCK(free_vmap_area_lock); /* Export for kexec only */ LIST_HEAD(vmap_area_list); -static struct rb_root vmap_area_root =3D RB_ROOT; static bool vmap_initialized __read_mostly; =20 static struct rb_root purge_vmap_area_root =3D RB_ROOT; @@ -823,7 +821,7 @@ addr_to_cpu(unsigned long addr) } =20 static inline struct cpu_vmap_zone * -__maybe_unused addr_to_cvz(unsigned long addr) +addr_to_cvz(unsigned long addr) { return &per_cpu(cpu_vmap_zone, addr_to_cpu(addr)); } @@ -859,10 +857,10 @@ unsigned long vmalloc_nr_pages(void) } =20 /* Look up the first VA which satisfies addr < va_end, NULL if none. */ -static struct vmap_area *find_vmap_area_exceed_addr(unsigned long addr) +static struct vmap_area *find_vmap_area_exceed_addr(unsigned long addr, st= ruct rb_root *root) { struct vmap_area *va =3D NULL; - struct rb_node *n =3D vmap_area_root.rb_node; + struct rb_node *n =3D root->rb_node; =20 addr =3D (unsigned long)kasan_reset_tag((void *)addr); =20 @@ -1608,12 +1606,14 @@ __alloc_vmap_area(struct rb_root *root, struct list= _head *head, */ static void free_vmap_area(struct vmap_area *va) { + struct cpu_vmap_zone *z =3D addr_to_cvz(va->va_start); + /* * Remove from the busy tree/list. */ - spin_lock(&vmap_area_lock); - unlink_va(va, &vmap_area_root); - spin_unlock(&vmap_area_lock); + fbl_lock(z, BUSY); + unlink_va(va, &fbl_root(z, BUSY)); + fbl_unlock(z, BUSY); =20 /* * Insert/Merge it back to the free tree/list. @@ -1656,6 +1656,7 @@ static struct vmap_area *alloc_vmap_area(unsigned lon= g size, int node, gfp_t gfp_mask, unsigned long va_flags) { + struct cpu_vmap_zone *z; struct vmap_area *va; unsigned long freed; unsigned long addr; @@ -1701,9 +1702,11 @@ static struct vmap_area *alloc_vmap_area(unsigned lo= ng size, va->vm =3D NULL; va->flags =3D va_flags; =20 - spin_lock(&vmap_area_lock); - insert_vmap_area(va, &vmap_area_root, &vmap_area_list); - spin_unlock(&vmap_area_lock); + z =3D addr_to_cvz(va->va_start); + + fbl_lock(z, BUSY); + insert_vmap_area(va, &fbl_root(z, BUSY), &fbl_head(z, BUSY)); + fbl_unlock(z, BUSY); =20 BUG_ON(!IS_ALIGNED(va->va_start, align)); BUG_ON(va->va_start < vstart); @@ -1926,24 +1929,26 @@ static void free_unmap_vmap_area(struct vmap_area *= va) =20 struct vmap_area *find_vmap_area(unsigned long addr) { + struct cpu_vmap_zone *z =3D addr_to_cvz(addr); struct vmap_area *va; =20 - spin_lock(&vmap_area_lock); - va =3D __find_vmap_area(addr, &vmap_area_root); - spin_unlock(&vmap_area_lock); + fbl_lock(z, BUSY); + va =3D __find_vmap_area(addr, &fbl_root(z, BUSY)); + fbl_unlock(z, BUSY); =20 return va; } =20 static struct vmap_area *find_unlink_vmap_area(unsigned long addr) { + struct cpu_vmap_zone *z =3D addr_to_cvz(addr); struct vmap_area *va; =20 - spin_lock(&vmap_area_lock); - va =3D __find_vmap_area(addr, &vmap_area_root); + fbl_lock(z, BUSY); + va =3D __find_vmap_area(addr, &fbl_root(z, BUSY)); if (va) - unlink_va(va, &vmap_area_root); - spin_unlock(&vmap_area_lock); + unlink_va(va, &fbl_root(z, BUSY)); + fbl_unlock(z, BUSY); =20 return va; } @@ -2095,14 +2100,17 @@ static void *new_vmap_block(unsigned int order, gfp= _t gfp_mask) =20 static void free_vmap_block(struct vmap_block *vb) { + struct cpu_vmap_zone *z; struct vmap_block *tmp; =20 tmp =3D xa_erase(&vmap_blocks, addr_to_vb_idx(vb->va->va_start)); BUG_ON(tmp !=3D vb); =20 - spin_lock(&vmap_area_lock); - unlink_va(vb->va, &vmap_area_root); - spin_unlock(&vmap_area_lock); + z =3D addr_to_cvz(vb->va->va_start); + + fbl_lock(z, BUSY); + unlink_va(vb->va, &fbl_root(z, BUSY)); + fbl_unlock(z, BUSY); =20 free_vmap_area_noflush(vb->va); kfree_rcu(vb, rcu_head); @@ -2484,9 +2492,11 @@ static inline void setup_vmalloc_vm_locked(struct vm= _struct *vm, static void setup_vmalloc_vm(struct vm_struct *vm, struct vmap_area *va, unsigned long flags, const void *caller) { - spin_lock(&vmap_area_lock); + struct cpu_vmap_zone *z =3D addr_to_cvz(va->va_start); + + fbl_lock(z, BUSY); setup_vmalloc_vm_locked(vm, va, flags, caller); - spin_unlock(&vmap_area_lock); + fbl_unlock(z, BUSY); } =20 static void clear_vm_uninitialized_flag(struct vm_struct *vm) @@ -3605,6 +3615,7 @@ static void vmap_ram_vread(char *buf, char *addr, int= count, unsigned long flags */ long vread(char *buf, char *addr, unsigned long count) { + struct cpu_vmap_zone *z; struct vmap_area *va; struct vm_struct *vm; char *vaddr, *buf_start =3D buf; @@ -3617,8 +3628,11 @@ long vread(char *buf, char *addr, unsigned long coun= t) if ((unsigned long) addr + count < count) count =3D -(unsigned long) addr; =20 - spin_lock(&vmap_area_lock); - va =3D find_vmap_area_exceed_addr((unsigned long)addr); + /* Hooked to CPU0 because a cache is not activated. */ + z =3D &per_cpu(cpu_vmap_zone, 0); + fbl_lock(z, BUSY); + + va =3D find_vmap_area_exceed_addr((unsigned long)addr, &fbl_root(z, BUSY)= ); if (!va) goto finished; =20 @@ -3626,7 +3640,7 @@ long vread(char *buf, char *addr, unsigned long count) if ((unsigned long)addr + count <=3D va->va_start) goto finished; =20 - list_for_each_entry_from(va, &vmap_area_list, list) { + list_for_each_entry_from(va, &fbl_head(z, BUSY), list) { if (!count) break; =20 @@ -3674,7 +3688,7 @@ long vread(char *buf, char *addr, unsigned long count) count -=3D n; } finished: - spin_unlock(&vmap_area_lock); + fbl_unlock(z, BUSY); =20 if (buf =3D=3D buf_start) return 0; @@ -4014,14 +4028,15 @@ struct vm_struct **pcpu_get_vm_areas(const unsigned= long *offsets, } =20 /* insert all vm's */ - spin_lock(&vmap_area_lock); for (area =3D 0; area < nr_vms; area++) { - insert_vmap_area(vas[area], &vmap_area_root, &vmap_area_list); + struct cpu_vmap_zone *z =3D addr_to_cvz(vas[area]->va_start); =20 + fbl_lock(z, BUSY); + insert_vmap_area(vas[area], &fbl_root(z, BUSY), &fbl_head(z, BUSY)); setup_vmalloc_vm_locked(vms[area], vas[area], VM_ALLOC, pcpu_get_vm_areas); + fbl_unlock(z, BUSY); } - spin_unlock(&vmap_area_lock); =20 /* * Mark allocated areas as accessible. Do it now as a best-effort @@ -4145,24 +4160,24 @@ bool vmalloc_dump_obj(void *object) #ifdef CONFIG_PROC_FS static void *s_start(struct seq_file *m, loff_t *pos) __acquires(&vmap_purge_lock) - __acquires(&vmap_area_lock) + __acquires(&fbl(&per_cpu(cpu_vmap_zone, 0), BUSY, lock)) { mutex_lock(&vmap_purge_lock); - spin_lock(&vmap_area_lock); + fbl_lock((&per_cpu(cpu_vmap_zone, 0)), BUSY); =20 - return seq_list_start(&vmap_area_list, *pos); + return seq_list_start(&fbl_head((&per_cpu(cpu_vmap_zone, 0)), BUSY), *pos= ); } =20 static void *s_next(struct seq_file *m, void *p, loff_t *pos) { - return seq_list_next(p, &vmap_area_list, pos); + return seq_list_next(p, &fbl_head((&per_cpu(cpu_vmap_zone, 0)), BUSY), po= s); } =20 static void s_stop(struct seq_file *m, void *p) - __releases(&vmap_area_lock) + __releases(&fbl(&per_cpu(cpu_vmap_zone, 0), BUSY, lock)) __releases(&vmap_purge_lock) { - spin_unlock(&vmap_area_lock); + fbl_unlock((&per_cpu(cpu_vmap_zone, 0)), BUSY); mutex_unlock(&vmap_purge_lock); } =20 @@ -4258,7 +4273,7 @@ static int s_show(struct seq_file *m, void *p) * As a final step, dump "unpurged" areas. */ final: - if (list_is_last(&va->list, &vmap_area_list)) + if (list_is_last(&va->list, &fbl_head((&per_cpu(cpu_vmap_zone, 0)), BUSY)= )) show_purge_info(m); =20 return 0; @@ -4289,7 +4304,8 @@ static void vmap_init_free_space(void) { unsigned long vmap_start =3D 1; const unsigned long vmap_end =3D ULONG_MAX; - struct vmap_area *busy, *free; + struct vmap_area *free; + struct vm_struct *busy; =20 /* * B F B B B F @@ -4297,12 +4313,12 @@ static void vmap_init_free_space(void) * | The KVA space | * |<--------------------------------->| */ - list_for_each_entry(busy, &vmap_area_list, list) { - if (busy->va_start - vmap_start > 0) { + for (busy =3D vmlist; busy; busy =3D busy->next) { + if (busy->addr - vmap_start > 0) { free =3D kmem_cache_zalloc(vmap_area_cachep, GFP_NOWAIT); if (!WARN_ON_ONCE(!free)) { free->va_start =3D vmap_start; - free->va_end =3D busy->va_start; + free->va_end =3D (unsigned long) busy->addr; =20 insert_vmap_area_augment(free, NULL, &free_vmap_area_root, @@ -4310,7 +4326,7 @@ static void vmap_init_free_space(void) } } =20 - vmap_start =3D busy->va_end; + vmap_start =3D (unsigned long) busy->addr + busy->size; } =20 if (vmap_end - vmap_start > 0) { @@ -4326,6 +4342,22 @@ static void vmap_init_free_space(void) } } =20 +static void vmap_init_pcpu_zones(void) +{ + struct cpu_vmap_zone *z; + int i, j; + + for_each_possible_cpu(i) { + z =3D per_cpu_ptr(&cpu_vmap_zone, i); + + for (j =3D 0; j < ARRAY_SIZE(z->fbl); j++) { + INIT_LIST_HEAD(&z->fbl[j].head); + z->fbl[j].root =3D RB_ROOT; + spin_lock_init(&z->fbl[j].lock); + } + } +} + void __init vmalloc_init(void) { struct vmap_area *va; @@ -4349,8 +4381,15 @@ void __init vmalloc_init(void) INIT_WORK(&p->wq, delayed_vfree_work); } =20 + /* + * Setup per-cpu data before importing vmlist. + */ + vmap_init_pcpu_zones(); + /* Import existing vmlist entries. */ for (tmp =3D vmlist; tmp; tmp =3D tmp->next) { + struct cpu_vmap_zone *z; + va =3D kmem_cache_zalloc(vmap_area_cachep, GFP_NOWAIT); if (WARN_ON_ONCE(!va)) continue; @@ -4358,7 +4397,9 @@ void __init vmalloc_init(void) va->va_start =3D (unsigned long)tmp->addr; va->va_end =3D va->va_start + tmp->size; va->vm =3D tmp; - insert_vmap_area(va, &vmap_area_root, &vmap_area_list); + + z =3D addr_to_cvz(va->va_start); + insert_vmap_area(va, &fbl_root(z, BUSY), &fbl_head(z, BUSY)); } =20 /* --=20 2.30.2 From nobody Sun Feb 8 02:21:39 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id D0EBDC77B75 for ; Mon, 22 May 2023 11:15:15 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231678AbjEVLPN (ORCPT ); Mon, 22 May 2023 07:15:13 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58090 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232609AbjEVLOC (ORCPT ); Mon, 22 May 2023 07:14:02 -0400 Received: from mail-ed1-x529.google.com (mail-ed1-x529.google.com [IPv6:2a00:1450:4864:20::529]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id DDAB31BD1 for ; Mon, 22 May 2023 04:08:58 -0700 (PDT) Received: by mail-ed1-x529.google.com with SMTP id 4fb4d7f45d1cf-510b869fe0dso10509970a12.3 for ; Mon, 22 May 2023 04:08:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1684753737; x=1687345737; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=o/CF6SD8t4KOzLfI5LisrLifDfr954/t5B1JlATADoU=; b=Fi720qzQUzMzlB1l50k0vAlTsjRzBe/cg9qRDCZSutZdUCFmDwXU1qWUWjnhF6C+YJ IYmwGB/dBSBFcf+4n8lnS3JuELxCnqx/5KzHyZvkVrc0zdw6+T1veg4ZmjJxjg9CxrEV OKjrPtpIsPfNKe3Tk/UTfUWcUqfmc21TZhwMRhoz1hrlNz5snsaWIbFcpPMBqrvXzw+e BjnHOUkDc9QXislazQsjEiQDUHPub3bFS8PaRZShQXGb/Yb2GfHmsHP7ed4HMpnWLyiW rLbMYl8TEH+ya11TbsE8iAZdu8RNYvaI/k8Wdj555MFh7QE1iHfdGfjD0joUYCDSs+QZ 4VAQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1684753737; x=1687345737; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=o/CF6SD8t4KOzLfI5LisrLifDfr954/t5B1JlATADoU=; b=P06ktIl7nPoWMHbVMHWYfM4gVoS5HzGsi/giE8KswU23lAHx/Pvy2VHdQrFT6xck0J PzjowFZKY+EbtZJfl/o1RvdjeMqdSHGLs/AGBgSlN8truGcliJHOQnQS60PtQNiF9FEX g6g6RI66gcqTnynAsSRw6escN+m6Wvg7bQ/l+MEuEZ+7OSLQEGWgCmkfoGF6ONWf7xxC GioUdSGlzO05JDN1cO4DIl8DRTseIKkluw7ktTgfcZEZkB/UHE3No9nL7PIVY95csyiz 7XBIJB/xUP2jDY1T18Y2pBI8C7sm52YhXKqv4VWsd786sL3wwuPgZU2JT5u3PLB5SdlY eMMg== X-Gm-Message-State: AC+VfDz4ccvktdKc6aTgl75rt56yWjCsD3y2LJDHm4aZv4zvdqKrNiMz 3jXEgihcde3bll9Bt6z6Jro= X-Google-Smtp-Source: ACHHUZ4+1Ulvc5fCkI/Ge8lzhIjdkOo8SAA6ig4yBJZkl0gY7cfCTg6POEY2/5MBo0jvvQfpKEnDTQ== X-Received: by 2002:a17:907:8a12:b0:93e:fa12:aa1a with SMTP id sc18-20020a1709078a1200b0093efa12aa1amr10143550ejc.1.1684753736842; Mon, 22 May 2023 04:08:56 -0700 (PDT) Received: from pc638.lan ([155.137.26.201]) by smtp.gmail.com with ESMTPSA id lx19-20020a170906af1300b0094f07545d40sm2949453ejb.220.2023.05.22.04.08.56 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 22 May 2023 04:08:56 -0700 (PDT) From: "Uladzislau Rezki (Sony)" To: linux-mm@kvack.org, Andrew Morton Cc: LKML , Baoquan He , Lorenzo Stoakes , Christoph Hellwig , Matthew Wilcox , "Liam R . Howlett" , Dave Chinner , "Paul E . McKenney" , Joel Fernandes , Uladzislau Rezki , Oleksiy Avramchenko Subject: [PATCH 6/9] mm: vmalloc: Support multiple zones in vmallocinfo Date: Mon, 22 May 2023 13:08:46 +0200 Message-Id: <20230522110849.2921-7-urezki@gmail.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20230522110849.2921-1-urezki@gmail.com> References: <20230522110849.2921-1-urezki@gmail.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" A global vmap area busy tree has been removed and replaced by multiple per-cpu trees/lists, therefore we need to traversal all per-cpu busy-lists in order to dump all allocated objects. Please note, after this patch, dumped addresses of allocated areas are not sequential. See an example below: 0 1 2 0 1 2 |---|---|---|---|---|---|-> vmap space There 3 CPUs dumping is done per-CPU zone, as you can see address of zone_0 can be ahead of addresses residing in the zone_1 or zone_2. Signed-off-by: Uladzislau Rezki (Sony) --- mm/vmalloc.c | 90 ++++++++++++++++++++++++++-------------------------- 1 file changed, 45 insertions(+), 45 deletions(-) diff --git a/mm/vmalloc.c b/mm/vmalloc.c index a9170fe19909..dd83deb5ef4f 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -4159,26 +4159,18 @@ bool vmalloc_dump_obj(void *object) =20 #ifdef CONFIG_PROC_FS static void *s_start(struct seq_file *m, loff_t *pos) - __acquires(&vmap_purge_lock) - __acquires(&fbl(&per_cpu(cpu_vmap_zone, 0), BUSY, lock)) { - mutex_lock(&vmap_purge_lock); - fbl_lock((&per_cpu(cpu_vmap_zone, 0)), BUSY); - - return seq_list_start(&fbl_head((&per_cpu(cpu_vmap_zone, 0)), BUSY), *pos= ); + return *pos < 1 ? (void *)1 : NULL; } =20 static void *s_next(struct seq_file *m, void *p, loff_t *pos) { - return seq_list_next(p, &fbl_head((&per_cpu(cpu_vmap_zone, 0)), BUSY), po= s); + ++*pos; + return NULL; } =20 static void s_stop(struct seq_file *m, void *p) - __releases(&fbl(&per_cpu(cpu_vmap_zone, 0), BUSY, lock)) - __releases(&vmap_purge_lock) { - fbl_unlock((&per_cpu(cpu_vmap_zone, 0)), BUSY); - mutex_unlock(&vmap_purge_lock); } =20 static void show_numa_info(struct seq_file *m, struct vm_struct *v) @@ -4209,6 +4201,7 @@ static void show_purge_info(struct seq_file *m) { struct vmap_area *va; =20 + mutex_lock(&vmap_purge_lock); spin_lock(&purge_vmap_area_lock); list_for_each_entry(va, &purge_vmap_area_list, list) { seq_printf(m, "0x%pK-0x%pK %7ld unpurged vm_area\n", @@ -4216,65 +4209,72 @@ static void show_purge_info(struct seq_file *m) va->va_end - va->va_start); } spin_unlock(&purge_vmap_area_lock); + mutex_unlock(&vmap_purge_lock); } =20 static int s_show(struct seq_file *m, void *p) { + struct cpu_vmap_zone *z; struct vmap_area *va; struct vm_struct *v; + int i; =20 - va =3D list_entry(p, struct vmap_area, list); + for_each_possible_cpu(i) { + z =3D per_cpu_ptr(&cpu_vmap_zone, i); =20 - if (!va->vm) { - if (va->flags & VMAP_RAM) - seq_printf(m, "0x%pK-0x%pK %7ld vm_map_ram\n", - (void *)va->va_start, (void *)va->va_end, - va->va_end - va->va_start); + fbl_lock(z, BUSY); + list_for_each_entry(va, &fbl_head(z, BUSY), list) { + if (!va->vm) { + if (va->flags & VMAP_RAM) + seq_printf(m, "0x%pK-0x%pK %7ld vm_map_ram\n", + (void *)va->va_start, (void *)va->va_end, + va->va_end - va->va_start); =20 - goto final; - } + continue; + } =20 - v =3D va->vm; + v =3D va->vm; =20 - seq_printf(m, "0x%pK-0x%pK %7ld", - v->addr, v->addr + v->size, v->size); + seq_printf(m, "0x%pK-0x%pK %7ld", + v->addr, v->addr + v->size, v->size); =20 - if (v->caller) - seq_printf(m, " %pS", v->caller); + if (v->caller) + seq_printf(m, " %pS", v->caller); =20 - if (v->nr_pages) - seq_printf(m, " pages=3D%d", v->nr_pages); + if (v->nr_pages) + seq_printf(m, " pages=3D%d", v->nr_pages); =20 - if (v->phys_addr) - seq_printf(m, " phys=3D%pa", &v->phys_addr); + if (v->phys_addr) + seq_printf(m, " phys=3D%pa", &v->phys_addr); =20 - if (v->flags & VM_IOREMAP) - seq_puts(m, " ioremap"); + if (v->flags & VM_IOREMAP) + seq_puts(m, " ioremap"); =20 - if (v->flags & VM_ALLOC) - seq_puts(m, " vmalloc"); + if (v->flags & VM_ALLOC) + seq_puts(m, " vmalloc"); =20 - if (v->flags & VM_MAP) - seq_puts(m, " vmap"); + if (v->flags & VM_MAP) + seq_puts(m, " vmap"); =20 - if (v->flags & VM_USERMAP) - seq_puts(m, " user"); + if (v->flags & VM_USERMAP) + seq_puts(m, " user"); =20 - if (v->flags & VM_DMA_COHERENT) - seq_puts(m, " dma-coherent"); + if (v->flags & VM_DMA_COHERENT) + seq_puts(m, " dma-coherent"); =20 - if (is_vmalloc_addr(v->pages)) - seq_puts(m, " vpages"); + if (is_vmalloc_addr(v->pages)) + seq_puts(m, " vpages"); =20 - show_numa_info(m, v); - seq_putc(m, '\n'); + show_numa_info(m, v); + seq_putc(m, '\n'); + } + fbl_unlock(z, BUSY); + } =20 /* * As a final step, dump "unpurged" areas. */ -final: - if (list_is_last(&va->list, &fbl_head((&per_cpu(cpu_vmap_zone, 0)), BUSY)= )) - show_purge_info(m); + show_purge_info(m); =20 return 0; } --=20 2.30.2 From nobody Sun Feb 8 02:21:39 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id DD5E0C7EE2A for ; Mon, 22 May 2023 11:15:18 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232071AbjEVLPR (ORCPT ); Mon, 22 May 2023 07:15:17 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58092 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233017AbjEVLOC (ORCPT ); Mon, 22 May 2023 07:14:02 -0400 Received: from mail-ej1-x62e.google.com (mail-ej1-x62e.google.com [IPv6:2a00:1450:4864:20::62e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 69C6640F4 for ; Mon, 22 May 2023 04:08:59 -0700 (PDT) Received: by mail-ej1-x62e.google.com with SMTP id a640c23a62f3a-96f99222e80so434939666b.1 for ; Mon, 22 May 2023 04:08:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1684753738; x=1687345738; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=fdmHm9JMTKbIdAZzyhemwDZBEVKcMOyv00QDGf853lw=; b=W+JtWfgKVMFA959Uc5C8wiOgCsYc9+SYai8nupDO9KlapBo9VlFnbVWgJAQ004f/QK mZQgkKrt4awtWG7HH1AMVsq0BI7HDBAsUxUanMdXJuT1NjGXA8j91P6MNQVZ7/2q/uOl u8b46ysLxNwbLImafPVAYrKfHx5ge512UBdEXib49XuWenSq7+MfDhUI4cQN2Y29QWPG 8ukh8J/sV1fqVvb+o7S1LN2QlrXlkMocODGZyaNkalUBH3toAXu0mQeN6PrvNuypiv2Q zaaZIwTdJUy/uadYITY2t/C6AJEJ4AgkDslYAlw0hmiXnARf/cNIGN7mPPlYm11bAx6x 3eRA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1684753738; x=1687345738; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=fdmHm9JMTKbIdAZzyhemwDZBEVKcMOyv00QDGf853lw=; b=Lx6m+HdMS/Hwxc8nsiCZCqf+5OrtGd0GNPZp7yP3vX4SWQLx+Of9EE6kX9eAqNBPXS ddjXsV3jw40kkkRzWUCSP5nti6tn7MQRSZGOyLPHkk0UySnpPc95qSRzS4QL5tBr97bg llzIQ0YKwGKvpacQOLG8yiCvSRDkDavvl1aJ/n6Tg2VI7dYdHQNCBqrJFCAgB3Y2E0Ev t1ubtbXFgWMAfcUK69lBRAJXMIF3n02lBhbRrqRSswOar6wVGJdI/ojm85W9rPWDb74z Hi9uPOMlVBSWhLrg/yagusFcy9kgo2sr31MrghxeQyE7pvVQqs7P1zeAL4j8/PoW3U7n 7iLg== X-Gm-Message-State: AC+VfDxLxaHdCpzTR14qWJwIxrBc8oQ10n/H8TvjbeHWnM6mytlbJWT3 u+BKDdhk9vYWkl5jA+/OIBU= X-Google-Smtp-Source: ACHHUZ4lpJZI6b774mW0GGFUcH9S7bFeeu1TaXyRQOPcceEGL/AxG3iytu1ZZc95ZLsmsCWn+xcxSw== X-Received: by 2002:a17:907:3eaa:b0:96f:8d00:43be with SMTP id hs42-20020a1709073eaa00b0096f8d0043bemr6754703ejc.0.1684753737692; Mon, 22 May 2023 04:08:57 -0700 (PDT) Received: from pc638.lan ([155.137.26.201]) by smtp.gmail.com with ESMTPSA id lx19-20020a170906af1300b0094f07545d40sm2949453ejb.220.2023.05.22.04.08.57 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 22 May 2023 04:08:57 -0700 (PDT) From: "Uladzislau Rezki (Sony)" To: linux-mm@kvack.org, Andrew Morton Cc: LKML , Baoquan He , Lorenzo Stoakes , Christoph Hellwig , Matthew Wilcox , "Liam R . Howlett" , Dave Chinner , "Paul E . McKenney" , Joel Fernandes , Uladzislau Rezki , Oleksiy Avramchenko Subject: [PATCH 7/9] mm: vmalloc: Insert lazy-VA per-cpu zone Date: Mon, 22 May 2023 13:08:47 +0200 Message-Id: <20230522110849.2921-8-urezki@gmail.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20230522110849.2921-1-urezki@gmail.com> References: <20230522110849.2921-1-urezki@gmail.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Similar to busy VAs, lazy ones are stored per a CPU zone also. Freed address is converted into a correct zone it belongs to and resides there for further handling. Such approach does not require to have any global locking primitive, instead an access becomes scalable to number of CPUs. This patch removes a global purge-lock, global purge-tree and list. Signed-off-by: Uladzislau Rezki (Sony) --- mm/vmalloc.c | 127 ++++++++++++++++++++++++++++----------------------- 1 file changed, 71 insertions(+), 56 deletions(-) diff --git a/mm/vmalloc.c b/mm/vmalloc.c index dd83deb5ef4f..fe993c0561dd 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -734,10 +734,6 @@ static DEFINE_SPINLOCK(free_vmap_area_lock); LIST_HEAD(vmap_area_list); static bool vmap_initialized __read_mostly; =20 -static struct rb_root purge_vmap_area_root =3D RB_ROOT; -static LIST_HEAD(purge_vmap_area_list); -static DEFINE_SPINLOCK(purge_vmap_area_lock); - /* * This kmem_cache is used for vmap_area objects. Instead of * allocating from slab we reuse an object from this cache to @@ -1792,39 +1788,17 @@ static DEFINE_MUTEX(vmap_purge_lock); /* for per-CPU blocks */ static void purge_fragmented_blocks_allcpus(void); =20 -/* - * Purges all lazily-freed vmap areas. - */ -static bool __purge_vmap_area_lazy(unsigned long start, unsigned long end) +static unsigned long +purge_cpu_vmap_zone(struct cpu_vmap_zone *z) { - unsigned long resched_threshold; - unsigned int num_purged_areas =3D 0; - struct list_head local_purge_list; + unsigned long num_purged_areas =3D 0; struct vmap_area *va, *n_va; =20 - lockdep_assert_held(&vmap_purge_lock); - - spin_lock(&purge_vmap_area_lock); - purge_vmap_area_root =3D RB_ROOT; - list_replace_init(&purge_vmap_area_list, &local_purge_list); - spin_unlock(&purge_vmap_area_lock); - - if (unlikely(list_empty(&local_purge_list))) + if (list_empty(&z->purge_list)) goto out; =20 - start =3D min(start, - list_first_entry(&local_purge_list, - struct vmap_area, list)->va_start); - - end =3D max(end, - list_last_entry(&local_purge_list, - struct vmap_area, list)->va_end); - - flush_tlb_kernel_range(start, end); - resched_threshold =3D lazy_max_pages() << 1; - spin_lock(&free_vmap_area_lock); - list_for_each_entry_safe(va, n_va, &local_purge_list, list) { + list_for_each_entry_safe(va, n_va, &z->purge_list, list) { unsigned long nr =3D (va->va_end - va->va_start) >> PAGE_SHIFT; unsigned long orig_start =3D va->va_start; unsigned long orig_end =3D va->va_end; @@ -1846,13 +1820,57 @@ static bool __purge_vmap_area_lazy(unsigned long st= art, unsigned long end) =20 atomic_long_sub(nr, &vmap_lazy_nr); num_purged_areas++; - - if (atomic_long_read(&vmap_lazy_nr) < resched_threshold) - cond_resched_lock(&free_vmap_area_lock); } spin_unlock(&free_vmap_area_lock); =20 out: + return num_purged_areas; +} + +/* + * Purges all lazily-freed vmap areas. + */ +static bool __purge_vmap_area_lazy(unsigned long start, unsigned long end) +{ + unsigned long num_purged_areas =3D 0; + struct cpu_vmap_zone *z; + int need_purge =3D 0; + int i; + + lockdep_assert_held(&vmap_purge_lock); + + for_each_possible_cpu(i) { + z =3D per_cpu_ptr(&cpu_vmap_zone, i); + INIT_LIST_HEAD(&z->purge_list); + + if (RB_EMPTY_ROOT(&fbl_root(z, LAZY))) + continue; + + fbl_lock(z, LAZY); + WRITE_ONCE(fbl(z, LAZY, root.rb_node), NULL); + list_replace_init(&fbl_head(z, LAZY), &z->purge_list); + fbl_unlock(z, LAZY); + + start =3D min(start, + list_first_entry(&z->purge_list, + struct vmap_area, list)->va_start); + + end =3D max(end, + list_last_entry(&z->purge_list, + struct vmap_area, list)->va_end); + + need_purge++; + } + + if (need_purge) { + flush_tlb_kernel_range(start, end); + + for_each_possible_cpu(i) { + z =3D per_cpu_ptr(&cpu_vmap_zone, i); + num_purged_areas +=3D purge_cpu_vmap_zone(z); + } + } + trace_purge_vmap_area_lazy(start, end, num_purged_areas); return num_purged_areas > 0; } @@ -1870,16 +1888,9 @@ static void purge_vmap_area_lazy(void) =20 static void drain_vmap_area_work(struct work_struct *work) { - unsigned long nr_lazy; - - do { - mutex_lock(&vmap_purge_lock); - __purge_vmap_area_lazy(ULONG_MAX, 0); - mutex_unlock(&vmap_purge_lock); - - /* Recheck if further work is required. */ - nr_lazy =3D atomic_long_read(&vmap_lazy_nr); - } while (nr_lazy > lazy_max_pages()); + mutex_lock(&vmap_purge_lock); + __purge_vmap_area_lazy(ULONG_MAX, 0); + mutex_unlock(&vmap_purge_lock); } =20 /* @@ -1889,6 +1900,7 @@ static void drain_vmap_area_work(struct work_struct *= work) */ static void free_vmap_area_noflush(struct vmap_area *va) { + struct cpu_vmap_zone *z =3D addr_to_cvz(va->va_start); unsigned long nr_lazy_max =3D lazy_max_pages(); unsigned long va_start =3D va->va_start; unsigned long nr_lazy; @@ -1902,10 +1914,9 @@ static void free_vmap_area_noflush(struct vmap_area = *va) /* * Merge or place it to the purge tree/list. */ - spin_lock(&purge_vmap_area_lock); - merge_or_add_vmap_area(va, - &purge_vmap_area_root, &purge_vmap_area_list); - spin_unlock(&purge_vmap_area_lock); + fbl_lock(z, LAZY); + merge_or_add_vmap_area(va, &fbl_root(z, LAZY), &fbl_head(z, LAZY)); + fbl_unlock(z, LAZY); =20 trace_free_vmap_area_noflush(va_start, nr_lazy, nr_lazy_max); =20 @@ -4199,17 +4210,21 @@ static void show_numa_info(struct seq_file *m, stru= ct vm_struct *v) =20 static void show_purge_info(struct seq_file *m) { + struct cpu_vmap_zone *z; struct vmap_area *va; + int i; =20 - mutex_lock(&vmap_purge_lock); - spin_lock(&purge_vmap_area_lock); - list_for_each_entry(va, &purge_vmap_area_list, list) { - seq_printf(m, "0x%pK-0x%pK %7ld unpurged vm_area\n", - (void *)va->va_start, (void *)va->va_end, - va->va_end - va->va_start); + for_each_possible_cpu(i) { + z =3D per_cpu_ptr(&cpu_vmap_zone, i); + + fbl_lock(z, LAZY); + list_for_each_entry(va, &fbl_head(z, LAZY), list) { + seq_printf(m, "0x%pK-0x%pK %7ld unpurged vm_area\n", + (void *)va->va_start, (void *)va->va_end, + va->va_end - va->va_start); + } + fbl_unlock(z, LAZY); } - spin_unlock(&purge_vmap_area_lock); - mutex_unlock(&vmap_purge_lock); } =20 static int s_show(struct seq_file *m, void *p) --=20 2.30.2 From nobody Sun Feb 8 02:21:39 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 669B3C7EE2A for ; Mon, 22 May 2023 11:15:10 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233277AbjEVLO4 (ORCPT ); Mon, 22 May 2023 07:14:56 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58096 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233020AbjEVLOC (ORCPT ); Mon, 22 May 2023 07:14:02 -0400 Received: from mail-ej1-x629.google.com (mail-ej1-x629.google.com [IPv6:2a00:1450:4864:20::629]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 5C14D1BD3 for ; Mon, 22 May 2023 04:09:00 -0700 (PDT) Received: by mail-ej1-x629.google.com with SMTP id a640c23a62f3a-96fbe7fbdd4so291570466b.3 for ; Mon, 22 May 2023 04:09:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1684753739; x=1687345739; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=CbQClwJJa5Q5OTrsV+8XOhbG6pyJZXEhMwzuSrOaxe4=; b=Ad+N9/BxXUiItff9Ka8gp9m6Ut+hvQw/iiM/znN/E/FjnWbeKMX497Feia+wUTQL7P nn0HMTXM+eAOwCIXtFoVtxCNH0LHncvSzMz7xzwQ1Srikc1cgV966ZS1qa7YdSUE18l3 cG1SVgkhO1ucaJlN4YMO1/+KLZpOHs16VnpaYal4+QLxI/ktvW/6nIUZRewRfRy6T7uj t6Bkz1DHlo2XaUESkC1cNdMlQINss0j4ahV0Ld6LZrYiZAUobHEQcZF2xuwkm88kQ9RE EafR2f2o+2S2vnNz55Q7/rEKNFxFSEEhyB+7TfomnLLk3I+4DN8cvc7+ZtkoKPBd38H7 HuUg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1684753739; x=1687345739; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=CbQClwJJa5Q5OTrsV+8XOhbG6pyJZXEhMwzuSrOaxe4=; b=W9lBUUaziEYILHl3tGuuMchwrn5i6Eo4pq3Z5ljshPKtiOgI4YLIGqfc7WUa4ZnlTp 8IEZuBuX7st7UB6jFh1ItqCI3L0+d9rO4+VOoKV4lv3D8g+S1uFn1WUW32q48byxYMCR RgLf7ovGbQGpACI3uHjrpqUnq/9zC7O1wIuPlvmQDMAPRHwr2IwR2CCdJ+PSJ+m6HHzY Rsxs/37NZN7g5nXwsMqElSVmy2qyuNcy8a5ZhX5aKcu2FHCQGnI2g9wtcqqjnsLGQqp2 YtBLHROc4J6dVbvfVlFOpeqkgoJimiUOyxEKSjWAnPkU8vsp38rp41+7H9UXfebFrdtU 2Zzg== X-Gm-Message-State: AC+VfDzpmXVbb0c2YVj3m0ZgFmJUtMG8kobxUUErdn8uO7n0+NAJ9SnN DFd7QrTTlHZrzMfjafgM7YY= X-Google-Smtp-Source: ACHHUZ4pAASV3ZfWJMFVdFF/6izNH1fGukazCQNkIlZEbt/DA1kUADDrO481FwKNM2C/1frFuAV8LA== X-Received: by 2002:a17:907:1c07:b0:96f:d8a9:d045 with SMTP id nc7-20020a1709071c0700b0096fd8a9d045mr3465283ejc.59.1684753738602; Mon, 22 May 2023 04:08:58 -0700 (PDT) Received: from pc638.lan ([155.137.26.201]) by smtp.gmail.com with ESMTPSA id lx19-20020a170906af1300b0094f07545d40sm2949453ejb.220.2023.05.22.04.08.57 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 22 May 2023 04:08:58 -0700 (PDT) From: "Uladzislau Rezki (Sony)" To: linux-mm@kvack.org, Andrew Morton Cc: LKML , Baoquan He , Lorenzo Stoakes , Christoph Hellwig , Matthew Wilcox , "Liam R . Howlett" , Dave Chinner , "Paul E . McKenney" , Joel Fernandes , Uladzislau Rezki , Oleksiy Avramchenko Subject: [PATCH 8/9] mm: vmalloc: Offload free_vmap_area_lock global lock Date: Mon, 22 May 2023 13:08:48 +0200 Message-Id: <20230522110849.2921-9-urezki@gmail.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20230522110849.2921-1-urezki@gmail.com> References: <20230522110849.2921-1-urezki@gmail.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Introduce a fast path of allocation sequence, that consists of per-cpu path and fallback mechanism which is used when a request can not be accomplished by fast track. A fast track pre-loads a chunk from a global vmap heap directly into its per-cpu zone, following by clipping the chunk based on allocation request. This technique allows to offload a global free_vmap_area_lock making an allocation path to be serialized to number of CPUs in a system. Signed-off-by: Uladzislau Rezki (Sony) --- mm/vmalloc.c | 127 +++++++++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 123 insertions(+), 4 deletions(-) diff --git a/mm/vmalloc.c b/mm/vmalloc.c index fe993c0561dd..8054b8bf6c18 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -1642,6 +1642,93 @@ preload_this_cpu_lock(spinlock_t *lock, gfp_t gfp_ma= sk, int node) kmem_cache_free(vmap_area_cachep, va); } =20 +static unsigned long +this_cpu_zone_alloc_fill(struct cpu_vmap_zone *z, + unsigned long size, unsigned long align, + gfp_t gfp_mask, int node) +{ + unsigned long addr =3D VMALLOC_END; + struct vmap_area *va; + + /* + * It still can race. One task sets a progress to + * 1 a second one gets preempted on entry, the first + * zeroed the progress flag and second proceed with + * an extra prefetch. + */ + if (atomic_xchg(&z->fill_in_progress, 1)) + return addr; + + va =3D kmem_cache_alloc_node(vmap_area_cachep, gfp_mask, node); + if (unlikely(!va)) + goto out; + + spin_lock(&free_vmap_area_lock); + addr =3D __alloc_vmap_area(&free_vmap_area_root, &free_vmap_area_list, + cvz_size, 1, VMALLOC_START, VMALLOC_END); + spin_unlock(&free_vmap_area_lock); + + if (addr =3D=3D VMALLOC_END) { + kmem_cache_free(vmap_area_cachep, va); + goto out; + } + + va->va_start =3D addr; + va->va_end =3D addr + cvz_size; + + fbl_lock(z, FREE); + va =3D merge_or_add_vmap_area_augment(va, + &fbl_root(z, FREE), &fbl_head(z, FREE)); + addr =3D va_alloc(va, &fbl_root(z, FREE), &fbl_head(z, FREE), + size, align, VMALLOC_START, VMALLOC_END); + fbl_unlock(z, FREE); + +out: + atomic_set(&z->fill_in_progress, 0); + return addr; +} + +static unsigned long +this_cpu_zone_alloc(unsigned long size, unsigned long align, gfp_t gfp_mas= k, int node) +{ + struct cpu_vmap_zone *z =3D raw_cpu_ptr(&cpu_vmap_zone); + unsigned long extra =3D align > PAGE_SIZE ? align : 0; + unsigned long addr =3D VMALLOC_END, left =3D 0; + + /* + * It is disabled, fallback to a global heap. + */ + if (cvz_size =3D=3D ULONG_MAX) + return addr; + + /* + * Any allocation bigger/equal than one half of + * a zone-size will fallback to a global heap. + */ + if (cvz_size / (size + extra) < 3) + return addr; + + if (RB_EMPTY_ROOT(&fbl_root(z, FREE))) + goto fill; + + fbl_lock(z, FREE); + addr =3D __alloc_vmap_area(&fbl_root(z, FREE), &fbl_head(z, FREE), + size, align, VMALLOC_START, VMALLOC_END); + + if (addr =3D=3D VMALLOC_END) + left =3D get_subtree_max_size(fbl_root(z, FREE).rb_node); + fbl_unlock(z, FREE); + +fill: + /* + * A low watermark is 3 pages. + */ + if (addr =3D=3D VMALLOC_END && left < 4 * PAGE_SIZE) + addr =3D this_cpu_zone_alloc_fill(z, size, align, gfp_mask, node); + + return addr; +} + /* * Allocate a region of KVA of the specified size and alignment, within the * vstart and vend. @@ -1678,11 +1765,21 @@ static struct vmap_area *alloc_vmap_area(unsigned l= ong size, */ kmemleak_scan_area(&va->rb_node, SIZE_MAX, gfp_mask); =20 + /* + * Fast path allocation, start with it. + */ + if (vstart =3D=3D VMALLOC_START && vend =3D=3D VMALLOC_END) + addr =3D this_cpu_zone_alloc(size, align, gfp_mask, node); + else + addr =3D vend; + retry: - preload_this_cpu_lock(&free_vmap_area_lock, gfp_mask, node); - addr =3D __alloc_vmap_area(&free_vmap_area_root, &free_vmap_area_list, - size, align, vstart, vend); - spin_unlock(&free_vmap_area_lock); + if (addr =3D=3D vend) { + preload_this_cpu_lock(&free_vmap_area_lock, gfp_mask, node); + addr =3D __alloc_vmap_area(&free_vmap_area_root, &free_vmap_area_list, + size, align, vstart, vend); + spin_unlock(&free_vmap_area_lock); + } =20 trace_alloc_vmap_area(addr, size, align, vstart, vend, addr =3D=3D vend); =20 @@ -1827,6 +1924,27 @@ purge_cpu_vmap_zone(struct cpu_vmap_zone *z) return num_purged_areas; } =20 +static void +drop_cpu_vmap_cache(struct cpu_vmap_zone *z) +{ + struct vmap_area *va, *n_va; + LIST_HEAD(free_head); + + if (RB_EMPTY_ROOT(&fbl_root(z, FREE))) + return; + + fbl_lock(z, FREE); + WRITE_ONCE(fbl(z, FREE, root.rb_node), NULL); + list_replace_init(&fbl_head(z, FREE), &free_head); + fbl_unlock(z, FREE); + + spin_lock(&free_vmap_area_lock); + list_for_each_entry_safe(va, n_va, &free_head, list) + merge_or_add_vmap_area_augment(va, + &free_vmap_area_root, &free_vmap_area_list); + spin_unlock(&free_vmap_area_lock); +} + /* * Purges all lazily-freed vmap areas. */ @@ -1868,6 +1986,7 @@ static bool __purge_vmap_area_lazy(unsigned long star= t, unsigned long end) for_each_possible_cpu(i) { z =3D per_cpu_ptr(&cpu_vmap_zone, i); num_purged_areas +=3D purge_cpu_vmap_zone(z); + drop_cpu_vmap_cache(z); } } =20 --=20 2.30.2 From nobody Sun Feb 8 02:21:39 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 55F88C7EE2D for ; Mon, 22 May 2023 11:15:10 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233254AbjEVLOx (ORCPT ); Mon, 22 May 2023 07:14:53 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58350 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233118AbjEVLOC (ORCPT ); Mon, 22 May 2023 07:14:02 -0400 Received: from mail-ej1-x62f.google.com (mail-ej1-x62f.google.com [IPv6:2a00:1450:4864:20::62f]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0FE4D1BD7 for ; Mon, 22 May 2023 04:09:01 -0700 (PDT) Received: by mail-ej1-x62f.google.com with SMTP id a640c23a62f3a-96f683e8855so488862466b.2 for ; Mon, 22 May 2023 04:09:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1684753739; x=1687345739; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=f2bGuuQYL8b+xbvuLn4PcYekQUWt9yPN5iQ314l4lMI=; b=PvB7xNb9ygkZOuxzTtzRZdOhB6MfhJSa/XRAM+BYE1xoLfv+ewUOyX4VD8Bu0FXaOK AXlUTKkyIK2WVRsD39QMuEntv1HpM65yERdXpxKJXgRGMC2pW3R98Ogtx7X6mgbxBHMR ldToVDFezt0XeMtgZh1MaqlUYU2be1G2xPupT/rIWTKOSu/1KFkLMDIX1H3BLSGe29sO Fg3kOpF9fGfKN44fAOhFpBwP9Z7IQ03rJSGvaYH1M1Nh+mhzwxadyfvgGc5HplBYwh3q fzDEE5gFHzKH0XnCa9ZqMPOjoi2odYahJQNJL0GrEIc+/7U+m3d4QYr1A8eWf9czKONh 1f4A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1684753739; x=1687345739; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=f2bGuuQYL8b+xbvuLn4PcYekQUWt9yPN5iQ314l4lMI=; b=LM+YizE6+kL5AztjvEBeZDF2GFzwha7k4U7RwKQMWpQlC5JMNqhm1KAaaA4hnrtFXG I+ptpKfoouk7VR7SZ4jMnWH5LYpFJbEPIn25e2fgpOTir9iXRdc9x0XOiPxB1LdxkKVw 4JxJiZPhuaMmSJT9IH+CEFap64ewL8A4YegG5M925mtJjfbLzXnbZXmk7BxyGqCDO80q crpYZFF/lkCakuVppPU8RhlbCkwCaUIMlOPz7mkOAUa3CfE7Z+igvy0lxslRqXvH/38T sAPSZIo8sqZarYSnVIjPaVByT64/qm+gvC9vezWjPoAd/w9IQIILTwniNusmkN+z4ogW eltw== X-Gm-Message-State: AC+VfDxni/6dRANrTBhGmZVtLDUdQTocIkGfh3Lvth9xfnRtwkM7H1ap i9al701hyGLJ2IqiC8NtpXA= X-Google-Smtp-Source: ACHHUZ7OjSrTldnv+rUkfYY8jX9+e0NK8wBEQHN/GocoDZWHojfGcLerDTURKKOgb/QQDOHh1JKUqA== X-Received: by 2002:a17:907:a42a:b0:96f:a7d9:6e86 with SMTP id sg42-20020a170907a42a00b0096fa7d96e86mr6231547ejc.1.1684753739445; Mon, 22 May 2023 04:08:59 -0700 (PDT) Received: from pc638.lan ([155.137.26.201]) by smtp.gmail.com with ESMTPSA id lx19-20020a170906af1300b0094f07545d40sm2949453ejb.220.2023.05.22.04.08.58 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 22 May 2023 04:08:59 -0700 (PDT) From: "Uladzislau Rezki (Sony)" To: linux-mm@kvack.org, Andrew Morton Cc: LKML , Baoquan He , Lorenzo Stoakes , Christoph Hellwig , Matthew Wilcox , "Liam R . Howlett" , Dave Chinner , "Paul E . McKenney" , Joel Fernandes , Uladzislau Rezki , Oleksiy Avramchenko Subject: [PATCH 9/9] mm: vmalloc: Scale and activate cvz_size Date: Mon, 22 May 2023 13:08:49 +0200 Message-Id: <20230522110849.2921-10-urezki@gmail.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20230522110849.2921-1-urezki@gmail.com> References: <20230522110849.2921-1-urezki@gmail.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Scale a zone size to number of CPUs. Each power of two value adds extra 100 * PAGE_SIZE bytes to a zone size. A maximum zone size limit is set to 4M, it is reached if number of CPUs=3D1024 and PAGE_SIZE=3D4096 bytes. For 32-bit or single core systems cvz_size is set to ULONG_MAX. It means that a pre-loading technique is deactivated. Signed-off-by: Uladzislau Rezki (Sony) --- mm/vmalloc.c | 15 +++++++++++++-- 1 file changed, 13 insertions(+), 2 deletions(-) diff --git a/mm/vmalloc.c b/mm/vmalloc.c index 8054b8bf6c18..a5e6956a8da4 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -806,8 +806,6 @@ struct cpu_vmap_zone { }; =20 static DEFINE_PER_CPU(struct cpu_vmap_zone, cpu_vmap_zone); - -/* Disable a per-cpu caching. */ static __read_mostly unsigned long cvz_size =3D ULONG_MAX; =20 static inline unsigned int @@ -4490,6 +4488,19 @@ static void vmap_init_pcpu_zones(void) spin_lock_init(&z->fbl[j].lock); } } + + /* + * Scale a zone size to number of CPUs. Each power of two + * value adds extra 100 pages to a zone size. A high-threshold + * limit is set to 4M. It can be reached if number of CPUs=3D1024 + * whereas a PAGE_SIZE is 4096 bytes. + */ +#if BITS_PER_LONG =3D=3D 64 + if (i > 1) { + cvz_size =3D fls(i) * (100 * PAGE_SIZE); + cvz_size =3D min(cvz_size, (unsigned long) SZ_4M); + } +#endif } =20 void __init vmalloc_init(void) --=20 2.30.2