From nobody Fri May 8 06:50:14 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7FD8BC433EF for ; Mon, 9 May 2022 06:38:00 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229577AbiEIGlH (ORCPT ); Mon, 9 May 2022 02:41:07 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41240 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234638AbiEIGdc (ORCPT ); Mon, 9 May 2022 02:33:32 -0400 Received: from mail-pj1-x1036.google.com (mail-pj1-x1036.google.com [IPv6:2607:f8b0:4864:20::1036]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 466FE189951 for ; Sun, 8 May 2022 23:29:36 -0700 (PDT) Received: by mail-pj1-x1036.google.com with SMTP id iq2-20020a17090afb4200b001d93cf33ae9so16127453pjb.5 for ; Sun, 08 May 2022 23:29:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20210112.gappssmtp.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=ODWBdYqZzg8lChYiYiJ2sSIUo4NFfWVrgG6YTxSVP1A=; b=5VaRvE+1b0+qecOZ4GJucoJxjsjPux4wEKZiPchnZSOlzntRm47r+/fNfyNO94gM33 b2lF5essaJ0F0u7Yc3+XcwJ3ydFrKhYUKiGKsRV8SfCbsL08+ZifjY0kGM1Bwg6pxtTr iW5uTN2ddKymbe65caoHqSSKFdZBSnCQXHSFzM4CASc2YO1ihZmxvh8RKvlNktIgYEXn 1tp6D4xItTZeORAmkea3Cm2vzwvGkoZZ2RvAMyr4bAENdGLj8/KCQD0sxRGO152aPijt X49U+u+mQOasEcUMV/c5XBsqLXy1AzmQaatvdPpF2zUyAS4aQG9SjLl+lu2F2jAjSjet qgRQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=ODWBdYqZzg8lChYiYiJ2sSIUo4NFfWVrgG6YTxSVP1A=; b=fgZ1aMpvM5IMXisGnXTXDbq2OoVUaRUIqDfz1BNYs45qsRDauM5YDpxOexK+zrFgfz VbJTrKFY7X6qlXc6cCNQZAY6/6VMXx0EggHMe29cv0siuTI7mnPj5FhNEdJF2mCEMZys o7P8IKg6yUdv2ohLcOPTGYT35n553ZkzACqwnKyJm4Pt0ypes+Fx6PL1OfEcTeTYIXj0 6xM45+KvNpglv5CYsO6c0Ntx3+D2xBBG1hkdpbGKdgHhE+2Vw9mbA+hllP6XWQrTfviY I8CL5mrVYv0cOYJmQG/GtpGewAZ/pL4ACkqAnYiDMXfjrtFrdbeJpXJM3iRto8PLbgV8 HmaQ== X-Gm-Message-State: AOAM530CN/dt88t1O55cEERcGISjL7b7YcUmqeg76b5i+0nKtR4IZrLW 2ykHlyzyq16tUWhBLfQS1ZPMMw== X-Google-Smtp-Source: ABdhPJwNiK5mpR/izG+cjL30Z+YU1i7fHNaGnW9U69VmZcwf6IsqUGfxeQRhRibbPJ5pQZmZ1kOS/A== X-Received: by 2002:a17:903:2341:b0:15e:c7c4:2bb9 with SMTP id c1-20020a170903234100b0015ec7c42bb9mr14922190plh.108.1652077762701; Sun, 08 May 2022 23:29:22 -0700 (PDT) Received: from FVFYT0MHHV2J.bytedance.net ([139.177.225.250]) by smtp.gmail.com with ESMTPSA id k17-20020a170902ba9100b0015e8d4eb2c2sm6162001pls.268.2022.05.08.23.29.16 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 08 May 2022 23:29:22 -0700 (PDT) From: Muchun Song To: corbet@lwn.net, mike.kravetz@oracle.com, akpm@linux-foundation.org, mcgrof@kernel.org, keescook@chromium.org, yzaikin@google.com, osalvador@suse.de, david@redhat.com, masahiroy@kernel.org Cc: linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, duanxiongchun@bytedance.com, smuchun@gmail.com, Muchun Song Subject: [PATCH v10 1/4] mm: hugetlb_vmemmap: disable hugetlb_optimize_vmemmap when struct page crosses page boundaries Date: Mon, 9 May 2022 14:27:00 +0800 Message-Id: <20220509062703.64249-2-songmuchun@bytedance.com> X-Mailer: git-send-email 2.32.0 (Apple Git-132) In-Reply-To: <20220509062703.64249-1-songmuchun@bytedance.com> References: <20220509062703.64249-1-songmuchun@bytedance.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" If the size of "struct page" is not the power of two but with the feature of minimizing overhead of struct page associated with each HugeTLB is enabled, then the vmemmap pages of HugeTLB will be corrupted after remapping (panic is about to happen in theory). But this only exists when !CONFIG_MEMCG && !CONFIG_SLUB on x86_64. However, it is not a conventional configuration nowadays. So it is not a real word issue, just the result of a code review. But we cannot prevent anyone from configuring that combined configure. This hugetlb_optimize_vmemmap should be disable in this case to fix this issue. Signed-off-by: Muchun Song Reviewed-by: Mike Kravetz Acked-by: David Hildenbrand --- mm/hugetlb_vmemmap.c | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c index 29554c6ef2ae..6254bb2d4ae5 100644 --- a/mm/hugetlb_vmemmap.c +++ b/mm/hugetlb_vmemmap.c @@ -28,12 +28,6 @@ EXPORT_SYMBOL(hugetlb_optimize_vmemmap_key); =20 static int __init hugetlb_vmemmap_early_param(char *buf) { - /* We cannot optimize if a "struct page" crosses page boundaries. */ - if (!is_power_of_2(sizeof(struct page))) { - pr_warn("cannot free vmemmap pages because \"struct page\" crosses page = boundaries\n"); - return 0; - } - if (!buf) return -EINVAL; =20 @@ -119,6 +113,12 @@ void __init hugetlb_vmemmap_init(struct hstate *h) if (!hugetlb_optimize_vmemmap_enabled()) return; =20 + if (!is_power_of_2(sizeof(struct page))) { + pr_warn_once("cannot optimize vmemmap pages because \"struct page\" cros= ses page boundaries\n"); + static_branch_disable(&hugetlb_optimize_vmemmap_key); + return; + } + vmemmap_pages =3D (nr_pages * sizeof(struct page)) >> PAGE_SHIFT; /* * The head page is not to be freed to buddy allocator, the other tail --=20 2.11.0 From nobody Fri May 8 06:50:14 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 65B59C43219 for ; Mon, 9 May 2022 06:38:43 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235513AbiEIGkZ (ORCPT ); Mon, 9 May 2022 02:40:25 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41608 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234672AbiEIGdg (ORCPT ); Mon, 9 May 2022 02:33:36 -0400 Received: from mail-pf1-x42b.google.com (mail-pf1-x42b.google.com [IPv6:2607:f8b0:4864:20::42b]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0EFA818A69C for ; Sun, 8 May 2022 23:29:44 -0700 (PDT) Received: by mail-pf1-x42b.google.com with SMTP id j6so11405985pfe.13 for ; Sun, 08 May 2022 23:29:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20210112.gappssmtp.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=TxGePxN899QZhoK9/ghhqKhjvB6kwBatns9zvkXLNoM=; b=YxJIkB+tjVeh+3RaEuq0lx22G52hqRP2Q41/98aBO7PbNanU0ZAG/2VYybdqoy7A9G NMUXQFOd0c0pmsmXy+HaT2LDFlWv5g5lewOhRall6jZaP7EaHXIRTmKbhLMvQJu8U0Z2 KD0/h6+ozi2Qm+EGZh7LEP+B6OSR1gFw+hOA2mZy2JOIZOdQ9Gu5iF2EblTcQHtyBNHD HFdPX8PW+nM7stUaD0vTKqmz6t8ps5QUjKtNLW/JM7V457xk441c9ZwHf0Uaqa3hQqIJ sD5wJ69vhdY8VHBVEl7SGO6pZjj3slZUUYcA8gL9PtZ4GTtPMNGbsoC7QudoKYbgfAod B3aA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=TxGePxN899QZhoK9/ghhqKhjvB6kwBatns9zvkXLNoM=; b=e1w1hXb+EsjBuHZC+f8AVY7ESbDFF3hmMhFcWWbhQepLW9ywvZjPLzO5N2BJXNMmOF k7nxVDgRRb8lWssdg9tWFADHwpMQbU2hDROY0gxndB3ZranC0BR6JuGAlcxxJD65Bd79 T6ml6KCDZInA96qrUVxpf6DJ4LVY+PRJl5oz1ZxP4UobPiL3tRYhxZip3WND4yv2eaTI +vN2Jl16aoq0Hw1Gkot/6ZsbFS0cN5+exEKy4eOIg7G71eW+CPy934KTnBOsa7Btrajw OqK8j5DYFXdDN+HIKPQM+kuyoy9d28GyJ0QwOIYQuSw2DtdimQ48s/+cELbKEBMHmbOK mQDA== X-Gm-Message-State: AOAM5334KxnzPfCzrbT7XRNPPjxaDNa1qCH2NLRuUatRNtrWDo1k61cr 1RICkwzJAb+0XZrmlcYd2kdsTQ== X-Google-Smtp-Source: ABdhPJx/EFqVRRy5eL0t3+AS10kjTooGOitYJZKGi7Xtf2ozkJgteJkT0pgvGVy/dc2m1ku/jQuGXw== X-Received: by 2002:a65:6a16:0:b0:39d:4f3:67e6 with SMTP id m22-20020a656a16000000b0039d04f367e6mr12242601pgu.84.1652077768485; Sun, 08 May 2022 23:29:28 -0700 (PDT) Received: from FVFYT0MHHV2J.bytedance.net ([139.177.225.250]) by smtp.gmail.com with ESMTPSA id k17-20020a170902ba9100b0015e8d4eb2c2sm6162001pls.268.2022.05.08.23.29.23 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 08 May 2022 23:29:28 -0700 (PDT) From: Muchun Song To: corbet@lwn.net, mike.kravetz@oracle.com, akpm@linux-foundation.org, mcgrof@kernel.org, keescook@chromium.org, yzaikin@google.com, osalvador@suse.de, david@redhat.com, masahiroy@kernel.org Cc: linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, duanxiongchun@bytedance.com, smuchun@gmail.com, Muchun Song Subject: [PATCH v10 2/4] mm: memory_hotplug: override memmap_on_memory when hugetlb_free_vmemmap=on Date: Mon, 9 May 2022 14:27:01 +0800 Message-Id: <20220509062703.64249-3-songmuchun@bytedance.com> X-Mailer: git-send-email 2.32.0 (Apple Git-132) In-Reply-To: <20220509062703.64249-1-songmuchun@bytedance.com> References: <20220509062703.64249-1-songmuchun@bytedance.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Optimizing HugeTLB vmemmap pages is not compatible with allocating memmap on hot added memory. If "hugetlb_free_vmemmap=3Don" and memory_hotplug.memmap_on_memory" are both passed on the kernel command line, optimizing hugetlb pages takes precedence. However, the global variable memmap_on_memory will still be set to 1, even though we will not try to allocate memmap on hot added memory. Also introduce mhp_memmap_on_memory() helper to move the definition of "memmap_on_memory" to the scope of CONFIG_MHP_MEMMAP_ON_MEMORY. In the next patch, mhp_memmap_on_memory() will also be exported to be used in hugetlb_vmemmap.c. Signed-off-by: Muchun Song Acked-by: Mike Kravetz --- mm/memory_hotplug.c | 32 ++++++++++++++++++++++++++------ 1 file changed, 26 insertions(+), 6 deletions(-) diff --git a/mm/memory_hotplug.c b/mm/memory_hotplug.c index 111684878fd9..a6101ae402f9 100644 --- a/mm/memory_hotplug.c +++ b/mm/memory_hotplug.c @@ -42,14 +42,36 @@ #include "internal.h" #include "shuffle.h" =20 +#ifdef CONFIG_MHP_MEMMAP_ON_MEMORY +static int memmap_on_memory_set(const char *val, const struct kernel_param= *kp) +{ + if (hugetlb_optimize_vmemmap_enabled()) + return 0; + return param_set_bool(val, kp); +} + +static const struct kernel_param_ops memmap_on_memory_ops =3D { + .flags =3D KERNEL_PARAM_OPS_FL_NOARG, + .set =3D memmap_on_memory_set, + .get =3D param_get_bool, +}; =20 /* * memory_hotplug.memmap_on_memory parameter */ static bool memmap_on_memory __ro_after_init; -#ifdef CONFIG_MHP_MEMMAP_ON_MEMORY -module_param(memmap_on_memory, bool, 0444); +module_param_cb(memmap_on_memory, &memmap_on_memory_ops, &memmap_on_memory= , 0444); MODULE_PARM_DESC(memmap_on_memory, "Enable memmap on memory for memory hot= plug"); + +static inline bool mhp_memmap_on_memory(void) +{ + return memmap_on_memory; +} +#else +static inline bool mhp_memmap_on_memory(void) +{ + return false; +} #endif =20 enum { @@ -1263,9 +1285,7 @@ bool mhp_supports_memmap_on_memory(unsigned long size) * altmap as an alternative source of memory, and we do not exactly * populate a single PMD. */ - return memmap_on_memory && - !hugetlb_optimize_vmemmap_enabled() && - IS_ENABLED(CONFIG_MHP_MEMMAP_ON_MEMORY) && + return mhp_memmap_on_memory() && size =3D=3D memory_block_size_bytes() && IS_ALIGNED(vmemmap_size, PMD_SIZE) && IS_ALIGNED(remaining_size, (pageblock_nr_pages << PAGE_SHIFT)); @@ -2083,7 +2103,7 @@ static int __ref try_remove_memory(u64 start, u64 siz= e) * We only support removing memory added with MHP_MEMMAP_ON_MEMORY in * the same granularity it was added - a single memory block. */ - if (memmap_on_memory) { + if (mhp_memmap_on_memory()) { nr_vmemmap_pages =3D walk_memory_blocks(start, size, NULL, get_nr_vmemmap_pages_cb); if (nr_vmemmap_pages) { --=20 2.11.0 From nobody Fri May 8 06:50:14 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id E4303C46467 for ; Mon, 9 May 2022 06:38:43 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235903AbiEIGk5 (ORCPT ); Mon, 9 May 2022 02:40:57 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42044 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234679AbiEIGdm (ORCPT ); Mon, 9 May 2022 02:33:42 -0400 Received: from mail-pj1-x1032.google.com (mail-pj1-x1032.google.com [IPv6:2607:f8b0:4864:20::1032]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 74DED18A6BB for ; Sun, 8 May 2022 23:29:49 -0700 (PDT) Received: by mail-pj1-x1032.google.com with SMTP id z5-20020a17090a468500b001d2bc2743c4so11967133pjf.0 for ; Sun, 08 May 2022 23:29:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20210112.gappssmtp.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=NyzkOphQ+s3WVfI9qZfbl6Rk93jtvh1L9Nek6U2DnhA=; b=MAi1RD94OdpjwH1pi8YbdqfTnkUKhdIKyjN/91Iiop2H2NF4JDF3N5PanHUWjP88F1 fUcAqUykTqgJo80i3V/4pawA8SeixCYIwVsOt+YS97K7cBVFe6m8Bt6eovF7+rmv3Sh1 I70dZyLpAuCmziwAXdrz8QrbyLP79Fb7laHzZ8WE3bN2ooDGxsBQ4XCvj1YAVJjcTVz1 sZWHPm1Treje0k0lQSyJ5RKtLLtaTs+wEKfjknWRbarWvd0zVZljThZWlCdPDV+7j4mQ GaBv2VEhB6F6AdWxIM3Vqe2XkbD8Ias/esGjM9ITPdVc2SZFTXoFrmeSAQvLe8X1mSd6 yX8g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=NyzkOphQ+s3WVfI9qZfbl6Rk93jtvh1L9Nek6U2DnhA=; b=lrQ+8Sr0W/YzAfwNUevV444wAtYhM+/GG9+aU22njL0U/eMOHqjpTOyDtkcYwBmWws fwD86i/XoYGg/Xb1vzUrhVAbe/y359pOIs+ZBQzorx/B0pTh3mTUhL/ZbJRFa6dJ6gOQ dRBK6BhsbGbcdN+YHMRaaONpNfrIIx12faDQ4unUBLd7lVCvj9DbIX+Y/qtHX5c0LPkX i+bI2taobAlK0i4AOAJvAuEycUJSzt2lfxTyd39zR/a0RbhO/LYhhe2ciefRFwMtJHwD cTAdN3cn9t3Q3C8JCkIKwc7hgU/pQkNFVlJI9McLsk5Rf1jxTjO20cp5wdm+SGg1FR1p 4v0g== X-Gm-Message-State: AOAM531LGK/N0bgKAbcNfdk2ocrKw85KdRZ3MPvk8tfMoI9s4kaw/MsS MJRvE0u9tXjt+MyOb+0v4ijctbfj42cb7g== X-Google-Smtp-Source: ABdhPJyUeNp5lyUBRQZydy9WfLbxkuPhlCBdpMAI02q3YSVfY3kMmne5Gx42MkLs4yh2Hehx52YfiA== X-Received: by 2002:a17:903:31d1:b0:159:804:e852 with SMTP id v17-20020a17090331d100b001590804e852mr15083698ple.19.1652077773702; Sun, 08 May 2022 23:29:33 -0700 (PDT) Received: from FVFYT0MHHV2J.bytedance.net ([139.177.225.250]) by smtp.gmail.com with ESMTPSA id k17-20020a170902ba9100b0015e8d4eb2c2sm6162001pls.268.2022.05.08.23.29.28 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 08 May 2022 23:29:33 -0700 (PDT) From: Muchun Song To: corbet@lwn.net, mike.kravetz@oracle.com, akpm@linux-foundation.org, mcgrof@kernel.org, keescook@chromium.org, yzaikin@google.com, osalvador@suse.de, david@redhat.com, masahiroy@kernel.org Cc: linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, duanxiongchun@bytedance.com, smuchun@gmail.com, Muchun Song Subject: [PATCH v10 3/4] mm: hugetlb_vmemmap: use kstrtobool for hugetlb_vmemmap param parsing Date: Mon, 9 May 2022 14:27:02 +0800 Message-Id: <20220509062703.64249-4-songmuchun@bytedance.com> X-Mailer: git-send-email 2.32.0 (Apple Git-132) In-Reply-To: <20220509062703.64249-1-songmuchun@bytedance.com> References: <20220509062703.64249-1-songmuchun@bytedance.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Use kstrtobool rather than open coding "on" and "off" parsing in mm/hugetlb_vmemmap.c, which is more powerful to handle all kinds of parameters like 'Yy1Nn0' or [oO][NnFf] for "on" and "off". Signed-off-by: Muchun Song Reviewed-by: Mike Kravetz Acked-by: David Hildenbrand --- Documentation/admin-guide/kernel-parameters.txt | 6 +++--- mm/hugetlb_vmemmap.c | 10 +++++----- 2 files changed, 8 insertions(+), 8 deletions(-) diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentatio= n/admin-guide/kernel-parameters.txt index 308da668bbb1..43b8385073ad 100644 --- a/Documentation/admin-guide/kernel-parameters.txt +++ b/Documentation/admin-guide/kernel-parameters.txt @@ -1703,10 +1703,10 @@ enabled. Allows heavy hugetlb users to free up some more memory (7 * PAGE_SIZE for each 2MB hugetlb page). - Format: { on | off (default) } + Format: { [oO][Nn]/Y/y/1 | [oO][Ff]/N/n/0 (default) } =20 - on: enable the feature - off: disable the feature + [oO][Nn]/Y/y/1: enable the feature + [oO][Ff]/N/n/0: disable the feature =20 Built with CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP_DEFAULT_ON=3Dy, the default is on. diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c index 6254bb2d4ae5..cc4ec752ec16 100644 --- a/mm/hugetlb_vmemmap.c +++ b/mm/hugetlb_vmemmap.c @@ -28,15 +28,15 @@ EXPORT_SYMBOL(hugetlb_optimize_vmemmap_key); =20 static int __init hugetlb_vmemmap_early_param(char *buf) { - if (!buf) + bool enable; + + if (kstrtobool(buf, &enable)) return -EINVAL; =20 - if (!strcmp(buf, "on")) + if (enable) static_branch_enable(&hugetlb_optimize_vmemmap_key); - else if (!strcmp(buf, "off")) - static_branch_disable(&hugetlb_optimize_vmemmap_key); else - return -EINVAL; + static_branch_disable(&hugetlb_optimize_vmemmap_key); =20 return 0; } --=20 2.11.0 From nobody Fri May 8 06:50:14 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 40270C4332F for ; Mon, 9 May 2022 06:38:43 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235446AbiEIGkR (ORCPT ); Mon, 9 May 2022 02:40:17 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42226 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234729AbiEIGdp (ORCPT ); Mon, 9 May 2022 02:33:45 -0400 Received: from mail-pj1-x102f.google.com (mail-pj1-x102f.google.com [IPv6:2607:f8b0:4864:20::102f]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id BB20918A6BF for ; Sun, 8 May 2022 23:29:51 -0700 (PDT) Received: by mail-pj1-x102f.google.com with SMTP id x88so714718pjj.1 for ; Sun, 08 May 2022 23:29:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20210112.gappssmtp.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=3vUx5XYnNUQ4SpbVJ4KR5YqsE/7i6GdjQvDrH5RKigc=; b=TC82INHLWZrU49CJ9vAbOyzbFZYnVPtZhj4HVMh0/PQ11kYlduRnmVkgVpkkAmIUnW pp840rF3esS1AZXEMuittuqcKYnRssv1NCAHX+11MQu2rVZIYkFqwWTuqOqgiwjCNnBB gdieRlCPhZ+IRj3xCyGAW1f3VK5ARJswXCwe76rmm0+I4sLIkCfkOQfdIDsxEfT4U7Dk hACf8ULMirNyXDDJb1zw8YIoWhcmMXEMNjJxn3cd3XDd3M7dlOUFpz3jpcz1xnuXtzcA kiAErI82e+jpbRnHXUxTgoNhziz8XhCLewLt0JtkWY8bFQn2IRNUTQr9/ZbaaK4lnTKd +a2w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=3vUx5XYnNUQ4SpbVJ4KR5YqsE/7i6GdjQvDrH5RKigc=; b=JAV0CVg5JYEsRJZzV+wpsMgLZRKjnFQVcQ2nS2Tp6i06Lu2urq2vpPonfA5uJE8CP4 0wAmNam/hoTgRNkP6cGFl6oe7B9CcMAuOSaeHPnzbfm2Bh8TULoL9iL0V1RCBu5Xo9gt 7YcX7B8hhBr82QwRsYRXv7hRrUW1/qYCzQFan0YbOJW/0RoYId7MGeDRQ6rdzjVkZ+C3 2wU476H61uXxHQGQPG7FgQWjDGCvTq4y1prALzLxcMtvM37th7SMxP9tL/t98KSOcvyq oz51NjR7bXeWjRuBj0UzeCeEJf4xMuAf/M3lCnbQSBuoXzW8SPNTE7BOm8toQWWXpRZ+ xz6Q== X-Gm-Message-State: AOAM5311c+2qFK/bVO1ZXP+cDHrLSbwoVyGkpnyxm/Pt+kxHyOIy0iEr AVGIjv9Oia6fjhJx9w/H1C5/0w== X-Google-Smtp-Source: ABdhPJysbt69R8xvl2WHk4ESD/nLQCAsz2esg98Qalvazq3o1Jn35yLaPmDEGrRWUUQboLjeHbYzwg== X-Received: by 2002:a17:903:1210:b0:15e:8373:d4b8 with SMTP id l16-20020a170903121000b0015e8373d4b8mr14867375plh.11.1652077779569; Sun, 08 May 2022 23:29:39 -0700 (PDT) Received: from FVFYT0MHHV2J.bytedance.net ([139.177.225.250]) by smtp.gmail.com with ESMTPSA id k17-20020a170902ba9100b0015e8d4eb2c2sm6162001pls.268.2022.05.08.23.29.34 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 08 May 2022 23:29:39 -0700 (PDT) From: Muchun Song To: corbet@lwn.net, mike.kravetz@oracle.com, akpm@linux-foundation.org, mcgrof@kernel.org, keescook@chromium.org, yzaikin@google.com, osalvador@suse.de, david@redhat.com, masahiroy@kernel.org Cc: linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, duanxiongchun@bytedance.com, smuchun@gmail.com, Muchun Song Subject: [PATCH v10 4/4] mm: hugetlb_vmemmap: add hugetlb_optimize_vmemmap sysctl Date: Mon, 9 May 2022 14:27:03 +0800 Message-Id: <20220509062703.64249-5-songmuchun@bytedance.com> X-Mailer: git-send-email 2.32.0 (Apple Git-132) In-Reply-To: <20220509062703.64249-1-songmuchun@bytedance.com> References: <20220509062703.64249-1-songmuchun@bytedance.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" We must add hugetlb_free_vmemmap=3Don (or "off") to the boot cmdline and reboot the server to enable or disable the feature of optimizing vmemmap pages associated with HugeTLB pages. However, rebooting usually takes a long time. So add a sysctl to enable or disable the feature at runtime without rebooting. Why we need this? There are 3 use cases. 1) The feature of minimizing overhead of struct page associated with each HugeTLB is disabled by default without passing "hugetlb_free_vmemmap=3Don" to the boot cmdline. When we (ByteDance) deliver the servers to the users who want to enable this feature, they have to configure the grub (change boot cmdline) and reboot the servers, whereas rebooting usually takes a long time (we have thousands of servers). It's a very bad experience for the users. So we need a approach to enable this feature after rebooting. This is a use case in our practical environment. 2) Some use cases are that HugeTLB pages are allocated 'on the fly' instead of being pulled from the HugeTLB pool, those workloads would be affected with this feature enabled. Those workloads could be identified by the characteristics of they never explicitly allocating huge pages with 'nr_hugepages' but only set 'nr_overcommit_hugepages' and then let the pages be allocated from the buddy allocator at fault time. We can confirm it is a real use case from the commit 099730d67417. For those workloads, the page fault time could be ~2x slower than before. We suspect those users want to disable this feature if the system has enabled this before and they don't think the memory savings benefit is enough to make up for the performance drop. 3) If the workload which wants vmemmap pages to be optimized and the workload which wants to set 'nr_overcommit_hugepages' and does not want the extera overhead at fault time when the overcommitted pages be allocated from the buddy allocator are deployed in the same server. The user could enable this feature and set 'nr_hugepages' and 'nr_overcommit_hugepages', then disable the feature. In this case, the overcommited HugeTLB pages will not encounter the extra overhead at fault time. Signed-off-by: Muchun Song Reviewed-by: Mike Kravetz --- Documentation/admin-guide/sysctl/vm.rst | 39 ++++++++++++++ include/linux/memory_hotplug.h | 9 ++++ mm/hugetlb.c | 3 ++ mm/hugetlb_vmemmap.c | 93 +++++++++++++++++++++++++++++= ---- mm/memory_hotplug.c | 7 +-- 5 files changed, 136 insertions(+), 15 deletions(-) diff --git a/Documentation/admin-guide/sysctl/vm.rst b/Documentation/admin-= guide/sysctl/vm.rst index 747e325ebcd0..5c9aa171a0d3 100644 --- a/Documentation/admin-guide/sysctl/vm.rst +++ b/Documentation/admin-guide/sysctl/vm.rst @@ -562,6 +562,45 @@ Change the minimum size of the hugepage pool. See Documentation/admin-guide/mm/hugetlbpage.rst =20 =20 +hugetlb_optimize_vmemmap +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + +This knob is not available when memory_hotplug.memmap_on_memory (kernel pa= rameter) +is configured or the size of 'struct page' (a structure defined in +include/linux/mm_types.h) is not power of two (an unusual system config co= uld +result in this). + +Enable (set to 1) or disable (set to 0) the feature of optimizing vmemmap = pages +associated with each HugeTLB page. + +Once enabled, the vmemmap pages of subsequent allocation of HugeTLB pages = from +buddy allocator will be optimized (7 pages per 2MB HugeTLB page and 4095 p= ages +per 1GB HugeTLB page), whereas already allocated HugeTLB pages will not be +optimized. When those optimized HugeTLB pages are freed from the HugeTLB = pool +to the buddy allocator, the vmemmap pages representing that range needs to= be +remapped again and the vmemmap pages discarded earlier need to be rellocat= ed +again. If your use case is that HugeTLB pages are allocated 'on the fly' = (e.g. +never explicitly allocating HugeTLB pages with 'nr_hugepages' but only set +'nr_overcommit_hugepages', those overcommitted HugeTLB pages are allocated= 'on +the fly') instead of being pulled from the HugeTLB pool, you should weigh = the +benefits of memory savings against the more overhead (~2x slower than befo= re) +of allocation or freeing HugeTLB pages between the HugeTLB pool and the bu= ddy +allocator. Another behavior to note is that if the system is under heavy = memory +pressure, it could prevent the user from freeing HugeTLB pages from the Hu= geTLB +pool to the buddy allocator since the allocation of vmemmap pages could be +failed, you have to retry later if your system encounter this situation. + +Once disabled, the vmemmap pages of subsequent allocation of HugeTLB pages= from +buddy allocator will not be optimized meaning the extra overhead at alloca= tion +time from buddy allocator disappears, whereas already optimized HugeTLB pa= ges +will not be affected. If you want to make sure there are no optimized Hug= eTLB +pages, you can set "nr_hugepages" to 0 first and then disable this. Note = that +writing 0 to nr_hugepages will make any "in use" HugeTLB pages become surp= lus +pages. So, those surplus pages are still optimized until they are no long= er +in use. You would need to wait for those surplus pages to be released bef= ore +there are no optimized pages in the system. + + nr_hugepages_mempolicy =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D =20 diff --git a/include/linux/memory_hotplug.h b/include/linux/memory_hotplug.h index 029fb7e26504..917112661b5c 100644 --- a/include/linux/memory_hotplug.h +++ b/include/linux/memory_hotplug.h @@ -351,4 +351,13 @@ void arch_remove_linear_mapping(u64 start, u64 size); extern bool mhp_supports_memmap_on_memory(unsigned long size); #endif /* CONFIG_MEMORY_HOTPLUG */ =20 +#ifdef CONFIG_MHP_MEMMAP_ON_MEMORY +bool mhp_memmap_on_memory(void); +#else +static inline bool mhp_memmap_on_memory(void) +{ + return false; +} +#endif + #endif /* __LINUX_MEMORY_HOTPLUG_H */ diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 8605d7eb7f5c..86158eb9da70 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -1617,6 +1617,9 @@ static DECLARE_WORK(free_hpage_work, free_hpage_workf= n); =20 static inline void flush_free_hpage_work(struct hstate *h) { + if (!hugetlb_optimize_vmemmap_enabled()) + return; + if (hugetlb_optimize_vmemmap_pages(h)) flush_work(&free_hpage_work); } diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c index cc4ec752ec16..fcd9f7872064 100644 --- a/mm/hugetlb_vmemmap.c +++ b/mm/hugetlb_vmemmap.c @@ -10,6 +10,7 @@ */ #define pr_fmt(fmt) "HugeTLB: " fmt =20 +#include #include "hugetlb_vmemmap.h" =20 /* @@ -22,21 +23,40 @@ #define RESERVE_VMEMMAP_NR 1U #define RESERVE_VMEMMAP_SIZE (RESERVE_VMEMMAP_NR << PAGE_SHIFT) =20 +enum vmemmap_optimize_mode { + VMEMMAP_OPTIMIZE_OFF, + VMEMMAP_OPTIMIZE_ON, +}; + DEFINE_STATIC_KEY_MAYBE(CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP_DEFAULT_ON, hugetlb_optimize_vmemmap_key); EXPORT_SYMBOL(hugetlb_optimize_vmemmap_key); =20 +static enum vmemmap_optimize_mode vmemmap_optimize_mode =3D + IS_ENABLED(CONFIG_HUGETLB_PAGE_FREE_VMEMMAP_DEFAULT_ON); + +static void vmemmap_optimize_mode_switch(enum vmemmap_optimize_mode to) +{ + if (vmemmap_optimize_mode =3D=3D to) + return; + + if (to =3D=3D VMEMMAP_OPTIMIZE_OFF) + static_branch_dec(&hugetlb_optimize_vmemmap_key); + else + static_branch_inc(&hugetlb_optimize_vmemmap_key); + WRITE_ONCE(vmemmap_optimize_mode, to); +} + static int __init hugetlb_vmemmap_early_param(char *buf) { bool enable; + enum vmemmap_optimize_mode mode; =20 if (kstrtobool(buf, &enable)) return -EINVAL; =20 - if (enable) - static_branch_enable(&hugetlb_optimize_vmemmap_key); - else - static_branch_disable(&hugetlb_optimize_vmemmap_key); + mode =3D enable ? VMEMMAP_OPTIMIZE_ON : VMEMMAP_OPTIMIZE_OFF; + vmemmap_optimize_mode_switch(mode); =20 return 0; } @@ -69,8 +89,10 @@ int hugetlb_vmemmap_alloc(struct hstate *h, struct page = *head) */ ret =3D vmemmap_remap_alloc(vmemmap_addr, vmemmap_end, vmemmap_reuse, GFP_KERNEL | __GFP_NORETRY | __GFP_THISNODE); - if (!ret) + if (!ret) { ClearHPageVmemmapOptimized(head); + static_branch_dec(&hugetlb_optimize_vmemmap_key); + } =20 return ret; } @@ -84,6 +106,11 @@ void hugetlb_vmemmap_free(struct hstate *h, struct page= *head) if (!vmemmap_pages) return; =20 + if (READ_ONCE(vmemmap_optimize_mode) =3D=3D VMEMMAP_OPTIMIZE_OFF) + return; + + static_branch_inc(&hugetlb_optimize_vmemmap_key); + vmemmap_addr +=3D RESERVE_VMEMMAP_SIZE; vmemmap_end =3D vmemmap_addr + (vmemmap_pages << PAGE_SHIFT); vmemmap_reuse =3D vmemmap_addr - PAGE_SIZE; @@ -93,7 +120,9 @@ void hugetlb_vmemmap_free(struct hstate *h, struct page = *head) * to the page which @vmemmap_reuse is mapped to, then free the pages * which the range [@vmemmap_addr, @vmemmap_end] is mapped to. */ - if (!vmemmap_remap_free(vmemmap_addr, vmemmap_end, vmemmap_reuse)) + if (vmemmap_remap_free(vmemmap_addr, vmemmap_end, vmemmap_reuse)) + static_branch_dec(&hugetlb_optimize_vmemmap_key); + else SetHPageVmemmapOptimized(head); } =20 @@ -110,9 +139,6 @@ void __init hugetlb_vmemmap_init(struct hstate *h) BUILD_BUG_ON(__NR_USED_SUBPAGE >=3D RESERVE_VMEMMAP_SIZE / sizeof(struct page)); =20 - if (!hugetlb_optimize_vmemmap_enabled()) - return; - if (!is_power_of_2(sizeof(struct page))) { pr_warn_once("cannot optimize vmemmap pages because \"struct page\" cros= ses page boundaries\n"); static_branch_disable(&hugetlb_optimize_vmemmap_key); @@ -134,3 +160,52 @@ void __init hugetlb_vmemmap_init(struct hstate *h) pr_info("can optimize %d vmemmap pages for %s\n", h->optimize_vmemmap_pages, h->name); } + +#ifdef CONFIG_PROC_SYSCTL +static int hugetlb_optimize_vmemmap_handler(struct ctl_table *table, int w= rite, + void *buffer, size_t *length, + loff_t *ppos) +{ + int ret; + enum vmemmap_optimize_mode mode; + static DEFINE_MUTEX(sysctl_mutex); + + if (write && !capable(CAP_SYS_ADMIN)) + return -EPERM; + + mutex_lock(&sysctl_mutex); + mode =3D vmemmap_optimize_mode; + table->data =3D &mode; + ret =3D proc_dointvec_minmax(table, write, buffer, length, ppos); + if (write && !ret) + vmemmap_optimize_mode_switch(mode); + mutex_unlock(&sysctl_mutex); + + return ret; +} + +static struct ctl_table hugetlb_vmemmap_sysctls[] =3D { + { + .procname =3D "hugetlb_optimize_vmemmap", + .maxlen =3D sizeof(enum vmemmap_optimize_mode), + .mode =3D 0644, + .proc_handler =3D hugetlb_optimize_vmemmap_handler, + .extra1 =3D SYSCTL_ZERO, + .extra2 =3D SYSCTL_ONE, + }, + { } +}; + +static __init int hugetlb_vmemmap_sysctls_init(void) +{ + /* + * If "memory_hotplug.memmap_on_memory" is enabled or "struct page" + * crosses page boundaries, the vmemmap pages cannot be optimized. + */ + if (!mhp_memmap_on_memory() && is_power_of_2(sizeof(struct page))) + register_sysctl_init("vm", hugetlb_vmemmap_sysctls); + + return 0; +} +late_initcall(hugetlb_vmemmap_sysctls_init); +#endif /* CONFIG_PROC_SYSCTL */ diff --git a/mm/memory_hotplug.c b/mm/memory_hotplug.c index a6101ae402f9..c72070cdd055 100644 --- a/mm/memory_hotplug.c +++ b/mm/memory_hotplug.c @@ -63,15 +63,10 @@ static bool memmap_on_memory __ro_after_init; module_param_cb(memmap_on_memory, &memmap_on_memory_ops, &memmap_on_memory= , 0444); MODULE_PARM_DESC(memmap_on_memory, "Enable memmap on memory for memory hot= plug"); =20 -static inline bool mhp_memmap_on_memory(void) +bool mhp_memmap_on_memory(void) { return memmap_on_memory; } -#else -static inline bool mhp_memmap_on_memory(void) -{ - return false; -} #endif =20 enum { --=20 2.11.0