From nobody Tue Dec 16 03:00:38 2025 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 67F10C54EE9 for ; Tue, 13 Sep 2022 19:58:11 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229535AbiIMT6G (ORCPT ); Tue, 13 Sep 2022 15:58:06 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46164 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229732AbiIMT6A (ORCPT ); Tue, 13 Sep 2022 15:58:00 -0400 Received: from mail-qt1-x82a.google.com (mail-qt1-x82a.google.com [IPv6:2607:f8b0:4864:20::82a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E578672EF8; Tue, 13 Sep 2022 12:57:58 -0700 (PDT) Received: by mail-qt1-x82a.google.com with SMTP id r20so8904690qtn.12; Tue, 13 Sep 2022 12:57:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date; bh=rWqwJImtlBlKgE2e4HCMCjZ57hDnUIYZvYnGoYO3Ko0=; b=d2+jGtD0TKLMdHImIPIOFRc/CCsLBd1zTd7Z32acL/4ee7Q///DRfLAeWKVby8PAWE fgb6z8hXf1k5PM+mERKCcjYlhJUc7YcVmkzbKybhsAlNUjEqr+U7/BKoMqeCMB/HNIHI g9CIZx3cpM7r7QtrN3niww/63rwDZ4ABYIOm7CcpFGEgc/xw03gOtoM8VQgFq7CdtHOy PnM3xHgjGzGq4Cl6QAxH02OPTmRMVzFrDgXO8l6Rjf0jEf15sxaNGCsaRPE9K0g1rGX0 Q9q2x8LOpIavofodITDiPKTwnIXo7FGcbxZ+dhF4xLlkpGof9K3nuM7AOF386w+Nu9xd TF4A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date; bh=rWqwJImtlBlKgE2e4HCMCjZ57hDnUIYZvYnGoYO3Ko0=; b=htp07O+LUzlLcuRDBo/jLs0Qna4JbMm1SQP1g5z6O3NJVGsbQRLGt/FroqyvjADLwz LEMGzTv6UD9Tos1We7dGLItHAVse6cwnI4Ire47DJpRmvCcxQEh9CKd1xPIW1kZ9p3fE KUnenjT3BnrC9g32o4+NcyjVTrZwGRK3uR9PcCnI98eg8+7loYIdZ2q0hzrZB+GaV5VL aYEJ96G6ubneVTjKtjzKAuahTwZZ6gzVTUOJqTQ/uDEnLNMvHIqU4/rVUk5ROOj9KwAa e5oGpTGDJrmJnijMsHOymLBVMRPrQuNV1RA1FAor6jUg983Pxs6iT+vpyXQfOV+n0D1s atUg== X-Gm-Message-State: ACgBeo3twSvFvU9kaYE+n/RZ7xkEDk+fcfFmuP2i1U2V6Nxumu+VUctb GXQmW0fP6MXVoDMkZFYpeQo= X-Google-Smtp-Source: AA6agR5i53agCunM1nP5BNWdgMMXAxgKBBgHP+YifFsQRT8ONLZddir0hPCAFH311ymBKWli946Oxg== X-Received: by 2002:a05:622a:40e:b0:343:7769:5895 with SMTP id n14-20020a05622a040e00b0034377695895mr30562094qtx.467.1663099078015; Tue, 13 Sep 2022 12:57:58 -0700 (PDT) Received: from stbirv-lnx-3.igp.broadcom.net ([192.19.223.252]) by smtp.gmail.com with ESMTPSA id s11-20020a05620a29cb00b006b8e049cf08sm276305qkp.2.2022.09.13.12.57.54 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 13 Sep 2022 12:57:57 -0700 (PDT) From: Doug Berger To: Andrew Morton Cc: Jonathan Corbet , Rob Herring , Krzysztof Kozlowski , Frank Rowand , Mike Kravetz , Muchun Song , Mike Rapoport , Christoph Hellwig , Marek Szyprowski , Robin Murphy , Borislav Petkov , "Paul E. McKenney" , Neeraj Upadhyay , Randy Dunlap , Damien Le Moal , Doug Berger , Florian Fainelli , David Hildenbrand , Zi Yan , Oscar Salvador , Hari Bathini , Kees Cook , - , KOSAKI Motohiro , Mel Gorman , linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, devicetree@vger.kernel.org, linux-mm@kvack.org, iommu@lists.linux.dev Subject: [PATCH 01/21] mm/page_isolation: protect cma from isolate_single_pageblock Date: Tue, 13 Sep 2022 12:54:48 -0700 Message-Id: <20220913195508.3511038-2-opendmb@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220913195508.3511038-1-opendmb@gmail.com> References: <20220913195508.3511038-1-opendmb@gmail.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" The function set_migratetype_isolate() has special handling for pageblocks of MIGRATE_CMA type that protects them from being isolated for MIGRATE_MOVABLE requests. Since isolate_single_pageblock() doesn't receive the migratetype argument of start_isolate_page_range() it used the migratetype of the pageblock instead of the requested migratetype which defeats this MIGRATE_CMA check. This allows an attempt to create a gigantic page within a CMA region to change the migratetype of the first and last pageblocks from MIGRATE_CMA to MIGRATE_MOVABLE when they are restored after failure, which corrupts the CMA region. The calls to (un)set_migratetype_isolate() for the first and last pageblocks of the start_isolate_page_range() are moved back into that function to allow access to its migratetype argument and make it easier to see how all of the pageblocks in the range are isolated. Fixes: b2c9e2fbba32 ("mm: make alloc_contig_range work at pageblock granula= rity") Signed-off-by: Doug Berger Reported-by: kernel test robot --- mm/page_isolation.c | 75 +++++++++++++++++++++------------------------ 1 file changed, 35 insertions(+), 40 deletions(-) diff --git a/mm/page_isolation.c b/mm/page_isolation.c index 9d73dc38e3d7..8e16aa22cb61 100644 --- a/mm/page_isolation.c +++ b/mm/page_isolation.c @@ -286,8 +286,6 @@ __first_valid_page(unsigned long pfn, unsigned long nr_= pages) * @flags: isolation flags * @gfp_flags: GFP flags used for migrating pages * @isolate_before: isolate the pageblock before the boundary_pfn - * @skip_isolation: the flag to skip the pageblock isolation in second - * isolate_single_pageblock() * * Free and in-use pages can be as big as MAX_ORDER-1 and contain more tha= n one * pageblock. When not all pageblocks within a page are isolated at the sa= me @@ -302,9 +300,8 @@ __first_valid_page(unsigned long pfn, unsigned long nr_= pages) * the in-use page then splitting the free page. */ static int isolate_single_pageblock(unsigned long boundary_pfn, int flags, - gfp_t gfp_flags, bool isolate_before, bool skip_isolation) + gfp_t gfp_flags, bool isolate_before) { - unsigned char saved_mt; unsigned long start_pfn; unsigned long isolate_pageblock; unsigned long pfn; @@ -328,18 +325,6 @@ static int isolate_single_pageblock(unsigned long boun= dary_pfn, int flags, start_pfn =3D max(ALIGN_DOWN(isolate_pageblock, MAX_ORDER_NR_PAGES), zone->zone_start_pfn); =20 - saved_mt =3D get_pageblock_migratetype(pfn_to_page(isolate_pageblock)); - - if (skip_isolation) - VM_BUG_ON(!is_migrate_isolate(saved_mt)); - else { - ret =3D set_migratetype_isolate(pfn_to_page(isolate_pageblock), saved_mt= , flags, - isolate_pageblock, isolate_pageblock + pageblock_nr_pages); - - if (ret) - return ret; - } - /* * Bail out early when the to-be-isolated pageblock does not form * a free or in-use page across boundary_pfn: @@ -428,7 +413,7 @@ static int isolate_single_pageblock(unsigned long bound= ary_pfn, int flags, ret =3D set_migratetype_isolate(page, page_mt, flags, head_pfn, head_pfn + nr_pages); if (ret) - goto failed; + return ret; } =20 ret =3D __alloc_contig_migrate_range(&cc, head_pfn, @@ -443,7 +428,7 @@ static int isolate_single_pageblock(unsigned long bound= ary_pfn, int flags, unset_migratetype_isolate(page, page_mt); =20 if (ret) - goto failed; + return -EBUSY; /* * reset pfn to the head of the free page, so * that the free page handling code above can split @@ -459,24 +444,19 @@ static int isolate_single_pageblock(unsigned long bou= ndary_pfn, int flags, while (!PageBuddy(pfn_to_page(outer_pfn))) { /* stop if we cannot find the free page */ if (++order >=3D MAX_ORDER) - goto failed; + return -EBUSY; outer_pfn &=3D ~0UL << order; } pfn =3D outer_pfn; continue; } else #endif - goto failed; + return -EBUSY; } =20 pfn++; } return 0; -failed: - /* restore the original migratetype */ - if (!skip_isolation) - unset_migratetype_isolate(pfn_to_page(isolate_pageblock), saved_mt); - return -EBUSY; } =20 /** @@ -534,21 +514,30 @@ int start_isolate_page_range(unsigned long start_pfn,= unsigned long end_pfn, unsigned long isolate_start =3D ALIGN_DOWN(start_pfn, pageblock_nr_pages); unsigned long isolate_end =3D ALIGN(end_pfn, pageblock_nr_pages); int ret; - bool skip_isolation =3D false; =20 /* isolate [isolate_start, isolate_start + pageblock_nr_pages) pageblock = */ - ret =3D isolate_single_pageblock(isolate_start, flags, gfp_flags, false, = skip_isolation); + ret =3D set_migratetype_isolate(pfn_to_page(isolate_start), migratetype, + flags, isolate_start, isolate_start + pageblock_nr_pages); if (ret) return ret; - - if (isolate_start =3D=3D isolate_end - pageblock_nr_pages) - skip_isolation =3D true; + ret =3D isolate_single_pageblock(isolate_start, flags, gfp_flags, false); + if (ret) + goto unset_start_block; =20 /* isolate [isolate_end - pageblock_nr_pages, isolate_end) pageblock */ - ret =3D isolate_single_pageblock(isolate_end, flags, gfp_flags, true, ski= p_isolation); + pfn =3D isolate_end - pageblock_nr_pages; + if (isolate_start !=3D pfn) { + ret =3D set_migratetype_isolate(pfn_to_page(pfn), migratetype, + flags, pfn, pfn + pageblock_nr_pages); + if (ret) + goto unset_start_block; + } + ret =3D isolate_single_pageblock(isolate_end, flags, gfp_flags, true); if (ret) { - unset_migratetype_isolate(pfn_to_page(isolate_start), migratetype); - return ret; + if (isolate_start !=3D pfn) + goto unset_end_block; + else + goto unset_start_block; } =20 /* skip isolated pageblocks at the beginning and end */ @@ -557,15 +546,21 @@ int start_isolate_page_range(unsigned long start_pfn,= unsigned long end_pfn, pfn +=3D pageblock_nr_pages) { page =3D __first_valid_page(pfn, pageblock_nr_pages); if (page && set_migratetype_isolate(page, migratetype, flags, - start_pfn, end_pfn)) { - undo_isolate_page_range(isolate_start, pfn, migratetype); - unset_migratetype_isolate( - pfn_to_page(isolate_end - pageblock_nr_pages), - migratetype); - return -EBUSY; - } + start_pfn, end_pfn)) + goto unset_isolated_blocks; } return 0; + +unset_isolated_blocks: + ret =3D -EBUSY; + undo_isolate_page_range(isolate_start + pageblock_nr_pages, pfn, + migratetype); +unset_end_block: + unset_migratetype_isolate(pfn_to_page(isolate_end - pageblock_nr_pages), + migratetype); +unset_start_block: + unset_migratetype_isolate(pfn_to_page(isolate_start), migratetype); + return ret; } =20 /* --=20 2.25.1 From nobody Tue Dec 16 03:00:38 2025 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 90AD4ECAAD8 for ; Tue, 13 Sep 2022 19:58:17 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229568AbiIMT6P (ORCPT ); Tue, 13 Sep 2022 15:58:15 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46272 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229764AbiIMT6D (ORCPT ); Tue, 13 Sep 2022 15:58:03 -0400 Received: from mail-qt1-x830.google.com (mail-qt1-x830.google.com [IPv6:2607:f8b0:4864:20::830]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8CE7472697; Tue, 13 Sep 2022 12:58:02 -0700 (PDT) Received: by mail-qt1-x830.google.com with SMTP id cb8so9637682qtb.0; Tue, 13 Sep 2022 12:58:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date; bh=Ae0x/cYtHCzUmIbnlOleUipnU3BBydliG8CbSfbHHfI=; b=lQy156yj9m33v2+lsCppN/1SSiL1+HSKUeGu4OKDvKzsWQP9uaxSZeRF6KYff4/ZX1 T8ar9K8mrC7poQnpcOlicqZBPrnyrX4lQmy/efXceAo8c/r+O7+kebNQPiRqxM6b8RHk nqDdqIjYqMl2UkmBaLyC3jNoxqdBf+LTtzsYTc4L2U/V/TdvsthKIhCb1FgggCffFcVL 4UCRPgfEkwehGvOaibbczRrkl3etaXkViPMT1ZepP3U6LDqZx2jP70X41REaFOYLplOf cOzr55DR2HMCT6kc/P98YRx0lPP2AD9Ny8hyMOuXORVZJrRjvmaKZrJZ1BXySGhbtpI9 HDig== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date; bh=Ae0x/cYtHCzUmIbnlOleUipnU3BBydliG8CbSfbHHfI=; b=DhK79UjZYFX6oR2ZfYRzZaRib+JuOiQw8zC14/4oqBwzjoVvBskxTnNUCUo+YU570S B5CZYKkrdBf5SwmIMYGyhv/n0nQVNWtcdtummZTCM0UGL0qkAs/xpdg/H4boH+pE0V9V yNnJE8yAHHm5NHtpw73ESZKl9QW0K8FTlUT77nRa8MmIZjyn7k80pomSnX/iXKsxf/jt eMcbYT9SF2INEc+LzvA4qoehtyBZ95C/UEdreXGb1AJ+P7CpAgZ8SaYBy7XlXjzhAbD1 b4/+ggHwwyaafLccNik9c5gSOMZXg42FzxbmAC0/kJXemAWbVjbqBVRP2pIP+UUYFtSk QSng== X-Gm-Message-State: ACgBeo2JCDymnw+Bopy8tAou9papj5jEbvfJOtWdQd8u6UDfWCnRq0uE HXtocqxXjAP+AHFJ9lJ0vIw= X-Google-Smtp-Source: AA6agR5RQ3Vz73r+e9pp4sPmIM/iMa/zHS6QVY1LvvDFqMY99XQUTUB/nmtdr1ce7F9gfpRjTDrD5Q== X-Received: by 2002:ac8:5790:0:b0:35b:fd92:6fb8 with SMTP id v16-20020ac85790000000b0035bfd926fb8mr4051143qta.53.1663099081690; Tue, 13 Sep 2022 12:58:01 -0700 (PDT) Received: from stbirv-lnx-3.igp.broadcom.net ([192.19.223.252]) by smtp.gmail.com with ESMTPSA id s11-20020a05620a29cb00b006b8e049cf08sm276305qkp.2.2022.09.13.12.57.58 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 13 Sep 2022 12:58:01 -0700 (PDT) From: Doug Berger To: Andrew Morton Cc: Jonathan Corbet , Rob Herring , Krzysztof Kozlowski , Frank Rowand , Mike Kravetz , Muchun Song , Mike Rapoport , Christoph Hellwig , Marek Szyprowski , Robin Murphy , Borislav Petkov , "Paul E. McKenney" , Neeraj Upadhyay , Randy Dunlap , Damien Le Moal , Doug Berger , Florian Fainelli , David Hildenbrand , Zi Yan , Oscar Salvador , Hari Bathini , Kees Cook , - , KOSAKI Motohiro , Mel Gorman , linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, devicetree@vger.kernel.org, linux-mm@kvack.org, iommu@lists.linux.dev Subject: [PATCH 02/21] mm/hugetlb: correct max_huge_pages accounting on demote Date: Tue, 13 Sep 2022 12:54:49 -0700 Message-Id: <20220913195508.3511038-3-opendmb@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220913195508.3511038-1-opendmb@gmail.com> References: <20220913195508.3511038-1-opendmb@gmail.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" When demoting a hugepage to a smaller order, the number of pages added to the target hstate will be the size of the large page divided by the size of the smaller page. Fixes: 8531fc6f52f5 ("hugetlb: add hugetlb demote page support") Signed-off-by: Doug Berger --- mm/hugetlb.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index e070b8593b37..79949893ac12 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -3472,7 +3472,8 @@ static int demote_free_huge_page(struct hstate *h, st= ruct page *page) * based on pool changes for the demoted page. */ h->max_huge_pages--; - target_hstate->max_huge_pages +=3D pages_per_huge_page(h); + target_hstate->max_huge_pages +=3D pages_per_huge_page(h) / + pages_per_huge_page(target_hstate); =20 return rc; } --=20 2.25.1 From nobody Tue Dec 16 03:00:38 2025 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 97A54C54EE9 for ; Tue, 13 Sep 2022 19:58:38 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229937AbiIMT6a (ORCPT ); Tue, 13 Sep 2022 15:58:30 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46740 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229723AbiIMT6Y (ORCPT ); Tue, 13 Sep 2022 15:58:24 -0400 Received: from mail-qv1-xf2e.google.com (mail-qv1-xf2e.google.com [IPv6:2607:f8b0:4864:20::f2e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 69A507392C; Tue, 13 Sep 2022 12:58:06 -0700 (PDT) Received: by mail-qv1-xf2e.google.com with SMTP id q8so10090370qvr.9; Tue, 13 Sep 2022 12:58:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date; bh=dgAC9wdSWdNnQd7c6lUo6ch6gbgfZ8Ik+m13B5omrFE=; b=pgU7qB1cfn3bbb6aqaPKVsP0QULztBXM6ZTdmlLajCShk8VBE78cxzG8Cv0iNse9pr raJ8GgLPcDyEkXzvMHViIvbgw2EPxCCCxmBWVydWXsbvHPXhV9LnTIQcxrhx+uvf4Xru 5WYEg1qQVSMXl/Ik2LXwDiiTB+uN5XnQ7jp32HYymQ2HZJZADmguB3tnteKHk9qIukRc jpn8LdBlaPcxTkVAmOKzkV+NLvIVifwROYLBZ5KLKak/eoCCJdA+KloOJgYdmZJeYMVG xPc5vPtISXEXXxdKiktdhoBVSWbwV6cguyuU1u/JwyBG7hVEPpa11URawNPjZH6cK9Db YImw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date; bh=dgAC9wdSWdNnQd7c6lUo6ch6gbgfZ8Ik+m13B5omrFE=; b=B6u2/MEiC4pV8psVGPVSD88rjHk/IaFxLFt9DSmwztJxmOBOQ29whM6/9WL5oPgNju Ytwri3PoF0ssUebbuJc8S0sZLZzFZRI/aZtyV9JK6V3VFUPFfg8YLHEvMd6du48OYrSk Ea+C66o6ypmPRi/fSi6gvKMo8moBkr9/qBgTXhD35luusRvDsjnWWGc29uVC4X8YKslK uC6wHIidWuQmowN79vPl4MAWCtJWG33aRyClCK8tVyisEKmqxBlcHfHcvL01Jo/rSxWc gL/N6P2yaDJ4VM8v9oDUdYg9IPVkzqZ0f04Gfic3TpRHdodT6gifAxAssYMzGsCPxqKN g9lg== X-Gm-Message-State: ACgBeo3yXKV81tjYZmKY4cxWDuvTmGVW/jcVwiycnZPmHVv4Yp85sQza iYVtOy1JxBKkbFeGu5TfnAE= X-Google-Smtp-Source: AA6agR5N0IDxlcIGSy6m2BrDrpoJ44hk3cGJOE0+/g5noyEETZjDxRdlg2yegkHVtpvIIdmASzximw== X-Received: by 2002:a0c:a901:0:b0:4aa:a283:ef4a with SMTP id y1-20020a0ca901000000b004aaa283ef4amr28465376qva.53.1663099085481; Tue, 13 Sep 2022 12:58:05 -0700 (PDT) Received: from stbirv-lnx-3.igp.broadcom.net ([192.19.223.252]) by smtp.gmail.com with ESMTPSA id s11-20020a05620a29cb00b006b8e049cf08sm276305qkp.2.2022.09.13.12.58.01 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 13 Sep 2022 12:58:04 -0700 (PDT) From: Doug Berger To: Andrew Morton Cc: Jonathan Corbet , Rob Herring , Krzysztof Kozlowski , Frank Rowand , Mike Kravetz , Muchun Song , Mike Rapoport , Christoph Hellwig , Marek Szyprowski , Robin Murphy , Borislav Petkov , "Paul E. McKenney" , Neeraj Upadhyay , Randy Dunlap , Damien Le Moal , Doug Berger , Florian Fainelli , David Hildenbrand , Zi Yan , Oscar Salvador , Hari Bathini , Kees Cook , - , KOSAKI Motohiro , Mel Gorman , linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, devicetree@vger.kernel.org, linux-mm@kvack.org, iommu@lists.linux.dev Subject: [PATCH 03/21] mm/hugetlb: correct demote page offset logic Date: Tue, 13 Sep 2022 12:54:50 -0700 Message-Id: <20220913195508.3511038-4-opendmb@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220913195508.3511038-1-opendmb@gmail.com> References: <20220913195508.3511038-1-opendmb@gmail.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" With gigantic pages it may not be true that struct page structures are contiguous across the entire gigantic page. The mem_map_offset function is used here in place of direct pointer arithmetic to correct for this. Fixes: 8531fc6f52f5 ("hugetlb: add hugetlb demote page support") Signed-off-by: Doug Berger Acked-by: Muchun Song --- mm/hugetlb.c | 14 ++++++++------ 1 file changed, 8 insertions(+), 6 deletions(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 79949893ac12..a1d51a1f0404 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -3420,6 +3420,7 @@ static int demote_free_huge_page(struct hstate *h, st= ruct page *page) { int i, nid =3D page_to_nid(page); struct hstate *target_hstate; + struct page *subpage; int rc =3D 0; =20 target_hstate =3D size_to_hstate(PAGE_SIZE << h->demote_order); @@ -3453,15 +3454,16 @@ static int demote_free_huge_page(struct hstate *h, = struct page *page) mutex_lock(&target_hstate->resize_lock); for (i =3D 0; i < pages_per_huge_page(h); i +=3D pages_per_huge_page(target_hstate)) { + subpage =3D mem_map_offset(page, i); if (hstate_is_gigantic(target_hstate)) - prep_compound_gigantic_page_for_demote(page + i, + prep_compound_gigantic_page_for_demote(subpage, target_hstate->order); else - prep_compound_page(page + i, target_hstate->order); - set_page_private(page + i, 0); - set_page_refcounted(page + i); - prep_new_huge_page(target_hstate, page + i, nid); - put_page(page + i); + prep_compound_page(subpage, target_hstate->order); + set_page_private(subpage, 0); + set_page_refcounted(subpage); + prep_new_huge_page(target_hstate, subpage, nid); + put_page(subpage); } mutex_unlock(&target_hstate->resize_lock); =20 --=20 2.25.1 From nobody Tue Dec 16 03:00:38 2025 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id BFFC0C6FA82 for ; Tue, 13 Sep 2022 19:58:45 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229863AbiIMT6m (ORCPT ); Tue, 13 Sep 2022 15:58:42 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46882 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229833AbiIMT6Z (ORCPT ); Tue, 13 Sep 2022 15:58:25 -0400 Received: from mail-qk1-x72c.google.com (mail-qk1-x72c.google.com [IPv6:2607:f8b0:4864:20::72c]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id DA5DA74DD4; Tue, 13 Sep 2022 12:58:10 -0700 (PDT) Received: by mail-qk1-x72c.google.com with SMTP id y2so6829968qkl.11; Tue, 13 Sep 2022 12:58:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date; bh=yHgeZ0+Ocv6GrFoaruOXhrYrRwLmKzzR+EPUoXoCFLk=; b=MO7TGhjFAweThFGVYteWrV+nJ7j5y3LZRa4C7ACnJ1qWyIG54wf5kg5kO3xvw/FqDQ s4wKsy9yFNOEBO191kejMvdr64z7Q9PD3aPBFXDJT69Mpap/Qm7BagM/tmOC2sbVk5Ek f646vwKxfSqOitl6VKXJ3CuYfAma7EN7IglgyNYbVTkgDe6A8Ejr2VK5Z6ft+DXrx+l4 fQRhw7+DxWu3uXxtthnmrlFhvdM2zwLPoqSspHMHQaTzl30h2Y5TuDS9T3a1oeXGiuia JD/Wvi1/kbP+I5YXUEfkPCFyR+Q73ACUQMRNqUffeivDuZhRpUvnXKzfC7RG3PK+ZTef LsyA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date; bh=yHgeZ0+Ocv6GrFoaruOXhrYrRwLmKzzR+EPUoXoCFLk=; b=cnBDhhQIFNXguJJ23rGROJmch75Eqsn/VfAHAHyhxlDTq1/ZCt/mJ+u8LtJSYNSG7Y /VSibe22cxCkCk+frsTgtFPD7sppKKsow7FCaXp8IPtfNqlXeCfCifr4YCmPkO49mVvF tr0CyYaf+WvAsS3TRIIH+r7XiQY/v5IAxxu7rzSgkD6Eup5Sd3ob4ChU1R6cC4N3WdAC qnWruRmMnEmAvfdNH6FnDhbHnxqJBokE+0ojoAVZrAAhBRpscRPhr0ZjlOljSCO/RlM+ r/XFbrjCZEp2W528aRhk2XdbfQwe6a/ibJ7nFw+H48p+eMRlZLUJjoBfHaamgNJPEsmQ WZaA== X-Gm-Message-State: ACgBeo0EEXE3Rb3FJ/9hVh3SOJkUEWDhK2YpLwU3Wiv2QvI7vFQ2NDBQ fzaW9yYfpVFKQ3Q4E1BD++A= X-Google-Smtp-Source: AA6agR5MEfLRKDjJcxpoJCjM2jEAW8H06DRSio04kdGXrEcQ2RvULscAsN8q5RMRJ4UP1SunQTdfRQ== X-Received: by 2002:a05:620a:488f:b0:6ce:634e:8963 with SMTP id ea15-20020a05620a488f00b006ce634e8963mr3462100qkb.16.1663099089149; Tue, 13 Sep 2022 12:58:09 -0700 (PDT) Received: from stbirv-lnx-3.igp.broadcom.net ([192.19.223.252]) by smtp.gmail.com with ESMTPSA id s11-20020a05620a29cb00b006b8e049cf08sm276305qkp.2.2022.09.13.12.58.05 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 13 Sep 2022 12:58:08 -0700 (PDT) From: Doug Berger To: Andrew Morton Cc: Jonathan Corbet , Rob Herring , Krzysztof Kozlowski , Frank Rowand , Mike Kravetz , Muchun Song , Mike Rapoport , Christoph Hellwig , Marek Szyprowski , Robin Murphy , Borislav Petkov , "Paul E. McKenney" , Neeraj Upadhyay , Randy Dunlap , Damien Le Moal , Doug Berger , Florian Fainelli , David Hildenbrand , Zi Yan , Oscar Salvador , Hari Bathini , Kees Cook , - , KOSAKI Motohiro , Mel Gorman , linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, devicetree@vger.kernel.org, linux-mm@kvack.org, iommu@lists.linux.dev Subject: [PATCH 04/21] mm/hugetlb: refactor alloc_and_dissolve_huge_page Date: Tue, 13 Sep 2022 12:54:51 -0700 Message-Id: <20220913195508.3511038-5-opendmb@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220913195508.3511038-1-opendmb@gmail.com> References: <20220913195508.3511038-1-opendmb@gmail.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" The alloc_replacement_page() and replace_hugepage() functions are created from code in the alloc_and_dissolve_huge_page() function to allow their reuse by the next commit. Signed-off-by: Doug Berger --- mm/hugetlb.c | 84 +++++++++++++++++++++++++++++++--------------------- 1 file changed, 51 insertions(+), 33 deletions(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index a1d51a1f0404..f232a37df4b6 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -2709,32 +2709,22 @@ void restore_reserve_on_error(struct hstate *h, str= uct vm_area_struct *vma, } =20 /* - * alloc_and_dissolve_huge_page - Allocate a new page and dissolve the old= one - * @h: struct hstate old page belongs to - * @old_page: Old page to dissolve - * @list: List to isolate the page in case we need to - * Returns 0 on success, otherwise negated error. + * Before dissolving the page, we need to allocate a new one for the + * pool to remain stable. Here, we allocate the page and 'prep' it + * by doing everything but actually updating counters and adding to + * the pool. This simplifies and let us do most of the processing + * under the lock. */ -static int alloc_and_dissolve_huge_page(struct hstate *h, struct page *old= _page, - struct list_head *list) +static struct page *alloc_replacement_page(struct hstate *h, int nid) { gfp_t gfp_mask =3D htlb_alloc_mask(h) | __GFP_THISNODE; - int nid =3D page_to_nid(old_page); bool alloc_retry =3D false; struct page *new_page; - int ret =3D 0; =20 - /* - * Before dissolving the page, we need to allocate a new one for the - * pool to remain stable. Here, we allocate the page and 'prep' it - * by doing everything but actually updating counters and adding to - * the pool. This simplifies and let us do most of the processing - * under the lock. - */ alloc_retry: new_page =3D alloc_buddy_huge_page(h, gfp_mask, nid, NULL, NULL); if (!new_page) - return -ENOMEM; + return ERR_PTR(-ENOMEM); /* * If all goes well, this page will be directly added to the free * list in the pool. For this the ref count needs to be zero. @@ -2748,7 +2738,7 @@ static int alloc_and_dissolve_huge_page(struct hstate= *h, struct page *old_page, SetHPageTemporary(new_page); if (!put_page_testzero(new_page)) { if (alloc_retry) - return -EBUSY; + return ERR_PTR(-EBUSY); =20 alloc_retry =3D true; goto alloc_retry; @@ -2757,6 +2747,48 @@ static int alloc_and_dissolve_huge_page(struct hstat= e *h, struct page *old_page, =20 __prep_new_huge_page(h, new_page); =20 + return new_page; +} + +static void replace_hugepage(struct hstate *h, int nid, struct page *old_p= age, + struct page *new_page) +{ + lockdep_assert_held(&hugetlb_lock); + /* + * Ok, old_page is still a genuine free hugepage. Remove it from + * the freelist and decrease the counters. These will be + * incremented again when calling __prep_account_new_huge_page() + * and enqueue_huge_page() for new_page. The counters will remain + * stable since this happens under the lock. + */ + remove_hugetlb_page(h, old_page, false); + + /* + * Ref count on new page is already zero as it was dropped + * earlier. It can be directly added to the pool free list. + */ + __prep_account_new_huge_page(h, nid); + enqueue_huge_page(h, new_page); +} + +/* + * alloc_and_dissolve_huge_page - Allocate a new page and dissolve the old= one + * @h: struct hstate old page belongs to + * @old_page: Old page to dissolve + * @list: List to isolate the page in case we need to + * Returns 0 on success, otherwise negated error. + */ +static int alloc_and_dissolve_huge_page(struct hstate *h, struct page *old= _page, + struct list_head *list) +{ + int nid =3D page_to_nid(old_page); + struct page *new_page; + int ret =3D 0; + + new_page =3D alloc_replacement_page(h, nid); + if (IS_ERR(new_page)) + return PTR_ERR(new_page); + retry: spin_lock_irq(&hugetlb_lock); if (!PageHuge(old_page)) { @@ -2783,21 +2815,7 @@ static int alloc_and_dissolve_huge_page(struct hstat= e *h, struct page *old_page, cond_resched(); goto retry; } else { - /* - * Ok, old_page is still a genuine free hugepage. Remove it from - * the freelist and decrease the counters. These will be - * incremented again when calling __prep_account_new_huge_page() - * and enqueue_huge_page() for new_page. The counters will remain - * stable since this happens under the lock. - */ - remove_hugetlb_page(h, old_page, false); - - /* - * Ref count on new page is already zero as it was dropped - * earlier. It can be directly added to the pool free list. - */ - __prep_account_new_huge_page(h, nid); - enqueue_huge_page(h, new_page); + replace_hugepage(h, nid, old_page, new_page); =20 /* * Pages have been replaced, we can safely free the old one. --=20 2.25.1 From nobody Tue Dec 16 03:00:38 2025 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id F0003C54EE9 for ; Tue, 13 Sep 2022 19:58:52 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229931AbiIMT6s (ORCPT ); Tue, 13 Sep 2022 15:58:48 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46832 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229854AbiIMT60 (ORCPT ); Tue, 13 Sep 2022 15:58:26 -0400 Received: from mail-qv1-xf2a.google.com (mail-qv1-xf2a.google.com [IPv6:2607:f8b0:4864:20::f2a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 65A9E7696F; Tue, 13 Sep 2022 12:58:14 -0700 (PDT) Received: by mail-qv1-xf2a.google.com with SMTP id g4so10091039qvo.3; Tue, 13 Sep 2022 12:58:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date; bh=ivekPqFwDTKnplQubqOcOnwSj5SwqgEqKMLIx8G0mTA=; b=kvUC28iyEDcMoby9xrnJHhocivoBZj9gqY+ui6eXHvBbq0tXS2dzYAAmqBdXCNT1wy Q0fTtOM1kwBq1neZJJDtv8OVb2E4WTXaookEyrn9UWfnw0gFlPMPSsLKdxzfJXEQRdps TBY/dxDfKDXG5nzytFERu/dU7OZ7FBxZvIzoh4dkW/QYs2YW41IfiKdkkwlFW8nnTbi2 RA0eP5SekM1KC2ny0+Tc+vzS7nujN6k/2bx9b8Rj/IE1ke6REKO17fe0+no3v3V+6DPQ hTcXXufvBklDdoPK3SevxPEu7k9onHuyK1D6iBkExepS0BnC5h2ewbJv8V7xNvAFJRmS S5FA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date; bh=ivekPqFwDTKnplQubqOcOnwSj5SwqgEqKMLIx8G0mTA=; b=2BjZVmk/glp/qQCTxmgbJKANL0YD5f0xJY7Lo8jW5wO9sZcBmsQolXQgmzXkY2oeCM XeAZ52Cat0Sr32rElTUfs8I0UBzfu4mjp9jeZl53hy/zLxPJrDKBrqkgIlf+XDPsxKm6 8UYIcf1gn7XbyJvvdSP/yck6op18DF2gc9oJy5WPP2Lienwv6Yuf2LknLCpaU9C8G34m EhSZQC0kVTub7OaaTngdqpdrGZicbLHNi23ZCg1gn/bg0XDVpFCJViPc033QXJgY0aQQ Y9w5XgzT6wh99Nis7MlrObfb9oDwWLGtDIRwBE/dooYpK+J4Tu/ILyyAC8e28881ZXWA mJAA== X-Gm-Message-State: ACgBeo3baPJEVT12L7e4mPTvKiOTxdRtvw4ZF2Ln+bxcS8g9VZ4AIOVO y/nghyGNUgCe0oHXMTVZ6c4= X-Google-Smtp-Source: AA6agR5Uplm7zgaGZn1afr+syf2bXEr/tRhasRhXystutC9gt9duBJzLy1nlXABisQhehw9c0OtbIg== X-Received: by 2002:a05:6214:1c09:b0:4ac:9160:7484 with SMTP id u9-20020a0562141c0900b004ac91607484mr17181443qvc.13.1663099092959; Tue, 13 Sep 2022 12:58:12 -0700 (PDT) Received: from stbirv-lnx-3.igp.broadcom.net ([192.19.223.252]) by smtp.gmail.com with ESMTPSA id s11-20020a05620a29cb00b006b8e049cf08sm276305qkp.2.2022.09.13.12.58.09 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 13 Sep 2022 12:58:12 -0700 (PDT) From: Doug Berger To: Andrew Morton Cc: Jonathan Corbet , Rob Herring , Krzysztof Kozlowski , Frank Rowand , Mike Kravetz , Muchun Song , Mike Rapoport , Christoph Hellwig , Marek Szyprowski , Robin Murphy , Borislav Petkov , "Paul E. McKenney" , Neeraj Upadhyay , Randy Dunlap , Damien Le Moal , Doug Berger , Florian Fainelli , David Hildenbrand , Zi Yan , Oscar Salvador , Hari Bathini , Kees Cook , - , KOSAKI Motohiro , Mel Gorman , linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, devicetree@vger.kernel.org, linux-mm@kvack.org, iommu@lists.linux.dev Subject: [PATCH 05/21] mm/hugetlb: allow migrated hugepage to dissolve when freed Date: Tue, 13 Sep 2022 12:54:52 -0700 Message-Id: <20220913195508.3511038-6-opendmb@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220913195508.3511038-1-opendmb@gmail.com> References: <20220913195508.3511038-1-opendmb@gmail.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" There is no isolation mechanism for hugepages so a hugepage that is migrated is returned to its hugepage freelist. This creates problems for alloc_contig_range() because migrated hugepages can be allocated as migrate targets for subsequent hugepage migration attempts. Even if the migration succeeds the alloc_contig_range() attempt will fail because test_pages_isolated() will find the now free hugepages haven't been dissolved. A subsequent attempt by alloc_contig_range() is necessary for the isolate_migratepages_range() function to find the freed hugepage and dissolve it (assuming it has not been reallocated). A workqueue is introduced to perform the equivalent functionality of alloc_and_dissolve_huge_page() for a migrated hugepage when it is freed so that the pages can be released to the isolated page lists of the buddy allocator allowing the alloc_contig_range() attempt to succeed. The HPG_dissolve hugepage flag is introduced to allow tagging migratable hugepages that should be dissolved when freed. Signed-off-by: Doug Berger --- include/linux/hugetlb.h | 5 +++ mm/hugetlb.c | 72 ++++++++++++++++++++++++++++++++++++++--- mm/migrate.c | 1 + mm/page_alloc.c | 1 + 4 files changed, 75 insertions(+), 4 deletions(-) diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h index 3ec981a0d8b3..0e6e21805e51 100644 --- a/include/linux/hugetlb.h +++ b/include/linux/hugetlb.h @@ -222,6 +222,7 @@ unsigned long hugetlb_change_protection(struct vm_area_= struct *vma, =20 bool is_hugetlb_entry_migration(pte_t pte); void hugetlb_unshare_all_pmds(struct vm_area_struct *vma); +void sync_hugetlb_dissolve(void); =20 #else /* !CONFIG_HUGETLB_PAGE */ =20 @@ -430,6 +431,8 @@ static inline vm_fault_t hugetlb_fault(struct mm_struct= *mm, =20 static inline void hugetlb_unshare_all_pmds(struct vm_area_struct *vma) { } =20 +static inline void sync_hugetlb_dissolve(void) { } + #endif /* !CONFIG_HUGETLB_PAGE */ /* * hugepages at page global directory. If arch support @@ -574,6 +577,7 @@ enum hugetlb_page_flags { HPG_freed, HPG_vmemmap_optimized, HPG_raw_hwp_unreliable, + HPG_dissolve, __NR_HPAGEFLAGS, }; =20 @@ -621,6 +625,7 @@ HPAGEFLAG(Temporary, temporary) HPAGEFLAG(Freed, freed) HPAGEFLAG(VmemmapOptimized, vmemmap_optimized) HPAGEFLAG(RawHwpUnreliable, raw_hwp_unreliable) +HPAGEFLAG(Dissolve, dissolve) =20 #ifdef CONFIG_HUGETLB_PAGE =20 diff --git a/mm/hugetlb.c b/mm/hugetlb.c index f232a37df4b6..da80889e1436 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -1582,6 +1582,10 @@ static void __update_and_free_page(struct hstate *h,= struct page *page) } } =20 +static LLIST_HEAD(hpage_dissolvelist); +static void dissolve_hpage_workfn(struct work_struct *work); +static DECLARE_WORK(dissolve_hpage_work, dissolve_hpage_workfn); + /* * As update_and_free_page() can be called under any context, so we cannot * use GFP_KERNEL to allocate vmemmap pages. However, we can defer the @@ -1628,6 +1632,8 @@ static inline void flush_free_hpage_work(struct hstat= e *h) { if (hugetlb_vmemmap_optimizable(h)) flush_work(&free_hpage_work); + if (!hstate_is_gigantic(h)) + flush_work(&dissolve_hpage_work); } =20 static void update_and_free_page(struct hstate *h, struct page *page, @@ -1679,7 +1685,7 @@ void free_huge_page(struct page *page) struct hstate *h =3D page_hstate(page); int nid =3D page_to_nid(page); struct hugepage_subpool *spool =3D hugetlb_page_subpool(page); - bool restore_reserve; + bool restore_reserve, dissolve; unsigned long flags; =20 VM_BUG_ON_PAGE(page_count(page), page); @@ -1691,6 +1697,8 @@ void free_huge_page(struct page *page) page->mapping =3D NULL; restore_reserve =3D HPageRestoreReserve(page); ClearHPageRestoreReserve(page); + dissolve =3D HPageDissolve(page); + ClearHPageDissolve(page); =20 /* * If HPageRestoreReserve was set on page, page allocation consumed a @@ -1729,6 +1737,11 @@ void free_huge_page(struct page *page) remove_hugetlb_page(h, page, true); spin_unlock_irqrestore(&hugetlb_lock, flags); update_and_free_page(h, page, true); + } else if (dissolve) { + spin_unlock_irqrestore(&hugetlb_lock, flags); + if (llist_add((struct llist_node *)&page->mapping, + &hpage_dissolvelist)) + schedule_work(&dissolve_hpage_work); } else { arch_clear_hugepage_flags(page); enqueue_huge_page(h, page); @@ -2771,6 +2784,49 @@ static void replace_hugepage(struct hstate *h, int n= id, struct page *old_page, enqueue_huge_page(h, new_page); } =20 +static void dissolve_hpage_workfn(struct work_struct *work) +{ + struct llist_node *node; + + node =3D llist_del_all(&hpage_dissolvelist); + + while (node) { + struct page *oldpage, *newpage; + struct hstate *h; + int nid; + + oldpage =3D container_of((struct address_space **)node, + struct page, mapping); + node =3D node->next; + oldpage->mapping =3D NULL; + + h =3D page_hstate(oldpage); + nid =3D page_to_nid(oldpage); + + newpage =3D alloc_replacement_page(h, nid); + + spin_lock_irq(&hugetlb_lock); + /* finish freeing oldpage */ + arch_clear_hugepage_flags(oldpage); + enqueue_huge_page(h, oldpage); + if (IS_ERR(newpage)) { + /* cannot dissolve so just leave free */ + spin_unlock_irq(&hugetlb_lock); + goto next; + } + + replace_hugepage(h, nid, oldpage, newpage); + + /* + * Pages have been replaced, we can safely free the old one. + */ + spin_unlock_irq(&hugetlb_lock); + __update_and_free_page(h, oldpage); +next: + cond_resched(); + } +} + /* * alloc_and_dissolve_huge_page - Allocate a new page and dissolve the old= one * @h: struct hstate old page belongs to @@ -2803,6 +2859,7 @@ static int alloc_and_dissolve_huge_page(struct hstate= *h, struct page *old_page, */ spin_unlock_irq(&hugetlb_lock); ret =3D isolate_hugetlb(old_page, list); + SetHPageDissolve(old_page); spin_lock_irq(&hugetlb_lock); goto free_new; } else if (!HPageFreed(old_page)) { @@ -2864,14 +2921,21 @@ int isolate_or_dissolve_huge_page(struct page *page= , struct list_head *list) if (hstate_is_gigantic(h)) return -ENOMEM; =20 - if (page_count(head) && !isolate_hugetlb(head, list)) + if (page_count(head) && !isolate_hugetlb(head, list)) { + SetHPageDissolve(head); ret =3D 0; - else if (!page_count(head)) + } else if (!page_count(head)) { ret =3D alloc_and_dissolve_huge_page(h, head, list); - + } return ret; } =20 +void sync_hugetlb_dissolve(void) +{ + flush_work(&free_hpage_work); + flush_work(&dissolve_hpage_work); +} + struct page *alloc_huge_page(struct vm_area_struct *vma, unsigned long addr, int avoid_reserve) { diff --git a/mm/migrate.c b/mm/migrate.c index 6a1597c92261..b6c6123e614c 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -141,6 +141,7 @@ void putback_movable_pages(struct list_head *l) =20 list_for_each_entry_safe(page, page2, l, lru) { if (unlikely(PageHuge(page))) { + ClearHPageDissolve(page); putback_active_hugepage(page); continue; } diff --git a/mm/page_alloc.c b/mm/page_alloc.c index e5486d47406e..6bf76bbc0308 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -9235,6 +9235,7 @@ int alloc_contig_range(unsigned long start, unsigned = long end, if (ret && ret !=3D -EBUSY) goto done; ret =3D 0; + sync_hugetlb_dissolve(); =20 /* * Pages from [start, end) are within a pageblock_nr_pages --=20 2.25.1 From nobody Tue Dec 16 03:00:38 2025 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6BA37ECAAD8 for ; Tue, 13 Sep 2022 19:58:57 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229511AbiIMT6z (ORCPT ); Tue, 13 Sep 2022 15:58:55 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46748 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229895AbiIMT62 (ORCPT ); Tue, 13 Sep 2022 15:58:28 -0400 Received: from mail-qk1-x72b.google.com (mail-qk1-x72b.google.com [IPv6:2607:f8b0:4864:20::72b]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0D24F78200; Tue, 13 Sep 2022 12:58:18 -0700 (PDT) Received: by mail-qk1-x72b.google.com with SMTP id d15so9307283qka.9; Tue, 13 Sep 2022 12:58:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date; bh=5478I3WyUfDJcI1zhJnzCnFwBV1ia1XR+sLHu3XJI6Q=; b=ngWpsEYVUBqyRCK+w+5Ame50HZp1DYM8gstxDexc5gHtIO2fEu4wX/+7Noqq7EWBX5 SX4JOXX5B1E8Qf1s/pvzk9Bz2iJJBgv4Hk3Ull+tKjmtldfM+s3z+vbM1B3di5w6gxjz dcVCWO2NZ50adOCK8bPYxw3tCqWOsDx7M7cTAjTnYf/V4tSaPDhi8fqtVljCEiP5o1JE EBkXOKZZNuxNzz6s5U3pOqHYezTLOaL4mQLClkKUJN8hT0raCVH32ZKrxQcROVGentqi 38kFA0gMphAG/6qqKzQ2VCttdNXcqVd0gmzjLhB0hmlMfPBCKPcE2Rytw17lBrFXHH3/ rqCA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date; bh=5478I3WyUfDJcI1zhJnzCnFwBV1ia1XR+sLHu3XJI6Q=; b=ZdpfxQH5JMWc9atFLaVl342kjTClvqmVLm6tWN8jUEJSadOqlZRCjWzEKZ2gqlrXun svhU85L1/UsjszGpxALGj7CavklIt/24+En7eaWEUs5VinJYbxMFgGZWmXOq6Jd91K+i 39r4he9C3wSa5mPX6jZ8AgJ3o0JM/frfMgOrLVGR4kGF/dcqJQbJ2Umv27AH9vmJKWw2 oZqHXTGes5RTFpEMNcNS7ldLFJtXfcLAqtvf3xpQ6kJwgzoqBkV3ty3ETMi9fCnks8px aRtf5KgXyQkUx7Kn0VoolTHaRA7sxep65RQK6UyXYap8F5uYDdqMPx75F7cMo7/ei1qj MALA== X-Gm-Message-State: ACgBeo3gHm0/c7qvp8ADIW3PPtIMJxy+4sB8U8lcgx5ldy86SKdunGTy MYSxew3nscFHdY2tZ+F0N58= X-Google-Smtp-Source: AA6agR5yg7LQIQNFFYJ0sDmMy6v2CLDWuXatdLv4m+yPVXKNj8MYR/nHEcTFAbCbbhbRcvfawH5dww== X-Received: by 2002:a05:620a:4496:b0:6bb:8db4:6b49 with SMTP id x22-20020a05620a449600b006bb8db46b49mr24594751qkp.703.1663099096778; Tue, 13 Sep 2022 12:58:16 -0700 (PDT) Received: from stbirv-lnx-3.igp.broadcom.net ([192.19.223.252]) by smtp.gmail.com with ESMTPSA id s11-20020a05620a29cb00b006b8e049cf08sm276305qkp.2.2022.09.13.12.58.13 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 13 Sep 2022 12:58:16 -0700 (PDT) From: Doug Berger To: Andrew Morton Cc: Jonathan Corbet , Rob Herring , Krzysztof Kozlowski , Frank Rowand , Mike Kravetz , Muchun Song , Mike Rapoport , Christoph Hellwig , Marek Szyprowski , Robin Murphy , Borislav Petkov , "Paul E. McKenney" , Neeraj Upadhyay , Randy Dunlap , Damien Le Moal , Doug Berger , Florian Fainelli , David Hildenbrand , Zi Yan , Oscar Salvador , Hari Bathini , Kees Cook , - , KOSAKI Motohiro , Mel Gorman , linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, devicetree@vger.kernel.org, linux-mm@kvack.org, iommu@lists.linux.dev Subject: [PATCH 06/21] mm/hugetlb: add hugepage isolation support Date: Tue, 13 Sep 2022 12:54:53 -0700 Message-Id: <20220913195508.3511038-7-opendmb@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220913195508.3511038-1-opendmb@gmail.com> References: <20220913195508.3511038-1-opendmb@gmail.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" When a range of pageblocks is isolated there is at most one hugepage that has only tail pages overlapping that range (i.e. a hugepage that overlaps the beginning of the range). However, that hugepage is the first migration target for an alloc_contig_range() attempt so it already receives special attention. Checking whether the pageblock containing the head of a hugepage is isolated is an inexpensive way to avoid hugepage allocations from isolated pageblocks which makes alloc_contig_range() more efficient. Signed-off-by: Doug Berger --- mm/hugetlb.c | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index da80889e1436..2f354423f50f 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -33,6 +33,7 @@ #include #include #include +#include =20 #include #include @@ -1135,6 +1136,10 @@ static struct page *dequeue_huge_page_node_exact(str= uct hstate *h, int nid) if (PageHWPoison(page)) continue; =20 + /* Check head pageblock isolation */ + if (is_migrate_isolate_page(page)) + continue; + list_move(&page->lru, &h->hugepage_activelist); set_page_refcounted(page); ClearHPageFreed(page); --=20 2.25.1 From nobody Tue Dec 16 03:00:38 2025 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 341AAC54EE9 for ; Tue, 13 Sep 2022 19:59:04 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229964AbiIMT7C (ORCPT ); Tue, 13 Sep 2022 15:59:02 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47026 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229953AbiIMT6a (ORCPT ); Tue, 13 Sep 2022 15:58:30 -0400 Received: from mail-qv1-xf2d.google.com (mail-qv1-xf2d.google.com [IPv6:2607:f8b0:4864:20::f2d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E76167822B; Tue, 13 Sep 2022 12:58:21 -0700 (PDT) Received: by mail-qv1-xf2d.google.com with SMTP id ml1so10116352qvb.1; Tue, 13 Sep 2022 12:58:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date; bh=hB/8CHaP3epa66ZDI7eIxcYHinx9lai7E+SzdR8JK34=; b=DH5wOpdpiHaFFXY8TAT3JRecX64XiPNxiR/CcG1LrVu07Readvn0PWViKgBmDA60Pb UBITk9aWlYro2ikNI4WmLJPOCXwQOqtprdUB0CzoJZ72kknTQx9o/RitCRSffQDxQ5Q6 VTZyKVoIHd28mrFZHIQl9dz1uLmu560sdtEcQWDtP1q9dt0hIGQCuqCTjE9fttuhjONt 0HIT34gX8eS66xaVSIttBxYAN/YQSf5UtfsLj6aZ6r0xFgT3C7geS9uBf8xzXOro8Ch1 QuB8O5yFa+P3URJ5XpTXammfwNsUv1UpGQMU/vjO4KV44uIZqk+ZBhqofvaHP1BLFOQP ilEQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date; bh=hB/8CHaP3epa66ZDI7eIxcYHinx9lai7E+SzdR8JK34=; b=lsGCDxEQmPS3bvGU1uYNwbVPxKKD4cCzXQitNCv83Z4hxM31dWH2JuVcNPq3jKetCo VaxxISkv0XiakRffnHQ0FIJe6o3lsQ7Gl+IGNoJlmAcd1I6yk9PFRhUuR2JQSWnGZ7Cx wye9WTEZcu6J6uPg6oR+Uwkch+yhwSIEDMUEkk79+75u8Wn3+ss/tnvOK8NHSofHMcJq G1OVzI+tCyZgwIKtqOZBPPw12G4qyQMBTB4Lca8ldOaadXlTFzGGReQ+WhNHF5hIee7v eusCLXx6vH2/XpM245Vz/hpIcSr8letmTh5Ybk9YiuNia9+VGxFIq7W6cuH9yMn9TqBE sxkA== X-Gm-Message-State: ACgBeo1agUzYEYbEGSyuozDVO4qiGXFPyX5MF8wrs/uzC6jW64WDzDgR c+ajKkxtZLuUfQzWUmZ+TF0= X-Google-Smtp-Source: AA6agR4VzptNv0IdYnihlFGaNyhbElSt3SCLpuN9uI3qoIO24r77G7BDUS/ieSa0RDNa+EBnGiO84g== X-Received: by 2002:a0c:c98a:0:b0:4ac:9f4b:efed with SMTP id b10-20020a0cc98a000000b004ac9f4befedmr14045129qvk.35.1663099100395; Tue, 13 Sep 2022 12:58:20 -0700 (PDT) Received: from stbirv-lnx-3.igp.broadcom.net ([192.19.223.252]) by smtp.gmail.com with ESMTPSA id s11-20020a05620a29cb00b006b8e049cf08sm276305qkp.2.2022.09.13.12.58.16 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 13 Sep 2022 12:58:19 -0700 (PDT) From: Doug Berger To: Andrew Morton Cc: Jonathan Corbet , Rob Herring , Krzysztof Kozlowski , Frank Rowand , Mike Kravetz , Muchun Song , Mike Rapoport , Christoph Hellwig , Marek Szyprowski , Robin Murphy , Borislav Petkov , "Paul E. McKenney" , Neeraj Upadhyay , Randy Dunlap , Damien Le Moal , Doug Berger , Florian Fainelli , David Hildenbrand , Zi Yan , Oscar Salvador , Hari Bathini , Kees Cook , - , KOSAKI Motohiro , Mel Gorman , linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, devicetree@vger.kernel.org, linux-mm@kvack.org, iommu@lists.linux.dev Subject: [PATCH 07/21] lib/show_mem.c: display MovableOnly Date: Tue, 13 Sep 2022 12:54:54 -0700 Message-Id: <20220913195508.3511038-8-opendmb@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220913195508.3511038-1-opendmb@gmail.com> References: <20220913195508.3511038-1-opendmb@gmail.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" The comment for commit c78e93630d15 ("mm: do not walk all of system memory during show_mem") indicates it "also corrects the reporting of HighMem as HighMem/MovableOnly as ZONE_MOVABLE has similar problems to HighMem with respect to lowmem/highmem exhaustion." Presuming the similar problems are with regard to the general exclusion of kernel allocations from either zone, I believe it makes sense to include all ZONE_MOVABLE memory even on systems without HighMem. To the extent that this was the intent of the original commit I have included a "Fixes" tag, but it seems unnecessary to submit to linux-stable. Fixes: c78e93630d15 ("mm: do not walk all of system memory during show_mem") Signed-off-by: Doug Berger --- lib/show_mem.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/show_mem.c b/lib/show_mem.c index 1c26c14ffbb9..337c870a5e59 100644 --- a/lib/show_mem.c +++ b/lib/show_mem.c @@ -27,7 +27,7 @@ void show_mem(unsigned int filter, nodemask_t *nodemask) total +=3D zone->present_pages; reserved +=3D zone->present_pages - zone_managed_pages(zone); =20 - if (is_highmem_idx(zoneid)) + if (zoneid =3D=3D ZONE_MOVABLE || is_highmem_idx(zoneid)) highmem +=3D zone->present_pages; } } --=20 2.25.1 From nobody Tue Dec 16 03:00:38 2025 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 08BC0C54EE9 for ; Tue, 13 Sep 2022 19:59:10 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230076AbiIMT7I (ORCPT ); Tue, 13 Sep 2022 15:59:08 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46838 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229998AbiIMT6c (ORCPT ); Tue, 13 Sep 2022 15:58:32 -0400 Received: from mail-qv1-xf35.google.com (mail-qv1-xf35.google.com [IPv6:2607:f8b0:4864:20::f35]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1B3B174B81; Tue, 13 Sep 2022 12:58:25 -0700 (PDT) Received: by mail-qv1-xf35.google.com with SMTP id o13so10086435qvw.12; Tue, 13 Sep 2022 12:58:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date; bh=6/+RylwAUS1L6kGkdbzNevLkV7SCUovxqo9fjOaS6JU=; b=IIvzQBml/wGri/ZBWsuig4iC5yLnXMtk1VHesQ1eh/9HzasiMgV8YxfdpWHpGIoYep M/INFOPjeTVC3OEHHCjy2E1Hs8kTN4G53N8xRA/QBMd1ulJjTypxrYcp9Vasl1Q/eOJB qHAj6LLFWCEJm7FVRIifJLn+g0xAhtpqiGJt/0Lazj8LpFDzEnyQtx4bqcugERvtsdZG jeSnKtUCKGP4SXlnb3HB7CkPQRbT8a6KgjjUz3Q32zw4SQOfC5UckDKNJQGxML3W3hgv kauff9Q/Ua47CAvmWSZZgrFSzvsXxKLS/PIpqLn25rigG7/KetjxEqCiT7a4LgLAH6NO U6Ng== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date; bh=6/+RylwAUS1L6kGkdbzNevLkV7SCUovxqo9fjOaS6JU=; b=qgVFZ8T7kukAfgpncvsvudRZ+DSjxdQyvYvQfMSlqLSIr/2/KakJQAmUJ8NlVxcvWR +77JmhrQ1attjTjhQ3C5VvwdQZEEdPQJTULyidCKIHt5nw5dSdSDq59/m0yLivBefH5w 2qlgHgBuKxfa8SCzWSb/YbTG4bk+HfCGTMvvdtYAN5U80VH188Lo9HqMi9BQVAuAzb4G ZVYMf4svKyDsaz9n/+Fthqi9sizW5jQyFW2NA1VDXh3Ur8S8kNa9o2OEcH+c5OIJrGHz OjTpY220DOnkOwQIdBAJbg+S/X7xf9dhbgL4zt+fSY6OtZzzv9o627JwM+/kDEh148E3 PsAA== X-Gm-Message-State: ACgBeo0SytFwwStcyHTs6CBGDEqpUIA1rjh6K5K7T6QqcKXa4GzsHjEs xohwT4YTTPzuY41nJ6q2gCg= X-Google-Smtp-Source: AA6agR7HhxHAKHSAet4vBy7WKGiBHiOPyi3Gu3lpC/Y1Ix8I/hDUvAWKeNeCYEL5N+Wcelu80g5S7w== X-Received: by 2002:a0c:d987:0:b0:4a5:192c:faff with SMTP id y7-20020a0cd987000000b004a5192cfaffmr29341373qvj.106.1663099104077; Tue, 13 Sep 2022 12:58:24 -0700 (PDT) Received: from stbirv-lnx-3.igp.broadcom.net ([192.19.223.252]) by smtp.gmail.com with ESMTPSA id s11-20020a05620a29cb00b006b8e049cf08sm276305qkp.2.2022.09.13.12.58.20 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 13 Sep 2022 12:58:23 -0700 (PDT) From: Doug Berger To: Andrew Morton Cc: Jonathan Corbet , Rob Herring , Krzysztof Kozlowski , Frank Rowand , Mike Kravetz , Muchun Song , Mike Rapoport , Christoph Hellwig , Marek Szyprowski , Robin Murphy , Borislav Petkov , "Paul E. McKenney" , Neeraj Upadhyay , Randy Dunlap , Damien Le Moal , Doug Berger , Florian Fainelli , David Hildenbrand , Zi Yan , Oscar Salvador , Hari Bathini , Kees Cook , - , KOSAKI Motohiro , Mel Gorman , linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, devicetree@vger.kernel.org, linux-mm@kvack.org, iommu@lists.linux.dev Subject: [PATCH 08/21] mm/vmstat: show start_pfn when zone spans pages Date: Tue, 13 Sep 2022 12:54:55 -0700 Message-Id: <20220913195508.3511038-9-opendmb@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220913195508.3511038-1-opendmb@gmail.com> References: <20220913195508.3511038-1-opendmb@gmail.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" A zone that overlaps with another zone may span a range of pages that are not present. In this case, displaying the start_pfn of the zone allows the zone page range to be identified. Signed-off-by: Doug Berger --- mm/vmstat.c | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/mm/vmstat.c b/mm/vmstat.c index 90af9a8572f5..e2f19f2b7615 100644 --- a/mm/vmstat.c +++ b/mm/vmstat.c @@ -1717,6 +1717,11 @@ static void zoneinfo_show_print(struct seq_file *m, = pg_data_t *pgdat, =20 /* If unpopulated, no other information is useful */ if (!populated_zone(zone)) { + /* Show start_pfn for empty overlapped zones */ + if (zone->spanned_pages) + seq_printf(m, + "\n start_pfn: %lu", + zone->zone_start_pfn); seq_putc(m, '\n'); return; } --=20 2.25.1 From nobody Tue Dec 16 03:00:38 2025 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id B9CD7C54EE9 for ; Tue, 13 Sep 2022 19:59:19 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229770AbiIMT7S (ORCPT ); Tue, 13 Sep 2022 15:59:18 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46842 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230029AbiIMT6g (ORCPT ); Tue, 13 Sep 2022 15:58:36 -0400 Received: from mail-qk1-x72f.google.com (mail-qk1-x72f.google.com [IPv6:2607:f8b0:4864:20::72f]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B64C874CE6; Tue, 13 Sep 2022 12:58:28 -0700 (PDT) Received: by mail-qk1-x72f.google.com with SMTP id s9so7062214qkg.4; Tue, 13 Sep 2022 12:58:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date; bh=kefl6D8eKpN1GxKRqpPd+2PbW8yMD7NaxrgEOy2EJKU=; b=byR/eekYsW9DhH0NqLev2fS+tpkFMN+hxgB8j3wmXvKTocIMCm6zZesYvYgQ87wjhP q+4AfKw8TXxcj2+vnpVcn5ECPh4qGk7Jtzj+zNbyb6YCFLIiwPOzRUIUuAqm/vme1CGD HXFFRmWffOsc2lXoXpa+V0KHw2MwGlrFjaT+dLfIFP4Da4Lspw/3GTIWmZkSLzazc/cQ hUYdezyBusY92UGvKeXjT7Rg0vsemlXyaK6ln+reN6yeiO3uNu7F3nGVR2B+OsucHch6 d5MKRdQDkKjHro2ts06Z/prjBlf3W1AQZuN3IrXE7aLh74QN4KD+aUTWaAkLm41rgl56 mOWw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date; bh=kefl6D8eKpN1GxKRqpPd+2PbW8yMD7NaxrgEOy2EJKU=; b=pxILh4RiPrkaLKWtwcO0Stiql600sgOP4HLqmk9oo8eEU9flyY8wE9iF6hahCeU4kU BgQrrntlmdJOTU0LWOqmn1MZ1o6/AhMI80fAkQZCq1SOXmIOxOrnLCUeSN/7BKjWrNKw L0+CNoG6rdkEn2CaBtz3TDYC3GyJMgIu57FUXe4vDWuqCle0fY/BLuCDJ9OggJOklSAD U2f2SPFlS5jzDtwEbShQgfxFCfFbhM8r8r0BfBM3W2y9IbjDyW37iEAsmCu8lkdqNy2f RUzvFwhFR/HQEDDGdJoESoFb+EzX+1cfFnsYrJhBxnvrZsL+BLZ+qSupdiVMsSN0YPfT Ruzw== X-Gm-Message-State: ACgBeo0+TIBXew2b9EwIELFX2rXqbBjea6U+B2Fcz+zZa1MrWuOlAEYA l5lhjXkWmfzpSCl7iUNeF8s= X-Google-Smtp-Source: AA6agR4kCtYZpwUOe+kdoRL5XUeDY9Udr6YA+L+RrDdkM+ZkGSBIa8BGCY7yWmaW8ey8FgVIise+hw== X-Received: by 2002:a37:916:0:b0:6ce:5f1c:d5d9 with SMTP id 22-20020a370916000000b006ce5f1cd5d9mr5678519qkj.737.1663099107720; Tue, 13 Sep 2022 12:58:27 -0700 (PDT) Received: from stbirv-lnx-3.igp.broadcom.net ([192.19.223.252]) by smtp.gmail.com with ESMTPSA id s11-20020a05620a29cb00b006b8e049cf08sm276305qkp.2.2022.09.13.12.58.24 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 13 Sep 2022 12:58:27 -0700 (PDT) From: Doug Berger To: Andrew Morton Cc: Jonathan Corbet , Rob Herring , Krzysztof Kozlowski , Frank Rowand , Mike Kravetz , Muchun Song , Mike Rapoport , Christoph Hellwig , Marek Szyprowski , Robin Murphy , Borislav Petkov , "Paul E. McKenney" , Neeraj Upadhyay , Randy Dunlap , Damien Le Moal , Doug Berger , Florian Fainelli , David Hildenbrand , Zi Yan , Oscar Salvador , Hari Bathini , Kees Cook , - , KOSAKI Motohiro , Mel Gorman , linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, devicetree@vger.kernel.org, linux-mm@kvack.org, iommu@lists.linux.dev Subject: [PATCH 09/21] mm/page_alloc: calculate node_spanned_pages from pfns Date: Tue, 13 Sep 2022 12:54:56 -0700 Message-Id: <20220913195508.3511038-10-opendmb@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220913195508.3511038-1-opendmb@gmail.com> References: <20220913195508.3511038-1-opendmb@gmail.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Since the start and end pfns of the node are passed as arguments to calculate_node_totalpages() they might as well be used to specify the node_spanned_pages value for the node rather than accumulating the spans of member zones. This prevents the need for additional adjustments if zones are allowed to overlap. Signed-off-by: Doug Berger --- mm/page_alloc.c | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 6bf76bbc0308..b6074961fb59 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -7452,7 +7452,7 @@ static void __init calculate_node_totalpages(struct p= glist_data *pgdat, unsigned long node_start_pfn, unsigned long node_end_pfn) { - unsigned long realtotalpages =3D 0, totalpages =3D 0; + unsigned long realtotalpages =3D 0; enum zone_type i; =20 for (i =3D 0; i < MAX_NR_ZONES; i++) { @@ -7483,11 +7483,10 @@ static void __init calculate_node_totalpages(struct= pglist_data *pgdat, zone->present_early_pages =3D real_size; #endif =20 - totalpages +=3D size; realtotalpages +=3D real_size; } =20 - pgdat->node_spanned_pages =3D totalpages; + pgdat->node_spanned_pages =3D node_end_pfn - node_start_pfn; pgdat->node_present_pages =3D realtotalpages; pr_debug("On node %d totalpages: %lu\n", pgdat->node_id, realtotalpages); } --=20 2.25.1 From nobody Tue Dec 16 03:00:38 2025 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8554CC54EE9 for ; Tue, 13 Sep 2022 19:59:23 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230101AbiIMT7W (ORCPT ); Tue, 13 Sep 2022 15:59:22 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46748 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229499AbiIMT6i (ORCPT ); Tue, 13 Sep 2022 15:58:38 -0400 Received: from mail-qt1-x82e.google.com (mail-qt1-x82e.google.com [IPv6:2607:f8b0:4864:20::82e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3DDF172FE8; Tue, 13 Sep 2022 12:58:32 -0700 (PDT) Received: by mail-qt1-x82e.google.com with SMTP id a20so6960420qtw.10; Tue, 13 Sep 2022 12:58:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date; bh=bjPZ9urefAsJcftBbAgQB1diXayOj809VCbylW0WU+4=; b=ElB8vGfwnTgMrOHPV8jOPMjX9e4dyOnpsA/daCy84nADITgBEDs0AWwUlYd7LmzJ4R Fd+wihlwr+eHGiOYJevIadmukrrhPQQQbV14He/EgVL8UId/ZjEQspu78E84lCBkqAk3 9AGpDbnJvnF63g83iAedqf+9zINjZ62a64qbSfO2T1vB7Ckgl7brAPRPzZAIUuLk9YsF YnnaDAUhOF2TFJBs1+hdGFIi2/6/xETSCDXaO1b95XuqZwbhceSd0bqWp4uVAOWzs4lk zAxkAToGZXGzD1zml0W1YllS9/kHr+k5y9Z8yZENt9c1+33raLf8akoDdgYfGMOxfC4A ITFg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date; bh=bjPZ9urefAsJcftBbAgQB1diXayOj809VCbylW0WU+4=; b=2vKeVej398VF1KxT6CmF+E7esLxfw/Fj1hH+T2RFtOUPWXojcmoxLBGtE2zN/QU0OO dgSFdeQ3aUPKjIxnZgqFws4nUn3YQWEL2ldTWe7mEnVKoj5uVDtheVMFgEZeSPVg0Phy 5I0yVqsxMLqFcy2oqH/ao5I1e1GzkCAyJJ7l5zUzXEmTF10QNpGKAp8hTLIndVVthv1p s0GurfFepRyfXNTwgX/5yaIGIxpIhQzw93fHlCcKAaqBVl7gd5WFyL4D/AkeK5e15ePd DShJVf0f8tACEOEofsP3lP7Vs/hoJXB9STRpbIQOGWkinUrEK5R02dn7ET4WfXnlpHGk f7AQ== X-Gm-Message-State: ACgBeo3haF0AkNEgOStpbETGk8eX+5kQ86HA89ABPA0nWIIYviAjKVNh 5/0OVF2wnbhas/Yo7UisHvs= X-Google-Smtp-Source: AA6agR4w6aZMPi6cpzcCwIclFPfEwGVo8/KdfU9B4JBAlrDz19GDQ32yg1HkC6aVwcDa1gVOoM50cA== X-Received: by 2002:a05:622a:174b:b0:343:1fc:14d8 with SMTP id l11-20020a05622a174b00b0034301fc14d8mr29317121qtk.579.1663099111308; Tue, 13 Sep 2022 12:58:31 -0700 (PDT) Received: from stbirv-lnx-3.igp.broadcom.net ([192.19.223.252]) by smtp.gmail.com with ESMTPSA id s11-20020a05620a29cb00b006b8e049cf08sm276305qkp.2.2022.09.13.12.58.27 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 13 Sep 2022 12:58:30 -0700 (PDT) From: Doug Berger To: Andrew Morton Cc: Jonathan Corbet , Rob Herring , Krzysztof Kozlowski , Frank Rowand , Mike Kravetz , Muchun Song , Mike Rapoport , Christoph Hellwig , Marek Szyprowski , Robin Murphy , Borislav Petkov , "Paul E. McKenney" , Neeraj Upadhyay , Randy Dunlap , Damien Le Moal , Doug Berger , Florian Fainelli , David Hildenbrand , Zi Yan , Oscar Salvador , Hari Bathini , Kees Cook , - , KOSAKI Motohiro , Mel Gorman , linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, devicetree@vger.kernel.org, linux-mm@kvack.org, iommu@lists.linux.dev Subject: [PATCH 10/21] mm/page_alloc.c: allow oversized movablecore Date: Tue, 13 Sep 2022 12:54:57 -0700 Message-Id: <20220913195508.3511038-11-opendmb@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220913195508.3511038-1-opendmb@gmail.com> References: <20220913195508.3511038-1-opendmb@gmail.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Now that the error in computation of corepages has been corrected by commit 9fd745d450e7 ("mm: fix overflow in find_zone_movable_pfns_for_nodes()"), oversized specifications of movablecore will result in a zero value for required_kernelcore if it is not also specified. It is unintuitive for such a request to lead to no ZONE_MOVABLE memory when the kernel parameters are clearly requesting some. The current behavior when requesting an oversized kernelcore is to classify all of the pages in movable_zone as kernelcore. The new behavior when requesting an oversized movablecore (when not also specifying kernelcore) is to similarly classify all of the pages in movable_zone as movablecore. Signed-off-by: Doug Berger --- mm/page_alloc.c | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index b6074961fb59..ad38a81203e5 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -8041,13 +8041,13 @@ static void __init find_zone_movable_pfns_for_nodes= (void) corepages =3D totalpages - required_movablecore; =20 required_kernelcore =3D max(required_kernelcore, corepages); + } else if (!required_kernelcore) { + /* If kernelcore was not specified, there is no ZONE_MOVABLE */ + goto out; } =20 - /* - * If kernelcore was not specified or kernelcore size is larger - * than totalpages, there is no ZONE_MOVABLE. - */ - if (!required_kernelcore || required_kernelcore >=3D totalpages) + /* If kernelcore size exceeds totalpages, there is no ZONE_MOVABLE */ + if (required_kernelcore >=3D totalpages) goto out; =20 /* usable_startpfn is the lowest possible pfn ZONE_MOVABLE can be at */ --=20 2.25.1 From nobody Tue Dec 16 03:00:38 2025 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 54941ECAAD8 for ; Tue, 13 Sep 2022 19:59:30 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230109AbiIMT72 (ORCPT ); Tue, 13 Sep 2022 15:59:28 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47194 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229768AbiIMT6j (ORCPT ); Tue, 13 Sep 2022 15:58:39 -0400 Received: from mail-qv1-xf2c.google.com (mail-qv1-xf2c.google.com [IPv6:2607:f8b0:4864:20::f2c]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 132A075CD8; Tue, 13 Sep 2022 12:58:35 -0700 (PDT) Received: by mail-qv1-xf2c.google.com with SMTP id q8so10091435qvr.9; Tue, 13 Sep 2022 12:58:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date; bh=KeLoRQJB3eAOdyXC4NTd1TP4on3QCykrOlScFxq8ZPE=; b=FeFlqfsiEQU3HSeYb1n4gkpg1qmnB5WeDR3FybX0zPrw0OvjL/48yUYTNjXdAaMlAU JdyQzl2XCc8s8RP5OO/H5J7S/0xDcSLDPZ/g9EIPbcsw9LhFgl8pcNCL7ia4xS0HbBWk UrWVqsTOfQjwUdrXVkxZ/eY8sVsnX0gm6/fSQE07CBzEH1IsIWTpZVsPIgNRukD5QnzD c6mLxWXf85eq1Q9U3EFaUBdh/wEyMg/Ynk1wbDDmvjWml5kn5SQbnOB+ZusyeRT2gzZB 30p18F8NjILejz6FdnbFmeCsFxJLxim3pDKbO19ZAHgQZI8N1Ooww3p9oQH3KqqVXxOk 7Eog== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date; bh=KeLoRQJB3eAOdyXC4NTd1TP4on3QCykrOlScFxq8ZPE=; b=73KjvX/hW9b5RIPytF7VDZZ7S9LzWUCqudyw8RUyKfzCrGd7gkafXXeBavDigf4GBj +SVaqRADsGAZHfiNfxsflokn9wb9vAFNSGIygClAFJFwd57lWGwuVg5Q24TPJWd/v/2p MYiMhO8D17H8ltry2pZ4Ccufwbj3cZSDTgZwQgfx6JxVeeqNYizA+Ows5gzI3P1058Qw /xzwh4+hT2pQl3y8YpHrXDPAQtOokDevGK4nQd7NOahZX0d+iD5pN4BabwGc3rRlOOeG XmCeAmTeXJ8/SGlxX9p+dXKvSM9pRdbHieA4xMoRWsBdWkmeopsrYE2corD+GuECEJmv V+pg== X-Gm-Message-State: ACgBeo1od7ThmtacegZzr+jd83r5Zf2XBxYxgTruXNP71wOOO7MifYy6 ihRahicxK6tfQyxIngXEprQ= X-Google-Smtp-Source: AA6agR6LdySz/okFv68gwh3HYvltSAZalCbl/B/p5X5bZPgANvOoXngUoiwr94jsBmavX1SN+2thWQ== X-Received: by 2002:a05:6214:301a:b0:4ac:a4ec:b8b1 with SMTP id ke26-20020a056214301a00b004aca4ecb8b1mr13793217qvb.122.1663099114897; Tue, 13 Sep 2022 12:58:34 -0700 (PDT) Received: from stbirv-lnx-3.igp.broadcom.net ([192.19.223.252]) by smtp.gmail.com with ESMTPSA id s11-20020a05620a29cb00b006b8e049cf08sm276305qkp.2.2022.09.13.12.58.31 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 13 Sep 2022 12:58:34 -0700 (PDT) From: Doug Berger To: Andrew Morton Cc: Jonathan Corbet , Rob Herring , Krzysztof Kozlowski , Frank Rowand , Mike Kravetz , Muchun Song , Mike Rapoport , Christoph Hellwig , Marek Szyprowski , Robin Murphy , Borislav Petkov , "Paul E. McKenney" , Neeraj Upadhyay , Randy Dunlap , Damien Le Moal , Doug Berger , Florian Fainelli , David Hildenbrand , Zi Yan , Oscar Salvador , Hari Bathini , Kees Cook , - , KOSAKI Motohiro , Mel Gorman , linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, devicetree@vger.kernel.org, linux-mm@kvack.org, iommu@lists.linux.dev Subject: [PATCH 11/21] mm/page_alloc: introduce init_reserved_pageblock() Date: Tue, 13 Sep 2022 12:54:58 -0700 Message-Id: <20220913195508.3511038-12-opendmb@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220913195508.3511038-1-opendmb@gmail.com> References: <20220913195508.3511038-1-opendmb@gmail.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Most of the implementation of init_cma_reserved_pageblock() is common to the initialization of any reserved pageblock for use by the page allocator. This commit breaks that functionality out into the new common function init_reserved_pageblock() for use by code other than CMA. The CMA specific code is relocated from page_alloc to the point where init_cma_reserved_pageblock() was invoked and the new function is used there instead. The error path is also updated to use the function to operate on pageblocks rather than pages. Signed-off-by: Doug Berger --- include/linux/gfp.h | 5 +---- mm/cma.c | 15 +++++++++++---- mm/page_alloc.c | 8 ++------ 3 files changed, 14 insertions(+), 14 deletions(-) diff --git a/include/linux/gfp.h b/include/linux/gfp.h index f314be58fa77..71ed687be406 100644 --- a/include/linux/gfp.h +++ b/include/linux/gfp.h @@ -367,9 +367,6 @@ extern struct page *alloc_contig_pages(unsigned long nr= _pages, gfp_t gfp_mask, #endif void free_contig_range(unsigned long pfn, unsigned long nr_pages); =20 -#ifdef CONFIG_CMA -/* CMA stuff */ -extern void init_cma_reserved_pageblock(struct page *page); -#endif +extern void init_reserved_pageblock(struct page *page); =20 #endif /* __LINUX_GFP_H */ diff --git a/mm/cma.c b/mm/cma.c index 4a978e09547a..6208a3e1cd9d 100644 --- a/mm/cma.c +++ b/mm/cma.c @@ -31,6 +31,7 @@ #include #include #include +#include #include =20 #include "cma.h" @@ -116,8 +117,13 @@ static void __init cma_activate_area(struct cma *cma) } =20 for (pfn =3D base_pfn; pfn < base_pfn + cma->count; - pfn +=3D pageblock_nr_pages) - init_cma_reserved_pageblock(pfn_to_page(pfn)); + pfn +=3D pageblock_nr_pages) { + struct page *page =3D pfn_to_page(pfn); + + set_pageblock_migratetype(page, MIGRATE_CMA); + init_reserved_pageblock(page); + page_zone(page)->cma_pages +=3D pageblock_nr_pages; + } =20 spin_lock_init(&cma->lock); =20 @@ -133,8 +139,9 @@ static void __init cma_activate_area(struct cma *cma) out_error: /* Expose all pages to the buddy, they are useless for CMA. */ if (!cma->reserve_pages_on_error) { - for (pfn =3D base_pfn; pfn < base_pfn + cma->count; pfn++) - free_reserved_page(pfn_to_page(pfn)); + for (pfn =3D base_pfn; pfn < base_pfn + cma->count; + pfn +=3D pageblock_nr_pages) + init_reserved_pageblock(pfn_to_page(pfn)); } totalcma_pages -=3D cma->count; cma->count =3D 0; diff --git a/mm/page_alloc.c b/mm/page_alloc.c index ad38a81203e5..1682d8815efa 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -2302,9 +2302,8 @@ void __init page_alloc_init_late(void) set_zone_contiguous(zone); } =20 -#ifdef CONFIG_CMA -/* Free whole pageblock and set its migration type to MIGRATE_CMA. */ -void __init init_cma_reserved_pageblock(struct page *page) +/* Free whole pageblock */ +void __init init_reserved_pageblock(struct page *page) { unsigned i =3D pageblock_nr_pages; struct page *p =3D page; @@ -2314,14 +2313,11 @@ void __init init_cma_reserved_pageblock(struct page= *page) set_page_count(p, 0); } while (++p, --i); =20 - set_pageblock_migratetype(page, MIGRATE_CMA); set_page_refcounted(page); __free_pages(page, pageblock_order); =20 adjust_managed_page_count(page, pageblock_nr_pages); - page_zone(page)->cma_pages +=3D pageblock_nr_pages; } -#endif =20 /* * The order of subdivision here is critical for the IO subsystem. --=20 2.25.1 From nobody Tue Dec 16 03:00:38 2025 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9D547ECAAD8 for ; Tue, 13 Sep 2022 19:59:44 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229640AbiIMT7m (ORCPT ); Tue, 13 Sep 2022 15:59:42 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47004 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229903AbiIMT6o (ORCPT ); Tue, 13 Sep 2022 15:58:44 -0400 Received: from mail-qt1-x82b.google.com (mail-qt1-x82b.google.com [IPv6:2607:f8b0:4864:20::82b]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1239877EB4; Tue, 13 Sep 2022 12:58:39 -0700 (PDT) Received: by mail-qt1-x82b.google.com with SMTP id r20so8906218qtn.12; Tue, 13 Sep 2022 12:58:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date; bh=5WWAAzNcGRX09yf3QcgLeiV4ytIgw0QIAz2zMov/ZDY=; b=Yz1mtV6YiuvArbfHbQBXNKT9eLELbvn7bnniGDLD1BumkiBsr2ZDlVa36+Wy8wKT5o ryYhGAJx49vUh11TKsorymepYWGSO76xJbhP0rWVGH3ntJ/cZBgl0HG4OoxTwsRCKi5P r8fZHkmi9dhCwHqvtScDgQwe7gg6nWmJBlbT6hYOH0ew7iRaIAjSwvoSk6SKm+tRfQu+ rKZsLcxmkz1cCAli2W9VyR6gyYWGuJOVoZMSgPsY0YmHve5v7DOrmmIyu3LMUZg6pUfw lIU/zKc3QMqDUaQMXnCFOtpC/YhN4nYgSjeSX0Y2GgcVNB6wvWWTn+zhjMxvZzDNv7zs 69cQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date; bh=5WWAAzNcGRX09yf3QcgLeiV4ytIgw0QIAz2zMov/ZDY=; b=keVK2qTyr0mAp68rSoSKQu2s0EYZtJ0bTuSBrK5zTl1JgbNXoxxQl1YT+pY8QQzoz0 saVuhpTmIfoHIlBlo8DPQph6N9bE1/SKbIQ0mQEzvJ6CSVLrj7KOolr1QEdWAj7xha3k jfiloyjwHoBL9F+DpmL44ThLdRC+qo8HZ/o+79vkLb9dmAzWkXqy0EM1fIQEw4dJ+v92 kHtBXq0M4tJ1FmigwLMPiaWcUMZniDCaQFa3pXHMp+hSsvOMjSvsGgOQdOEgFRi988b5 N076arQ4q2iblI1BbvgMK6xXc6PgFMdsZloZwzQXQwuhoitoR9uLj5eSXeinyEvHrJ2R po1Q== X-Gm-Message-State: ACgBeo2xHUdK/BT7ZkzDVqAXW7ZG+QvWOf73qetsowJQVpNQB96yeXcY Z+r+U5DL2URTREHK2PAfiUM= X-Google-Smtp-Source: AA6agR6JFHHtW1Pd7kJf3d4lhxO33YWLvSfzGCmyAGQLZ1dEkgSuVHnr1X0agG5L8d/cKj0D7xYuZA== X-Received: by 2002:a05:622a:1998:b0:343:6452:dbd9 with SMTP id u24-20020a05622a199800b003436452dbd9mr30254647qtc.423.1663099118855; Tue, 13 Sep 2022 12:58:38 -0700 (PDT) Received: from stbirv-lnx-3.igp.broadcom.net ([192.19.223.252]) by smtp.gmail.com with ESMTPSA id s11-20020a05620a29cb00b006b8e049cf08sm276305qkp.2.2022.09.13.12.58.35 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 13 Sep 2022 12:58:38 -0700 (PDT) From: Doug Berger To: Andrew Morton Cc: Jonathan Corbet , Rob Herring , Krzysztof Kozlowski , Frank Rowand , Mike Kravetz , Muchun Song , Mike Rapoport , Christoph Hellwig , Marek Szyprowski , Robin Murphy , Borislav Petkov , "Paul E. McKenney" , Neeraj Upadhyay , Randy Dunlap , Damien Le Moal , Doug Berger , Florian Fainelli , David Hildenbrand , Zi Yan , Oscar Salvador , Hari Bathini , Kees Cook , - , KOSAKI Motohiro , Mel Gorman , linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, devicetree@vger.kernel.org, linux-mm@kvack.org, iommu@lists.linux.dev Subject: [PATCH 12/21] memblock: introduce MEMBLOCK_MOVABLE flag Date: Tue, 13 Sep 2022 12:54:59 -0700 Message-Id: <20220913195508.3511038-13-opendmb@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220913195508.3511038-1-opendmb@gmail.com> References: <20220913195508.3511038-1-opendmb@gmail.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" The MEMBLOCK_MOVABLE flag is introduced to designate a memblock as only supporting movable allocations by the page allocator. Signed-off-by: Doug Berger --- include/linux/memblock.h | 8 ++++++++ mm/memblock.c | 24 ++++++++++++++++++++++++ 2 files changed, 32 insertions(+) diff --git a/include/linux/memblock.h b/include/linux/memblock.h index 50ad19662a32..8eb3ca32dfa7 100644 --- a/include/linux/memblock.h +++ b/include/linux/memblock.h @@ -47,6 +47,7 @@ enum memblock_flags { MEMBLOCK_MIRROR =3D 0x2, /* mirrored region */ MEMBLOCK_NOMAP =3D 0x4, /* don't add to kernel direct mapping */ MEMBLOCK_DRIVER_MANAGED =3D 0x8, /* always detected via a driver */ + MEMBLOCK_MOVABLE =3D 0x10, /* designated movable block */ }; =20 /** @@ -125,6 +126,8 @@ int memblock_clear_hotplug(phys_addr_t base, phys_addr_= t size); int memblock_mark_mirror(phys_addr_t base, phys_addr_t size); int memblock_mark_nomap(phys_addr_t base, phys_addr_t size); int memblock_clear_nomap(phys_addr_t base, phys_addr_t size); +int memblock_mark_movable(phys_addr_t base, phys_addr_t size); +int memblock_clear_movable(phys_addr_t base, phys_addr_t size); =20 void memblock_free_all(void); void memblock_free(void *ptr, size_t size); @@ -265,6 +268,11 @@ static inline bool memblock_is_driver_managed(struct m= emblock_region *m) return m->flags & MEMBLOCK_DRIVER_MANAGED; } =20 +static inline bool memblock_is_movable(struct memblock_region *m) +{ + return m->flags & MEMBLOCK_MOVABLE; +} + int memblock_search_pfn_nid(unsigned long pfn, unsigned long *start_pfn, unsigned long *end_pfn); void __next_mem_pfn_range(int *idx, int nid, unsigned long *out_start_pfn, diff --git a/mm/memblock.c b/mm/memblock.c index b5d3026979fc..5d6a210d98ec 100644 --- a/mm/memblock.c +++ b/mm/memblock.c @@ -979,6 +979,30 @@ int __init_memblock memblock_clear_nomap(phys_addr_t b= ase, phys_addr_t size) return memblock_setclr_flag(base, size, 0, MEMBLOCK_NOMAP); } =20 +/** + * memblock_mark_movable - Mark designated movable block with MEMBLOCK_MOV= ABLE. + * @base: the base phys addr of the region + * @size: the size of the region + * + * Return: 0 on success, -errno on failure. + */ +int __init_memblock memblock_mark_movable(phys_addr_t base, phys_addr_t si= ze) +{ + return memblock_setclr_flag(base, size, 1, MEMBLOCK_MOVABLE); +} + +/** + * memblock_clear_movable - Clear flag MEMBLOCK_MOVABLE for a specified re= gion. + * @base: the base phys addr of the region + * @size: the size of the region + * + * Return: 0 on success, -errno on failure. + */ +int __init_memblock memblock_clear_movable(phys_addr_t base, phys_addr_t s= ize) +{ + return memblock_setclr_flag(base, size, 0, MEMBLOCK_MOVABLE); +} + static bool should_skip_region(struct memblock_type *type, struct memblock_region *m, int nid, int flags) --=20 2.25.1 From nobody Tue Dec 16 03:00:38 2025 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1CB14C6FA8D for ; Tue, 13 Sep 2022 19:59:40 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229925AbiIMT7i (ORCPT ); Tue, 13 Sep 2022 15:59:38 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47388 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229904AbiIMT6o (ORCPT ); Tue, 13 Sep 2022 15:58:44 -0400 Received: from mail-qk1-x72c.google.com (mail-qk1-x72c.google.com [IPv6:2607:f8b0:4864:20::72c]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id DC5C47697B; Tue, 13 Sep 2022 12:58:42 -0700 (PDT) Received: by mail-qk1-x72c.google.com with SMTP id y2so6831082qkl.11; Tue, 13 Sep 2022 12:58:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date; bh=WT2HgOEfMbFWghWK9yDd+nkL0DbhssI0liQaO6tZEZs=; b=NfYu0WDbln3rXPAMbeUyjmnvdIGA79s9mtSia48vz0sEDfu2+4xfWqtDXeYpTUlyRc 54Asfh9SWgfuW8gGqmBaFd7ieggdhyEGugonHr+HKUAimN2KvkcgvqS3KbvRNk7jbeiU pGj8A2IHi3i4nWgWwdQY/62/SBJ4kOo+XJKjOoujv76opKObBHDRoqHOvK9jveceTEPN xT/IY+PpVqUyvoH2+Nkw/JGST2FyicYuP52lylnWYbkXJXT6cJrrig7qu89oh2NMLK/r lFceQECtLR20FuN/NHPRY6nDYri8WtxZ1z09emAxpBR0TCKH3LCGsCDGO13oleU0JeEj NOVA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date; bh=WT2HgOEfMbFWghWK9yDd+nkL0DbhssI0liQaO6tZEZs=; b=wWVJDcrlPivRbCzZ7aJ73KQ1lU2FBSiElyr6KjP/B7TKl0I4v+FD0bD3KP2WX67nK0 kxWDJEG1ksHeDWp3UGq+F82qknVkX9UPVzusEWeDrBVOCBzdWWxtmYJmvP9nu8t1VF9/ +7BrpfOXc12VUxoNa15GxY7/nvfUQV+nMRqYttcHErCj0GROtReb0zuhXMA1A84578Pp 0HvEfdhRNJKT/POAWCMnYju30pG6AAvvbu21vruznNNRWji/cmcvOO30KuiuabvQFOo1 wx0erOtd0S6EGF0x56DIYv8MM+qsJZZERO2QhQd5RFYRptTYZcRrh3N3xEmFbtf/aAGi DVig== X-Gm-Message-State: ACgBeo1h3LLeewt+mWhJ1HLG61gvOQQeCay7ycPVQVk2wDf/353Oem3e AaIIvmbvsGoGqjl+llENqgkulPVdtgY= X-Google-Smtp-Source: AA6agR7Nra03ZBRrpuwPuFCXQWjHEJPitoW4JIxgLux1CjcxnJdmZWLlhKoopGRGctCWLJD9uLTPGQ== X-Received: by 2002:a37:aac4:0:b0:6cb:d070:7842 with SMTP id t187-20020a37aac4000000b006cbd0707842mr19201938qke.386.1663099122426; Tue, 13 Sep 2022 12:58:42 -0700 (PDT) Received: from stbirv-lnx-3.igp.broadcom.net ([192.19.223.252]) by smtp.gmail.com with ESMTPSA id s11-20020a05620a29cb00b006b8e049cf08sm276305qkp.2.2022.09.13.12.58.39 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 13 Sep 2022 12:58:41 -0700 (PDT) From: Doug Berger To: Andrew Morton Cc: Jonathan Corbet , Rob Herring , Krzysztof Kozlowski , Frank Rowand , Mike Kravetz , Muchun Song , Mike Rapoport , Christoph Hellwig , Marek Szyprowski , Robin Murphy , Borislav Petkov , "Paul E. McKenney" , Neeraj Upadhyay , Randy Dunlap , Damien Le Moal , Doug Berger , Florian Fainelli , David Hildenbrand , Zi Yan , Oscar Salvador , Hari Bathini , Kees Cook , - , KOSAKI Motohiro , Mel Gorman , linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, devicetree@vger.kernel.org, linux-mm@kvack.org, iommu@lists.linux.dev Subject: [PATCH 13/21] mm/dmb: Introduce Designated Movable Blocks Date: Tue, 13 Sep 2022 12:55:00 -0700 Message-Id: <20220913195508.3511038-14-opendmb@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220913195508.3511038-1-opendmb@gmail.com> References: <20220913195508.3511038-1-opendmb@gmail.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Designated Movable Blocks are blocks of memory that are composed of one or more adjacent memblocks that have the MEMBLOCK_MOVABLE designation. These blocks must be reserved before receiving that designation and will be located in the ZONE_MOVABLE zone rather than any other zone that may span them. Signed-off-by: Doug Berger --- include/linux/dmb.h | 28 ++++++++++++++ mm/Kconfig | 12 ++++++ mm/Makefile | 1 + mm/dmb.c | 92 +++++++++++++++++++++++++++++++++++++++++++++ mm/memblock.c | 6 ++- mm/page_alloc.c | 84 ++++++++++++++++++++++++++++++++++------- 6 files changed, 209 insertions(+), 14 deletions(-) create mode 100644 include/linux/dmb.h create mode 100644 mm/dmb.c diff --git a/include/linux/dmb.h b/include/linux/dmb.h new file mode 100644 index 000000000000..eecc90e7f884 --- /dev/null +++ b/include/linux/dmb.h @@ -0,0 +1,28 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef __DMB_H__ +#define __DMB_H__ + +#include + +/* + * the buddy -- especially pageblock merging and alloc_contig_range() + * -- can deal with only some pageblocks of a higher-order page being + * MIGRATE_MOVABLE, we can use pageblock_nr_pages. + */ +#define DMB_MIN_ALIGNMENT_PAGES pageblock_nr_pages +#define DMB_MIN_ALIGNMENT_BYTES (PAGE_SIZE * DMB_MIN_ALIGNMENT_PAGES) + +enum { + DMB_DISJOINT =3D 0, + DMB_INTERSECTS, + DMB_MIXED, +}; + +struct dmb; + +extern int dmb_reserve(phys_addr_t base, phys_addr_t size, + struct dmb **res_dmb); +extern int dmb_intersects(unsigned long spfn, unsigned long epfn); +extern void dmb_init_region(struct memblock_region *region); + +#endif diff --git a/mm/Kconfig b/mm/Kconfig index 0331f1461f81..7739edde5d4d 100644 --- a/mm/Kconfig +++ b/mm/Kconfig @@ -868,6 +868,18 @@ config CMA_AREAS =20 If unsure, leave the default value "7" in UMA and "19" in NUMA. =20 +config DMB_COUNT + int "Maximum count of Designated Movable Blocks" + default 19 if NUMA + default 7 + help + Designated Movable Blocks are blocks of memory that can be used + by the page allocator exclusively for movable pages. They are + managed in ZONE_MOVABLE but may overlap with other zones. This + parameter sets the maximum number of DMBs in the system. + + If unsure, leave the default value "7" in UMA and "19" in NUMA. + config MEM_SOFT_DIRTY bool "Track memory changes" depends on CHECKPOINT_RESTORE && HAVE_ARCH_SOFT_DIRTY && PROC_FS diff --git a/mm/Makefile b/mm/Makefile index 9a564f836403..d0b469a494f2 100644 --- a/mm/Makefile +++ b/mm/Makefile @@ -67,6 +67,7 @@ obj-y +=3D page-alloc.o obj-y +=3D init-mm.o obj-y +=3D memblock.o obj-y +=3D $(memory-hotplug-y) +obj-y +=3D dmb.o =20 ifdef CONFIG_MMU obj-$(CONFIG_ADVISE_SYSCALLS) +=3D madvise.o diff --git a/mm/dmb.c b/mm/dmb.c new file mode 100644 index 000000000000..9d9fd31089d2 --- /dev/null +++ b/mm/dmb.c @@ -0,0 +1,92 @@ +// SPDX-License-Identifier: GPL-2.0+ +/* + * Designated Movable Block + */ + +#define pr_fmt(fmt) "dmb: " fmt + +#include + +struct dmb { + unsigned long start_pfn; + unsigned long end_pfn; + void *owner; +}; + +static struct dmb dmb_areas[CONFIG_DMB_COUNT]; +static unsigned int dmb_area_count; + +int __init dmb_reserve(phys_addr_t base, phys_addr_t size, + struct dmb **res_dmb) +{ + struct dmb *dmb; + + /* Sanity checks */ + if (dmb_area_count =3D=3D ARRAY_SIZE(dmb_areas)) { + pr_warn("Not enough slots for DMB reserved regions!\n"); + return -ENOSPC; + } + + if (!size || !memblock_is_region_reserved(base, size)) + return -EINVAL; + + /* ensure minimal alignment required by mm core */ + if (!IS_ALIGNED(base | size, DMB_MIN_ALIGNMENT_BYTES)) + return -EINVAL; + + /* + * Each reserved area must be initialised later, when more kernel + * subsystems (like slab allocator) are available. + */ + dmb =3D &dmb_areas[dmb_area_count++]; + + dmb->start_pfn =3D PFN_DOWN(base); + dmb->end_pfn =3D PFN_DOWN(base + size); + if (res_dmb) + *res_dmb =3D dmb; + + memblock_mark_movable(base, size); + return 0; +} + +int dmb_intersects(unsigned long spfn, unsigned long epfn) +{ + int i; + struct dmb *dmb; + + if (spfn >=3D epfn) + return DMB_DISJOINT; + + for (i =3D 0; i < dmb_area_count; i++) { + dmb =3D &dmb_areas[i]; + if (spfn >=3D dmb->end_pfn) + continue; + if (epfn <=3D dmb->start_pfn) + return DMB_DISJOINT; + if (spfn >=3D dmb->start_pfn && epfn <=3D dmb->end_pfn) + return DMB_INTERSECTS; + else + return DMB_MIXED; + } + + return DMB_DISJOINT; +} +EXPORT_SYMBOL(dmb_intersects); + +void __init dmb_init_region(struct memblock_region *region) +{ + unsigned long pfn; + int i; + + for (pfn =3D memblock_region_memory_base_pfn(region); + pfn < memblock_region_memory_end_pfn(region); + pfn +=3D pageblock_nr_pages) { + struct page *page =3D pfn_to_page(pfn); + + for (i =3D 0; i < pageblock_nr_pages; i++) + set_page_zone(page + i, ZONE_MOVABLE); + + /* free reserved pageblocks to page allocator */ + init_reserved_pageblock(page); + } +} diff --git a/mm/memblock.c b/mm/memblock.c index 5d6a210d98ec..9eb91acdeb75 100644 --- a/mm/memblock.c +++ b/mm/memblock.c @@ -16,6 +16,7 @@ #include #include #include +#include =20 #include #include @@ -2090,13 +2091,16 @@ static void __init memmap_init_reserved_pages(void) for_each_reserved_mem_range(i, &start, &end) reserve_bootmem_region(start, end); =20 - /* and also treat struct pages for the NOMAP regions as PageReserved */ for_each_mem_region(region) { + /* treat struct pages for the NOMAP regions as PageReserved */ if (memblock_is_nomap(region)) { start =3D region->base; end =3D start + region->size; reserve_bootmem_region(start, end); } + /* move Designated Movable Block pages to ZONE_MOVABLE */ + if (memblock_is_movable(region)) + dmb_init_region(region); } } =20 diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 1682d8815efa..e723094d1e1e 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -75,6 +75,7 @@ #include #include #include +#include #include #include #include @@ -433,6 +434,7 @@ static unsigned long required_kernelcore __initdata; static unsigned long required_kernelcore_percent __initdata; static unsigned long required_movablecore __initdata; static unsigned long required_movablecore_percent __initdata; +static unsigned long min_dmb_pfn[MAX_NUMNODES] __initdata; static unsigned long zone_movable_pfn[MAX_NUMNODES] __initdata; bool mirrored_kernelcore __initdata_memblock; =20 @@ -2165,7 +2167,7 @@ static int __init deferred_init_memmap(void *data) } zone_empty: /* Sanity check that the next zone really is unpopulated */ - WARN_ON(++zid < MAX_NR_ZONES && populated_zone(++zone)); + WARN_ON(++zid < ZONE_MOVABLE && populated_zone(++zone)); =20 pr_info("node %d deferred pages initialised in %ums\n", pgdat->node_id, jiffies_to_msecs(jiffies - start)); @@ -6899,6 +6901,10 @@ static void __init memmap_init_zone_range(struct zon= e *zone, unsigned long zone_end_pfn =3D zone_start_pfn + zone->spanned_pages; int nid =3D zone_to_nid(zone), zone_id =3D zone_idx(zone); =20 + /* Skip overlap of ZONE_MOVABLE */ + if (zone_id =3D=3D ZONE_MOVABLE && zone_start_pfn < *hole_pfn) + zone_start_pfn =3D *hole_pfn; + start_pfn =3D clamp(start_pfn, zone_start_pfn, zone_end_pfn); end_pfn =3D clamp(end_pfn, zone_start_pfn, zone_end_pfn); =20 @@ -7348,6 +7354,9 @@ static unsigned long __init zone_spanned_pages_in_nod= e(int nid, node_start_pfn, node_end_pfn, zone_start_pfn, zone_end_pfn); =20 + if (zone_type =3D=3D ZONE_MOVABLE && min_dmb_pfn[nid]) + *zone_start_pfn =3D min(*zone_start_pfn, min_dmb_pfn[nid]); + /* Check that this node has pages within the zone's required range */ if (*zone_end_pfn < node_start_pfn || *zone_start_pfn > node_end_pfn) return 0; @@ -7416,12 +7425,17 @@ static unsigned long __init zone_absent_pages_in_no= de(int nid, &zone_start_pfn, &zone_end_pfn); nr_absent =3D __absent_pages_in_range(nid, zone_start_pfn, zone_end_pfn); =20 + if (zone_type =3D=3D ZONE_MOVABLE && min_dmb_pfn[nid]) { + zone_start_pfn =3D min(zone_start_pfn, min_dmb_pfn[nid]); + nr_absent +=3D zone_movable_pfn[nid] - zone_start_pfn; + } + /* * ZONE_MOVABLE handling. - * Treat pages to be ZONE_MOVABLE in ZONE_NORMAL as absent pages + * Treat pages to be ZONE_MOVABLE in other zones as absent pages * and vice versa. */ - if (mirrored_kernelcore && zone_movable_pfn[nid]) { + if (zone_movable_pfn[nid]) { unsigned long start_pfn, end_pfn; struct memblock_region *r; =20 @@ -7431,6 +7445,21 @@ static unsigned long __init zone_absent_pages_in_nod= e(int nid, end_pfn =3D clamp(memblock_region_memory_end_pfn(r), zone_start_pfn, zone_end_pfn); =20 + if (memblock_is_movable(r)) { + if (zone_type !=3D ZONE_MOVABLE) { + nr_absent +=3D end_pfn - start_pfn; + continue; + } + + end_pfn =3D min(end_pfn, zone_movable_pfn[nid]); + if (start_pfn < zone_movable_pfn[nid]) + nr_absent -=3D end_pfn - start_pfn; + continue; + } + + if (!mirrored_kernelcore) + continue; + if (zone_type =3D=3D ZONE_MOVABLE && memblock_is_mirror(r)) nr_absent +=3D end_pfn - start_pfn; @@ -7450,6 +7479,15 @@ static void __init calculate_node_totalpages(struct = pglist_data *pgdat, { unsigned long realtotalpages =3D 0; enum zone_type i; + int nid =3D pgdat->node_id; + + /* + * If Designated Movable Blocks are defined on this node, ensure that + * zone_movable_pfn is also defined for this node. + */ + if (min_dmb_pfn[nid] && !zone_movable_pfn[nid]) + zone_movable_pfn[nid] =3D min(node_end_pfn, + arch_zone_highest_possible_pfn[movable_zone]); =20 for (i =3D 0; i < MAX_NR_ZONES; i++) { struct zone *zone =3D pgdat->node_zones + i; @@ -7457,12 +7495,12 @@ static void __init calculate_node_totalpages(struct= pglist_data *pgdat, unsigned long spanned, absent; unsigned long size, real_size; =20 - spanned =3D zone_spanned_pages_in_node(pgdat->node_id, i, + spanned =3D zone_spanned_pages_in_node(nid, i, node_start_pfn, node_end_pfn, &zone_start_pfn, &zone_end_pfn); - absent =3D zone_absent_pages_in_node(pgdat->node_id, i, + absent =3D zone_absent_pages_in_node(nid, i, node_start_pfn, node_end_pfn); =20 @@ -7922,15 +7960,23 @@ unsigned long __init find_min_pfn_with_active_regio= ns(void) static unsigned long __init early_calculate_totalpages(void) { unsigned long totalpages =3D 0; - unsigned long start_pfn, end_pfn; - int i, nid; + struct memblock_region *r; =20 - for_each_mem_pfn_range(i, MAX_NUMNODES, &start_pfn, &end_pfn, &nid) { - unsigned long pages =3D end_pfn - start_pfn; + for_each_mem_region(r) { + unsigned long start_pfn, end_pfn, pages; + int nid; =20 - totalpages +=3D pages; - if (pages) + nid =3D memblock_get_region_node(r); + start_pfn =3D memblock_region_memory_base_pfn(r); + end_pfn =3D memblock_region_memory_end_pfn(r); + + pages =3D end_pfn - start_pfn; + if (pages) { + totalpages +=3D pages; node_set_state(nid, N_MEMORY); + if (memblock_is_movable(r) && !min_dmb_pfn[nid]) + min_dmb_pfn[nid] =3D start_pfn; + } } return totalpages; } @@ -7943,7 +7989,7 @@ static unsigned long __init early_calculate_totalpage= s(void) */ static void __init find_zone_movable_pfns_for_nodes(void) { - int i, nid; + int nid; unsigned long usable_startpfn; unsigned long kernelcore_node, kernelcore_remaining; /* save the state before borrow the nodemask */ @@ -8071,13 +8117,24 @@ static void __init find_zone_movable_pfns_for_nodes= (void) kernelcore_remaining =3D kernelcore_node; =20 /* Go through each range of PFNs within this node */ - for_each_mem_pfn_range(i, nid, &start_pfn, &end_pfn, NULL) { + for_each_mem_region(r) { unsigned long size_pages; =20 + if (memblock_get_region_node(r) !=3D nid) + continue; + + start_pfn =3D memblock_region_memory_base_pfn(r); + end_pfn =3D memblock_region_memory_end_pfn(r); start_pfn =3D max(start_pfn, zone_movable_pfn[nid]); if (start_pfn >=3D end_pfn) continue; =20 + /* Skip over Designated Movable Blocks */ + if (memblock_is_movable(r)) { + zone_movable_pfn[nid] =3D end_pfn; + continue; + } + /* Account for what is only usable for kernelcore */ if (start_pfn < usable_startpfn) { unsigned long kernel_pages; @@ -8226,6 +8283,7 @@ void __init free_area_init(unsigned long *max_zone_pf= n) } =20 /* Find the PFNs that ZONE_MOVABLE begins at in each node */ + memset(min_dmb_pfn, 0, sizeof(min_dmb_pfn)); memset(zone_movable_pfn, 0, sizeof(zone_movable_pfn)); find_zone_movable_pfns_for_nodes(); =20 --=20 2.25.1 From nobody Tue Dec 16 03:00:38 2025 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id C5D4CC6FA8A for ; Tue, 13 Sep 2022 19:59:59 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229482AbiIMT7z (ORCPT ); Tue, 13 Sep 2022 15:59:55 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46916 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230058AbiIMT7E (ORCPT ); Tue, 13 Sep 2022 15:59:04 -0400 Received: from mail-qt1-x82e.google.com (mail-qt1-x82e.google.com [IPv6:2607:f8b0:4864:20::82e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6CBF6786C3; Tue, 13 Sep 2022 12:58:47 -0700 (PDT) Received: by mail-qt1-x82e.google.com with SMTP id c11so9603342qtw.8; Tue, 13 Sep 2022 12:58:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date; bh=Q4ZU5cexm9Q08qLiEijmmMTjwuY3GlJ1Cn90C3bzJMU=; b=DPtm4ok3PSf82mq7JMYxEELlSbe2Rh/g3fQ51ooEmzdbhvAmX7bq1p/PDmYadgyeC1 zi9y/jcGd3D1R6ZJkZ/FQsWIwfFLD+Pb0MJITbf3fgUqnK83ONtjfJ+HMIIxaq6Be8T0 7yW02a5Gpkn6GV6C9TtbKwXcBYYQfckEhhDlRw3kQGNLccCEAsysOi1687eQCVwa1Snz UzUeXpguUoApWzimdM3XamY7zoYoXwniOBrFXnFh1C31eUhw4i9vT9n/3okZ8ZMJuUbE x2Q9CpCURykZLJagLzDO5v/o6/9ctT6KeQ6BSDQrVAvGz6WE7hB0JQGNvnw+QEyD9e4R S75w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date; bh=Q4ZU5cexm9Q08qLiEijmmMTjwuY3GlJ1Cn90C3bzJMU=; b=cVm3jcqIl8VrX5pxqWFz1PosmEr89rAV3mFawA6Wzvbu7hpKrsszOFHhK7seocYhpB yuRt0vq9dmseifi5j1L9IAIynCuNTuc3fIthb3pCje4U5HSQZJAOrr8ttXTieu0xlykV oL/jFMtvpfFyoxAVw5AypICxzHX3hw5Tbzwhn9mmedfQbM8KzcWd8dLKZN4X0hfZjxIT KGoEXtFY+ZCHAEhHah43UJWd5IM+wy5SUm8Riz0YVRJMPdv+h2Hwbu+agGpbvZN/pnED SbkLrHX0Zh6IU0Lcs+8a0NJHoDtc2hZ4CM7rR9Qs8W6oacxG05WDIaBIWT3qMJeL+h8Z Seqw== X-Gm-Message-State: ACgBeo0K2zcoPOw+bF5Q6YRE5KxJEWI0D1wA8RTt+HORRGTKmtXAY1Xl SEIqkpmzG+izjsc2KVypZYg= X-Google-Smtp-Source: AA6agR79LKnSKwsuEjaHOZbqMf9sDJeff1s7ltAoeOd0yWLYcv6FI23B3yp+SehZLzBtT5Q7988TlQ== X-Received: by 2002:a05:622a:174b:b0:343:1fc:14d8 with SMTP id l11-20020a05622a174b00b0034301fc14d8mr29317695qtk.579.1663099126048; Tue, 13 Sep 2022 12:58:46 -0700 (PDT) Received: from stbirv-lnx-3.igp.broadcom.net ([192.19.223.252]) by smtp.gmail.com with ESMTPSA id s11-20020a05620a29cb00b006b8e049cf08sm276305qkp.2.2022.09.13.12.58.42 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 13 Sep 2022 12:58:45 -0700 (PDT) From: Doug Berger To: Andrew Morton Cc: Jonathan Corbet , Rob Herring , Krzysztof Kozlowski , Frank Rowand , Mike Kravetz , Muchun Song , Mike Rapoport , Christoph Hellwig , Marek Szyprowski , Robin Murphy , Borislav Petkov , "Paul E. McKenney" , Neeraj Upadhyay , Randy Dunlap , Damien Le Moal , Doug Berger , Florian Fainelli , David Hildenbrand , Zi Yan , Oscar Salvador , Hari Bathini , Kees Cook , - , KOSAKI Motohiro , Mel Gorman , linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, devicetree@vger.kernel.org, linux-mm@kvack.org, iommu@lists.linux.dev Subject: [PATCH 14/21] mm/page_alloc: make alloc_contig_pages DMB aware Date: Tue, 13 Sep 2022 12:55:01 -0700 Message-Id: <20220913195508.3511038-15-opendmb@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220913195508.3511038-1-opendmb@gmail.com> References: <20220913195508.3511038-1-opendmb@gmail.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Designated Movable Blocks are skipped when attempting to allocate contiguous pages. Doing per page validation across all spanned pages within a zone can be extra inefficient when Designated Movable Blocks create large overlaps between zones. Use dmb_intersects() within pfn_range_valid_contig as an early check to signal the range is not valid. The zone_movable_pfn array which represents the start of non- overlapped ZONE_MOVABLE on the node is now preserved to be used at runtime to skip over any DMB-only portion of the zone. Signed-off-by: Doug Berger --- mm/page_alloc.c | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index e723094d1e1e..69753cc51e19 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -435,7 +435,7 @@ static unsigned long required_kernelcore_percent __init= data; static unsigned long required_movablecore __initdata; static unsigned long required_movablecore_percent __initdata; static unsigned long min_dmb_pfn[MAX_NUMNODES] __initdata; -static unsigned long zone_movable_pfn[MAX_NUMNODES] __initdata; +static unsigned long zone_movable_pfn[MAX_NUMNODES]; bool mirrored_kernelcore __initdata_memblock; =20 /* movable_zone is the "real" zone pages in ZONE_MOVABLE are taken from */ @@ -9370,6 +9370,9 @@ static bool pfn_range_valid_contig(struct zone *z, un= signed long start_pfn, unsigned long i, end_pfn =3D start_pfn + nr_pages; struct page *page; =20 + if (dmb_intersects(start_pfn, end_pfn)) + return false; + for (i =3D start_pfn; i < end_pfn; i++) { page =3D pfn_to_online_page(i); if (!page) @@ -9426,7 +9429,10 @@ struct page *alloc_contig_pages(unsigned long nr_pag= es, gfp_t gfp_mask, gfp_zone(gfp_mask), nodemask) { spin_lock_irqsave(&zone->lock, flags); =20 - pfn =3D ALIGN(zone->zone_start_pfn, nr_pages); + if (zone_idx(zone) =3D=3D ZONE_MOVABLE && zone_movable_pfn[nid]) + pfn =3D ALIGN(zone_movable_pfn[nid], nr_pages); + else + pfn =3D ALIGN(zone->zone_start_pfn, nr_pages); while (zone_spans_last_pfn(zone, pfn, nr_pages)) { if (pfn_range_valid_contig(zone, pfn, nr_pages)) { /* --=20 2.25.1 From nobody Tue Dec 16 03:00:38 2025 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 97C84C6FA82 for ; Tue, 13 Sep 2022 20:00:04 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230190AbiIMUAC (ORCPT ); Tue, 13 Sep 2022 16:00:02 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47098 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230079AbiIMT7I (ORCPT ); Tue, 13 Sep 2022 15:59:08 -0400 Received: from mail-qk1-x733.google.com (mail-qk1-x733.google.com [IPv6:2607:f8b0:4864:20::733]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B56A1786FF; Tue, 13 Sep 2022 12:58:50 -0700 (PDT) Received: by mail-qk1-x733.google.com with SMTP id q11so6264314qkc.12; Tue, 13 Sep 2022 12:58:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date; bh=ed+iX/nv/XMEFygJ1SlfRX9x+uzs7Q6SqUw+PcSFA+Y=; b=kTC2V8I0KbKQgYVpiMXWXobqHukHN5wc9G7pyTq0OXIlmxgzs1DdLdIuMurl376JOh TI+sLZOsyYvKvgQV1mtsgOKDhgkCHY9vgTN75V+209rBR7AYRtArLxFXLQfDpntS8+77 1VJuC85XaAr9lDOJi4JPuZBYlxos4LmgTFUlBZzjlAHRPleIyKNPrsw1Q/TybKJ6LhLy P7jZiatOzHC+C4lsh8jLuUsBJv4lBg9GwtABHfhEamUAOb2Eu3rURCtfJTb4hzSEiYzf Mf68rOnlVxAV76vW2E0E1N4Dtdqb6V3ShGLPb0HbK61ur+zHBF+wQNqRrMeYo3aOEqsI sQUw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date; bh=ed+iX/nv/XMEFygJ1SlfRX9x+uzs7Q6SqUw+PcSFA+Y=; b=T20Otj5Ly/TC8a+VsTAB82tQyD56hLYzJ8A4iZBRUOyYuwJeTDyvVkJfEHZchkr9o6 ovTG9933VUDtGHgYWjS02Q2PzvykKjGg+0Xcf5ibS8YFftj7Lvdo8mI8IyeqEdEQQqmD eOHN3hJvGlb59TdFRhlOU2NPZKO8xO9Rgg7k3BHXEV9VLL1gspkQTrZ8u7QmR3JApcXq XoGDa/otK7QtsgqNQ1tNSYzu61A3bg76kCI76EIg2UuKYl+NfACgAVj5jBaZ1gSY8yTj hZ8telwUlJk6v3baBvwa27UpL1fVEfRFib1tJe7DaAJNOEsRtvf4NYmtjFLdZfckWgpy YmBQ== X-Gm-Message-State: ACgBeo2WYpA/Dl/zJgmMXFGqBPH6+kwbgNEU7VqoDjxR5ATWpXEDnboa HFc2tVUeLx1KXCIVCtXzQ6w= X-Google-Smtp-Source: AA6agR6oL/h20WlhqK4D/I9yr1owRo+hnv02mOT7OtmgSO2hWSXNGrz7MzYBNL87UH+0Ca4e8ZJtlg== X-Received: by 2002:a05:620a:2956:b0:6ce:60f5:d887 with SMTP id n22-20020a05620a295600b006ce60f5d887mr4614697qkp.303.1663099129650; Tue, 13 Sep 2022 12:58:49 -0700 (PDT) Received: from stbirv-lnx-3.igp.broadcom.net ([192.19.223.252]) by smtp.gmail.com with ESMTPSA id s11-20020a05620a29cb00b006b8e049cf08sm276305qkp.2.2022.09.13.12.58.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 13 Sep 2022 12:58:49 -0700 (PDT) From: Doug Berger To: Andrew Morton Cc: Jonathan Corbet , Rob Herring , Krzysztof Kozlowski , Frank Rowand , Mike Kravetz , Muchun Song , Mike Rapoport , Christoph Hellwig , Marek Szyprowski , Robin Murphy , Borislav Petkov , "Paul E. McKenney" , Neeraj Upadhyay , Randy Dunlap , Damien Le Moal , Doug Berger , Florian Fainelli , David Hildenbrand , Zi Yan , Oscar Salvador , Hari Bathini , Kees Cook , - , KOSAKI Motohiro , Mel Gorman , linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, devicetree@vger.kernel.org, linux-mm@kvack.org, iommu@lists.linux.dev Subject: [PATCH 15/21] mm/page_alloc: allow base for movablecore Date: Tue, 13 Sep 2022 12:55:02 -0700 Message-Id: <20220913195508.3511038-16-opendmb@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220913195508.3511038-1-opendmb@gmail.com> References: <20220913195508.3511038-1-opendmb@gmail.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" A Designated Movable Block can be created by including the base address of the block when specifying a movablecore range on the kernel command line. Signed-off-by: Doug Berger --- .../admin-guide/kernel-parameters.txt | 14 ++++++- mm/page_alloc.c | 38 ++++++++++++++++--- 2 files changed, 45 insertions(+), 7 deletions(-) diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentatio= n/admin-guide/kernel-parameters.txt index 426fa892d311..8141fac7c7cb 100644 --- a/Documentation/admin-guide/kernel-parameters.txt +++ b/Documentation/admin-guide/kernel-parameters.txt @@ -3312,7 +3312,7 @@ reporting absolute coordinates, such as tablets =20 movablecore=3D [KNL,X86,IA-64,PPC] - Format: nn[KMGTPE] | nn% + Format: nn[KMGTPE] | nn[KMGTPE]@ss[KMGTPE] | nn% This parameter is the complement to kernelcore=3D, it specifies the amount of memory used for migratable allocations. If both kernelcore and movablecore is @@ -3322,6 +3322,18 @@ that the amount of memory usable for all allocations is not too small. =20 + If @ss[KMGTPE] is included, memory within the region + from ss to ss+nn will be designated as a movable block + and included in ZONE_MOVABLE. Designated Movable Blocks + must be aligned to pageblock_order. Designated Movable + Blocks take priority over values of kernelcore=3D and are + considered part of any memory specified by more general + movablecore=3D values. + Multiple Designated Movable Blocks may be specified, + comma delimited. + Example: + movablecore=3D100M@2G,100M@3G,1G@1024G + movable_node [KNL] Boot-time switch to make hotplugable memory NUMA nodes to be movable. This means that the memory of such nodes will be usable only for movable diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 69753cc51e19..e38dd1b32771 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -8370,9 +8370,9 @@ void __init free_area_init(unsigned long *max_zone_pf= n) } =20 static int __init cmdline_parse_core(char *p, unsigned long *core, - unsigned long *percent) + unsigned long *percent, bool movable) { - unsigned long long coremem; + unsigned long long coremem, address; char *endptr; =20 if (!p) @@ -8387,6 +8387,17 @@ static int __init cmdline_parse_core(char *p, unsign= ed long *core, *percent =3D coremem; } else { coremem =3D memparse(p, &p); + if (movable && *p =3D=3D '@') { + address =3D memparse(++p, &p); + if (*p !=3D '\0' || + !memblock_is_region_memory(address, coremem) || + memblock_is_region_reserved(address, coremem)) + return -EINVAL; + memblock_reserve(address, coremem); + return dmb_reserve(address, coremem, NULL); + } else if (*p !=3D '\0') { + return -EINVAL; + } /* Paranoid check that UL is enough for the coremem value */ WARN_ON((coremem >> PAGE_SHIFT) > ULONG_MAX); =20 @@ -8409,17 +8420,32 @@ static int __init cmdline_parse_kernelcore(char *p) } =20 return cmdline_parse_core(p, &required_kernelcore, - &required_kernelcore_percent); + &required_kernelcore_percent, false); } =20 /* * movablecore=3Dsize sets the amount of memory for use for allocations th= at - * can be reclaimed or migrated. + * can be reclaimed or migrated. movablecore=3Dsize@base defines a Designa= ted + * Movable Block. */ static int __init cmdline_parse_movablecore(char *p) { - return cmdline_parse_core(p, &required_movablecore, - &required_movablecore_percent); + int ret =3D -EINVAL; + + while (p) { + char *k =3D strchr(p, ','); + + if (k) + *k++ =3D 0; + + ret =3D cmdline_parse_core(p, &required_movablecore, + &required_movablecore_percent, true); + if (ret) + break; + p =3D k; + } + + return ret; } =20 early_param("kernelcore", cmdline_parse_kernelcore); --=20 2.25.1 From nobody Tue Dec 16 03:00:38 2025 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id C8641C54EE9 for ; Tue, 13 Sep 2022 20:00:13 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229883AbiIMUAK (ORCPT ); Tue, 13 Sep 2022 16:00:10 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46960 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229743AbiIMT7R (ORCPT ); Tue, 13 Sep 2022 15:59:17 -0400 Received: from mail-qk1-x735.google.com (mail-qk1-x735.google.com [IPv6:2607:f8b0:4864:20::735]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 5D55678BF6; Tue, 13 Sep 2022 12:58:54 -0700 (PDT) Received: by mail-qk1-x735.google.com with SMTP id d17so8539481qko.13; Tue, 13 Sep 2022 12:58:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date; bh=0ytqVsMPjuVvTMmUHQDgw++YWpDMZOL8Z97lPlE2FVY=; b=io9Nt4/RcZ2YZMphnT4cANVUYCQB6U7tpDHWWA0ymeRAMaiACQe59CDdxV3fKa+Hsn xNls20lsFyfw6rNS33zM5vtcFodTEOrCxgBbNhbWX8T1mEeMX+VwfC33CIm+xfP758LM zM7bYSmAZtJOGaBc2/pCs2H6YwEWsP5kO079m4wM+f6s/zfDbHxnmlkNGrxVR7sLq2J2 vi+KzltoxbcgQVb6NYjPdS8rs20FudQRcZ6AiEV/jd58a8RhxFomEIPJSEoUVrySG9CU fGCpRyRxdV4msiy4sXSvtH2LCG/1JOnoJrLiDkFSx8Zr9e7tZ4S5+YfhjAOUxZaXHtxy O3kw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date; bh=0ytqVsMPjuVvTMmUHQDgw++YWpDMZOL8Z97lPlE2FVY=; b=ZQs7vf9Islv6Wbory1SqXuxXCb2rDP4tYv2q0p7RXJaDcwPblba2PqaDF/QkC5Q5aU ldZ5JsmPRWL6PEprxVb/gLa6nE498IBUU6Qy58fFSQLaNbINQ7ljgyOUQCUbTKg5QPaL felKmzP9PAglUrKUOryFmpnkAkmy+YzGZuoV4P5KIEBngbVlRnQIiz6QAERn6b3u6Gnb TLZdWOHhXga0Zr8DC3MQU1sU9x5wpscl6nRKM619Hq5SdPZ33kK0xEy0+ykBSFtlRZHJ 6SZdkDzwx3ve0qHpmSyuQ1Bp1xd5zvSaXf0tpz4N2n26ReYN7Cje8U1Zq8kdcua/EXis rCIA== X-Gm-Message-State: ACgBeo2kxXrTsxSI/YzgtnPtx8AJIce6cBZkX9Ct4xJvGQslSpRVRQl0 b7gB1WqYbHMBxwXgvXn3o9k= X-Google-Smtp-Source: AA6agR6d/FiwNOHoDQ8OKNxH916Sryp0qXJllILzqqegobMzkZRMGgPUcF2Qjw5zlXSo5I0Sdx+6eA== X-Received: by 2002:a05:620a:4407:b0:6cb:e111:32c2 with SMTP id v7-20020a05620a440700b006cbe11132c2mr18923399qkp.415.1663099133214; Tue, 13 Sep 2022 12:58:53 -0700 (PDT) Received: from stbirv-lnx-3.igp.broadcom.net ([192.19.223.252]) by smtp.gmail.com with ESMTPSA id s11-20020a05620a29cb00b006b8e049cf08sm276305qkp.2.2022.09.13.12.58.49 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 13 Sep 2022 12:58:52 -0700 (PDT) From: Doug Berger To: Andrew Morton Cc: Jonathan Corbet , Rob Herring , Krzysztof Kozlowski , Frank Rowand , Mike Kravetz , Muchun Song , Mike Rapoport , Christoph Hellwig , Marek Szyprowski , Robin Murphy , Borislav Petkov , "Paul E. McKenney" , Neeraj Upadhyay , Randy Dunlap , Damien Le Moal , Doug Berger , Florian Fainelli , David Hildenbrand , Zi Yan , Oscar Salvador , Hari Bathini , Kees Cook , - , KOSAKI Motohiro , Mel Gorman , linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, devicetree@vger.kernel.org, linux-mm@kvack.org, iommu@lists.linux.dev Subject: [PATCH 16/21] dt-bindings: reserved-memory: introduce designated-movable-block Date: Tue, 13 Sep 2022 12:55:03 -0700 Message-Id: <20220913195508.3511038-17-opendmb@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220913195508.3511038-1-opendmb@gmail.com> References: <20220913195508.3511038-1-opendmb@gmail.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Introduce designated-movable-block.yaml to document the devicetree binding for Designated Movable Block children of the reserved-memory node. Signed-off-by: Doug Berger --- .../designated-movable-block.yaml | 51 +++++++++++++++++++ 1 file changed, 51 insertions(+) create mode 100644 Documentation/devicetree/bindings/reserved-memory/desig= nated-movable-block.yaml diff --git a/Documentation/devicetree/bindings/reserved-memory/designated-m= ovable-block.yaml b/Documentation/devicetree/bindings/reserved-memory/desig= nated-movable-block.yaml new file mode 100644 index 000000000000..42f846069a2e --- /dev/null +++ b/Documentation/devicetree/bindings/reserved-memory/designated-movable-= block.yaml @@ -0,0 +1,51 @@ +# SPDX-License-Identifier: (GPL-2.0 OR BSD-2-Clause) +%YAML 1.2 +--- +$id: http://devicetree.org/schemas/reserved-memory/designated-movable-bloc= k.yaml# +$schema: http://devicetree.org/meta-schemas/core.yaml# + +title: /reserved-memory Designated Movable Block node binding + +maintainers: + - devicetree-spec@vger.kernel.org + +allOf: + - $ref: "reserved-memory.yaml" + +properties: + compatible: + const: designated-movable-block + description: + This indicates a region of memory meant to be placed into + ZONE_MOVABLE. + +unevaluatedProperties: false + +required: + - compatible + - reusable + +examples: + - | + reserved-memory { + #address-cells =3D <0x2>; + #size-cells =3D <0x2>; + + DMB0@10800000 { + compatible =3D "designated-movable-block"; + reusable; + reg =3D <0x0 0x10800000 0x0 0x2d800000>; + }; + + DMB1@40000000 { + compatible =3D "designated-movable-block"; + reusable; + reg =3D <0x0 0x40000000 0x0 0x30000000>; + }; + + DMB2@80000000 { + compatible =3D "designated-movable-block"; + reusable; + reg =3D <0x0 0x80000000 0x0 0x2fc00000>; + }; + }; --=20 2.25.1 From nobody Tue Dec 16 03:00:38 2025 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id A0C5EC54EE9 for ; Tue, 13 Sep 2022 20:00:19 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229572AbiIMUAQ (ORCPT ); Tue, 13 Sep 2022 16:00:16 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47254 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230029AbiIMT7T (ORCPT ); Tue, 13 Sep 2022 15:59:19 -0400 Received: from mail-qk1-x733.google.com (mail-qk1-x733.google.com [IPv6:2607:f8b0:4864:20::733]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C8E71792D0; Tue, 13 Sep 2022 12:58:57 -0700 (PDT) Received: by mail-qk1-x733.google.com with SMTP id i3so4616430qkl.3; Tue, 13 Sep 2022 12:58:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date; bh=U4jOrSu6bqJBytWJghR6t0ljWrrZGLSB9ph/v+e9V8I=; b=D/mTJk6kZw64eyUhHPaSFfxR9Msa5UBxyOTlcsmCwhUVjiAhTk7oqnxftuemkUJs0o e/3ecILYiDZtf1ZduEhchjc87lM/UzBWANhyHjDCeSCCZZ+OsXTubHr/HBHsItOzKdH7 p/GQO/8nQMFRqeejgZsBOVzCwSXLRfYO2jhkP7CDsujDhgqd9989s5l2qAcx+CNYHM2D 3FzlZrQmr76F2aSmz5wNYnpRQxzHx962ONlutGAvPr/+dXcOvxUiHsk1k1ee7yaWbHVI 2LCJzHzqV9IGkV4gb1i+myYCzvATpnUG8sQPbUiE3I3lJUeSHhrVFH3mITfi1tyJU0b1 X7Mw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date; bh=U4jOrSu6bqJBytWJghR6t0ljWrrZGLSB9ph/v+e9V8I=; b=6tGerPA+8CLTaSTurXxfHH11caWZcpKkMd0d7u6ZQznwAYm2Lqstgix8JAC9XY2Ias PjBfiEnd2F57RC8cTQvztOyRLGzn+KN25OIc5glV8l8sZ8aCuSXM1z+3FOLa/rVNr8Z2 ZQ4MUlcCZoEf8JbHCUi+QirGDEZppS3zHNO/x7BcodkzQ+Of+5hOZONZxILgYLq14eOU Nhle7ekgTmWIP57OvyhCMkVxLVhoouA+fbxnT68XH9yll0MK7Lr+BRC9os5aDaCGrNpe QnkIi9LK8nZPqffy6uKUYMa7+l6fn8Q4roBC3aqYDbb4+ZDet5EPhm6DibH1EgAJoQCL bsGw== X-Gm-Message-State: ACgBeo0u1sLYwclovlWZGsAQhjYhUGpEZ5wc506+TDJwKe4CSJ8D2sva 2jos7PqMzIl8gLZjY6e3pRo= X-Google-Smtp-Source: AA6agR42tNhbqU65weBklFefGDPSfeY7AMPrK26FTuyeT6UMnlp5fPm7LJh/3FGN4dhzcFbZNJ8Esw== X-Received: by 2002:a05:620a:2452:b0:6bb:d8ba:ca65 with SMTP id h18-20020a05620a245200b006bbd8baca65mr24199159qkn.263.1663099136924; Tue, 13 Sep 2022 12:58:56 -0700 (PDT) Received: from stbirv-lnx-3.igp.broadcom.net ([192.19.223.252]) by smtp.gmail.com with ESMTPSA id s11-20020a05620a29cb00b006b8e049cf08sm276305qkp.2.2022.09.13.12.58.53 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 13 Sep 2022 12:58:56 -0700 (PDT) From: Doug Berger To: Andrew Morton Cc: Jonathan Corbet , Rob Herring , Krzysztof Kozlowski , Frank Rowand , Mike Kravetz , Muchun Song , Mike Rapoport , Christoph Hellwig , Marek Szyprowski , Robin Murphy , Borislav Petkov , "Paul E. McKenney" , Neeraj Upadhyay , Randy Dunlap , Damien Le Moal , Doug Berger , Florian Fainelli , David Hildenbrand , Zi Yan , Oscar Salvador , Hari Bathini , Kees Cook , - , KOSAKI Motohiro , Mel Gorman , linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, devicetree@vger.kernel.org, linux-mm@kvack.org, iommu@lists.linux.dev Subject: [PATCH 17/21] mm/dmb: introduce rmem designated-movable-block Date: Tue, 13 Sep 2022 12:55:04 -0700 Message-Id: <20220913195508.3511038-18-opendmb@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220913195508.3511038-1-opendmb@gmail.com> References: <20220913195508.3511038-1-opendmb@gmail.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" This commit allows Designated Movable Blocks to be created by including reserved-memory child nodes in the device tree with the "designated-movable-block" compatible string. Signed-off-by: Doug Berger --- drivers/of/of_reserved_mem.c | 15 ++++++--- mm/dmb.c | 64 ++++++++++++++++++++++++++++++++++++ 2 files changed, 74 insertions(+), 5 deletions(-) diff --git a/drivers/of/of_reserved_mem.c b/drivers/of/of_reserved_mem.c index 65f3b02a0e4e..0eb9e8898d7b 100644 --- a/drivers/of/of_reserved_mem.c +++ b/drivers/of/of_reserved_mem.c @@ -23,6 +23,7 @@ #include #include #include +#include =20 #include "of_private.h" =20 @@ -113,12 +114,16 @@ static int __init __reserved_mem_alloc_size(unsigned = long node, =20 nomap =3D of_get_flat_dt_prop(node, "no-map", NULL) !=3D NULL; =20 - /* Need adjust the alignment to satisfy the CMA requirement */ - if (IS_ENABLED(CONFIG_CMA) - && of_flat_dt_is_compatible(node, "shared-dma-pool") - && of_get_flat_dt_prop(node, "reusable", NULL) - && !nomap) + if (of_flat_dt_is_compatible(node, "designated-movable-block")) { + /* Need adjust the alignment to satisfy the DMB requirement */ + align =3D max_t(phys_addr_t, align, DMB_MIN_ALIGNMENT_BYTES); + } else if (IS_ENABLED(CONFIG_CMA) + && of_flat_dt_is_compatible(node, "shared-dma-pool") + && of_get_flat_dt_prop(node, "reusable", NULL) + && !nomap) { + /* Need adjust the alignment to satisfy the CMA requirement */ align =3D max_t(phys_addr_t, align, CMA_MIN_ALIGNMENT_BYTES); + } =20 prop =3D of_get_flat_dt_prop(node, "alloc-ranges", &len); if (prop) { diff --git a/mm/dmb.c b/mm/dmb.c index 9d9fd31089d2..8132d18542a0 100644 --- a/mm/dmb.c +++ b/mm/dmb.c @@ -90,3 +90,67 @@ void __init dmb_init_region(struct memblock_region *regi= on) init_reserved_pageblock(page); } } + +/* + * Support for reserved memory regions defined in device tree + */ +#ifdef CONFIG_OF_RESERVED_MEM +#include +#include +#include + +#undef pr_fmt +#define pr_fmt(fmt) fmt + +static int rmem_dmb_device_init(struct reserved_mem *rmem, struct device *= dev) +{ + struct dmb *dmb; + + dmb =3D (struct dmb *)rmem->priv; + if (dmb->owner) + return -EBUSY; + + dmb->owner =3D dev; + return 0; +} + +static void rmem_dmb_device_release(struct reserved_mem *rmem, + struct device *dev) +{ + struct dmb *dmb; + + dmb =3D (struct dmb *)rmem->priv; + if (dmb->owner =3D=3D (void *)dev) + dmb->owner =3D NULL; +} + +static const struct reserved_mem_ops rmem_dmb_ops =3D { + .device_init =3D rmem_dmb_device_init, + .device_release =3D rmem_dmb_device_release, +}; + +static int __init rmem_dmb_setup(struct reserved_mem *rmem) +{ + unsigned long node =3D rmem->fdt_node; + struct dmb *dmb; + int err; + + if (!of_get_flat_dt_prop(node, "reusable", NULL) || + of_get_flat_dt_prop(node, "no-map", NULL)) + return -EINVAL; + + err =3D dmb_reserve(rmem->base, rmem->size, &dmb); + if (err) { + pr_err("Reserved memory: unable to setup DMB region\n"); + return err; + } + + rmem->priv =3D dmb; + rmem->ops =3D &rmem_dmb_ops; + pr_info("Reserved memory: created DMB at %pa, size %ld MiB\n", + &rmem->base, (unsigned long)rmem->size / SZ_1M); + + return 0; +} +RESERVEDMEM_OF_DECLARE(dmb, "designated-movable-block", rmem_dmb_setup); +#endif --=20 2.25.1 From nobody Tue Dec 16 03:00:38 2025 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id DAF84ECAAD8 for ; Tue, 13 Sep 2022 20:00:36 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230136AbiIMUAe (ORCPT ); Tue, 13 Sep 2022 16:00:34 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47030 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230135AbiIMT7l (ORCPT ); Tue, 13 Sep 2022 15:59:41 -0400 Received: from mail-qt1-x829.google.com (mail-qt1-x829.google.com [IPv6:2607:f8b0:4864:20::829]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 198A578223; Tue, 13 Sep 2022 12:59:01 -0700 (PDT) Received: by mail-qt1-x829.google.com with SMTP id w2so5985587qtv.9; Tue, 13 Sep 2022 12:59:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date; bh=fHU5ZUx5v+GumCekHv98dt5HtboTaZeeu0Gp9AlVmvA=; b=ZkQM2Xi16fac9Kxsd6JfpJUyj9dFTpL2a9fMxPQYb5tsP1AGTq9LrS3GQ23PdeUUAq KiAJoUd0eFmdtlIe31jI3xWYF3lNu2z6iPVbkhYktSoNmKAD7mOQ11g6lhzdx8PeDKqG lPt1LWxw7/FsEvYMWeWC/86MaywQjplVNZrIXdmUyLe5Nr4lwjgNum55ZdLf4luz/yKw pKJIZBfhsG2I2a6u/v5b8+uct94R+RrovFIPHm9V4i65FXFauBq/I0QO/UcUPxNQEMMR iJaQjUMSz8SAuIngfV/EGkXgYLEYcOdGnoAVuWBboXNtLXNpt0Nqru8cUMiF0ZW7269J 6y+g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date; bh=fHU5ZUx5v+GumCekHv98dt5HtboTaZeeu0Gp9AlVmvA=; b=TLiGnzbzNnpVtdmLnUhd0AFA8QWa+PBw87tuV3W/LvTzOLgBhsN1JIt3QFJlFHy4w/ Jo2cVSqDq426f/upNG84xxTH4silnHaN24IeqkYzivx6NDmgaRhW8y/2QgahAI/DGvQI ks/beEcY4PevVKLnCbEy9vROa0FcNUpZ734bb58qS/yT/EqR2RIZbKpstA1qtZW37O+b cAhxo7seSqYK0pEihEwWLs5Y5noVu8Tzu3Z8rXASG9EJoXldExjLRAU/k0SynMd/RkiV LTaFFQyJTQ+XRO0Y9+0oWcFk2bp8dg8BXba9D/Xx7XlzcHh3qzSrftHooMluxHtalzxI L9/g== X-Gm-Message-State: ACgBeo3zt1eF5PUrQqYYXF4hrJS+5Avj7KHMqzGs0L82OcJAomdOBh+I u7Q1J0A+gXeck3OzfwsG3LI= X-Google-Smtp-Source: AA6agR7yxxGK0L7lb4UY6ElTAIAR0o3jzl9zqXW4dIwQkZBtshIlrdqdJVz3avbMktN2rc9zUTYXIg== X-Received: by 2002:ac8:570a:0:b0:343:73af:7cd8 with SMTP id 10-20020ac8570a000000b0034373af7cd8mr30203309qtw.16.1663099140678; Tue, 13 Sep 2022 12:59:00 -0700 (PDT) Received: from stbirv-lnx-3.igp.broadcom.net ([192.19.223.252]) by smtp.gmail.com with ESMTPSA id s11-20020a05620a29cb00b006b8e049cf08sm276305qkp.2.2022.09.13.12.58.57 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 13 Sep 2022 12:59:00 -0700 (PDT) From: Doug Berger To: Andrew Morton Cc: Jonathan Corbet , Rob Herring , Krzysztof Kozlowski , Frank Rowand , Mike Kravetz , Muchun Song , Mike Rapoport , Christoph Hellwig , Marek Szyprowski , Robin Murphy , Borislav Petkov , "Paul E. McKenney" , Neeraj Upadhyay , Randy Dunlap , Damien Le Moal , Doug Berger , Florian Fainelli , David Hildenbrand , Zi Yan , Oscar Salvador , Hari Bathini , Kees Cook , - , KOSAKI Motohiro , Mel Gorman , linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, devicetree@vger.kernel.org, linux-mm@kvack.org, iommu@lists.linux.dev Subject: [PATCH 18/21] mm/cma: support CMA in Designated Movable Blocks Date: Tue, 13 Sep 2022 12:55:05 -0700 Message-Id: <20220913195508.3511038-19-opendmb@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220913195508.3511038-1-opendmb@gmail.com> References: <20220913195508.3511038-1-opendmb@gmail.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" This commit allows for different page allocator handling for CMA areas that are within Designated Movable Blocks. Specifically, the pageblocks are allowed to remain migratetype MIGRATE_MOVABLE to allow more aggressive utilization by the page allocator. This also means that the page allocator should not consider these pages as part of the nr_free_cma metric it uses for managing MIGRATE_CMA type pageblocks. This leads to the decision to remove these areas from the CmaTotal metrics after initialization to avoid confusion. Signed-off-by: Doug Berger Reported-by: kernel test robot --- include/linux/cma.h | 13 ++++++--- mm/cma.c | 55 +++++++++++++++++++++++++----------- mm/page_alloc.c | 69 +++++++++++++++++++++++++++++---------------- 3 files changed, 92 insertions(+), 45 deletions(-) diff --git a/include/linux/cma.h b/include/linux/cma.h index 63873b93deaa..ffbb8ea2c5f8 100644 --- a/include/linux/cma.h +++ b/include/linux/cma.h @@ -31,11 +31,13 @@ extern phys_addr_t cma_get_base(const struct cma *cma); extern unsigned long cma_get_size(const struct cma *cma); extern const char *cma_get_name(const struct cma *cma); =20 -extern int __init cma_declare_contiguous_nid(phys_addr_t base, +extern int __init __cma_declare_contiguous_nid(phys_addr_t base, phys_addr_t size, phys_addr_t limit, phys_addr_t alignment, unsigned int order_per_bit, bool fixed, const char *name, struct cma **res_cma, - int nid); + int nid, bool in_dmb); +#define cma_declare_contiguous_nid(b, s, l, a, o, f, n, r_c, nid) \ + __cma_declare_contiguous_nid(b, s, l, a, o, f, n, r_c, nid, false) static inline int __init cma_declare_contiguous(phys_addr_t base, phys_addr_t size, phys_addr_t limit, phys_addr_t alignment, unsigned int order_per_bit, @@ -44,10 +46,13 @@ static inline int __init cma_declare_contiguous(phys_ad= dr_t base, return cma_declare_contiguous_nid(base, size, limit, alignment, order_per_bit, fixed, name, res_cma, NUMA_NO_NODE); } -extern int cma_init_reserved_mem(phys_addr_t base, phys_addr_t size, +extern int __cma_init_reserved_mem(phys_addr_t base, phys_addr_t size, unsigned int order_per_bit, const char *name, - struct cma **res_cma); + struct cma **res_cma, + bool in_dmb); +#define cma_init_reserved_mem(base, size, order, name, res_cma) \ + __cma_init_reserved_mem(base, size, order, name, res_cma, 0) extern struct page *cma_alloc(struct cma *cma, unsigned long count, unsign= ed int align, bool no_warn); extern bool cma_pages_valid(struct cma *cma, const struct page *pages, uns= igned long count); diff --git a/mm/cma.c b/mm/cma.c index 6208a3e1cd9d..4f33cd54db9e 100644 --- a/mm/cma.c +++ b/mm/cma.c @@ -33,6 +33,7 @@ #include #include #include +#include =20 #include "cma.h" =20 @@ -98,6 +99,10 @@ static void __init cma_activate_area(struct cma *cma) { unsigned long base_pfn =3D cma->base_pfn, pfn; struct zone *zone; + int is_dmb =3D dmb_intersects(base_pfn, base_pfn + cma->count); + + if (is_dmb =3D=3D DMB_MIXED) + goto out_error; =20 cma->bitmap =3D bitmap_zalloc(cma_bitmap_maxno(cma), GFP_KERNEL); if (!cma->bitmap) @@ -116,13 +121,17 @@ static void __init cma_activate_area(struct cma *cma) goto not_in_zone; } =20 - for (pfn =3D base_pfn; pfn < base_pfn + cma->count; - pfn +=3D pageblock_nr_pages) { - struct page *page =3D pfn_to_page(pfn); + if (is_dmb =3D=3D DMB_INTERSECTS) { + totalcma_pages -=3D cma->count; + } else { + for (pfn =3D base_pfn; pfn < base_pfn + cma->count; + pfn +=3D pageblock_nr_pages) { + struct page *page =3D pfn_to_page(pfn); =20 - set_pageblock_migratetype(page, MIGRATE_CMA); - init_reserved_pageblock(page); - page_zone(page)->cma_pages +=3D pageblock_nr_pages; + set_pageblock_migratetype(page, MIGRATE_CMA); + init_reserved_pageblock(page); + page_zone(page)->cma_pages +=3D pageblock_nr_pages; + } } =20 spin_lock_init(&cma->lock); @@ -141,7 +150,8 @@ static void __init cma_activate_area(struct cma *cma) if (!cma->reserve_pages_on_error) { for (pfn =3D base_pfn; pfn < base_pfn + cma->count; pfn +=3D pageblock_nr_pages) - init_reserved_pageblock(pfn_to_page(pfn)); + if (!dmb_intersects(pfn, pfn + pageblock_nr_pages)) + init_reserved_pageblock(pfn_to_page(pfn)); } totalcma_pages -=3D cma->count; cma->count =3D 0; @@ -166,7 +176,7 @@ void __init cma_reserve_pages_on_error(struct cma *cma) } =20 /** - * cma_init_reserved_mem() - create custom contiguous area from reserved m= emory + * __cma_init_reserved_mem() - create custom contiguous area in reserved m= emory * @base: Base address of the reserved area * @size: Size of the reserved area (in bytes), * @order_per_bit: Order of pages represented by one bit on bitmap. @@ -174,15 +184,18 @@ void __init cma_reserve_pages_on_error(struct cma *cm= a) * the area will be set to "cmaN", where N is a running counter of * used areas. * @res_cma: Pointer to store the created cma region. + * @in_dmb: Designate the reserved memory as a Designated Movable Block. * * This function creates custom contiguous area from already reserved memo= ry. */ -int __init cma_init_reserved_mem(phys_addr_t base, phys_addr_t size, - unsigned int order_per_bit, - const char *name, - struct cma **res_cma) +int __init __cma_init_reserved_mem(phys_addr_t base, phys_addr_t size, + unsigned int order_per_bit, + const char *name, + struct cma **res_cma, + bool in_dmb) { struct cma *cma; + int err; =20 /* Sanity checks */ if (cma_area_count =3D=3D ARRAY_SIZE(cma_areas)) { @@ -201,6 +214,14 @@ int __init cma_init_reserved_mem(phys_addr_t base, phy= s_addr_t size, if (!IS_ALIGNED(base | size, CMA_MIN_ALIGNMENT_BYTES)) return -EINVAL; =20 + if (in_dmb) { + err =3D dmb_reserve(base, size, NULL); + if (err) { + pr_err("Cannot reserve DMB for CMA!\n"); + return err; + } + } + /* * Each reserved area must be initialised later, when more kernel * subsystems (like slab allocator) are available. @@ -223,7 +244,7 @@ int __init cma_init_reserved_mem(phys_addr_t base, phys= _addr_t size, } =20 /** - * cma_declare_contiguous_nid() - reserve custom contiguous area + * __cma_declare_contiguous_nid() - reserve custom contiguous area * @base: Base address of the reserved area optional, use 0 for any * @size: Size of the reserved area (in bytes), * @limit: End address of the reserved memory (optional, 0 for any). @@ -233,6 +254,7 @@ int __init cma_init_reserved_mem(phys_addr_t base, phys= _addr_t size, * @name: The name of the area. See function cma_init_reserved_mem() * @res_cma: Pointer to store the created cma region. * @nid: nid of the free area to find, %NUMA_NO_NODE for any node + * @in_dmb: Designate the reserved memory as a Designated Movable Block. * * This function reserves memory from early allocator. It should be * called by arch specific code once the early allocator (memblock or boot= mem) @@ -242,11 +264,11 @@ int __init cma_init_reserved_mem(phys_addr_t base, ph= ys_addr_t size, * If @fixed is true, reserve contiguous area at exactly @base. If false, * reserve in range from @base to @limit. */ -int __init cma_declare_contiguous_nid(phys_addr_t base, +int __init __cma_declare_contiguous_nid(phys_addr_t base, phys_addr_t size, phys_addr_t limit, phys_addr_t alignment, unsigned int order_per_bit, bool fixed, const char *name, struct cma **res_cma, - int nid) + int nid, bool in_dmb) { phys_addr_t memblock_end =3D memblock_end_of_DRAM(); phys_addr_t highmem_start; @@ -374,7 +396,8 @@ int __init cma_declare_contiguous_nid(phys_addr_t base, base =3D addr; } =20 - ret =3D cma_init_reserved_mem(base, size, order_per_bit, name, res_cma); + ret =3D __cma_init_reserved_mem(base, size, order_per_bit, name, res_cma, + in_dmb); if (ret) goto free_mem; =20 diff --git a/mm/page_alloc.c b/mm/page_alloc.c index e38dd1b32771..09d00c178bc8 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -9233,29 +9233,8 @@ int __alloc_contig_migrate_range(struct compact_cont= rol *cc, return 0; } =20 -/** - * alloc_contig_range() -- tries to allocate given range of pages - * @start: start PFN to allocate - * @end: one-past-the-last PFN to allocate - * @migratetype: migratetype of the underlying pageblocks (either - * #MIGRATE_MOVABLE or #MIGRATE_CMA). All pageblocks - * in range must have the same migratetype and it must - * be either of the two. - * @gfp_mask: GFP mask to use during compaction - * - * The PFN range does not have to be pageblock aligned. The PFN range must - * belong to a single zone. - * - * The first thing this routine does is attempt to MIGRATE_ISOLATE all - * pageblocks in the range. Once isolated, the pageblocks should not - * be modified by others. - * - * Return: zero on success or negative error code. On success all - * pages which PFN is in [start, end) are allocated for the caller and - * need to be freed with free_contig_range(). - */ -int alloc_contig_range(unsigned long start, unsigned long end, - unsigned migratetype, gfp_t gfp_mask) +int _alloc_contig_range(unsigned long start, unsigned long end, + unsigned int migratetype, gfp_t gfp_mask) { unsigned long outer_start, outer_end; int order; @@ -9379,6 +9358,46 @@ int alloc_contig_range(unsigned long start, unsigned= long end, undo_isolate_page_range(start, end, migratetype); return ret; } + +/** + * alloc_contig_range() -- tries to allocate given range of pages + * @start: start PFN to allocate + * @end: one-past-the-last PFN to allocate + * @migratetype: migratetype of the underlying pageblocks (either + * #MIGRATE_MOVABLE or #MIGRATE_CMA). All pageblocks + * in range must have the same migratetype and it must + * be either of the two. + * @gfp_mask: GFP mask to use during compaction + * + * The PFN range does not have to be pageblock aligned. The PFN range must + * belong to a single zone. + * + * The first thing this routine does is attempt to MIGRATE_ISOLATE all + * pageblocks in the range. Once isolated, the pageblocks should not + * be modified by others. + * + * Return: zero on success or negative error code. On success all + * pages which PFN is in [start, end) are allocated for the caller and + * need to be freed with free_contig_range(). + */ +int alloc_contig_range(unsigned long start, unsigned long end, + unsigned int migratetype, gfp_t gfp_mask) +{ + switch (dmb_intersects(start, end)) { + case DMB_DISJOINT: + break; + case DMB_INTERSECTS: + if (migratetype =3D=3D MIGRATE_CMA) + migratetype =3D MIGRATE_MOVABLE; + else + return -EBUSY; + break; + default: + return -EBUSY; + } + + return _alloc_contig_range(start, end, migratetype, gfp_mask); +} EXPORT_SYMBOL(alloc_contig_range); =20 static int __alloc_contig_pages(unsigned long start_pfn, @@ -9386,8 +9405,8 @@ static int __alloc_contig_pages(unsigned long start_p= fn, { unsigned long end_pfn =3D start_pfn + nr_pages; =20 - return alloc_contig_range(start_pfn, end_pfn, MIGRATE_MOVABLE, - gfp_mask); + return _alloc_contig_range(start_pfn, end_pfn, MIGRATE_MOVABLE, + gfp_mask); } =20 static bool pfn_range_valid_contig(struct zone *z, unsigned long start_pfn, --=20 2.25.1 From nobody Tue Dec 16 03:00:38 2025 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id AF6B5ECAAD8 for ; Tue, 13 Sep 2022 20:00:42 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230154AbiIMUAl (ORCPT ); Tue, 13 Sep 2022 16:00:41 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47056 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229982AbiIMT7v (ORCPT ); Tue, 13 Sep 2022 15:59:51 -0400 Received: from mail-qk1-x731.google.com (mail-qk1-x731.google.com [IPv6:2607:f8b0:4864:20::731]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 36D9A7437A; Tue, 13 Sep 2022 12:59:05 -0700 (PDT) Received: by mail-qk1-x731.google.com with SMTP id u28so6371503qku.2; Tue, 13 Sep 2022 12:59:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date; bh=Iv6LAHuQtTXeYWyq0DEDKZbOD/1nb3ztqrePD2HtkIE=; b=YrAFBuuyJkwRgGb5yJxtoTNNKh/7y4BOHe54UWWbwGw/7zVyvD5YS3xh6wOizb8P5y WPvQ65cSNAdgAQQ5wDi83N9jAKpum/P9kqfXIHyP2JYI4r5D1linVF+pAeO7tMTGeNJd s10XvBpPDdlS4NQdOIfsc4Y86zu8zbnj/iRe9JIr+LJQn5lsr3jvpDq8YkHXv8CVoVQ9 l1Y7VxuX7aqVyF4T9IwYMUdPvuZE/+WFrgImMS1687EPItdw+VwpQNOwUoolbQ3Hf9aR Ox2qAlcyZj/+2EVV8HvGsz7p+nWwMiSQtDYItBS1H3jfaKi/gJ7kAUpfhR4jWWYZpfwT Vmow== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date; bh=Iv6LAHuQtTXeYWyq0DEDKZbOD/1nb3ztqrePD2HtkIE=; b=A5hoLNND822FCudX2ihAYymlfGK7yfBKgzASY9nWsW4RyOUx9pJrGxe1oWXoPLDGrn n9QMHEvcjkJP9QFiI602C29eEQ7zbrft1bm9ozpIUrJd+ESBvZE9urWmoZUWHru0obVd LvPDaBgVzdREnhNZ435dbSVOGH+KKVIWC1/iJTZPIpGLOvCdxcQ5JGtMqDnPA8Q32R3q ZHSoQ1m2GIItCGRBXcLNDyteTt16DL3XFybem2w2iwluVF56wVatG/rk4eHZNaKUjSi7 V2LTSmSa/XngYdssC51zP2M4DWaqQnkrsjNq6V51CPL93IugfbBbvs5f73tBf9OyAiKU W4hw== X-Gm-Message-State: ACgBeo163S6u9sLJ7xkyJuyxNzyCpJNm308cw2CT06JiXqOycO4d6eYX 1q/wHT16vw10JdFRJS3e6KM= X-Google-Smtp-Source: AA6agR6fK874H1rNDTk5i/hsxGcmtY4AZlr+R557N3WVBFwFecth0fyDkTeEea5ukgS/JNJqcxXZjQ== X-Received: by 2002:a37:916:0:b0:6ce:5f1c:d5d9 with SMTP id 22-20020a370916000000b006ce5f1cd5d9mr5679706qkj.737.1663099144354; Tue, 13 Sep 2022 12:59:04 -0700 (PDT) Received: from stbirv-lnx-3.igp.broadcom.net ([192.19.223.252]) by smtp.gmail.com with ESMTPSA id s11-20020a05620a29cb00b006b8e049cf08sm276305qkp.2.2022.09.13.12.59.00 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 13 Sep 2022 12:59:03 -0700 (PDT) From: Doug Berger To: Andrew Morton Cc: Jonathan Corbet , Rob Herring , Krzysztof Kozlowski , Frank Rowand , Mike Kravetz , Muchun Song , Mike Rapoport , Christoph Hellwig , Marek Szyprowski , Robin Murphy , Borislav Petkov , "Paul E. McKenney" , Neeraj Upadhyay , Randy Dunlap , Damien Le Moal , Doug Berger , Florian Fainelli , David Hildenbrand , Zi Yan , Oscar Salvador , Hari Bathini , Kees Cook , - , KOSAKI Motohiro , Mel Gorman , linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, devicetree@vger.kernel.org, linux-mm@kvack.org, iommu@lists.linux.dev Subject: [PATCH 19/21] dt-bindings: reserved-memory: shared-dma-pool: support DMB Date: Tue, 13 Sep 2022 12:55:06 -0700 Message-Id: <20220913195508.3511038-20-opendmb@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220913195508.3511038-1-opendmb@gmail.com> References: <20220913195508.3511038-1-opendmb@gmail.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" The shared-dmb-pool compatible string creates a Designated Movable Block to contain a shared pool of DMA buffers. Signed-off-by: Doug Berger --- .../bindings/reserved-memory/shared-dma-pool.yaml | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/Documentation/devicetree/bindings/reserved-memory/shared-dma-p= ool.yaml b/Documentation/devicetree/bindings/reserved-memory/shared-dma-poo= l.yaml index 618105f079be..85824fe05ac9 100644 --- a/Documentation/devicetree/bindings/reserved-memory/shared-dma-pool.yaml +++ b/Documentation/devicetree/bindings/reserved-memory/shared-dma-pool.yaml @@ -22,6 +22,14 @@ properties: operating system to instantiate the necessary pool management subsystem if necessary. =20 + - const: shared-dmb-pool + description: > + This indicates a shared-dma-pool region that is located within + a Designated Movable Block. The operating system is free to + use unallocated memory for movable allocations in this region. + Devices need to be tolerant of allocation latency to use this + pool. + - const: restricted-dma-pool description: > This indicates a region of memory meant to be used as a pool --=20 2.25.1 From nobody Tue Dec 16 03:00:38 2025 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 703B2ECAAD8 for ; Tue, 13 Sep 2022 20:00:54 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230202AbiIMUAw (ORCPT ); Tue, 13 Sep 2022 16:00:52 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51150 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230031AbiIMT77 (ORCPT ); Tue, 13 Sep 2022 15:59:59 -0400 Received: from mail-qk1-x72e.google.com (mail-qk1-x72e.google.com [IPv6:2607:f8b0:4864:20::72e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6CE15792DE; Tue, 13 Sep 2022 12:59:09 -0700 (PDT) Received: by mail-qk1-x72e.google.com with SMTP id k12so9304691qkj.8; Tue, 13 Sep 2022 12:59:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date; bh=lc6C5FZ3R6bpGdrTQfUMqHUthTTami+g742/pfBsTic=; b=RlsLmEiP3JVWhyXjCMSzF7A4AVjyq1jRjWCW9x5GNVTee31Y+zsIXxNeUFqHaiWHyg 8J6hP8RFMjTeETwSHV9ENC6pqQalmeYj6JweS9Tsv7OxuqvPlsMarmRSfwGBaWF+XcBE la0+kyQTMUqV4xn7LwqYCphDGQkUMF9O3ZqeG7FJEIlcBtt95RRsh5UONSCeySm6f9no qWos+YkhM9jVDPaT5oAH5RbFh07TQsfPjWCoTiWZNYZktg0UxgNqKCsIm4np7G5/9pqK Gqu0dk9U2X717zjfg5WZXwdvFki/e2iGeK+mAGJNsof/XN1i9elvjnIO7qYgQkqpgSuo A90g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date; bh=lc6C5FZ3R6bpGdrTQfUMqHUthTTami+g742/pfBsTic=; b=EmoYVb/xHZp2zy8lEHzJh8jE83PJfTP69p41jXm4mQpEDZu1WdoiK5qFFFbYi45VI+ sPg9vh+z8TFR8Cs1Qs3VKLxl4T6+9Xul6fWbBFGYctSSQKWcSOaZBpOZtCl7+0TcGfGU 9Wt9+iBw2FMZ7eGECxd/CjcsgkUtVJ3pt6JK79GfpmV3f0HhTQPem8UxPHG/eMvRxlaD xFjxzy4pWvw5yBa6F289yTFh9yrAkHfE24pGzDeu0Xl57BKbqV8jrYYeSrbWs3tqrAMa YNF/iqg+6WuLHtCLPPWq74cPnjdiCZ/JrQyRWWpWOVrieS/Vg9vqR4GsVx/aXlAUSNAk EOew== X-Gm-Message-State: ACgBeo1EIKsfjIHRTQTk9VOrHIRJiDUq0u0hzp5q5n9+B6QdEWzWARKc QdlaVa+5Cc9whLgv2B0zcpc= X-Google-Smtp-Source: AA6agR4JgnZFDc4dYiN2cHvYsrY/zIDRf84OViiKl4IHPDPBj/qqA4B2wHLv+IBu33IWNCFBvl5WRQ== X-Received: by 2002:a05:620a:22b0:b0:6cd:f06a:7978 with SMTP id p16-20020a05620a22b000b006cdf06a7978mr13621240qkh.106.1663099148142; Tue, 13 Sep 2022 12:59:08 -0700 (PDT) Received: from stbirv-lnx-3.igp.broadcom.net ([192.19.223.252]) by smtp.gmail.com with ESMTPSA id s11-20020a05620a29cb00b006b8e049cf08sm276305qkp.2.2022.09.13.12.59.04 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 13 Sep 2022 12:59:07 -0700 (PDT) From: Doug Berger To: Andrew Morton Cc: Jonathan Corbet , Rob Herring , Krzysztof Kozlowski , Frank Rowand , Mike Kravetz , Muchun Song , Mike Rapoport , Christoph Hellwig , Marek Szyprowski , Robin Murphy , Borislav Petkov , "Paul E. McKenney" , Neeraj Upadhyay , Randy Dunlap , Damien Le Moal , Doug Berger , Florian Fainelli , David Hildenbrand , Zi Yan , Oscar Salvador , Hari Bathini , Kees Cook , - , KOSAKI Motohiro , Mel Gorman , linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, devicetree@vger.kernel.org, linux-mm@kvack.org, iommu@lists.linux.dev Subject: [PATCH 20/21] mm/cma: introduce rmem shared-dmb-pool Date: Tue, 13 Sep 2022 12:55:07 -0700 Message-Id: <20220913195508.3511038-21-opendmb@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220913195508.3511038-1-opendmb@gmail.com> References: <20220913195508.3511038-1-opendmb@gmail.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" A 'shared-dmb-pool' reserved-memory device tree node defines a Designated Movable Block for use by an associated Contiguous Memory Allocator. Devices access the CMA region in the same manner as a 'shared- dma-pool', but the kernel page allocator is free to use the memory like any other ZONE_MOVABLE memory. Signed-off-by: Doug Berger --- drivers/of/of_reserved_mem.c | 5 +++++ kernel/dma/contiguous.c | 33 ++++++++++++++++++++++++++++----- 2 files changed, 33 insertions(+), 5 deletions(-) diff --git a/drivers/of/of_reserved_mem.c b/drivers/of/of_reserved_mem.c index 0eb9e8898d7b..700c0dc0d3b6 100644 --- a/drivers/of/of_reserved_mem.c +++ b/drivers/of/of_reserved_mem.c @@ -123,6 +123,11 @@ static int __init __reserved_mem_alloc_size(unsigned l= ong node, && !nomap) { /* Need adjust the alignment to satisfy the CMA requirement */ align =3D max_t(phys_addr_t, align, CMA_MIN_ALIGNMENT_BYTES); + } else if (IS_ENABLED(CONFIG_CMA) + && of_flat_dt_is_compatible(node, "shared-dmb-pool")) { + /* Need adjust the alignment to satisfy CMA/DMB requirements */ + align =3D max_t(phys_addr_t, align, CMA_MIN_ALIGNMENT_BYTES); + align =3D max_t(phys_addr_t, align, DMB_MIN_ALIGNMENT_BYTES); } =20 prop =3D of_get_flat_dt_prop(node, "alloc-ranges", &len); diff --git a/kernel/dma/contiguous.c b/kernel/dma/contiguous.c index 6ea80ae42622..65dda12752a7 100644 --- a/kernel/dma/contiguous.c +++ b/kernel/dma/contiguous.c @@ -50,6 +50,7 @@ #include #include #include +#include =20 #ifdef CONFIG_CMA_SIZE_MBYTES #define CMA_SIZE_MBYTES CONFIG_CMA_SIZE_MBYTES @@ -397,10 +398,11 @@ static const struct reserved_mem_ops rmem_cma_ops =3D= { .device_release =3D rmem_cma_device_release, }; =20 -static int __init rmem_cma_setup(struct reserved_mem *rmem) +static int __init _rmem_cma_setup(struct reserved_mem *rmem, bool in_dmb) { unsigned long node =3D rmem->fdt_node; bool default_cma =3D of_get_flat_dt_prop(node, "linux,cma-default", NULL); + phys_addr_t align =3D CMA_MIN_ALIGNMENT_BYTES; struct cma *cma; int err; =20 @@ -414,16 +416,25 @@ static int __init rmem_cma_setup(struct reserved_mem = *rmem) of_get_flat_dt_prop(node, "no-map", NULL)) return -EINVAL; =20 - if (!IS_ALIGNED(rmem->base | rmem->size, CMA_MIN_ALIGNMENT_BYTES)) { + if (in_dmb) { + if (default_cma) { + pr_err("Reserved memory: cma-default cannot be DMB\n"); + return -EINVAL; + } + align =3D max_t(phys_addr_t, align, DMB_MIN_ALIGNMENT_BYTES); + } + if (!IS_ALIGNED(rmem->base | rmem->size, align)) { pr_err("Reserved memory: incorrect alignment of CMA region\n"); return -EINVAL; } =20 - err =3D cma_init_reserved_mem(rmem->base, rmem->size, 0, rmem->name, &cma= ); + err =3D __cma_init_reserved_mem(rmem->base, rmem->size, 0, rmem->name, + &cma, in_dmb); if (err) { pr_err("Reserved memory: unable to setup CMA region\n"); return err; } + /* Architecture specific contiguous memory fixup. */ dma_contiguous_early_fixup(rmem->base, rmem->size); =20 @@ -433,10 +444,22 @@ static int __init rmem_cma_setup(struct reserved_mem = *rmem) rmem->ops =3D &rmem_cma_ops; rmem->priv =3D cma; =20 - pr_info("Reserved memory: created CMA memory pool at %pa, size %ld MiB\n", - &rmem->base, (unsigned long)rmem->size / SZ_1M); + pr_info("Reserved memory: created %s memory pool at %pa, size %ld MiB\n", + in_dmb ? "DMB" : "CMA", &rmem->base, + (unsigned long)rmem->size / SZ_1M); =20 return 0; } + +static int __init rmem_cma_setup(struct reserved_mem *rmem) +{ + return _rmem_cma_setup(rmem, false); +} RESERVEDMEM_OF_DECLARE(cma, "shared-dma-pool", rmem_cma_setup); + +static int __init rmem_cma_in_dmb_setup(struct reserved_mem *rmem) +{ + return _rmem_cma_setup(rmem, true); +} +RESERVEDMEM_OF_DECLARE(cma_in_dmb, "shared-dmb-pool", rmem_cma_in_dmb_setu= p); #endif --=20 2.25.1 From nobody Tue Dec 16 03:00:38 2025 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id DDF39C54EE9 for ; Tue, 13 Sep 2022 20:01:00 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229638AbiIMUA5 (ORCPT ); Tue, 13 Sep 2022 16:00:57 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46868 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229744AbiIMUAD (ORCPT ); Tue, 13 Sep 2022 16:00:03 -0400 Received: from mail-qv1-xf2a.google.com (mail-qv1-xf2a.google.com [IPv6:2607:f8b0:4864:20::f2a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id AB6B6796AA; Tue, 13 Sep 2022 12:59:12 -0700 (PDT) Received: by mail-qv1-xf2a.google.com with SMTP id l5so10085275qvs.13; Tue, 13 Sep 2022 12:59:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date; bh=koBwJdR4HF2Mh9OkMa3nN+ZpGb/cJotv47IHPg3aLjM=; b=H9fVlQTVY6EaU+njTimAq+ibJtvJjx2vWGkJikgtxKpK4lpRvh7Gs0dhq+SfYU7Y1l ol227J1p7+D7fHpafEcMsArwK6zpQesHNLeCyw8pBBnlfTeZXRH0gTB01wVhDLo6ZqZ0 2mt9ehwJJq5f5bEntKn/0aquyhMi35hMoNg4MgcTAFw56eJYhoioxREYotQhcXVzv4ef 26N8v6sdGt7Ksl3t6WmZRbYcEffyv4LTbLh0c7Zn4Y23wzA2w1INQgl13/vOL+Z8k7Uy rhOg4rC2RHnt2n62JjXnioNT4+kK415lG500sJWQkISGfWvSLQb972JndYtjJi8pBB0H qioA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date; bh=koBwJdR4HF2Mh9OkMa3nN+ZpGb/cJotv47IHPg3aLjM=; b=HN7lhz6SD+hT8b1DBWvUrJNtZdiu4wDF7bHx9lrwgcYGTgSOc9npRxclx1uIhOeAm/ piFgg7I0qM+7uuFTbwTcWzP9U+X37VOMOF18k7Z1D1yQGZJGyaNNn4onTUV5J8Th551p 4gdxT/E+aBKOjYJheyA4m31wPDDwaZRD2xEJnTY5xETTAmV94uC4j32OtCkbyae3S8X9 3ZAVw2kfdi96qY6vAUR0REO3fQCFLxRduOFcSh39HdTtdjl54THciYIjQnSWhml+qgH2 qEnrb7SSxvrQeqjiwyp+ZKEWS+kj+Rfqsjno6sE1Zc1Vr10CMWw9AhTKBF6sjsJ/c8tq tAUw== X-Gm-Message-State: ACgBeo0ybWQ69iXreBTql3F2ddDt1pdWNkqPzneq7FOVfr/I3cHY5oCE Boyb2a0M2EpQxprcLY5ENB8= X-Google-Smtp-Source: AA6agR4KLeMksXEqgCvXkwrSYo6YaUFxsEGC4nazCYouj2Y6nxYnhGX19vjDfEjEBR3X6wdqMlktzg== X-Received: by 2002:a05:6214:1c0a:b0:49d:87ff:e77c with SMTP id u10-20020a0562141c0a00b0049d87ffe77cmr28164232qvc.54.1663099151805; Tue, 13 Sep 2022 12:59:11 -0700 (PDT) Received: from stbirv-lnx-3.igp.broadcom.net ([192.19.223.252]) by smtp.gmail.com with ESMTPSA id s11-20020a05620a29cb00b006b8e049cf08sm276305qkp.2.2022.09.13.12.59.08 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 13 Sep 2022 12:59:11 -0700 (PDT) From: Doug Berger To: Andrew Morton Cc: Jonathan Corbet , Rob Herring , Krzysztof Kozlowski , Frank Rowand , Mike Kravetz , Muchun Song , Mike Rapoport , Christoph Hellwig , Marek Szyprowski , Robin Murphy , Borislav Petkov , "Paul E. McKenney" , Neeraj Upadhyay , Randy Dunlap , Damien Le Moal , Doug Berger , Florian Fainelli , David Hildenbrand , Zi Yan , Oscar Salvador , Hari Bathini , Kees Cook , - , KOSAKI Motohiro , Mel Gorman , linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, devicetree@vger.kernel.org, linux-mm@kvack.org, iommu@lists.linux.dev Subject: [PATCH 21/21] mm/hugetlb: introduce hugetlb_dmb Date: Tue, 13 Sep 2022 12:55:08 -0700 Message-Id: <20220913195508.3511038-22-opendmb@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220913195508.3511038-1-opendmb@gmail.com> References: <20220913195508.3511038-1-opendmb@gmail.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" If specified on the kernel command line the hugetlb_dmb parameter modifies the behavior of the hugetlb_cma parameter to use the Contiguous Memory Allocator within Designated Movable Blocks for gigantic page allocation. This allows the kernel page allocator to use the memory more agressively than traditional CMA memory pools at the cost of potentially increased allocation latency. Signed-off-by: Doug Berger --- Documentation/admin-guide/kernel-parameters.txt | 3 +++ mm/hugetlb.c | 16 +++++++++++++--- 2 files changed, 16 insertions(+), 3 deletions(-) diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentatio= n/admin-guide/kernel-parameters.txt index 8141fac7c7cb..b29d1fa253d6 100644 --- a/Documentation/admin-guide/kernel-parameters.txt +++ b/Documentation/admin-guide/kernel-parameters.txt @@ -1732,6 +1732,9 @@ hugepages using the CMA allocator. If enabled, the boot-time allocation of gigantic hugepages is skipped. =20 + hugetlb_dmb [HW,CMA] Causes hugetlb_cma to use Designated Movable + Blocks for any CMA areas it reserves. + hugetlb_free_vmemmap=3D [KNL] Reguires CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP enabled. diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 2f354423f50f..d3fb8b1f443f 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -54,6 +54,7 @@ struct hstate hstates[HUGE_MAX_HSTATE]; #ifdef CONFIG_CMA static struct cma *hugetlb_cma[MAX_NUMNODES]; static unsigned long hugetlb_cma_size_in_node[MAX_NUMNODES] __initdata; +static bool hugetlb_dmb __initdata; static bool hugetlb_cma_page(struct page *page, unsigned int order) { return cma_pages_valid(hugetlb_cma[page_to_nid(page)], page, @@ -7321,6 +7322,14 @@ static int __init cmdline_parse_hugetlb_cma(char *p) =20 early_param("hugetlb_cma", cmdline_parse_hugetlb_cma); =20 +static int __init cmdline_parse_hugetlb_dmb(char *p) +{ + hugetlb_dmb =3D true; + return 0; +} + +early_param("hugetlb_dmb", cmdline_parse_hugetlb_dmb); + void __init hugetlb_cma_reserve(int order) { unsigned long size, reserved, per_node; @@ -7396,10 +7405,11 @@ void __init hugetlb_cma_reserve(int order) * may be returned to CMA allocator in the case of * huge page demotion. */ - res =3D cma_declare_contiguous_nid(0, size, 0, + res =3D __cma_declare_contiguous_nid(0, size, 0, PAGE_SIZE << HUGETLB_PAGE_ORDER, - 0, false, name, - &hugetlb_cma[nid], nid); + 0, false, name, + &hugetlb_cma[nid], nid, + hugetlb_dmb); if (res) { pr_warn("hugetlb_cma: reservation failed: err %d, node %d", res, nid); --=20 2.25.1