From nobody Mon Apr 27 04:46:48 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 97BF4C433EF for ; Thu, 16 Jun 2022 17:48:57 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236518AbiFPRsz (ORCPT ); Thu, 16 Jun 2022 13:48:55 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:53572 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235647AbiFPRsu (ORCPT ); Thu, 16 Jun 2022 13:48:50 -0400 Received: from mail-pf1-x432.google.com (mail-pf1-x432.google.com [IPv6:2607:f8b0:4864:20::432]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id AA3D44BFCF for ; Thu, 16 Jun 2022 10:48:49 -0700 (PDT) Received: by mail-pf1-x432.google.com with SMTP id w21so2147439pfc.0 for ; Thu, 16 Jun 2022 10:48:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=gQUColGb/PRKYuEV4lE8XSFjbzf2io5XZTvMnHxGXf0=; b=H8066BRclkgv160HBGUSksGljw5jEx74P81Wyl7DeklPUcfVRAzzn4ytpohhWZvS6Z bpso2AnaKouGt/5mw84y3KUdYhbE9jmtHQatT9ICKrULazn80V2zMIJSCZPItFdRljXh H6aFJpbnfQughZDnUFhPMMhlzqd0YguP5GLK2+gCQuBe0X6KLNggSWTXUJR9bdZxcrR7 WTTG+kcWkBdkQyWYbhl2priPPsi5ijjv+dcXEljTRUC2eP4QUXR4ZAyjeeVsvkurn61y TJjtSPV52f5tCl1bh01NNGhVozBWuNUlJ2X4U+VASX8ogw4/5DUTI2jK+RriHKc3J8IY DBug== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=gQUColGb/PRKYuEV4lE8XSFjbzf2io5XZTvMnHxGXf0=; b=hoZU+/ty5qt6LpnWlliR/M67JDM13QacYktHLUq54hwJ0c0LHbAMR5Ecl3kA8deO5k 0b44LY/SLkauOuqS+QuBkGQkgQiYgEnshblnnlDMUw/Dp7PMIckUfmLXJ7mOFdViTIhe /6aPg3+dNxsdE06XBgb6EzWiPWExLLxFojqEIXSKzkPZF/yzwUckPOKBBukCyJNzXzHw ia3gizDICqAnhpr0Aln/D6+H8QDceJE+ox7nITDRnDNsIIBUdHokxTblHuG6Yzk3Kct/ 8nwO6cQDnzfd7PwTGp+4s+9QxHYx7GJeUsG9pd9PodlnrrWPWLN22962Wa50snjpxg0q R/HQ== X-Gm-Message-State: AJIora/3DlQWSZJKZy4OYoM5xWaQc5Zy7xMPCPTVzMoYy2DW1HE7St1U 4yyJ8aDmlqIJ+IgzESvSNFY= X-Google-Smtp-Source: AGRyM1vVU3rKwXy9n+PUA6CJmwhrnakM7UqbwEUBt0tkabuU8YP6EHHL4ouJ2CBwwzJvSF9Zcj+yCQ== X-Received: by 2002:a05:6a00:2392:b0:51b:fe0e:2b8 with SMTP id f18-20020a056a00239200b0051bfe0e02b8mr5799350pfc.84.1655401729182; Thu, 16 Jun 2022 10:48:49 -0700 (PDT) Received: from localhost.localdomain (c-67-174-241-145.hsd1.ca.comcast.net. [67.174.241.145]) by smtp.gmail.com with ESMTPSA id z21-20020a17090a8b9500b001e8520b211bsm1818289pjn.53.2022.06.16.10.48.47 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 16 Jun 2022 10:48:48 -0700 (PDT) From: Yang Shi To: vbabka@suse.cz, kirill.shutemov@linux.intel.com, willy@infradead.org, zokeefe@google.com, linmiaohe@huawei.com, akpm@linux-foundation.org Cc: shy828301@gmail.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [v5 PATCH 1/7] mm: khugepaged: check THP flag in hugepage_vma_check() Date: Thu, 16 Jun 2022 10:48:34 -0700 Message-Id: <20220616174840.1202070-2-shy828301@gmail.com> X-Mailer: git-send-email 2.26.3 In-Reply-To: <20220616174840.1202070-1-shy828301@gmail.com> References: <20220616174840.1202070-1-shy828301@gmail.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Currently the THP flag check in hugepage_vma_check() will fallthrough if the flag is NEVER and VM_HUGEPAGE is set. This is not a problem for now since all the callers have the flag checked before or can't be invoked if the flag is NEVER. However, the following patch will call hugepage_vma_check() in more places, for example, page fault, so this flag must be checked in hugepge_vma_check(). Reviewed-by: Zach O'Keefe Reviewed-by: Miaohe Lin Signed-off-by: Yang Shi --- mm/khugepaged.c | 3 +++ 1 file changed, 3 insertions(+) diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 476d79360101..b1dab94c0f1e 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -458,6 +458,9 @@ bool hugepage_vma_check(struct vm_area_struct *vma, if (shmem_file(vma->vm_file)) return shmem_huge_enabled(vma); =20 + if (!khugepaged_enabled()) + return false; + /* THP settings require madvise. */ if (!(vm_flags & VM_HUGEPAGE) && !khugepaged_always()) return false; --=20 2.26.3 From nobody Mon Apr 27 04:46:48 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7808BCCA47A for ; Thu, 16 Jun 2022 17:49:02 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237777AbiFPRtB (ORCPT ); Thu, 16 Jun 2022 13:49:01 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:53600 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235967AbiFPRsv (ORCPT ); Thu, 16 Jun 2022 13:48:51 -0400 Received: from mail-pf1-x42d.google.com (mail-pf1-x42d.google.com [IPv6:2607:f8b0:4864:20::42d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 33E814BFCF for ; Thu, 16 Jun 2022 10:48:51 -0700 (PDT) Received: by mail-pf1-x42d.google.com with SMTP id u2so2123544pfc.2 for ; Thu, 16 Jun 2022 10:48:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=Lf1Kp53gV5xVxcPrzuVVyrzHKBnqiH74ZtDbvhyF8zw=; b=qHbvu3U4WI6XTL+Bb7EcWbfSgzpXVwmBwdYi5b4fTDSMY/tMFb3YRbJy105vXefOI1 v+XLHWPz/vB0UgVN0u2mh1l+NLdiJ88GcEjUAoGZvoskG4KtWLQvWM29G0qBeynD9M9l ejsjQKiDU16xatYVZWK11/4EP942U/xh+kghEmAO2eS/VqNHbCd9twMGhSrNDNbfL3Fx 8AuHMCmNpCKAfw1BDm1meeWm/ujGolhpZ8paAlpESIC987HvF+vMDiRVmzs8ryP6UOof 4f4TUCiMMsSW1Qf6i2NxxcKj798ou3uvQlxSTGuXQ1ILAc8RKJd5hZjzKuHbjwjJ1504 8VjA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=Lf1Kp53gV5xVxcPrzuVVyrzHKBnqiH74ZtDbvhyF8zw=; b=yarRhz5uXHo5NVwEhQ7MFQ4GasG77d/ajPnLPrsdpBL9Zjr0bzmNuMBn8aZrMLjNHf vH6h0gGfDL99Axi2AadH8AmauDsU5R5BhQf/atFs6X34wXoaZCXwn17rr2s5yz3FRLB3 EVvYT3bg1SUWzRA0x5E21bKr0kW36foD78mx2hE5ueqqLq+jcGWWkpcVIKovMxV8K5iv fnG+bkdC2sRv/9GSop+9Pun8hiewEw2lPUaWLcUc4MOvwtdWJRkP38FCeuVVTWbtnKp+ Bg5qEVEdbBr7qE1fVHROjGfqb7AcRVHLyxkxU/N/6h4wFbyvYC6cXcmT8JeohfHP4Cpr Vqzw== X-Gm-Message-State: AJIora9Ik28smitMheU61QLDuguXu7xcfW2TyEVYMTIkK55BqmOKDTl2 w3dya5HQ0k+YDr9VZ0LA4a8= X-Google-Smtp-Source: AGRyM1sILDxv0QZoQXrOLOSuJVQZEjJnBYzToTfeZrKnwvvpIW3ALa81CvsweBmzReuWF3hmAbqADA== X-Received: by 2002:a63:894a:0:b0:3fc:a724:578c with SMTP id v71-20020a63894a000000b003fca724578cmr5543251pgd.499.1655401730679; Thu, 16 Jun 2022 10:48:50 -0700 (PDT) Received: from localhost.localdomain (c-67-174-241-145.hsd1.ca.comcast.net. [67.174.241.145]) by smtp.gmail.com with ESMTPSA id z21-20020a17090a8b9500b001e8520b211bsm1818289pjn.53.2022.06.16.10.48.49 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 16 Jun 2022 10:48:50 -0700 (PDT) From: Yang Shi To: vbabka@suse.cz, kirill.shutemov@linux.intel.com, willy@infradead.org, zokeefe@google.com, linmiaohe@huawei.com, akpm@linux-foundation.org Cc: shy828301@gmail.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [v5 PATCH 2/7] mm: thp: consolidate vma size check to transhuge_vma_suitable Date: Thu, 16 Jun 2022 10:48:35 -0700 Message-Id: <20220616174840.1202070-3-shy828301@gmail.com> X-Mailer: git-send-email 2.26.3 In-Reply-To: <20220616174840.1202070-1-shy828301@gmail.com> References: <20220616174840.1202070-1-shy828301@gmail.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" There are couple of places that check whether the vma size is ok for THP or whether address fits, they are open coded and duplicate, use transhuge_vma_suitable() to do the job by passing in (vma->end - HPAGE_PMD_SIZE). Move vma size check into hugepage_vma_check(). This will make khugepaged_enter() is as same as khugepaged_enter_vma(). There is just one caller for khugepaged_enter(), replace it to khugepaged_enter_vma() and remove khugepaged_enter(). Reviewed-by: Zach O'Keefe Signed-off-by: Yang Shi --- include/linux/huge_mm.h | 11 +++++++++++ include/linux/khugepaged.h | 14 -------------- mm/huge_memory.c | 2 +- mm/khugepaged.c | 19 ++++++------------- 4 files changed, 18 insertions(+), 28 deletions(-) diff --git a/include/linux/huge_mm.h b/include/linux/huge_mm.h index 648cb3ce7099..8a5a8bfce0f5 100644 --- a/include/linux/huge_mm.h +++ b/include/linux/huge_mm.h @@ -116,6 +116,17 @@ extern struct kobj_attribute shmem_enabled_attr; =20 extern unsigned long transparent_hugepage_flags; =20 +/* + * Do the below checks: + * - For file vma, check if the linear page offset of vma is + * HPAGE_PMD_NR aligned within the file. The hugepage is + * guaranteed to be hugepage-aligned within the file, but we must + * check that the PMD-aligned addresses in the VMA map to + * PMD-aligned offsets within the file, else the hugepage will + * not be PMD-mappable. + * - For all vmas, check if the haddr is in an aligned HPAGE_PMD_SIZE + * area. + */ static inline bool transhuge_vma_suitable(struct vm_area_struct *vma, unsigned long addr) { diff --git a/include/linux/khugepaged.h b/include/linux/khugepaged.h index 392d34c3c59a..31ca8a7f78f4 100644 --- a/include/linux/khugepaged.h +++ b/include/linux/khugepaged.h @@ -51,16 +51,6 @@ static inline void khugepaged_exit(struct mm_struct *mm) if (test_bit(MMF_VM_HUGEPAGE, &mm->flags)) __khugepaged_exit(mm); } - -static inline void khugepaged_enter(struct vm_area_struct *vma, - unsigned long vm_flags) -{ - if (!test_bit(MMF_VM_HUGEPAGE, &vma->vm_mm->flags) && - khugepaged_enabled()) { - if (hugepage_vma_check(vma, vm_flags)) - __khugepaged_enter(vma->vm_mm); - } -} #else /* CONFIG_TRANSPARENT_HUGEPAGE */ static inline void khugepaged_fork(struct mm_struct *mm, struct mm_struct = *oldmm) { @@ -68,10 +58,6 @@ static inline void khugepaged_fork(struct mm_struct *mm,= struct mm_struct *oldmm static inline void khugepaged_exit(struct mm_struct *mm) { } -static inline void khugepaged_enter(struct vm_area_struct *vma, - unsigned long vm_flags) -{ -} static inline void khugepaged_enter_vma(struct vm_area_struct *vma, unsigned long vm_flags) { diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 4f9bbb4eab23..b530462c4493 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -726,7 +726,7 @@ vm_fault_t do_huge_pmd_anonymous_page(struct vm_fault *= vmf) return VM_FAULT_FALLBACK; if (unlikely(anon_vma_prepare(vma))) return VM_FAULT_OOM; - khugepaged_enter(vma, vma->vm_flags); + khugepaged_enter_vma(vma, vma->vm_flags); =20 if (!(vmf->flags & FAULT_FLAG_WRITE) && !mm_forbids_zeropage(vma->vm_mm) && diff --git a/mm/khugepaged.c b/mm/khugepaged.c index b1dab94c0f1e..db0b334a7d1f 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -450,8 +450,8 @@ bool hugepage_vma_check(struct vm_area_struct *vma, if (vma_is_dax(vma)) return false; =20 - if (vma->vm_file && !IS_ALIGNED((vma->vm_start >> PAGE_SHIFT) - - vma->vm_pgoff, HPAGE_PMD_NR)) + /* Check alignment for file vma and size for both file and anon vma */ + if (!transhuge_vma_suitable(vma, (vma->vm_end - HPAGE_PMD_SIZE))) return false; =20 /* Enabled via shmem mount options or sysfs settings. */ @@ -512,9 +512,7 @@ void khugepaged_enter_vma(struct vm_area_struct *vma, unsigned long vm_flags) { if (!test_bit(MMF_VM_HUGEPAGE, &vma->vm_mm->flags) && - khugepaged_enabled() && - (((vma->vm_start + ~HPAGE_PMD_MASK) & HPAGE_PMD_MASK) < - (vma->vm_end & HPAGE_PMD_MASK))) { + khugepaged_enabled()) { if (hugepage_vma_check(vma, vm_flags)) __khugepaged_enter(vma->vm_mm); } @@ -950,7 +948,6 @@ static int hugepage_vma_revalidate(struct mm_struct *mm= , unsigned long address, struct vm_area_struct **vmap) { struct vm_area_struct *vma; - unsigned long hstart, hend; =20 if (unlikely(khugepaged_test_exit(mm))) return SCAN_ANY_PROCESS; @@ -959,9 +956,7 @@ static int hugepage_vma_revalidate(struct mm_struct *mm= , unsigned long address, if (!vma) return SCAN_VMA_NULL; =20 - hstart =3D (vma->vm_start + ~HPAGE_PMD_MASK) & HPAGE_PMD_MASK; - hend =3D vma->vm_end & HPAGE_PMD_MASK; - if (address < hstart || address + HPAGE_PMD_SIZE > hend) + if (!transhuge_vma_suitable(vma, address)) return SCAN_ADDRESS_RANGE; if (!hugepage_vma_check(vma, vma->vm_flags)) return SCAN_VMA_CHECK; @@ -2147,10 +2142,8 @@ static unsigned int khugepaged_scan_mm_slot(unsigned= int pages, progress++; continue; } - hstart =3D (vma->vm_start + ~HPAGE_PMD_MASK) & HPAGE_PMD_MASK; - hend =3D vma->vm_end & HPAGE_PMD_MASK; - if (hstart >=3D hend) - goto skip; + hstart =3D round_up(vma->vm_start, HPAGE_PMD_SIZE); + hend =3D round_down(vma->vm_end, HPAGE_PMD_SIZE); if (khugepaged_scan.address > hend) goto skip; if (khugepaged_scan.address < hstart) --=20 2.26.3 From nobody Mon Apr 27 04:46:48 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1B34CCCA47A for ; Thu, 16 Jun 2022 17:49:08 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238818AbiFPRtG (ORCPT ); Thu, 16 Jun 2022 13:49:06 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:53716 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237099AbiFPRs4 (ORCPT ); Thu, 16 Jun 2022 13:48:56 -0400 Received: from mail-pj1-x102d.google.com (mail-pj1-x102d.google.com [IPv6:2607:f8b0:4864:20::102d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E09C33ED23 for ; Thu, 16 Jun 2022 10:48:52 -0700 (PDT) Received: by mail-pj1-x102d.google.com with SMTP id o7-20020a17090a0a0700b001ebad897457so1015301pjo.0 for ; Thu, 16 Jun 2022 10:48:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=IVDYrUdqiRm9UcRbpXweUBj4/hDRVoO4CK1KpBPozys=; b=YoZ1bssMZl0ds5wLwIPKg0IItqCPaKU17iyetjJF+hT+7KvhMVimEkgRt95TZkLAxO FnqS18PmPzsWKkIght4RbuqwygilGASy2GLKqKg6k/vzT6t+DtM4UxAQl5b7P9A3TNNf eNO3UdSwBlFGrUig73CtXZrfbbh+sfQ3yu/6CT/tziWSALK5brivuz9r9Oul1PmCzrfZ vwzv81Kzs6hCvJuSzDap1wEz3A/BmNXbZucyPTub2avd33WiHFdjm29e8KsZtbG6Ob56 R8nESRo3DArWhf+2Xo28r7HWLmRlqKlf75OJ1nRX8ig9JYoj6D/6DI+3HTWP9r+gSf7n AW3Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=IVDYrUdqiRm9UcRbpXweUBj4/hDRVoO4CK1KpBPozys=; b=fAxnUT3pQhUL/BqM7oEfVsrGILpoB9jRYSJ21l0lmNvFEFvcA34rXWhsW2LOazWNug LltnyDY00IqhsTTtV0e0I9eeCTL8yPgkCrenobQc/W40x4xFSDUc+qqIkr/S3DkA8oIH ecYp7aWV6KSDnlchfDPiGWDYuEUdklvqS/MI7AreFY+a4irJofgsBEHQPSQm/FQ6LM/Z gAm9rLRHeg10dHeDR1cRQxJl667cM3f8c12OTTOY+XTQjHM/ntfaNRux9PThXyCxhp/C sQsG5YCYubz0Ek9tX8hnf5BySHhFUqAhNZwaQHNzgQMTWkoHXRxG7f3cPIy9jX1DYZFI w1hQ== X-Gm-Message-State: AJIora8IHb2PHcKkV7PxZMv+JWN9DMkoHUQ99r0Wa83k3rMer2WU8agg H4a5T2h0iBzuGrQmkzDWl0Q= X-Google-Smtp-Source: AGRyM1vxGK17GWzftPYeur/8Me6GEPOXRVJXvmUbi9GWcVUiu6WhWb2FJDsYzyvI0N7smHaYh/uyEg== X-Received: by 2002:a17:902:c403:b0:167:4a9f:2799 with SMTP id k3-20020a170902c40300b001674a9f2799mr5716030plk.56.1655401732435; Thu, 16 Jun 2022 10:48:52 -0700 (PDT) Received: from localhost.localdomain (c-67-174-241-145.hsd1.ca.comcast.net. [67.174.241.145]) by smtp.gmail.com with ESMTPSA id z21-20020a17090a8b9500b001e8520b211bsm1818289pjn.53.2022.06.16.10.48.50 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 16 Jun 2022 10:48:51 -0700 (PDT) From: Yang Shi To: vbabka@suse.cz, kirill.shutemov@linux.intel.com, willy@infradead.org, zokeefe@google.com, linmiaohe@huawei.com, akpm@linux-foundation.org Cc: shy828301@gmail.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [v5 PATCH 3/7] mm: khugepaged: better comments for anon vma check in hugepage_vma_revalidate Date: Thu, 16 Jun 2022 10:48:36 -0700 Message-Id: <20220616174840.1202070-4-shy828301@gmail.com> X-Mailer: git-send-email 2.26.3 In-Reply-To: <20220616174840.1202070-1-shy828301@gmail.com> References: <20220616174840.1202070-1-shy828301@gmail.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" The hugepage_vma_revalidate() needs to check if the vma is still anonymous vma or not since the address may be unmapped then remapped to file before khugepaged reaquired the mmap_lock. The old comment is not quite helpful, elaborate this with better comment. Reviewed-by: Zach O'Keefe Signed-off-by: Yang Shi --- mm/khugepaged.c | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/mm/khugepaged.c b/mm/khugepaged.c index db0b334a7d1f..5baa394e34c8 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -960,7 +960,13 @@ static int hugepage_vma_revalidate(struct mm_struct *m= m, unsigned long address, return SCAN_ADDRESS_RANGE; if (!hugepage_vma_check(vma, vma->vm_flags)) return SCAN_VMA_CHECK; - /* Anon VMA expected */ + /* + * Anon VMA expected, the address may be unmapped then + * remapped to file after khugepaged reaquired the mmap_lock. + * + * hugepage_vma_check may return true for qualified file + * vmas. + */ if (!vma->anon_vma || !vma_is_anonymous(vma)) return SCAN_VMA_CHECK; return 0; --=20 2.26.3 From nobody Mon Apr 27 04:46:48 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1D122C433EF for ; Thu, 16 Jun 2022 17:49:14 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S242282AbiFPRtK (ORCPT ); Thu, 16 Jun 2022 13:49:10 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:53722 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237297AbiFPRs5 (ORCPT ); Thu, 16 Jun 2022 13:48:57 -0400 Received: from mail-pl1-x631.google.com (mail-pl1-x631.google.com [IPv6:2607:f8b0:4864:20::631]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 814324D258 for ; Thu, 16 Jun 2022 10:48:54 -0700 (PDT) Received: by mail-pl1-x631.google.com with SMTP id k7so1837126plg.7 for ; Thu, 16 Jun 2022 10:48:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=SKzP8GvCz2PJf6EFnSMPoShRtGtH8yTLiOh2Lf/1mNw=; b=o6VtXrucgvo9V9vrl6em7ARq7LIYKBa6YZlKMEdy4FgPRadKHeJHGuino/92SIAf47 W5YJH0o3FkV2OlxUTftRIXQ1z6ePSIK0b0pozHwxajn/2+eAWl7VpUe1oEoxCWUQGpfh PGWBSbZBfsBXCM+Z2YGdzyXzGlIGxUxiENsmowMvYOBPiex+eS6XOOZAm1gjvkx47fAU p6memXJ0Vuq1uyU51/mX+zdh503EFoSnWzQYT2CrhScToA6fjxvN2ssJNUaR4hpqAzDp Ht/rlKGsHZYDj3WzSMKmKyXpAK8eEafaKStomvzNxTOr42LCJjlReO1s0H9E/t8Kf9kb Rfjg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=SKzP8GvCz2PJf6EFnSMPoShRtGtH8yTLiOh2Lf/1mNw=; b=G6G+LvK9lJ2ozwr7XfA5NW38krGesrR8kd5jQVzBRh0NHtxzhqp4eDx0wCq2Oh5+HI 6XgokftkCH8jxswc6tkdx9hS7oKrPX6DUgHgP0Lzs3tB/pzICMKfijBYx4xrbj4S0rkj /mzseAuj26rTuokxHAGTwuBg9e6IOSGY8QZ1VDSm+WCjKLZYyviLRJUcTDJGu7KHIQXB udGz+VKCRxSxzSBS5ubhzUql0jzMeUZ1FfNuHEefmszytWOe+w6AJ1zKfMpTqVFi1iAi 2GG6AP2f5XwE3vdYsbnD24DIix1COFRTJ8WZKK4nL9bWzSYbzEIO2ZNiFR8k5Q0hHqlv AewA== X-Gm-Message-State: AJIora9dByWV3xMh1FUCqqdIvrA9a6o3LPmxD6Bo+3mFxZn9xWsSKik8 roSPzWouFObwV0UScN70FrQ= X-Google-Smtp-Source: AGRyM1tD+obgwCVCrFhpKnWZxWNyZcVnZs4BoSmGflr9GWj6ak0KmkrqfKKE387/dApUt9IsDJQPzg== X-Received: by 2002:a17:902:76c2:b0:16a:205:f0cc with SMTP id j2-20020a17090276c200b0016a0205f0ccmr500472plt.171.1655401733973; Thu, 16 Jun 2022 10:48:53 -0700 (PDT) Received: from localhost.localdomain (c-67-174-241-145.hsd1.ca.comcast.net. [67.174.241.145]) by smtp.gmail.com with ESMTPSA id z21-20020a17090a8b9500b001e8520b211bsm1818289pjn.53.2022.06.16.10.48.52 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 16 Jun 2022 10:48:53 -0700 (PDT) From: Yang Shi To: vbabka@suse.cz, kirill.shutemov@linux.intel.com, willy@infradead.org, zokeefe@google.com, linmiaohe@huawei.com, akpm@linux-foundation.org Cc: shy828301@gmail.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [v5 PATCH 4/7] mm: thp: kill transparent_hugepage_active() Date: Thu, 16 Jun 2022 10:48:37 -0700 Message-Id: <20220616174840.1202070-5-shy828301@gmail.com> X-Mailer: git-send-email 2.26.3 In-Reply-To: <20220616174840.1202070-1-shy828301@gmail.com> References: <20220616174840.1202070-1-shy828301@gmail.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" The transparent_hugepage_active() was introduced to show THP eligibility bit in smaps in proc, smaps is the only user. But it actually does the similar check as hugepage_vma_check() which is used by khugepaged. We definitely don't have to maintain two similar checks, so kill transparent_hugepage_active(). This patch also fixed the wrong behavior for VM_NO_KHUGEPAGED vmas. Also move hugepage_vma_check() to huge_memory.c and huge_mm.h since it is not only for khugepaged anymore. Reviewed-by: Zach O'Keefe Signed-off-by: Yang Shi --- fs/proc/task_mmu.c | 2 +- include/linux/huge_mm.h | 16 +++++++----- include/linux/khugepaged.h | 2 -- mm/huge_memory.c | 50 +++++++++++++++++++++++++++++++------- mm/khugepaged.c | 48 +++--------------------------------- 5 files changed, 56 insertions(+), 62 deletions(-) diff --git a/fs/proc/task_mmu.c b/fs/proc/task_mmu.c index 37ccb5c9f4f8..39a40ec181e7 100644 --- a/fs/proc/task_mmu.c +++ b/fs/proc/task_mmu.c @@ -863,7 +863,7 @@ static int show_smap(struct seq_file *m, void *v) __show_smap(m, &mss, false); =20 seq_printf(m, "THPeligible: %d\n", - transparent_hugepage_active(vma)); + hugepage_vma_check(vma, vma->vm_flags, true)); =20 if (arch_pkeys_enabled()) seq_printf(m, "ProtectionKey: %8u\n", vma_pkey(vma)); diff --git a/include/linux/huge_mm.h b/include/linux/huge_mm.h index 8a5a8bfce0f5..64487bcd0c7b 100644 --- a/include/linux/huge_mm.h +++ b/include/linux/huge_mm.h @@ -202,7 +202,9 @@ static inline bool file_thp_enabled(struct vm_area_stru= ct *vma) !inode_is_open_for_write(inode) && S_ISREG(inode->i_mode); } =20 -bool transparent_hugepage_active(struct vm_area_struct *vma); +bool hugepage_vma_check(struct vm_area_struct *vma, + unsigned long vm_flags, + bool smaps); =20 #define transparent_hugepage_use_zero_page() \ (transparent_hugepage_flags & \ @@ -351,11 +353,6 @@ static inline bool __transparent_hugepage_enabled(stru= ct vm_area_struct *vma) return false; } =20 -static inline bool transparent_hugepage_active(struct vm_area_struct *vma) -{ - return false; -} - static inline bool transhuge_vma_suitable(struct vm_area_struct *vma, unsigned long addr) { @@ -368,6 +365,13 @@ static inline bool transhuge_vma_enabled(struct vm_are= a_struct *vma, return false; } =20 +static inline bool hugepage_vma_check(struct vm_area_struct *vma, + unsigned long vm_flags, + bool smaps) +{ + return false; +} + static inline void prep_transhuge_page(struct page *page) {} =20 #define transparent_hugepage_flags 0UL diff --git a/include/linux/khugepaged.h b/include/linux/khugepaged.h index 31ca8a7f78f4..ea5fd4c398f7 100644 --- a/include/linux/khugepaged.h +++ b/include/linux/khugepaged.h @@ -10,8 +10,6 @@ extern struct attribute_group khugepaged_attr_group; extern int khugepaged_init(void); extern void khugepaged_destroy(void); extern int start_stop_khugepaged(void); -extern bool hugepage_vma_check(struct vm_area_struct *vma, - unsigned long vm_flags); extern void __khugepaged_enter(struct mm_struct *mm); extern void __khugepaged_exit(struct mm_struct *mm); extern void khugepaged_enter_vma(struct vm_area_struct *vma, diff --git a/mm/huge_memory.c b/mm/huge_memory.c index b530462c4493..a28c6100b491 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -69,21 +69,53 @@ static atomic_t huge_zero_refcount; struct page *huge_zero_page __read_mostly; unsigned long huge_zero_pfn __read_mostly =3D ~0UL; =20 -bool transparent_hugepage_active(struct vm_area_struct *vma) +bool hugepage_vma_check(struct vm_area_struct *vma, + unsigned long vm_flags, + bool smaps) { - /* The addr is used to check if the vma size fits */ - unsigned long addr =3D (vma->vm_end & HPAGE_PMD_MASK) - HPAGE_PMD_SIZE; + if (!transhuge_vma_enabled(vma, vm_flags)) + return false; + + if (vm_flags & VM_NO_KHUGEPAGED) + return false; + + /* Don't run khugepaged against DAX vma */ + if (vma_is_dax(vma)) + return false; =20 - if (!transhuge_vma_suitable(vma, addr)) + /* Check alignment for file vma and size for both file and anon vma */ + if (!transhuge_vma_suitable(vma, (vma->vm_end - HPAGE_PMD_SIZE))) return false; - if (vma_is_anonymous(vma)) - return __transparent_hugepage_enabled(vma); - if (vma_is_shmem(vma)) + + /* Enabled via shmem mount options or sysfs settings. */ + if (shmem_file(vma->vm_file)) return shmem_huge_enabled(vma); - if (transhuge_vma_enabled(vma, vma->vm_flags) && file_thp_enabled(vma)) + + if (!khugepaged_enabled()) + return false; + + /* THP settings require madvise. */ + if (!(vm_flags & VM_HUGEPAGE) && !khugepaged_always()) + return false; + + /* Only regular file is valid */ + if (file_thp_enabled(vma)) return true; =20 - return false; + if (!vma_is_anonymous(vma)) + return false; + + if (vma_is_temporary_stack(vma)) + return false; + + /* + * THPeligible bit of smaps should show 1 for proper VMAs even + * though anon_vma is not initialized yet. + */ + if (!vma->anon_vma) + return smaps; + + return true; } =20 static bool get_huge_zero_page(void) diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 5baa394e34c8..3afd87f8c0b1 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -437,46 +437,6 @@ static inline int khugepaged_test_exit(struct mm_struc= t *mm) return atomic_read(&mm->mm_users) =3D=3D 0; } =20 -bool hugepage_vma_check(struct vm_area_struct *vma, - unsigned long vm_flags) -{ - if (!transhuge_vma_enabled(vma, vm_flags)) - return false; - - if (vm_flags & VM_NO_KHUGEPAGED) - return false; - - /* Don't run khugepaged against DAX vma */ - if (vma_is_dax(vma)) - return false; - - /* Check alignment for file vma and size for both file and anon vma */ - if (!transhuge_vma_suitable(vma, (vma->vm_end - HPAGE_PMD_SIZE))) - return false; - - /* Enabled via shmem mount options or sysfs settings. */ - if (shmem_file(vma->vm_file)) - return shmem_huge_enabled(vma); - - if (!khugepaged_enabled()) - return false; - - /* THP settings require madvise. */ - if (!(vm_flags & VM_HUGEPAGE) && !khugepaged_always()) - return false; - - /* Only regular file is valid */ - if (file_thp_enabled(vma)) - return true; - - if (!vma->anon_vma || !vma_is_anonymous(vma)) - return false; - if (vma_is_temporary_stack(vma)) - return false; - - return true; -} - void __khugepaged_enter(struct mm_struct *mm) { struct mm_slot *mm_slot; @@ -513,7 +473,7 @@ void khugepaged_enter_vma(struct vm_area_struct *vma, { if (!test_bit(MMF_VM_HUGEPAGE, &vma->vm_mm->flags) && khugepaged_enabled()) { - if (hugepage_vma_check(vma, vm_flags)) + if (hugepage_vma_check(vma, vm_flags, false)) __khugepaged_enter(vma->vm_mm); } } @@ -958,7 +918,7 @@ static int hugepage_vma_revalidate(struct mm_struct *mm= , unsigned long address, =20 if (!transhuge_vma_suitable(vma, address)) return SCAN_ADDRESS_RANGE; - if (!hugepage_vma_check(vma, vma->vm_flags)) + if (!hugepage_vma_check(vma, vma->vm_flags, false)) return SCAN_VMA_CHECK; /* * Anon VMA expected, the address may be unmapped then @@ -1448,7 +1408,7 @@ void collapse_pte_mapped_thp(struct mm_struct *mm, un= signed long addr) * the valid THP. Add extra VM_HUGEPAGE so hugepage_vma_check() * will not fail the vma for missing VM_HUGEPAGE */ - if (!hugepage_vma_check(vma, vma->vm_flags | VM_HUGEPAGE)) + if (!hugepage_vma_check(vma, vma->vm_flags | VM_HUGEPAGE, false)) return; =20 /* Keep pmd pgtable for uffd-wp; see comment in retract_page_tables() */ @@ -2143,7 +2103,7 @@ static unsigned int khugepaged_scan_mm_slot(unsigned = int pages, progress++; break; } - if (!hugepage_vma_check(vma, vma->vm_flags)) { + if (!hugepage_vma_check(vma, vma->vm_flags, false)) { skip: progress++; continue; --=20 2.26.3 From nobody Mon Apr 27 04:46:48 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7BF65C433EF for ; Thu, 16 Jun 2022 17:49:21 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238485AbiFPRtT (ORCPT ); Thu, 16 Jun 2022 13:49:19 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:53732 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237406AbiFPRs5 (ORCPT ); Thu, 16 Jun 2022 13:48:57 -0400 Received: from mail-pg1-x52c.google.com (mail-pg1-x52c.google.com [IPv6:2607:f8b0:4864:20::52c]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 309854D60F for ; Thu, 16 Jun 2022 10:48:56 -0700 (PDT) Received: by mail-pg1-x52c.google.com with SMTP id h192so1898934pgc.4 for ; Thu, 16 Jun 2022 10:48:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=9rMZNhDmVhFfYI3dZDinmpFWJ+654cXj29ZWajToGJQ=; b=k54UC37gpxXu1HC6M7ieaY9VAO7o/YPUOu3hXBYkVkv9C9ybJ7kPZYEOP8IFLnqa/K b7SIBpXhgLFueHs3dPgXuUiOOqsWOJml11FXjJAmxamtSAesoQTB/a6eYS0IOrmmlW3e baHSJK3GJFK4JfnGIsv3RSKvHe4rsDTJkVQQNJb2ag7cv7IfjK4EtZiTm8uCh9tK/t1R hPTvYNDOs8R4yWRNcO9HttLDz1hx57EdefHssGDrstlaCNpwL44FwCRn5in6w/PrA+3V 6T/kd19emhBUlP+XHDsb2pnQCRpRfCDXfhv9VXTz+/fHjJJsirE5df6xjGI9jd5nA5UQ nIgA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=9rMZNhDmVhFfYI3dZDinmpFWJ+654cXj29ZWajToGJQ=; b=VZajzeYdrC5EWS40+HFiOvr3fHzbzz9VpPaSDh9Wd/2EM+XodzF5OiovvKDaRIeCW3 KqPmwA6jASyPiLUJasy8c4/aqevTzPLVrJkkdoSTcdz+6iOuH35OesKFjkWZeej/KqKe MyV1uwX8Mn6DdK0/x60R2+ve6EuZnAUa8QUX7zlmrT2QfeG4KA9ICd4EAS5fsSf8QnXb thWiOfKeBToe8lRz2ox1dL/aLvWRF5p1YocF98ZIy8J03ZseySZkHgS5Ybwt/xc4DbPh kDKlD5GnxYIiaB0WwA0SBAojjpEyxIcHtyPK8yi6XfWSKWOWEBKiGPnjuP/n0IxODWYb dlaA== X-Gm-Message-State: AJIora/qAeatectW2vF6HA2VQMbx/fkjVzhg7IlrDlJYaJ2Q5sZgGqwY v9/O2O4WHRmn5ZKqj3fDHA9KIuMa3sQ= X-Google-Smtp-Source: AGRyM1vl4tOydvP7hK0KqRHMn/OYMwWHFtGqeiwYKuHJr2LotWCPXbJStaJQm1Mior+02JTeRp6U3w== X-Received: by 2002:a05:6a00:98b:b0:51b:d730:c58 with SMTP id u11-20020a056a00098b00b0051bd7300c58mr6008432pfg.23.1655401735644; Thu, 16 Jun 2022 10:48:55 -0700 (PDT) Received: from localhost.localdomain (c-67-174-241-145.hsd1.ca.comcast.net. [67.174.241.145]) by smtp.gmail.com with ESMTPSA id z21-20020a17090a8b9500b001e8520b211bsm1818289pjn.53.2022.06.16.10.48.54 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 16 Jun 2022 10:48:54 -0700 (PDT) From: Yang Shi To: vbabka@suse.cz, kirill.shutemov@linux.intel.com, willy@infradead.org, zokeefe@google.com, linmiaohe@huawei.com, akpm@linux-foundation.org Cc: shy828301@gmail.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [v5 PATCH 5/7] mm: thp: kill __transhuge_page_enabled() Date: Thu, 16 Jun 2022 10:48:38 -0700 Message-Id: <20220616174840.1202070-6-shy828301@gmail.com> X-Mailer: git-send-email 2.26.3 In-Reply-To: <20220616174840.1202070-1-shy828301@gmail.com> References: <20220616174840.1202070-1-shy828301@gmail.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" The page fault path checks THP eligibility with __transhuge_page_enabled() which does the similar thing as hugepage_vma_check(), so use hugepage_vma_check() instead. However page fault allows DAX and !anon_vma cases, so added a new flag, in_pf, to hugepage_vma_check() to make page fault work correctly. The in_pf flag is also used to skip shmem and file THP for page fault since shmem handles THP in its own shmem_fault() and file THP allocation on fault is not supported yet. Also remove hugepage_vma_enabled() since hugepage_vma_check() is the only caller now, it is not necessary to have a helper function. Reviewed-by: Zach O'Keefe Signed-off-by: Yang Shi --- fs/proc/task_mmu.c | 2 +- include/linux/huge_mm.h | 57 ++--------------------------------------- mm/huge_memory.c | 51 ++++++++++++++++++++++++++++-------- mm/khugepaged.c | 8 +++--- mm/memory.c | 7 +++-- 5 files changed, 52 insertions(+), 73 deletions(-) diff --git a/fs/proc/task_mmu.c b/fs/proc/task_mmu.c index 39a40ec181e7..cef72e49acc5 100644 --- a/fs/proc/task_mmu.c +++ b/fs/proc/task_mmu.c @@ -863,7 +863,7 @@ static int show_smap(struct seq_file *m, void *v) __show_smap(m, &mss, false); =20 seq_printf(m, "THPeligible: %d\n", - hugepage_vma_check(vma, vma->vm_flags, true)); + hugepage_vma_check(vma, vma->vm_flags, true, false)); =20 if (arch_pkeys_enabled()) seq_printf(m, "ProtectionKey: %8u\n", vma_pkey(vma)); diff --git a/include/linux/huge_mm.h b/include/linux/huge_mm.h index 64487bcd0c7b..cd8a6c5d9fe5 100644 --- a/include/linux/huge_mm.h +++ b/include/linux/huge_mm.h @@ -146,48 +146,6 @@ static inline bool transhuge_vma_suitable(struct vm_ar= ea_struct *vma, return true; } =20 -static inline bool transhuge_vma_enabled(struct vm_area_struct *vma, - unsigned long vm_flags) -{ - /* Explicitly disabled through madvise. */ - if ((vm_flags & VM_NOHUGEPAGE) || - test_bit(MMF_DISABLE_THP, &vma->vm_mm->flags)) - return false; - return true; -} - -/* - * to be used on vmas which are known to support THP. - * Use transparent_hugepage_active otherwise - */ -static inline bool __transparent_hugepage_enabled(struct vm_area_struct *v= ma) -{ - - /* - * If the hardware/firmware marked hugepage support disabled. - */ - if (transparent_hugepage_flags & (1 << TRANSPARENT_HUGEPAGE_NEVER_DAX)) - return false; - - if (!transhuge_vma_enabled(vma, vma->vm_flags)) - return false; - - if (vma_is_temporary_stack(vma)) - return false; - - if (transparent_hugepage_flags & (1 << TRANSPARENT_HUGEPAGE_FLAG)) - return true; - - if (vma_is_dax(vma)) - return true; - - if (transparent_hugepage_flags & - (1 << TRANSPARENT_HUGEPAGE_REQ_MADV_FLAG)) - return !!(vma->vm_flags & VM_HUGEPAGE); - - return false; -} - static inline bool file_thp_enabled(struct vm_area_struct *vma) { struct inode *inode; @@ -204,7 +162,7 @@ static inline bool file_thp_enabled(struct vm_area_stru= ct *vma) =20 bool hugepage_vma_check(struct vm_area_struct *vma, unsigned long vm_flags, - bool smaps); + bool smaps, bool in_pf); =20 #define transparent_hugepage_use_zero_page() \ (transparent_hugepage_flags & \ @@ -348,26 +306,15 @@ static inline bool folio_test_pmd_mappable(struct fol= io *folio) return false; } =20 -static inline bool __transparent_hugepage_enabled(struct vm_area_struct *v= ma) -{ - return false; -} - static inline bool transhuge_vma_suitable(struct vm_area_struct *vma, unsigned long addr) { return false; } =20 -static inline bool transhuge_vma_enabled(struct vm_area_struct *vma, - unsigned long vm_flags) -{ - return false; -} - static inline bool hugepage_vma_check(struct vm_area_struct *vma, unsigned long vm_flags, - bool smaps) + bool smaps, bool in_pf) { return false; } diff --git a/mm/huge_memory.c b/mm/huge_memory.c index a28c6100b491..d0c37d99917b 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -71,24 +71,50 @@ unsigned long huge_zero_pfn __read_mostly =3D ~0UL; =20 bool hugepage_vma_check(struct vm_area_struct *vma, unsigned long vm_flags, - bool smaps) + bool smaps, bool in_pf) { - if (!transhuge_vma_enabled(vma, vm_flags)) + /* + * Explicitly disabled through madvise or prctl, or some + * architectures may disable THP for some mappings, for + * example, s390 kvm. + * */ + if ((vm_flags & VM_NOHUGEPAGE) || + test_bit(MMF_DISABLE_THP, &vma->vm_mm->flags)) return false; - - if (vm_flags & VM_NO_KHUGEPAGED) + /* + * If the hardware/firmware marked hugepage support disabled. + */ + if (transparent_hugepage_flags & (1 << TRANSPARENT_HUGEPAGE_NEVER_DAX)) return false; =20 - /* Don't run khugepaged against DAX vma */ + /* khugepaged doesn't collapse DAX vma, but page fault is fine. */ if (vma_is_dax(vma)) + return in_pf; + + /* + * Special VMA and hugetlb VMA. + * Must be checked after dax since some dax mappings may have + * VM_MIXEDMAP set. + */ + if (vm_flags & VM_NO_KHUGEPAGED) return false; =20 - /* Check alignment for file vma and size for both file and anon vma */ - if (!transhuge_vma_suitable(vma, (vma->vm_end - HPAGE_PMD_SIZE))) + /* + * Check alignment for file vma and size for both file and anon vma. + * + * Skip the check for page fault. Huge fault does the check in fault + * handlers. And this check is not suitable for huge PUD fault. + */ + if (!in_pf && + !transhuge_vma_suitable(vma, (vma->vm_end - HPAGE_PMD_SIZE))) return false; =20 - /* Enabled via shmem mount options or sysfs settings. */ - if (shmem_file(vma->vm_file)) + /* + * Enabled via shmem mount options or sysfs settings. + * Must be done before hugepage flags check since shmem has its + * own flags. + */ + if (!in_pf && shmem_file(vma->vm_file)) return shmem_huge_enabled(vma); =20 if (!khugepaged_enabled()) @@ -99,7 +125,7 @@ bool hugepage_vma_check(struct vm_area_struct *vma, return false; =20 /* Only regular file is valid */ - if (file_thp_enabled(vma)) + if (!in_pf && file_thp_enabled(vma)) return true; =20 if (!vma_is_anonymous(vma)) @@ -111,9 +137,12 @@ bool hugepage_vma_check(struct vm_area_struct *vma, /* * THPeligible bit of smaps should show 1 for proper VMAs even * though anon_vma is not initialized yet. + * + * Allow page fault since anon_vma may be not initialized until + * the first page fault. */ if (!vma->anon_vma) - return smaps; + return (smaps || in_pf); =20 return true; } diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 3afd87f8c0b1..2a676f37c921 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -473,7 +473,7 @@ void khugepaged_enter_vma(struct vm_area_struct *vma, { if (!test_bit(MMF_VM_HUGEPAGE, &vma->vm_mm->flags) && khugepaged_enabled()) { - if (hugepage_vma_check(vma, vm_flags, false)) + if (hugepage_vma_check(vma, vm_flags, false, false)) __khugepaged_enter(vma->vm_mm); } } @@ -918,7 +918,7 @@ static int hugepage_vma_revalidate(struct mm_struct *mm= , unsigned long address, =20 if (!transhuge_vma_suitable(vma, address)) return SCAN_ADDRESS_RANGE; - if (!hugepage_vma_check(vma, vma->vm_flags, false)) + if (!hugepage_vma_check(vma, vma->vm_flags, false, false)) return SCAN_VMA_CHECK; /* * Anon VMA expected, the address may be unmapped then @@ -1408,7 +1408,7 @@ void collapse_pte_mapped_thp(struct mm_struct *mm, un= signed long addr) * the valid THP. Add extra VM_HUGEPAGE so hugepage_vma_check() * will not fail the vma for missing VM_HUGEPAGE */ - if (!hugepage_vma_check(vma, vma->vm_flags | VM_HUGEPAGE, false)) + if (!hugepage_vma_check(vma, vma->vm_flags | VM_HUGEPAGE, false, false)) return; =20 /* Keep pmd pgtable for uffd-wp; see comment in retract_page_tables() */ @@ -2103,7 +2103,7 @@ static unsigned int khugepaged_scan_mm_slot(unsigned = int pages, progress++; break; } - if (!hugepage_vma_check(vma, vma->vm_flags, false)) { + if (!hugepage_vma_check(vma, vma->vm_flags, false, false)) { skip: progress++; continue; diff --git a/mm/memory.c b/mm/memory.c index be724238a9d3..fee2884481f2 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -4985,6 +4985,7 @@ static vm_fault_t __handle_mm_fault(struct vm_area_st= ruct *vma, .gfp_mask =3D __get_fault_gfp_mask(vma), }; struct mm_struct *mm =3D vma->vm_mm; + unsigned long vm_flags =3D vma->vm_flags; pgd_t *pgd; p4d_t *p4d; vm_fault_t ret; @@ -4998,7 +4999,8 @@ static vm_fault_t __handle_mm_fault(struct vm_area_st= ruct *vma, if (!vmf.pud) return VM_FAULT_OOM; retry_pud: - if (pud_none(*vmf.pud) && __transparent_hugepage_enabled(vma)) { + if (pud_none(*vmf.pud) && + hugepage_vma_check(vma, vm_flags, false, true)) { ret =3D create_huge_pud(&vmf); if (!(ret & VM_FAULT_FALLBACK)) return ret; @@ -5031,7 +5033,8 @@ static vm_fault_t __handle_mm_fault(struct vm_area_st= ruct *vma, if (pud_trans_unstable(vmf.pud)) goto retry_pud; =20 - if (pmd_none(*vmf.pmd) && __transparent_hugepage_enabled(vma)) { + if (pmd_none(*vmf.pmd) && + hugepage_vma_check(vma, vm_flags, false, true)) { ret =3D create_huge_pmd(&vmf); if (!(ret & VM_FAULT_FALLBACK)) return ret; --=20 2.26.3 From nobody Mon Apr 27 04:46:48 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 43E02C43334 for ; Thu, 16 Jun 2022 17:49:28 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237297AbiFPRt1 (ORCPT ); Thu, 16 Jun 2022 13:49:27 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:53790 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237614AbiFPRtA (ORCPT ); Thu, 16 Jun 2022 13:49:00 -0400 Received: from mail-pf1-x432.google.com (mail-pf1-x432.google.com [IPv6:2607:f8b0:4864:20::432]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id AF2064CD69 for ; Thu, 16 Jun 2022 10:48:57 -0700 (PDT) Received: by mail-pf1-x432.google.com with SMTP id y196so2111057pfb.6 for ; Thu, 16 Jun 2022 10:48:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=QRTCWUz/v80cwRev6O447pCBMSdMFGV94XhB5oNBreQ=; b=Fm5abIw414WczmkiARiuOIB9/0ASIunFH2zFKcRTz+KBNKvcBYcmyc+rwz37WtNCMN Q91YdwCifI/utdZj36nMuMzQ4+xEtVBzz4U+2d8nLuOBijN4D6KkGIrGsfWfJqgzx88i N4serWC67hGg9tGb1K1E+W3IM/ktOQLXj8AjMMv2xSPVvxkLY8yTbIBB23hujhfKIjuF qR8R2Fl5Vg/GOWk7BTeueqJwqM3OPvf3ZTwxKrBpWmtBKIO+6AXuItMuxGrDC5dm3FVJ yOYhpdT39E1dzHRcviaiO05a4p0BqyMWfcINITHFuS6WnG9CxsdiuWgtrO7U4uWx+vZy 185Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=QRTCWUz/v80cwRev6O447pCBMSdMFGV94XhB5oNBreQ=; b=RcmLyNfMWN8i2OHDt9mbyCtdo/B5lUCdOZ2vFyY7vUXBoicyJjMCapRPsBugCQVa+/ d1Tlvqu94RtkYMolSD/xFKj9Ozk1ZRCpaZp5LaA5XMEuDkpMcMKdMS740VLG0Egu40GB U5sqJMt0VsuLZ4hZo/H6+ciojfxlcy72nEbjymcCehelEzM7hDHY9buWKE1S/087h7g7 RX2hMgmGbGrbd+L7+Q2BEQ9TfGtx61ACvAI8fETB+H3qjqaqjWXBpVmNu+qvxiF76+jg ngOV0AJ0mvdljzTu0q41hqTJyKpawAEDdMAlOheJsVB8UNFcNEPz71B67sBJgVBfm1X4 ugvQ== X-Gm-Message-State: AJIora+HfmWwW9bDFVXE3m1IrSUb+wU0nnPU+J3tUlm5LPZD6kls069A 5QfkKkfCNeCSo9UgGJgiztU= X-Google-Smtp-Source: AGRyM1vnH5OfaqcNXG9WeXIL0jSqmq4SmAlVZPgdAgD+zxW+JwxF7guIJVE0p/eybVg5QN/5LR3qRg== X-Received: by 2002:a05:6a00:1513:b0:51c:3ca8:47a4 with SMTP id q19-20020a056a00151300b0051c3ca847a4mr5967135pfu.48.1655401737201; Thu, 16 Jun 2022 10:48:57 -0700 (PDT) Received: from localhost.localdomain (c-67-174-241-145.hsd1.ca.comcast.net. [67.174.241.145]) by smtp.gmail.com with ESMTPSA id z21-20020a17090a8b9500b001e8520b211bsm1818289pjn.53.2022.06.16.10.48.55 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 16 Jun 2022 10:48:56 -0700 (PDT) From: Yang Shi To: vbabka@suse.cz, kirill.shutemov@linux.intel.com, willy@infradead.org, zokeefe@google.com, linmiaohe@huawei.com, akpm@linux-foundation.org Cc: shy828301@gmail.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [v5 PATCH 6/7] mm: khugepaged: reorg some khugepaged helpers Date: Thu, 16 Jun 2022 10:48:39 -0700 Message-Id: <20220616174840.1202070-7-shy828301@gmail.com> X-Mailer: git-send-email 2.26.3 In-Reply-To: <20220616174840.1202070-1-shy828301@gmail.com> References: <20220616174840.1202070-1-shy828301@gmail.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" The khugepaged_{enabled|always|req_madv} are not khugepaged only anymore, move them to huge_mm.h and rename to hugepage_flags_xxx, and remove khugepaged_req_madv due to no users. Also move khugepaged_defrag to khugepaged.c since its only caller is in that file, it doesn't have to be in a header file. Reviewed-by: Zach O'Keefe Signed-off-by: Yang Shi --- include/linux/huge_mm.h | 8 ++++++++ include/linux/khugepaged.h | 14 -------------- mm/huge_memory.c | 4 ++-- mm/khugepaged.c | 18 +++++++++++------- 4 files changed, 21 insertions(+), 23 deletions(-) diff --git a/include/linux/huge_mm.h b/include/linux/huge_mm.h index cd8a6c5d9fe5..ae3d8e2fd9e2 100644 --- a/include/linux/huge_mm.h +++ b/include/linux/huge_mm.h @@ -116,6 +116,14 @@ extern struct kobj_attribute shmem_enabled_attr; =20 extern unsigned long transparent_hugepage_flags; =20 +#define hugepage_flags_enabled() \ + (transparent_hugepage_flags & \ + ((1<flags)) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index d0c37d99917b..0f2cce2d7041 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -117,11 +117,11 @@ bool hugepage_vma_check(struct vm_area_struct *vma, if (!in_pf && shmem_file(vma->vm_file)) return shmem_huge_enabled(vma); =20 - if (!khugepaged_enabled()) + if (!hugepage_flags_enabled()) return false; =20 /* THP settings require madvise. */ - if (!(vm_flags & VM_HUGEPAGE) && !khugepaged_always()) + if (!(vm_flags & VM_HUGEPAGE) && !hugepage_flags_always()) return false; =20 /* Only regular file is valid */ diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 2a676f37c921..d8ebb60aae36 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -472,7 +472,7 @@ void khugepaged_enter_vma(struct vm_area_struct *vma, unsigned long vm_flags) { if (!test_bit(MMF_VM_HUGEPAGE, &vma->vm_mm->flags) && - khugepaged_enabled()) { + hugepage_flags_enabled()) { if (hugepage_vma_check(vma, vm_flags, false, false)) __khugepaged_enter(vma->vm_mm); } @@ -763,6 +763,10 @@ static bool khugepaged_scan_abort(int nid) return false; } =20 +#define khugepaged_defrag() \ + (transparent_hugepage_flags & \ + (1< X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 12208C43334 for ; Thu, 16 Jun 2022 17:49:35 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1347485AbiFPRtc (ORCPT ); Thu, 16 Jun 2022 13:49:32 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:53846 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238181AbiFPRtC (ORCPT ); Thu, 16 Jun 2022 13:49:02 -0400 Received: from mail-pf1-x42d.google.com (mail-pf1-x42d.google.com [IPv6:2607:f8b0:4864:20::42d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 417764D621 for ; Thu, 16 Jun 2022 10:48:59 -0700 (PDT) Received: by mail-pf1-x42d.google.com with SMTP id u2so2123544pfc.2 for ; Thu, 16 Jun 2022 10:48:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=cbrfBk2owseXTl2TqGluY3ivP+501euUdLqfR5FVpCc=; b=F7+G3ESb9pcGIuGXoF2O/H03YvYsGpLhfNOWSQqYVP5yFLNGleH39xqFQ8cWIZuFho xg3heTbq6ePKtRc8GFH0juCyjezdh2hqyFjPBLzT9941kxHwTK7yLUeKVkiQKo6mDCxu se3LMMGqxW6HiTvu93X33WpwUBs73ecdVSU001ye3aHuc0r0loPqN+9Uj09NMTU9BWna ENe8DNgwYHvtpA6ITqyCZUYZN8mAOAXoWKj0lirO78K389Cubg2uLLRLQv4c9eITuXpW hzCgRhi/7HJwNubd85XqcCfY+TlKwnYPFZ+SI68tWv28Ig/OEhTEMpCtXj3Cf1NQPEVQ Vnzg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=cbrfBk2owseXTl2TqGluY3ivP+501euUdLqfR5FVpCc=; b=ekDrxr157AJEaTTiz0OPhPOuIA4q6udd4h29O/6yfcrEEa//ud6wnYaAKnPXHnVIgr YXeIoVtpapnietsgiErQy9dfMBI2aCBVBEBIxIzDgjVbGjMEH27I5ddypl4EYysqat0u 6GwcFrJ6Wgy364n/Za3Co1WYW+P3b8kDgLkYwgxEP5+K7pRFXcfGcjmRuehJ6neMqLG5 88+hd6j0uoO5Oe9MXftR11d9FOa04f+wMI5N1tyPiNMOYm3XxzQ9NbuzKYubRomQ3Q+8 xv50MClGSLtaTYnXKcwXlnpS40tiqFcUz/lBThk2HtvWttZt6A0g9NALUGR2F7rzaCRy JV8w== X-Gm-Message-State: AJIora92FRQ7oxvDSvyZHxDBpjVuUfEBMeYqGKveavyHniKK4XCCyzHQ 2rzBuC//FKb2wbI3O0zR1Xo= X-Google-Smtp-Source: AGRyM1vtFhWJ5FLzx5f3p3o/qmSs8NJRmjg3vyIZZSoaHxETIbvnT3f9vP9XDu44HCWoj/kKASXWTg== X-Received: by 2002:aa7:88cd:0:b0:51c:188e:fbeb with SMTP id k13-20020aa788cd000000b0051c188efbebmr5782611pff.81.1655401738804; Thu, 16 Jun 2022 10:48:58 -0700 (PDT) Received: from localhost.localdomain (c-67-174-241-145.hsd1.ca.comcast.net. [67.174.241.145]) by smtp.gmail.com with ESMTPSA id z21-20020a17090a8b9500b001e8520b211bsm1818289pjn.53.2022.06.16.10.48.57 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 16 Jun 2022 10:48:58 -0700 (PDT) From: Yang Shi To: vbabka@suse.cz, kirill.shutemov@linux.intel.com, willy@infradead.org, zokeefe@google.com, linmiaohe@huawei.com, akpm@linux-foundation.org Cc: shy828301@gmail.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [v5 PATCH 7/7] doc: proc: fix the description to THPeligible Date: Thu, 16 Jun 2022 10:48:40 -0700 Message-Id: <20220616174840.1202070-8-shy828301@gmail.com> X-Mailer: git-send-email 2.26.3 In-Reply-To: <20220616174840.1202070-1-shy828301@gmail.com> References: <20220616174840.1202070-1-shy828301@gmail.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" The THPeligible bit shows 1 if and only if the VMA is eligible for allocating THP and the THP is also PMD mappable. Some misaligned file VMAs may be eligible for allocating THP but the THP can't be mapped by PMD. Make this more explictly to avoid ambiguity. Reviewed-by: Zach O'Keefe Signed-off-by: Yang Shi --- Documentation/filesystems/proc.rst | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/Documentation/filesystems/proc.rst b/Documentation/filesystems= /proc.rst index 1bc91fb8c321..a5e41e636a1a 100644 --- a/Documentation/filesystems/proc.rst +++ b/Documentation/filesystems/proc.rst @@ -514,8 +514,10 @@ replaced by copy-on-write) part of the underlying shme= m object out on swap. "SwapPss" shows proportional swap share of this mapping. Unlike "Swap", th= is does not take into account swapped out page of underlying shmem objects. "Locked" indicates whether the mapping is locked in memory or not. + "THPeligible" indicates whether the mapping is eligible for allocating THP -pages - 1 if true, 0 otherwise. It just shows the current status. +pages as well as the THP is PMD mappable or not - 1 if true, 0 otherwise. +It just shows the current status. =20 "VmFlags" field deserves a separate description. This member represents the kernel flags associated with the particular virtual memory area in two let= ter --=20 2.26.3