From nobody Tue Sep 16 08:40:27 2025 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id F04F7C54EBC for ; Wed, 4 Jan 2023 22:56:57 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234661AbjADW43 (ORCPT ); Wed, 4 Jan 2023 17:56:29 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55380 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234576AbjADW4L (ORCPT ); Wed, 4 Jan 2023 17:56:11 -0500 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7CFE24437D for ; Wed, 4 Jan 2023 14:55:25 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1672872924; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=iUQecF0p0KDaGb3ic+3dlRjRyqZfOtSt4XLNwuIXR0Y=; b=eKUy64RdDxZuGHGRVfko63JMu8nXyYKDgA8xffbzTgoMxxK91i7THAk2ztsUgR8iQWuRY7 AMD/ucEp788aDf49h0BvU8D8XSF/maGYquUYrc7dhfv4uEhGesHBlB4TsDxKiiJuO3TKyb 3E5aVl5MGn7yBzDPKPDvrlkS3pWiNhY= Received: from mail-qv1-f69.google.com (mail-qv1-f69.google.com [209.85.219.69]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_128_GCM_SHA256) id us-mta-625-QnrUK-fnMuKlk5MHAX28yA-1; Wed, 04 Jan 2023 17:52:13 -0500 X-MC-Unique: QnrUK-fnMuKlk5MHAX28yA-1 Received: by mail-qv1-f69.google.com with SMTP id o95-20020a0c9068000000b005320eb4e959so471614qvo.16 for ; Wed, 04 Jan 2023 14:52:13 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=iUQecF0p0KDaGb3ic+3dlRjRyqZfOtSt4XLNwuIXR0Y=; b=65SkchlFOIpuEPwKDD5DPfLgCYckfo+sd+ZF6TfIyGW6TBTH/pm4a5wsIS12kn57uf it9/pYGKv267DB/G0Q/820LcVk+DWsvRSittCkm3Tf5Aylk4jsTncfe5tdu9oKHHbBvl Zi96C2F8tUMmaTGglZIIzH0X44SjfBJLRAsv0fFoxJEs1xwGGnwt0VshHub3e+GQwUL/ 1T8KofVkaKSMi9UIktQgEumSb/+K65z9ii3loN3le9x7dSJfEdEt7PQRXbNz1YFJ4OBV IV0sISWZV5X/e7J5w7QNGHCmAHuPCf77dmGbPEVPz27pJJUwiso8uhNOLTP0PS52x0e2 RmQA== X-Gm-Message-State: AFqh2kr8gz9TT5gE2WiDLi3acFTEZ3dkncx9OhqsFvs3Qz1DidDYj4LI r42Y0hOV01p/ZqvPaC7k0gtyK3n9lxkN/v1FBOHi55n8qhZDiSUAQhhAVzmWNkmTo1mZDxJc9sb sQ2nWn4XSzl73OsYKqpo8FIoQ X-Received: by 2002:a0c:f3ce:0:b0:531:7b1e:cdaa with SMTP id f14-20020a0cf3ce000000b005317b1ecdaamr56474332qvm.44.1672872732866; Wed, 04 Jan 2023 14:52:12 -0800 (PST) X-Google-Smtp-Source: AMrXdXvpKKemP1V8IrIbogqNKolIRqZbJHS4f3LCuRtRb0RPSmf1nkmMiOVux8MqIUMl3pEtxplS2A== X-Received: by 2002:a0c:f3ce:0:b0:531:7b1e:cdaa with SMTP id f14-20020a0cf3ce000000b005317b1ecdaamr56474310qvm.44.1672872732634; Wed, 04 Jan 2023 14:52:12 -0800 (PST) Received: from x1n.redhat.com (bras-base-aurron9127w-grc-39-70-52-228-144.dsl.bell.ca. [70.52.228.144]) by smtp.gmail.com with ESMTPSA id r1-20020a05620a298100b006eeb3165565sm24654710qkp.80.2023.01.04.14.52.10 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 04 Jan 2023 14:52:11 -0800 (PST) From: Peter Xu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Mike Kravetz , Muchun Song , peterx@redhat.com, Nadav Amit , Andrea Arcangeli , David Hildenbrand , James Houghton , Axel Rasmussen , Andrew Morton , linux-stable Subject: [PATCH 1/3] mm/hugetlb: Pre-allocate pgtable pages for uffd wr-protects Date: Wed, 4 Jan 2023 17:52:05 -0500 Message-Id: <20230104225207.1066932-2-peterx@redhat.com> X-Mailer: git-send-email 2.37.3 In-Reply-To: <20230104225207.1066932-1-peterx@redhat.com> References: <20230104225207.1066932-1-peterx@redhat.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Userfaultfd-wp uses pte markers to mark wr-protected pages for both shmem and hugetlb. Shmem has pre-allocation ready for markers, but hugetlb path was overlooked. Doing so by calling huge_pte_alloc() if the initial pgtable walk fails to find the huge ptep. It's possible that huge_pte_alloc() can fail with high memory pressure, in that case stop the loop immediately and fail silently. This is not the most ideal solution but it matches with what we do with shmem meanwhile it avoids the splat in dmesg. Cc: linux-stable # 5.19+ Fixes: 60dfaad65aa9 ("mm/hugetlb: allow uffd wr-protect none ptes") Reported-by: James Houghton Signed-off-by: Peter Xu Acked-by: David Hildenbrand Acked-by: James Houghton Reviewed-by: Mike Kravetz --- mm/hugetlb.c | 13 +++++++++++-- 1 file changed, 11 insertions(+), 2 deletions(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index bf7a1f628357..017d9159cddf 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -6649,8 +6649,17 @@ unsigned long hugetlb_change_protection(struct vm_ar= ea_struct *vma, spinlock_t *ptl; ptep =3D hugetlb_walk(vma, address, psize); if (!ptep) { - address |=3D last_addr_mask; - continue; + if (!uffd_wp) { + address |=3D last_addr_mask; + continue; + } + /* + * Userfaultfd wr-protect requires pgtable + * pre-allocations to install pte markers. + */ + ptep =3D huge_pte_alloc(mm, vma, address, psize); + if (!ptep) + break; } ptl =3D huge_pte_lock(h, mm, ptep); if (huge_pmd_unshare(mm, vma, address, ptep)) { --=20 2.37.3 From nobody Tue Sep 16 08:40:27 2025 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0CF65C54EBE for ; Wed, 4 Jan 2023 22:56:58 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234770AbjADW4e (ORCPT ); Wed, 4 Jan 2023 17:56:34 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55436 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234571AbjADW4L (ORCPT ); Wed, 4 Jan 2023 17:56:11 -0500 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7F7644730C for ; Wed, 4 Jan 2023 14:55:28 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1672872927; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=YDqbXjeAi3fC7Raz+Oh8InMZYaeFp0RXTarC/jVXe5Y=; b=E4808Jt6ZVAbUbRgITjCOarR9mwP7JuMY0dtjEyiRbWcBhIeJ1fSpKCFjfwBpOzWw2QKhe rlLfsehmBF6Zi4Bo0+PT6MD/Bd+hFjVD4S4qAMCw93B4DFZWihdkptDcIbMk26pC5F9bnf ODYC7K+QdpCrt/5IaPuv2O1R7qxfKr8= Received: from mail-qk1-f198.google.com (mail-qk1-f198.google.com [209.85.222.198]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_128_GCM_SHA256) id us-mta-582-27EBYC0hPCGPXt5ZQm7vjA-1; Wed, 04 Jan 2023 17:52:15 -0500 X-MC-Unique: 27EBYC0hPCGPXt5ZQm7vjA-1 Received: by mail-qk1-f198.google.com with SMTP id v7-20020a05620a0f0700b006faffce43b2so23816237qkl.9 for ; Wed, 04 Jan 2023 14:52:15 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=YDqbXjeAi3fC7Raz+Oh8InMZYaeFp0RXTarC/jVXe5Y=; b=vD3/78GlTir0y7kNibwUvoHI6pei0hR2kiEVQsnqrHqzksyx+gjJ9X3t6DywKd5RLk C+6//M5rPsnohrHNQjhz/0SLMUYN76jRJhZhCse131NjlDDYcIhTX1k9vKw94Y/PH8mR ZVUHEI/SNmKW4GwWtSMAEHqa1cWQHMHFVg80R0dqjs67znLhYV5ZyVEk83Ce8rQd6n1y OATKH+DPpjJMfKOTtcuJvYUREpJ/jjvmphIkAQ+FSmyDgm+vh9Sturushx017lkj7OMI 1nNc3ShNJrUJVkf8ygJoZJE/X/JHYrA9OBWxYELhrUzyxJl+mg0e5RyC4+6mWimPtjcb MSpA== X-Gm-Message-State: AFqh2kr4ul5JSispevfv/FoJVve2R9GU/GPN0/rpRqAVmXQp0jHp575N K3qefJBo9tYsRA1N/uUr75A8K3eh2lgnFZWwPK2l9/00la3wO+a+lsBlvWlgxROdrIPWZjpQIYx C1p1kPhdyDk2fLsmlKrcN9ZTs X-Received: by 2002:a05:622a:5a98:b0:3ab:8c3f:328b with SMTP id fz24-20020a05622a5a9800b003ab8c3f328bmr54270798qtb.4.1672872735075; Wed, 04 Jan 2023 14:52:15 -0800 (PST) X-Google-Smtp-Source: AMrXdXs1Sxv8n1AMZb/GH6LHO27LqRI5J2ATLlS4R+n+Mc6li+QeToMA1bMR//t6nBuvUswKomKeqQ== X-Received: by 2002:a05:622a:5a98:b0:3ab:8c3f:328b with SMTP id fz24-20020a05622a5a9800b003ab8c3f328bmr54270772qtb.4.1672872734836; Wed, 04 Jan 2023 14:52:14 -0800 (PST) Received: from x1n.redhat.com (bras-base-aurron9127w-grc-39-70-52-228-144.dsl.bell.ca. [70.52.228.144]) by smtp.gmail.com with ESMTPSA id r1-20020a05620a298100b006eeb3165565sm24654710qkp.80.2023.01.04.14.52.12 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 04 Jan 2023 14:52:14 -0800 (PST) From: Peter Xu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Mike Kravetz , Muchun Song , peterx@redhat.com, Nadav Amit , Andrea Arcangeli , David Hildenbrand , James Houghton , Axel Rasmussen , Andrew Morton Subject: [PATCH 2/3] mm/mprotect: Use long for page accountings and retval Date: Wed, 4 Jan 2023 17:52:06 -0500 Message-Id: <20230104225207.1066932-3-peterx@redhat.com> X-Mailer: git-send-email 2.37.3 In-Reply-To: <20230104225207.1066932-1-peterx@redhat.com> References: <20230104225207.1066932-1-peterx@redhat.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Switch to use type "long" for page accountings and retval across the whole procedure of change_protection(). The change should have shrinked the possible maximum page number to be half comparing to previous (ULONG_MAX / 2), but it shouldn't overflow on any system either because the maximum possible pages touched by change protection should be ULONG_MAX / PAGE_SIZE. Two reasons to switch from "unsigned long" to "long": 1. It suites better on count_vm_numa_events(), whose 2nd parameter takes a long type. 2. It paves way for returning negative (error) values in the future. Currently the only caller that consumes this retval is change_prot_numa(), where the unsigned long was converted to an int. Since at it, touching up the numa code to also take a long, so it'll avoid any possible overflow too during the int-size convertion. Signed-off-by: Peter Xu Acked-by: James Houghton Acked-by: Mike Kravetz --- include/linux/hugetlb.h | 4 ++-- include/linux/mm.h | 2 +- mm/hugetlb.c | 4 ++-- mm/mempolicy.c | 2 +- mm/mprotect.c | 26 +++++++++++++------------- 5 files changed, 19 insertions(+), 19 deletions(-) diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h index b6b10101bea7..e3aa336df900 100644 --- a/include/linux/hugetlb.h +++ b/include/linux/hugetlb.h @@ -248,7 +248,7 @@ void hugetlb_vma_lock_release(struct kref *kref); =20 int pmd_huge(pmd_t pmd); int pud_huge(pud_t pud); -unsigned long hugetlb_change_protection(struct vm_area_struct *vma, +long hugetlb_change_protection(struct vm_area_struct *vma, unsigned long address, unsigned long end, pgprot_t newprot, unsigned long cp_flags); =20 @@ -437,7 +437,7 @@ static inline void move_hugetlb_state(struct folio *old= _folio, { } =20 -static inline unsigned long hugetlb_change_protection( +static inline long hugetlb_change_protection( struct vm_area_struct *vma, unsigned long address, unsigned long end, pgprot_t newprot, unsigned long cp_flags) diff --git a/include/linux/mm.h b/include/linux/mm.h index c37f9330f14e..86fe17e6ded7 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -2132,7 +2132,7 @@ static inline bool vma_wants_manual_pte_write_upgrade= (struct vm_area_struct *vma } bool can_change_pte_writable(struct vm_area_struct *vma, unsigned long add= r, pte_t pte); -extern unsigned long change_protection(struct mmu_gather *tlb, +extern long change_protection(struct mmu_gather *tlb, struct vm_area_struct *vma, unsigned long start, unsigned long end, unsigned long cp_flags); extern int mprotect_fixup(struct mmu_gather *tlb, struct vm_area_struct *v= ma, diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 017d9159cddf..84bc665c7c86 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -6613,7 +6613,7 @@ long follow_hugetlb_page(struct mm_struct *mm, struct= vm_area_struct *vma, return i ? i : err; } =20 -unsigned long hugetlb_change_protection(struct vm_area_struct *vma, +long hugetlb_change_protection(struct vm_area_struct *vma, unsigned long address, unsigned long end, pgprot_t newprot, unsigned long cp_flags) { @@ -6622,7 +6622,7 @@ unsigned long hugetlb_change_protection(struct vm_are= a_struct *vma, pte_t *ptep; pte_t pte; struct hstate *h =3D hstate_vma(vma); - unsigned long pages =3D 0, psize =3D huge_page_size(h); + long pages =3D 0, psize =3D huge_page_size(h); bool shared_pmd =3D false; struct mmu_notifier_range range; unsigned long last_addr_mask; diff --git a/mm/mempolicy.c b/mm/mempolicy.c index d3558248a0f0..a86b8f15e2f0 100644 --- a/mm/mempolicy.c +++ b/mm/mempolicy.c @@ -631,7 +631,7 @@ unsigned long change_prot_numa(struct vm_area_struct *v= ma, unsigned long addr, unsigned long end) { struct mmu_gather tlb; - int nr_updated; + long nr_updated; =20 tlb_gather_mmu(&tlb, vma->vm_mm); =20 diff --git a/mm/mprotect.c b/mm/mprotect.c index 71358e45a742..0af22ab59ea8 100644 --- a/mm/mprotect.c +++ b/mm/mprotect.c @@ -80,13 +80,13 @@ bool can_change_pte_writable(struct vm_area_struct *vma= , unsigned long addr, return pte_dirty(pte); } =20 -static unsigned long change_pte_range(struct mmu_gather *tlb, +static long change_pte_range(struct mmu_gather *tlb, struct vm_area_struct *vma, pmd_t *pmd, unsigned long addr, unsigned long end, pgprot_t newprot, unsigned long cp_flags) { pte_t *pte, oldpte; spinlock_t *ptl; - unsigned long pages =3D 0; + long pages =3D 0; int target_node =3D NUMA_NO_NODE; bool prot_numa =3D cp_flags & MM_CP_PROT_NUMA; bool uffd_wp =3D cp_flags & MM_CP_UFFD_WP; @@ -353,13 +353,13 @@ uffd_wp_protect_file(struct vm_area_struct *vma, unsi= gned long cp_flags) } \ } while (0) =20 -static inline unsigned long change_pmd_range(struct mmu_gather *tlb, +static inline long change_pmd_range(struct mmu_gather *tlb, struct vm_area_struct *vma, pud_t *pud, unsigned long addr, unsigned long end, pgprot_t newprot, unsigned long cp_flags) { pmd_t *pmd; unsigned long next; - unsigned long pages =3D 0; + long pages =3D 0; unsigned long nr_huge_updates =3D 0; struct mmu_notifier_range range; =20 @@ -367,7 +367,7 @@ static inline unsigned long change_pmd_range(struct mmu= _gather *tlb, =20 pmd =3D pmd_offset(pud, addr); do { - unsigned long this_pages; + long this_pages; =20 next =3D pmd_addr_end(addr, end); =20 @@ -437,13 +437,13 @@ static inline unsigned long change_pmd_range(struct m= mu_gather *tlb, return pages; } =20 -static inline unsigned long change_pud_range(struct mmu_gather *tlb, +static inline long change_pud_range(struct mmu_gather *tlb, struct vm_area_struct *vma, p4d_t *p4d, unsigned long addr, unsigned long end, pgprot_t newprot, unsigned long cp_flags) { pud_t *pud; unsigned long next; - unsigned long pages =3D 0; + long pages =3D 0; =20 pud =3D pud_offset(p4d, addr); do { @@ -458,13 +458,13 @@ static inline unsigned long change_pud_range(struct m= mu_gather *tlb, return pages; } =20 -static inline unsigned long change_p4d_range(struct mmu_gather *tlb, +static inline long change_p4d_range(struct mmu_gather *tlb, struct vm_area_struct *vma, pgd_t *pgd, unsigned long addr, unsigned long end, pgprot_t newprot, unsigned long cp_flags) { p4d_t *p4d; unsigned long next; - unsigned long pages =3D 0; + long pages =3D 0; =20 p4d =3D p4d_offset(pgd, addr); do { @@ -479,14 +479,14 @@ static inline unsigned long change_p4d_range(struct m= mu_gather *tlb, return pages; } =20 -static unsigned long change_protection_range(struct mmu_gather *tlb, +static long change_protection_range(struct mmu_gather *tlb, struct vm_area_struct *vma, unsigned long addr, unsigned long end, pgprot_t newprot, unsigned long cp_flags) { struct mm_struct *mm =3D vma->vm_mm; pgd_t *pgd; unsigned long next; - unsigned long pages =3D 0; + long pages =3D 0; =20 BUG_ON(addr >=3D end); pgd =3D pgd_offset(mm, addr); @@ -505,12 +505,12 @@ static unsigned long change_protection_range(struct m= mu_gather *tlb, return pages; } =20 -unsigned long change_protection(struct mmu_gather *tlb, +long change_protection(struct mmu_gather *tlb, struct vm_area_struct *vma, unsigned long start, unsigned long end, unsigned long cp_flags) { pgprot_t newprot =3D vma->vm_page_prot; - unsigned long pages; + long pages; =20 BUG_ON((cp_flags & MM_CP_UFFD_WP_ALL) =3D=3D MM_CP_UFFD_WP_ALL); =20 --=20 2.37.3 From nobody Tue Sep 16 08:40:27 2025 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id E947FC54EBC for ; Wed, 4 Jan 2023 22:57:29 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234513AbjADW5F (ORCPT ); Wed, 4 Jan 2023 17:57:05 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55426 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229515AbjADW4N (ORCPT ); Wed, 4 Jan 2023 17:56:13 -0500 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3C8F347309 for ; Wed, 4 Jan 2023 14:55:28 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1672872927; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=w8KxT0fw2Kti4PKRWkIZPhnz1STfAelBM/L1tGxVI0U=; b=MOITBMECp9zVkGJmoUvi91eYFk5JfQkkr6MyFm/Ua8P3Fx4rJphpSD4zEb8L8z5J3d1pvR 54jc1q1TqchdaC50QbU1U4UOe3dPDTJUmTiBJzvqrJbsHrb++P0myb2ain4J3uw5URhjbG 6Z5S8uC9+yeGI2aPY6DcX5U2cYzrjz4= Received: from mail-qk1-f200.google.com (mail-qk1-f200.google.com [209.85.222.200]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_128_GCM_SHA256) id us-mta-472-pG8ygPPxPoi4tIKZskSTOg-1; Wed, 04 Jan 2023 17:52:18 -0500 X-MC-Unique: pG8ygPPxPoi4tIKZskSTOg-1 Received: by mail-qk1-f200.google.com with SMTP id az39-20020a05620a172700b006ff85c3b19eso24037206qkb.18 for ; Wed, 04 Jan 2023 14:52:18 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=w8KxT0fw2Kti4PKRWkIZPhnz1STfAelBM/L1tGxVI0U=; b=YopcAdPmjsLnv/qVC5cXcEUEgw6wQuMAYvvKVSzw+YFg09+kT6hbSL/LDrviWXG63v 6Vv0i6pALTjK53+bNKaz3nmJ/3TcC0IKcGj0OXyUVQ6qdwPWX+RCePwJnp95VBaMsloB GK3ScltY/H6URu4C3FekA92K2IrO2Syo+dGkBhwrXUCgUH0bJezenzIDoD2Dr3YvpdLa QzBx+YvwjerlArhb8BYG2lpQDVCStKMkrdMxhsTa5D1W2dXZ0DvrwdtmotV917l6MFGY NE27U38OFA29ChLuNje3EdJCgsDZ49I403NFJgTuNiD2XY37boQ/wM1L2xNhn4Neeb// fWsQ== X-Gm-Message-State: AFqh2kqy2BY5YKVF2kUaK5es5hnRCv8z0GG9EveMs9VQSlGtb6PiV5KZ Po+wry++uWX7ON5hof3NBpChp5fK4nTaBKU80zsyhp2Khx+a+fYndBI8R1fbIEx9Ml3CPT8M/jb 8Z0t5yLKUlVlB062+wNNNkxWG X-Received: by 2002:a05:6214:8e3:b0:521:ae4d:ea6a with SMTP id dr3-20020a05621408e300b00521ae4dea6amr65901329qvb.20.1672872737446; Wed, 04 Jan 2023 14:52:17 -0800 (PST) X-Google-Smtp-Source: AMrXdXu3dqYKFJ4Eo3TV1tw30PJqBr+Re6V+2aX8VJI591b6bZwEcKu5VZE1Ef0m6xJeQKmWoMv59Q== X-Received: by 2002:a05:6214:8e3:b0:521:ae4d:ea6a with SMTP id dr3-20020a05621408e300b00521ae4dea6amr65901311qvb.20.1672872737186; Wed, 04 Jan 2023 14:52:17 -0800 (PST) Received: from x1n.redhat.com (bras-base-aurron9127w-grc-39-70-52-228-144.dsl.bell.ca. [70.52.228.144]) by smtp.gmail.com with ESMTPSA id r1-20020a05620a298100b006eeb3165565sm24654710qkp.80.2023.01.04.14.52.14 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 04 Jan 2023 14:52:16 -0800 (PST) From: Peter Xu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Mike Kravetz , Muchun Song , peterx@redhat.com, Nadav Amit , Andrea Arcangeli , David Hildenbrand , James Houghton , Axel Rasmussen , Andrew Morton Subject: [PATCH 3/3] mm/uffd: Detect pgtable allocation failures Date: Wed, 4 Jan 2023 17:52:07 -0500 Message-Id: <20230104225207.1066932-4-peterx@redhat.com> X-Mailer: git-send-email 2.37.3 In-Reply-To: <20230104225207.1066932-1-peterx@redhat.com> References: <20230104225207.1066932-1-peterx@redhat.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Before this patch, when there's any pgtable allocation issues happened during change_protection(), the error will be ignored from the syscall. For shmem, there will be an error dumped into the host dmesg. Two issues with that: (1) Doing a trace dump when allocation fails is not anything close to grace.. (2) The user should be notified with any kind of such error, so the user can trap it and decide what to do next, either by retrying, or stop the process properly, or anything else. For userfault users, this will change the API of UFFDIO_WRITEPROTECT when pgtable allocation failure happened. It should not normally break anyone, though. If it breaks, then in good ways. One man-page update will be on the way to introduce the new -ENOMEM for UFFDIO_WRITEPROTECT. Not marking stable so we keep the old behavior on the 5.19-till-now kernels. Reported-by: James Houghton Signed-off-by: Peter Xu Acked-by: James Houghton --- include/linux/userfaultfd_k.h | 2 +- mm/hugetlb.c | 6 ++- mm/mempolicy.c | 2 +- mm/mprotect.c | 69 +++++++++++++++++++++++------------ mm/userfaultfd.c | 16 +++++--- 5 files changed, 62 insertions(+), 33 deletions(-) diff --git a/include/linux/userfaultfd_k.h b/include/linux/userfaultfd_k.h index 9df0b9a762cc..3767f18114ef 100644 --- a/include/linux/userfaultfd_k.h +++ b/include/linux/userfaultfd_k.h @@ -73,7 +73,7 @@ extern ssize_t mcopy_continue(struct mm_struct *dst_mm, u= nsigned long dst_start, extern int mwriteprotect_range(struct mm_struct *dst_mm, unsigned long start, unsigned long len, bool enable_wp, atomic_t *mmap_changing); -extern void uffd_wp_range(struct mm_struct *dst_mm, struct vm_area_struct = *vma, +extern long uffd_wp_range(struct mm_struct *dst_mm, struct vm_area_struct = *vma, unsigned long start, unsigned long len, bool enable_wp); =20 /* mm helpers */ diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 84bc665c7c86..d82d97e03eae 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -6658,8 +6658,10 @@ long hugetlb_change_protection(struct vm_area_struct= *vma, * pre-allocations to install pte markers. */ ptep =3D huge_pte_alloc(mm, vma, address, psize); - if (!ptep) + if (!ptep) { + pages =3D -ENOMEM; break; + } } ptl =3D huge_pte_lock(h, mm, ptep); if (huge_pmd_unshare(mm, vma, address, ptep)) { @@ -6749,7 +6751,7 @@ long hugetlb_change_protection(struct vm_area_struct = *vma, hugetlb_vma_unlock_write(vma); mmu_notifier_invalidate_range_end(&range); =20 - return pages << h->order; + return pages > 0 ? (pages << h->order) : pages; } =20 /* Return true if reservation was successful, false otherwise. */ diff --git a/mm/mempolicy.c b/mm/mempolicy.c index a86b8f15e2f0..85a34f1f3ab8 100644 --- a/mm/mempolicy.c +++ b/mm/mempolicy.c @@ -636,7 +636,7 @@ unsigned long change_prot_numa(struct vm_area_struct *v= ma, tlb_gather_mmu(&tlb, vma->vm_mm); =20 nr_updated =3D change_protection(&tlb, vma, addr, end, MM_CP_PROT_NUMA); - if (nr_updated) + if (nr_updated > 0) count_vm_numa_events(NUMA_PTE_UPDATES, nr_updated); =20 tlb_finish_mmu(&tlb); diff --git a/mm/mprotect.c b/mm/mprotect.c index 0af22ab59ea8..ade0d5f85a36 100644 --- a/mm/mprotect.c +++ b/mm/mprotect.c @@ -330,28 +330,34 @@ uffd_wp_protect_file(struct vm_area_struct *vma, unsi= gned long cp_flags) /* * If wr-protecting the range for file-backed, populate pgtable for the ca= se * when pgtable is empty but page cache exists. When {pte|pmd|...}_alloc() - * failed it means no memory, we don't have a better option but stop. + * failed we treat it the same way as pgtable allocation failures during + * page faults by kicking OOM and returning error. */ #define change_pmd_prepare(vma, pmd, cp_flags) \ - do { \ + ({ \ + long err =3D 0; \ if (unlikely(uffd_wp_protect_file(vma, cp_flags))) { \ - if (WARN_ON_ONCE(pte_alloc(vma->vm_mm, pmd))) \ - break; \ + if (pte_alloc(vma->vm_mm, pmd)) \ + err =3D -ENOMEM; \ } \ - } while (0) + err; \ + }) + /* * This is the general pud/p4d/pgd version of change_pmd_prepare(). We nee= d to * have separate change_pmd_prepare() because pte_alloc() returns 0 on suc= cess, * while {pmd|pud|p4d}_alloc() returns the valid pointer on success. */ #define change_prepare(vma, high, low, addr, cp_flags) \ - do { \ - if (unlikely(uffd_wp_protect_file(vma, cp_flags))) { \ - low##_t *p =3D low##_alloc(vma->vm_mm, high, addr); \ - if (WARN_ON_ONCE(p =3D=3D NULL)) \ - break; \ - } \ - } while (0) + ({ \ + long err =3D 0; \ + if (unlikely(uffd_wp_protect_file(vma, cp_flags))) { \ + low##_t *p =3D low##_alloc(vma->vm_mm, high, addr); \ + if (p =3D=3D NULL) \ + err =3D -ENOMEM; \ + } \ + err; \ + }) =20 static inline long change_pmd_range(struct mmu_gather *tlb, struct vm_area_struct *vma, pud_t *pud, unsigned long addr, @@ -367,11 +373,15 @@ static inline long change_pmd_range(struct mmu_gather= *tlb, =20 pmd =3D pmd_offset(pud, addr); do { - long this_pages; + long ret; =20 next =3D pmd_addr_end(addr, end); =20 - change_pmd_prepare(vma, pmd, cp_flags); + ret =3D change_pmd_prepare(vma, pmd, cp_flags); + if (ret) { + pages =3D ret; + break; + } /* * Automatic NUMA balancing walks the tables with mmap_lock * held for read. It's possible a parallel update to occur @@ -401,7 +411,11 @@ static inline long change_pmd_range(struct mmu_gather = *tlb, * cleared; make sure pmd populated if * necessary, then fall-through to pte level. */ - change_pmd_prepare(vma, pmd, cp_flags); + ret =3D change_pmd_prepare(vma, pmd, cp_flags); + if (ret) { + pages =3D ret; + break; + } } else { /* * change_huge_pmd() does not defer TLB flushes, @@ -422,9 +436,8 @@ static inline long change_pmd_range(struct mmu_gather *= tlb, } /* fall through, the trans huge pmd just split */ } - this_pages =3D change_pte_range(tlb, vma, pmd, addr, next, - newprot, cp_flags); - pages +=3D this_pages; + pages +=3D change_pte_range(tlb, vma, pmd, addr, next, + newprot, cp_flags); next: cond_resched(); } while (pmd++, addr =3D next, addr !=3D end); @@ -443,12 +456,14 @@ static inline long change_pud_range(struct mmu_gather= *tlb, { pud_t *pud; unsigned long next; - long pages =3D 0; + long pages =3D 0, ret; =20 pud =3D pud_offset(p4d, addr); do { next =3D pud_addr_end(addr, end); - change_prepare(vma, pud, pmd, addr, cp_flags); + ret =3D change_prepare(vma, pud, pmd, addr, cp_flags); + if (ret) + return ret; if (pud_none_or_clear_bad(pud)) continue; pages +=3D change_pmd_range(tlb, vma, pud, addr, next, newprot, @@ -464,12 +479,14 @@ static inline long change_p4d_range(struct mmu_gather= *tlb, { p4d_t *p4d; unsigned long next; - long pages =3D 0; + long pages =3D 0, ret; =20 p4d =3D p4d_offset(pgd, addr); do { next =3D p4d_addr_end(addr, end); - change_prepare(vma, p4d, pud, addr, cp_flags); + ret =3D change_prepare(vma, p4d, pud, addr, cp_flags); + if (ret) + return ret; if (p4d_none_or_clear_bad(p4d)) continue; pages +=3D change_pud_range(tlb, vma, p4d, addr, next, newprot, @@ -486,14 +503,18 @@ static long change_protection_range(struct mmu_gather= *tlb, struct mm_struct *mm =3D vma->vm_mm; pgd_t *pgd; unsigned long next; - long pages =3D 0; + long pages =3D 0, ret; =20 BUG_ON(addr >=3D end); pgd =3D pgd_offset(mm, addr); tlb_start_vma(tlb, vma); do { next =3D pgd_addr_end(addr, end); - change_prepare(vma, pgd, p4d, addr, cp_flags); + ret =3D change_prepare(vma, pgd, p4d, addr, cp_flags); + if (ret) { + pages =3D ret; + break; + } if (pgd_none_or_clear_bad(pgd)) continue; pages +=3D change_p4d_range(tlb, vma, pgd, addr, next, newprot, diff --git a/mm/userfaultfd.c b/mm/userfaultfd.c index 65ad172add27..53c3d916ff66 100644 --- a/mm/userfaultfd.c +++ b/mm/userfaultfd.c @@ -710,11 +710,12 @@ ssize_t mcopy_continue(struct mm_struct *dst_mm, unsi= gned long start, mmap_changing, 0); } =20 -void uffd_wp_range(struct mm_struct *dst_mm, struct vm_area_struct *dst_vm= a, +long uffd_wp_range(struct mm_struct *dst_mm, struct vm_area_struct *dst_vm= a, unsigned long start, unsigned long len, bool enable_wp) { unsigned int mm_cp_flags; struct mmu_gather tlb; + long ret; =20 if (enable_wp) mm_cp_flags =3D MM_CP_UFFD_WP; @@ -730,8 +731,10 @@ void uffd_wp_range(struct mm_struct *dst_mm, struct vm= _area_struct *dst_vma, if (!enable_wp && vma_wants_manual_pte_write_upgrade(dst_vma)) mm_cp_flags |=3D MM_CP_TRY_CHANGE_WRITABLE; tlb_gather_mmu(&tlb, dst_mm); - change_protection(&tlb, dst_vma, start, start + len, mm_cp_flags); + ret =3D change_protection(&tlb, dst_vma, start, start + len, mm_cp_flags); tlb_finish_mmu(&tlb); + + return ret; } =20 int mwriteprotect_range(struct mm_struct *dst_mm, unsigned long start, @@ -740,7 +743,7 @@ int mwriteprotect_range(struct mm_struct *dst_mm, unsig= ned long start, { struct vm_area_struct *dst_vma; unsigned long page_mask; - int err; + long err; =20 /* * Sanitize the command parameters: @@ -779,9 +782,12 @@ int mwriteprotect_range(struct mm_struct *dst_mm, unsi= gned long start, goto out_unlock; } =20 - uffd_wp_range(dst_mm, dst_vma, start, len, enable_wp); + err =3D uffd_wp_range(dst_mm, dst_vma, start, len, enable_wp); + + /* Return 0 on success, <0 on failures */ + if (err > 0) + err =3D 0; =20 - err =3D 0; out_unlock: mmap_read_unlock(dst_mm); return err; --=20 2.37.3