From nobody Tue Jun 30 01:42:28 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id BC562C433FE for ; Fri, 28 Jan 2022 04:54:34 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1346109AbiA1Eye (ORCPT ); Thu, 27 Jan 2022 23:54:34 -0500 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]:58865 "EHLO us-smtp-delivery-124.mimecast.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1346106AbiA1Eyc (ORCPT ); Thu, 27 Jan 2022 23:54:32 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1643345672; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=r0OLEdIrUqYcaY6DYkFVTZPLD4+1hLYjm8QOXt2Ctgs=; b=DqcQDMxcgN4MZAcskziob3S2ldDRW8aAWg5MhNyrtNGd6+LLSbSA11/3vLnayDB6cR7fj7 AhHOTz4tjhzWYCEMrHZaX6kYg7tumHUoXCGUEogW0qZAmvjiN2cOfprURzUYb1y3PafehF V+fgxVIze1zHMHzzxCWcelQWr14AvLQ= Received: from mail-wr1-f70.google.com (mail-wr1-f70.google.com [209.85.221.70]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-642-fi-u0vRNOVamLqRpHkrtrg-1; Thu, 27 Jan 2022 23:54:30 -0500 X-MC-Unique: fi-u0vRNOVamLqRpHkrtrg-1 Received: by mail-wr1-f70.google.com with SMTP id v28-20020adfa1dc000000b001dd1cb24081so1782514wrv.10 for ; Thu, 27 Jan 2022 20:54:30 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=r0OLEdIrUqYcaY6DYkFVTZPLD4+1hLYjm8QOXt2Ctgs=; b=CbaRXXiGBHAbgqcU7qKroTw+S86u5l8CrEB2wvsIjQTiFtEyWdqWMK4jayJ/dHUMW4 pxnc4Jznzy5z0tNHPFONBhEP2ST51Rkk+CKstWtlvPd1x2uw6yMFKVCeA7zw791Knc98 vGrkRboHy5ayIMX9AXi+uoxPNSE0o78CxU1oGfMooh3QBxhY9LH7SCvgbb7+cvJTm15k za5kHW7qTl2wRN6VywZi9HIK5VKc8ePly9Qj4qaTVl4cr5SJEPUkq3ID/3icIkHsnIIy olI2v3oE4TgIT2W11xEKuKUX4WvOP+fEgjvZKp3AamWv+TpK18KXafdfAAZjsjZ72qid 8/NQ== X-Gm-Message-State: AOAM530vH2JFvQNPNKItbhChU3s4tZDa8iubfKmWROLnIFyZ0PNZrrcB xw5krZNGmN4QXszVVxo2jjdxkSQ1VTCvWvNW2UCiCXqcxjLotPtOeqtsxMlwKlevY21sjr6ofOR 03VtuKvCiCzhxpBYOv5cTLZqP X-Received: by 2002:a5d:6da4:: with SMTP id u4mr5486419wrs.611.1643345669617; Thu, 27 Jan 2022 20:54:29 -0800 (PST) X-Google-Smtp-Source: ABdhPJzNjlbcr557L0JzcCx8cJn1ypoGGBxeXy3L3cPs1G3/t3aoCJ8N911EUy9hAc8Wvze/o8vnfQ== X-Received: by 2002:a5d:6da4:: with SMTP id u4mr5486406wrs.611.1643345669412; Thu, 27 Jan 2022 20:54:29 -0800 (PST) Received: from localhost.localdomain ([64.64.123.9]) by smtp.gmail.com with ESMTPSA id i13sm814014wrf.3.2022.01.27.20.54.24 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Thu, 27 Jan 2022 20:54:29 -0800 (PST) From: Peter Xu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: peterx@redhat.com, Alistair Popple , Andrew Morton , Andrea Arcangeli , David Hildenbrand , Matthew Wilcox , John Hubbard , Hugh Dickins , Vlastimil Babka , Yang Shi , "Kirill A . Shutemov" Subject: [PATCH v3 1/4] mm: Don't skip swap entry even if zap_details specified Date: Fri, 28 Jan 2022 12:54:09 +0800 Message-Id: <20220128045412.18695-2-peterx@redhat.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20220128045412.18695-1-peterx@redhat.com> References: <20220128045412.18695-1-peterx@redhat.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" The "details" pointer shouldn't be the token to decide whether we should sk= ip swap entries. For example, when the user specified details->zap_mapping=3D= =3DNULL, it means the user wants to zap all the pages (including COWed pages), then = we need to look into swap entries because there can be private COWed pages that was swapped out. Skipping some swap entries when details is non-NULL may lead to wrongly lea= ving some of the swap entries while we should have zapped them. A reproducer of the problem: =3D=3D=3D8<=3D=3D=3D #define _GNU_SOURCE /* See feature_test_macros(7) */ #include #include #include #include #include int page_size; int shmem_fd; char *buffer; void main(void) { int ret; char val; page_size =3D getpagesize(); shmem_fd =3D memfd_create("test", 0); assert(shmem_fd >=3D 0); ret =3D ftruncate(shmem_fd, page_size * 2); assert(ret =3D=3D 0); buffer =3D mmap(NULL, page_size * 2, PROT_READ | PROT_WRITE, MAP_PRIVATE, shmem_fd, 0); assert(buffer !=3D MAP_FAILED); /* Write private page, swap it out */ buffer[page_size] =3D 1; madvise(buffer, page_size * 2, MADV_PAGEOUT); /* This should drop private buffer[page_size] already */ ret =3D ftruncate(shmem_fd, page_size); assert(ret =3D=3D 0); /* Recover the size */ ret =3D ftruncate(shmem_fd, page_size * 2); assert(ret =3D=3D 0); /* Re-read the data, it should be all zero */ val =3D buffer[page_size]; if (val =3D=3D 0) printf("Good\n"); else printf("BUG\n"); } =3D=3D=3D8<=3D=3D=3D We don't need to touch up the pmd path, because pmd never had a issue with = swap entries. For example, shmem pmd migration will always be split into pte le= vel, and same to swapping on anonymous. Add another helper should_zap_cows() so that we can also check whether we should zap private mappings when there's no page pointer specified. This patch drops that trick, so we handle swap ptes coherently. Meanwhile = we should do the same check upon migration entry, hwpoison entry and genuine s= wap entries too. To be explicit, we should still remember to keep the private entries if even_cows=3D=3Dfalse, and always zap them when even_cows=3D=3Dtr= ue. The issue seems to exist starting from the initial commit of git. Fixes: 1da177e4c3f4 ("Linux-2.6.12-rc2") Signed-off-by: Peter Xu --- mm/memory.c | 45 ++++++++++++++++++++++++++++++++++++--------- 1 file changed, 36 insertions(+), 9 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index c125c4969913..4bfeaca7cbc7 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -1313,6 +1313,17 @@ struct zap_details { struct folio *single_folio; /* Locked folio to be unmapped */ }; =20 +/* Whether we should zap all COWed (private) pages too */ +static inline bool should_zap_cows(struct zap_details *details) +{ + /* By default, zap all pages */ + if (!details) + return true; + + /* Or, we zap COWed pages only if the caller wants to */ + return !details->zap_mapping; +} + /* * We set details->zap_mapping when we want to unmap shared but keep priva= te * pages. Return true if skip zapping this page, false otherwise. @@ -1320,11 +1331,15 @@ struct zap_details { static inline bool zap_skip_check_mapping(struct zap_details *details, struct page *page) { - if (!details || !page) + /* If we can make a decision without *page.. */ + if (should_zap_cows(details)) return false; =20 - return details->zap_mapping && - (details->zap_mapping !=3D page_rmapping(page)); + /* E.g. zero page */ + if (!page) + return false; + + return details->zap_mapping !=3D page_rmapping(page); } =20 static unsigned long zap_pte_range(struct mmu_gather *tlb, @@ -1405,17 +1420,29 @@ static unsigned long zap_pte_range(struct mmu_gathe= r *tlb, continue; } =20 - /* If details->check_mapping, we leave swap entries. */ - if (unlikely(details)) - continue; - - if (!non_swap_entry(entry)) + if (!non_swap_entry(entry)) { + /* + * If this is a genuine swap entry, then it must be an + * private anon page. If the caller wants to skip + * COWed pages, ignore it. + */ + if (!should_zap_cows(details)) + continue; rss[MM_SWAPENTS]--; - else if (is_migration_entry(entry)) { + } else if (is_migration_entry(entry)) { struct page *page; =20 page =3D pfn_swap_entry_to_page(entry); + if (zap_skip_check_mapping(details, page)) + continue; rss[mm_counter(page)]--; + } else if (is_hwpoison_entry(entry)) { + /* If the caller wants to skip COWed pages, ignore it */ + if (!should_zap_cows(details)) + continue; + } else { + /* We should have covered all the swap entry types */ + WARN_ON_ONCE(1); } if (unlikely(!free_swap_and_cache(entry))) print_bad_pte(vma, addr, ptent, NULL); --=20 2.32.0 From nobody Tue Jun 30 01:42:28 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 53374C433F5 for ; Fri, 28 Jan 2022 04:54:47 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1346112AbiA1Eyq (ORCPT ); Thu, 27 Jan 2022 23:54:46 -0500 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]:54532 "EHLO us-smtp-delivery-124.mimecast.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1346117AbiA1Eyi (ORCPT ); Thu, 27 Jan 2022 23:54:38 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1643345677; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=X4WXHh7pV56dMHSpF/O5K7LVTClsvKIQ79sBisfSRRA=; b=i77TYSfjBOLuBN4uUTRu2JUpJ8vizn3/zzMfbLJDrgDnS6p88pgRKKmQ5Odwxp6oqiOHd7 YxUeiDj6e4RRGsf9vVgI4Ty2/4OQEjDR7m/GeJUAoLfOO6y4xXKr/8JZETevPycFx5iiGR hOGzR6LU+3JvCLIRH/+DVw2sc6OzHTo= Received: from mail-wr1-f71.google.com (mail-wr1-f71.google.com [209.85.221.71]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-339-Qxdr3tMyNoqy4XvnRmZgNw-1; Thu, 27 Jan 2022 23:54:35 -0500 X-MC-Unique: Qxdr3tMyNoqy4XvnRmZgNw-1 Received: by mail-wr1-f71.google.com with SMTP id z1-20020adfbbc1000000b001df54394cebso950626wrg.20 for ; Thu, 27 Jan 2022 20:54:35 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=X4WXHh7pV56dMHSpF/O5K7LVTClsvKIQ79sBisfSRRA=; b=cCmoZktTlOzYSzkvrzCE1i6ZV431k3bSTmp9E2zZhKVs930VQW6w+asJA+Xd5N28F4 AIAdfRIqYiadIbsQhvwAX4wBezIdo/+yA/iX222NNs9ZZ+QhqeD0LLqyw8l30tXCkWdC EBrOk4vQqCddkN6rivBrI19fOmmSCvJ/CsR7r4u8pRb53eYQ+6DmYOAesEEUVNYzpqD8 F+1/27XVCA8ti9u2C+ycnUuJ7soXtnpiIusJfzPFmR7ZN4wH+bfMASDdOz3dAcDESY1G R158yiBP6a0sIW1fg9Hgb4tJjldqS7pXnuRpLigkzqPAx+qxMCurwL3vBLeXllncMx5P Lu6w== X-Gm-Message-State: AOAM530/Qf45/FprWtauxxDAhC88QR6hl5M4/F0XMdgFAJELCxuLVFjU OLoJTzu+eV7bb1FSTa3PCg47XFFSg7U2Dc74wiErjDtKcvGN8ZaiES3M4mPTDpzBG3+ZC46JHRU TNZhT9ob6kkPAzX6UJcF+ISWS X-Received: by 2002:a1c:a90d:: with SMTP id s13mr5787849wme.32.1643345674572; Thu, 27 Jan 2022 20:54:34 -0800 (PST) X-Google-Smtp-Source: ABdhPJzUZOPum7MMLETcw25d/iBx+n7SScJ596aZWjbvcjraz7IAo8lenVlYxR/jtIOkJT4M5mmwpg== X-Received: by 2002:a1c:a90d:: with SMTP id s13mr5787840wme.32.1643345674342; Thu, 27 Jan 2022 20:54:34 -0800 (PST) Received: from localhost.localdomain ([64.64.123.9]) by smtp.gmail.com with ESMTPSA id i13sm814014wrf.3.2022.01.27.20.54.29 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Thu, 27 Jan 2022 20:54:34 -0800 (PST) From: Peter Xu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: peterx@redhat.com, Alistair Popple , Andrew Morton , Andrea Arcangeli , David Hildenbrand , Matthew Wilcox , John Hubbard , Hugh Dickins , Vlastimil Babka , Yang Shi , "Kirill A . Shutemov" Subject: [PATCH v3 2/4] mm: Rename zap_skip_check_mapping() to should_zap_page() Date: Fri, 28 Jan 2022 12:54:10 +0800 Message-Id: <20220128045412.18695-3-peterx@redhat.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20220128045412.18695-1-peterx@redhat.com> References: <20220128045412.18695-1-peterx@redhat.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" The previous name is against the natural way people think. Invert the mean= ing and also the return value. No functional change intended. Suggested-by: Hugh Dickins Signed-off-by: Peter Xu Reviewed-by: David Hildenbrand --- mm/memory.c | 17 ++++++++--------- 1 file changed, 8 insertions(+), 9 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index 4bfeaca7cbc7..14d8428ff4db 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -1326,20 +1326,19 @@ static inline bool should_zap_cows(struct zap_detai= ls *details) =20 /* * We set details->zap_mapping when we want to unmap shared but keep priva= te - * pages. Return true if skip zapping this page, false otherwise. + * pages. Return true if we should zap this page, false otherwise. */ -static inline bool -zap_skip_check_mapping(struct zap_details *details, struct page *page) +static inline bool should_zap_page(struct zap_details *details, struct pag= e *page) { /* If we can make a decision without *page.. */ if (should_zap_cows(details)) - return false; + return true; =20 /* E.g. zero page */ if (!page) - return false; + return true; =20 - return details->zap_mapping !=3D page_rmapping(page); + return details->zap_mapping =3D=3D page_rmapping(page); } =20 static unsigned long zap_pte_range(struct mmu_gather *tlb, @@ -1374,7 +1373,7 @@ static unsigned long zap_pte_range(struct mmu_gather = *tlb, struct page *page; =20 page =3D vm_normal_page(vma, addr, ptent); - if (unlikely(zap_skip_check_mapping(details, page))) + if (unlikely(!should_zap_page(details, page))) continue; ptent =3D ptep_get_and_clear_full(mm, addr, pte, tlb->fullmm); @@ -1408,7 +1407,7 @@ static unsigned long zap_pte_range(struct mmu_gather = *tlb, is_device_exclusive_entry(entry)) { struct page *page =3D pfn_swap_entry_to_page(entry); =20 - if (unlikely(zap_skip_check_mapping(details, page))) + if (unlikely(!should_zap_page(details, page))) continue; pte_clear_not_present_full(mm, addr, pte, tlb->fullmm); rss[mm_counter(page)]--; @@ -1433,7 +1432,7 @@ static unsigned long zap_pte_range(struct mmu_gather = *tlb, struct page *page; =20 page =3D pfn_swap_entry_to_page(entry); - if (zap_skip_check_mapping(details, page)) + if (!should_zap_page(details, page)) continue; rss[mm_counter(page)]--; } else if (is_hwpoison_entry(entry)) { --=20 2.32.0 From nobody Tue Jun 30 01:42:28 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5874CC433F5 for ; Fri, 28 Jan 2022 04:54:49 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1346124AbiA1Eys (ORCPT ); Thu, 27 Jan 2022 23:54:48 -0500 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]:25297 "EHLO us-smtp-delivery-124.mimecast.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1346126AbiA1Eyo (ORCPT ); Thu, 27 Jan 2022 23:54:44 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1643345683; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=H1DVV/S8YSQjtZLKWoD0HN3kRcNDUylmDF/rAE9teCo=; b=MC1wQej2vHJ4/VAf8obkkL8dhWBqIY0Mwkb5qx+9JM0HQgE/IbtQpZoXwtiM3bN8YiIKgr 8uUKh5P1gKP5qi+Bjp736ERaWbMJZ2mlrMHxmJF+LJUN8KUWzXgMscDkKgg8MiAOhRp8wp axDPX3T4XjsRV/s0tpMKAguaI+WTg/g= Received: from mail-wm1-f72.google.com (mail-wm1-f72.google.com [209.85.128.72]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-22-hD0wDvE9PKW2tP0zzm0ayg-1; Thu, 27 Jan 2022 23:54:41 -0500 X-MC-Unique: hD0wDvE9PKW2tP0zzm0ayg-1 Received: by mail-wm1-f72.google.com with SMTP id w5-20020a1cf605000000b0034b8cb1f55eso5330921wmc.0 for ; Thu, 27 Jan 2022 20:54:41 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=H1DVV/S8YSQjtZLKWoD0HN3kRcNDUylmDF/rAE9teCo=; b=41OcEc1O86WdWwef3taKcLQEbI8gu0I12y1UekoKg33ej6Rxxd73K1y0jQbyEslEPo +uggv7AlUjM6MZKheO5HYmFEUCvXO753C2BjAFg9zxiee6p9WKEcS3o/90wEp29isA+n Brh/Y7+wtN5hCkOpDXQe7qbDSC/fmARXTIKM/7N0/MmnBtjnRPGjbc/by/ysZAqSPBJj 8VCShnHLZAYe9s0sYJ/Vq0FDlV+RlAbmgc1nQwCy+HF4cQRM5qzsqglgc6qn6koF8uAg erwdpa5Lk14nM6I7538fpMtBJ+sJ8H8AVpJjSg+rXL7E6FqSMprGZfH4bTJDWCOLIWAN TetA== X-Gm-Message-State: AOAM533G+ujdTjhfRvI+cyoaBruintpYJb8TINFG6eNwCDfXkPPJjhaV bp4rqiCLpm5hCcx8Xtlg39MGzJ/JGDRCyE7X/tlDuUy5vhvnmOQDpawonbrVi5sskfZQb4t9KCG caG2ZQ1YaHreHFS1tjyx2qyr6 X-Received: by 2002:a1c:2b06:: with SMTP id r6mr14345442wmr.4.1643345680337; Thu, 27 Jan 2022 20:54:40 -0800 (PST) X-Google-Smtp-Source: ABdhPJxSmguUCLAJc3uPMchWo2O4qWDyEBL0Tp/mOd4YhtXSNiPDsTt43pf61km0GV/X6qLuqocw1g== X-Received: by 2002:a1c:2b06:: with SMTP id r6mr14345423wmr.4.1643345680044; Thu, 27 Jan 2022 20:54:40 -0800 (PST) Received: from localhost.localdomain ([64.64.123.9]) by smtp.gmail.com with ESMTPSA id i13sm814014wrf.3.2022.01.27.20.54.34 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Thu, 27 Jan 2022 20:54:39 -0800 (PST) From: Peter Xu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: peterx@redhat.com, Alistair Popple , Andrew Morton , Andrea Arcangeli , David Hildenbrand , Matthew Wilcox , John Hubbard , Hugh Dickins , Vlastimil Babka , Yang Shi , "Kirill A . Shutemov" Subject: [PATCH v3 3/4] mm: Change zap_details.zap_mapping into even_cows Date: Fri, 28 Jan 2022 12:54:11 +0800 Message-Id: <20220128045412.18695-4-peterx@redhat.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20220128045412.18695-1-peterx@redhat.com> References: <20220128045412.18695-1-peterx@redhat.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Currently we have a zap_mapping pointer maintained in zap_details, when it = is specified we only want to zap the pages that has the same mapping with what= the caller has specified. But what we want to do is actually simpler: we want to skip zapping private (COW-ed) pages in some cases. We can refer to unmap_mapping_pages() callers where we could have passed in different even_cows values. The other user is unmap_mapping_folio() where we always want to skip private pages. According to Hugh, we used a mapping pointer for historical reason, as explained here: https://lore.kernel.org/lkml/391aa58d-ce84-9d4-d68d-d98a9c533255@google.c= om/ Quotting partly from Hugh: Which raises the question again of why I did not just use a boolean flag there originally: aah, I think I've found why. In those days there was a horrible "optimization", for better performance on some benchmark I guess, which when you read from /dev/zero into a private mapping, would map the = zero page there (look up read_zero_pagealigned() and zeromap_page_range() if y= ou dare). So there was another category of page to be skipped along with the anon COWs, and I didn't want multiple tests in the zap loop, so checking check_mapping against page->mapping did both. I think nowadays you could= do it by checking for PageAnon page (or genuine swap entry) instead. This patch replaced the zap_details.zap_mapping pointer into the even_cows boolean, then we check it against PageAnon. Suggested-by: Hugh Dickins Signed-off-by: Peter Xu --- mm/memory.c | 16 +++++++--------- 1 file changed, 7 insertions(+), 9 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index 14d8428ff4db..ffa8c7dfe9ad 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -1309,8 +1309,8 @@ copy_page_range(struct vm_area_struct *dst_vma, struc= t vm_area_struct *src_vma) * Parameter block passed down to zap_pte_range in exceptional cases. */ struct zap_details { - struct address_space *zap_mapping; /* Check page->mapping if set */ struct folio *single_folio; /* Locked folio to be unmapped */ + bool even_cows; /* Zap COWed private pages too? */ }; =20 /* Whether we should zap all COWed (private) pages too */ @@ -1321,13 +1321,10 @@ static inline bool should_zap_cows(struct zap_detai= ls *details) return true; =20 /* Or, we zap COWed pages only if the caller wants to */ - return !details->zap_mapping; + return details->even_cows; } =20 -/* - * We set details->zap_mapping when we want to unmap shared but keep priva= te - * pages. Return true if we should zap this page, false otherwise. - */ +/* Decides whether we should zap this page with the page pointer specified= */ static inline bool should_zap_page(struct zap_details *details, struct pag= e *page) { /* If we can make a decision without *page.. */ @@ -1338,7 +1335,8 @@ static inline bool should_zap_page(struct zap_details= *details, struct page *pag if (!page) return true; =20 - return details->zap_mapping =3D=3D page_rmapping(page); + /* Otherwise we should only zap non-anon pages */ + return !PageAnon(page); } =20 static unsigned long zap_pte_range(struct mmu_gather *tlb, @@ -3403,7 +3401,7 @@ void unmap_mapping_folio(struct folio *folio) first_index =3D folio->index; last_index =3D folio->index + folio_nr_pages(folio) - 1; =20 - details.zap_mapping =3D mapping; + details.even_cows =3D false; details.single_folio =3D folio; =20 i_mmap_lock_write(mapping); @@ -3432,7 +3430,7 @@ void unmap_mapping_pages(struct address_space *mappin= g, pgoff_t start, pgoff_t first_index =3D start; pgoff_t last_index =3D start + nr - 1; =20 - details.zap_mapping =3D even_cows ? NULL : mapping; + details.even_cows =3D even_cows; if (last_index < first_index) last_index =3D ULONG_MAX; =20 --=20 2.32.0 From nobody Tue Jun 30 01:42:28 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 57A11C433F5 for ; Fri, 28 Jan 2022 04:54:58 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1346132AbiA1Ey4 (ORCPT ); Thu, 27 Jan 2022 23:54:56 -0500 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]:50776 "EHLO us-smtp-delivery-124.mimecast.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1346130AbiA1Eyu (ORCPT ); Thu, 27 Jan 2022 23:54:50 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1643345690; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=rxjE8nQUBN11kwzcRplXdSY7IFspDWL4Rd9u3ETZvZU=; b=YO7nBw5xLmY93LEDYURC5Tu8S+JCtCoLoyLkdhOjyx3ri0BbqMy9MYKTgvU/113ooA3a7j dzqO9UuWlcXYe1s9eUo/iIpC3FN+acJ5AiwMFtgqhvh7oS2EZmW6760w8fIplT8qjtYpRg WKGE+DytE40X+g68pPxXvIZF8CFn51U= Received: from mail-wm1-f69.google.com (mail-wm1-f69.google.com [209.85.128.69]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-194-ThQMNl68Mi-vjCdiYHy5FQ-1; Thu, 27 Jan 2022 23:54:46 -0500 X-MC-Unique: ThQMNl68Mi-vjCdiYHy5FQ-1 Received: by mail-wm1-f69.google.com with SMTP id l20-20020a05600c1d1400b0035153bf34c3so2562864wms.2 for ; Thu, 27 Jan 2022 20:54:46 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=rxjE8nQUBN11kwzcRplXdSY7IFspDWL4Rd9u3ETZvZU=; b=ZawArv9dZ0WttI80RFD3g5mCcfWChcSR81uz4sfoNaj32xeFrlds3VSpUhV9Lv8U+X ZSCl1LJEH6Jg6/EJRZvlisUI8eHXPElb5nGSME+H/hI9510yqDS4ZDCup9WcSEDASxGv QZGax17YAnjbVeyTcadx4UYbrUnzFCd3owtNjYlhk4wd6OX0gnwE8j9Fr/U4OFEtuCmH dN4ZB48yH7KBvZbJzmXAymcTY8GM51uJFwMSVy/lIP/L/HauRLTEaPaCz++KID8Nm5Xv hDeAQ7PLVeDvCrfpcEltHRjIwaV9RAlrICBfx1Q2TSVg2hYYJxys5cPvGNM3KyJxGfWU vBmw== X-Gm-Message-State: AOAM533jKew3ubNvafvmRWu19mL3oFFfJtkxiqZ2k5KwLO5CFghV0nIz Sm+FBvdnOteOMaaAQxAUv5dfwvks2pG6oeL8S6ii4vfCJ9H2NGPRIr/DjZ9BHv7i05vpDqlaCrE dF+jU6tohKWjo4HcRpGRH2+Td X-Received: by 2002:a05:6000:1a89:: with SMTP id f9mr5884820wry.251.1643345685470; Thu, 27 Jan 2022 20:54:45 -0800 (PST) X-Google-Smtp-Source: ABdhPJzvpMqKTZBmuLjr5YO2EFIJM9G9DWCeuPiKqQMCAYXt0G0OqDw+Xws+L0tBDAR/T/8LmGetKw== X-Received: by 2002:a05:6000:1a89:: with SMTP id f9mr5884805wry.251.1643345685283; Thu, 27 Jan 2022 20:54:45 -0800 (PST) Received: from localhost.localdomain ([64.64.123.9]) by smtp.gmail.com with ESMTPSA id i13sm814014wrf.3.2022.01.27.20.54.40 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Thu, 27 Jan 2022 20:54:45 -0800 (PST) From: Peter Xu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: peterx@redhat.com, Alistair Popple , Andrew Morton , Andrea Arcangeli , David Hildenbrand , Matthew Wilcox , John Hubbard , Hugh Dickins , Vlastimil Babka , Yang Shi , "Kirill A . Shutemov" Subject: [PATCH v3 4/4] mm: Rework swap handling of zap_pte_range Date: Fri, 28 Jan 2022 12:54:12 +0800 Message-Id: <20220128045412.18695-5-peterx@redhat.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20220128045412.18695-1-peterx@redhat.com> References: <20220128045412.18695-1-peterx@redhat.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Clean the code up by merging the device private/exclusive swap entry handli= ng with the rest, then we merge the pte clear operation too. struct* page is defined in multiple places in the function, move it upward. free_swap_and_cache() is only useful for !non_swap_entry() case, put it into the condition. No functional change intended. Signed-off-by: Peter Xu --- mm/memory.c | 21 ++++++--------------- 1 file changed, 6 insertions(+), 15 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index ffa8c7dfe9ad..cade96024349 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -1361,6 +1361,8 @@ static unsigned long zap_pte_range(struct mmu_gather = *tlb, arch_enter_lazy_mmu_mode(); do { pte_t ptent =3D *pte; + struct page *page; + if (pte_none(ptent)) continue; =20 @@ -1368,8 +1370,6 @@ static unsigned long zap_pte_range(struct mmu_gather = *tlb, break; =20 if (pte_present(ptent)) { - struct page *page; - page =3D vm_normal_page(vma, addr, ptent); if (unlikely(!should_zap_page(details, page))) continue; @@ -1403,21 +1403,14 @@ static unsigned long zap_pte_range(struct mmu_gathe= r *tlb, entry =3D pte_to_swp_entry(ptent); if (is_device_private_entry(entry) || is_device_exclusive_entry(entry)) { - struct page *page =3D pfn_swap_entry_to_page(entry); - + page =3D pfn_swap_entry_to_page(entry); if (unlikely(!should_zap_page(details, page))) continue; - pte_clear_not_present_full(mm, addr, pte, tlb->fullmm); rss[mm_counter(page)]--; - if (is_device_private_entry(entry)) page_remove_rmap(page, false); - put_page(page); - continue; - } - - if (!non_swap_entry(entry)) { + } else if (!non_swap_entry(entry)) { /* * If this is a genuine swap entry, then it must be an * private anon page. If the caller wants to skip @@ -1426,9 +1419,9 @@ static unsigned long zap_pte_range(struct mmu_gather = *tlb, if (!should_zap_cows(details)) continue; rss[MM_SWAPENTS]--; + if (unlikely(!free_swap_and_cache(entry))) + print_bad_pte(vma, addr, ptent, NULL); } else if (is_migration_entry(entry)) { - struct page *page; - page =3D pfn_swap_entry_to_page(entry); if (!should_zap_page(details, page)) continue; @@ -1441,8 +1434,6 @@ static unsigned long zap_pte_range(struct mmu_gather = *tlb, /* We should have covered all the swap entry types */ WARN_ON_ONCE(1); } - if (unlikely(!free_swap_and_cache(entry))) - print_bad_pte(vma, addr, ptent, NULL); pte_clear_not_present_full(mm, addr, pte, tlb->fullmm); } while (pte++, addr +=3D PAGE_SIZE, addr !=3D end); =20 --=20 2.32.0