From nobody Sun Jun 28 01:49:55 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 14584C433EF for ; Wed, 16 Feb 2022 09:48:59 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232429AbiBPJtI (ORCPT ); Wed, 16 Feb 2022 04:49:08 -0500 Received: from gmail-smtp-in.l.google.com ([23.128.96.19]:35720 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232421AbiBPJtF (ORCPT ); Wed, 16 Feb 2022 04:49:05 -0500 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTP id 306A32B4059 for ; Wed, 16 Feb 2022 01:48:53 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1645004932; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=r0OLEdIrUqYcaY6DYkFVTZPLD4+1hLYjm8QOXt2Ctgs=; b=Wzd/ZrYycohIFIg9aOqh54zhVaoF2gzrP2SKn+5Mo7zRbcOW3PRKDJFFtigME93NvulK/c K8T7qE+PJACAa5YlqOW+VWe9M4x2/s0AFP5iHOUzTBtHjF7drFOE9jV6QdR5L2QbbeNsxi aehIz2C/ztFOMswiaObVEkLZtjc0Abk= Received: from mail-pj1-f72.google.com (mail-pj1-f72.google.com [209.85.216.72]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-428--igpvnesOy-yVz2uT6RGBg-1; Wed, 16 Feb 2022 04:48:51 -0500 X-MC-Unique: -igpvnesOy-yVz2uT6RGBg-1 Received: by mail-pj1-f72.google.com with SMTP id q40-20020a17090a17ab00b001bafa89b70aso1102946pja.2 for ; Wed, 16 Feb 2022 01:48:51 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=r0OLEdIrUqYcaY6DYkFVTZPLD4+1hLYjm8QOXt2Ctgs=; b=QLm2kvPmXw2M8K4TBN2B2eXpHEu2QN15x9EkIeJ191r85l7Dz/JvSZksIRvIL4YnSI RxNp7VZQXGGmjCnxgNFTs3BBhQ6M/F8wHcZ8gbHIgB5AC63vTTBAH0y+lRUiN6lPTHpi PYx9rIdqHOJPHGrfQPUvCQimCvip/2g/SdZJ/YkcE8B88JUbzTc1YSGy5eKa43tKt19A sw8/ZKkAXxm/dR1wDCdZ/VaQU9m+/vWVpkfiHNOYnEXb5+lwtDXtxyvDMbEtzFr1ztRp jw782qAV3HKx1khUPHId8ry0OWOjHsXMCfMiUmkTiuhsL1XYPhUMuhsI5d5HR9XfLaSf huxw== X-Gm-Message-State: AOAM532qfbWgVbwNLFEs46X1CUGEQzckWAUQ3Vy+Tj5p/SiV1tWi55QO 27jPcqjuYNbNRnvJusWQo+V0SCGq+kOzw7KwQLgEHeaua9uMSqRZYUA9tLS7kzUkzs/Zx4gPkuL O1coTr9P+kiYIAZxntVr3cQKED9HonoiG5o8qPr66ak+10FJbAXSlw7GpnLcwNPQZerRqwqdcjQ == X-Received: by 2002:a17:902:6acc:b0:149:8f60:a526 with SMTP id i12-20020a1709026acc00b001498f60a526mr2088583plt.25.1645004929947; Wed, 16 Feb 2022 01:48:49 -0800 (PST) X-Google-Smtp-Source: ABdhPJy8ro/PQOMeJ2c0+5tdk79xjZJksrx44v8baDKpAy/uaeFgFPs2vRvtMpMFMhDbUf7ALQrB3g== X-Received: by 2002:a17:902:6acc:b0:149:8f60:a526 with SMTP id i12-20020a1709026acc00b001498f60a526mr2088543plt.25.1645004929405; Wed, 16 Feb 2022 01:48:49 -0800 (PST) Received: from localhost.localdomain ([64.64.123.81]) by smtp.gmail.com with ESMTPSA id qe7sm11567835pjb.25.2022.02.16.01.48.36 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Wed, 16 Feb 2022 01:48:49 -0800 (PST) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Andrew Morton , "Kirill A . Shutemov" , Matthew Wilcox , Yang Shi , Andrea Arcangeli , peterx@redhat.com, John Hubbard , Alistair Popple , David Hildenbrand , Vlastimil Babka , Hugh Dickins Subject: [PATCH v4 1/4] mm: Don't skip swap entry even if zap_details specified Date: Wed, 16 Feb 2022 17:48:07 +0800 Message-Id: <20220216094810.60572-2-peterx@redhat.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20220216094810.60572-1-peterx@redhat.com> References: <20220216094810.60572-1-peterx@redhat.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" The "details" pointer shouldn't be the token to decide whether we should sk= ip swap entries. For example, when the user specified details->zap_mapping=3D= =3DNULL, it means the user wants to zap all the pages (including COWed pages), then = we need to look into swap entries because there can be private COWed pages that was swapped out. Skipping some swap entries when details is non-NULL may lead to wrongly lea= ving some of the swap entries while we should have zapped them. A reproducer of the problem: =3D=3D=3D8<=3D=3D=3D #define _GNU_SOURCE /* See feature_test_macros(7) */ #include #include #include #include #include int page_size; int shmem_fd; char *buffer; void main(void) { int ret; char val; page_size =3D getpagesize(); shmem_fd =3D memfd_create("test", 0); assert(shmem_fd >=3D 0); ret =3D ftruncate(shmem_fd, page_size * 2); assert(ret =3D=3D 0); buffer =3D mmap(NULL, page_size * 2, PROT_READ | PROT_WRITE, MAP_PRIVATE, shmem_fd, 0); assert(buffer !=3D MAP_FAILED); /* Write private page, swap it out */ buffer[page_size] =3D 1; madvise(buffer, page_size * 2, MADV_PAGEOUT); /* This should drop private buffer[page_size] already */ ret =3D ftruncate(shmem_fd, page_size); assert(ret =3D=3D 0); /* Recover the size */ ret =3D ftruncate(shmem_fd, page_size * 2); assert(ret =3D=3D 0); /* Re-read the data, it should be all zero */ val =3D buffer[page_size]; if (val =3D=3D 0) printf("Good\n"); else printf("BUG\n"); } =3D=3D=3D8<=3D=3D=3D We don't need to touch up the pmd path, because pmd never had a issue with = swap entries. For example, shmem pmd migration will always be split into pte le= vel, and same to swapping on anonymous. Add another helper should_zap_cows() so that we can also check whether we should zap private mappings when there's no page pointer specified. This patch drops that trick, so we handle swap ptes coherently. Meanwhile = we should do the same check upon migration entry, hwpoison entry and genuine s= wap entries too. To be explicit, we should still remember to keep the private entries if even_cows=3D=3Dfalse, and always zap them when even_cows=3D=3Dtr= ue. The issue seems to exist starting from the initial commit of git. Fixes: 1da177e4c3f4 ("Linux-2.6.12-rc2") Signed-off-by: Peter Xu Reviewed-by: John Hubbard --- mm/memory.c | 45 ++++++++++++++++++++++++++++++++++++--------- 1 file changed, 36 insertions(+), 9 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index c125c4969913..4bfeaca7cbc7 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -1313,6 +1313,17 @@ struct zap_details { struct folio *single_folio; /* Locked folio to be unmapped */ }; =20 +/* Whether we should zap all COWed (private) pages too */ +static inline bool should_zap_cows(struct zap_details *details) +{ + /* By default, zap all pages */ + if (!details) + return true; + + /* Or, we zap COWed pages only if the caller wants to */ + return !details->zap_mapping; +} + /* * We set details->zap_mapping when we want to unmap shared but keep priva= te * pages. Return true if skip zapping this page, false otherwise. @@ -1320,11 +1331,15 @@ struct zap_details { static inline bool zap_skip_check_mapping(struct zap_details *details, struct page *page) { - if (!details || !page) + /* If we can make a decision without *page.. */ + if (should_zap_cows(details)) return false; =20 - return details->zap_mapping && - (details->zap_mapping !=3D page_rmapping(page)); + /* E.g. zero page */ + if (!page) + return false; + + return details->zap_mapping !=3D page_rmapping(page); } =20 static unsigned long zap_pte_range(struct mmu_gather *tlb, @@ -1405,17 +1420,29 @@ static unsigned long zap_pte_range(struct mmu_gathe= r *tlb, continue; } =20 - /* If details->check_mapping, we leave swap entries. */ - if (unlikely(details)) - continue; - - if (!non_swap_entry(entry)) + if (!non_swap_entry(entry)) { + /* + * If this is a genuine swap entry, then it must be an + * private anon page. If the caller wants to skip + * COWed pages, ignore it. + */ + if (!should_zap_cows(details)) + continue; rss[MM_SWAPENTS]--; - else if (is_migration_entry(entry)) { + } else if (is_migration_entry(entry)) { struct page *page; =20 page =3D pfn_swap_entry_to_page(entry); + if (zap_skip_check_mapping(details, page)) + continue; rss[mm_counter(page)]--; + } else if (is_hwpoison_entry(entry)) { + /* If the caller wants to skip COWed pages, ignore it */ + if (!should_zap_cows(details)) + continue; + } else { + /* We should have covered all the swap entry types */ + WARN_ON_ONCE(1); } if (unlikely(!free_swap_and_cache(entry))) print_bad_pte(vma, addr, ptent, NULL); --=20 2.32.0 From nobody Sun Jun 28 01:49:55 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1DDFEC433EF for ; Wed, 16 Feb 2022 09:49:10 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232443AbiBPJtU (ORCPT ); Wed, 16 Feb 2022 04:49:20 -0500 Received: from gmail-smtp-in.l.google.com ([23.128.96.19]:37052 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232421AbiBPJtT (ORCPT ); Wed, 16 Feb 2022 04:49:19 -0500 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTP id DE4EF2B406E for ; Wed, 16 Feb 2022 01:49:06 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1645004945; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=trYMhQgsw7O8qP2aA5tHFBOoRzKbhngS39UuP85axlI=; b=AWOERnF+lk+S9qxCiNrcm0nwbaXY/NfT2G+8u+CNHqlq0G0SzvJHF9Sf+fFhyJkXO/ZHNT SVMKPZP4s59uNWOcTkOh9zyQlnLKBXwwyyP76G4ZS33ygMHo4FjxtZ6yNBl74YGBORGTYB t9/vSSl5kAX1dQH4ehrxt4EbKtbNlwk= Received: from mail-pf1-f200.google.com (mail-pf1-f200.google.com [209.85.210.200]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-418-ZLy2-bLmNyCcLR92Gyxfyg-1; Wed, 16 Feb 2022 04:49:04 -0500 X-MC-Unique: ZLy2-bLmNyCcLR92Gyxfyg-1 Received: by mail-pf1-f200.google.com with SMTP id i7-20020a626d07000000b004e07b61362bso1201948pfc.6 for ; Wed, 16 Feb 2022 01:49:04 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=trYMhQgsw7O8qP2aA5tHFBOoRzKbhngS39UuP85axlI=; b=B/ylR6eYYVuO85Da6W3EAVBHTjaW/bHFbs7MbysMGm3oKj0KPEmmV8YrHQqDv+URHc 5dVQvLC5oq12AfrNERo+CIvk88JO+wflQwk7tDleL7XOjjthELGPMNKmsN/y73x5HTVK GTXK0ZpOLSVULYYYE44ERCPU0FHAondoBJkhLZMLC5DU5xuBLde88kwD0mJhx1tQSvcH LCn7QutUvqWoj7xdRWav7pKriOje6J3tUVJNmKNPNCpts4JpNdn9Lh/cndW+Js4GrwlX fkNIql1ul+OzsTJhCHffADuObuEj9UY+HqvAOwE4diSuwZu3mIx3nEKmrjcR46a0fkCo T//Q== X-Gm-Message-State: AOAM531+p/JqO9TAIaM9inUYkLc45+7UqcCwn0ZrXv2o+Op9Ch1ugJbM 7UJSAvFkL5rLPzbefwj+1aY73AvUeG0s0ywN2x87FpBmqv0QhQ/yJo1L59ILigujDCMavRjXeNK wss1zKV2pZvBitTJcGSTaS+TvwYvhtlB3pubcV0b+myoeLxz/b8iVAma+ogm6EC5ppeSB3slGyw == X-Received: by 2002:a17:903:228a:b0:14d:aa04:2278 with SMTP id b10-20020a170903228a00b0014daa042278mr1850399plh.58.1645004943551; Wed, 16 Feb 2022 01:49:03 -0800 (PST) X-Google-Smtp-Source: ABdhPJwp9Rn/eyC2VMT306FaqLkb/8rUoQRaxAT2awvvDCzl/7i40J6BsqMQKml7qFHRkaY20DhzAA== X-Received: by 2002:a17:903:228a:b0:14d:aa04:2278 with SMTP id b10-20020a170903228a00b0014daa042278mr1850363plh.58.1645004943140; Wed, 16 Feb 2022 01:49:03 -0800 (PST) Received: from localhost.localdomain ([64.64.123.81]) by smtp.gmail.com with ESMTPSA id qe7sm11567835pjb.25.2022.02.16.01.48.50 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Wed, 16 Feb 2022 01:49:02 -0800 (PST) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Andrew Morton , "Kirill A . Shutemov" , Matthew Wilcox , Yang Shi , Andrea Arcangeli , peterx@redhat.com, John Hubbard , Alistair Popple , David Hildenbrand , Vlastimil Babka , Hugh Dickins Subject: [PATCH v4 2/4] mm: Rename zap_skip_check_mapping() to should_zap_page() Date: Wed, 16 Feb 2022 17:48:08 +0800 Message-Id: <20220216094810.60572-3-peterx@redhat.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20220216094810.60572-1-peterx@redhat.com> References: <20220216094810.60572-1-peterx@redhat.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" The previous name is against the natural way people think. Invert the mean= ing and also the return value. No functional change intended. Suggested-by: David Hildenbrand Suggested-by: Hugh Dickins Reviewed-by: David Hildenbrand Signed-off-by: Peter Xu Reviewed-by: John Hubbard --- mm/memory.c | 17 ++++++++--------- 1 file changed, 8 insertions(+), 9 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index 4bfeaca7cbc7..14d8428ff4db 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -1326,20 +1326,19 @@ static inline bool should_zap_cows(struct zap_detai= ls *details) =20 /* * We set details->zap_mapping when we want to unmap shared but keep priva= te - * pages. Return true if skip zapping this page, false otherwise. + * pages. Return true if we should zap this page, false otherwise. */ -static inline bool -zap_skip_check_mapping(struct zap_details *details, struct page *page) +static inline bool should_zap_page(struct zap_details *details, struct pag= e *page) { /* If we can make a decision without *page.. */ if (should_zap_cows(details)) - return false; + return true; =20 /* E.g. zero page */ if (!page) - return false; + return true; =20 - return details->zap_mapping !=3D page_rmapping(page); + return details->zap_mapping =3D=3D page_rmapping(page); } =20 static unsigned long zap_pte_range(struct mmu_gather *tlb, @@ -1374,7 +1373,7 @@ static unsigned long zap_pte_range(struct mmu_gather = *tlb, struct page *page; =20 page =3D vm_normal_page(vma, addr, ptent); - if (unlikely(zap_skip_check_mapping(details, page))) + if (unlikely(!should_zap_page(details, page))) continue; ptent =3D ptep_get_and_clear_full(mm, addr, pte, tlb->fullmm); @@ -1408,7 +1407,7 @@ static unsigned long zap_pte_range(struct mmu_gather = *tlb, is_device_exclusive_entry(entry)) { struct page *page =3D pfn_swap_entry_to_page(entry); =20 - if (unlikely(zap_skip_check_mapping(details, page))) + if (unlikely(!should_zap_page(details, page))) continue; pte_clear_not_present_full(mm, addr, pte, tlb->fullmm); rss[mm_counter(page)]--; @@ -1433,7 +1432,7 @@ static unsigned long zap_pte_range(struct mmu_gather = *tlb, struct page *page; =20 page =3D pfn_swap_entry_to_page(entry); - if (zap_skip_check_mapping(details, page)) + if (!should_zap_page(details, page)) continue; rss[mm_counter(page)]--; } else if (is_hwpoison_entry(entry)) { --=20 2.32.0 From nobody Sun Jun 28 01:49:55 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3CCAEC433EF for ; Wed, 16 Feb 2022 09:49:34 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232483AbiBPJtn (ORCPT ); Wed, 16 Feb 2022 04:49:43 -0500 Received: from gmail-smtp-in.l.google.com ([23.128.96.19]:38828 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232430AbiBPJtd (ORCPT ); Wed, 16 Feb 2022 04:49:33 -0500 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTP id 9A37E2B4072 for ; Wed, 16 Feb 2022 01:49:21 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1645004960; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=oZfOtFEoi6jPUWDy50tF9fEsGXQRQqyIANlBUu6Ts5c=; b=faA21TKRiY+fR9ZiteElUmfmuHrq+1s40BNwUyotMovb6BNvWP4IaGgmJ/9TBQEUi0qAiR c/kHst1yVKPBgFPD0KOROw0roAgXoHW4MDBL5yVrWwTOG+kvJaClXKqveA0LfRgEgTBMqR UnJtR9MsGXmg0ddrZz7JDGgS5OTpD4I= Received: from mail-pf1-f199.google.com (mail-pf1-f199.google.com [209.85.210.199]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-616-eLwqPexkMZy7DyX59K8Rqg-1; Wed, 16 Feb 2022 04:49:19 -0500 X-MC-Unique: eLwqPexkMZy7DyX59K8Rqg-1 Received: by mail-pf1-f199.google.com with SMTP id h3-20020a628303000000b004e12f44a262so1172826pfe.21 for ; Wed, 16 Feb 2022 01:49:19 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=oZfOtFEoi6jPUWDy50tF9fEsGXQRQqyIANlBUu6Ts5c=; b=KuMdRYelMC5t5yvwL54xWSskemp4qe0h1Sw85Cd9l0vq86PMk+SlBPAsRwu5I7Gg2j WWwEFylPrtl7B1LLfxlVM+Ahr2PiaSQnnxTGutfQuhTA14ZTjNzjx7nqSjZVHb596SHW m4oKTc6k1OZRLmXZHzd2qqSOAByszxk1WeKm1vxf2F4a5Vb8DUAi8NGMWJ+BeljTHWYw b/6klyVcIxObGjnC13a5s/X3+hxlhLo0IxbJTk6N07712VSCq5Sv8L+L5NR+JzW0YwkV Bo60IRXnQLycOTz81VShyWU3tQHN5lRsnVXc4A5uytH6uUWbjO9Srz3ZHkR3KCIm8R2q dk0A== X-Gm-Message-State: AOAM530igqjvrg/ASC9W4+99/PKT58gO9+CA0DCkHo3fWHHLijf/6f0G UdvvH3FPS+PF2aWWs7G0xnaJ8sqvJbc6GvqJ0qMjE/AevOawKzcoEjj7NRSOCH8uhPTudy951X1 43Gfbt/GJfuaZqTkDWnHz7alB4l306Q1GBez7ce/bC4k/tEsKhzQem0iwqTnG++IZuTtLHU4i6w == X-Received: by 2002:a17:90a:e7ca:b0:1b9:66c8:982c with SMTP id kb10-20020a17090ae7ca00b001b966c8982cmr784818pjb.30.1645004958385; Wed, 16 Feb 2022 01:49:18 -0800 (PST) X-Google-Smtp-Source: ABdhPJxr7nMoK1mV1Zu2nttOBw8/qblPs/Y5s3WL1TIEPfmM1m5wytDOSBV+uQUMnFliIrLh3OSfJg== X-Received: by 2002:a17:90a:e7ca:b0:1b9:66c8:982c with SMTP id kb10-20020a17090ae7ca00b001b966c8982cmr784784pjb.30.1645004958039; Wed, 16 Feb 2022 01:49:18 -0800 (PST) Received: from localhost.localdomain ([64.64.123.81]) by smtp.gmail.com with ESMTPSA id qe7sm11567835pjb.25.2022.02.16.01.49.04 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Wed, 16 Feb 2022 01:49:17 -0800 (PST) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Andrew Morton , "Kirill A . Shutemov" , Matthew Wilcox , Yang Shi , Andrea Arcangeli , peterx@redhat.com, John Hubbard , Alistair Popple , David Hildenbrand , Vlastimil Babka , Hugh Dickins Subject: [PATCH v4 3/4] mm: Change zap_details.zap_mapping into even_cows Date: Wed, 16 Feb 2022 17:48:09 +0800 Message-Id: <20220216094810.60572-4-peterx@redhat.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20220216094810.60572-1-peterx@redhat.com> References: <20220216094810.60572-1-peterx@redhat.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Currently we have a zap_mapping pointer maintained in zap_details, when it = is specified we only want to zap the pages that has the same mapping with what= the caller has specified. But what we want to do is actually simpler: we want to skip zapping private (COW-ed) pages in some cases. We can refer to unmap_mapping_pages() callers where we could have passed in different even_cows values. The other user is unmap_mapping_folio() where we always want to skip private pages. According to Hugh, we used a mapping pointer for historical reason, as explained here: https://lore.kernel.org/lkml/391aa58d-ce84-9d4-d68d-d98a9c533255@google.c= om/ Quoting partly from Hugh: Which raises the question again of why I did not just use a boolean flag there originally: aah, I think I've found why. In those days there was a horrible "optimization", for better performance on some benchmark I guess, which when you read from /dev/zero into a private mapping, would map the = zero page there (look up read_zero_pagealigned() and zeromap_page_range() if y= ou dare). So there was another category of page to be skipped along with the anon COWs, and I didn't want multiple tests in the zap loop, so checking check_mapping against page->mapping did both. I think nowadays you could= do it by checking for PageAnon page (or genuine swap entry) instead. This patch replaced the zap_details.zap_mapping pointer into the even_cows boolean, then we check it against PageAnon. Suggested-by: Hugh Dickins Signed-off-by: Peter Xu Reviewed-by: John Hubbard --- mm/memory.c | 16 +++++++--------- 1 file changed, 7 insertions(+), 9 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index 14d8428ff4db..ffa8c7dfe9ad 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -1309,8 +1309,8 @@ copy_page_range(struct vm_area_struct *dst_vma, struc= t vm_area_struct *src_vma) * Parameter block passed down to zap_pte_range in exceptional cases. */ struct zap_details { - struct address_space *zap_mapping; /* Check page->mapping if set */ struct folio *single_folio; /* Locked folio to be unmapped */ + bool even_cows; /* Zap COWed private pages too? */ }; =20 /* Whether we should zap all COWed (private) pages too */ @@ -1321,13 +1321,10 @@ static inline bool should_zap_cows(struct zap_detai= ls *details) return true; =20 /* Or, we zap COWed pages only if the caller wants to */ - return !details->zap_mapping; + return details->even_cows; } =20 -/* - * We set details->zap_mapping when we want to unmap shared but keep priva= te - * pages. Return true if we should zap this page, false otherwise. - */ +/* Decides whether we should zap this page with the page pointer specified= */ static inline bool should_zap_page(struct zap_details *details, struct pag= e *page) { /* If we can make a decision without *page.. */ @@ -1338,7 +1335,8 @@ static inline bool should_zap_page(struct zap_details= *details, struct page *pag if (!page) return true; =20 - return details->zap_mapping =3D=3D page_rmapping(page); + /* Otherwise we should only zap non-anon pages */ + return !PageAnon(page); } =20 static unsigned long zap_pte_range(struct mmu_gather *tlb, @@ -3403,7 +3401,7 @@ void unmap_mapping_folio(struct folio *folio) first_index =3D folio->index; last_index =3D folio->index + folio_nr_pages(folio) - 1; =20 - details.zap_mapping =3D mapping; + details.even_cows =3D false; details.single_folio =3D folio; =20 i_mmap_lock_write(mapping); @@ -3432,7 +3430,7 @@ void unmap_mapping_pages(struct address_space *mappin= g, pgoff_t start, pgoff_t first_index =3D start; pgoff_t last_index =3D start + nr - 1; =20 - details.zap_mapping =3D even_cows ? NULL : mapping; + details.even_cows =3D even_cows; if (last_index < first_index) last_index =3D ULONG_MAX; =20 --=20 2.32.0 From nobody Sun Jun 28 01:49:55 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7656CC433EF for ; Wed, 16 Feb 2022 09:49:48 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232462AbiBPJt6 (ORCPT ); Wed, 16 Feb 2022 04:49:58 -0500 Received: from gmail-smtp-in.l.google.com ([23.128.96.19]:40804 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232492AbiBPJtw (ORCPT ); Wed, 16 Feb 2022 04:49:52 -0500 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTP id 681582B4D86 for ; Wed, 16 Feb 2022 01:49:37 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1645004976; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=rxjE8nQUBN11kwzcRplXdSY7IFspDWL4Rd9u3ETZvZU=; b=Ir615H0RNSiExfYgbPzQlwbLcyQDK1T2Ef/yh/wVID4o1voW+93/gqpEJiDVtjHIzrYI98 a5tjEcD3Qszr5dTkFzoxRJgSmXfawvbWeYYADkXaAgV/jBL6BNErKjwUPo9kKqPxcOy67n Jrxm/C6UPdZfSZpnnZK6CWlIdUKNSvk= Received: from mail-pf1-f199.google.com (mail-pf1-f199.google.com [209.85.210.199]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-617-H7wICrN8P6SCSHgk5fLM7Q-1; Wed, 16 Feb 2022 04:49:35 -0500 X-MC-Unique: H7wICrN8P6SCSHgk5fLM7Q-1 Received: by mail-pf1-f199.google.com with SMTP id y28-20020aa793dc000000b004e160274e3eso1179448pff.18 for ; Wed, 16 Feb 2022 01:49:35 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=rxjE8nQUBN11kwzcRplXdSY7IFspDWL4Rd9u3ETZvZU=; b=MhlfvKAfylQPcEwTjxEGlfwNT2p5jTjFgkqhpy3BEqlTpKcAYPWvxdd54ysWKdT1Ne YyW4gutPXP/OZPb1gSF/33F4y2rm+RhAMsQ9nUu08w3ON2dJBKrHWx0fHwR2dd/O6UBs EiXHw7RypY4wCEBH/v960x4n1R0YIA7tYjjJch0fcvX0RCRonJxfHmW5v0ejKD+c/gnO KWYa7CSzequfs+yffTy9eyx/m+ouTHyZsbH5980uBPIa1dkMn8SWSFKwG6wNJHbiTTm/ hYvG8Ww0JJhyZpxXSdRF/Kankl1EsoBl48Z/0dJByQdy4N4cpUze4NLlGZjqfruE3nRU chTw== X-Gm-Message-State: AOAM531hPVx6Zz5R/QbNv5IIGD7THn4Ac+VS1F9FMBdEJX219QCJuYDd bxck2lsDDiwWTjR2Ef8s3mQ4USzaBlucmhUfW5q5LBXy3sAy0EoL0kPRPGNG+QY4ALbqVr/NF7l nDL840EhGWZeUgPkGyNbu80WWM5K0Fe4VtA5n0veDeI0JzMKvPT+o2WBGXHRQOSxrR9nmKSYDQA == X-Received: by 2002:a17:902:7892:b0:14e:c520:e47d with SMTP id q18-20020a170902789200b0014ec520e47dmr1620168pll.105.1645004973437; Wed, 16 Feb 2022 01:49:33 -0800 (PST) X-Google-Smtp-Source: ABdhPJyU/EfLN18Cu4rotd2bXfEqIGUN8J9OwxkyYql/PYAkRf8rWS5DdGmn2URaeB50VX2oPQZFOw== X-Received: by 2002:a17:902:7892:b0:14e:c520:e47d with SMTP id q18-20020a170902789200b0014ec520e47dmr1620133pll.105.1645004973050; Wed, 16 Feb 2022 01:49:33 -0800 (PST) Received: from localhost.localdomain ([64.64.123.81]) by smtp.gmail.com with ESMTPSA id qe7sm11567835pjb.25.2022.02.16.01.49.19 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Wed, 16 Feb 2022 01:49:32 -0800 (PST) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Andrew Morton , "Kirill A . Shutemov" , Matthew Wilcox , Yang Shi , Andrea Arcangeli , peterx@redhat.com, John Hubbard , Alistair Popple , David Hildenbrand , Vlastimil Babka , Hugh Dickins Subject: [PATCH v4 4/4] mm: Rework swap handling of zap_pte_range Date: Wed, 16 Feb 2022 17:48:10 +0800 Message-Id: <20220216094810.60572-5-peterx@redhat.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20220216094810.60572-1-peterx@redhat.com> References: <20220216094810.60572-1-peterx@redhat.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Clean the code up by merging the device private/exclusive swap entry handli= ng with the rest, then we merge the pte clear operation too. struct* page is defined in multiple places in the function, move it upward. free_swap_and_cache() is only useful for !non_swap_entry() case, put it into the condition. No functional change intended. Signed-off-by: Peter Xu Reviewed-by: John Hubbard --- mm/memory.c | 21 ++++++--------------- 1 file changed, 6 insertions(+), 15 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index ffa8c7dfe9ad..cade96024349 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -1361,6 +1361,8 @@ static unsigned long zap_pte_range(struct mmu_gather = *tlb, arch_enter_lazy_mmu_mode(); do { pte_t ptent =3D *pte; + struct page *page; + if (pte_none(ptent)) continue; =20 @@ -1368,8 +1370,6 @@ static unsigned long zap_pte_range(struct mmu_gather = *tlb, break; =20 if (pte_present(ptent)) { - struct page *page; - page =3D vm_normal_page(vma, addr, ptent); if (unlikely(!should_zap_page(details, page))) continue; @@ -1403,21 +1403,14 @@ static unsigned long zap_pte_range(struct mmu_gathe= r *tlb, entry =3D pte_to_swp_entry(ptent); if (is_device_private_entry(entry) || is_device_exclusive_entry(entry)) { - struct page *page =3D pfn_swap_entry_to_page(entry); - + page =3D pfn_swap_entry_to_page(entry); if (unlikely(!should_zap_page(details, page))) continue; - pte_clear_not_present_full(mm, addr, pte, tlb->fullmm); rss[mm_counter(page)]--; - if (is_device_private_entry(entry)) page_remove_rmap(page, false); - put_page(page); - continue; - } - - if (!non_swap_entry(entry)) { + } else if (!non_swap_entry(entry)) { /* * If this is a genuine swap entry, then it must be an * private anon page. If the caller wants to skip @@ -1426,9 +1419,9 @@ static unsigned long zap_pte_range(struct mmu_gather = *tlb, if (!should_zap_cows(details)) continue; rss[MM_SWAPENTS]--; + if (unlikely(!free_swap_and_cache(entry))) + print_bad_pte(vma, addr, ptent, NULL); } else if (is_migration_entry(entry)) { - struct page *page; - page =3D pfn_swap_entry_to_page(entry); if (!should_zap_page(details, page)) continue; @@ -1441,8 +1434,6 @@ static unsigned long zap_pte_range(struct mmu_gather = *tlb, /* We should have covered all the swap entry types */ WARN_ON_ONCE(1); } - if (unlikely(!free_swap_and_cache(entry))) - print_bad_pte(vma, addr, ptent, NULL); pte_clear_not_present_full(mm, addr, pte, tlb->fullmm); } while (pte++, addr +=3D PAGE_SIZE, addr !=3D end); =20 --=20 2.32.0