From nobody Sun Jun 28 00:50:38 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 24AB8C433FE for ; Thu, 17 Feb 2022 06:08:11 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234802AbiBQGIX (ORCPT ); Thu, 17 Feb 2022 01:08:23 -0500 Received: from mxb-00190b01.gslb.pphosted.com ([23.128.96.19]:45410 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232897AbiBQGIR (ORCPT ); Thu, 17 Feb 2022 01:08:17 -0500 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTP id 07CDAC7D4F for ; Wed, 16 Feb 2022 22:08:03 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1645078083; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=ZSU30vCJtk7GotZgltWgn1HG21KfRp7j8JvnndBOhWw=; b=CsHaBznw49UKzgWzxyqbZ/Tw2tgCHqUTiWMH0vNLr2TxyAW47dUwY6CyRu3XkQX8qQBSxM Tinz+XG2BOojfNFFMSnUDgUt8Y7cUUj8xAtKm6v57lkMqmfC9VPox1Tghyx5fOFTTaEhbM gBjBVk0arPnnxv3izmsXa3du+EJW/BU= Received: from mail-pl1-f199.google.com (mail-pl1-f199.google.com [209.85.214.199]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-629-AN7owxkMOPS3goyTwb2E8A-1; Thu, 17 Feb 2022 01:08:01 -0500 X-MC-Unique: AN7owxkMOPS3goyTwb2E8A-1 Received: by mail-pl1-f199.google.com with SMTP id w3-20020a170902a70300b0014cced44ad7so2106815plq.6 for ; Wed, 16 Feb 2022 22:08:01 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=ZSU30vCJtk7GotZgltWgn1HG21KfRp7j8JvnndBOhWw=; b=IlYqKPGPX4CytPIrWPAV8EzNqpHlCwv/y5dwY5UmCrpGbE1KgXTwlhhNF1T6NQs2CX VcVLdxERR8wn4rbIvmGZQBeuEMrlLX6ZrqPDoHBPp6dbaq/28Bc7IsH419G1rJCO82HR QuWB/kKTuyCqiPjj0suULq9vU1AipmQ8k0YIA7Brtxiksomn9jl7eGrhZ6MsC2ruNQDL BGqFQ85t4cBBFf52SgZISBdxNneyjpArADdtijqWd803BBwat6qSuHD6pZIPC57XW/iv o9OlXMsnv/+RmpVBJaoH3puH35t1iP5ZIN1lEMNefAwDTP6KfhtnickE8mWsLIOczHzx AxLw== X-Gm-Message-State: AOAM531EwGLUJFqMvPqwFBuLbrRtMhwLN+o1iEAzhZt3RyepJ3pyFHW/ MRr5ElYNiFnXRR3yFCJKaj8xcqvfXVydVXmu+9R9zv5QI5UAE03kFyN5mMuYndnBxuqLw1b5K50 lKn0SXhHx32YuBD7gYXAJnoZq X-Received: by 2002:a17:902:da91:b0:14d:58de:5667 with SMTP id j17-20020a170902da9100b0014d58de5667mr1392087plx.95.1645078080606; Wed, 16 Feb 2022 22:08:00 -0800 (PST) X-Google-Smtp-Source: ABdhPJyDUnDyEDIU63kvmjH9FX83zNUGtXLAelwg+LX3uRrYzwGaMsuEvfOVDWTrPD9itq0ndmmSlQ== X-Received: by 2002:a17:902:da91:b0:14d:58de:5667 with SMTP id j17-20020a170902da9100b0014d58de5667mr1392070plx.95.1645078080296; Wed, 16 Feb 2022 22:08:00 -0800 (PST) Received: from localhost.localdomain ([94.177.118.126]) by smtp.gmail.com with ESMTPSA id j8sm224230pjc.11.2022.02.16.22.07.55 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Wed, 16 Feb 2022 22:08:00 -0800 (PST) From: Peter Xu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Alistair Popple , Matthew Wilcox , peterx@redhat.com, David Hildenbrand , Andrea Arcangeli , Hugh Dickins , Yang Shi , Vlastimil Babka , John Hubbard , Andrew Morton , "Kirill A . Shutemov" Subject: [PATCH v5 1/4] mm: Don't skip swap entry even if zap_details specified Date: Thu, 17 Feb 2022 14:07:43 +0800 Message-Id: <20220217060746.71256-2-peterx@redhat.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20220217060746.71256-1-peterx@redhat.com> References: <20220217060746.71256-1-peterx@redhat.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" The "details" pointer shouldn't be the token to decide whether we should skip swap entries. For example, when the callers specified details->zap_mapping=3D=3DNULL, it means the user wants to zap all the pages (including COWed pages), then we need to look into swap entries because there can be private COWed pages that was swapped out. Skipping some swap entries when details is non-NULL may lead to wrongly leaving some of the swap entries while we should have zapped them. A reproducer of the problem: =3D=3D=3D8<=3D=3D=3D #define _GNU_SOURCE /* See feature_test_macros(7) */ #include #include #include #include #include int page_size; int shmem_fd; char *buffer; void main(void) { int ret; char val; page_size =3D getpagesize(); shmem_fd =3D memfd_create("test", 0); assert(shmem_fd >=3D 0); ret =3D ftruncate(shmem_fd, page_size * 2); assert(ret =3D=3D 0); buffer =3D mmap(NULL, page_size * 2, PROT_READ | PROT_WRITE, MAP_PRIVATE, shmem_fd, 0); assert(buffer !=3D MAP_FAILED); /* Write private page, swap it out */ buffer[page_size] =3D 1; madvise(buffer, page_size * 2, MADV_PAGEOUT); /* This should drop private buffer[page_size] already */ ret =3D ftruncate(shmem_fd, page_size); assert(ret =3D=3D 0); /* Recover the size */ ret =3D ftruncate(shmem_fd, page_size * 2); assert(ret =3D=3D 0); /* Re-read the data, it should be all zero */ val =3D buffer[page_size]; if (val =3D=3D 0) printf("Good\n"); else printf("BUG\n"); } =3D=3D=3D8<=3D=3D=3D We don't need to touch up the pmd path, because pmd never had a issue with swap entries. For example, shmem pmd migration will always be split into pte level, and same to swapping on anonymous. Add another helper should_zap_cows() so that we can also check whether we should zap private mappings when there's no page pointer specified. This patch drops that trick, so we handle swap ptes coherently. Meanwhile we should do the same check upon migration entry, hwpoison entry and genuine swap entries too. To be explicit, we should still remember to keep the private entries if even_cows=3D=3Dfalse, and always zap them when even_cows=3D=3Dtrue. The issue seems to exist starting from the initial commit of git. Fixes: 1da177e4c3f4 ("Linux-2.6.12-rc2") Reviewed-by: John Hubbard Signed-off-by: Peter Xu --- mm/memory.c | 40 +++++++++++++++++++++++++++++++--------- 1 file changed, 31 insertions(+), 9 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index c125c4969913..533da5d6c32c 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -1313,6 +1313,17 @@ struct zap_details { struct folio *single_folio; /* Locked folio to be unmapped */ }; =20 +/* Whether we should zap all COWed (private) pages too */ +static inline bool should_zap_cows(struct zap_details *details) +{ + /* By default, zap all pages */ + if (!details) + return true; + + /* Or, we zap COWed pages only if the caller wants to */ + return !details->zap_mapping; +} + /* * We set details->zap_mapping when we want to unmap shared but keep priva= te * pages. Return true if skip zapping this page, false otherwise. @@ -1320,11 +1331,15 @@ struct zap_details { static inline bool zap_skip_check_mapping(struct zap_details *details, struct page *page) { - if (!details || !page) + /* If we can make a decision without *page.. */ + if (should_zap_cows(details)) + return false; + + /* E.g. the caller passes NULL for the case of a zero page */ + if (!page) return false; =20 - return details->zap_mapping && - (details->zap_mapping !=3D page_rmapping(page)); + return details->zap_mapping !=3D page_rmapping(page); } =20 static unsigned long zap_pte_range(struct mmu_gather *tlb, @@ -1405,17 +1420,24 @@ static unsigned long zap_pte_range(struct mmu_gathe= r *tlb, continue; } =20 - /* If details->check_mapping, we leave swap entries. */ - if (unlikely(details)) - continue; - - if (!non_swap_entry(entry)) + if (!non_swap_entry(entry)) { + /* Genuine swap entry, hence a private anon page */ + if (!should_zap_cows(details)) + continue; rss[MM_SWAPENTS]--; - else if (is_migration_entry(entry)) { + } else if (is_migration_entry(entry)) { struct page *page; =20 page =3D pfn_swap_entry_to_page(entry); + if (zap_skip_check_mapping(details, page)) + continue; rss[mm_counter(page)]--; + } else if (is_hwpoison_entry(entry)) { + if (!should_zap_cows(details)) + continue; + } else { + /* We should have covered all the swap entry types */ + WARN_ON_ONCE(1); } if (unlikely(!free_swap_and_cache(entry))) print_bad_pte(vma, addr, ptent, NULL); --=20 2.32.0 From nobody Sun Jun 28 00:50:38 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 28B3EC433EF for ; Thu, 17 Feb 2022 06:08:15 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234838AbiBQGI1 (ORCPT ); Thu, 17 Feb 2022 01:08:27 -0500 Received: from mxb-00190b01.gslb.pphosted.com ([23.128.96.19]:45948 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234796AbiBQGIX (ORCPT ); Thu, 17 Feb 2022 01:08:23 -0500 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTP id 97D04C7D49 for ; Wed, 16 Feb 2022 22:08:09 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1645078088; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=GodqcRh1SHgqGa/hH7oS86cAqo3cWdJIC+9/AWlAQQI=; b=TaxuKaAqllgT1XfTfFg+JEr/+CA+4G1Lb5L3HdquzPNUoF9+yxaMIyKzeC/XsQLuWvlmc+ VQctkbr3jyYrZb0FvFRnIFJimWh0EOr+egDS8xeD9sLfizyTKMsCNTgdU91XXNo2XD1rE+ k83zseY/JWxdYIpuqcIwj6VQKGil5nw= Received: from mail-pl1-f197.google.com (mail-pl1-f197.google.com [209.85.214.197]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-192-1_SDuT1LOg6ggc15A_vpKA-1; Thu, 17 Feb 2022 01:08:07 -0500 X-MC-Unique: 1_SDuT1LOg6ggc15A_vpKA-1 Received: by mail-pl1-f197.google.com with SMTP id v20-20020a1709028d9400b0014ca63d0f10so2110776plo.5 for ; Wed, 16 Feb 2022 22:08:07 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=GodqcRh1SHgqGa/hH7oS86cAqo3cWdJIC+9/AWlAQQI=; b=ZnX+Odl3kJjDwdWV7BMgsoX2QIyIhnfZKh/CTOjEW27+Z08uC3yRUpfJ+JGW4TBBwW aATINwwUJATSZWxN9W02uZYcfhy+kL/rTlb6GgLmso/lCtTLQLK6kDxyIUpgMAc+4MPj Uw9L8BQo+22O3ryjnywldW6uZrOBI88ODlgDwUBtqIuZnEIwNp5SouRiiOOnRMIWAHwe OAazsPKxs2zEKLb7D8TysQLfzoZTJKlD0m0BZZ0Z607zqviIg78wudkOCkb9jBaazRut zbcHBHAz6X2qpcoonz89uHy/XxzwXkAMJSUBo1WVCQyMiu4idg4YCGLJcl62M79Hy3Bu LAkQ== X-Gm-Message-State: AOAM530361IKyx8rIWSauCdZ5H/TDQwRHCZht7CLR5CM58D9TRUrFLOv uvfx44D+YpXAi+VawM7QQd9V/zkutZGfbN4YE4gz7z7/YXFVkzjepxLP6DoCb/r/skZDTcoInv8 Uh5Ev6sYw8CFX8m63vdP1KiYg X-Received: by 2002:a17:902:be0a:b0:14d:5db0:7a14 with SMTP id r10-20020a170902be0a00b0014d5db07a14mr1368615pls.155.1645078085889; Wed, 16 Feb 2022 22:08:05 -0800 (PST) X-Google-Smtp-Source: ABdhPJwxmgTyql91syHq0iBmulNuYN38si4IUsKVfqHWUowF2A+HcKnj8DOPPRsO38J/B724ZyqK7Q== X-Received: by 2002:a17:902:be0a:b0:14d:5db0:7a14 with SMTP id r10-20020a170902be0a00b0014d5db07a14mr1368590pls.155.1645078085600; Wed, 16 Feb 2022 22:08:05 -0800 (PST) Received: from localhost.localdomain ([94.177.118.126]) by smtp.gmail.com with ESMTPSA id j8sm224230pjc.11.2022.02.16.22.08.00 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Wed, 16 Feb 2022 22:08:05 -0800 (PST) From: Peter Xu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Alistair Popple , Matthew Wilcox , peterx@redhat.com, David Hildenbrand , Andrea Arcangeli , Hugh Dickins , Yang Shi , Vlastimil Babka , John Hubbard , Andrew Morton , "Kirill A . Shutemov" Subject: [PATCH v5 2/4] mm: Rename zap_skip_check_mapping() to should_zap_page() Date: Thu, 17 Feb 2022 14:07:44 +0800 Message-Id: <20220217060746.71256-3-peterx@redhat.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20220217060746.71256-1-peterx@redhat.com> References: <20220217060746.71256-1-peterx@redhat.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" The previous name is against the natural way people think. Invert the meaning and also the return value. No functional change intended. Suggested-by: David Hildenbrand Suggested-by: Hugh Dickins Reviewed-by: David Hildenbrand Reviewed-by: John Hubbard Signed-off-by: Peter Xu --- mm/memory.c | 17 ++++++++--------- 1 file changed, 8 insertions(+), 9 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index 533da5d6c32c..3728632ea993 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -1326,20 +1326,19 @@ static inline bool should_zap_cows(struct zap_detai= ls *details) =20 /* * We set details->zap_mapping when we want to unmap shared but keep priva= te - * pages. Return true if skip zapping this page, false otherwise. + * pages. Return true if we should zap this page, false otherwise. */ -static inline bool -zap_skip_check_mapping(struct zap_details *details, struct page *page) +static inline bool should_zap_page(struct zap_details *details, struct pag= e *page) { /* If we can make a decision without *page.. */ if (should_zap_cows(details)) - return false; + return true; =20 /* E.g. the caller passes NULL for the case of a zero page */ if (!page) - return false; + return true; =20 - return details->zap_mapping !=3D page_rmapping(page); + return details->zap_mapping =3D=3D page_rmapping(page); } =20 static unsigned long zap_pte_range(struct mmu_gather *tlb, @@ -1374,7 +1373,7 @@ static unsigned long zap_pte_range(struct mmu_gather = *tlb, struct page *page; =20 page =3D vm_normal_page(vma, addr, ptent); - if (unlikely(zap_skip_check_mapping(details, page))) + if (unlikely(!should_zap_page(details, page))) continue; ptent =3D ptep_get_and_clear_full(mm, addr, pte, tlb->fullmm); @@ -1408,7 +1407,7 @@ static unsigned long zap_pte_range(struct mmu_gather = *tlb, is_device_exclusive_entry(entry)) { struct page *page =3D pfn_swap_entry_to_page(entry); =20 - if (unlikely(zap_skip_check_mapping(details, page))) + if (unlikely(!should_zap_page(details, page))) continue; pte_clear_not_present_full(mm, addr, pte, tlb->fullmm); rss[mm_counter(page)]--; @@ -1429,7 +1428,7 @@ static unsigned long zap_pte_range(struct mmu_gather = *tlb, struct page *page; =20 page =3D pfn_swap_entry_to_page(entry); - if (zap_skip_check_mapping(details, page)) + if (!should_zap_page(details, page)) continue; rss[mm_counter(page)]--; } else if (is_hwpoison_entry(entry)) { --=20 2.32.0 From nobody Sun Jun 28 00:50:38 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id DC42CC433FE for ; Thu, 17 Feb 2022 06:08:20 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234855AbiBQGId (ORCPT ); Thu, 17 Feb 2022 01:08:33 -0500 Received: from mxb-00190b01.gslb.pphosted.com ([23.128.96.19]:46366 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234842AbiBQGI2 (ORCPT ); Thu, 17 Feb 2022 01:08:28 -0500 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTP id 755E4C7D6A for ; Wed, 16 Feb 2022 22:08:14 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1645078093; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=A1vbZxsyzMifaaJqgkhHlizU7SHJnrILT5bLrmNnzMQ=; b=OZMeYcVqlq4pNdvhAGzBxH81Z0YZ8+d6IYZV24ocHZ7Da+Ei2ODTIySnJH9ejfvXEWeTOb erv/7vRGIBBaFZuPJ0ntq/Lu/NLHn9jLMmlGGnx1Zq1b6mFIVU50MHR8IwP5w85IaEFUtd v5yarUhFHMP68iqiJ82SF/HqprOoJ2s= Received: from mail-pj1-f69.google.com (mail-pj1-f69.google.com [209.85.216.69]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-410-6hXjHtCtP9m5i4wAPwLTrQ-1; Thu, 17 Feb 2022 01:08:12 -0500 X-MC-Unique: 6hXjHtCtP9m5i4wAPwLTrQ-1 Received: by mail-pj1-f69.google.com with SMTP id t16-20020a17090b019000b001b8af627800so5834347pjs.8 for ; Wed, 16 Feb 2022 22:08:12 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=A1vbZxsyzMifaaJqgkhHlizU7SHJnrILT5bLrmNnzMQ=; b=RDqX5W1HfxEsFd6wALKoXkkQnePEZfiwMeie367FvX7C3f90AsyT+Htrl/a6QxYUOd kH6GMdw2HoS0jujOw5HgEHT1p5gQUyLXMwJnPLqTDNfxNWJmqiUySjM1477szH/dr5IV Gsjza/BdHlWErIhZn3dgnvfcPI/DOD2fe4Ekv/irzevNm5uqk89ctpjwnyTT7Fk7ARZm fS6E4CuPhCI7yPaaV8nPi4MxtKmlSxYuk1TG25TpIj+tUc4ciNhJ6y/COKr1AaMzdG1I IHcUIz4Bpgy5OPchit+8A+P7DifJMctSlNlpW5gglwhuf9C+DZwFMnBq0z40zF3ED4qz WB6Q== X-Gm-Message-State: AOAM5322qqQCaYn0mHHO+znNhfqbI1dclt2bbJGoKGqkS+0jyh5unfN3 JkE7am+E6w4Vwffo9fKUxUnEdhFn300THW+7uxKVJK+QRKsT3wk3tcbyovdcrzfkTxG+elg7PN/ Xfxb7bAm18i0ZAu+A/CdmzjFa X-Received: by 2002:a17:90a:ec10:b0:1b8:9da7:3d13 with SMTP id l16-20020a17090aec1000b001b89da73d13mr1531870pjy.194.1645078091161; Wed, 16 Feb 2022 22:08:11 -0800 (PST) X-Google-Smtp-Source: ABdhPJxJ+MRoSejjpjSfq3CLN/Ow3BCwyndgoRTeQRVbvX65z98rR9y1roji3xMcenHn/prKAwWNxg== X-Received: by 2002:a17:90a:ec10:b0:1b8:9da7:3d13 with SMTP id l16-20020a17090aec1000b001b89da73d13mr1531852pjy.194.1645078090846; Wed, 16 Feb 2022 22:08:10 -0800 (PST) Received: from localhost.localdomain ([94.177.118.126]) by smtp.gmail.com with ESMTPSA id j8sm224230pjc.11.2022.02.16.22.08.06 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Wed, 16 Feb 2022 22:08:10 -0800 (PST) From: Peter Xu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Alistair Popple , Matthew Wilcox , peterx@redhat.com, David Hildenbrand , Andrea Arcangeli , Hugh Dickins , Yang Shi , Vlastimil Babka , John Hubbard , Andrew Morton , "Kirill A . Shutemov" Subject: [PATCH v5 3/4] mm: Change zap_details.zap_mapping into even_cows Date: Thu, 17 Feb 2022 14:07:45 +0800 Message-Id: <20220217060746.71256-4-peterx@redhat.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20220217060746.71256-1-peterx@redhat.com> References: <20220217060746.71256-1-peterx@redhat.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Currently we have a zap_mapping pointer maintained in zap_details, when it is specified we only want to zap the pages that has the same mapping with what the caller has specified. But what we want to do is actually simpler: we want to skip zapping private (COW-ed) pages in some cases. It comes from unmap_mapping_pages() where we could have passed in different even_cows values. The other user is unmap_mapping_folio() where we always want to skip private pages. According to Hugh, we used a mapping pointer for historical reason, as explained here: https://lore.kernel.org/lkml/391aa58d-ce84-9d4-d68d-d98a9c533255@google.c= om/ Quoting partly from Hugh: Which raises the question again of why I did not just use a boolean flag there originally: aah, I think I've found why. In those days there was a horrible "optimization", for better performance on some benchmark I guess, which when you read from /dev/zero into a private mapping, would map the zero page there (look up read_zero_pagealigned() and zeromap_page_range() if you dare). So there was another category of page to be skipped along with the anon COWs, and I didn't want multiple tests in the zap loop, so checking check_mapping against page->mapping did both. I think nowadays you could do it by checking for PageAnon page (or genuine swap entry) instead. This patch replaced the zap_details.zap_mapping pointer into the even_cows boolean, then we check it against PageAnon. Suggested-by: Hugh Dickins Reviewed-by: John Hubbard Signed-off-by: Peter Xu --- mm/memory.c | 16 +++++++--------- 1 file changed, 7 insertions(+), 9 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index 3728632ea993..c2defe8a1472 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -1309,8 +1309,8 @@ copy_page_range(struct vm_area_struct *dst_vma, struc= t vm_area_struct *src_vma) * Parameter block passed down to zap_pte_range in exceptional cases. */ struct zap_details { - struct address_space *zap_mapping; /* Check page->mapping if set */ struct folio *single_folio; /* Locked folio to be unmapped */ + bool even_cows; /* Zap COWed private pages too? */ }; =20 /* Whether we should zap all COWed (private) pages too */ @@ -1321,13 +1321,10 @@ static inline bool should_zap_cows(struct zap_detai= ls *details) return true; =20 /* Or, we zap COWed pages only if the caller wants to */ - return !details->zap_mapping; + return details->even_cows; } =20 -/* - * We set details->zap_mapping when we want to unmap shared but keep priva= te - * pages. Return true if we should zap this page, false otherwise. - */ +/* Decides whether we should zap this page with the page pointer specified= */ static inline bool should_zap_page(struct zap_details *details, struct pag= e *page) { /* If we can make a decision without *page.. */ @@ -1338,7 +1335,8 @@ static inline bool should_zap_page(struct zap_details= *details, struct page *pag if (!page) return true; =20 - return details->zap_mapping =3D=3D page_rmapping(page); + /* Otherwise we should only zap non-anon pages */ + return !PageAnon(page); } =20 static unsigned long zap_pte_range(struct mmu_gather *tlb, @@ -3398,7 +3396,7 @@ void unmap_mapping_folio(struct folio *folio) first_index =3D folio->index; last_index =3D folio->index + folio_nr_pages(folio) - 1; =20 - details.zap_mapping =3D mapping; + details.even_cows =3D false; details.single_folio =3D folio; =20 i_mmap_lock_write(mapping); @@ -3427,7 +3425,7 @@ void unmap_mapping_pages(struct address_space *mappin= g, pgoff_t start, pgoff_t first_index =3D start; pgoff_t last_index =3D start + nr - 1; =20 - details.zap_mapping =3D even_cows ? NULL : mapping; + details.even_cows =3D even_cows; if (last_index < first_index) last_index =3D ULONG_MAX; =20 --=20 2.32.0 From nobody Sun Jun 28 00:50:38 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3A06AC433EF for ; Thu, 17 Feb 2022 06:08:25 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234885AbiBQGIh (ORCPT ); Thu, 17 Feb 2022 01:08:37 -0500 Received: from mxb-00190b01.gslb.pphosted.com ([23.128.96.19]:46804 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234859AbiBQGId (ORCPT ); Thu, 17 Feb 2022 01:08:33 -0500 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTP id 5D457C7D49 for ; Wed, 16 Feb 2022 22:08:19 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1645078098; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=7Iw+/fHKv6eLf6uybneXrBCZa8a9WBXNMPZFyFhXGnU=; b=aj62FZAwnNoQM7MT1S59Cu1sOXe9PJSCKkSa8DToRtUAwFKxl9E/H08yd7qbCjmWtoQU3k dWGLcu5F6JIlIKWSmeE/rAiE0gtW1qFx/0v5MdMruyA4dEoCvPKuMrtf2kp51HRUZwTW9f EyFaZVjTxNqdB73mlRIS195yZXAvyKM= Received: from mail-pg1-f198.google.com (mail-pg1-f198.google.com [209.85.215.198]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-97-0i0WF9IRNh-o2wmz3Xw2FQ-1; Thu, 17 Feb 2022 01:08:17 -0500 X-MC-Unique: 0i0WF9IRNh-o2wmz3Xw2FQ-1 Received: by mail-pg1-f198.google.com with SMTP id 27-20020a63135b000000b0036285f54b6aso2454416pgt.19 for ; Wed, 16 Feb 2022 22:08:17 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=7Iw+/fHKv6eLf6uybneXrBCZa8a9WBXNMPZFyFhXGnU=; b=iuhVDMAzkJRt7FHL3cKFhwJU8kPejt7TT1wMiXuQ7tzien8Kxl4iYD1TwPAhWFzKLR vMsBgnDSlvevLxxkyGV9GlTzG6EE9BIohrnqJ+QO0TCmlflJeem/MOvJP86QaEe5Fm/h dFoHnzPNANUbIIqphT6Sn/1VtzqbSWCNAJljOD6mBDQ+AOwZLUdEJpoBUsf+ShOoUP9+ nBjfdcDdDSTrigz8zSdchm1nR8NdaDFvg6oXEQTdpHXdi+eMmSd28eIBrC3BwPzkanM7 LMAd439OeT6+aSu5vxrUJ2H/2ViPy7DYoU2MiWjtwWm5l6G5HdExzJ5wYvxaETbOzyyM u3nA== X-Gm-Message-State: AOAM532JFJPP3+FAQ2idN5EVc4YT8hJAnWnobDaDp3fO8BuBHCt4Nlhr rYWUJCwFdjkSL/+MrfIAhBkP/N+UfLMK9rCUV3sI0Jf1mdmUeW0ufWdRmHmSY48McaSFUXtC5oi 2cH4n47MgTYOARlJsFvfbg3wL X-Received: by 2002:a05:6a00:8c5:b0:4c7:f9a5:ebc6 with SMTP id s5-20020a056a0008c500b004c7f9a5ebc6mr1690177pfu.34.1645078096100; Wed, 16 Feb 2022 22:08:16 -0800 (PST) X-Google-Smtp-Source: ABdhPJzJq01ZMuxcXHBVvWHdhve6h2WPzO7RXn4GgLcncsq4yBWdeLBav4ZSIjD78cJP8QrYAxhIEQ== X-Received: by 2002:a05:6a00:8c5:b0:4c7:f9a5:ebc6 with SMTP id s5-20020a056a0008c500b004c7f9a5ebc6mr1690157pfu.34.1645078095837; Wed, 16 Feb 2022 22:08:15 -0800 (PST) Received: from localhost.localdomain ([94.177.118.126]) by smtp.gmail.com with ESMTPSA id j8sm224230pjc.11.2022.02.16.22.08.11 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Wed, 16 Feb 2022 22:08:15 -0800 (PST) From: Peter Xu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Alistair Popple , Matthew Wilcox , peterx@redhat.com, David Hildenbrand , Andrea Arcangeli , Hugh Dickins , Yang Shi , Vlastimil Babka , John Hubbard , Andrew Morton , "Kirill A . Shutemov" Subject: [PATCH v5 4/4] mm: Rework swap handling of zap_pte_range Date: Thu, 17 Feb 2022 14:07:46 +0800 Message-Id: <20220217060746.71256-5-peterx@redhat.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20220217060746.71256-1-peterx@redhat.com> References: <20220217060746.71256-1-peterx@redhat.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Clean the code up by merging the device private/exclusive swap entry handling with the rest, then we merge the pte clear operation too. struct* page is defined in multiple places in the function, move it upward. free_swap_and_cache() is only useful for !non_swap_entry() case, put it into the condition. No functional change intended. Reviewed-by: John Hubbard Signed-off-by: Peter Xu --- mm/memory.c | 21 ++++++--------------- 1 file changed, 6 insertions(+), 15 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index c2defe8a1472..6aa51ee2a307 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -1361,6 +1361,8 @@ static unsigned long zap_pte_range(struct mmu_gather = *tlb, arch_enter_lazy_mmu_mode(); do { pte_t ptent =3D *pte; + struct page *page; + if (pte_none(ptent)) continue; =20 @@ -1368,8 +1370,6 @@ static unsigned long zap_pte_range(struct mmu_gather = *tlb, break; =20 if (pte_present(ptent)) { - struct page *page; - page =3D vm_normal_page(vma, addr, ptent); if (unlikely(!should_zap_page(details, page))) continue; @@ -1403,28 +1403,21 @@ static unsigned long zap_pte_range(struct mmu_gathe= r *tlb, entry =3D pte_to_swp_entry(ptent); if (is_device_private_entry(entry) || is_device_exclusive_entry(entry)) { - struct page *page =3D pfn_swap_entry_to_page(entry); - + page =3D pfn_swap_entry_to_page(entry); if (unlikely(!should_zap_page(details, page))) continue; - pte_clear_not_present_full(mm, addr, pte, tlb->fullmm); rss[mm_counter(page)]--; - if (is_device_private_entry(entry)) page_remove_rmap(page, false); - put_page(page); - continue; - } - - if (!non_swap_entry(entry)) { + } else if (!non_swap_entry(entry)) { /* Genuine swap entry, hence a private anon page */ if (!should_zap_cows(details)) continue; rss[MM_SWAPENTS]--; + if (unlikely(!free_swap_and_cache(entry))) + print_bad_pte(vma, addr, ptent, NULL); } else if (is_migration_entry(entry)) { - struct page *page; - page =3D pfn_swap_entry_to_page(entry); if (!should_zap_page(details, page)) continue; @@ -1436,8 +1429,6 @@ static unsigned long zap_pte_range(struct mmu_gather = *tlb, /* We should have covered all the swap entry types */ WARN_ON_ONCE(1); } - if (unlikely(!free_swap_and_cache(entry))) - print_bad_pte(vma, addr, ptent, NULL); pte_clear_not_present_full(mm, addr, pte, tlb->fullmm); } while (pte++, addr +=3D PAGE_SIZE, addr !=3D end); =20 --=20 2.32.0