From nobody Sat Apr 11 10:47:11 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id E8C87C19F2A for ; Thu, 11 Aug 2022 16:42:27 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238984AbiHKQmZ (ORCPT ); Thu, 11 Aug 2022 12:42:25 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:52672 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S239162AbiHKQkz (ORCPT ); Thu, 11 Aug 2022 12:40:55 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTP id 7D55BA5C4E for ; Thu, 11 Aug 2022 09:13:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1660234417; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=nfi/Ecb+oYo1RXzYzaJhIF6aEHnKWbHUlt161URhhFA=; b=GhsOGOx+UWqgdk2hLvFdA/eje74sCFQsPeRXdDNjHvzK3Otd+Mrpav1B0lpKp30PgCg2wa btm7fZwBHcS040weXYswe/FFkU71n+WM5P4a7vtmKfZUZm+OaoHQgl3n8ew5BA36ineajT YGPkKdNXFqkbrkRMUQfoWriJsUbg+Qo= Received: from mail-io1-f72.google.com (mail-io1-f72.google.com [209.85.166.72]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_128_GCM_SHA256) id us-mta-586-0DYtKXsdMQ2QSdY2jwh4QA-1; Thu, 11 Aug 2022 12:13:36 -0400 X-MC-Unique: 0DYtKXsdMQ2QSdY2jwh4QA-1 Received: by mail-io1-f72.google.com with SMTP id f4-20020a5d8784000000b00684f917b026so5683668ion.12 for ; Thu, 11 Aug 2022 09:13:36 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc; bh=nfi/Ecb+oYo1RXzYzaJhIF6aEHnKWbHUlt161URhhFA=; b=0MsanOpOrzrOYhyUyYK0EWnUi65FCAhSN4IHpM0W+Ag+kg5gF6KKNlU0Fw9FAysWeG FzrYy3UpaUNRCSZ9UlBTfOq6/ekyPDkVSyS0byUdBAplyuQT/ZRu9/9YO3Vw+frRGFnJ c9BJQfaKHPvzIbMWGuMTA96747U72Q65Sq6/hSx/rW7nLIvu74EeVKWcnziAOoIDMp30 fFiPONi9SrcVDm3nio62SvPTmUB3k4FgoyGs44z32AujcShneUimI3ysWsGJkrQrFiEo yv+ZE5Ji+zSAH4tysKCfCcXh3m96sHHgQS3LuGLUnORdhuT3Oxsp3NAP93uNfTzGAg4K 40oA== X-Gm-Message-State: ACgBeo3QcOMw6hcm3vrzhY0MmsNqJKVTqK4/0SEw7utD9lG5bfgySFTd i6vNVWiyx9jIu3ArzY5JxihXGbnNuoxV1H4jtH8NvigcoB5R+yxlSpbylczY152dlIFVMUXQqtJ zjHCMki0m90X4oj84cKRcU3XF X-Received: by 2002:a05:6602:4019:b0:684:4cde:9e74 with SMTP id bk25-20020a056602401900b006844cde9e74mr16654iob.53.1660234415563; Thu, 11 Aug 2022 09:13:35 -0700 (PDT) X-Google-Smtp-Source: AA6agR6aerHoxC67XZ45dRi+nnFHBgH3aaV4pC70vmG8Qq33dGhvpVJki9kiKQSL4lwRWMdaL0bZUw== X-Received: by 2002:a05:6602:4019:b0:684:4cde:9e74 with SMTP id bk25-20020a056602401900b006844cde9e74mr16649iob.53.1660234415385; Thu, 11 Aug 2022 09:13:35 -0700 (PDT) Received: from localhost.localdomain (bras-base-aurron9127w-grc-35-70-27-3-10.dsl.bell.ca. [70.27.3.10]) by smtp.gmail.com with ESMTPSA id t1-20020a92ca81000000b002dd1c3c5c46sm3415429ilo.73.2022.08.11.09.13.33 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Thu, 11 Aug 2022 09:13:34 -0700 (PDT) From: Peter Xu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Hugh Dickins , "Kirill A . Shutemov" , Alistair Popple , peterx@redhat.com, Andrea Arcangeli , Minchan Kim , Andrew Morton , David Hildenbrand , Andi Kleen , Nadav Amit , Huang Ying , Vlastimil Babka Subject: [PATCH v4 1/7] mm/x86: Use SWP_TYPE_BITS in 3-level swap macros Date: Thu, 11 Aug 2022 12:13:25 -0400 Message-Id: <20220811161331.37055-2-peterx@redhat.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20220811161331.37055-1-peterx@redhat.com> References: <20220811161331.37055-1-peterx@redhat.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Replace all the magic "5" with the macro. Reviewed-by: David Hildenbrand Reviewed-by: Huang Ying Signed-off-by: Peter Xu Tested-by: Raghavendra K T --- arch/x86/include/asm/pgtable-3level.h | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/arch/x86/include/asm/pgtable-3level.h b/arch/x86/include/asm/p= gtable-3level.h index e896ebef8c24..28421a887209 100644 --- a/arch/x86/include/asm/pgtable-3level.h +++ b/arch/x86/include/asm/pgtable-3level.h @@ -256,10 +256,10 @@ static inline pud_t native_pudp_get_and_clear(pud_t *= pudp) /* We always extract/encode the offset by shifting it all the way up, and = then down again */ #define SWP_OFFSET_SHIFT (SWP_OFFSET_FIRST_BIT + SWP_TYPE_BITS) =20 -#define MAX_SWAPFILES_CHECK() BUILD_BUG_ON(MAX_SWAPFILES_SHIFT > 5) -#define __swp_type(x) (((x).val) & 0x1f) -#define __swp_offset(x) ((x).val >> 5) -#define __swp_entry(type, offset) ((swp_entry_t){(type) | (offset) << 5}) +#define MAX_SWAPFILES_CHECK() BUILD_BUG_ON(MAX_SWAPFILES_SHIFT > SWP_TYPE_= BITS) +#define __swp_type(x) (((x).val) & ((1UL << SWP_TYPE_BITS) - 1)) +#define __swp_offset(x) ((x).val >> SWP_TYPE_BITS) +#define __swp_entry(type, offset) ((swp_entry_t){(type) | (offset) << SWP_= TYPE_BITS}) =20 /* * Normally, __swp_entry() converts from arch-independent swp_entry_t to --=20 2.32.0 From nobody Sat Apr 11 10:47:11 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 882FDC19F2A for ; Thu, 11 Aug 2022 16:42:31 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S239020AbiHKQma (ORCPT ); Thu, 11 Aug 2022 12:42:30 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:52822 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S239182AbiHKQk6 (ORCPT ); Thu, 11 Aug 2022 12:40:58 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTP id 7F781A5C5D for ; Thu, 11 Aug 2022 09:13:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1660234419; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=m9J0o2gxjPkVRafIB7R87NmoXI0VU2ltkrT9Ru6AEjI=; b=foQFPNwdWgpeCALFHqPbqDZ/O3iGjk/aO71b7h5s6k0Iu79g86AZ7+mpwIDzl2v4tphOZD CYtEKltjrVQ1MC/AEBzSnDHBO1rMxBL9kXFeiWvV/kyenBmX7nreVs9yYdgMUGNmwLlz+q YBGJ6JU8J4BrTbA8HS0OXEnNDZOSAj4= Received: from mail-il1-f197.google.com (mail-il1-f197.google.com [209.85.166.197]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_128_GCM_SHA256) id us-mta-64-qfaHNue-NFGA8MWcKMBfAw-1; Thu, 11 Aug 2022 12:13:38 -0400 X-MC-Unique: qfaHNue-NFGA8MWcKMBfAw-1 Received: by mail-il1-f197.google.com with SMTP id w6-20020a056e021a6600b002dea6904708so12726372ilv.6 for ; Thu, 11 Aug 2022 09:13:38 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc; bh=m9J0o2gxjPkVRafIB7R87NmoXI0VU2ltkrT9Ru6AEjI=; b=ZMeRT0VhOmeDVNMPqpGCfqU1FGcC28Dazd1dWF5r7JyE7+MiFp4LuYsY1uj3VJpcQD gDnA5hqf5kmca/V8XRUYpMn5Vitkv6s8ehyKOby291o6TSv5TCNRSbMW0/ccCN3RelFz zWUq0g3/iKqh1K6lYzzguNWnWyecL0BV34jg9enXlmpshKapQ9RMGNDl9bFWADws7cy6 ZJh4FCTvPvZ/H8xA5a1Oytc+Ua9+jej9EY5WenfJmG7yOYowFdMD8LDT5cyTQKqfV6Sl Cy6G1ACq1hK4hsBzV0KJzh05fHaDjPN8NzqVYreSt/uqDT/IFvBeVpXHzv0NGfzIKm86 ANGg== X-Gm-Message-State: ACgBeo3aKl5CFxwAtUhGnIVIVan0FBFIiCnqaOv6tsTR4CfbFl4Vjrjm BxUS7kwuK2RNgrsC+rqOWN7hbUBBlAX6cLVHVmt0G6R4IdGe3p678I1gCvpQr1mN2igc0n3hgbY Ve5jLcmJeZqxL5ThX+4WMEtgi X-Received: by 2002:a6b:c505:0:b0:67c:dcd:a5b2 with SMTP id v5-20020a6bc505000000b0067c0dcda5b2mr17914iof.37.1660234417482; Thu, 11 Aug 2022 09:13:37 -0700 (PDT) X-Google-Smtp-Source: AA6agR4/bnpktdH+KIUEDaTM72sZY/jQsDJec0sJqNd+QvV0NBcU4DHXpzsUzSl7BCeITxJOQpuLrg== X-Received: by 2002:a6b:c505:0:b0:67c:dcd:a5b2 with SMTP id v5-20020a6bc505000000b0067c0dcda5b2mr17905iof.37.1660234417290; Thu, 11 Aug 2022 09:13:37 -0700 (PDT) Received: from localhost.localdomain (bras-base-aurron9127w-grc-35-70-27-3-10.dsl.bell.ca. [70.27.3.10]) by smtp.gmail.com with ESMTPSA id t1-20020a92ca81000000b002dd1c3c5c46sm3415429ilo.73.2022.08.11.09.13.35 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Thu, 11 Aug 2022 09:13:36 -0700 (PDT) From: Peter Xu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Hugh Dickins , "Kirill A . Shutemov" , Alistair Popple , peterx@redhat.com, Andrea Arcangeli , Minchan Kim , Andrew Morton , David Hildenbrand , Andi Kleen , Nadav Amit , Huang Ying , Vlastimil Babka Subject: [PATCH v4 2/7] mm/swap: Comment all the ifdef in swapops.h Date: Thu, 11 Aug 2022 12:13:26 -0400 Message-Id: <20220811161331.37055-3-peterx@redhat.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20220811161331.37055-1-peterx@redhat.com> References: <20220811161331.37055-1-peterx@redhat.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" swapops.h contains quite a few layers of ifdef, some of the "else" and "endif" doesn't get proper comment on the macro so it's hard to follow on what are they referring to. Add the comments. Suggested-by: Nadav Amit Reviewed-by: Huang Ying Signed-off-by: Peter Xu Reviewed-by: Alistair Popple Tested-by: Raghavendra K T --- include/linux/swapops.h | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/include/linux/swapops.h b/include/linux/swapops.h index a3d435bf9f97..3a2901ff4f1e 100644 --- a/include/linux/swapops.h +++ b/include/linux/swapops.h @@ -247,8 +247,8 @@ extern void migration_entry_wait(struct mm_struct *mm, = pmd_t *pmd, #ifdef CONFIG_HUGETLB_PAGE extern void __migration_entry_wait_huge(pte_t *ptep, spinlock_t *ptl); extern void migration_entry_wait_huge(struct vm_area_struct *vma, pte_t *p= te); -#endif -#else +#endif /* CONFIG_HUGETLB_PAGE */ +#else /* CONFIG_MIGRATION */ static inline swp_entry_t make_readable_migration_entry(pgoff_t offset) { return swp_entry(0, 0); @@ -276,7 +276,7 @@ static inline void migration_entry_wait(struct mm_struc= t *mm, pmd_t *pmd, #ifdef CONFIG_HUGETLB_PAGE static inline void __migration_entry_wait_huge(pte_t *ptep, spinlock_t *pt= l) { } static inline void migration_entry_wait_huge(struct vm_area_struct *vma, p= te_t *pte) { } -#endif +#endif /* CONFIG_HUGETLB_PAGE */ static inline int is_writable_migration_entry(swp_entry_t entry) { return 0; @@ -286,7 +286,7 @@ static inline int is_readable_migration_entry(swp_entry= _t entry) return 0; } =20 -#endif +#endif /* CONFIG_MIGRATION */ =20 typedef unsigned long pte_marker; =20 @@ -426,7 +426,7 @@ static inline int is_pmd_migration_entry(pmd_t pmd) { return is_swap_pmd(pmd) && is_migration_entry(pmd_to_swp_entry(pmd)); } -#else +#else /* CONFIG_ARCH_ENABLE_THP_MIGRATION */ static inline int set_pmd_migration_entry(struct page_vma_mapped_walk *pvm= w, struct page *page) { @@ -455,7 +455,7 @@ static inline int is_pmd_migration_entry(pmd_t pmd) { return 0; } -#endif +#endif /* CONFIG_ARCH_ENABLE_THP_MIGRATION */ =20 #ifdef CONFIG_MEMORY_FAILURE =20 @@ -495,7 +495,7 @@ static inline void num_poisoned_pages_sub(long i) atomic_long_sub(i, &num_poisoned_pages); } =20 -#else +#else /* CONFIG_MEMORY_FAILURE */ =20 static inline swp_entry_t make_hwpoison_entry(struct page *page) { @@ -514,7 +514,7 @@ static inline void num_poisoned_pages_inc(void) static inline void num_poisoned_pages_sub(long i) { } -#endif +#endif /* CONFIG_MEMORY_FAILURE */ =20 static inline int non_swap_entry(swp_entry_t entry) { --=20 2.32.0 From nobody Sat Apr 11 10:47:11 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id A1315C19F2A for ; Thu, 11 Aug 2022 16:42:40 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S239077AbiHKQmi (ORCPT ); Thu, 11 Aug 2022 12:42:38 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35864 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S239194AbiHKQlB (ORCPT ); Thu, 11 Aug 2022 12:41:01 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTP id 2F97DA50C1 for ; Thu, 11 Aug 2022 09:13:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1660234421; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=tV5La4wEOuDKwImLzug6SnOpAoLEybFChef0YUGwxZI=; b=afmzxUi4OsRPgSJ5tVXhXpxC1rH3gmxQpIhSHxMdUytfPZlYlParCfqF2rx7yi0WsF4xUt U4R1+yp/3dxDfCMqHalfuzDJ2yUspEU9nB+yOL9EYQBhyyMVfSoaOl31wbyscjQCu/AYUP muZxEMn6CtKXkZHcB/C00f56eZnwtAg= Received: from mail-il1-f199.google.com (mail-il1-f199.google.com [209.85.166.199]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_128_GCM_SHA256) id us-mta-21-vGKLcBSbPXOGIcLJaddMkA-1; Thu, 11 Aug 2022 12:13:40 -0400 X-MC-Unique: vGKLcBSbPXOGIcLJaddMkA-1 Received: by mail-il1-f199.google.com with SMTP id n13-20020a056e02140d00b002dfa5464967so11412917ilo.19 for ; Thu, 11 Aug 2022 09:13:39 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc; bh=tV5La4wEOuDKwImLzug6SnOpAoLEybFChef0YUGwxZI=; b=hrc1fxSgYWO86wYpXL/v7iYu84sk0HmhSB9dNVfrIt3X1ryJol6HO4B5ZnzZJhPVD/ nzyVpXRzqYB/zVEKXZ4xOQ2+d91QQoMMOj05jJ57IgjTkjlUYwcgkrktVQXEkeDyH/dT mrDbpKRRl/xMUZVEHBIoYx3LII/frA7XJ3+3sUD5WapGKruEnappKmIBWnARuplIwXU7 7enNmhH1mNhzyVWUkAOTtp+Ne78TtcOdX1r3FEq6AYIcIclIVEhdl78qBe7UTWGbpwWY J0w7dOLgU5u4Aok09Psj8lrx/161poVI4DwTM61sQsU0/43wJ4aMeYy0iuQtg+Gazd1+ ICzA== X-Gm-Message-State: ACgBeo1xPehMDADvKY2NUSzFniITsQUFEZpWqBcsI606v6m5ckhvHz1R nRYVKMJUdrYOM4yUV+zwfkOWKTNwbMwEQ9/irm633qbynSKhX/1HXGKFE/SbP3eE3uw1o9Qj5nv R8IKeI1dJsKa15jrAm+qjx0tW X-Received: by 2002:a05:6602:2182:b0:684:fb55:663f with SMTP id b2-20020a056602218200b00684fb55663fmr31092iob.60.1660234419273; Thu, 11 Aug 2022 09:13:39 -0700 (PDT) X-Google-Smtp-Source: AA6agR5jIQLvYUpX1q44iT+XDMj9ZK15CNYkE0bFGQEL63lTXbeHWM+wVqA+5wMIWo1nNzyBJbp0sw== X-Received: by 2002:a05:6602:2182:b0:684:fb55:663f with SMTP id b2-20020a056602218200b00684fb55663fmr31074iob.60.1660234419026; Thu, 11 Aug 2022 09:13:39 -0700 (PDT) Received: from localhost.localdomain (bras-base-aurron9127w-grc-35-70-27-3-10.dsl.bell.ca. [70.27.3.10]) by smtp.gmail.com with ESMTPSA id t1-20020a92ca81000000b002dd1c3c5c46sm3415429ilo.73.2022.08.11.09.13.37 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Thu, 11 Aug 2022 09:13:38 -0700 (PDT) From: Peter Xu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Hugh Dickins , "Kirill A . Shutemov" , Alistair Popple , peterx@redhat.com, Andrea Arcangeli , Minchan Kim , Andrew Morton , David Hildenbrand , Andi Kleen , Nadav Amit , Huang Ying , Vlastimil Babka Subject: [PATCH v4 3/7] mm/swap: Add swp_offset_pfn() to fetch PFN from swap entry Date: Thu, 11 Aug 2022 12:13:27 -0400 Message-Id: <20220811161331.37055-4-peterx@redhat.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20220811161331.37055-1-peterx@redhat.com> References: <20220811161331.37055-1-peterx@redhat.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" We've got a bunch of special swap entries that stores PFN inside the swap offset fields. To fetch the PFN, normally the user just calls swp_offset() assuming that'll be the PFN. Add a helper swp_offset_pfn() to fetch the PFN instead, fetching only the max possible length of a PFN on the host, meanwhile doing proper check with MAX_PHYSMEM_BITS to make sure the swap offsets can actually store the PFNs properly always using the BUILD_BUG_ON() in is_pfn_swap_entry(). One reason to do so is we never tried to sanitize whether swap offset can really fit for storing PFN. At the meantime, this patch also prepares us with the future possibility to store more information inside the swp offset field, so assuming "swp_offset(entry)" to be the PFN will not stand any more very soon. Replace many of the swp_offset() callers to use swp_offset_pfn() where proper. Note that many of the existing users are not candidates for the replacement, e.g.: (1) When the swap entry is not a pfn swap entry at all, or, (2) when we wanna keep the whole swp_offset but only change the swp type. For the latter, it can happen when fork() triggered on a write-migration swap entry pte, we may want to only change the migration type from write->read but keep the rest, so it's not "fetching PFN" but "changing swap type only". They're left aside so that when there're more information within the swp offset they'll be carried over naturally in those cases. Since at it, dropping hwpoison_entry_to_pfn() because that's exactly what the new swp_offset_pfn() is about. Signed-off-by: Peter Xu Reviewed-by: "Huang, Ying" Tested-by: Raghavendra K T --- arch/arm64/mm/hugetlbpage.c | 2 +- fs/proc/task_mmu.c | 20 +++++++++++++++++--- include/linux/swapops.h | 35 +++++++++++++++++++++++++++++------ mm/hmm.c | 2 +- mm/memory-failure.c | 2 +- mm/page_vma_mapped.c | 6 +++--- 6 files changed, 52 insertions(+), 15 deletions(-) diff --git a/arch/arm64/mm/hugetlbpage.c b/arch/arm64/mm/hugetlbpage.c index 0795028f017c..35e9a468d13e 100644 --- a/arch/arm64/mm/hugetlbpage.c +++ b/arch/arm64/mm/hugetlbpage.c @@ -245,7 +245,7 @@ static inline struct folio *hugetlb_swap_entry_to_folio= (swp_entry_t entry) { VM_BUG_ON(!is_migration_entry(entry) && !is_hwpoison_entry(entry)); =20 - return page_folio(pfn_to_page(swp_offset(entry))); + return page_folio(pfn_to_page(swp_offset_pfn(entry))); } =20 void set_huge_pte_at(struct mm_struct *mm, unsigned long addr, diff --git a/fs/proc/task_mmu.c b/fs/proc/task_mmu.c index d56c65f98d00..b3e79128fca0 100644 --- a/fs/proc/task_mmu.c +++ b/fs/proc/task_mmu.c @@ -1419,9 +1419,19 @@ static pagemap_entry_t pte_to_pagemap_entry(struct p= agemapread *pm, if (pte_swp_uffd_wp(pte)) flags |=3D PM_UFFD_WP; entry =3D pte_to_swp_entry(pte); - if (pm->show_pfn) + if (pm->show_pfn) { + pgoff_t offset; + /* + * For PFN swap offsets, keeping the offset field + * to be PFN only to be compatible with old smaps. + */ + if (is_pfn_swap_entry(entry)) + offset =3D swp_offset_pfn(entry); + else + offset =3D swp_offset(entry); frame =3D swp_type(entry) | - (swp_offset(entry) << MAX_SWAPFILES_SHIFT); + (offset << MAX_SWAPFILES_SHIFT); + } flags |=3D PM_SWAP; migration =3D is_migration_entry(entry); if (is_pfn_swap_entry(entry)) @@ -1478,7 +1488,11 @@ static int pagemap_pmd_range(pmd_t *pmdp, unsigned l= ong addr, unsigned long end, unsigned long offset; =20 if (pm->show_pfn) { - offset =3D swp_offset(entry) + + if (is_pfn_swap_entry(entry)) + offset =3D swp_offset_pfn(entry); + else + offset =3D swp_offset(entry); + offset =3D offset + ((addr & ~PMD_MASK) >> PAGE_SHIFT); frame =3D swp_type(entry) | (offset << MAX_SWAPFILES_SHIFT); diff --git a/include/linux/swapops.h b/include/linux/swapops.h index 3a2901ff4f1e..bd4c6f0c2103 100644 --- a/include/linux/swapops.h +++ b/include/linux/swapops.h @@ -23,6 +23,20 @@ #define SWP_TYPE_SHIFT (BITS_PER_XA_VALUE - MAX_SWAPFILES_SHIFT) #define SWP_OFFSET_MASK ((1UL << SWP_TYPE_SHIFT) - 1) =20 +/* + * Definitions only for PFN swap entries (see is_pfn_swap_entry()). To + * store PFN, we only need SWP_PFN_BITS bits. Each of the pfn swap entries + * can use the extra bits to store other information besides PFN. + */ +#ifdef MAX_PHYSMEM_BITS +#define SWP_PFN_BITS (MAX_PHYSMEM_BITS - PAGE_SHIFT) +#else /* MAX_PHYSMEM_BITS */ +#define SWP_PFN_BITS (BITS_PER_LONG - PAGE_SHIFT) +#endif /* MAX_PHYSMEM_BITS */ +#define SWP_PFN_MASK (BIT(SWP_PFN_BITS) - 1) + +static inline bool is_pfn_swap_entry(swp_entry_t entry); + /* Clear all flags but only keep swp_entry_t related information */ static inline pte_t pte_swp_clear_flags(pte_t pte) { @@ -64,6 +78,17 @@ static inline pgoff_t swp_offset(swp_entry_t entry) return entry.val & SWP_OFFSET_MASK; } =20 +/* + * This should only be called upon a pfn swap entry to get the PFN stored + * in the swap entry. Please refers to is_pfn_swap_entry() for definition + * of pfn swap entry. + */ +static inline unsigned long swp_offset_pfn(swp_entry_t entry) +{ + VM_BUG_ON(!is_pfn_swap_entry(entry)); + return swp_offset(entry) & SWP_PFN_MASK; +} + /* check whether a pte points to a swap entry */ static inline int is_swap_pte(pte_t pte) { @@ -369,7 +394,7 @@ static inline int pte_none_mostly(pte_t pte) =20 static inline struct page *pfn_swap_entry_to_page(swp_entry_t entry) { - struct page *p =3D pfn_to_page(swp_offset(entry)); + struct page *p =3D pfn_to_page(swp_offset_pfn(entry)); =20 /* * Any use of migration entries may only occur while the @@ -387,6 +412,9 @@ static inline struct page *pfn_swap_entry_to_page(swp_e= ntry_t entry) */ static inline bool is_pfn_swap_entry(swp_entry_t entry) { + /* Make sure the swp offset can always store the needed fields */ + BUILD_BUG_ON(SWP_TYPE_SHIFT < SWP_PFN_BITS); + return is_migration_entry(entry) || is_device_private_entry(entry) || is_device_exclusive_entry(entry); } @@ -475,11 +503,6 @@ static inline int is_hwpoison_entry(swp_entry_t entry) return swp_type(entry) =3D=3D SWP_HWPOISON; } =20 -static inline unsigned long hwpoison_entry_to_pfn(swp_entry_t entry) -{ - return swp_offset(entry); -} - static inline void num_poisoned_pages_inc(void) { atomic_long_inc(&num_poisoned_pages); diff --git a/mm/hmm.c b/mm/hmm.c index f2aa63b94d9b..3850fb625dda 100644 --- a/mm/hmm.c +++ b/mm/hmm.c @@ -253,7 +253,7 @@ static int hmm_vma_handle_pte(struct mm_walk *walk, uns= igned long addr, cpu_flags =3D HMM_PFN_VALID; if (is_writable_device_private_entry(entry)) cpu_flags |=3D HMM_PFN_WRITE; - *hmm_pfn =3D swp_offset(entry) | cpu_flags; + *hmm_pfn =3D swp_offset_pfn(entry) | cpu_flags; return 0; } =20 diff --git a/mm/memory-failure.c b/mm/memory-failure.c index 0dfed9d7b273..e48f6f6a259d 100644 --- a/mm/memory-failure.c +++ b/mm/memory-failure.c @@ -632,7 +632,7 @@ static int check_hwpoisoned_entry(pte_t pte, unsigned l= ong addr, short shift, swp_entry_t swp =3D pte_to_swp_entry(pte); =20 if (is_hwpoison_entry(swp)) - pfn =3D hwpoison_entry_to_pfn(swp); + pfn =3D swp_offset_pfn(swp); } =20 if (!pfn || pfn !=3D poisoned_pfn) diff --git a/mm/page_vma_mapped.c b/mm/page_vma_mapped.c index 8e9e574d535a..93e13fc17d3c 100644 --- a/mm/page_vma_mapped.c +++ b/mm/page_vma_mapped.c @@ -86,7 +86,7 @@ static bool check_pte(struct page_vma_mapped_walk *pvmw) !is_device_exclusive_entry(entry)) return false; =20 - pfn =3D swp_offset(entry); + pfn =3D swp_offset_pfn(entry); } else if (is_swap_pte(*pvmw->pte)) { swp_entry_t entry; =20 @@ -96,7 +96,7 @@ static bool check_pte(struct page_vma_mapped_walk *pvmw) !is_device_exclusive_entry(entry)) return false; =20 - pfn =3D swp_offset(entry); + pfn =3D swp_offset_pfn(entry); } else { if (!pte_present(*pvmw->pte)) return false; @@ -221,7 +221,7 @@ bool page_vma_mapped_walk(struct page_vma_mapped_walk *= pvmw) return not_found(pvmw); entry =3D pmd_to_swp_entry(pmde); if (!is_migration_entry(entry) || - !check_pmd(swp_offset(entry), pvmw)) + !check_pmd(swp_offset_pfn(entry), pvmw)) return not_found(pvmw); return true; } --=20 2.32.0 From nobody Sat Apr 11 10:47:11 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id C8F64C19F2A for ; Thu, 11 Aug 2022 16:42:23 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238963AbiHKQmV (ORCPT ); Thu, 11 Aug 2022 12:42:21 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50704 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S239210AbiHKQlC (ORCPT ); Thu, 11 Aug 2022 12:41:02 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTP id E5964A5C63 for ; Thu, 11 Aug 2022 09:13:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1660234422; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=3RIarNO4bl/SlYdwgU6TGCIMSMDAhYZFkoNFGsJp8ZY=; b=Ve3XDgeHzhWl1vN60y2FbtItfPts8jfi6EAA2rmqQ1gwGq9VPqMZmI6sAcqNLeHTob5pkG iDAd1LmFqFFnjYCbLAChj6sLqWTUMIwRseYb37zqxHfPOUPPyhaU6IymFMVG7/h+rLRy/E kiJNpP6QimDI6AA1Jrgf6CmF8y4C9w4= Received: from mail-io1-f69.google.com (mail-io1-f69.google.com [209.85.166.69]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_128_GCM_SHA256) id us-mta-28-Qg9sZUtAONSi6vCROLRXPQ-1; Thu, 11 Aug 2022 12:13:41 -0400 X-MC-Unique: Qg9sZUtAONSi6vCROLRXPQ-1 Received: by mail-io1-f69.google.com with SMTP id j8-20020a6b7948000000b0067c2923d1b8so9797860iop.6 for ; Thu, 11 Aug 2022 09:13:41 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc; bh=3RIarNO4bl/SlYdwgU6TGCIMSMDAhYZFkoNFGsJp8ZY=; b=iQ+q4YCOKNlVEOVMrt0qwc3oHTyKFqITFcXBgCckhFPcJlct/bmFmotRqkF4aQmW3/ shkIQNiJhonnkdWfzJR8PTOWvZ7j3ZSxvu1ILYVg4yWnKVfkdZt8xVIst8Ex+gvuqVrr noolZAnaFWajahyqz1AjiwQbnR3EryqpiR2fC+SDk1SbfRxge3ggXAL9akqhjYGVn2rB mnDXLswuYA+ehlN3nG4jbwC++V4nTe9ayNf3N/OwTUbNsSx2iWGoxeDn3TNgTg9pdsJt JkBhN/Ll9hMB6wwJXAmd6I4JkASk14AoycXaokHBRszBcklZsOzIPvKguE0DombgCmqB 5Ozg== X-Gm-Message-State: ACgBeo2IlN2qCWl7FjnF+H2ITanZL/UFOAh8Jq8YaFpGJ8ZB5MrBsmiF IC1Z8P3T7hdPpS2uGTB+AUj3gokMuoSEELz+WQhPtKR4v1e4CHg1kmyLi2WIsW7ZRxIUhT5NUGn CKgXSeu9MBVoE4oGNgufK+dft X-Received: by 2002:a6b:c343:0:b0:67c:6033:a682 with SMTP id t64-20020a6bc343000000b0067c6033a682mr25999iof.148.1660234420822; Thu, 11 Aug 2022 09:13:40 -0700 (PDT) X-Google-Smtp-Source: AA6agR6oIBb2BB4Q/hh/sdEHHNGipMyiCcZTfi7mlLc8HUt82uGH8takFsvaPCiWYe/hV/AqCvZxHw== X-Received: by 2002:a6b:c343:0:b0:67c:6033:a682 with SMTP id t64-20020a6bc343000000b0067c6033a682mr25990iof.148.1660234420598; Thu, 11 Aug 2022 09:13:40 -0700 (PDT) Received: from localhost.localdomain (bras-base-aurron9127w-grc-35-70-27-3-10.dsl.bell.ca. [70.27.3.10]) by smtp.gmail.com with ESMTPSA id t1-20020a92ca81000000b002dd1c3c5c46sm3415429ilo.73.2022.08.11.09.13.39 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Thu, 11 Aug 2022 09:13:40 -0700 (PDT) From: Peter Xu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Hugh Dickins , "Kirill A . Shutemov" , Alistair Popple , peterx@redhat.com, Andrea Arcangeli , Minchan Kim , Andrew Morton , David Hildenbrand , Andi Kleen , Nadav Amit , Huang Ying , Vlastimil Babka Subject: [PATCH v4 4/7] mm/thp: Carry over dirty bit when thp splits on pmd Date: Thu, 11 Aug 2022 12:13:28 -0400 Message-Id: <20220811161331.37055-5-peterx@redhat.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20220811161331.37055-1-peterx@redhat.com> References: <20220811161331.37055-1-peterx@redhat.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Carry over the dirty bit from pmd to pte when a huge pmd splits. It shouldn't be a correctness issue since when pmd_dirty() we'll have the page marked dirty anyway, however having dirty bit carried over helps the next initial writes of split ptes on some archs like x86. Reviewed-by: Huang Ying Signed-off-by: Peter Xu Tested-by: Raghavendra K T --- mm/huge_memory.c | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 3222b40a0f6d..2f68e034ddec 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -2027,7 +2027,7 @@ static void __split_huge_pmd_locked(struct vm_area_st= ruct *vma, pmd_t *pmd, pgtable_t pgtable; pmd_t old_pmd, _pmd; bool young, write, soft_dirty, pmd_migration =3D false, uffd_wp =3D false; - bool anon_exclusive =3D false; + bool anon_exclusive =3D false, dirty =3D false; unsigned long addr; int i; =20 @@ -2116,8 +2116,10 @@ static void __split_huge_pmd_locked(struct vm_area_s= truct *vma, pmd_t *pmd, uffd_wp =3D pmd_swp_uffd_wp(old_pmd); } else { page =3D pmd_page(old_pmd); - if (pmd_dirty(old_pmd)) + if (pmd_dirty(old_pmd)) { + dirty =3D true; SetPageDirty(page); + } write =3D pmd_write(old_pmd); young =3D pmd_young(old_pmd); soft_dirty =3D pmd_soft_dirty(old_pmd); @@ -2183,6 +2185,9 @@ static void __split_huge_pmd_locked(struct vm_area_st= ruct *vma, pmd_t *pmd, entry =3D pte_wrprotect(entry); if (!young) entry =3D pte_mkold(entry); + /* NOTE: this may set soft-dirty too on some archs */ + if (dirty) + entry =3D pte_mkdirty(entry); if (soft_dirty) entry =3D pte_mksoft_dirty(entry); if (uffd_wp) --=20 2.32.0 From nobody Sat Apr 11 10:47:11 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id D2BAAC19F2A for ; Thu, 11 Aug 2022 16:42:36 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S239041AbiHKQme (ORCPT ); Thu, 11 Aug 2022 12:42:34 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50954 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S239235AbiHKQlG (ORCPT ); Thu, 11 Aug 2022 12:41:06 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTP id 5BA09A5C6D for ; Thu, 11 Aug 2022 09:13:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1660234424; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=wcBPlgoBrj0uOf3oLcjW7xdC8Px/qOVmL+drDcWQ9zU=; b=ac8Gq8eGLD70CwFRaAEckItyLGHvfAz82ZcxywSOoZdsaqu7ZxHfGM/Oj3mV9p65Ow79lm Jf4arGEX/ZfXa2vFdRPZ3sWg2sKOcQ6seNeuofPzg/ZrsYDiyfB2ZDxp4m+TCIKaugVdVZ fWItRg8Ty3UzKG0j8Pgp4IOk634Fdfc= Received: from mail-il1-f198.google.com (mail-il1-f198.google.com [209.85.166.198]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_128_GCM_SHA256) id us-mta-445-beIqJO_QNNGC9Z3eo-WMvQ-1; Thu, 11 Aug 2022 12:13:43 -0400 X-MC-Unique: beIqJO_QNNGC9Z3eo-WMvQ-1 Received: by mail-il1-f198.google.com with SMTP id d6-20020a056e020c0600b002deca741bc2so12840771ile.18 for ; Thu, 11 Aug 2022 09:13:43 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc; bh=wcBPlgoBrj0uOf3oLcjW7xdC8Px/qOVmL+drDcWQ9zU=; b=EMyUZxJbHgyruDWOJGD5uEXLojvjpRcX2BoBuoVdWMJ/L+4B19jWeHZLTJ6JmM57+7 UqKQaolVB83xvXLJwTZChBvRyL6Ur9dwMkDbi8sf1S19VvAdfLNM+TPevHQva/rkQLNX gpFI+6jgK8cSDcD8TKwk2KzdDmYSh3XpYpeWq1mBof89SfEqSp7b+loBGwnBgHgjWHct q1UAsnENaLpQr9/vK+IOTeG1cQ0/Zxl/3GQLnXvrmKbnnOc5DdSWY/I/V9MvAOIHlm1Y sapYJju3iqouZIq/f6QV4VIz4hr8CTt89np1k2jbhjhG5txCRm03hECPvOKDswpN6B5O qnmQ== X-Gm-Message-State: ACgBeo0tFhiDuiYPUMnD1Lmz7OrTVa7wkM9ezVuq4MTDFjxB9EaqyYaF v+a9UcnV7v1JS1NR2aO6mOugCk5GDtV1DQCNi5eYPkDrkq0Wb/u8JnXKnloDvF/T7gaoCQtPAfF 6/2MLQDKBU+csLSSRsFss6O+S X-Received: by 2002:a05:6602:3c8:b0:672:4e60:7294 with SMTP id g8-20020a05660203c800b006724e607294mr19032iov.17.1660234422697; Thu, 11 Aug 2022 09:13:42 -0700 (PDT) X-Google-Smtp-Source: AA6agR4a4wc03JiaOd+Wy1g/LSTkpve80CiM3YnLVxORIPuirDVkEdT56xr6TTENWxL/H/ODMSTXkQ== X-Received: by 2002:a05:6602:3c8:b0:672:4e60:7294 with SMTP id g8-20020a05660203c800b006724e607294mr19017iov.17.1660234422392; Thu, 11 Aug 2022 09:13:42 -0700 (PDT) Received: from localhost.localdomain (bras-base-aurron9127w-grc-35-70-27-3-10.dsl.bell.ca. [70.27.3.10]) by smtp.gmail.com with ESMTPSA id t1-20020a92ca81000000b002dd1c3c5c46sm3415429ilo.73.2022.08.11.09.13.40 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Thu, 11 Aug 2022 09:13:41 -0700 (PDT) From: Peter Xu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Hugh Dickins , "Kirill A . Shutemov" , Alistair Popple , peterx@redhat.com, Andrea Arcangeli , Minchan Kim , Andrew Morton , David Hildenbrand , Andi Kleen , Nadav Amit , Huang Ying , Vlastimil Babka Subject: [PATCH v4 5/7] mm: Remember young/dirty bit for page migrations Date: Thu, 11 Aug 2022 12:13:29 -0400 Message-Id: <20220811161331.37055-6-peterx@redhat.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20220811161331.37055-1-peterx@redhat.com> References: <20220811161331.37055-1-peterx@redhat.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" When page migration happens, we always ignore the young/dirty bit settings in the old pgtable, and marking the page as old in the new page table using either pte_mkold() or pmd_mkold(), and keeping the pte clean. That's fine from functional-wise, but that's not friendly to page reclaim because the moving page can be actively accessed within the procedure. Not to mention hardware setting the young bit can bring quite some overhead on some systems, e.g. x86_64 needs a few hundreds nanoseconds to set the bit. The same slowdown problem to dirty bits when the memory is first written after page migration happened. Actually we can easily remember the A/D bit configuration and recover the information after the page is migrated. To achieve it, define a new set of bits in the migration swap offset field to cache the A/D bits for old pte. Then when removing/recovering the migration entry, we can recover the A/D bits even if the page changed. One thing to mention is that here we used max_swapfile_size() to detect how many swp offset bits we have, and we'll only enable this feature if we know the swp offset is big enough to store both the PFN value and the A/D bits. Otherwise the A/D bits are dropped like before. Signed-off-by: Peter Xu Reviewed-by: "Huang, Ying" Tested-by: Raghavendra K T --- include/linux/swapops.h | 99 +++++++++++++++++++++++++++++++++++++++++ mm/huge_memory.c | 18 +++++++- mm/migrate.c | 6 ++- mm/migrate_device.c | 6 +++ mm/rmap.c | 5 ++- 5 files changed, 130 insertions(+), 4 deletions(-) diff --git a/include/linux/swapops.h b/include/linux/swapops.h index bd4c6f0c2103..36e462e116af 100644 --- a/include/linux/swapops.h +++ b/include/linux/swapops.h @@ -8,6 +8,10 @@ =20 #ifdef CONFIG_MMU =20 +#ifdef CONFIG_SWAP +#include +#endif /* CONFIG_SWAP */ + /* * swapcache pages are stored in the swapper_space radix tree. We want to * get good packing density in that tree, so the index should be dense in @@ -35,6 +39,31 @@ #endif /* MAX_PHYSMEM_BITS */ #define SWP_PFN_MASK (BIT(SWP_PFN_BITS) - 1) =20 +/** + * Migration swap entry specific bitfield definitions. Layout: + * + * |----------+--------------------| + * | swp_type | swp_offset | + * |----------+--------+-+-+-------| + * | | resv |D|A| PFN | + * |----------+--------+-+-+-------| + * + * @SWP_MIG_YOUNG_BIT: Whether the page used to have young bit set (bit A) + * @SWP_MIG_DIRTY_BIT: Whether the page used to have dirty bit set (bit D) + * + * Note: A/D bits will be stored in migration entries iff there're enough + * free bits in arch specific swp offset. By default we'll ignore A/D bits + * when migrating a page. Please refer to migration_entry_supports_ad() + * for more information. If there're more bits besides PFN and A/D bits, + * they should be reserved and always be zeros. + */ +#define SWP_MIG_YOUNG_BIT (SWP_PFN_BITS) +#define SWP_MIG_DIRTY_BIT (SWP_PFN_BITS + 1) +#define SWP_MIG_TOTAL_BITS (SWP_PFN_BITS + 2) + +#define SWP_MIG_YOUNG BIT(SWP_MIG_YOUNG_BIT) +#define SWP_MIG_DIRTY BIT(SWP_MIG_DIRTY_BIT) + static inline bool is_pfn_swap_entry(swp_entry_t entry); =20 /* Clear all flags but only keep swp_entry_t related information */ @@ -265,6 +294,57 @@ static inline swp_entry_t make_writable_migration_entr= y(pgoff_t offset) return swp_entry(SWP_MIGRATION_WRITE, offset); } =20 +/* + * Returns whether the host has large enough swap offset field to support + * carrying over pgtable A/D bits for page migrations. The result is + * pretty much arch specific. + */ +static inline bool migration_entry_supports_ad(void) +{ + /* + * max_swapfile_size() returns the max supported swp-offset plus 1. + * We can support the migration A/D bits iff the pfn swap entry has + * the offset large enough to cover all of them (PFN, A & D bits). + */ +#ifdef CONFIG_SWAP + return max_swapfile_size() >=3D (1UL << SWP_MIG_TOTAL_BITS); +#else /* CONFIG_SWAP */ + return false; +#endif /* CONFIG_SWAP */ +} + +static inline swp_entry_t make_migration_entry_young(swp_entry_t entry) +{ + if (migration_entry_supports_ad()) + return swp_entry(swp_type(entry), + swp_offset(entry) | SWP_MIG_YOUNG); + return entry; +} + +static inline bool is_migration_entry_young(swp_entry_t entry) +{ + if (migration_entry_supports_ad()) + return swp_offset(entry) & SWP_MIG_YOUNG; + /* Keep the old behavior of aging page after migration */ + return false; +} + +static inline swp_entry_t make_migration_entry_dirty(swp_entry_t entry) +{ + if (migration_entry_supports_ad()) + return swp_entry(swp_type(entry), + swp_offset(entry) | SWP_MIG_DIRTY); + return entry; +} + +static inline bool is_migration_entry_dirty(swp_entry_t entry) +{ + if (migration_entry_supports_ad()) + return swp_offset(entry) & SWP_MIG_DIRTY; + /* Keep the old behavior of clean page after migration */ + return false; +} + extern void __migration_entry_wait(struct mm_struct *mm, pte_t *ptep, spinlock_t *ptl); extern void migration_entry_wait(struct mm_struct *mm, pmd_t *pmd, @@ -311,6 +391,25 @@ static inline int is_readable_migration_entry(swp_entr= y_t entry) return 0; } =20 +static inline swp_entry_t make_migration_entry_young(swp_entry_t entry) +{ + return entry; +} + +static inline bool is_migration_entry_young(swp_entry_t entry) +{ + return false; +} + +static inline swp_entry_t make_migration_entry_dirty(swp_entry_t entry) +{ + return entry; +} + +static inline bool is_migration_entry_dirty(swp_entry_t entry) +{ + return false; +} #endif /* CONFIG_MIGRATION */ =20 typedef unsigned long pte_marker; diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 2f68e034ddec..ac858fd9c1f1 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -2111,7 +2111,8 @@ static void __split_huge_pmd_locked(struct vm_area_st= ruct *vma, pmd_t *pmd, write =3D is_writable_migration_entry(entry); if (PageAnon(page)) anon_exclusive =3D is_readable_exclusive_migration_entry(entry); - young =3D false; + young =3D is_migration_entry_young(entry); + dirty =3D is_migration_entry_dirty(entry); soft_dirty =3D pmd_swp_soft_dirty(old_pmd); uffd_wp =3D pmd_swp_uffd_wp(old_pmd); } else { @@ -2171,6 +2172,10 @@ static void __split_huge_pmd_locked(struct vm_area_s= truct *vma, pmd_t *pmd, else swp_entry =3D make_readable_migration_entry( page_to_pfn(page + i)); + if (young) + swp_entry =3D make_migration_entry_young(swp_entry); + if (dirty) + swp_entry =3D make_migration_entry_dirty(swp_entry); entry =3D swp_entry_to_pte(swp_entry); if (soft_dirty) entry =3D pte_swp_mksoft_dirty(entry); @@ -3180,6 +3185,10 @@ int set_pmd_migration_entry(struct page_vma_mapped_w= alk *pvmw, entry =3D make_readable_exclusive_migration_entry(page_to_pfn(page)); else entry =3D make_readable_migration_entry(page_to_pfn(page)); + if (pmd_young(pmdval)) + entry =3D make_migration_entry_young(entry); + if (pmd_dirty(pmdval)) + entry =3D make_migration_entry_dirty(entry); pmdswp =3D swp_entry_to_pmd(entry); if (pmd_soft_dirty(pmdval)) pmdswp =3D pmd_swp_mksoft_dirty(pmdswp); @@ -3205,13 +3214,18 @@ void remove_migration_pmd(struct page_vma_mapped_wa= lk *pvmw, struct page *new) =20 entry =3D pmd_to_swp_entry(*pvmw->pmd); get_page(new); - pmde =3D pmd_mkold(mk_huge_pmd(new, READ_ONCE(vma->vm_page_prot))); + pmde =3D mk_huge_pmd(new, READ_ONCE(vma->vm_page_prot)); if (pmd_swp_soft_dirty(*pvmw->pmd)) pmde =3D pmd_mksoft_dirty(pmde); if (is_writable_migration_entry(entry)) pmde =3D maybe_pmd_mkwrite(pmde, vma); if (pmd_swp_uffd_wp(*pvmw->pmd)) pmde =3D pmd_wrprotect(pmd_mkuffd_wp(pmde)); + if (!is_migration_entry_young(entry)) + pmde =3D pmd_mkold(pmde); + /* NOTE: this may contain setting soft-dirty on some archs */ + if (PageDirty(new) && is_migration_entry_dirty(entry)) + pmde =3D pmd_mkdirty(pmde); =20 if (PageAnon(new)) { rmap_t rmap_flags =3D RMAP_COMPOUND; diff --git a/mm/migrate.c b/mm/migrate.c index 6a1597c92261..0433a71d2bee 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -198,7 +198,7 @@ static bool remove_migration_pte(struct folio *folio, #endif =20 folio_get(folio); - pte =3D pte_mkold(mk_pte(new, READ_ONCE(vma->vm_page_prot))); + pte =3D mk_pte(new, READ_ONCE(vma->vm_page_prot)); if (pte_swp_soft_dirty(*pvmw.pte)) pte =3D pte_mksoft_dirty(pte); =20 @@ -206,6 +206,10 @@ static bool remove_migration_pte(struct folio *folio, * Recheck VMA as permissions can change since migration started */ entry =3D pte_to_swp_entry(*pvmw.pte); + if (!is_migration_entry_young(entry)) + pte =3D pte_mkold(pte); + if (folio_test_dirty(folio) && is_migration_entry_dirty(entry)) + pte =3D pte_mkdirty(pte); if (is_writable_migration_entry(entry)) pte =3D maybe_mkwrite(pte, vma); else if (pte_swp_uffd_wp(*pvmw.pte)) diff --git a/mm/migrate_device.c b/mm/migrate_device.c index 27fb37d65476..e450b318b01b 100644 --- a/mm/migrate_device.c +++ b/mm/migrate_device.c @@ -221,6 +221,12 @@ static int migrate_vma_collect_pmd(pmd_t *pmdp, else entry =3D make_readable_migration_entry( page_to_pfn(page)); + if (pte_present(pte)) { + if (pte_young(pte)) + entry =3D make_migration_entry_young(entry); + if (pte_dirty(pte)) + entry =3D make_migration_entry_dirty(entry); + } swp_pte =3D swp_entry_to_pte(entry); if (pte_present(pte)) { if (pte_soft_dirty(pte)) diff --git a/mm/rmap.c b/mm/rmap.c index af775855e58f..28aef434ea41 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -2065,7 +2065,10 @@ static bool try_to_migrate_one(struct folio *folio, = struct vm_area_struct *vma, else entry =3D make_readable_migration_entry( page_to_pfn(subpage)); - + if (pte_young(pteval)) + entry =3D make_migration_entry_young(entry); + if (pte_dirty(pteval)) + entry =3D make_migration_entry_dirty(entry); swp_pte =3D swp_entry_to_pte(entry); if (pte_soft_dirty(pteval)) swp_pte =3D pte_swp_mksoft_dirty(swp_pte); --=20 2.32.0 From nobody Sat Apr 11 10:47:11 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0770BC3F6B0 for ; Thu, 11 Aug 2022 16:42:19 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238939AbiHKQmR (ORCPT ); Thu, 11 Aug 2022 12:42:17 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35654 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S239280AbiHKQlN (ORCPT ); Thu, 11 Aug 2022 12:41:13 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTP id A4E7CA61D2 for ; Thu, 11 Aug 2022 09:13:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1660234429; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=/Huxd7IPCPnmA3ZqjXCerOPqYjMrNourUX2B7v06rhc=; b=XJGWxRejrCYVzyK6dIdGSsJ4OuX4L4sncbi2UZ+wij7Gh1D7F2QbXHHwDm0c2yy+nMACnZ /gKXI/q5Ad08mY5oC7Z/jMuWwM6y/3MianCP5hKisHPsgDLAFW/SRhKdqGOqD9MlZbTeFr wE0r1+4YXoyk03NNEPaeah2rSitJjkY= Received: from mail-il1-f197.google.com (mail-il1-f197.google.com [209.85.166.197]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_128_GCM_SHA256) id us-mta-446-HN37TWv9OBi_S03WOuowgQ-1; Thu, 11 Aug 2022 12:13:45 -0400 X-MC-Unique: HN37TWv9OBi_S03WOuowgQ-1 Received: by mail-il1-f197.google.com with SMTP id i12-20020a056e021d0c00b002df2d676974so12730071ila.5 for ; Thu, 11 Aug 2022 09:13:45 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc; bh=/Huxd7IPCPnmA3ZqjXCerOPqYjMrNourUX2B7v06rhc=; b=aP3l1Xfj8TC0RuOCq3rvR1SjnTG2xjsegvremxzdLgU6CI/NFslkNwcg3/bIyXdyXA VkAJIufIWTUtY/vcp4TSctMIi+FfJOK4hSmnBjU2fdJ0RhfpN/HKym+gtuAskEu7xRi8 Ln6E8Nl8VP9SZs43eB4K9hfj7+jaip/ydzY+ZEWeGTmgOrb12Z3c+x/nofT78bqntwU2 pE5kroog6bWUEdI3SMGDTnZhMaxptBSLW8WhXB8H/sQOJGZPP7VI/ccuOYKuonlwF7Er zCLi2dBbQ88IqAtzX3hhkC0j/TRpyYeGgP3Io5uuMAi5Asb/LBJgq++nmLG6fUK2IHZ5 elbg== X-Gm-Message-State: ACgBeo1Qh4c2V8ruVCuwlay06YiJXvFAfIOJpdjewjsYsqFOXzXvq65v rr5LhrhrQK0NmVa+Od7C3moC8JYs2mDjxNyIU52lJ6630xh1PFXxScU+gfzfkRK6+TONUfL4le7 RrSYqtzgu8YLN632io3FEIhhP X-Received: by 2002:a05:6638:1301:b0:342:c20d:3b15 with SMTP id r1-20020a056638130100b00342c20d3b15mr21422jad.59.1660234424221; Thu, 11 Aug 2022 09:13:44 -0700 (PDT) X-Google-Smtp-Source: AA6agR4T6gWtzxZxHWUIVbSUYS7vdfLC8nfHajGAKsVlnVfjs8zkeYJ9u3kFIxUuxfWF4Czl1LgvQg== X-Received: by 2002:a05:6638:1301:b0:342:c20d:3b15 with SMTP id r1-20020a056638130100b00342c20d3b15mr21411jad.59.1660234424004; Thu, 11 Aug 2022 09:13:44 -0700 (PDT) Received: from localhost.localdomain (bras-base-aurron9127w-grc-35-70-27-3-10.dsl.bell.ca. [70.27.3.10]) by smtp.gmail.com with ESMTPSA id t1-20020a92ca81000000b002dd1c3c5c46sm3415429ilo.73.2022.08.11.09.13.42 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Thu, 11 Aug 2022 09:13:43 -0700 (PDT) From: Peter Xu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Hugh Dickins , "Kirill A . Shutemov" , Alistair Popple , peterx@redhat.com, Andrea Arcangeli , Minchan Kim , Andrew Morton , David Hildenbrand , Andi Kleen , Nadav Amit , Huang Ying , Vlastimil Babka Subject: [PATCH v4 6/7] mm/swap: Cache maximum swapfile size when init swap Date: Thu, 11 Aug 2022 12:13:30 -0400 Message-Id: <20220811161331.37055-7-peterx@redhat.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20220811161331.37055-1-peterx@redhat.com> References: <20220811161331.37055-1-peterx@redhat.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" We used to have swapfile_maximum_size() fetching a maximum value of swapfile size per-arch. As the caller of max_swapfile_size() grows, this patch introduce a variable "swapfile_maximum_size" and cache the value of old max_swapfile_size(), so that we don't need to calculate the value every time. Caching the value in swapfile_init() is safe because when reaching the phase we should have initialized all the relevant information. Here the major arch to take care of is x86, which defines the max swapfile size based on L1TF mitigation. Here both X86_BUG_L1TF or l1tf_mitigation should have been setup properly when reaching swapfile_init(). As a reference, the code path looks like this for x86: - start_kernel - setup_arch - early_cpu_init - early_identify_cpu --> setup X86_BUG_L1TF - parse_early_param - l1tf_cmdline --> set l1tf_mitigation - check_bugs - l1tf_select_mitigation --> set l1tf_mitigation - arch_call_rest_init - rest_init - kernel_init - kernel_init_freeable - do_basic_setup - do_initcalls --> calls swapfile_init() (initcall level 4) The swapfile size only depends on swp pte format on non-x86 archs, so caching it is safe too. Since at it, rename max_swapfile_size() to arch_max_swapfile_size() because arch can define its own function, so it's more straightforward to have "arch_" as its prefix. At the meantime, export swapfile_maximum_size to replace the old usages of max_swapfile_size(). Signed-off-by: Peter Xu Reviewed-by: "Huang, Ying" Tested-by: Raghavendra K T --- arch/x86/mm/init.c | 2 +- include/linux/swapfile.h | 3 ++- include/linux/swapops.h | 2 +- mm/swapfile.c | 7 +++++-- 4 files changed, 9 insertions(+), 5 deletions(-) diff --git a/arch/x86/mm/init.c b/arch/x86/mm/init.c index 82a042c03824..9121bc1b9453 100644 --- a/arch/x86/mm/init.c +++ b/arch/x86/mm/init.c @@ -1054,7 +1054,7 @@ void update_cache_mode_entry(unsigned entry, enum pag= e_cache_mode cache) } =20 #ifdef CONFIG_SWAP -unsigned long max_swapfile_size(void) +unsigned long arch_max_swapfile_size(void) { unsigned long pages; =20 diff --git a/include/linux/swapfile.h b/include/linux/swapfile.h index 54078542134c..165e0bd04862 100644 --- a/include/linux/swapfile.h +++ b/include/linux/swapfile.h @@ -8,6 +8,7 @@ */ extern struct swap_info_struct *swap_info[]; extern unsigned long generic_max_swapfile_size(void); -extern unsigned long max_swapfile_size(void); +/* Maximum swapfile size supported for the arch (not inclusive). */ +extern unsigned long swapfile_maximum_size; =20 #endif /* _LINUX_SWAPFILE_H */ diff --git a/include/linux/swapops.h b/include/linux/swapops.h index 36e462e116af..f25b566643f1 100644 --- a/include/linux/swapops.h +++ b/include/linux/swapops.h @@ -307,7 +307,7 @@ static inline bool migration_entry_supports_ad(void) * the offset large enough to cover all of them (PFN, A & D bits). */ #ifdef CONFIG_SWAP - return max_swapfile_size() >=3D (1UL << SWP_MIG_TOTAL_BITS); + return swapfile_maximum_size >=3D (1UL << SWP_MIG_TOTAL_BITS); #else /* CONFIG_SWAP */ return false; #endif /* CONFIG_SWAP */ diff --git a/mm/swapfile.c b/mm/swapfile.c index 1fdccd2f1422..3cc64399df44 100644 --- a/mm/swapfile.c +++ b/mm/swapfile.c @@ -63,6 +63,7 @@ EXPORT_SYMBOL_GPL(nr_swap_pages); /* protected with swap_lock. reading in vm_swap_full() doesn't need lock */ long total_swap_pages; static int least_priority =3D -1; +unsigned long swapfile_maximum_size; =20 static const char Bad_file[] =3D "Bad swap file entry "; static const char Unused_file[] =3D "Unused swap file entry "; @@ -2816,7 +2817,7 @@ unsigned long generic_max_swapfile_size(void) } =20 /* Can be overridden by an architecture for additional checks. */ -__weak unsigned long max_swapfile_size(void) +__weak unsigned long arch_max_swapfile_size(void) { return generic_max_swapfile_size(); } @@ -2856,7 +2857,7 @@ static unsigned long read_swap_header(struct swap_inf= o_struct *p, p->cluster_next =3D 1; p->cluster_nr =3D 0; =20 - maxpages =3D max_swapfile_size(); + maxpages =3D swapfile_maximum_size; last_page =3D swap_header->info.last_page; if (!last_page) { pr_warn("Empty swap-file\n"); @@ -3677,6 +3678,8 @@ static int __init swapfile_init(void) for_each_node(nid) plist_head_init(&swap_avail_heads[nid]); =20 + swapfile_maximum_size =3D arch_max_swapfile_size(); + return 0; } subsys_initcall(swapfile_init); --=20 2.32.0 From nobody Sat Apr 11 10:47:11 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 65C62C25B06 for ; Thu, 11 Aug 2022 16:42:15 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238867AbiHKQmM (ORCPT ); Thu, 11 Aug 2022 12:42:12 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:52600 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S239285AbiHKQlO (ORCPT ); Thu, 11 Aug 2022 12:41:14 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTP id C0F8EA61DC for ; Thu, 11 Aug 2022 09:13:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1660234429; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Kvn+70I7yPIly7NIcpI+E4BYwfh3vfPaXmRAy92AF5w=; b=hVTN7i0QMKFBh392bac4jgfDb9v6Stf77skSImDft/H2S9/YrAmTxie6y+yysl3hue34rX D7vxaR3YmB3QBPB8v7/q/gafzHMI/KYJYff/osB3qs64VorXQWGuLXU0Fvi+AGgk1cAgjk aDWQpi0nGivaYsrkGwxgC+HCzwkYqk0= Received: from mail-il1-f200.google.com (mail-il1-f200.google.com [209.85.166.200]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_128_GCM_SHA256) id us-mta-660-GKQMlPldOKq3pJ_IvPSDHA-1; Thu, 11 Aug 2022 12:13:48 -0400 X-MC-Unique: GKQMlPldOKq3pJ_IvPSDHA-1 Received: by mail-il1-f200.google.com with SMTP id x7-20020a056e021ca700b002ded2e6331aso13023929ill.20 for ; Thu, 11 Aug 2022 09:13:46 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc; bh=Kvn+70I7yPIly7NIcpI+E4BYwfh3vfPaXmRAy92AF5w=; b=cczcbIcih1Oc2hhZ5G7QkoPklNPPYzk2khBqf5fPcWY14Q7rNZiRGl7t2/cLfN92Hl YKBmCIw9fFs4L8LlLsI9fw/8auejIdOkwrENCmJbzcVRduhERuIj2SjTCX08Fqkztisp 4GU9NzMSbIMJStE+FsqZ8N9+EwUfQBK4N1JyHrBnGdgSU3oUDlvBLo6J1LfUCpmNN9TG dikcuxH785qSxADkt5BZMybRoq7/6TDcVkUMe1nypavtTXVgx5rA9VFqzPNuCUXBHx6D 1auPEUewD/N3f6JAMR41Z6aCeHtkYC0yEjOAtBH9vFTETGjcBk50NV8uFS/rRVP5deMP cuoA== X-Gm-Message-State: ACgBeo384bJTAFqZwuy7CPJwJKJv5CEtG7NzxUeBvXDctLYSlqfMk9gq HdwfiHbEKC4N7u2uaHHYoP27Oc8FYRiZURjLXyF2lKwDMLPoEzFjdT8M+/WhudRUQMv2hDWOzxC V6yxZmP6hTIksNnDoNLo+qTG8 X-Received: by 2002:a92:ce50:0:b0:2dd:dc8e:1f36 with SMTP id a16-20020a92ce50000000b002dddc8e1f36mr16218642ilr.34.1660234425789; Thu, 11 Aug 2022 09:13:45 -0700 (PDT) X-Google-Smtp-Source: AA6agR4ZzxcH25ORg0vfYxNfi+ta3V6GDOQvV28uQeT5mwGBGdlqyFRzS44blQE276L9QlyIzt5Fvg== X-Received: by 2002:a92:ce50:0:b0:2dd:dc8e:1f36 with SMTP id a16-20020a92ce50000000b002dddc8e1f36mr16218626ilr.34.1660234425555; Thu, 11 Aug 2022 09:13:45 -0700 (PDT) Received: from localhost.localdomain (bras-base-aurron9127w-grc-35-70-27-3-10.dsl.bell.ca. [70.27.3.10]) by smtp.gmail.com with ESMTPSA id t1-20020a92ca81000000b002dd1c3c5c46sm3415429ilo.73.2022.08.11.09.13.44 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Thu, 11 Aug 2022 09:13:45 -0700 (PDT) From: Peter Xu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Hugh Dickins , "Kirill A . Shutemov" , Alistair Popple , peterx@redhat.com, Andrea Arcangeli , Minchan Kim , Andrew Morton , David Hildenbrand , Andi Kleen , Nadav Amit , Huang Ying , Vlastimil Babka Subject: [PATCH v4 7/7] mm/swap: Cache swap migration A/D bits support Date: Thu, 11 Aug 2022 12:13:31 -0400 Message-Id: <20220811161331.37055-8-peterx@redhat.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20220811161331.37055-1-peterx@redhat.com> References: <20220811161331.37055-1-peterx@redhat.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Introduce a variable swap_migration_ad_supported to cache whether the arch supports swap migration A/D bits. Here one thing to mention is that SWP_MIG_TOTAL_BITS will internally reference the other macro MAX_PHYSMEM_BITS, which is a function call on x86 (constant on all the rest of archs). It's safe to reference it in swapfile_init() because when reaching here we're already during initcalls level 4 so we must have initialized 5-level pgtable for x86_64 (right after early_identify_cpu() finishes). - start_kernel - setup_arch - early_cpu_init - get_cpu_cap --> fetch from CPUID (including X86_FEATURE_LA57) - early_identify_cpu --> clear X86_FEATURE_LA57 (if early lvl5 not en= abled (USE_EARLY_PGTABLE_L5)) - arch_call_rest_init - rest_init - kernel_init - kernel_init_freeable - do_basic_setup - do_initcalls --> calls swapfile_init() (initcall level 4) This should slightly speed up the migration swap entry handlings. Signed-off-by: Peter Xu Tested-by: Raghavendra K T --- include/linux/swapfile.h | 2 ++ include/linux/swapops.h | 7 +------ mm/swapfile.c | 8 ++++++++ 3 files changed, 11 insertions(+), 6 deletions(-) diff --git a/include/linux/swapfile.h b/include/linux/swapfile.h index 165e0bd04862..2fbcc9afd814 100644 --- a/include/linux/swapfile.h +++ b/include/linux/swapfile.h @@ -10,5 +10,7 @@ extern struct swap_info_struct *swap_info[]; extern unsigned long generic_max_swapfile_size(void); /* Maximum swapfile size supported for the arch (not inclusive). */ extern unsigned long swapfile_maximum_size; +/* Whether swap migration entry supports storing A/D bits for the arch */ +extern bool swap_migration_ad_supported; =20 #endif /* _LINUX_SWAPFILE_H */ diff --git a/include/linux/swapops.h b/include/linux/swapops.h index f25b566643f1..dbf9df854124 100644 --- a/include/linux/swapops.h +++ b/include/linux/swapops.h @@ -301,13 +301,8 @@ static inline swp_entry_t make_writable_migration_entr= y(pgoff_t offset) */ static inline bool migration_entry_supports_ad(void) { - /* - * max_swapfile_size() returns the max supported swp-offset plus 1. - * We can support the migration A/D bits iff the pfn swap entry has - * the offset large enough to cover all of them (PFN, A & D bits). - */ #ifdef CONFIG_SWAP - return swapfile_maximum_size >=3D (1UL << SWP_MIG_TOTAL_BITS); + return swap_migration_ad_supported; #else /* CONFIG_SWAP */ return false; #endif /* CONFIG_SWAP */ diff --git a/mm/swapfile.c b/mm/swapfile.c index 3cc64399df44..263b19e693cf 100644 --- a/mm/swapfile.c +++ b/mm/swapfile.c @@ -64,6 +64,9 @@ EXPORT_SYMBOL_GPL(nr_swap_pages); long total_swap_pages; static int least_priority =3D -1; unsigned long swapfile_maximum_size; +#ifdef CONFIG_MIGRATION +bool swap_migration_ad_supported; +#endif /* CONFIG_MIGRATION */ =20 static const char Bad_file[] =3D "Bad swap file entry "; static const char Unused_file[] =3D "Unused swap file entry "; @@ -3680,6 +3683,11 @@ static int __init swapfile_init(void) =20 swapfile_maximum_size =3D arch_max_swapfile_size(); =20 +#ifdef CONFIG_MIGRATION + if (swapfile_maximum_size >=3D (1UL << SWP_MIG_TOTAL_BITS)) + swap_migration_ad_supported =3D true; +#endif /* CONFIG_MIGRATION */ + return 0; } subsys_initcall(swapfile_init); --=20 2.32.0