From nobody Mon Jun 8 19:49:21 2026 Received: from va-1-113.ptr.blmpb.com (va-1-113.ptr.blmpb.com [209.127.230.113]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0150633AD99 for ; Wed, 3 Jun 2026 08:03:54 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.127.230.113 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780473836; cv=none; b=fXYAbNad+e833uAb3L1o/VY4AogchogfdY29tgWBw8Nqcyc+ESn6BX7+MBBhcuUUWqa8Fm0bqTTlgDNm/cQSAjuMHFlf8PyrSM6JgF3ZRlILroDNWBY6L3ZjQQukqzxrb5A4a5VYmjBMmqu7Vk7vX2OCoBlaXtQY8cELhv3rLgc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780473836; c=relaxed/simple; bh=CKJrgkBXuJL76tNULqACYxiYPuJim8xWXoUjKVuwOWo=; h=Cc:Message-Id:In-Reply-To:From:Date:References:Mime-Version: Content-Type:To:Subject; b=Uc+ValeMGB0KQSNaF8/EMJZPR5PXNqShEt3sFxTV0I1zv4sNwpRqbssxGIKy/T5FHYB+xIXjZlaRARJkLKNnVVx5JoS33mLblEcSIaCEYLyawq1xBcXDok77V9dxV8hriop+vem4Y/8/Jaw++kSo0YCTKHelStAT+/Fu65g0QsE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=DWwfEf7Q; arc=none smtp.client-ip=209.127.230.113 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="DWwfEf7Q" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; s=2212171451; d=bytedance.com; t=1780473824; h=from:subject: mime-version:from:date:message-id:subject:to:cc:reply-to:content-type: mime-version:in-reply-to:message-id; bh=g/rmtPD1MKrWC7BjiewTdsAoLxWW7dzbcFZmhvRKNSg=; b=DWwfEf7QMoGjXqoTXelNAgI/iLEGIR9X5nbC2Epz1dDNoWtgSK6JJc1/kL46HxLEQ49lMG YjNAqK7lNtmte7uWYLGELUWyRQIJyZwPXvs7/sGnJUvZMY+7jqYT7ZwjjSWff7XSySlxaw XITt71tgA389mo2kRmrCgHveYE4I7P2Qfr873wMhcooitIY/Rik2EbpPUIU3liIvgdwK1C ZKJu9+Gj3a9+7JYWKbhq3+xNN6p6FapbLmb56awEeo7TU7iaBOCBCjMOIEaI3TXthHHp/+ G6agip45lUq44QDNRsYSkyAzr5kjHzzw9fdgoR37WIXZmtIaANzqswAI4ztXJA== Cc: , , , , , Message-Id: <20260603080152.64728-2-lizhe.67@bytedance.com> X-Original-From: Li Zhe X-Mailer: git-send-email 2.45.2 In-Reply-To: <20260603080152.64728-1-lizhe.67@bytedance.com> From: "Li Zhe" Date: Wed, 3 Jun 2026 16:01:45 +0800 References: <20260603080152.64728-1-lizhe.67@bytedance.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 Content-Transfer-Encoding: quoted-printable To: , , , , , , , , , Subject: [PATCH v4 1/8] mm: fix stale ZONE_DEVICE refcount comment X-Lms-Return-Path: Content-Type: text/plain; charset="utf-8" The comment in __init_zone_device_page() still uses the old MEMORY_TYPE_* names and implies that FS_DAX pages regain a refcount of 1 in the free path. That no longer matches the code. Update the comment to describe the current policy correctly: MEMORY_DEVICE_GENERIC pages regain a refcount of 1 in the free path, while the remaining ZONE_DEVICE types start from 0 here and raise the count again when the allocator or driver hands the page out. No functional change intended. Signed-off-by: Li Zhe --- mm/mm_init.c | 10 +++------- 1 file changed, 3 insertions(+), 7 deletions(-) diff --git a/mm/mm_init.c b/mm/mm_init.c index f9f8e1af921c..35de3b6a186d 100644 --- a/mm/mm_init.c +++ b/mm/mm_init.c @@ -1027,13 +1027,9 @@ static void __ref __init_zone_device_page(struct pag= e *page, unsigned long pfn, } =20 /* - * ZONE_DEVICE pages other than MEMORY_TYPE_GENERIC are released - * directly to the driver page allocator which will set the page count - * to 1 when allocating the page. - * - * MEMORY_TYPE_GENERIC and MEMORY_TYPE_FS_DAX pages automatically have - * their refcount reset to one whenever they are freed (ie. after - * their refcount drops to 0). + * MEMORY_DEVICE_GENERIC pages regain a refcount of 1 in the free + * path. The remaining ZONE_DEVICE types start from 0 here and raise + * the count again when the allocator or driver hands the page out. */ switch (pgmap->type) { case MEMORY_DEVICE_FS_DAX: --=20 2.20.1 From nobody Mon Jun 8 19:49:21 2026 Received: from va-1-114.ptr.blmpb.com (va-1-114.ptr.blmpb.com [209.127.230.114]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C4120346E4A for ; Wed, 3 Jun 2026 08:04:30 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.127.230.114 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780473872; cv=none; b=dK65x3VbQ04htwcIsG+UWPgyxC9vqfqSS9hDuYBaAFSfvJLath9Ut6fMnMNjidpaBxW3r5ZYnGKYmzma8Wk7kyI9JvpjHgdA1er0bonLvyVvXElxnRHBYtl8ILFAnVi6H4Ps9I9Lvo/1BwX4xIolrxWMfdm3KyIfB8tXRvPRw0k= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780473872; c=relaxed/simple; bh=zEwM4ExvxDP39KFMRfs9oCMwfWnr8A/tcgD3Hi6W860=; h=Subject:References:Message-Id:Cc:From:Mime-Version:To:Date: In-Reply-To:Content-Type; b=G0UA+KRz1d+h2wlqn70U57SA3TksqxynuSF6qktM+dqkGhx9VjjIFyMlB1IMJwswj/GQsEz07V1JrB95slUotick6lIEqY7LpmmCUvJsX65Zhh/5dXa3lWEqomJJUSCnw+0JjvZar6lHxI0KI1BZTf/GHzk6upMOeItw1rcOXwI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=nXc4Z1Jj; arc=none smtp.client-ip=209.127.230.114 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="nXc4Z1Jj" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; s=2212171451; d=bytedance.com; t=1780473860; h=from:subject: mime-version:from:date:message-id:subject:to:cc:reply-to:content-type: mime-version:in-reply-to:message-id; bh=PAoGwOogI0538Jjz3Jr5mPaMM02ePyByPYJiX1kFuRI=; b=nXc4Z1Jju0zHBJOvEeYJbZHEg9IWLUkiIUT9Y5bIJxEQsqz/L2DmvHPWjZckPV8v+RMxKa OgET/x03w4pmPnmfN8PEr4I6+BDxFDg1o4b1j/WnvD1K8gtEMM5snPrJHDP/5Z8fx9/iyo 7Yp0fMCsBr9c9Zb7ftKJ/sJ+cIB5f/dUHLpmO/F1lIpwXL0rC9UvcU/vqdupPDFbJ+gZa1 svfmi2VZZdjyoM+OzAjHWHjMgNkjq1gpl9/vIaQ9bGCHyye2h3vUh4UBalVqJLAikEIfs7 nXlzMu+uxtMELsyyP/8iAMJ/gI9TGdtQJJzXFI+nEmYs9qjsSykorhFplaSpwA== Subject: [PATCH v4 2/8] mm: factor zone-device page init helpers out of __init_zone_device_page References: <20260603080152.64728-1-lizhe.67@bytedance.com> Message-Id: <20260603080152.64728-3-lizhe.67@bytedance.com> Cc: , , , , , From: "Li Zhe" Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Original-From: Li Zhe To: , , , , , , , , , Date: Wed, 3 Jun 2026 16:01:46 +0800 X-Lms-Return-Path: In-Reply-To: <20260603080152.64728-1-lizhe.67@bytedance.com> X-Mailer: git-send-email 2.45.2 Content-Type: text/plain; charset="utf-8" memmap_init_zone_device() currently mixes refcount policy, core ZONE_DEVICE page setup, and pageblock metadata handling in a single helper. Factor the refcount-reset predicate into pagemap_resets_refcount(), move the common page initialization into __zone_device_page_init(), split pageblock handling into zone_device_page_init_pageblock(), and wrap the existing slow path in zone_device_page_init_slow(). This keeps the slow-path behaviour unchanged and gives later patches reusable helper boundaries. No functional change intended. Signed-off-by: Li Zhe Reviewed-by: Mike Rapoport (Microsoft) --- mm/mm_init.c | 62 ++++++++++++++++++++++++++++++++++++---------------- 1 file changed, 43 insertions(+), 19 deletions(-) diff --git a/mm/mm_init.c b/mm/mm_init.c index 35de3b6a186d..2e5899c5cf35 100644 --- a/mm/mm_init.c +++ b/mm/mm_init.c @@ -987,11 +987,38 @@ static void __init memmap_init(void) } =20 #ifdef CONFIG_ZONE_DEVICE -static void __ref __init_zone_device_page(struct page *page, unsigned long= pfn, +/* + * Return true when the free path for this pagemap type restores the page + * refcount to 1, so memmap_init_zone_device() can keep the count set by + * __init_single_page(). Otherwise initialize the refcount to 0 and leave + * it to the allocator or pgmap callbacks to raise it when the page is + * handed out again. + */ +static inline bool pagemap_resets_refcount(const struct dev_pagemap *pgmap) +{ + /* + * MEMORY_DEVICE_GENERIC pages regain a refcount of 1 in the free + * path. The remaining ZONE_DEVICE types start from 0 here and raise + * the count again when the allocator or driver hands the page out. + */ + switch (pgmap->type) { + case MEMORY_DEVICE_FS_DAX: + case MEMORY_DEVICE_PRIVATE: + case MEMORY_DEVICE_COHERENT: + case MEMORY_DEVICE_PCI_P2PDMA: + return false; + case MEMORY_DEVICE_GENERIC: + return true; + default: + WARN_ONCE(1, "Unknown memory type!"); + return true; + } +} + +static void __ref __zone_device_page_init(struct page *page, unsigned long= pfn, unsigned long zone_idx, int nid, struct dev_pagemap *pgmap) { - __init_single_page(page, pfn, zone_idx, nid); =20 /* @@ -1010,7 +1037,11 @@ static void __ref __init_zone_device_page(struct pag= e *page, unsigned long pfn, */ page_folio(page)->pgmap =3D pgmap; page->zone_device_data =3D NULL; +} =20 +static void __ref zone_device_page_init_pageblock(struct page *page, + unsigned long pfn) +{ /* * Mark the block movable so that blocks are reserved for * movable at startup. This will force kernel allocations @@ -1025,23 +1056,16 @@ static void __ref __init_zone_device_page(struct pa= ge *page, unsigned long pfn, init_pageblock_migratetype(page, MIGRATE_MOVABLE, false); cond_resched(); } +} =20 - /* - * MEMORY_DEVICE_GENERIC pages regain a refcount of 1 in the free - * path. The remaining ZONE_DEVICE types start from 0 here and raise - * the count again when the allocator or driver hands the page out. - */ - switch (pgmap->type) { - case MEMORY_DEVICE_FS_DAX: - case MEMORY_DEVICE_PRIVATE: - case MEMORY_DEVICE_COHERENT: - case MEMORY_DEVICE_PCI_P2PDMA: +static void __ref zone_device_page_init_slow(struct page *page, + unsigned long pfn, unsigned long zone_idx, int nid, + struct dev_pagemap *pgmap) +{ + __zone_device_page_init(page, pfn, zone_idx, nid, pgmap); + if (!pagemap_resets_refcount(pgmap)) set_page_count(page, 0); - break; - - case MEMORY_DEVICE_GENERIC: - break; - } + zone_device_page_init_pageblock(page, pfn); } =20 /* @@ -1080,7 +1104,7 @@ static void __ref memmap_init_compound(struct page *h= ead, for (pfn =3D head_pfn + 1; pfn < end_pfn; pfn++) { struct page *page =3D pfn_to_page(pfn); =20 - __init_zone_device_page(page, pfn, zone_idx, nid, pgmap); + zone_device_page_init_slow(page, pfn, zone_idx, nid, pgmap); prep_compound_tail(page, head, order); set_page_count(page, 0); } @@ -1116,7 +1140,7 @@ void __ref memmap_init_zone_device(struct zone *zone, for (pfn =3D start_pfn; pfn < end_pfn; pfn +=3D pfns_per_compound) { struct page *page =3D pfn_to_page(pfn); =20 - __init_zone_device_page(page, pfn, zone_idx, nid, pgmap); + zone_device_page_init_slow(page, pfn, zone_idx, nid, pgmap); =20 if (pfns_per_compound =3D=3D 1) continue; --=20 2.20.1 From nobody Mon Jun 8 19:49:21 2026 Received: from va-1-115.ptr.blmpb.com (va-1-115.ptr.blmpb.com [209.127.230.115]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5E118407CE2 for ; Wed, 3 Jun 2026 08:04:59 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.127.230.115 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780473900; cv=none; b=eGTv3nmxgd86EIPAILuT9KqDgViO+BRatwumKQWbf6vQ3BTcIYxnNZkj4LpYqk2h0QQ0n/aRfq/qgR2lElyLYJHENv6yrE/e/AIWrm32CYyIII/V7gzJxyXatvZgkPRYShInVsBHYFwGp5K3hwMNPtuD6xlLJqtcbm/7KoxkUes= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780473900; c=relaxed/simple; bh=TV9B3FF8uyNzCN29bSzOpcQAbaH6iSVb+B/b0Zh9c0w=; h=To:In-Reply-To:Content-Type:Cc:From:Message-Id:Mime-Version: Subject:Date:References; b=ODgBSHzJQCytK7+2od/AzwhQTwrRXRNMtic8b9M7MGQkx/CfesYxZu8fDaS6C7p1Jb0sIX2/MLiuZA5GDSPe1oXe9f1p/Tse8ZZ6Jps+f6o8bMf9sLqvA5v0CopMINuA4HoeMZOXMR8E8yGSq6vFKUpQN/hHZo1GKOj6KjiXGYI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=fOClk2CL; arc=none smtp.client-ip=209.127.230.115 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="fOClk2CL" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; s=2212171451; d=bytedance.com; t=1780473894; h=from:subject: mime-version:from:date:message-id:subject:to:cc:reply-to:content-type: mime-version:in-reply-to:message-id; bh=Wk9MRIIObAkXzp6VZckmtUtqVfitvWEYaMcAPnuhTg0=; b=fOClk2CL1fCwzT3+vuaOHqsI7g5da5AScnzI84M2/HrGKXqxQrH4IPvNIHKS3TxVjyxM2Z VwcAqVELhm1mJS5Jrmlx2v2SgVi7W8wZU3PkB+nodYQ/qhJ3BAm9UHoTvdckGxPm2D6L9x egKnpB+fJeGVp48JQRIr4kUNcyMq8Go7tDmCSgORlhH8367V2nOEoSgZ/KNVOZ/2HRg/qt VfryU/jR3qHqofObntNIVieU04cbqrnmdgbKUL2HiCGvfpjyQXpwmXEYdrQhYMBJMG8zx1 rX+OEjJ7trpkY2JANC+C2RpU8o5q/J3iGZNIxu5udZ2lGtPjynn1E0LD1Bvmgg== To: , , , , , , , , , In-Reply-To: <20260603080152.64728-1-lizhe.67@bytedance.com> Content-Transfer-Encoding: quoted-printable X-Mailer: git-send-email 2.45.2 Cc: , , , , , From: "Li Zhe" Message-Id: <20260603080152.64728-4-lizhe.67@bytedance.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 Subject: [PATCH v4 3/8] mm: add a set_page_section_from_pfn() helper Date: Wed, 3 Jun 2026 16:01:47 +0800 References: <20260603080152.64728-1-lizhe.67@bytedance.com> X-Original-From: Li Zhe X-Lms-Return-Path: Content-Type: text/plain; charset="utf-8" Callers that want to update section bits from a PFN currently need to open-code: set_page_section(page, pfn_to_section_nr(pfn)); and guard that sequence with #ifdef SECTION_IN_PAGE_FLAGS. Add set_page_section_from_pfn() to wrap that update in one place. When section bits are stored in page flags, the helper derives the section number from the PFN and updates the page flags. Otherwise it degrades to a no-op. Convert set_page_links() to use the new helper so later ZONE_DEVICE fast-path patches can also update section bits without open-coding SECTION_IN_PAGE_FLAGS at each callsite. This keeps the PFN-to-section translation local to the configurations that actually store section bits in struct page flags, and avoids exposing that detail to generic callers. No functional change intended. Signed-off-by: Li Zhe Reviewed-by: Mike Rapoport (Microsoft) --- include/linux/mm.h | 19 ++++++++++++++++--- 1 file changed, 16 insertions(+), 3 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 06bbe9eba636..1b0de1eef4c9 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -2507,11 +2507,26 @@ static inline void set_page_section(struct page *pa= ge, unsigned long section) page->flags.f |=3D (section & SECTIONS_MASK) << SECTIONS_PGSHIFT; } =20 +static inline void set_page_section_from_pfn(struct page *page, + unsigned long pfn) +{ + set_page_section(page, pfn_to_section_nr(pfn)); +} + static inline unsigned long memdesc_section(memdesc_flags_t mdf) { return (mdf.f >> SECTIONS_PGSHIFT) & SECTIONS_MASK; } #else /* !SECTION_IN_PAGE_FLAGS */ +static inline void set_page_section(struct page *page, unsigned long secti= on) +{ +} + +static inline void set_page_section_from_pfn(struct page *page, + unsigned long pfn) +{ +} + static inline unsigned long memdesc_section(memdesc_flags_t mdf) { return 0; @@ -2734,9 +2749,7 @@ static inline void set_page_links(struct page *page, = enum zone_type zone, { set_page_zone(page, zone); set_page_node(page, node); -#ifdef SECTION_IN_PAGE_FLAGS - set_page_section(page, pfn_to_section_nr(pfn)); -#endif + set_page_section_from_pfn(page, pfn); } =20 /** --=20 2.20.1 From nobody Mon Jun 8 19:49:21 2026 Received: from va-1-115.ptr.blmpb.com (va-1-115.ptr.blmpb.com [209.127.230.115]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D8ADB407CCE for ; Wed, 3 Jun 2026 08:05:36 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.127.230.115 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780473938; cv=none; b=ZTmPbG7AjdSJTE41gRQ2bCkeb0tM/FWGu1Y0yXCZNGnqvT9EZ1oInwLNV6MofEPTmR9DWuWpdTFOxzTe9W82w4SmKIOaIafdvQk7YIlch0KgfSy26+EatXhfVNCbqM5hlOC/9a/2e19p9ewM5K4JnCa8rI0z6Idoe15EfNcUmMI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780473938; c=relaxed/simple; bh=/pYgzuRQ6iq6nYtxlWbo/GZKyUbyCI63h8deMoKd5M0=; h=To:Subject:Message-Id:In-Reply-To:References:Date:From: Content-Type:Cc:Mime-Version; b=UfgRTi0D9EEPzI0T3AxQwdkG7xa3ZdfqqVpRownAYpo6hQI1H8zd57HC7A3Q9uxpARNo2kztkxN+Q2+zpJalFCsXjk86c7P6II6pVLgKdn/E0+sqlBTCbPv5Kfa5MLFTnKYICufnEhPAU/3Ucp7vvqsNxBwJ02xP2lEAQbAVHrc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=ZLrF88yr; arc=none smtp.client-ip=209.127.230.115 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="ZLrF88yr" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; s=2212171451; d=bytedance.com; t=1780473932; h=from:subject: mime-version:from:date:message-id:subject:to:cc:reply-to:content-type: mime-version:in-reply-to:message-id; bh=yy6XV8hK8oauYu/xVX9D1lPDIbV+Br2ZbBHBJaftx+0=; b=ZLrF88yrw0Nzm2arTnuzExC5kIY/vk+nde66tD5n4K7wAdlp8Y4KcqGSZTG+Ron/yDXItV GoEPagz8RFHgpXEFcKw8l44nkDsEnRI0f0h3Qy7Ke5b+k+3awfWUhUV7zyvZ2c6fK24d4E CznItthwfvCx5Di0TfFWL2mt4vyATW7tjB0kN8DokSactI8q1U7dKDmBMqMgvBrRVdVTpL PxIL4JmeIUKa9WLnwZF1gVlV6NvJjWdBHs8jL2oxj+fcwEB1Y5zXI5P0QOgHGpIXgonH71 OvQWensr1Q/3a0Et0oPJFBj0FjKGp5noK19664J4PsS4BSO49ekAaIDU7LkH/w== Content-Transfer-Encoding: quoted-printable To: , , , , , , , , , Subject: [PATCH v4 4/8] mm: add a template-based fast path for zone-device page init Message-Id: <20260603080152.64728-5-lizhe.67@bytedance.com> In-Reply-To: <20260603080152.64728-1-lizhe.67@bytedance.com> References: <20260603080152.64728-1-lizhe.67@bytedance.com> X-Original-From: Li Zhe Date: Wed, 3 Jun 2026 16:01:48 +0800 From: "Li Zhe" X-Lms-Return-Path: X-Mailer: git-send-email 2.45.2 Cc: , , , , , Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 Content-Type: text/plain; charset="utf-8" memmap_init_zone_device() repeats nearly identical head-page initialization for each PFN. Prepare one reusable ZONE_DEVICE head-page template through the existing slow path, refresh the PFN-dependent fields in that template before each copy, and memcpy it into each destination page. The optimized path assigns _refcount through the copied template, so keep it disabled when the page_ref_set tracepoint is enabled. This patch accelerates head-page initialization. The pfns_per_compound =3D=3D 1 case gets the full benefit here, compound tails are handled in the next patch. Tested in a VM with a 100 GB fsdax namespace device configured with map=3Ddev on Intel Ice Lake server. This test exercises the nd_pmem rebind path (pfns_per_compound =3D=3D 1). Test procedure: Rebind the nd_pmem driver 30 times and collect the memmap initialization time from the pr_debug() output of memmap_init_zone_device(). Base(v7.1-rc6): First binding: 1466 ms Average of subsequent rebinds: 262.12 ms With this patch and its prerequisites applied: First binding: 1432 ms Average of subsequent rebinds: 246.30 ms This reduces the average rebind time from 262.12 ms to 246.30 ms, or about 6%. Signed-off-by: Li Zhe --- mm/mm_init.c | 76 ++++++++++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 74 insertions(+), 2 deletions(-) diff --git a/mm/mm_init.c b/mm/mm_init.c index 2e5899c5cf35..56fe71e966ae 100644 --- a/mm/mm_init.c +++ b/mm/mm_init.c @@ -1068,6 +1068,51 @@ static void __ref zone_device_page_init_slow(struct = page *page, zone_device_page_init_pageblock(page, pfn); } =20 +static inline bool zone_device_page_init_optimization_enabled(void) +{ + /* + * The template fast path copies a preinitialized struct page image. + * Skip it when the page_ref_set tracepoint is enabled. + */ + return !page_ref_tracepoint_active(page_ref_set); +} + +static inline void zone_device_template_page_init(struct page *template, + struct page *src) +{ + memcpy(template, src, sizeof(*template)); +} + +/* + * 'template' is a reusable page prototype rather than a strictly immutable + * object. Most ZONE_DEVICE fields stay constant across the pages covered = by + * the current template, but section bits and page->virtual may still depe= nd + * on the PFN. Refresh those PFN-dependent fields in the template before + * copying it into @page. + */ +static inline void zone_device_page_update_template(struct page *template, + unsigned long pfn) +{ + set_page_section_from_pfn(template, pfn); +#ifdef WANT_PAGE_VIRTUAL + if (!is_highmem_idx(ZONE_DEVICE)) + set_page_address(template, __va(pfn << PAGE_SHIFT)); +#endif +} + +static void zone_device_page_init_from_template(struct page *page, + unsigned long pfn, struct page *template) +{ + /* + * 'template' carries the invariant portion of a ZONE_DEVICE struct + * page. Update the PFN-dependent fields in place before copying it + * to the destination page. + */ + zone_device_page_update_template(template, pfn); + memcpy(page, template, sizeof(*page)); + zone_device_page_init_pageblock(page, pfn); +} + /* * With compound page geometry and when struct pages are stored in ram most * tail pages are reused. Consequently, the amount of unique struct pages = to @@ -1116,6 +1161,7 @@ void __ref memmap_init_zone_device(struct zone *zone, unsigned long nr_pages, struct dev_pagemap *pgmap) { + bool use_template =3D zone_device_page_init_optimization_enabled(); unsigned long pfn, end_pfn =3D start_pfn + nr_pages; struct pglist_data *pgdat =3D zone->zone_pgdat; struct vmem_altmap *altmap =3D pgmap_altmap(pgmap); @@ -1123,6 +1169,7 @@ void __ref memmap_init_zone_device(struct zone *zone, unsigned long zone_idx =3D zone_idx(zone); unsigned long start =3D jiffies; int nid =3D pgdat->node_id; + struct page template; =20 if (WARN_ON_ONCE(!pgmap || zone_idx !=3D ZONE_DEVICE)) return; @@ -1137,10 +1184,35 @@ void __ref memmap_init_zone_device(struct zone *zon= e, nr_pages =3D end_pfn - start_pfn; } =20 - for (pfn =3D start_pfn; pfn < end_pfn; pfn +=3D pfns_per_compound) { - struct page *page =3D pfn_to_page(pfn); + if (!nr_pages) + return; + + pfn =3D start_pfn; + /* + * Seed the reusable head-page template from the first real struct + * page, because the existing page-init and pageblock helpers expect + * a real memmap entry rather than a stack object. + */ + if (use_template) { + struct page *page =3D pfn_to_page(start_pfn); =20 zone_device_page_init_slow(page, pfn, zone_idx, nid, pgmap); + zone_device_template_page_init(&template, page); + if (pfns_per_compound !=3D 1) + memmap_init_compound(page, pfn, zone_idx, nid, pgmap, + compound_nr_pages(altmap, pgmap)); + pfn +=3D pfns_per_compound; + } + + for (; pfn < end_pfn; pfn +=3D pfns_per_compound) { + struct page *page =3D pfn_to_page(pfn); + + if (use_template) + zone_device_page_init_from_template(page, pfn, + &template); + else + zone_device_page_init_slow(page, pfn, zone_idx, + nid, pgmap); =20 if (pfns_per_compound =3D=3D 1) continue; --=20 2.20.1 From nobody Mon Jun 8 19:49:21 2026 Received: from va-1-115.ptr.blmpb.com (va-1-115.ptr.blmpb.com [209.127.230.115]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8FCC9397E86 for ; Wed, 3 Jun 2026 08:06:29 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.127.230.115 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780473991; cv=none; b=GH1Glo+TAkLPfLfyUv7NfdfnPb2jDcr1QCDd0JPR/zOXYmFkek4LcIAkvi2UBKpSOJUWI0tPt8cc3oIziMsuexEjsP9VDWuAzC9+Dcjt37Kfdzc/Isc97ATABF9dome9Ild0PhiSY9VlmBv3VG+M9vziv4Q9JJ0TaaEUa/KViUs= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780473991; c=relaxed/simple; bh=KlIrYySv42sjQ374Xuf4sWKDanycr+WDeKknoYsTVic=; h=Content-Type:From:References:In-Reply-To:To:Subject:Date:Cc: Message-Id:Mime-Version; b=WOBiHr5qRNUmrWvTEf27sssdScVisXx8NMdeQL6yRRFAqvral7t3+h+c8VcAkpKl0sNQhMSo9Vctg3w9pSKVy00vGrvhOF19XO4KeLvg04jkFat9Qs6EdlsYBGufaQVmCdOmQJYIzS+UigxeYerl6+I+ifeSzvwkAEks05okq3w= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=CkDYJ4X/; arc=none smtp.client-ip=209.127.230.115 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="CkDYJ4X/" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; s=2212171451; d=bytedance.com; t=1780473984; h=from:subject: mime-version:from:date:message-id:subject:to:cc:reply-to:content-type: mime-version:in-reply-to:message-id; bh=0+pxu4QONrexAlpcc34zjw3GPn7T+Jri0V38RE3OFI0=; b=CkDYJ4X/V8qxLlchNcKL8fTOAaZ4SJAT0AeDzNjKgbuv+HwOZ6ihzOAovB0owB1QkxfNZ9 tVTJ+gDHEMddKMXnFvr5PEdiSqviN+/Qwp92BzlD+geecV13gUF6GXs8PxomqT5Q6FmIuH 8lhCJBHPh8aiUP74KkseSaQNWo6YD9dSLRfcPyIA/utuGYqg289KZym8pAExSH7WA3Kpmg RNr28bCo97u9isEoHZKdEhQkuY5IZ0xppZ9IarAIspLg6VDJGPnSsj69S2A7zTmJ8mdkuH JDWLImM+rCaaHlxbRn44RN2opAML2ZrfR1DoqyebT6FMrVkZdFhNcXZo3owtlQ== From: "Li Zhe" References: <20260603080152.64728-1-lizhe.67@bytedance.com> Content-Transfer-Encoding: quoted-printable X-Mailer: git-send-email 2.45.2 X-Original-From: Li Zhe In-Reply-To: <20260603080152.64728-1-lizhe.67@bytedance.com> X-Lms-Return-Path: To: , , , , , , , , , Subject: [PATCH v4 5/8] mm: extend the template fast path to zone-device compound tails Date: Wed, 3 Jun 2026 16:01:49 +0800 Cc: , , , , , Message-Id: <20260603080152.64728-6-lizhe.67@bytedance.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 Content-Type: text/plain; charset="utf-8" The template fast path from the previous patch only accelerates head pages. Compound tails in memmap_init_compound() still go through the slow path one by one. Build separate head and tail templates and reuse one prepared tail template across the tail pages in a compound range. Head pages preserve the existing refcount policy, while compound tails always start with a refcount of 0 after prep_compound_tail(). This extends the template-copy fast path to pfns_per_compound > 1 without changing the existing slow path. Tail-page PFN-dependent fields are refreshed in the reusable tail template before each copy. Tested in a VM with a 100 GB devdax namespace (align=3D2097152) on Intel Ice Lake server. This test exercises the dax_pmem rebind path and measures memmap initialization latency. Test procedure: Unbind and rebind the dax_pmem driver 30 times, collect memmap initialization time from the pr_debug() output of memmap_init_zone_device(). Base(v7.1-rc6): First binding: 1430 ms Average of subsequent rebinds: 229.12 ms With this patch and its prerequisites applied: First binding: 1336 ms Average of subsequent rebinds: 180.00 ms This reduces the average rebind time from 229.12 ms to 180.00 ms, or about 21.4%. Signed-off-by: Li Zhe --- mm/mm_init.c | 46 +++++++++++++++++++++++++++++++++++++++------- 1 file changed, 39 insertions(+), 7 deletions(-) diff --git a/mm/mm_init.c b/mm/mm_init.c index 56fe71e966ae..ad078ee354fb 100644 --- a/mm/mm_init.c +++ b/mm/mm_init.c @@ -1083,6 +1083,16 @@ static inline void zone_device_template_page_init(st= ruct page *template, memcpy(template, src, sizeof(*template)); } =20 +static inline void zone_device_tail_page_init(struct page *page, + unsigned long pfn, unsigned long zone_idx, int nid, + struct dev_pagemap *pgmap, const struct page *head, + unsigned int order) +{ + zone_device_page_init_slow(page, pfn, zone_idx, nid, pgmap); + prep_compound_tail(page, head, order); + set_page_count(page, 0); +} + /* * 'template' is a reusable page prototype rather than a strictly immutable * object. Most ZONE_DEVICE fields stay constant across the pages covered = by @@ -1134,10 +1144,12 @@ static void __ref memmap_init_compound(struct page = *head, unsigned long head_pfn, unsigned long zone_idx, int nid, struct dev_pagemap *pgmap, - unsigned long nr_pages) + unsigned long nr_pages, + bool use_template) { unsigned long pfn, end_pfn =3D head_pfn + nr_pages; unsigned int order =3D pgmap->vmemmap_shift; + struct page template; =20 /* * We have to initialize the pages, including setting up page links. @@ -1146,12 +1158,31 @@ static void __ref memmap_init_compound(struct page = *head, * the pages in the same go. */ __SetPageHead(head); - for (pfn =3D head_pfn + 1; pfn < end_pfn; pfn++) { + + pfn =3D head_pfn + 1; + /* + * All tails of the same compound page share the state established by + * prep_compound_tail(). Reuse one tail template for the whole range and + * refresh only the PFN-dependent fields in that template before each cop= y. + */ + if (use_template) { struct page *page =3D pfn_to_page(pfn); =20 - zone_device_page_init_slow(page, pfn, zone_idx, nid, pgmap); - prep_compound_tail(page, head, order); - set_page_count(page, 0); + zone_device_tail_page_init(page, pfn, zone_idx, nid, + pgmap, head, order); + zone_device_template_page_init(&template, page); + pfn++; + } + + for (; pfn < end_pfn; pfn++) { + struct page *page =3D pfn_to_page(pfn); + + if (use_template) + zone_device_page_init_from_template(page, pfn, + &template); + else + zone_device_tail_page_init(page, pfn, zone_idx, nid, + pgmap, head, order); } prep_compound_head(head, order); } @@ -1200,7 +1231,7 @@ void __ref memmap_init_zone_device(struct zone *zone, zone_device_template_page_init(&template, page); if (pfns_per_compound !=3D 1) memmap_init_compound(page, pfn, zone_idx, nid, pgmap, - compound_nr_pages(altmap, pgmap)); + compound_nr_pages(altmap, pgmap), use_template); pfn +=3D pfns_per_compound; } =20 @@ -1218,7 +1249,8 @@ void __ref memmap_init_zone_device(struct zone *zone, continue; =20 memmap_init_compound(page, pfn, zone_idx, nid, pgmap, - compound_nr_pages(altmap, pgmap)); + compound_nr_pages(altmap, pgmap), + use_template); } =20 pr_debug("%s initialised %lu pages in %ums\n", __func__, --=20 2.20.1 From nobody Mon Jun 8 19:49:21 2026 Received: from va-2-114.ptr.blmpb.com (va-2-114.ptr.blmpb.com [209.127.231.114]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D0C57403EA2 for ; Wed, 3 Jun 2026 08:07:07 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.127.231.114 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780474029; cv=none; b=AbmnhMfdLdj58/M233XCGz0GXLp+8fuktgmmvjgRRfcv1Mui9d/cRB1DGVWltfqmec0jIMBOane4v5ECe43FLZnqgib8OCa83PKFmI/Y5+fsKZXb0okLqWKagr/Xptu4x6lVH080BMjEbBcSdVsCTL/4R3FRjirFR/D1SGEeDmY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780474029; c=relaxed/simple; bh=63NMpe0PRElzthaZjCab7XFUgcysrLnekDQoCS6vnp4=; h=From:Subject:Date:Content-Type:Cc:In-Reply-To:References: Mime-Version:To:Message-Id; b=KdAfNG+pk3z4gXmAOroFM0sRoAlCpq71Xsb0nlUmkkMXOOO2OfD0D8oupDt4b7TXqr6cxSD6ZkvcYOTHTlFZO2gKBuJXVMtyN5WXqf5ttLhAv7QW+yWrk/S2YllZdCAFCiflR+olxFLzm6WFDjBDX5+HsaDpe9Lvt4JFpXPqdec= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=U4dMfXKX; arc=none smtp.client-ip=209.127.231.114 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="U4dMfXKX" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; s=2212171451; d=bytedance.com; t=1780474023; h=from:subject: mime-version:from:date:message-id:subject:to:cc:reply-to:content-type: mime-version:in-reply-to:message-id; bh=QbTAW4+JtusN/1ajk2i57VFeY4ZMdb/3rGS57KirNj4=; b=U4dMfXKXNBR99SZJTA3fZeC7T+ReleTpxNpkP5iVjZwBtp1OeC6g2aTZGuDRc+eJwpED+i NY6uRJ1TGh20Ek4WJOxe1ZtTdNydHqNmQ7+CcoK3Eq3zLcc4rD8MoKZoIfeF79HUymeyev UQ/4IXBCLuG8l0mcM5JLX8xABZaAKFw3ZZRdrbZCGjHx0xiAmiGAeMrMdvFFYu19TVH+RS xxacHLnKPIRxM6D+0iuqSgRAqq5/ccz1Twu+4Q/m95WqO4bV+DHwiLb/G63thl5jc+0J+4 nJkB4DiJ8wR07xqGb1Zl1lIiOrD4UoHnqIeFHvnxcqo+/gMATzvX4xToDNsBuA== From: "Li Zhe" Subject: [PATCH v4 6/8] string: introduce memcpy_streaming() helpers Date: Wed, 3 Jun 2026 16:01:50 +0800 Content-Transfer-Encoding: quoted-printable Cc: , , , , , In-Reply-To: <20260603080152.64728-1-lizhe.67@bytedance.com> References: <20260603080152.64728-1-lizhe.67@bytedance.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 X-Lms-Return-Path: To: , , , , , , , , , X-Original-From: Li Zhe X-Mailer: git-send-email 2.45.2 Message-Id: <20260603080152.64728-7-lizhe.67@bytedance.com> Content-Type: text/plain; charset="utf-8" Introduce a generic memcpy_streaming() interface for write-once copy sites that can fall back to memcpy() when no architecture-specific optimization is available, or when an architecture-specific backend cannot safely handle a given transfer. Add memcpy_streaming_drain() alongside it so callers can separate the copy primitive from any required ordering point. On x86, use memcpy_flushcache() and sfence only for aligned transfers that can stay entirely on the non-temporal store path; otherwise fall back to memcpy() so the generic API does not expose flushcache semantics on cached head/tail fragments. Callers are responsible for invoking memcpy_streaming_drain() before later normal stores that must be ordered after the streaming copy. Signed-off-by: Li Zhe --- arch/x86/include/asm/string_64.h | 32 ++++++++++++++++++++++++++++++++ include/linux/string.h | 20 ++++++++++++++++++++ 2 files changed, 52 insertions(+) diff --git a/arch/x86/include/asm/string_64.h b/arch/x86/include/asm/string= _64.h index 4635616863f5..aee63108577f 100644 --- a/arch/x86/include/asm/string_64.h +++ b/arch/x86/include/asm/string_64.h @@ -4,6 +4,7 @@ =20 #ifdef __KERNEL__ #include +#include =20 /* Written 2002 by Andi Kleen */ =20 @@ -100,6 +101,37 @@ static __always_inline void memcpy_flushcache(void *ds= t, const void *src, size_t } __memcpy_flushcache(dst, src, cnt); } + +/* + * Only map memcpy_streaming() to memcpy_flushcache() when the destination + * is already 8-byte aligned and the size can be handled without cached + * head/tail fragments in __memcpy_flushcache(). + */ +static __always_inline bool memcpy_flushcache_nt_safe(const void *dst, + size_t cnt) +{ + unsigned long d =3D (unsigned long)dst; + + return cnt && IS_ALIGNED(d, 8) && IS_ALIGNED(cnt, 4); +} + +#define __HAVE_ARCH_MEMCPY_STREAMING 1 +static __always_inline void memcpy_streaming(void *dst, const void *src, + size_t cnt) +{ + if (!cnt) + return; + + if (memcpy_flushcache_nt_safe(dst, cnt)) + memcpy_flushcache(dst, src, cnt); + else + memcpy(dst, src, cnt); +} + +static __always_inline void memcpy_streaming_drain(void) +{ + asm volatile("sfence" : : : "memory"); +} #endif =20 #endif /* __KERNEL__ */ diff --git a/include/linux/string.h b/include/linux/string.h index b850bd91b3d8..a4c2d4347f58 100644 --- a/include/linux/string.h +++ b/include/linux/string.h @@ -281,6 +281,26 @@ static inline void memcpy_flushcache(void *dst, const = void *src, size_t cnt) } #endif =20 +#ifndef __HAVE_ARCH_MEMCPY_STREAMING +/* + * memcpy_streaming() is for write-once copy sites that may use + * non-temporal stores on some architectures. Callers must follow it + * with memcpy_streaming_drain() before later normal stores that need to + * be ordered after the streaming copy. Implementations may fall back to + * memcpy() when a specialized backend cannot safely handle the given + * transfer, and backends that use regular cached stores can make the + * drain a no-op. + */ +static inline void memcpy_streaming(void *dst, const void *src, size_t cnt) +{ + memcpy(dst, src, cnt); +} + +static inline void memcpy_streaming_drain(void) +{ +} +#endif + void *memchr_inv(const void *s, int c, size_t n); char *strreplace(char *str, char old, char new); =20 --=20 2.20.1 From nobody Mon Jun 8 19:49:21 2026 Received: from va-1-114.ptr.blmpb.com (va-1-114.ptr.blmpb.com [209.127.230.114]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id CF1E7405C44 for ; Wed, 3 Jun 2026 08:07:45 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.127.230.114 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780474067; cv=none; b=VAhGvIlauiquZEFtOgykdq+B9dig1DK0UbOIlDKabWjUUImJW2j1LAQQL8byRYa1zUNUGOqyrqw6jnkCcgMWeQdY1khOVuGSGZ5+cCyvig5S6dGbLXgoS4ly7Wibwvr80iaB80MhoNBEADlhEMUz3SAuHD++3ho1A0J04/Vsh4s= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780474067; c=relaxed/simple; bh=z/Ea2k7fpYsGEUt9hh88F/a4jksvVj//QreE6FccEYs=; h=Subject:Date:From:In-Reply-To:Message-Id:Mime-Version:To: References:Content-Type:Cc; b=WLZdrW/yG+gVsO0kSrXCQWKdZ7GrUxXHDPTCdQY6acIO3HFYqxIBuib3CNLQL2uDE73akgPmRyKv+z+9H1xtcXgt4+SyIE3IlUbCBG1QOYhQi21vtS8bo8aJ+lVurjY0+Lg9vf3FNw+ZG8ph2li1Z7qM6CQMBt1ALmCvdA4Wjus= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=UzbACo83; arc=none smtp.client-ip=209.127.230.114 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="UzbACo83" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; s=2212171451; d=bytedance.com; t=1780474061; h=from:subject: mime-version:from:date:message-id:subject:to:cc:reply-to:content-type: mime-version:in-reply-to:message-id; bh=gRJ/J0Nb17+iEYOrMVREQ16d2zbIiv+jVQRVIwQKMg0=; b=UzbACo833+fZrHQgsSodG9/cGosdHc9uxTR0mzQTdV/YYzjBVArtNlLsAR1Fuwvu2ewzTK 6D4DJIRRTAKPLSjDCDgpzkgLVxy9k1KmkL4cqXxHo1WmZdeNhK5oAdFZF6h8Qq21bMk39d Jq9qOAMpmD9cog0JqIBZCyaVcg/6k9LU4A7Dksc5p8/Yc38fs/hQKHkpthMYuMC4praukD 8dpIf16uBEtZnyCmclYuhgprQ9MXD9HMRZv60Khhwg1M885ZWxHa54mnR+BW2Xpv1ebjte IUdRAAkeeJuY89e3CqOtB70jLu/u0qi1qrX6aCiDnhulkYt8lE/bPpaGwsQOug== Subject: [PATCH v4 7/8] x86/string: extend memcpy_flushcache() fixed-size fastpaths Date: Wed, 3 Jun 2026 16:01:51 +0800 From: "Li Zhe" In-Reply-To: <20260603080152.64728-1-lizhe.67@bytedance.com> X-Mailer: git-send-email 2.45.2 Message-Id: <20260603080152.64728-8-lizhe.67@bytedance.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 X-Original-From: Li Zhe To: , , , , , , , , , X-Lms-Return-Path: Content-Transfer-Encoding: quoted-printable References: <20260603080152.64728-1-lizhe.67@bytedance.com> Cc: , , , , , Content-Type: text/plain; charset="utf-8" Small constant-sized flushcache copies currently inline only the 4, 8, and 16-byte cases. Larger constant sizes fall back to __memcpy_flushcache() even when the destination is naturally aligned. Factor the movnti sequences into 4/8/16/32/64-byte helpers and extend the inline fastpath coverage to the additional aligned constant sizes 32, 48, 64, 80, and 96 bytes. Keep the existing 4/8/16-byte cases handled directly in memcpy_flushcache() so they do not pick up the extra alignment gating used by the larger fixed-size helpers. Because memcpy_streaming() maps aligned transfers to memcpy_flushcache(), these additional fixed-size cases also stay on the inline movnti path for that helper. Issue the stores in ascending address order so write-combining sees a forward stream. Keep all other sizes on __memcpy_flushcache(), and keep zero-length copies returning immediately. Signed-off-by: Li Zhe --- arch/x86/include/asm/string_64.h | 122 ++++++++++++++++++++++++++----- 1 file changed, 104 insertions(+), 18 deletions(-) diff --git a/arch/x86/include/asm/string_64.h b/arch/x86/include/asm/string= _64.h index aee63108577f..16d1aac2da24 100644 --- a/arch/x86/include/asm/string_64.h +++ b/arch/x86/include/asm/string_64.h @@ -83,24 +83,6 @@ int strcmp(const char *cs, const char *ct); #ifdef CONFIG_ARCH_HAS_UACCESS_FLUSHCACHE #define __HAVE_ARCH_MEMCPY_FLUSHCACHE 1 void __memcpy_flushcache(void *dst, const void *src, size_t cnt); -static __always_inline void memcpy_flushcache(void *dst, const void *src, = size_t cnt) -{ - if (__builtin_constant_p(cnt)) { - switch (cnt) { - case 4: - asm ("movntil %1, %0" : "=3Dm"(*(u32 *)dst) : "r"(*(u32 *)src)); - return; - case 8: - asm ("movntiq %1, %0" : "=3Dm"(*(u64 *)dst) : "r"(*(u64 *)src)); - return; - case 16: - asm ("movntiq %1, %0" : "=3Dm"(*(u64 *)dst) : "r"(*(u64 *)src)); - asm ("movntiq %1, %0" : "=3Dm"(*(u64 *)(dst + 8)) : "r"(*(u64 *)(src += 8))); - return; - } - } - __memcpy_flushcache(dst, src, cnt); -} =20 /* * Only map memcpy_streaming() to memcpy_flushcache() when the destination @@ -115,6 +97,110 @@ static __always_inline bool memcpy_flushcache_nt_safe(= const void *dst, return cnt && IS_ALIGNED(d, 8) && IS_ALIGNED(cnt, 4); } =20 +static __always_inline void memcpy_flushcache_4(void *dst, const void *src) +{ + asm volatile("movntil %1, %0" + : "=3Dm"(*(u32 *)dst) + : "r"(*(const u32 *)src) + : "memory"); +} + +static __always_inline void memcpy_flushcache_8(void *dst, const void *src) +{ + asm volatile("movntiq %1, %0" + : "=3Dm"(*(u64 *)dst) + : "r"(*(const u64 *)src) + : "memory"); +} + +static __always_inline void memcpy_flushcache_16(void *dst, + const void *src) +{ + memcpy_flushcache_8(dst, src); + memcpy_flushcache_8(dst + 8, src + 8); +} + +static __always_inline void memcpy_flushcache_32(void *dst, + const void *src) +{ + memcpy_flushcache_16(dst, src); + memcpy_flushcache_16(dst + 16, src + 16); +} + +static __always_inline void memcpy_flushcache_64(void *dst, + const void *src) +{ + memcpy_flushcache_32(dst, src); + memcpy_flushcache_32(dst + 32, src + 32); +} + +/* + * Keep the additional aligned fixed-size cases on the inline movnti path. + * Leave the existing 4/8/16-byte cases handled directly in + * memcpy_flushcache() so they do not pick up the extra alignment gating + * used by the larger fixed-size helpers. + */ +static __always_inline int memcpy_flushcache_large(void *dst, + const void *src, + size_t cnt) +{ + unsigned long d =3D (unsigned long)dst; + char *dptr =3D dst; + const char *sptr =3D src; + + if (!IS_ALIGNED(d, 8)) + return 0; + + switch (cnt) { + case 32: + memcpy_flushcache_32(dptr, sptr); + return 1; + case 48: + memcpy_flushcache_32(dptr, sptr); + memcpy_flushcache_16(dptr + 32, sptr + 32); + return 1; + case 64: + memcpy_flushcache_64(dptr, sptr); + return 1; + case 80: + memcpy_flushcache_64(dptr, sptr); + memcpy_flushcache_16(dptr + 64, sptr + 64); + return 1; + case 96: + memcpy_flushcache_64(dptr, sptr); + memcpy_flushcache_32(dptr + 64, sptr + 64); + return 1; + } + + return 0; +} + +static __always_inline void memcpy_flushcache(void *dst, const void *src, + size_t cnt) +{ + if (!cnt) + return; + + if (__builtin_constant_p(cnt)) { + switch (cnt) { + case 4: + memcpy_flushcache_4(dst, src); + return; + case 8: + memcpy_flushcache_8(dst, src); + return; + case 16: + memcpy_flushcache_16(dst, src); + return; + } + + if (memcpy_flushcache_large(dst, src, cnt)) + return; + } + + __memcpy_flushcache(dst, src, cnt); +} + #define __HAVE_ARCH_MEMCPY_STREAMING 1 static __always_inline void memcpy_streaming(void *dst, const void *src, size_t cnt) --=20 2.20.1 From nobody Mon Jun 8 19:49:21 2026 Received: from va-2-113.ptr.blmpb.com (va-2-113.ptr.blmpb.com [209.127.231.113]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 22A92407CE2 for ; Wed, 3 Jun 2026 08:08:12 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.127.231.113 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780474094; cv=none; b=DPOI8gdhGLSGrTAORp0hYvFHJLe86+uFGuTS01iyJjjWD5wbrK5CfWkv8xLYtDJZnMSIhcM0pogyqeytbwmoOrZZ9di9C0F7Gvf8NHrlRPVfdGrig455iF/Ze5+28xEZjULwPQDbkIVRGWMAO+ORRyIrnpx/ryUHv9GX0r8jIgE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780474094; c=relaxed/simple; bh=NDZ6qErnQG6w0k+nmX3Ucp0GetTk9yJWwLhq6duuSSU=; h=In-Reply-To:Content-Type:To:From:Subject:Date:Message-Id: References:Cc:Mime-Version; b=bRH3iYpgTv5521wK6EfkVx1s+0i/6/fLDI4XR1dLNuuNhziwf0SfBPE3mTpe67QoByzy9kEVBY6m4J7VF5MoiP3+rcYR5bg5nY3GgoGZpKWnkdY8fACIKk2x7CM0/BZWOo39wNkQqvI/73usjebvbzka7ChfKwRgxX9ev/fU4yU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=pB4S8BwY; arc=none smtp.client-ip=209.127.231.113 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="pB4S8BwY" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; s=2212171451; d=bytedance.com; t=1780474088; h=from:subject: mime-version:from:date:message-id:subject:to:cc:reply-to:content-type: mime-version:in-reply-to:message-id; bh=A5euipl6qc7ONaSWy009BEQvfxAQyFAQV+XgcOVI8kM=; b=pB4S8BwYAbNJ4NDAhr3y0o+XUY71E7wUogqlxhMSeC7adJ4B9e3UXJVbAjlOyy39srx9NF M6nBZgYHNkTxAxfQUI1EyJlHPBPk73tFzQqtarfY3fIDbYoKSx6wK05noC+cr7UjhqoJvf OnLbv6p13Dzpaeh/LC8msfhWHWaPViH4UO3Lk2/d9RueXY7+bnnVebgJP91lvv+So7GqdQ VHaJoMHp7BKTyxgdcPFmkVWL5QsIeNOBiQjVHgnbIht7lUfcjcCJOqHN7t06t1a/6vL5l6 kDZBlBjOLpTdY4UZ8Mk1bFearc1ysjWCtxe0ttTdZqnrqW/xi0oudyu8UhQkdA== X-Lms-Return-Path: Content-Transfer-Encoding: quoted-printable In-Reply-To: <20260603080152.64728-1-lizhe.67@bytedance.com> To: , , , , , , , , , From: "Li Zhe" X-Original-From: Li Zhe Subject: [PATCH v4 8/8] mm: use memcpy_streaming() in zone-device template copies Date: Wed, 3 Jun 2026 16:01:52 +0800 Message-Id: <20260603080152.64728-9-lizhe.67@bytedance.com> References: <20260603080152.64728-1-lizhe.67@bytedance.com> Cc: , , , , , Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 X-Mailer: git-send-email 2.45.2 Content-Type: text/plain; charset="utf-8" The template fast path still leaves the actual copy sequence up to the compiler. Use the streaming-copy helpers introduced in the previous patches for the ZONE_DEVICE template-copy path so common mm code can request a write-once copy primitive without embedding arch-specific store layout in the generic layer. ZONE_DEVICE memmap initialization is a write-once path: each struct page is populated once and is not expected to be reused from cache immediately afterwards. A regular cached copy can therefore incur write-allocate traffic and pollute the cache without much benefit. Using memcpy_streaming() lets this path use an architecture-optimized streaming copy where available, while still degrading to memcpy() on architectures that do not provide a specialized implementation. Keep pageblock-aligned PFNs on memcpy() so pageblock initialization can immediately read back page metadata without introducing a read-after-streaming dependency. For the remaining PFNs, use memcpy_streaming() so the hot path can avoid write-allocate traffic while still leaving unsupported or unsuitable cases to the fallback implementation. When the streaming backend uses non-temporal stores, order them before entering memmap_init_compound(), before prep_compound_head() updates the overlapping compound metadata, and before returning from memmap_init_zone_device(). Keep sanitized builds on the slow path so KASAN/KMSAN retain their instrumented stores. Tested in a VM with a 100 GB fsdax namespace device configured with map=3Ddev and a 100 GB devdax namespace (align=3D2097152) on Intel Ice Lake server. Test procedure: Rebind the nd_pmem and dax_pmem driver 30 times and collect the memmap initialization time from the pr_debug() output of memmap_init_zone_device(). Base(v7.1-rc6): First binding for nd_pmem driver: 1466 ms Average of subsequent rebinds: 262.12 ms First binding for dax_pmem driver: 1430 ms Average of subsequent rebinds: 229.12 ms With this series: First binding for nd_pmem driver: 1359 ms Average of subsequent rebinds: 108.36 ms First binding for dax_pmem driver: 1273 ms Average of subsequent rebinds: 100.17 ms This reduces the average rebind time by about 58.6% for nd_pmem and 56.3% for dax_pmem. Signed-off-by: Li Zhe --- mm/mm_init.c | 49 +++++++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 47 insertions(+), 2 deletions(-) diff --git a/mm/mm_init.c b/mm/mm_init.c index ad078ee354fb..fbc873284fb8 100644 --- a/mm/mm_init.c +++ b/mm/mm_init.c @@ -1070,11 +1070,21 @@ static void __ref zone_device_page_init_slow(struct= page *page, =20 static inline bool zone_device_page_init_optimization_enabled(void) { + /* + * Keep sanitized builds on the slow path so their stores stay + * instrumented. + */ + if (IS_ENABLED(CONFIG_KASAN) || IS_ENABLED(CONFIG_KMSAN)) + return false; + /* * The template fast path copies a preinitialized struct page image. * Skip it when the page_ref_set tracepoint is enabled. */ - return !page_ref_tracepoint_active(page_ref_set); + if (page_ref_tracepoint_active(page_ref_set)) + return false; + + return true; } =20 static inline void zone_device_template_page_init(struct page *template, @@ -1117,9 +1127,19 @@ static void zone_device_page_init_from_template(stru= ct page *page, * 'template' carries the invariant portion of a ZONE_DEVICE struct * page. Update the PFN-dependent fields in place before copying it * to the destination page. + * + * pageblock-aligned pages immediately feed + * init_pageblock_migratetype(), which reads back page metadata via + * helpers like page_zone(page). Avoid a read-after-streaming + * dependency for these rare pages by using regular cached stores + * instead of non-temporal ones. */ zone_device_page_update_template(template, pfn); - memcpy(page, template, sizeof(*page)); + if (unlikely(pageblock_aligned(pfn))) + memcpy(page, template, sizeof(*page)); + else + memcpy_streaming(page, template, sizeof(*page)); + zone_device_page_init_pageblock(page, pfn); } =20 @@ -1184,6 +1204,15 @@ static void __ref memmap_init_compound(struct page *= head, zone_device_tail_page_init(page, pfn, zone_idx, nid, pgmap, head, order); } + + /* + * When the template path is enabled, order the preceding tail-page copies + * before prep_compound_head() updates the overlapping compound metadata + * in the first tail-page descriptors. If memcpy_streaming() fell back to + * regular cached stores, memcpy_streaming_drain() may be a no-op. + */ + if (use_template) + memcpy_streaming_drain(); prep_compound_head(head, order); } =20 @@ -1248,10 +1277,26 @@ void __ref memmap_init_zone_device(struct zone *zon= e, if (pfns_per_compound =3D=3D 1) continue; =20 + /* + * When the template path is enabled, order the preceding head-page copy + * before memmap_init_compound(), which immediately updates compound-head + * metadata. If memcpy_streaming() fell back to regular cached stores, + * memcpy_streaming_drain() may be a no-op. + */ + if (use_template) + memcpy_streaming_drain(); + memmap_init_compound(page, pfn, zone_idx, nid, pgmap, compound_nr_pages(altmap, pgmap), use_template); } + /* + * Ensure any prior template copies are ordered before returning. + * On architectures where memcpy_streaming() used regular cached stores, + * memcpy_streaming_drain() may be a no-op. + */ + if (use_template) + memcpy_streaming_drain(); =20 pr_debug("%s initialised %lu pages in %ums\n", __func__, nr_pages, jiffies_to_msecs(jiffies - start)); --=20 2.20.1