From nobody Mon Jun 8 21:55:13 2026 Received: from mail-lf1-f42.google.com (mail-lf1-f42.google.com [209.85.167.42]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6A8FA3E3DA2 for ; Tue, 26 May 2026 11:37:39 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.167.42 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1779795465; cv=none; b=c6jIa+iaqvqz66Y7UM14h/DAT8FtOezfgvFuUg55vQdoDG3cc654DTgUUJ7Wjo+HYSydaxXLh5/XEPM5JdaQFeDlfu1kNXymQEPTU7zR/80PZlrHNfopm3wHpGCs6FBsfyh5kK6xD5mUMe3PFJS0Igwr6/jKkIR5mdYqtxQFwPw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1779795465; c=relaxed/simple; bh=5MIHgChRNy12kVUm5iWs8ChIekHBsvSP0fhUEXrwf9U=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=FuF1d2sfZMJeaOhKX7Sq1/JQCOpWSu+KflBvtJr2Qx2MYqUS9Am7ht8F7HzwcrUj8mWoFIckQ6vg7IazZ8T03Pe3OrZ/trlP0veuw220o/zHIDhyQl4e9oBwljshZ0oAZBAaZOrKxBaniYaV1xGIcun3oPebifFkxUtYITTaeqc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=rJQhF0c/; arc=none smtp.client-ip=209.85.167.42 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="rJQhF0c/" Received: by mail-lf1-f42.google.com with SMTP id 2adb3069b0e04-5aa0cf8bca3so9527527e87.0 for ; Tue, 26 May 2026 04:37:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1779795457; x=1780400257; darn=vger.kernel.org; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :mime-version:subject:date:from:from:to:cc:subject:date:message-id :reply-to; bh=RepOlA7uegAkHRX0rdEAbivBLVoGLgApQvWSVyuj8bk=; b=rJQhF0c/wNUvYl4FdibYFxZABQ4a1PwizuxWRg+q9h8Cbkt+dYbsnTNMO3bjlKf5cH diSP9ZV0qO3t9eIcVsfjURWbiA30lCDwbWkCHj37E/fCGyJhXBzETl/Ss8WJjc2TvSYS I8YF5WHHp8qZR3Py5hgp5i014cMmeBUFZkP4/r0AKWPLlHn/PKidrqEQ3Cf3NhmdinZ6 ZTaWKDNvTgYMBu4SOQop2oHobW/1+F7uGsMAC/ggxv86svSy0FDgDQDqNNECObEBowaa n7ptagAbTPkvPfZWPd6MgD2SwbpwuhZRKEhKTnG59DQI0CqpUFCtGJAjO1vGD+/5zoRt va7Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1779795457; x=1780400257; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :mime-version:subject:date:from:x-gm-gg:x-gm-message-state:from:to :cc:subject:date:message-id:reply-to; bh=RepOlA7uegAkHRX0rdEAbivBLVoGLgApQvWSVyuj8bk=; b=oNU5+WEDBjlsL8b3Xu3t3xP7xE8UZhTGfqtX6q5vPS0ZK3OsXUlqmtXDf21rordn8D Gj89/X2EXodCuZBtsqHP/mXjobtkj49jZ5rTsQlIuM+0z7BSt1mL0eLrnXVLYpjFE0qJ O3RjSlSqZa0nyH1GyRQ+NqgLr33WeoZXpYekktUMAlqVjm065omOAYjokMKAtWZtjEaP akqJaN2JdCb6tTX0TzeXJ8quDM9+YoSgYv1lc2wkc4snQWXSyRsgMU3dSABMwmuDpz1c VhdNbksasynK5z5oLF8gvzkhm9ALzV/i/vPBybRSKxjynUBw0Sermh/PbT8l+kitbnJQ Si3Q== X-Forwarded-Encrypted: i=1; AFNElJ9KkE1d7rdvM4SfSstOax11RlIDUjd8ZG1/Jo3M65MJGXFDJSmEdiKR0A/2ikYDBo1e+WLYX9C+UdWSaTg=@vger.kernel.org X-Gm-Message-State: AOJu0YzeTmpdGaw3hewmFjzgZT3hRTjbClUxcrCtFg5ONbl8hC8GnZ6D /3c8y4tr+WsTahnt8I3nO9QMgS3GlaHKQvDDpK83d5jfm/f6xn8Yl1Qe X-Gm-Gg: Acq92OGX69CiiyhwPqOaRK8Ny/vQw5OwUoYQ3EUD62mpK15mLabOgg46VBPnNTcku98 Jq4Ful2zITsQ56OmfmcN3rUjYs4SSryDVxqN6dFaaSrm43kIeCYxRLuoBb33a1Yn1HTMJ063Q10 ShznrLwCpVf+vOgvFKjSsXQUMrBaaxf9lM49TQOYmBEPgy63ZfANdzg7c+POK2WW6jZEIunNF0N wumXvrJ/KipY5HW8PnYrD2FuYYSIQCvyB53oydzmhajEgJJN15ffANaskcCrcgS2nJ2ko2FE1Wv g7VGSUWHTQHDXVWNXqv72Om7fAnbBjFyP646KFwmWwbhUccKmlqWdwhqFfn2gdwD7aRHJwjBP74 faC5HPl63e+/T5ixq5b9dbSsXZhmMFmmKrk6x5TEW/Lim4eEcPcY7UuUei8G76u97mi2mHb3GSk yxd8jtsxcFF78X8FkhLynTC6YPc3oL2ZUsx7NLCMGr614NygA5j00g1c6awNRU0xYNoE04 X-Received: by 2002:a05:6512:48cc:b0:5a8:87fc:2b34 with SMTP id 2adb3069b0e04-5aa323c3ee8mr3865629e87.32.1779795457167; Tue, 26 May 2026 04:37:37 -0700 (PDT) Received: from [127.0.1.1] (h-62-63-197-109.NA.cust.bahnhof.se. [62.63.197.109]) by smtp.gmail.com with ESMTPSA id 2adb3069b0e04-5aa46330d9asm376332e87.59.2026.05.26.04.37.35 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 26 May 2026 04:37:36 -0700 (PDT) From: Yiannis Nikolakopoulos Date: Tue, 26 May 2026 13:37:02 +0200 Subject: [PATCH RFC 1/3] mm, x86: support copying a folio using non-temporal stores Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20260526-rfc-nt-demote-v1-1-eb9c9422daef@zptcorp.com> References: <20260526-rfc-nt-demote-v1-0-eb9c9422daef@zptcorp.com> In-Reply-To: <20260526-rfc-nt-demote-v1-0-eb9c9422daef@zptcorp.com> To: Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , x86@kernel.org, "H. Peter Anvin" , Andrew Morton , David Hildenbrand , Lorenzo Stoakes , "Liam R. Howlett" , Vlastimil Babka , Mike Rapoport , Suren Baghdasaryan , Michal Hocko , Trond Myklebust , Anna Schumaker , Zi Yan , Matthew Brost , Joshua Hahn , Rakie Kim , Byungchul Park , Gregory Price , Ying Huang , Alistair Popple , Steven Rostedt , Masami Hiramatsu , Mathieu Desnoyers , Brendan Jackman , Johannes Weiner Cc: David Rientjes , Davidlohr Bueso , Fan Ni , Frank van der Linden , Jonathan Cameron , Raghavendra K T , "Rao, Bharata Bhasker" , SeongJae Park , Wei Xu , Xuezheng Chu , Yiannis Nikolakopoulos , dimitrios@palyvos.net, Ryan Roberts , linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-nfs@vger.kernel.org, linux-trace-kernel@vger.kernel.org, Yiannis Nikolakopoulos , Alirad Malek X-Mailer: b4 0.15.2 From: Alirad Malek In x86, use memcpy_flushcache (that uses non-temporal store instructions) to copy a folio. To achieve that, starting from folio_mc_copy down to copy_mc_to_kernel, create a series of helpers (named with an _nt suffix) that have similar behavior to the original counterparts. Signed-off-by: Alirad Malek Co-developed-by: Yiannis Nikolakopoulos Signed-off-by: Yiannis Nikolakopoulos --- arch/x86/include/asm/uaccess.h | 4 ++++ arch/x86/lib/copy_mc.c | 26 ++++++++++++++++++++++++++ include/linux/highmem.h | 32 ++++++++++++++++++++++++++++++++ include/linux/mm.h | 1 + mm/util.c | 17 +++++++++++++++++ 5 files changed, 80 insertions(+) diff --git a/arch/x86/include/asm/uaccess.h b/arch/x86/include/asm/uaccess.h index 367297b188c3..2d0938d3e372 100644 --- a/arch/x86/include/asm/uaccess.h +++ b/arch/x86/include/asm/uaccess.h @@ -494,6 +494,10 @@ unsigned long __must_check copy_mc_to_kernel(void *to, const void *from, unsigned len); #define copy_mc_to_kernel copy_mc_to_kernel =20 +unsigned long __must_check +copy_mc_to_kernel_nt(void *to, const void *from, unsigned len); +#define copy_mc_to_kernel_nt copy_mc_to_kernel_nt + unsigned long __must_check copy_mc_to_user(void __user *to, const void *from, unsigned len); #endif diff --git a/arch/x86/lib/copy_mc.c b/arch/x86/lib/copy_mc.c index 97e88e58567b..5a2ee5c2211e 100644 --- a/arch/x86/lib/copy_mc.c +++ b/arch/x86/lib/copy_mc.c @@ -81,6 +81,32 @@ unsigned long __must_check copy_mc_to_kernel(void *dst, = const void *src, unsigne } EXPORT_SYMBOL_GPL(copy_mc_to_kernel); =20 +/** + * copy_mc_to_kernel_nt - memory copy that handles source exceptions + * if enabled, otherwise uses non-temporal stores + * @dst: destination address + * @src: source address + * @len: number of bytes to copy + * + * Return 0 for success, or number of bytes not copied if there was an + * exception. + */ +unsigned long __must_check copy_mc_to_kernel_nt(void *dst, const void *src= , unsigned len) +{ + unsigned long ret; + + if (copy_mc_fragile_enabled) { + instrument_memcpy_before(dst, src, len); + ret =3D copy_mc_fragile(dst, src, len); + instrument_memcpy_after(dst, src, len, ret); + return ret; + } + + memcpy_flushcache(dst, src, len); + return 0; +} +EXPORT_SYMBOL_GPL(copy_mc_to_kernel_nt); + unsigned long __must_check copy_mc_to_user(void __user *dst, const void *s= rc, unsigned len) { unsigned long ret; diff --git a/include/linux/highmem.h b/include/linux/highmem.h index af03db851a1d..a5cb435b9ffe 100644 --- a/include/linux/highmem.h +++ b/include/linux/highmem.h @@ -468,6 +468,32 @@ static inline int copy_mc_highpage(struct page *to, st= ruct page *from) =20 return ret; } + +#ifdef copy_mc_to_kernel_nt +static inline int copy_mc_highpage_nt(struct page *to, struct page *from) +{ + unsigned long ret; + char *vfrom, *vto; + + vfrom =3D kmap_local_page(from); + vto =3D kmap_local_page(to); + ret =3D copy_mc_to_kernel_nt(vto, vfrom, PAGE_SIZE); + if (!ret) + kmsan_copy_page_meta(to, from); + kunmap_local(vto); + kunmap_local(vfrom); + + if (ret) + memory_failure_queue(page_to_pfn(from), 0); + + return ret; +} +#else +static inline int copy_mc_highpage_nt(struct page *to, struct page *from) +{ + return copy_mc_highpage(to, from); +} +#endif #else static inline int copy_mc_user_highpage(struct page *to, struct page *from, unsigned long vaddr, struct vm_area_struct *vma) @@ -481,6 +507,12 @@ static inline int copy_mc_highpage(struct page *to, st= ruct page *from) copy_highpage(to, from); return 0; } + +static inline int copy_mc_highpage_nt(struct page *to, struct page *from) +{ + copy_highpage(to, from); + return 0; +} #endif =20 static inline void memcpy_page(struct page *dst_page, size_t dst_off, diff --git a/include/linux/mm.h b/include/linux/mm.h index 5be3d8a8f806..d07ce478582d 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -1644,6 +1644,7 @@ void __folio_put(struct folio *folio); void split_page(struct page *page, unsigned int order); void folio_copy(struct folio *dst, struct folio *src); int folio_mc_copy(struct folio *dst, struct folio *src); +int folio_mc_copy_nt(struct folio *dst, struct folio *src); =20 unsigned long nr_free_buffer_pages(void); =20 diff --git a/mm/util.c b/mm/util.c index b05ab6f97e11..e09e9b5f8eee 100644 --- a/mm/util.c +++ b/mm/util.c @@ -749,6 +749,23 @@ int folio_mc_copy(struct folio *dst, struct folio *src) } EXPORT_SYMBOL(folio_mc_copy); =20 +int folio_mc_copy_nt(struct folio *dst, struct folio *src) +{ + long nr =3D folio_nr_pages(src); + long i =3D 0; + + for (;;) { + if (copy_mc_highpage_nt(folio_page(dst, i), folio_page(src, i))) + return -EHWPOISON; + if (++i =3D=3D nr) + break; + cond_resched(); + } + + return 0; +} +EXPORT_SYMBOL(folio_mc_copy_nt); + int sysctl_overcommit_memory __read_mostly =3D OVERCOMMIT_GUESS; static int sysctl_overcommit_ratio __read_mostly =3D 50; static unsigned long sysctl_overcommit_kbytes __read_mostly; --=20 2.43.0 From nobody Mon Jun 8 21:55:13 2026 Received: from mail-lf1-f53.google.com (mail-lf1-f53.google.com [209.85.167.53]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id EB16C3E2AC9 for ; Tue, 26 May 2026 11:37:45 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.167.53 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1779795482; cv=none; b=S5dQrdvRCJu9VWJl1YKHOYTBjdraSkZF+iXO02YPsim3cP88qGBypoAIiHU5Ta2MDi3qNk+TmFZx61KfavrxcGO/RgyqHjVpq7kl+oL4GUKtCySzc9tG/AP2mh9aJH6CaNaZfRxAFP78jMf7ar8D3MRNshsHMqpMg5IjLbh17ao= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1779795482; c=relaxed/simple; bh=YkONpgjarkoyetvAWGsRSwGxKORs8lGBbP3iuK2k+jI=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=YVg62XIfUHGHqDGQrfL3ELHrjFlYYweYQkYSjFd0n2wEQecfEJbH2NFImbkCcqhOzTxvM0AlF38CMZr3l5JECZ1bKEkDXzSDXWHlMxQTjvMbjCwYJj7YCwL42U/I6oNL+1dZtZOjZanLjvKhe/bjpw2yAFiEoOMP7OMOAdG9Pkc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=N2ZHD5Bt; arc=none smtp.client-ip=209.85.167.53 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="N2ZHD5Bt" Received: by mail-lf1-f53.google.com with SMTP id 2adb3069b0e04-5a8fbe18b1dso16702454e87.2 for ; Tue, 26 May 2026 04:37:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1779795460; x=1780400260; darn=vger.kernel.org; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :mime-version:subject:date:from:from:to:cc:subject:date:message-id :reply-to; bh=4YrtHMEYPI1tjDQiF/FNeUbCwB0JCeO1sQzq3zX3f00=; b=N2ZHD5Bt5an0mStRTPeVWIQaVmz6beWBYuD4n8NsNunqLk2ddmXIjc8Oa6pDhulG1O Isa0TtFuLFxfEtZzJWR9Uv7IhyIjdz9kGgpKRsuoISwQ9oXc/9k4zzQW/X56LdvUmejx obzfuHs82WGgJMksWEqG2tiD1L2g/ZoioAOa7IbLkoZ60K48GZOGpGOyvZGd5GRi2/4S NENjEAzmm+N9LIYuQHnkmYqi8X/RWD1xGCuuT+i4ZlY/HZWqRJklAZo+KDZ3hoWn2tMc sMEI18PB9pqV2TRBnnNtH84vEX/5ph7ycMGpNUPrgHRUcragwU/dU0VzV0DeIzMDLYs5 y7SA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1779795460; x=1780400260; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :mime-version:subject:date:from:x-gm-gg:x-gm-message-state:from:to :cc:subject:date:message-id:reply-to; bh=4YrtHMEYPI1tjDQiF/FNeUbCwB0JCeO1sQzq3zX3f00=; b=FA4I6XlUBU+wL1idRn7b//sj0zv+gE92tcBpcLn7xPes6Sxx3nJCj5KLLUOnegLR8I TtcfYYbbU7L7Vc7+d/Sm6IeuL6bm3doaRYZNyu9GNc7lR60tE7IkplPoKdjQ/Bo6h9Kb 93KX7fHczDTs9dD+R1vOFA7nr/meUHAihL1aTCUiYdq7QfgIzlIuQMXDyiEk4qsbUC4W j4h2GPyOfyMUkWt4XFNq9o3265Kte8WOLFcY00SJAHFbCCfdyBTTCAWgAjD6XM67GIK3 Hzotgh+Bp/7pzXLfF8j2YYoRCVZtAUdWwjkUXYGOglvhhlgQoimRHgrZtQ5zSN+axAy8 pqdw== X-Forwarded-Encrypted: i=1; AFNElJ+K7QZBXW0oni7QI77NRPkgCtkiCVNYFLfpHs9W3SVDPyHDulyPeIL8DeE+yqG5Zkb+qwj5Tde1rnhSoks=@vger.kernel.org X-Gm-Message-State: AOJu0YyHICmxyXG3NKb+LJz5/wBqJlE0GrBqhMrKzA7aiiXw8wQTOQQe ZLD5rtg1aClHHTffNcHw3UC3cQ/gPVx0sJjlMfmWDHgFNOyX6JYECqeu X-Gm-Gg: Acq92OGQrx+sCnpFWMkZv4VtE/nPM1zQD2YHQwIbwIGs3PvzACJ8nJm59UzgfYTT44w sI00hZIosdS4nCrRbB3nZwe4Cb9sB6RJ3dXtes+1pRPbsMudxeWEoVqCB91PTnZIGz1NJ7t6LdE 3KV9p4FeNXGZ4lewMirZyKvjY0G6FOO+OvbtEA6Q0vlZqlqSuJB124zWQix8NbCNbH3fZKiDacz lIWpaBI2+w5YoHo5rcpta0raOFJSebgpdFevBHWz6V39wEP4yWxFTiLe4dFhngfVNlQg5QoLlJm 1aJxP06VaITswmaZhZbBzccJMbB6CGnvrz056vK1AboZ7sEnjoxsru46Z36PW66f19ja3G0CTdD fgOrB/bBke1HqLz9QQV/8YLIF7E9KRhvvI5f82uykJhTGVSBPu6Spgkg+YOt6RTsJVDRSoBp7Sz 57qN+ZbOLWskrUHuVMM8/qS8v3Zua65tFT+Syvj74dF+/Ucid/SBQEQNO2b4ktBf9e3dAL X-Received: by 2002:a19:f010:0:b0:5a8:8f72:de76 with SMTP id 2adb3069b0e04-5aa3233bde4mr4472023e87.19.1779795459278; Tue, 26 May 2026 04:37:39 -0700 (PDT) Received: from [127.0.1.1] (h-62-63-197-109.NA.cust.bahnhof.se. [62.63.197.109]) by smtp.gmail.com with ESMTPSA id 2adb3069b0e04-5aa46330d9asm376332e87.59.2026.05.26.04.37.37 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 26 May 2026 04:37:38 -0700 (PDT) From: Yiannis Nikolakopoulos Date: Tue, 26 May 2026 13:37:03 +0200 Subject: [PATCH RFC 2/3] mm: new migrate_mode flag for async using non-temporal stores Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20260526-rfc-nt-demote-v1-2-eb9c9422daef@zptcorp.com> References: <20260526-rfc-nt-demote-v1-0-eb9c9422daef@zptcorp.com> In-Reply-To: <20260526-rfc-nt-demote-v1-0-eb9c9422daef@zptcorp.com> To: Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , x86@kernel.org, "H. Peter Anvin" , Andrew Morton , David Hildenbrand , Lorenzo Stoakes , "Liam R. Howlett" , Vlastimil Babka , Mike Rapoport , Suren Baghdasaryan , Michal Hocko , Trond Myklebust , Anna Schumaker , Zi Yan , Matthew Brost , Joshua Hahn , Rakie Kim , Byungchul Park , Gregory Price , Ying Huang , Alistair Popple , Steven Rostedt , Masami Hiramatsu , Mathieu Desnoyers , Brendan Jackman , Johannes Weiner Cc: David Rientjes , Davidlohr Bueso , Fan Ni , Frank van der Linden , Jonathan Cameron , Raghavendra K T , "Rao, Bharata Bhasker" , SeongJae Park , Wei Xu , Xuezheng Chu , Yiannis Nikolakopoulos , dimitrios@palyvos.net, Ryan Roberts , linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-nfs@vger.kernel.org, linux-trace-kernel@vger.kernel.org, Yiannis Nikolakopoulos , Alirad Malek X-Mailer: b4 0.15.2 From: Alirad Malek In preparation for the following patch, add a new migrate_mode which is still async but will use non-temporal stores. Add a helper function that checks for both async modes and replace all plain checks of MIGRATE_ASYNC. Signed-off-by: Alirad Malek Co-developed-by: Yiannis Nikolakopoulos Signed-off-by: Yiannis Nikolakopoulos --- fs/nfs/write.c | 2 +- include/linux/migrate_mode.h | 9 +++++++++ include/trace/events/migrate.h | 1 + mm/compaction.c | 18 +++++++++--------- mm/migrate.c | 12 ++++++------ 5 files changed, 26 insertions(+), 16 deletions(-) diff --git a/fs/nfs/write.c b/fs/nfs/write.c index 1ed4b3590b1a..beae4441e080 100644 --- a/fs/nfs/write.c +++ b/fs/nfs/write.c @@ -2119,7 +2119,7 @@ int nfs_migrate_folio(struct address_space *mapping, = struct folio *dst, } =20 if (folio_test_private_2(src)) { /* [DEPRECATED] */ - if (mode =3D=3D MIGRATE_ASYNC) + if (migrate_mode_is_async(mode)) return -EBUSY; folio_wait_private_2(src); } diff --git a/include/linux/migrate_mode.h b/include/linux/migrate_mode.h index 265c4328b36a..f7186e705b48 100644 --- a/include/linux/migrate_mode.h +++ b/include/linux/migrate_mode.h @@ -3,6 +3,8 @@ #define MIGRATE_MODE_H_INCLUDED /* * MIGRATE_ASYNC means never block + * MIGRATE_ASYNC_NON_TEMPORAL_STORES means never block and use non-temporal + * stores if supported by the architecture * MIGRATE_SYNC_LIGHT in the current implementation means to allow blocking * on most operations but not ->writepage as the potential stall time * is too significant @@ -10,10 +12,17 @@ */ enum migrate_mode { MIGRATE_ASYNC, + MIGRATE_ASYNC_NON_TEMPORAL_STORES, MIGRATE_SYNC_LIGHT, MIGRATE_SYNC, }; =20 +static inline bool migrate_mode_is_async(enum migrate_mode mode) +{ + return mode =3D=3D MIGRATE_ASYNC || + mode =3D=3D MIGRATE_ASYNC_NON_TEMPORAL_STORES; +} + enum migrate_reason { MR_COMPACTION, MR_MEMORY_FAILURE, diff --git a/include/trace/events/migrate.h b/include/trace/events/migrate.h index cd01dd7b3640..e493207a3f46 100644 --- a/include/trace/events/migrate.h +++ b/include/trace/events/migrate.h @@ -9,6 +9,7 @@ =20 #define MIGRATE_MODE \ EM( MIGRATE_ASYNC, "MIGRATE_ASYNC") \ + EM(MIGRATE_ASYNC_NON_TEMPORAL_STORES, "MIGRATE_ASYNC_NON_TEMPORAL_STORES= ") \ EM( MIGRATE_SYNC_LIGHT, "MIGRATE_SYNC_LIGHT") \ EMe(MIGRATE_SYNC, "MIGRATE_SYNC") =20 diff --git a/mm/compaction.c b/mm/compaction.c index 1e8f8eca318c..cd26781b7376 100644 --- a/mm/compaction.c +++ b/mm/compaction.c @@ -444,7 +444,7 @@ static void update_cached_migrate(struct compact_contro= l *cc, unsigned long pfn) /* Update where async and sync compaction should restart */ if (pfn > zone->compact_cached_migrate_pfn[0]) zone->compact_cached_migrate_pfn[0] =3D pfn; - if (cc->mode !=3D MIGRATE_ASYNC && + if (!migrate_mode_is_async(cc->mode) && pfn > zone->compact_cached_migrate_pfn[1]) zone->compact_cached_migrate_pfn[1] =3D pfn; } @@ -507,7 +507,7 @@ static bool compact_lock_irqsave(spinlock_t *lock, unsi= gned long *flags, __acquires(lock) { /* Track if the lock is contended in async mode */ - if (cc->mode =3D=3D MIGRATE_ASYNC && !cc->contended) { + if (migrate_mode_is_async(cc->mode) && !cc->contended) { if (spin_trylock_irqsave(lock, *flags)) return true; =20 @@ -864,7 +864,7 @@ isolate_migratepages_block(struct compact_control *cc, = unsigned long low_pfn, return -EAGAIN; =20 /* async migration should just abort */ - if (cc->mode =3D=3D MIGRATE_ASYNC) + if (migrate_mode_is_async(cc->mode)) return -EAGAIN; =20 reclaim_throttle(pgdat, VMSCAN_THROTTLE_ISOLATED); @@ -875,7 +875,7 @@ isolate_migratepages_block(struct compact_control *cc, = unsigned long low_pfn, =20 cond_resched(); =20 - if (cc->direct_compaction && (cc->mode =3D=3D MIGRATE_ASYNC)) { + if (cc->direct_compaction && migrate_mode_is_async(cc->mode)) { skip_on_failure =3D true; next_skip_pfn =3D block_end_pfn(low_pfn, cc->order); } @@ -1364,7 +1364,7 @@ static bool suitable_migration_source(struct compact_= control *cc, if (pageblock_skip_persistent(page)) return false; =20 - if ((cc->mode !=3D MIGRATE_ASYNC) || !cc->direct_compaction) + if (!migrate_mode_is_async(cc->mode) || !cc->direct_compaction) return true; =20 block_mt =3D get_pageblock_migratetype(page); @@ -1465,7 +1465,7 @@ fast_isolate_around(struct compact_control *cc, unsig= ned long pfn) return; =20 /* Minimise scanning during async compaction */ - if (cc->direct_compaction && cc->mode =3D=3D MIGRATE_ASYNC) + if (cc->direct_compaction && migrate_mode_is_async(cc->mode)) return; =20 /* Pageblock boundaries */ @@ -1705,7 +1705,7 @@ static void isolate_freepages(struct compact_control = *cc) block_end_pfn =3D min(block_start_pfn + pageblock_nr_pages, zone_end_pfn(zone)); low_pfn =3D pageblock_end_pfn(cc->migrate_pfn); - stride =3D cc->mode =3D=3D MIGRATE_ASYNC ? COMPACT_CLUSTER_MAX : 1; + stride =3D migrate_mode_is_async(cc->mode) ? COMPACT_CLUSTER_MAX : 1; =20 /* * Isolate free pages until enough are available to migrate the @@ -2514,7 +2514,7 @@ compact_zone(struct compact_control *cc, struct captu= re_control *capc) unsigned long start_pfn =3D cc->zone->zone_start_pfn; unsigned long end_pfn =3D zone_end_pfn(cc->zone); unsigned long last_migrated_pfn; - const bool sync =3D cc->mode !=3D MIGRATE_ASYNC; + const bool sync =3D !migrate_mode_is_async(cc->mode); bool update_cached; unsigned int nr_succeeded =3D 0, nr_migratepages; int order; @@ -2537,7 +2537,7 @@ compact_zone(struct compact_control *cc, struct captu= re_control *capc) ret =3D compaction_suit_allocation_order(cc->zone, cc->order, cc->highest_zoneidx, cc->alloc_flags, - cc->mode =3D=3D MIGRATE_ASYNC, + migrate_mode_is_async(cc->mode), !cc->direct_compaction); if (ret !=3D COMPACT_CONTINUE) return ret; diff --git a/mm/migrate.c b/mm/migrate.c index 2c3d489ecf51..ff6cf50e7b0b 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -907,7 +907,7 @@ static bool buffer_migrate_lock_buffers(struct buffer_h= ead *head, =20 do { if (!trylock_buffer(bh)) { - if (mode =3D=3D MIGRATE_ASYNC) + if (migrate_mode_is_async(mode)) goto unlock; if (mode =3D=3D MIGRATE_SYNC_LIGHT && !buffer_uptodate(bh)) goto unlock; @@ -1220,7 +1220,7 @@ static int migrate_folio_unmap(new_folio_t get_new_fo= lio, dst->private =3D NULL; =20 if (!folio_trylock(src)) { - if (mode =3D=3D MIGRATE_ASYNC) + if (migrate_mode_is_async(mode)) goto out; =20 /* @@ -1325,7 +1325,7 @@ static int migrate_folio_unmap(new_folio_t get_new_fo= lio, /* Establish migration ptes */ VM_BUG_ON_FOLIO(folio_test_anon(src) && !folio_test_ksm(src) && !anon_vma, src); - try_to_migrate(src, mode =3D=3D MIGRATE_ASYNC ? TTU_BATCH_FLUSH : 0); + try_to_migrate(src, migrate_mode_is_async(mode) ? TTU_BATCH_FLUSH : 0); old_page_state |=3D PAGE_WAS_MAPPED; } =20 @@ -1565,7 +1565,7 @@ static inline int try_split_folio(struct folio *folio= , struct list_head *split_f { int rc; =20 - if (mode =3D=3D MIGRATE_ASYNC) { + if (migrate_mode_is_async(mode)) { if (!folio_trylock(folio)) return -EAGAIN; } else { @@ -1799,7 +1799,7 @@ static int migrate_pages_batch(struct list_head *from, LIST_HEAD(dst_folios); bool nosplit =3D (reason =3D=3D MR_NUMA_MISPLACED); =20 - VM_WARN_ON_ONCE(mode !=3D MIGRATE_ASYNC && + VM_WARN_ON_ONCE(!migrate_mode_is_async(mode) && !list_empty(from) && !list_is_singular(from)); =20 for (pass =3D 0; pass < nr_pass && retry; pass++) { @@ -2107,7 +2107,7 @@ int migrate_pages(struct list_head *from, new_folio_t= get_new_folio, list_cut_before(&folios, from, &folio2->lru); else list_splice_init(from, &folios); - if (mode =3D=3D MIGRATE_ASYNC) + if (migrate_mode_is_async(mode)) rc =3D migrate_pages_batch(&folios, get_new_folio, put_new_folio, private, mode, reason, &ret_folios, &split_folios, &stats, --=20 2.43.0 From nobody Mon Jun 8 21:55:13 2026 Received: from mail-lf1-f49.google.com (mail-lf1-f49.google.com [209.85.167.49]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 533C83E3144 for ; Tue, 26 May 2026 11:37:46 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.167.49 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1779795476; cv=none; b=LmMydjtTL2Sti/DuUk/jf+EUsNGXV9B8PXpvjs1v/PQf5R0knqTVtDaUM1KT+CBK/NMCmnnyE79WEj3n1XAqDioT5jYn0m9GAIYO0cbj5F/5bWCyHR8Pu0I+U7zPtVEyN8PVT4q8FB12aclY26F5ISN6qQTUI8cjKHybTPY58LQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1779795476; c=relaxed/simple; bh=SwXzV8tK9BQ8QjgrG6CC7I98NvDSuaRSrOJYRNh4rFA=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=L0lNWqVPR1Dgyt6n57KSbyk6Mdom/aTdAEJrArRYPR94AmPVgS/8cC1xllfSelIibO0+W/hXrAgFkLkhjuTPpE5V3cGa0X9qy3qIxYHUTnUfTiXeX4UuidysmpuYciqwPEUSoqaVWi6wC9jVTTON8naNiFTCgH1eyQ0ZNdgCOOs= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=cPrQWuk7; arc=none smtp.client-ip=209.85.167.49 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="cPrQWuk7" Received: by mail-lf1-f49.google.com with SMTP id 2adb3069b0e04-5aa1b2327c8so12061765e87.0 for ; Tue, 26 May 2026 04:37:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1779795462; x=1780400262; darn=vger.kernel.org; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :mime-version:subject:date:from:from:to:cc:subject:date:message-id :reply-to; bh=FLeHkQsXAybNTL4Q4w5mxgSBDAtaR04/XoGzkQTY4uA=; b=cPrQWuk7sFJUVnkI4J/4xCExY/hk6221tlhx/fRQOGzXvYnbD8jjnmtRYkFbnZ/roK waeGXT5OZRhbCqSw5xnlFc0eASPxHOX7h50ogsv+uFqNtnKvGUUExbfF1La+W7rLN8ww Ldf1hFmkvdbphZQXxbd+1PFRGn37BZFi7x0doEwCnb+cJTP3EYWo/XgCk0/516EVhi36 al3XS5e55xVWGBm4EdvarjPTK7n49WFmqJpg6ExdgNoeeQlySQkrNgXcQ/7MiCLZRjvb 2TuEtQLcFmq6Tu9cGnsBBS55uskggQ3xq6pnvuIofrvHhlQTTN0BXOt++oBwCvRTzEQ5 xxGg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1779795462; x=1780400262; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :mime-version:subject:date:from:x-gm-gg:x-gm-message-state:from:to :cc:subject:date:message-id:reply-to; bh=FLeHkQsXAybNTL4Q4w5mxgSBDAtaR04/XoGzkQTY4uA=; b=tSt9/FLT9vvwE/h8q30x1zcbCdLalRgBDdarVGQdF5/h+Hy0YNnrWuXXFeDoeiqG+A Pot4ifiLABNyHaeSuZum2cP3ZqGaXnlhXQdp5iTIHBuTGQiBmde0IHMolZCXffJG98vR itO1R+J9QZEJ9JUPvekIB9b1BWQk9oKjqX6SUmIzUXyEsaJqUrfl7B/64MQ+eJiRDZmd uKqqBnl+4sYz9pPkdmxraSANxRVaLUVZkDHcEu/AYrfQUoG3xpeqeZEfFnuQRLM66be4 /XT2fE+AfnbDbPQd9tkBd+C7fjHDaReRQLcUsX7KnetWFD5Dj3tRZDKo3uPD/F0EW+OT e1+g== X-Forwarded-Encrypted: i=1; AFNElJ8jpV8EfTRD0A+llR8Sq5FXEVrJF5Q4RKTdqxAnnqkLoq8q1h35KE4POvbxV6nCFo4Im0K0SOf+e+NrShc=@vger.kernel.org X-Gm-Message-State: AOJu0YwTQ865mRa7+jb8Xys8D5pz92dGiLahE27Fp7aeQBUHUSFNVRLn x59nFj0XAUfsTrMosOA0u7p6OTC9oFb5XchqkZyIzuCdvXaVlyQ2XjlJ X-Gm-Gg: Acq92OEW2cAYHjywMou7muysXGI2XMvIUEIJf7V2kj0lC5tzPwX5VipEXN5xzErf56M L9YlHmO66JFRV78I7LKNs37ZsxuX0LDLpYmAH55Me17NNNXb1ZR+tp13PMIVX0P1xwWi7zDPFN5 h/fkK6ZDVdF0mlX88PbJLW2mMkjRv3pR893rCohD52Qon30A9viuMG2lWDrmzfFT31m6C2hZoms SI3S+NRrMBRiXwwLqU6bhVdsvOx5b4cmxlG8A1OI6U3wKwt6XIsBmYWfGKHrImIgPyPvzCz0gHc OQ3ax+uIlrDt0vf+71ZlVI0kBejdmnLKNhD/uWec5+v7jLMthcutfbHy4QRKaqM/066L5ay2rX+ C171wIXU1+BjmeBRCjAA9gTxndTU3bSt2/Bnc1aMMqtIKju1uLMGEQT8ldjJIOCFtYM9MfCgfjr AF/vck8xcxUTMJ3YJ+ShZBMSXjkQFJnsJUnYeDLvb0bEnPmZ7+FHNMPCrBymvbUcUZ9k+j X-Received: by 2002:a05:6512:3b21:b0:5a4:1a2:1d39 with SMTP id 2adb3069b0e04-5aa323143camr4952108e87.6.1779795461414; Tue, 26 May 2026 04:37:41 -0700 (PDT) Received: from [127.0.1.1] (h-62-63-197-109.NA.cust.bahnhof.se. [62.63.197.109]) by smtp.gmail.com with ESMTPSA id 2adb3069b0e04-5aa46330d9asm376332e87.59.2026.05.26.04.37.39 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 26 May 2026 04:37:41 -0700 (PDT) From: Yiannis Nikolakopoulos Date: Tue, 26 May 2026 13:37:04 +0200 Subject: [PATCH RFC 3/3] mm: use non-temporal stores for demotion Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20260526-rfc-nt-demote-v1-3-eb9c9422daef@zptcorp.com> References: <20260526-rfc-nt-demote-v1-0-eb9c9422daef@zptcorp.com> In-Reply-To: <20260526-rfc-nt-demote-v1-0-eb9c9422daef@zptcorp.com> To: Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , x86@kernel.org, "H. Peter Anvin" , Andrew Morton , David Hildenbrand , Lorenzo Stoakes , "Liam R. Howlett" , Vlastimil Babka , Mike Rapoport , Suren Baghdasaryan , Michal Hocko , Trond Myklebust , Anna Schumaker , Zi Yan , Matthew Brost , Joshua Hahn , Rakie Kim , Byungchul Park , Gregory Price , Ying Huang , Alistair Popple , Steven Rostedt , Masami Hiramatsu , Mathieu Desnoyers , Brendan Jackman , Johannes Weiner Cc: David Rientjes , Davidlohr Bueso , Fan Ni , Frank van der Linden , Jonathan Cameron , Raghavendra K T , "Rao, Bharata Bhasker" , SeongJae Park , Wei Xu , Xuezheng Chu , Yiannis Nikolakopoulos , dimitrios@palyvos.net, Ryan Roberts , linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-nfs@vger.kernel.org, linux-trace-kernel@vger.kernel.org, Yiannis Nikolakopoulos , Alirad Malek X-Mailer: b4 0.15.2 From: Alirad Malek Memory demoted to a lower tier is assumed to be cold and most likely out of the CPU's last level cache. Additionally, in certain demotion targets (e.g. CXL devices with compressed memory) the bandwidth can be negatively impacted by the eviction patterns of the last level cache when standard memcpy is used. When the feature is enabled, use the MIGRATE_ASYNC_NON_TEMPORAL_STORES flag in demotions to trigger the folio copy path using non-temporal stores. Signed-off-by: Alirad Malek Co-developed-by: Yiannis Nikolakopoulos Signed-off-by: Yiannis Nikolakopoulos --- mm/Kconfig | 8 ++++++++ mm/migrate.c | 9 ++++++++- 2 files changed, 16 insertions(+), 1 deletion(-) diff --git a/mm/Kconfig b/mm/Kconfig index ebd8ea353687..4b7a75b57f6e 100644 --- a/mm/Kconfig +++ b/mm/Kconfig @@ -645,6 +645,14 @@ config MIGRATION pages as migration can relocate pages to satisfy a huge page allocation instead of reclaiming. =20 +config DEMOTION_WITH_NON_TEMPORAL_STORES + bool "Use non-temporal stores for demotion" + default n + depends on MIGRATION + help + Enable non-temporal stores when migrating pages due to demotion. + If disabled, demotion uses regular migration copy paths. + config DEVICE_MIGRATION def_bool MIGRATION && ZONE_DEVICE =20 diff --git a/mm/migrate.c b/mm/migrate.c index ff6cf50e7b0b..368d40dc8772 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -862,7 +862,10 @@ static int __migrate_folio(struct address_space *mappi= ng, struct folio *dst, if (folio_ref_count(src) !=3D expected_count) return -EAGAIN; =20 - rc =3D folio_mc_copy(dst, src); + if (mode =3D=3D MIGRATE_ASYNC_NON_TEMPORAL_STORES) + rc =3D folio_mc_copy_nt(dst, src); + else + rc =3D folio_mc_copy(dst, src); if (unlikely(rc)) return rc; =20 @@ -2081,6 +2084,10 @@ int migrate_pages(struct list_head *from, new_folio_= t get_new_folio, LIST_HEAD(split_folios); struct migrate_pages_stats stats; =20 + if (IS_ENABLED(CONFIG_DEMOTION_WITH_NON_TEMPORAL_STORES) && + reason =3D=3D MR_DEMOTION && mode =3D=3D MIGRATE_ASYNC) + mode =3D MIGRATE_ASYNC_NON_TEMPORAL_STORES; + trace_mm_migrate_pages_start(mode, reason); =20 memset(&stats, 0, sizeof(stats)); --=20 2.43.0