From nobody Fri Oct 31 09:40:19 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) client-ip=192.237.175.120; envelope-from=xen-devel-bounces@lists.xenproject.org; helo=lists.xenproject.org; Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=pass(p=quarantine dis=none) header.from=suse.com ARC-Seal: i=1; a=rsa-sha256; t=1749119346; cv=none; d=zohomail.com; s=zohoarc; b=RDtpiG695JpUT0MctfaZx5Im26lKKxE9f50yWjwIRslkwMIF36q7QXCvzw3Bfc4M87mSixqdik9NCmt8zdBfYrhYDz8kK9OHLLa+tRclzdNgPNGHSn8AuX81kR5juurZgXoadAmq4RYqiwpSQ8uRwGltE4ezqLy49urHgfVnQLA= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1749119346; h=Content-Type:Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=4eEicFsdYkA5hJ8hFbKaqmBk1yERZphW9t6IYOWNxpY=; b=JTVSX9KQuI5b/1Ui6s3Y9QbMGSrVAT8p651mp+rHQ7N5x4A/LM3i0NoblSI/kbH/tIadPe0Jv45cbXboJDl87lo+1dDqOpD/xSJdZaV5lIHcJO1DZA4+LhbMus3H+m9rbw/M7e4C1v70eDAuMSzx/akX4JMMXmbCw1a9hgCU+pY= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=pass header.from= (p=quarantine dis=none) Return-Path: Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) by mx.zohomail.com with SMTPS id 1749119346162452.50260653591386; Thu, 5 Jun 2025 03:29:06 -0700 (PDT) Received: from list by lists.xenproject.org with outflank-mailman.1006582.1385798 (Exim 4.92) (envelope-from ) id 1uN7q6-0006wM-07; Thu, 05 Jun 2025 10:28:50 +0000 Received: by outflank-mailman (output) from mailman id 1006582.1385798; Thu, 05 Jun 2025 10:28:49 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1uN7q5-0006wF-SR; Thu, 05 Jun 2025 10:28:49 +0000 Received: by outflank-mailman (input) for mailman id 1006582; Thu, 05 Jun 2025 10:28:49 +0000 Received: from se1-gles-sth1-in.inumbo.com ([159.253.27.254] helo=se1-gles-sth1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1uN7q4-0006Ay-TC for xen-devel@lists.xenproject.org; Thu, 05 Jun 2025 10:28:49 +0000 Received: from mail-wr1-x430.google.com (mail-wr1-x430.google.com [2a00:1450:4864:20::430]) by se1-gles-sth1.inumbo.com (Halon) with ESMTPS id dd8a412d-41f7-11f0-a300-13f23c93f187; Thu, 05 Jun 2025 12:28:48 +0200 (CEST) Received: by mail-wr1-x430.google.com with SMTP id ffacd0b85a97d-3a35c894313so678430f8f.2 for ; Thu, 05 Jun 2025 03:28:47 -0700 (PDT) Received: from [10.156.60.236] (ip-037-024-206-209.um08.pools.vodafone-ip.de. [37.24.206.209]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-313319d45dfsm971765a91.32.2025.06.05.03.28.40 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Thu, 05 Jun 2025 03:28:46 -0700 (PDT) X-Outflank-Mailman: Message body and most headers restored to incoming version X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: dd8a412d-41f7-11f0-a300-13f23c93f187 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.com; s=google; t=1749119327; x=1749724127; darn=lists.xenproject.org; h=content-transfer-encoding:in-reply-to:autocrypt:content-language :references:cc:to:from:subject:user-agent:mime-version:date :message-id:from:to:cc:subject:date:message-id:reply-to; bh=4eEicFsdYkA5hJ8hFbKaqmBk1yERZphW9t6IYOWNxpY=; b=V9qzNbxkTnTPZSrk2mJzfGUX6VUlXCvExlosxw6e8L1oPsI5RijIhA2PxM3j9thEIT r4Lsnsbe7YcekE3zOfwbGzWDDU9sZ7i0UHhNya39+CDBdzqMcFlsRHb2oGw7JrE964nr sToHE7h4w1wPnWCG+ISKaYOmezmkjVKJTPF4A2PP6AMVwR6njY8Gb5XLtFcC9mOe4CXd xdcYJOSumhkChMeXTd+JgEZeFx2iqVew5Ldv4dp5dOzYb9Q2rhIEyyT4C6g1jNxqjRk/ G7b+PCrV2sxTKMo9u43vcyxVU02Rck4q72gYOyrD6dHogzbtnu/dyUiWF6s/b5O0Jkdr UoIQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1749119327; x=1749724127; h=content-transfer-encoding:in-reply-to:autocrypt:content-language :references:cc:to:from:subject:user-agent:mime-version:date :message-id:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=4eEicFsdYkA5hJ8hFbKaqmBk1yERZphW9t6IYOWNxpY=; b=MKS+nvJoARKN6P8+kL/iNtxs5hdG5d1uT27QV2PyTjioWzoG8vMQaVqBayOx9jgOcL DjLfm+EtSfqvUxIjm2zcFTZM+0+E9EnxgS8sr6xZlafH0NETrr/scufLGE1Jr3RxVrhY QAAP/6CTLb8srBMjC9rJn1FOWk2PP9CKi/CMPRRYbGBJ/K80AqNnIUdsn2JuED5XPOWl R0JBGuP2B/gGGCW3AQVvtD2vUvc8aII3Uv2p60vVwQpDgSgGLu4F90Ue/t+Yfp5Zous2 LIEpsXd5ZRpPt/QuV1R4A8nCeWuetAjLmsE8sOgF8s5vebocZGFdrePEXJOt1jJgJXdp JqOg== X-Gm-Message-State: AOJu0Yw3x5KvlsBLY/wrH/I7QhoY4z0OohuYgdd9Gb9L1norqhRi3jlI /XWOaEAyscViaJLTQznYlyLE7Dcpd8JnK4Ea7NeR91DgBuoWVQKbBbsSNE3kW4zD0lDiA+0ZuA0 1NuA= X-Gm-Gg: ASbGnct2Da4fxVxVBShgXcCzIdwFP5zX+2Va+PA2u0yaE6bSuTNu1/goakqkuHuEhwS c5lj2ClxfwQQqkdboG8ZPI7QoOIzBjn17gNhR6uyJ0z4w2V+lfrIR6S0gvhFNTAW+DC6sFc+aBc boS2tWt/4o6a0utQwVfYqmQNu7ArJby340EFaw4efZfYxabefuAqQ92HfkHYXyElGmhhFOwkHuZ Gl0R8eHwgR8cRQgGxEVJdtv9s3epXqZ5p4jkJW76n8sj6Hx2vzReRtP4418EGHrK5hoUnhi/xLF YgMvdOCd2efgBWz/EprynzwfcTI9TzhdfkZou2toWjRzUJBVMhImGzfXioGYYME83vYlfwt5BI+ m8wRgvr19bzxw9/+DxWmoBMvaBYNJ3/K/O47F9L10HmrRlkVYtdfHmODixQ== X-Google-Smtp-Source: AGHT+IFF2cBgmthnjaPY0d7ZcKwjaIvijwnZ6Vo0g+oi5WwQuWMEzpGV2GKkvMxJO2iG7gtswr52zg== X-Received: by 2002:a05:6000:3106:b0:3a4:f70d:a65e with SMTP id ffacd0b85a97d-3a51d95a4f1mr4785868f8f.37.1749119327240; Thu, 05 Jun 2025 03:28:47 -0700 (PDT) Message-ID: Date: Thu, 5 Jun 2025 12:28:36 +0200 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: [PATCH v5 6/6] mm: allow page scrubbing routine(s) to be arch controlled From: Jan Beulich To: "xen-devel@lists.xenproject.org" Cc: Andrew Cooper , Julien Grall , Stefano Stabellini , Anthony PERARD , Michal Orzel , =?UTF-8?Q?Roger_Pau_Monn=C3=A9?= , Bertrand Marquis , Volodymyr Babchuk , Oleksii Kurochko , Shawn Anastasio References: <73481cbf-337f-4e85-81d2-3487366cd822@suse.com> Content-Language: en-US Autocrypt: addr=jbeulich@suse.com; keydata= xsDiBFk3nEQRBADAEaSw6zC/EJkiwGPXbWtPxl2xCdSoeepS07jW8UgcHNurfHvUzogEq5xk hu507c3BarVjyWCJOylMNR98Yd8VqD9UfmX0Hb8/BrA+Hl6/DB/eqGptrf4BSRwcZQM32aZK 7Pj2XbGWIUrZrd70x1eAP9QE3P79Y2oLrsCgbZJfEwCgvz9JjGmQqQkRiTVzlZVCJYcyGGsD /0tbFCzD2h20ahe8rC1gbb3K3qk+LpBtvjBu1RY9drYk0NymiGbJWZgab6t1jM7sk2vuf0Py O9Hf9XBmK0uE9IgMaiCpc32XV9oASz6UJebwkX+zF2jG5I1BfnO9g7KlotcA/v5ClMjgo6Gl MDY4HxoSRu3i1cqqSDtVlt+AOVBJBACrZcnHAUSuCXBPy0jOlBhxPqRWv6ND4c9PH1xjQ3NP nxJuMBS8rnNg22uyfAgmBKNLpLgAGVRMZGaGoJObGf72s6TeIqKJo/LtggAS9qAUiuKVnygo 3wjfkS9A3DRO+SpU7JqWdsveeIQyeyEJ/8PTowmSQLakF+3fote9ybzd880fSmFuIEJldWxp Y2ggPGpiZXVsaWNoQHN1c2UuY29tPsJgBBMRAgAgBQJZN5xEAhsDBgsJCAcDAgQVAggDBBYC AwECHgECF4AACgkQoDSui/t3IH4J+wCfQ5jHdEjCRHj23O/5ttg9r9OIruwAn3103WUITZee e7Sbg12UgcQ5lv7SzsFNBFk3nEQQCACCuTjCjFOUdi5Nm244F+78kLghRcin/awv+IrTcIWF hUpSs1Y91iQQ7KItirz5uwCPlwejSJDQJLIS+QtJHaXDXeV6NI0Uef1hP20+y8qydDiVkv6l IreXjTb7DvksRgJNvCkWtYnlS3mYvQ9NzS9PhyALWbXnH6sIJd2O9lKS1Mrfq+y0IXCP10eS FFGg+Av3IQeFatkJAyju0PPthyTqxSI4lZYuJVPknzgaeuJv/2NccrPvmeDg6Coe7ZIeQ8Yj t0ARxu2xytAkkLCel1Lz1WLmwLstV30g80nkgZf/wr+/BXJW/oIvRlonUkxv+IbBM3dX2OV8 AmRv1ySWPTP7AAMFB/9PQK/VtlNUJvg8GXj9ootzrteGfVZVVT4XBJkfwBcpC/XcPzldjv+3 HYudvpdNK3lLujXeA5fLOH+Z/G9WBc5pFVSMocI71I8bT8lIAzreg0WvkWg5V2WZsUMlnDL9 mpwIGFhlbM3gfDMs7MPMu8YQRFVdUvtSpaAs8OFfGQ0ia3LGZcjA6Ik2+xcqscEJzNH+qh8V m5jjp28yZgaqTaRbg3M/+MTbMpicpZuqF4rnB0AQD12/3BNWDR6bmh+EkYSMcEIpQmBM51qM EKYTQGybRCjpnKHGOxG0rfFY1085mBDZCH5Kx0cl0HVJuQKC+dV2ZY5AqjcKwAxpE75MLFkr wkkEGBECAAkFAlk3nEQCGwwACgkQoDSui/t3IH7nnwCfcJWUDUFKdCsBH/E5d+0ZnMQi+G0A nAuWpQkjM1ASeQwSHEeAWPgskBQL In-Reply-To: <73481cbf-337f-4e85-81d2-3487366cd822@suse.com> Content-Transfer-Encoding: quoted-printable X-ZohoMail-DKIM: pass (identity @suse.com) X-ZM-MESSAGEID: 1749119347913116600 Content-Type: text/plain; charset="utf-8" Especially when dealing with large amounts of memory, memset() may not be very efficient; this can be bad enough that even for debug builds a custom function is warranted. We additionally want to distinguish "hot" and "cold" cases (with, as initial heuristic, "hot" being for any allocations a domain does for itself, assuming that in all other cases the page wouldn't be accessed [again] soon). The goal is for accesses of "cold" pages to not disturb caches (albeit finding a good balance between this and the higher latency looks to be difficult). Keep the default fallback to clear_page_*() in common code; this may want to be revisited down the road. Signed-off-by: Jan Beulich Acked-by: Julien Grall --- v4: Re-base. v3: Re-base. v2: New. --- The choice between hot and cold in scrub_one_page()'s callers is certainly up for discussion / improvement. --- a/xen/arch/arm/include/asm/page.h +++ b/xen/arch/arm/include/asm/page.h @@ -144,6 +144,12 @@ extern size_t dcache_line_bytes; =20 #define copy_page(dp, sp) memcpy(dp, sp, PAGE_SIZE) =20 +#define clear_page_hot clear_page +#define clear_page_cold clear_page + +#define scrub_page_hot(page) memset(page, SCRUB_BYTE_PATTERN, PAGE_SIZE) +#define scrub_page_cold scrub_page_hot + static inline size_t read_dcache_line_bytes(void) { register_t ctr; --- a/xen/arch/ppc/include/asm/page.h +++ b/xen/arch/ppc/include/asm/page.h @@ -188,6 +188,12 @@ static inline void invalidate_icache(voi #define clear_page(page) memset(page, 0, PAGE_SIZE) #define copy_page(dp, sp) memcpy(dp, sp, PAGE_SIZE) =20 +#define clear_page_hot clear_page +#define clear_page_cold clear_page + +#define scrub_page_hot(page) memset(page, SCRUB_BYTE_PATTERN, PAGE_SIZE) +#define scrub_page_cold scrub_page_hot + /* TODO: Flush the dcache for an entire page. */ static inline void flush_page_to_ram(unsigned long mfn, bool sync_icache) { --- a/xen/arch/riscv/include/asm/page.h +++ b/xen/arch/riscv/include/asm/page.h @@ -198,6 +198,12 @@ static inline void invalidate_icache(voi #define clear_page(page) memset((void *)(page), 0, PAGE_SIZE) #define copy_page(dp, sp) memcpy(dp, sp, PAGE_SIZE) =20 +#define clear_page_hot clear_page +#define clear_page_cold clear_page + +#define scrub_page_hot(page) memset(page, SCRUB_BYTE_PATTERN, PAGE_SIZE) +#define scrub_page_cold scrub_page_hot + static inline void flush_page_to_ram(unsigned long mfn, bool sync_icache) { const void *v =3D map_domain_page(_mfn(mfn)); --- a/xen/arch/x86/Makefile +++ b/xen/arch/x86/Makefile @@ -58,6 +58,7 @@ obj-y +=3D pci.o obj-y +=3D physdev.o obj-$(CONFIG_COMPAT) +=3D x86_64/physdev.o obj-$(CONFIG_X86_PSR) +=3D psr.o +obj-bin-$(CONFIG_DEBUG) +=3D scrub_page.o obj-y +=3D setup.o obj-y +=3D shutdown.o obj-y +=3D smp.o --- a/xen/arch/x86/include/asm/page.h +++ b/xen/arch/x86/include/asm/page.h @@ -226,6 +226,11 @@ void copy_page_sse2(void *to, const void #define clear_page(_p) clear_page_cold(_p) #define copy_page(_t, _f) copy_page_sse2(_t, _f) =20 +#ifdef CONFIG_DEBUG +void scrub_page_hot(void *); +void scrub_page_cold(void *); +#endif + /* Convert between Xen-heap virtual addresses and machine addresses. */ #define __pa(x) (virt_to_maddr(x)) #define __va(x) (maddr_to_virt(x)) --- /dev/null +++ b/xen/arch/x86/scrub_page.S @@ -0,0 +1,39 @@ + .file __FILE__ + +#include +#include +#include + +FUNC(scrub_page_cold) + mov $PAGE_SIZE/32, %ecx + mov $SCRUB_PATTERN, %rax + +0: movnti %rax, (%rdi) + movnti %rax, 8(%rdi) + movnti %rax, 16(%rdi) + movnti %rax, 24(%rdi) + add $32, %rdi + sub $1, %ecx + jnz 0b + + sfence + ret +END(scrub_page_cold) + + .macro scrub_page_stosb + mov $PAGE_SIZE, %ecx + mov $SCRUB_BYTE_PATTERN, %eax + rep stosb + ret + .endm + + .macro scrub_page_stosq + mov $PAGE_SIZE/8, %ecx + mov $SCRUB_PATTERN, %rax + rep stosq + ret + .endm + +FUNC(scrub_page_hot) + ALTERNATIVE scrub_page_stosq, scrub_page_stosb, X86_FEATURE_ERMS +END(scrub_page_hot) --- a/xen/common/page_alloc.c +++ b/xen/common/page_alloc.c @@ -135,6 +135,7 @@ #include #include #include +#include #include #include #include @@ -779,27 +780,31 @@ static void page_list_add_scrub(struct p page_list_add(pg, &heap(node, zone, order)); } =20 -/* SCRUB_PATTERN needs to be a repeating series of bytes. */ -#ifndef NDEBUG -#define SCRUB_PATTERN 0xc2c2c2c2c2c2c2c2ULL -#else -#define SCRUB_PATTERN 0ULL +/* + * While in debug builds we want callers to avoid relying on allocations + * returning zeroed pages, for a production build, clear_page_*() is the + * fastest way to scrub. + */ +#ifndef CONFIG_DEBUG +# undef scrub_page_hot +# define scrub_page_hot clear_page_hot +# undef scrub_page_cold +# define scrub_page_cold clear_page_cold #endif -#define SCRUB_BYTE_PATTERN (SCRUB_PATTERN & 0xff) =20 -static void scrub_one_page(const struct page_info *pg) +static void scrub_one_page(const struct page_info *pg, bool cold) { + void *ptr; + if ( unlikely(pg->count_info & PGC_broken) ) return; =20 -#ifndef NDEBUG - /* Avoid callers relying on allocations returning zeroed pages. */ - unmap_domain_page(memset(__map_domain_page(pg), - SCRUB_BYTE_PATTERN, PAGE_SIZE)); -#else - /* For a production build, clear_page() is the fastest way to scrub. */ - clear_domain_page(_mfn(page_to_mfn(pg))); -#endif + ptr =3D __map_domain_page(pg); + if ( cold ) + scrub_page_cold(ptr); + else + scrub_page_hot(ptr); + unmap_domain_page(ptr); } =20 static void poison_one_page(struct page_info *pg) @@ -1079,12 +1084,14 @@ static struct page_info *alloc_heap_page if ( first_dirty !=3D INVALID_DIRTY_IDX || (scrub_debug && !(memflags & MEMF_no_scrub)) ) { + bool cold =3D d && d !=3D current->domain; + for ( i =3D 0; i < (1U << order); i++ ) { if ( test_and_clear_bit(_PGC_need_scrub, &pg[i].count_info) ) { if ( !(memflags & MEMF_no_scrub) ) - scrub_one_page(&pg[i]); + scrub_one_page(&pg[i], cold); =20 dirty_cnt++; } @@ -1349,7 +1356,7 @@ bool scrub_free_pages(void) { if ( test_bit(_PGC_need_scrub, &pg[i].count_info) ) { - scrub_one_page(&pg[i]); + scrub_one_page(&pg[i], true); /* * We can modify count_info without holding heap * lock since we effectively locked this buddy by @@ -2074,7 +2081,7 @@ static struct page_info *alloc_color_hea if ( !(memflags & MEMF_no_scrub) ) { if ( need_scrub ) - scrub_one_page(pg); + scrub_one_page(pg, d !=3D current->domain); else check_one_page(pg); } @@ -2225,7 +2232,7 @@ static void __init cf_check smp_scrub_he if ( !mfn_valid(_mfn(mfn)) || !page_state_is(pg, free) ) continue; =20 - scrub_one_page(pg); + scrub_one_page(pg, true); } } =20 @@ -2930,7 +2937,7 @@ void unprepare_staticmem_pages(struct pa if ( need_scrub ) { /* TODO: asynchronous scrubbing for pages of static memory. */ - scrub_one_page(pg); + scrub_one_page(pg, true); } =20 pg[i].count_info |=3D PGC_static; --- /dev/null +++ b/xen/include/xen/scrub.h @@ -0,0 +1,24 @@ +#ifndef __XEN_SCRUB_H__ +#define __XEN_SCRUB_H__ + +#include + +/* SCRUB_PATTERN needs to be a repeating series of bytes. */ +#ifdef CONFIG_DEBUG +# define SCRUB_PATTERN _AC(0xc2c2c2c2c2c2c2c2,ULL) +#else +# define SCRUB_PATTERN _AC(0,ULL) +#endif +#define SCRUB_BYTE_PATTERN (SCRUB_PATTERN & 0xff) + +#endif /* __XEN_SCRUB_H__ */ + +/* + * Local variables: + * mode: C + * c-file-style: "BSD" + * c-basic-offset: 4 + * tab-width: 4 + * indent-tabs-mode: nil + * End: + */