From nobody Sun Feb 8 21:37:16 2026 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) client-ip=192.237.175.120; envelope-from=xen-devel-bounces@lists.xenproject.org; helo=lists.xenproject.org; Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=pass(p=quarantine dis=none) header.from=suse.com ARC-Seal: i=1; a=rsa-sha256; t=1751458561; cv=none; d=zohomail.com; s=zohoarc; b=XQU1+xwUi3unY7dkSGb62NjcYCglusSOINWkz+Ds0y087npDMbF69S5LQ2htHJzH48NfaxRikOp+RjjgagH6o08YlsDZPWs7lVYcIe9DtxohUBZddiJF7ZJ/gS/AjlczjYkLQ/aAMb5EjKo8h6TI7g9R6diCSAIn6ygEHmw+/y0= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1751458561; h=Content-Type:Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=xpn+GdDf2K4bj6rIROuU+aZcZQNprzBWNDxRLEP7f28=; b=YyXQVpGMSNz+GIxnA0U80LmYa/Giri0F13pRZ+uMAFDEB1HjqNLAEN7a4y/33qTIgqrFlu5AxGqPhCoJ4hgKeljK+4BNRwEwrBh/4ibruBjxIpT9Fx6RKcRZmmSKZPn4xyF8/xi8Ir+IMzZgdKlAvTRxJYkRgvp93r2KosXjVhg= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=pass header.from= (p=quarantine dis=none) Return-Path: Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) by mx.zohomail.com with SMTPS id 1751458561246374.44613019486474; Wed, 2 Jul 2025 05:16:01 -0700 (PDT) Received: from list by lists.xenproject.org with outflank-mailman.1031000.1404669 (Exim 4.92) (envelope-from ) id 1uWwNI-0008G2-PL; Wed, 02 Jul 2025 12:15:40 +0000 Received: by outflank-mailman (output) from mailman id 1031000.1404669; Wed, 02 Jul 2025 12:15:40 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1uWwNI-0008Fv-LN; Wed, 02 Jul 2025 12:15:40 +0000 Received: by outflank-mailman (input) for mailman id 1031000; Wed, 02 Jul 2025 12:15:38 +0000 Received: from se1-gles-sth1-in.inumbo.com ([159.253.27.254] helo=se1-gles-sth1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1uWwNG-0008Fp-RK for xen-devel@lists.xenproject.org; Wed, 02 Jul 2025 12:15:38 +0000 Received: from mail-wr1-x433.google.com (mail-wr1-x433.google.com [2a00:1450:4864:20::433]) by se1-gles-sth1.inumbo.com (Halon) with ESMTPS id 4343259f-573e-11f0-a313-13f23c93f187; Wed, 02 Jul 2025 14:15:37 +0200 (CEST) Received: by mail-wr1-x433.google.com with SMTP id ffacd0b85a97d-3a4e742dc97so4916687f8f.0 for ; Wed, 02 Jul 2025 05:15:37 -0700 (PDT) Received: from [10.156.60.236] (ip-037-024-206-209.um08.pools.vodafone-ip.de. [37.24.206.209]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-23acb39bb7dsm129018995ad.90.2025.07.02.05.15.34 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Wed, 02 Jul 2025 05:15:36 -0700 (PDT) X-Outflank-Mailman: Message body and most headers restored to incoming version X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: 4343259f-573e-11f0-a313-13f23c93f187 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.com; s=google; t=1751458537; x=1752063337; darn=lists.xenproject.org; h=content-transfer-encoding:in-reply-to:autocrypt:content-language :references:cc:to:from:subject:user-agent:mime-version:date :message-id:from:to:cc:subject:date:message-id:reply-to; bh=xpn+GdDf2K4bj6rIROuU+aZcZQNprzBWNDxRLEP7f28=; b=Gpu1UEZXlhjYMb+w3oawZeaqW2nqeRcCtQDTlT6CjeZqOOjpMuxL9NMyVuDMBGGL7p cXoBC9EHhkcskElfEf9psHmjCdY1GkFx5K42apQFYx7McQfR8Ann9sKDHeQGhcnDRSqU Rr1k2GG5vAyKcm9sqPw70uFdAVsWNz1B+UxtVlZZgYggqxTyiW3WW+/Jj3i5jzwIIBWU kjPqWf63hCM9hHpQ2NlZtoThYIY39hUKNBqaqApwBtkDDMQUA41Kzsl1c7oPkbR5/vXD PP10Nv30NtraT+ard48ozuRPCjd+8iqKskfJ1/+C6shoWA4LsaU5C96YHYah9WJvPvHb hvoQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1751458537; x=1752063337; h=content-transfer-encoding:in-reply-to:autocrypt:content-language :references:cc:to:from:subject:user-agent:mime-version:date :message-id:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=xpn+GdDf2K4bj6rIROuU+aZcZQNprzBWNDxRLEP7f28=; b=KZuaYWhSMRRQZ19/v3XsPa9Qwq8VA+dY675HNzM/8kWJCrwr5GjPMz0757qKxps+Sy DTqxndy3tdLOTOvpt4EvxVBiEDCRdkc8Si3cOhA7bUcMXc1R+b6EfX+8eqsAdO5On59s KspGoVZxE93NOApZGSWSXs8ShlJgwypDBatxWzkVM/LcXQdQOIbrF72zcFu/mDDenXEu Au6z7ivUrycc5bpCSaRofWEn99lE1uu1S6bgvb7xskzcAILPu0pdFB4lEqiDK+N+YEcO GSMUKaQ/iNfiHdAW77jKwzqOHfgtvAXkMD70apsBeZgdUq/baXzxg8JPfONII3vOcM5c S1EQ== X-Gm-Message-State: AOJu0YygvzRGg3WFwUsej5g3nisFMbEaANGMqEonbWcBSOdp9DZ5Kqd+ CeVONAKz5kNI40SxnJh8eErthDLyzEhazAWhm5/pv5WKINZAnFXpZpSXPh0MvyechJ6o4tM+WMh P/XI= X-Gm-Gg: ASbGnctp6MWMvtpqPEXiDofIView197cgcdKNWk+EqSf87kXytEICvJzDgsAfdrD8h8 d3nlJ/b/KGxQD0X4o8AAu5z0jm6eO1Oa7dmrqzSh6KeE7mOJRHaz0CVyQ9vFbDB70qxJ22hluRd +3lGBq9blo3c1OAtCWHAbB4ykQ/puw06fpu4m62M+YMkzIIR77T8YBwcRHUXYZ12QD87JLFkaH9 APn7mLxNTbXTdII0IXG9btvs4QSaPX61bRfKe/gNrXR0oh8F52dlYhAyO2z6Z208NYvUu1NSmZJ EOBKhQ4S4EEyF980PjjHixhibUJsBG4i565LHo8tWc/ufNaG++7YVCwQisPgYJxCflta7yWDaNm 9Knr4ScEtrZ8AqZ1Z5bXPGhboIxPWm0upkR2izz2DSYQgLg4= X-Google-Smtp-Source: AGHT+IGBfQRE5Qh4HUKgB9D1SjowhCPEFI2kOXdBpuXWIuAwOxvDaV9t5sfoxhZ/SxW45YDAWWN+fw== X-Received: by 2002:a05:6000:481b:b0:3a4:f607:a5ad with SMTP id ffacd0b85a97d-3b1f7ff7a30mr1662570f8f.19.1751458537111; Wed, 02 Jul 2025 05:15:37 -0700 (PDT) Message-ID: Date: Wed, 2 Jul 2025 14:15:30 +0200 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: [PATCH v7 1/7] x86: suppress ERMS for internal use when MISC_ENABLE.FAST_STRING is clear From: Jan Beulich To: "xen-devel@lists.xenproject.org" Cc: Andrew Cooper , =?UTF-8?Q?Roger_Pau_Monn=C3=A9?= References: <0b57c6f0-9fd0-4f8a-93f8-dc494a1e952c@suse.com> Content-Language: en-US Autocrypt: addr=jbeulich@suse.com; keydata= xsDiBFk3nEQRBADAEaSw6zC/EJkiwGPXbWtPxl2xCdSoeepS07jW8UgcHNurfHvUzogEq5xk hu507c3BarVjyWCJOylMNR98Yd8VqD9UfmX0Hb8/BrA+Hl6/DB/eqGptrf4BSRwcZQM32aZK 7Pj2XbGWIUrZrd70x1eAP9QE3P79Y2oLrsCgbZJfEwCgvz9JjGmQqQkRiTVzlZVCJYcyGGsD /0tbFCzD2h20ahe8rC1gbb3K3qk+LpBtvjBu1RY9drYk0NymiGbJWZgab6t1jM7sk2vuf0Py O9Hf9XBmK0uE9IgMaiCpc32XV9oASz6UJebwkX+zF2jG5I1BfnO9g7KlotcA/v5ClMjgo6Gl MDY4HxoSRu3i1cqqSDtVlt+AOVBJBACrZcnHAUSuCXBPy0jOlBhxPqRWv6ND4c9PH1xjQ3NP nxJuMBS8rnNg22uyfAgmBKNLpLgAGVRMZGaGoJObGf72s6TeIqKJo/LtggAS9qAUiuKVnygo 3wjfkS9A3DRO+SpU7JqWdsveeIQyeyEJ/8PTowmSQLakF+3fote9ybzd880fSmFuIEJldWxp Y2ggPGpiZXVsaWNoQHN1c2UuY29tPsJgBBMRAgAgBQJZN5xEAhsDBgsJCAcDAgQVAggDBBYC AwECHgECF4AACgkQoDSui/t3IH4J+wCfQ5jHdEjCRHj23O/5ttg9r9OIruwAn3103WUITZee e7Sbg12UgcQ5lv7SzsFNBFk3nEQQCACCuTjCjFOUdi5Nm244F+78kLghRcin/awv+IrTcIWF hUpSs1Y91iQQ7KItirz5uwCPlwejSJDQJLIS+QtJHaXDXeV6NI0Uef1hP20+y8qydDiVkv6l IreXjTb7DvksRgJNvCkWtYnlS3mYvQ9NzS9PhyALWbXnH6sIJd2O9lKS1Mrfq+y0IXCP10eS FFGg+Av3IQeFatkJAyju0PPthyTqxSI4lZYuJVPknzgaeuJv/2NccrPvmeDg6Coe7ZIeQ8Yj t0ARxu2xytAkkLCel1Lz1WLmwLstV30g80nkgZf/wr+/BXJW/oIvRlonUkxv+IbBM3dX2OV8 AmRv1ySWPTP7AAMFB/9PQK/VtlNUJvg8GXj9ootzrteGfVZVVT4XBJkfwBcpC/XcPzldjv+3 HYudvpdNK3lLujXeA5fLOH+Z/G9WBc5pFVSMocI71I8bT8lIAzreg0WvkWg5V2WZsUMlnDL9 mpwIGFhlbM3gfDMs7MPMu8YQRFVdUvtSpaAs8OFfGQ0ia3LGZcjA6Ik2+xcqscEJzNH+qh8V m5jjp28yZgaqTaRbg3M/+MTbMpicpZuqF4rnB0AQD12/3BNWDR6bmh+EkYSMcEIpQmBM51qM EKYTQGybRCjpnKHGOxG0rfFY1085mBDZCH5Kx0cl0HVJuQKC+dV2ZY5AqjcKwAxpE75MLFkr wkkEGBECAAkFAlk3nEQCGwwACgkQoDSui/t3IH7nnwCfcJWUDUFKdCsBH/E5d+0ZnMQi+G0A nAuWpQkjM1ASeQwSHEeAWPgskBQL In-Reply-To: <0b57c6f0-9fd0-4f8a-93f8-dc494a1e952c@suse.com> Content-Transfer-Encoding: quoted-printable X-ZohoMail-DKIM: pass (identity @suse.com) X-ZM-MESSAGEID: 1751458563141116600 Content-Type: text/plain; charset="utf-8" Before we start actually adjusting behavior when ERMS is available, follow Linux commit 161ec53c702c ("x86, mem, intel: Initialize Enhanced REP MOVSB/STOSB") and zap the CPUID-derived feature flag when the MSR bit is clear. Signed-off-by: Jan Beulich Reviewed-by: Jason Andryuk --- TBD: While with the use of host_cpu_policy "cpuid=3Dno-erms" now propagates to guest view, we've now lost the earlier "Don't extend the artificial clearing to guest view, though: Guests can take their own decision in this regard, as they can read (most of) MISC_ENABLE.". --- v7: Use host_cpu_policy in guest_common_default_feature_adjustments(). Change commentary. v5: Correct guest_common_max_feature_adjustments() addition. v4: Also adjust guest_common_max_feature_adjustments(). v3: New. --- a/xen/arch/x86/cpu/intel.c +++ b/xen/arch/x86/cpu/intel.c @@ -366,8 +366,18 @@ static void cf_check early_init_intel(st paddr_bits =3D 36; =20 if (c =3D=3D &boot_cpu_data) { + uint64_t misc_enable; + check_memory_type_self_snoop_errata(); =20 + /* + * If fast string is not enabled in IA32_MISC_ENABLE for any reason, + * clear the enhanced fast string CPU capability. + */ + rdmsrl(MSR_IA32_MISC_ENABLE, misc_enable); + if (!(misc_enable & MSR_IA32_MISC_ENABLE_FAST_STRING)) + setup_clear_cpu_cap(X86_FEATURE_ERMS); + intel_init_levelling(); } =20 --- a/xen/arch/x86/cpu-policy.c +++ b/xen/arch/x86/cpu-policy.c @@ -487,6 +487,12 @@ static void __init guest_common_max_feat */ if ( test_bit(X86_FEATURE_RTM, fs) ) __set_bit(X86_FEATURE_RTM_ALWAYS_ABORT, fs); + + /* + * ERMS is a performance hint. A VM which previously saw ERMS will + * function correctly when migrated here, even if ERMS isn't available. + */ + __set_bit(X86_FEATURE_ERMS, fs); } =20 static void __init guest_common_default_feature_adjustments(uint32_t *fs) @@ -562,6 +568,13 @@ static void __init guest_common_default_ __clear_bit(X86_FEATURE_RTM, fs); __set_bit(X86_FEATURE_RTM_ALWAYS_ABORT, fs); } + + /* + * ERMS is a performance hint, so is set unconditionally in the max + * policy. However, guests should default to the host setting. + */ + if ( !host_cpu_policy.feat.erms ) + __clear_bit(X86_FEATURE_ERMS, fs); } =20 static void __init guest_common_feature_adjustments(uint32_t *fs) --- a/xen/arch/x86/include/asm/msr-index.h +++ b/xen/arch/x86/include/asm/msr-index.h @@ -493,6 +493,7 @@ #define MSR_IA32_THERM_INTERRUPT 0x0000019b #define MSR_IA32_THERM_STATUS 0x0000019c #define MSR_IA32_MISC_ENABLE 0x000001a0 +#define MSR_IA32_MISC_ENABLE_FAST_STRING (1<<0) #define MSR_IA32_MISC_ENABLE_PERF_AVAIL (1<<7) #define MSR_IA32_MISC_ENABLE_BTS_UNAVAIL (1<<11) #define MSR_IA32_MISC_ENABLE_PEBS_UNAVAIL (1<<12) From nobody Sun Feb 8 21:37:16 2026 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) client-ip=192.237.175.120; envelope-from=xen-devel-bounces@lists.xenproject.org; helo=lists.xenproject.org; Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=pass(p=quarantine dis=none) header.from=suse.com ARC-Seal: i=1; a=rsa-sha256; t=1751458593; cv=none; d=zohomail.com; s=zohoarc; b=gObJEioHpZSscBEKgaNxvTjOXNp41tX8AS+U3SO3bO5VUO6G8EmUp5Dy37BtjLXgDv/kh5905S5IHstvwIv4GjJvn9GbAGqufPGVgT1BXHVPoPklP/ogmJnFMOEFOnkUWlu0d1mrdpFP1K2AvOHMW4DTEne146DkZEBzPbEHYFw= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1751458593; h=Content-Type:Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=PRY+zNrdOJssR5FV1ZBdSspoE8MCt50jDjF3DFcSHMs=; b=c6YOdgUpWv/gD4YE3gEtiMJhgRfxruLep60OioADU2rjCzvXhY10JrYIFJpDiF2SEOfg0FKC5FhEs3g5ug9wYZokTCLgOCgUoagchghtdJbXa6CutEBgSZwGs1/NOkc2xvGxvYSzzv7u9FAI8rwAuG0iqTzgqNrk5TVgCKI9EjQ= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=pass header.from= (p=quarantine dis=none) Return-Path: Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) by mx.zohomail.com with SMTPS id 17514585936631007.2114542515851; Wed, 2 Jul 2025 05:16:33 -0700 (PDT) Received: from list by lists.xenproject.org with outflank-mailman.1031006.1404677 (Exim 4.92) (envelope-from ) id 1uWwNv-0000GE-Vs; Wed, 02 Jul 2025 12:16:19 +0000 Received: by outflank-mailman (output) from mailman id 1031006.1404677; Wed, 02 Jul 2025 12:16:19 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1uWwNv-0000G7-TN; Wed, 02 Jul 2025 12:16:19 +0000 Received: by outflank-mailman (input) for mailman id 1031006; Wed, 02 Jul 2025 12:16:18 +0000 Received: from se1-gles-flk1-in.inumbo.com ([94.247.172.50] helo=se1-gles-flk1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1uWwNu-0000Fz-Ub for xen-devel@lists.xenproject.org; Wed, 02 Jul 2025 12:16:18 +0000 Received: from mail-wm1-x330.google.com (mail-wm1-x330.google.com [2a00:1450:4864:20::330]) by se1-gles-flk1.inumbo.com (Halon) with ESMTPS id 5aa685eb-573e-11f0-b894-0df219b8e170; Wed, 02 Jul 2025 14:16:17 +0200 (CEST) Received: by mail-wm1-x330.google.com with SMTP id 5b1f17b1804b1-453066fad06so45780015e9.2 for ; Wed, 02 Jul 2025 05:16:17 -0700 (PDT) Received: from [10.156.60.236] (ip-037-024-206-209.um08.pools.vodafone-ip.de. [37.24.206.209]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-318c13a509asm14558195a91.20.2025.07.02.05.16.13 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Wed, 02 Jul 2025 05:16:15 -0700 (PDT) X-Outflank-Mailman: Message body and most headers restored to incoming version X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: 5aa685eb-573e-11f0-b894-0df219b8e170 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.com; s=google; t=1751458576; x=1752063376; darn=lists.xenproject.org; h=content-transfer-encoding:in-reply-to:autocrypt:content-language :references:cc:to:from:subject:user-agent:mime-version:date :message-id:from:to:cc:subject:date:message-id:reply-to; bh=PRY+zNrdOJssR5FV1ZBdSspoE8MCt50jDjF3DFcSHMs=; b=BSTV8eNDcj3Z/upncgOfn8cRwNVj/Hvz0B+54X8elN+ktse8ATvF+Ii2vZsl5gtA1o 4Q+Ez1OEFejGdlvoeYgbA5Udi1r40Jku/H19KeBmYbz5NzcODviEqLUvkszoyAAzTQBe 8q2nUZvjIkJFhLQInBe96PQhDA7kr1eLm5j/rd6wVl72w8j3tJZgz+3OcQ6NWLyFuTmb CmnmQvHW2aITWm0gCr8EEAX8dFloJL8dZ+bROLLq1MIpq9QXx0XuasGZAFP7kvNv3/Xh Cg0NY4S5NCvd3iLopGG8tZn4rkLLT4ImgCkFsLhP+MoyDr84PPgRcBM59PqpapgD4l38 Z8Lg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1751458576; x=1752063376; h=content-transfer-encoding:in-reply-to:autocrypt:content-language :references:cc:to:from:subject:user-agent:mime-version:date :message-id:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=PRY+zNrdOJssR5FV1ZBdSspoE8MCt50jDjF3DFcSHMs=; b=UXvgaQeDEgKuP51RFkExtrRHTFLZnAk4lF86odu7v30mWL6ZuAGrTeKi83X5UhI5UW dlPbRTNQKRSjaRyy6ggJtnDRwuFkQoYqUdFiZq2kph/C/j+nhrr1R/arHzvxckXHHNVb f2kdK7c3gFwTIOornEzgEhvRHappsRqX0lLOtBBO15rttDsfvUPCDyTHQrbRjWKO0s/f 3QkSayOak8osy9eXRgXg4RIMz2hHS5k58j+IirTzZgGm5ojbXdyBehTu6V2tGSwGMBpP aZlWAy1D73YxID5dMyX285Z5yTZtQThKtuCO4JUr9kPotvybawleHpERTt2Y4RWSvS+O jrhQ== X-Gm-Message-State: AOJu0YxHxSgqORrC1suehLDwv3zLymxKZK6PAd6UFj2aiRH3mDxLrRLh 2/3WexyEnvnazhkr1YV3EX0NiuJanU09aiD1vtcvV1mVmsQmLyuzjj0P1I6gnakFvmIjWp0Vawz zQCk= X-Gm-Gg: ASbGncsMPONkepYTyL9qi7CqVt5w7l30Fx9057XHSFhWR3lrNMPsGAk2ZGaIUSgJT05 krgmMC6bHk915e8qlRECCas2N18Nq/Cx9HKBcGZbfSyHTDvfR+K4VJZv7Zcg3309EBLlyr6mm3a 8XFDu3fod2+3wJJAnLrzZ77yEHi0VtRRwmdA2hff/j9bl1ePcdCAC0sLi46nVYtiws3y9dc6sF6 zxJ/asiFOzzKDpRizQk3ocICO2F92sLQ0HliLSLpnoLZNOJqeZF5fgaQVOuaY6t3FWB/ZUko6Ea tdObRXK/epHiAwl+AZoNQLPP4EsxTaPR7ushn4CKnZKT/wSBTngPed+74tfjC7gSsbpNBolSXiZ QWSzkkDuriLbsBjEDYMsCpJJdAcJof7OXGjZQHYcQZWfpt9s= X-Google-Smtp-Source: AGHT+IGc2CgXwlHd+Bez7F409OcAi+ZlHzorV8utjIrmquyoqLxYgNFxJCtIehW6tffwQFy1Slbtuw== X-Received: by 2002:a05:6000:2809:b0:3a5:232c:6976 with SMTP id ffacd0b85a97d-3b200865de2mr1636662f8f.44.1751458576196; Wed, 02 Jul 2025 05:16:16 -0700 (PDT) Message-ID: Date: Wed, 2 Jul 2025 14:16:09 +0200 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: [PATCH v7 2/7] x86/alternatives: serialize after (self-)modifying code From: Jan Beulich To: "xen-devel@lists.xenproject.org" Cc: Andrew Cooper , =?UTF-8?Q?Roger_Pau_Monn=C3=A9?= References: <0b57c6f0-9fd0-4f8a-93f8-dc494a1e952c@suse.com> Content-Language: en-US Autocrypt: addr=jbeulich@suse.com; keydata= xsDiBFk3nEQRBADAEaSw6zC/EJkiwGPXbWtPxl2xCdSoeepS07jW8UgcHNurfHvUzogEq5xk hu507c3BarVjyWCJOylMNR98Yd8VqD9UfmX0Hb8/BrA+Hl6/DB/eqGptrf4BSRwcZQM32aZK 7Pj2XbGWIUrZrd70x1eAP9QE3P79Y2oLrsCgbZJfEwCgvz9JjGmQqQkRiTVzlZVCJYcyGGsD /0tbFCzD2h20ahe8rC1gbb3K3qk+LpBtvjBu1RY9drYk0NymiGbJWZgab6t1jM7sk2vuf0Py O9Hf9XBmK0uE9IgMaiCpc32XV9oASz6UJebwkX+zF2jG5I1BfnO9g7KlotcA/v5ClMjgo6Gl MDY4HxoSRu3i1cqqSDtVlt+AOVBJBACrZcnHAUSuCXBPy0jOlBhxPqRWv6ND4c9PH1xjQ3NP nxJuMBS8rnNg22uyfAgmBKNLpLgAGVRMZGaGoJObGf72s6TeIqKJo/LtggAS9qAUiuKVnygo 3wjfkS9A3DRO+SpU7JqWdsveeIQyeyEJ/8PTowmSQLakF+3fote9ybzd880fSmFuIEJldWxp Y2ggPGpiZXVsaWNoQHN1c2UuY29tPsJgBBMRAgAgBQJZN5xEAhsDBgsJCAcDAgQVAggDBBYC AwECHgECF4AACgkQoDSui/t3IH4J+wCfQ5jHdEjCRHj23O/5ttg9r9OIruwAn3103WUITZee e7Sbg12UgcQ5lv7SzsFNBFk3nEQQCACCuTjCjFOUdi5Nm244F+78kLghRcin/awv+IrTcIWF hUpSs1Y91iQQ7KItirz5uwCPlwejSJDQJLIS+QtJHaXDXeV6NI0Uef1hP20+y8qydDiVkv6l IreXjTb7DvksRgJNvCkWtYnlS3mYvQ9NzS9PhyALWbXnH6sIJd2O9lKS1Mrfq+y0IXCP10eS FFGg+Av3IQeFatkJAyju0PPthyTqxSI4lZYuJVPknzgaeuJv/2NccrPvmeDg6Coe7ZIeQ8Yj t0ARxu2xytAkkLCel1Lz1WLmwLstV30g80nkgZf/wr+/BXJW/oIvRlonUkxv+IbBM3dX2OV8 AmRv1ySWPTP7AAMFB/9PQK/VtlNUJvg8GXj9ootzrteGfVZVVT4XBJkfwBcpC/XcPzldjv+3 HYudvpdNK3lLujXeA5fLOH+Z/G9WBc5pFVSMocI71I8bT8lIAzreg0WvkWg5V2WZsUMlnDL9 mpwIGFhlbM3gfDMs7MPMu8YQRFVdUvtSpaAs8OFfGQ0ia3LGZcjA6Ik2+xcqscEJzNH+qh8V m5jjp28yZgaqTaRbg3M/+MTbMpicpZuqF4rnB0AQD12/3BNWDR6bmh+EkYSMcEIpQmBM51qM EKYTQGybRCjpnKHGOxG0rfFY1085mBDZCH5Kx0cl0HVJuQKC+dV2ZY5AqjcKwAxpE75MLFkr wkkEGBECAAkFAlk3nEQCGwwACgkQoDSui/t3IH7nnwCfcJWUDUFKdCsBH/E5d+0ZnMQi+G0A nAuWpQkjM1ASeQwSHEeAWPgskBQL In-Reply-To: <0b57c6f0-9fd0-4f8a-93f8-dc494a1e952c@suse.com> Content-Transfer-Encoding: quoted-printable X-ZohoMail-DKIM: pass (identity @suse.com) X-ZM-MESSAGEID: 1751458595345116600 Content-Type: text/plain; charset="utf-8" While supposedly safe via enforcing a control flow change when modifying already prefetched code, it may not really be. Afaik a request is pending to drop the first of the two options in the SDM's "Handling Self- and Cross-Modifying Code" section (still present there as of version 087). Insert a serializing instruction there, and remove the "noinline" in exchange. Signed-off-by: Jan Beulich Reviewed-by: Jason Andryuk --- Strictly speaking with LIVEPATCH=3Dn the if() (but not its body) is dead code. Adding IS_ENABLED(CONFIG_LIVEPATCH) would feel quite a bit like extra clutter, though. Plus Misra rule 2.2 is globally deviated anyway. --- v6: Split off. Extend commentary. Load deterministic value into CR2. Drop "noinline". --- a/xen/arch/x86/alternative.c +++ b/xen/arch/x86/alternative.c @@ -194,13 +194,18 @@ void *place_ret(void *ptr) * You should run this with interrupts disabled or on code that is not * executing. * - * "noinline" to cause control flow change and thus invalidate I$ and - * cause refetch after modification. + * While the SDM continues to suggest using "noinline" would be sufficient= , it + * may not be, e.g. due to errata. Issue a serializing insn afterwards, u= nless + * this is for live-patching, where we modify code before it goes live. I= ssue + * a serializing insn which is unlikely to be intercepted by a hypervisor,= in + * case we run virtualized ourselves. */ -static void init_or_livepatch noinline +static void init_or_livepatch text_poke(void *addr, const void *opcode, size_t len) { memcpy(addr, opcode, len); + if ( system_state < SYS_STATE_active ) + asm volatile ( "mov %0, %%cr2" :: "r" (0L) : "memory" ); } =20 extern void *const __initdata_cf_clobber_start[]; From nobody Sun Feb 8 21:37:16 2026 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) client-ip=192.237.175.120; envelope-from=xen-devel-bounces@lists.xenproject.org; helo=lists.xenproject.org; Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=pass(p=quarantine dis=none) header.from=suse.com ARC-Seal: i=1; a=rsa-sha256; t=1751458623; cv=none; d=zohomail.com; s=zohoarc; b=MS9n8PDTJTg6T0BNtjfUOiWfVqy7CnLwnoVzkNVrziQ8+v068nzLRJOaLEz+EPlT1b0fFnanpxNQy141vNviiC1CvrhspJAROJDniavS3JEsOiO//ScAG0vqWq3po4pTh55ljOra2Fyo+hcHU9ShmXtZLYWHZ8LGGboFbMKdR7I= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1751458623; h=Content-Type:Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=SE8xoyNdEL534537qH4i+x2FYrDmfoG9y/hWuSaiBfU=; b=P1Q9uI1FBEfCQ6b28K+p8GmTZp1Ndl5HbwFEuCqNefrZzN1aJS6p66tInDVd1ummIagRd32AicdAPhU/dVDT8FVR7YFsbZhrSsEeajsX6TUJpB4Jy5N2/IwSfmgdt1eKoVNSdj4ux20dy43GugFJAGv03pLfxa63Yq4lF9UyujA= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=pass header.from= (p=quarantine dis=none) Return-Path: Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) by mx.zohomail.com with SMTPS id 1751458623277400.2410994026327; Wed, 2 Jul 2025 05:17:03 -0700 (PDT) Received: from list by lists.xenproject.org with outflank-mailman.1031010.1404689 (Exim 4.92) (envelope-from ) id 1uWwOO-0000jG-8S; Wed, 02 Jul 2025 12:16:48 +0000 Received: by outflank-mailman (output) from mailman id 1031010.1404689; Wed, 02 Jul 2025 12:16:48 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1uWwOO-0000j9-4K; Wed, 02 Jul 2025 12:16:48 +0000 Received: by outflank-mailman (input) for mailman id 1031010; Wed, 02 Jul 2025 12:16:46 +0000 Received: from se1-gles-flk1-in.inumbo.com ([94.247.172.50] helo=se1-gles-flk1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1uWwOM-0000Fz-QS for xen-devel@lists.xenproject.org; Wed, 02 Jul 2025 12:16:46 +0000 Received: from mail-wr1-x433.google.com (mail-wr1-x433.google.com [2a00:1450:4864:20::433]) by se1-gles-flk1.inumbo.com (Halon) with ESMTPS id 6b6acd79-573e-11f0-b894-0df219b8e170; Wed, 02 Jul 2025 14:16:45 +0200 (CEST) Received: by mail-wr1-x433.google.com with SMTP id ffacd0b85a97d-3a5257748e1so2937044f8f.2 for ; Wed, 02 Jul 2025 05:16:45 -0700 (PDT) Received: from [10.156.60.236] (ip-037-024-206-209.um08.pools.vodafone-ip.de. [37.24.206.209]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-74af56ceb74sm14601400b3a.135.2025.07.02.05.16.41 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Wed, 02 Jul 2025 05:16:43 -0700 (PDT) X-Outflank-Mailman: Message body and most headers restored to incoming version X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: 6b6acd79-573e-11f0-b894-0df219b8e170 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.com; s=google; t=1751458604; x=1752063404; darn=lists.xenproject.org; h=content-transfer-encoding:in-reply-to:autocrypt:content-language :references:cc:to:from:subject:user-agent:mime-version:date :message-id:from:to:cc:subject:date:message-id:reply-to; bh=SE8xoyNdEL534537qH4i+x2FYrDmfoG9y/hWuSaiBfU=; b=Ux7RmnKmw1pg52xb6xd0wH+fzSk+vRKFsjkT01mQyzojeqmIEzTbYrM4sGwL0OEyIP J7n3QESkdYJdl9ualB/G0DqB+DzreE2+FnuxOvj9opSvnNZB63I1/1R+k0RRaEm7tNGB 3gXBkbQ3ktTnI0czp/btLzcKnabiIW11jnncdKn+mqskWaqnNtoDXyUQGR30oh+cfm1d 41GbiSpspKJC4yYrv0pjm2LRrIhGrfzlCo533dQ38MlSI7mzHm+DqnEjE/NkHN0yo+d8 YagB318AxwvTqgjaDK1YmK1v59VGrUVIyNSRl4Ea/xUOJrbbBLNsowdzxOaxgGrqyuKu 2Pyw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1751458604; x=1752063404; h=content-transfer-encoding:in-reply-to:autocrypt:content-language :references:cc:to:from:subject:user-agent:mime-version:date :message-id:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=SE8xoyNdEL534537qH4i+x2FYrDmfoG9y/hWuSaiBfU=; b=bphFQVsru2zxCTvxcastkEpCsu51Susr6pcxvAqe+e52GuNqWq4qh4wQFkikoqZYcF sWDXQ+c+kgUnFdmEfwAfQaXvxxye/AC3Pv5hfgxLLtJ/w5yMOjb7tdh/ZZqatAikVsP8 dtY17zy5SYuUtdOhkRHakIV6eQZ/hoEVksF4pVlUNpyNsgGS7QpOW5DK+c+WV5k9IvjI enKoP5wiYJQSdp7kQWPxtebxSbvC0T9zA/UlhfOz1ODSAzn3AuMd/aKNsmD9hRvlwv2D kHUYcq/LjETH8yGVkHGfFwQYeP8AW2Yt6fzOASrhKz6YsayeWBcvGG5/p8gitz5uLKFA 0tAA== X-Gm-Message-State: AOJu0Yww7hHgsvppzjOa3Fz+w4aLIAkEdXZBq1DCFUeV1+K/AL8BpyIt izHrwGlaVolZdobsfT3EMV3F8ibnSZATQbkSfp/6/xx7lVOTo+W18ZZj//2Do2DMs/kI7KMzcYC c/ds= X-Gm-Gg: ASbGncv1P60UOIwkQ244+uGjw3pNFaqCnB/EkYKFPeOPXoDVLe1qvasdRP436tHnSPG Dcy8mUVJ1yX0ywiacNahSBqt7BV8yc6N3gp/xOUh9AuD5Vb5ALnT8GtfHGEsKsq1ONrJGikbjqs BT/WaMUiWlqfYUXt/kIFfl04nWHWfVQgCNcWa9URdTptJt+m2564jrIVFa3iQM12kukta/dhkjb eW02U6r3yTfy8+56YsvJLgsO66ZZunfmaZUF0NBOA0jCxsyCyOG/I09Ec4ViZ4KKC/mnBp09i9W 2gJio5TnnFzquC5miQzliWEkUX8w9icnDyqpMtUrFp1TIcc9KbhYkBeIm70r6rImPk3JUHqX3GB QASU8u7xqz4d+YTxOtaE9twLfbH1rsXp3MYS9Hmelz9psUus= X-Google-Smtp-Source: AGHT+IGhy45QJk1iqOLx0KbGMNzCiLI49h3C97l8MTWZf3BXAB4e8yTx+ULA3+l2/Ex/QZ20D6NZWw== X-Received: by 2002:a5d:5888:0:b0:3a4:d0ed:257b with SMTP id ffacd0b85a97d-3b1ff42f9b6mr2096512f8f.22.1751458604454; Wed, 02 Jul 2025 05:16:44 -0700 (PDT) Message-ID: Date: Wed, 2 Jul 2025 14:16:37 +0200 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: [PATCH v7 3/7] x86: re-work memset() From: Jan Beulich To: "xen-devel@lists.xenproject.org" Cc: Andrew Cooper , =?UTF-8?Q?Roger_Pau_Monn=C3=A9?= References: <0b57c6f0-9fd0-4f8a-93f8-dc494a1e952c@suse.com> Content-Language: en-US Autocrypt: addr=jbeulich@suse.com; keydata= xsDiBFk3nEQRBADAEaSw6zC/EJkiwGPXbWtPxl2xCdSoeepS07jW8UgcHNurfHvUzogEq5xk hu507c3BarVjyWCJOylMNR98Yd8VqD9UfmX0Hb8/BrA+Hl6/DB/eqGptrf4BSRwcZQM32aZK 7Pj2XbGWIUrZrd70x1eAP9QE3P79Y2oLrsCgbZJfEwCgvz9JjGmQqQkRiTVzlZVCJYcyGGsD /0tbFCzD2h20ahe8rC1gbb3K3qk+LpBtvjBu1RY9drYk0NymiGbJWZgab6t1jM7sk2vuf0Py O9Hf9XBmK0uE9IgMaiCpc32XV9oASz6UJebwkX+zF2jG5I1BfnO9g7KlotcA/v5ClMjgo6Gl MDY4HxoSRu3i1cqqSDtVlt+AOVBJBACrZcnHAUSuCXBPy0jOlBhxPqRWv6ND4c9PH1xjQ3NP nxJuMBS8rnNg22uyfAgmBKNLpLgAGVRMZGaGoJObGf72s6TeIqKJo/LtggAS9qAUiuKVnygo 3wjfkS9A3DRO+SpU7JqWdsveeIQyeyEJ/8PTowmSQLakF+3fote9ybzd880fSmFuIEJldWxp Y2ggPGpiZXVsaWNoQHN1c2UuY29tPsJgBBMRAgAgBQJZN5xEAhsDBgsJCAcDAgQVAggDBBYC AwECHgECF4AACgkQoDSui/t3IH4J+wCfQ5jHdEjCRHj23O/5ttg9r9OIruwAn3103WUITZee e7Sbg12UgcQ5lv7SzsFNBFk3nEQQCACCuTjCjFOUdi5Nm244F+78kLghRcin/awv+IrTcIWF hUpSs1Y91iQQ7KItirz5uwCPlwejSJDQJLIS+QtJHaXDXeV6NI0Uef1hP20+y8qydDiVkv6l IreXjTb7DvksRgJNvCkWtYnlS3mYvQ9NzS9PhyALWbXnH6sIJd2O9lKS1Mrfq+y0IXCP10eS FFGg+Av3IQeFatkJAyju0PPthyTqxSI4lZYuJVPknzgaeuJv/2NccrPvmeDg6Coe7ZIeQ8Yj t0ARxu2xytAkkLCel1Lz1WLmwLstV30g80nkgZf/wr+/BXJW/oIvRlonUkxv+IbBM3dX2OV8 AmRv1ySWPTP7AAMFB/9PQK/VtlNUJvg8GXj9ootzrteGfVZVVT4XBJkfwBcpC/XcPzldjv+3 HYudvpdNK3lLujXeA5fLOH+Z/G9WBc5pFVSMocI71I8bT8lIAzreg0WvkWg5V2WZsUMlnDL9 mpwIGFhlbM3gfDMs7MPMu8YQRFVdUvtSpaAs8OFfGQ0ia3LGZcjA6Ik2+xcqscEJzNH+qh8V m5jjp28yZgaqTaRbg3M/+MTbMpicpZuqF4rnB0AQD12/3BNWDR6bmh+EkYSMcEIpQmBM51qM EKYTQGybRCjpnKHGOxG0rfFY1085mBDZCH5Kx0cl0HVJuQKC+dV2ZY5AqjcKwAxpE75MLFkr wkkEGBECAAkFAlk3nEQCGwwACgkQoDSui/t3IH7nnwCfcJWUDUFKdCsBH/E5d+0ZnMQi+G0A nAuWpQkjM1ASeQwSHEeAWPgskBQL In-Reply-To: <0b57c6f0-9fd0-4f8a-93f8-dc494a1e952c@suse.com> Content-Transfer-Encoding: quoted-printable X-ZohoMail-DKIM: pass (identity @suse.com) X-ZM-MESSAGEID: 1751458625756116600 Content-Type: text/plain; charset="utf-8" Move the function to its own assembly file. Having it in C just for the entire body to be an asm() isn't really helpful. Then have two flavors: A "basic" version using qword steps for the bulk of the operation, and an ERMS version for modern hardware, to be substituted in via alternatives patching. For RET to be usable in an alternative's replacement code, extend the CALL/JMP patching to cover the case of "JMP __x86_return_thunk" coming last in replacement code. Signed-off-by: Jan Beulich Reviewed-by: Jason Andryuk --- We may want to consider branching over the REP STOSQ as well, if the number of qwords turns out to be zero. We may also want to consider using non-REP STOS{L,W,B} for the tail. --- v5: Re-base. v4: Use %r8 instead of %rsi in a few places. v3: Re-base. --- a/xen/arch/x86/Makefile +++ b/xen/arch/x86/Makefile @@ -47,6 +47,7 @@ obj-$(CONFIG_RETURN_THUNK) +=3D indirect-t obj-$(CONFIG_PV) +=3D ioport_emulate.o obj-y +=3D irq.o obj-$(CONFIG_KEXEC) +=3D machine_kexec.o +obj-y +=3D memset.o obj-y +=3D mm.o x86_64/mm.o obj-$(CONFIG_VM_EVENT) +=3D monitor.o obj-y +=3D mpparse.o --- a/xen/arch/x86/alternative.c +++ b/xen/arch/x86/alternative.c @@ -351,6 +351,12 @@ static int init_or_livepatch _apply_alte /* 0xe8/0xe9 are relative branches; fix the offset. */ if ( a->repl_len >=3D 5 && (*buf & 0xfe) =3D=3D 0xe8 ) *(int32_t *)(buf + 1) +=3D repl - orig; + else if ( IS_ENABLED(CONFIG_RETURN_THUNK) && + a->repl_len > 5 && buf[a->repl_len - 5] =3D=3D 0xe9 && + ((long)repl + a->repl_len + + *(int32_t *)(buf + a->repl_len - 4) =3D=3D + (long)__x86_return_thunk) ) + *(int32_t *)(buf + a->repl_len - 4) +=3D repl - orig; =20 a->priv =3D 1; =20 --- /dev/null +++ b/xen/arch/x86/memset.S @@ -0,0 +1,30 @@ +#include + +.macro memset + and $7, %edx + shr $3, %rcx + movzbl %sil, %esi + mov $0x0101010101010101, %rax + imul %rsi, %rax + mov %rdi, %r8 + rep stosq + or %edx, %ecx + jz 0f + rep stosb +0: + mov %r8, %rax + RET +.endm + +.macro memset_erms + mov %esi, %eax + mov %rdi, %r8 + rep stosb + mov %r8, %rax + RET +.endm + +FUNC(memset) + mov %rdx, %rcx + ALTERNATIVE memset, memset_erms, X86_FEATURE_ERMS +END(memset) --- a/xen/arch/x86/string.c +++ b/xen/arch/x86/string.c @@ -22,19 +22,6 @@ void *(memcpy)(void *dest, const void *s return dest; } =20 -void *(memset)(void *s, int c, size_t n) -{ - long d0, d1; - - asm volatile ( - "rep stosb" - : "=3D&c" (d0), "=3D&D" (d1) - : "a" (c), "1" (s), "0" (n) - : "memory"); - - return s; -} - void *(memmove)(void *dest, const void *src, size_t n) { long d0, d1, d2; From nobody Sun Feb 8 21:37:16 2026 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) client-ip=192.237.175.120; envelope-from=xen-devel-bounces@lists.xenproject.org; helo=lists.xenproject.org; Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=pass(p=quarantine dis=none) header.from=suse.com ARC-Seal: i=1; a=rsa-sha256; t=1751458641; cv=none; d=zohomail.com; s=zohoarc; b=H4XTsXpeFsjE1mgtOeImX0aTfdQOcqeWPWLz2/Eq+s4QFSd8Ujjh0NgjQDH12ucdoteFoXyVipv1v2xGFgPk4Ox5sZLjDozFMkOYklsVadjqJb6ZSIUIrHSEBwtx0LoeLe/ZR7MmuuPvdv5wAw7TfTGcUrLpKFewncfwBqyb9Ck= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1751458641; h=Content-Type:Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=RZeSTtfnims00cJDfKtOlaDsz46iqIOaCHm3irCBO80=; b=e36CbVkggzZ/nIHjH6wSAQS0b3QaFwLJn0KKYtc59YY2qgchFJJ5M0kKyf0BdR/8TZrl2Jtlm0VAEuDV72iFP6m/FMJkkBUVaNncDvxFacXSkxldTwkS/nBcCCf2T3JaVGiIvGjm1lIazqUGYnww88zotZG/ALd3iONsjnHiMmQ= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=pass header.from= (p=quarantine dis=none) Return-Path: Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) by mx.zohomail.com with SMTPS id 1751458641505890.9486637681161; Wed, 2 Jul 2025 05:17:21 -0700 (PDT) Received: from list by lists.xenproject.org with outflank-mailman.1031016.1404698 (Exim 4.92) (envelope-from ) id 1uWwOl-0001En-Ll; Wed, 02 Jul 2025 12:17:11 +0000 Received: by outflank-mailman (output) from mailman id 1031016.1404698; Wed, 02 Jul 2025 12:17:11 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1uWwOl-0001Eg-HW; Wed, 02 Jul 2025 12:17:11 +0000 Received: by outflank-mailman (input) for mailman id 1031016; Wed, 02 Jul 2025 12:17:10 +0000 Received: from se1-gles-sth1-in.inumbo.com ([159.253.27.254] helo=se1-gles-sth1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1uWwOk-0000ib-9f for xen-devel@lists.xenproject.org; Wed, 02 Jul 2025 12:17:10 +0000 Received: from mail-wr1-x433.google.com (mail-wr1-x433.google.com [2a00:1450:4864:20::433]) by se1-gles-sth1.inumbo.com (Halon) with ESMTPS id 7a1f8bb7-573e-11f0-a313-13f23c93f187; Wed, 02 Jul 2025 14:17:09 +0200 (CEST) Received: by mail-wr1-x433.google.com with SMTP id ffacd0b85a97d-3a54690d369so4118814f8f.3 for ; Wed, 02 Jul 2025 05:17:09 -0700 (PDT) Received: from [10.156.60.236] (ip-037-024-206-209.um08.pools.vodafone-ip.de. [37.24.206.209]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-23acb2f1b25sm134818915ad.63.2025.07.02.05.17.06 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Wed, 02 Jul 2025 05:17:08 -0700 (PDT) X-Outflank-Mailman: Message body and most headers restored to incoming version X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: 7a1f8bb7-573e-11f0-a313-13f23c93f187 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.com; s=google; t=1751458629; x=1752063429; darn=lists.xenproject.org; h=content-transfer-encoding:in-reply-to:autocrypt:content-language :references:cc:to:from:subject:user-agent:mime-version:date :message-id:from:to:cc:subject:date:message-id:reply-to; bh=RZeSTtfnims00cJDfKtOlaDsz46iqIOaCHm3irCBO80=; b=SWhJfc10Sumelm55jwcMTVo2F1jW6/CU/KttdypzDbEdJfiXOzfLapmwzZ3wSMqn0j NgnCOkkY7jWKvKjeDk0w6/dEHN6OHUwU8D8HHHh3bRGqe4vhEOpL7tagJhrCnp8HuFQ1 JwwCfUy3UeA09hiBHqfaHXKujcNOfF3ClszI8TM5FOK/X7VZWP/LrqyrSvBmI2J574Yb J7TYpSCa3pnfV/uQd0qaQnVatQhMJC/4fDrtjc2yAccrvl3/lN9+QcjaIB4WvsyaRaba zjz2hfECTwLtYw5w2MSw2atMHa4ja5+X3Xas2hAFerilwQJsvlUtvwFgPZicoCptmh/w UDUQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1751458629; x=1752063429; h=content-transfer-encoding:in-reply-to:autocrypt:content-language :references:cc:to:from:subject:user-agent:mime-version:date :message-id:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=RZeSTtfnims00cJDfKtOlaDsz46iqIOaCHm3irCBO80=; b=qOFQ8+Q1GtUGVgfWwo3t5ZRaugzZZGGj6ljP79pBKPfM/tm2Usvb+wOpJS2VFqa+FJ 7wbid0fbF1V13ORn/5yVG8BPdDkSgTVOdJabSnUrG3Ml9vQAGo9mCLITs0/2ut2m7iOv OsWpzAFBs3c3BABW7QlJjlecY4biIcP5NM2mmI8kdbEl/9JSLZLV/HZxJll+2Fz/JscL f6l1AGHtm8rdvyE+5GwJ66RpnOglHT7ydN7g0MGgeYnia+C9km8Vwb4WZxPZ+TfFOddf WVyU3qRszv9Jh91qnYXcVw9VBMbQ1kzl444YxhsF4ikYQvMsg9Wcv3VjJrVdsVjs4peP dRBg== X-Gm-Message-State: AOJu0Yx4sRgaaBipe/Np0n0Ot3hsCqc0Z4dQ8uZyfymjKHjRBvNkt0RV /UIa/mpTAnRdZA78ptFBmBrT/X1qZqRSa9YuJA/EQw0iYxrRxHJAZfX7IbpylWDbVp9+ZOpQuuc Quko= X-Gm-Gg: ASbGncvhGLbGlVa1MdahshW/JnLIGUXOziegaewl7U7pNsoQSiuL/744MRzuDSxwVEz aJCQMc1QfmVgR6doiKUxnhsqJb7cNykF0xVtjO/gK+zEvT4iWYWy/s+h+hDVg9E9g9Om4LtYXbF qYQRdcnTj+rjYygs3M1Bg7WGpEBR5mzHj0j+JcwH9bUGsQjsik3kkKaHuaOKlaCId/qm+GLfoZK SDFJHIzH1VRqeT9E4KlArXdvP5yE+EFEIy4ElUeJ5at4h9pDTrfaVAG94ClOosgpOrI2F81p/85 R9zKMdqItmvxA8JYu4VdKFOHPnaR5VAFRUF4PpetLSHhqz9IScOU2ybN2cQVdGrJyLmmIO6VkpP qtvMWbBxMU5jdXsyLxQ7VIkMMweKu1UbRIjiER+IB7g61H7k= X-Google-Smtp-Source: AGHT+IG8uZko8oI04YwbwWvUjWY/Ph7lnYNq6Zzw8cMIdBVS54rMdAaYNmvgkh5Tr8T9zGuQPmkOpg== X-Received: by 2002:a05:6000:310b:b0:3a6:e2d5:f14c with SMTP id ffacd0b85a97d-3b20048e696mr1860341f8f.30.1751458629160; Wed, 02 Jul 2025 05:17:09 -0700 (PDT) Message-ID: Date: Wed, 2 Jul 2025 14:17:02 +0200 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: [PATCH v7 4/7] x86: re-work memcpy() From: Jan Beulich To: "xen-devel@lists.xenproject.org" Cc: Andrew Cooper , =?UTF-8?Q?Roger_Pau_Monn=C3=A9?= References: <0b57c6f0-9fd0-4f8a-93f8-dc494a1e952c@suse.com> Content-Language: en-US Autocrypt: addr=jbeulich@suse.com; keydata= xsDiBFk3nEQRBADAEaSw6zC/EJkiwGPXbWtPxl2xCdSoeepS07jW8UgcHNurfHvUzogEq5xk hu507c3BarVjyWCJOylMNR98Yd8VqD9UfmX0Hb8/BrA+Hl6/DB/eqGptrf4BSRwcZQM32aZK 7Pj2XbGWIUrZrd70x1eAP9QE3P79Y2oLrsCgbZJfEwCgvz9JjGmQqQkRiTVzlZVCJYcyGGsD /0tbFCzD2h20ahe8rC1gbb3K3qk+LpBtvjBu1RY9drYk0NymiGbJWZgab6t1jM7sk2vuf0Py O9Hf9XBmK0uE9IgMaiCpc32XV9oASz6UJebwkX+zF2jG5I1BfnO9g7KlotcA/v5ClMjgo6Gl MDY4HxoSRu3i1cqqSDtVlt+AOVBJBACrZcnHAUSuCXBPy0jOlBhxPqRWv6ND4c9PH1xjQ3NP nxJuMBS8rnNg22uyfAgmBKNLpLgAGVRMZGaGoJObGf72s6TeIqKJo/LtggAS9qAUiuKVnygo 3wjfkS9A3DRO+SpU7JqWdsveeIQyeyEJ/8PTowmSQLakF+3fote9ybzd880fSmFuIEJldWxp Y2ggPGpiZXVsaWNoQHN1c2UuY29tPsJgBBMRAgAgBQJZN5xEAhsDBgsJCAcDAgQVAggDBBYC AwECHgECF4AACgkQoDSui/t3IH4J+wCfQ5jHdEjCRHj23O/5ttg9r9OIruwAn3103WUITZee e7Sbg12UgcQ5lv7SzsFNBFk3nEQQCACCuTjCjFOUdi5Nm244F+78kLghRcin/awv+IrTcIWF hUpSs1Y91iQQ7KItirz5uwCPlwejSJDQJLIS+QtJHaXDXeV6NI0Uef1hP20+y8qydDiVkv6l IreXjTb7DvksRgJNvCkWtYnlS3mYvQ9NzS9PhyALWbXnH6sIJd2O9lKS1Mrfq+y0IXCP10eS FFGg+Av3IQeFatkJAyju0PPthyTqxSI4lZYuJVPknzgaeuJv/2NccrPvmeDg6Coe7ZIeQ8Yj t0ARxu2xytAkkLCel1Lz1WLmwLstV30g80nkgZf/wr+/BXJW/oIvRlonUkxv+IbBM3dX2OV8 AmRv1ySWPTP7AAMFB/9PQK/VtlNUJvg8GXj9ootzrteGfVZVVT4XBJkfwBcpC/XcPzldjv+3 HYudvpdNK3lLujXeA5fLOH+Z/G9WBc5pFVSMocI71I8bT8lIAzreg0WvkWg5V2WZsUMlnDL9 mpwIGFhlbM3gfDMs7MPMu8YQRFVdUvtSpaAs8OFfGQ0ia3LGZcjA6Ik2+xcqscEJzNH+qh8V m5jjp28yZgaqTaRbg3M/+MTbMpicpZuqF4rnB0AQD12/3BNWDR6bmh+EkYSMcEIpQmBM51qM EKYTQGybRCjpnKHGOxG0rfFY1085mBDZCH5Kx0cl0HVJuQKC+dV2ZY5AqjcKwAxpE75MLFkr wkkEGBECAAkFAlk3nEQCGwwACgkQoDSui/t3IH7nnwCfcJWUDUFKdCsBH/E5d+0ZnMQi+G0A nAuWpQkjM1ASeQwSHEeAWPgskBQL In-Reply-To: <0b57c6f0-9fd0-4f8a-93f8-dc494a1e952c@suse.com> Content-Transfer-Encoding: quoted-printable X-ZohoMail-DKIM: pass (identity @suse.com) X-ZM-MESSAGEID: 1751458643732116600 Content-Type: text/plain; charset="utf-8" Move the function to its own assembly file. Having it in C just for the entire body to be an asm() isn't really helpful. Then have two flavors: A "basic" version using qword steps for the bulk of the operation, and an ERMS version for modern hardware, to be substituted in via alternatives patching. Alternatives patching, however, requires an extra precaution: It uses memcpy() itself, and hence the function may patch itself. Luckily the patched-in code only replaces the prolog of the original function. Make sure this remains this way. Signed-off-by: Jan Beulich Reviewed-by: Teddy Astie Reviewed-by: Jason Andryuk --- We may want to consider branching over the REP MOVSQ as well, if the number of qwords turns out to be zero. We may also want to consider using non-REP MOVS{L,W,B} for the tail. TBD: We may further need a workaround similar to Linux'es 8ca97812c3c8 ("x86/mce: Work around an erratum on fast string copy instructions"). TBD: Some older AMD CPUs have an issue with REP MOVS when source and destination are misaligned with one another (modulo 32?), which may require a separate memcpy() flavor. --- v6: Move alternatives.c change to a separate patch. v5: Re-base. v4: Use CR2 write as serializing insn, and limit its use to boot time. v3: Re-base. --- a/xen/arch/x86/Makefile +++ b/xen/arch/x86/Makefile @@ -47,6 +47,7 @@ obj-$(CONFIG_RETURN_THUNK) +=3D indirect-t obj-$(CONFIG_PV) +=3D ioport_emulate.o obj-y +=3D irq.o obj-$(CONFIG_KEXEC) +=3D machine_kexec.o +obj-y +=3D memcpy.o obj-y +=3D memset.o obj-y +=3D mm.o x86_64/mm.o obj-$(CONFIG_VM_EVENT) +=3D monitor.o --- /dev/null +++ b/xen/arch/x86/memcpy.S @@ -0,0 +1,20 @@ +#include + +FUNC(memcpy) + mov %rdx, %rcx + mov %rdi, %rax + /* + * We need to be careful here: memcpy() is involved in alternatives + * patching, so the code doing the actual copying (i.e. past setti= ng + * up registers) may not be subject to patching (unless further + * precautions were taken). + */ + ALTERNATIVE "and $7, %edx; shr $3, %rcx", \ + STR(rep movsb; RET), X86_FEATURE_ERMS + rep movsq + or %edx, %ecx + jz 1f + rep movsb +1: + RET +END(memcpy) --- a/xen/arch/x86/string.c +++ b/xen/arch/x86/string.c @@ -7,21 +7,6 @@ =20 #include =20 -void *(memcpy)(void *dest, const void *src, size_t n) -{ - long d0, d1, d2; - - asm volatile ( - " rep ; movs"__OS" ; " - " mov %k4,%k3 ; " - " rep ; movsb " - : "=3D&c" (d0), "=3D&D" (d1), "=3D&S" (d2) - : "0" (n/BYTES_PER_LONG), "r" (n%BYTES_PER_LONG), "1" (dest), "2" = (src) - : "memory" ); - - return dest; -} - void *(memmove)(void *dest, const void *src, size_t n) { long d0, d1, d2; From nobody Sun Feb 8 21:37:16 2026 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) client-ip=192.237.175.120; envelope-from=xen-devel-bounces@lists.xenproject.org; helo=lists.xenproject.org; Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=pass(p=quarantine dis=none) header.from=suse.com ARC-Seal: i=1; a=rsa-sha256; t=1751458698; cv=none; d=zohomail.com; s=zohoarc; b=VGFZ8ATL1GVUGZuP/T1/bKOYpqrYE27NFtHTuK1GlMDT1XEwNbVojdH+JqIxeQDF8GBTwsaRUP/TQl1sQWSc1iRoOQhDQKeKDeORO9sCPbj9Kg9aGqQavf1NQ8i8FqXxDA4Q8IsMb3bIxWxEtugoCMjjF6X5DzQjNr4JSOiXa8Y= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1751458698; h=Content-Type:Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=Z7gcE9ICgrUlPBGCD5woSZRLeZLOTdIQyX79rN8wCOY=; b=BdEkHeh0L0qL6vcxiSx+U6mgQXhXfYDLjjrWe0pJBqyq7L2KAO3T5Geb1Jg49Xf0nrfQS40kMP5YdQBIvOvkV0yacO9Fy2RdhcdKYhE9iYN3pvD6du8MNr+KluqnMKhwWUsZIci1UD0CMRXUnetuKr2TdlwkcsgAPPUosNvPHUA= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=pass header.from= (p=quarantine dis=none) Return-Path: Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) by mx.zohomail.com with SMTPS id 17514586984061023.0975630523491; Wed, 2 Jul 2025 05:18:18 -0700 (PDT) Received: from list by lists.xenproject.org with outflank-mailman.1031026.1404707 (Exim 4.92) (envelope-from ) id 1uWwPE-0001od-RN; Wed, 02 Jul 2025 12:17:40 +0000 Received: by outflank-mailman (output) from mailman id 1031026.1404707; Wed, 02 Jul 2025 12:17:40 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1uWwPE-0001oW-Oo; Wed, 02 Jul 2025 12:17:40 +0000 Received: by outflank-mailman (input) for mailman id 1031026; Wed, 02 Jul 2025 12:17:38 +0000 Received: from se1-gles-sth1-in.inumbo.com ([159.253.27.254] helo=se1-gles-sth1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1uWwPC-0000ib-Qa for xen-devel@lists.xenproject.org; Wed, 02 Jul 2025 12:17:38 +0000 Received: from mail-wr1-x432.google.com (mail-wr1-x432.google.com [2a00:1450:4864:20::432]) by se1-gles-sth1.inumbo.com (Halon) with ESMTPS id 8b0bc667-573e-11f0-a313-13f23c93f187; Wed, 02 Jul 2025 14:17:38 +0200 (CEST) Received: by mail-wr1-x432.google.com with SMTP id ffacd0b85a97d-3a6e8b1fa37so3773906f8f.2 for ; Wed, 02 Jul 2025 05:17:38 -0700 (PDT) Received: from [10.156.60.236] (ip-037-024-206-209.um08.pools.vodafone-ip.de. [37.24.206.209]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-b34e31da5f2sm12760548a12.58.2025.07.02.05.17.30 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Wed, 02 Jul 2025 05:17:36 -0700 (PDT) X-Outflank-Mailman: Message body and most headers restored to incoming version X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: 8b0bc667-573e-11f0-a313-13f23c93f187 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.com; s=google; t=1751458658; x=1752063458; darn=lists.xenproject.org; h=content-transfer-encoding:in-reply-to:autocrypt:content-language :references:cc:to:from:subject:user-agent:mime-version:date :message-id:from:to:cc:subject:date:message-id:reply-to; bh=Z7gcE9ICgrUlPBGCD5woSZRLeZLOTdIQyX79rN8wCOY=; b=S7d2r1Qv+OGD/m5BsL9f4gUfsQPUZ/Ve1rWnw2qa3e6dpu+UdCdvySzuqhJKHgshct vZ6cTP6lcfJdml/TDA8XDLTnm6thquMh0G9ykyyZAW0cz0q7iNk9NV+/Pi5X/7vYXna+ unf82t+uIboO/Bgh507uAAXDkkirdrcFjTNiGOYjen2gwt1QjByOQvKFSjAQITABEXYk re8092BhJUeF3tFVnpcuMtJ2ZJxgHy/mTA3vtijk6+mDTO84b0R43OWSsw6MD+QyxDpN Iuk52u59BGlXls6hj6Sufp92lrU9MHuiK4VYBTT6RkY+lJJs2/q62dnvR/dhBrmZGMw5 L/Eg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1751458658; x=1752063458; h=content-transfer-encoding:in-reply-to:autocrypt:content-language :references:cc:to:from:subject:user-agent:mime-version:date :message-id:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=Z7gcE9ICgrUlPBGCD5woSZRLeZLOTdIQyX79rN8wCOY=; b=gRtdNao01nahn3xJhbR02rSX2NdjvwKrdehtcjIxSQRY9Go013RMOEQnYGBoFilPxs 31WrT0SGlO6Uh4MW4X/soE7J/WMN0terX5Nj582JsyMBNYZ4sqZH8J3JFG9DSzwecIkz utb9qnI7bIK+EdlGc/izhYGn+3ZXiC4NM0U1u+jVbT4K94yDDcyV7rF28EPpa8ScVtrM A9jwI2KGLMJmWzZFUz0dpv6RhAMjZHW1K1eKeufj0I338DfmkymsVdif658qDPd5r9TW B1wEpDo1VuXSuHm8XF0TlZJH6FR3xoKvvpQEyzyQ5qbAqzDds1O9mZJiBX1TUjKSLg3H koRg== X-Gm-Message-State: AOJu0YwNk9lHb8/SsaLKwyRgmMGfHf3hxf4Lf8TUkdZ5FTqGaJNxrybe IVJl8tyNHMfRyI7HrSUmv7GCTRottTiRflMImsfCcYmQWtCGm0/g6C+sf00OG7+0LAsu8/mrW+h zeu4= X-Gm-Gg: ASbGncsHlAIKVmR8a5ZpD5M2B2JS3xHXjTKRO/EqpxPLnyqVxxsFEMeuXBw7p01M3xd HKtbEmKE54SaVie+zZcOBVrhyvmS71ZejyxpIqjmJvbeTLZQaWfm2QOetfcr6EifKDNBdoCjMOe bt95Hae0NvkbPZFi0IA+pBqd/9LHwCT4QHjNfJL2qyJaeMUgpxTSYa1+YotbYnrazpn+ZY3eBLG cYzQq4BG9ovvyxOL1Ae1XNKhorv0ETghqTEIwdCmIhppdBeuIbbVUp5+OxBSc9Plcp8jq5zZ/uo 4QHBFDdwUHgwKO6rfMpggkxMstkAIcPdxu5RGjytzdzORNfQLoV2DqaLzAHmv1dnqLtKd8lxaXA H1OveA0WI57YReCTW3dYXXMS9Lu1OlIR8/DpfsmI2hnSHopc= X-Google-Smtp-Source: AGHT+IHzoH1tP4FyHzzh7758qKumA1283hbOfoPr+w6nt2GvdfwYzadlKiKpp2QZHERG0VmVfkf8ig== X-Received: by 2002:a05:6000:481c:b0:3a4:e68e:d33c with SMTP id ffacd0b85a97d-3b200e29fe0mr2111662f8f.47.1751458657562; Wed, 02 Jul 2025 05:17:37 -0700 (PDT) Message-ID: <7acea351-2a87-461b-9423-e157f1850450@suse.com> Date: Wed, 2 Jul 2025 14:17:26 +0200 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: [PATCH v7 5/7] x86: control memset() and memcpy() inlining From: Jan Beulich To: "xen-devel@lists.xenproject.org" Cc: Andrew Cooper , =?UTF-8?Q?Roger_Pau_Monn=C3=A9?= References: <0b57c6f0-9fd0-4f8a-93f8-dc494a1e952c@suse.com> Content-Language: en-US Autocrypt: addr=jbeulich@suse.com; keydata= xsDiBFk3nEQRBADAEaSw6zC/EJkiwGPXbWtPxl2xCdSoeepS07jW8UgcHNurfHvUzogEq5xk hu507c3BarVjyWCJOylMNR98Yd8VqD9UfmX0Hb8/BrA+Hl6/DB/eqGptrf4BSRwcZQM32aZK 7Pj2XbGWIUrZrd70x1eAP9QE3P79Y2oLrsCgbZJfEwCgvz9JjGmQqQkRiTVzlZVCJYcyGGsD /0tbFCzD2h20ahe8rC1gbb3K3qk+LpBtvjBu1RY9drYk0NymiGbJWZgab6t1jM7sk2vuf0Py O9Hf9XBmK0uE9IgMaiCpc32XV9oASz6UJebwkX+zF2jG5I1BfnO9g7KlotcA/v5ClMjgo6Gl MDY4HxoSRu3i1cqqSDtVlt+AOVBJBACrZcnHAUSuCXBPy0jOlBhxPqRWv6ND4c9PH1xjQ3NP nxJuMBS8rnNg22uyfAgmBKNLpLgAGVRMZGaGoJObGf72s6TeIqKJo/LtggAS9qAUiuKVnygo 3wjfkS9A3DRO+SpU7JqWdsveeIQyeyEJ/8PTowmSQLakF+3fote9ybzd880fSmFuIEJldWxp Y2ggPGpiZXVsaWNoQHN1c2UuY29tPsJgBBMRAgAgBQJZN5xEAhsDBgsJCAcDAgQVAggDBBYC AwECHgECF4AACgkQoDSui/t3IH4J+wCfQ5jHdEjCRHj23O/5ttg9r9OIruwAn3103WUITZee e7Sbg12UgcQ5lv7SzsFNBFk3nEQQCACCuTjCjFOUdi5Nm244F+78kLghRcin/awv+IrTcIWF hUpSs1Y91iQQ7KItirz5uwCPlwejSJDQJLIS+QtJHaXDXeV6NI0Uef1hP20+y8qydDiVkv6l IreXjTb7DvksRgJNvCkWtYnlS3mYvQ9NzS9PhyALWbXnH6sIJd2O9lKS1Mrfq+y0IXCP10eS FFGg+Av3IQeFatkJAyju0PPthyTqxSI4lZYuJVPknzgaeuJv/2NccrPvmeDg6Coe7ZIeQ8Yj t0ARxu2xytAkkLCel1Lz1WLmwLstV30g80nkgZf/wr+/BXJW/oIvRlonUkxv+IbBM3dX2OV8 AmRv1ySWPTP7AAMFB/9PQK/VtlNUJvg8GXj9ootzrteGfVZVVT4XBJkfwBcpC/XcPzldjv+3 HYudvpdNK3lLujXeA5fLOH+Z/G9WBc5pFVSMocI71I8bT8lIAzreg0WvkWg5V2WZsUMlnDL9 mpwIGFhlbM3gfDMs7MPMu8YQRFVdUvtSpaAs8OFfGQ0ia3LGZcjA6Ik2+xcqscEJzNH+qh8V m5jjp28yZgaqTaRbg3M/+MTbMpicpZuqF4rnB0AQD12/3BNWDR6bmh+EkYSMcEIpQmBM51qM EKYTQGybRCjpnKHGOxG0rfFY1085mBDZCH5Kx0cl0HVJuQKC+dV2ZY5AqjcKwAxpE75MLFkr wkkEGBECAAkFAlk3nEQCGwwACgkQoDSui/t3IH7nnwCfcJWUDUFKdCsBH/E5d+0ZnMQi+G0A nAuWpQkjM1ASeQwSHEeAWPgskBQL In-Reply-To: <0b57c6f0-9fd0-4f8a-93f8-dc494a1e952c@suse.com> Content-Transfer-Encoding: quoted-printable X-ZohoMail-DKIM: pass (identity @suse.com) X-ZM-MESSAGEID: 1751458701692116600 Content-Type: text/plain; charset="utf-8" Stop the compiler from inlining non-trivial memset() and memcpy() (for memset() see e.g. map_vcpu_info() or kimage_load_segments() for examples). This way we even keep the compiler from using REP STOSQ / REP MOVSQ when we'd prefer REP STOSB / REP MOVSB (when ERMS is available). With gcc10 this yields a modest .text size reduction (release build) of around 2k. Unfortunately these options aren't understood by the clang versions I have readily available for testing with; I'm unaware of equivalents. Note also that using cc-option-add is not an option here, or at least I couldn't make things work with it (in case the option was not supported by the compiler): The embedded comma in the option looks to be getting in the way. Requested-by: Andrew Cooper Signed-off-by: Jan Beulich Reviewed-by: Jason Andryuk --- v3: Re-base. v2: New. --- The boundary values are of course up for discussion - I wasn't really certain whether to use 16 or 32; I'd be less certain about using yet larger values. A respective Linux patch [1] uses 256 ... Similarly whether to permit the compiler to emit REP STOSQ / REP MOVSQ for known size, properly aligned blocks is up for discussion. [1] https://lore.kernel.org/lkml/20250605164733.737543-1-mjguzik@gmail.com/= T/#u --- a/xen/arch/x86/arch.mk +++ b/xen/arch/x86/arch.mk @@ -58,6 +58,9 @@ endif $(call cc-option-add,CFLAGS_stack_boundary,CC,-mpreferred-stack-boundary= =3D3) export CFLAGS_stack_boundary =20 +CFLAGS +=3D $(call cc-option,$(CC),-mmemcpy-strategy=3Dunrolled_loop:16:no= align$(comma)libcall:-1:noalign) +CFLAGS +=3D $(call cc-option,$(CC),-mmemset-strategy=3Dunrolled_loop:16:no= align$(comma)libcall:-1:noalign) + ifeq ($(CONFIG_UBSAN),y) # Don't enable alignment sanitisation. x86 has efficient unaligned access= es, # and various things (ACPI tables, hypercall pages, stubs, etc) are wont-f= ix. From nobody Sun Feb 8 21:37:16 2026 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) client-ip=192.237.175.120; envelope-from=xen-devel-bounces@lists.xenproject.org; helo=lists.xenproject.org; Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=pass(p=quarantine dis=none) header.from=suse.com ARC-Seal: i=1; a=rsa-sha256; t=1751458701; cv=none; d=zohomail.com; s=zohoarc; b=iIjzdZ3i1Xd/l2m1+btAOB6RODjDbefLeH4te61mg+RUsz707pAZo8lRpH/5aKOCRI74ifAlkP7ahJUZDGLhIc7Tcxs+HsJ+daDX5Cwv1WIcnyqV35kkOGuBz9Li5vwgX4uYBPgnb8wO89fUF8zmgdA8e9RWssxoiQM+fGCWlHo= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1751458701; h=Content-Type:Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=gDGtlTyrnPjFWjh7qBWn4eYhkW3UxiX63jCVI/KDses=; b=TZnTLJTJ5+Tr6u9vRPabhErpnXBnj99VwId01eI2zP7skyymIe26X/ytzbfQkhFq4SMraO+LZTEHkz8znoi9ZnJ6PiYe9ADZq3SVlTa40zVonf7Q0CegskzOjf/5jK+FWASKPonIXGyMXJu+YvUXCZ0X4WC3UrZ+hboOJ034WIw= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=pass header.from= (p=quarantine dis=none) Return-Path: Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) by mx.zohomail.com with SMTPS id 1751458701456494.8628833838834; Wed, 2 Jul 2025 05:18:21 -0700 (PDT) Received: from list by lists.xenproject.org with outflank-mailman.1031032.1404718 (Exim 4.92) (envelope-from ) id 1uWwPh-0002J0-2z; Wed, 02 Jul 2025 12:18:09 +0000 Received: by outflank-mailman (output) from mailman id 1031032.1404718; Wed, 02 Jul 2025 12:18:09 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1uWwPh-0002It-02; Wed, 02 Jul 2025 12:18:09 +0000 Received: by outflank-mailman (input) for mailman id 1031032; Wed, 02 Jul 2025 12:18:07 +0000 Received: from se1-gles-sth1-in.inumbo.com ([159.253.27.254] helo=se1-gles-sth1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1uWwPf-0000ib-Jw for xen-devel@lists.xenproject.org; Wed, 02 Jul 2025 12:18:07 +0000 Received: from mail-wm1-x331.google.com (mail-wm1-x331.google.com [2a00:1450:4864:20::331]) by se1-gles-sth1.inumbo.com (Halon) with ESMTPS id 9c28960d-573e-11f0-a313-13f23c93f187; Wed, 02 Jul 2025 14:18:07 +0200 (CEST) Received: by mail-wm1-x331.google.com with SMTP id 5b1f17b1804b1-451d54214adso48220795e9.3 for ; Wed, 02 Jul 2025 05:18:06 -0700 (PDT) Received: from [10.156.60.236] (ip-037-024-206-209.um08.pools.vodafone-ip.de. [37.24.206.209]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-23acb3c6e09sm128147445ad.227.2025.07.02.05.18.03 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Wed, 02 Jul 2025 05:18:05 -0700 (PDT) X-Outflank-Mailman: Message body and most headers restored to incoming version X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: 9c28960d-573e-11f0-a313-13f23c93f187 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.com; s=google; t=1751458686; x=1752063486; darn=lists.xenproject.org; h=content-transfer-encoding:in-reply-to:autocrypt:content-language :references:cc:to:from:subject:user-agent:mime-version:date :message-id:from:to:cc:subject:date:message-id:reply-to; bh=gDGtlTyrnPjFWjh7qBWn4eYhkW3UxiX63jCVI/KDses=; b=HBhrQT//sXU0Hdxmz/9yS157KJ118AKM39ImRcN/fK4RaU8hIw9x+5Z4u8F6NwCzS3 7hTVjxit+PgJlAFAPK2PPh/km8XZ+CoLt0mKTtH6iTeJDcz0ku9/FV60oVW3wS3LXUnQ mssVj2kt/sKdi95ltZlGd+Um3lZzTzvDsNXZVat00n8VszpNxhJ7WtvBdJXDJvAO/Eud ut5cqPdPGc4eGWLjiR0Z0obk34DdEmCm+CHZmNkB3goLb1Xvh90kvDZEd3368TqD9u1c McFcmzzvavU+oCeBfvRvjVsJla00+HgOl28r5zP0T3B0XPtFSl6utFsFT02tAnFeTftp 60Nw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1751458686; x=1752063486; h=content-transfer-encoding:in-reply-to:autocrypt:content-language :references:cc:to:from:subject:user-agent:mime-version:date :message-id:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=gDGtlTyrnPjFWjh7qBWn4eYhkW3UxiX63jCVI/KDses=; b=M7WrHm7pZkm8Upornx3n5kaKnyVEPVojj7jsvKwFN43Z86ZOWlajs3wSW8fHyBYCll sKWcWhk1vRge9N2GpHWxZ3qYuBoqZCv1dEn8DhwTo3KRyGBx5gtnIh523VOKF9yZmRvX BODExEuo0gcYbyQu1l049kyFtrAxmBNBtUxt75b05QP1iBxqUQz65FCkmU8SHmgficoy eiZPv1V2tyou30tv/VR4yflqTvsD8pixJt7K94r3jOODlG10H3viG3QBxnmP0DZsj0J5 /xvokWuGMmzAJRK6AeAmpN1b95OqRbMHocIJNxBcSxpQzSP1GkTV3t2gViyvFDVNowZ7 7+OA== X-Gm-Message-State: AOJu0Yy/y4cCTIYV6JwpGJ+YlX2O25JBK6tyL1H/yqaPsFoX6KB/naAT 8XoGwIRYBeEm24y/+svFr4rzWuUAYyAWepGQwV0EyrOynefEfpD06G/yl4plhBDeQJdPhh8hPSD fOfU= X-Gm-Gg: ASbGnct2xXYmpriVun87MJI8GCL5jIGhE17QQdNjpXgB603m+mumLKwIltrNXE93KIs qeIrhYkR8YTNewDBGtHXzhRE2A51kkK+XZDpazkFqP0XObcioU9R8Ybwkfj/rFjzFQwxJFkjvN3 AOrIE6x6Aqj7kxAzN0Qlpr66E65ADwnnigHVX/ZVyW69HHoZK8ETUe47d9oathfiCt/1Z1IeJlW 8nrQRyClAQd1X3hVCzTl2unhOOb6zshpN55tYd/CUg9iZGkNs6fMCK+72/1zG54kA3DBOJVpzLn n1wlzXhaNo0qoED+PIIuv4Sm4XboODn9yqG2c5YoLEoqaX3pXUbkvWXcWOqCnLg5CvgbO9AE6Z3 JiigAwRQSdrssfdEqiW+vpxvbZhtlyzsttt+FhMkA7ayp2pE= X-Google-Smtp-Source: AGHT+IEhGfU8kcE5jNxKKY4ZypA5Juy+BK8XfGut3MtTZLHyMcTKN56mhm0UsETKXDKYgFuEYSJrXA== X-Received: by 2002:a5d:5f81:0:b0:3a4:c2e4:11b with SMTP id ffacd0b85a97d-3b20095cf64mr2118462f8f.51.1751458686134; Wed, 02 Jul 2025 05:18:06 -0700 (PDT) Message-ID: <65db1bac-7f37-44f5-a727-cc5e344eb320@suse.com> Date: Wed, 2 Jul 2025 14:17:59 +0200 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: [PATCH v7 6/7] x86: introduce "hot" and "cold" page clearing functions From: Jan Beulich To: "xen-devel@lists.xenproject.org" Cc: Andrew Cooper , =?UTF-8?Q?Roger_Pau_Monn=C3=A9?= References: <0b57c6f0-9fd0-4f8a-93f8-dc494a1e952c@suse.com> Content-Language: en-US Autocrypt: addr=jbeulich@suse.com; keydata= xsDiBFk3nEQRBADAEaSw6zC/EJkiwGPXbWtPxl2xCdSoeepS07jW8UgcHNurfHvUzogEq5xk hu507c3BarVjyWCJOylMNR98Yd8VqD9UfmX0Hb8/BrA+Hl6/DB/eqGptrf4BSRwcZQM32aZK 7Pj2XbGWIUrZrd70x1eAP9QE3P79Y2oLrsCgbZJfEwCgvz9JjGmQqQkRiTVzlZVCJYcyGGsD /0tbFCzD2h20ahe8rC1gbb3K3qk+LpBtvjBu1RY9drYk0NymiGbJWZgab6t1jM7sk2vuf0Py O9Hf9XBmK0uE9IgMaiCpc32XV9oASz6UJebwkX+zF2jG5I1BfnO9g7KlotcA/v5ClMjgo6Gl MDY4HxoSRu3i1cqqSDtVlt+AOVBJBACrZcnHAUSuCXBPy0jOlBhxPqRWv6ND4c9PH1xjQ3NP nxJuMBS8rnNg22uyfAgmBKNLpLgAGVRMZGaGoJObGf72s6TeIqKJo/LtggAS9qAUiuKVnygo 3wjfkS9A3DRO+SpU7JqWdsveeIQyeyEJ/8PTowmSQLakF+3fote9ybzd880fSmFuIEJldWxp Y2ggPGpiZXVsaWNoQHN1c2UuY29tPsJgBBMRAgAgBQJZN5xEAhsDBgsJCAcDAgQVAggDBBYC AwECHgECF4AACgkQoDSui/t3IH4J+wCfQ5jHdEjCRHj23O/5ttg9r9OIruwAn3103WUITZee e7Sbg12UgcQ5lv7SzsFNBFk3nEQQCACCuTjCjFOUdi5Nm244F+78kLghRcin/awv+IrTcIWF hUpSs1Y91iQQ7KItirz5uwCPlwejSJDQJLIS+QtJHaXDXeV6NI0Uef1hP20+y8qydDiVkv6l IreXjTb7DvksRgJNvCkWtYnlS3mYvQ9NzS9PhyALWbXnH6sIJd2O9lKS1Mrfq+y0IXCP10eS FFGg+Av3IQeFatkJAyju0PPthyTqxSI4lZYuJVPknzgaeuJv/2NccrPvmeDg6Coe7ZIeQ8Yj t0ARxu2xytAkkLCel1Lz1WLmwLstV30g80nkgZf/wr+/BXJW/oIvRlonUkxv+IbBM3dX2OV8 AmRv1ySWPTP7AAMFB/9PQK/VtlNUJvg8GXj9ootzrteGfVZVVT4XBJkfwBcpC/XcPzldjv+3 HYudvpdNK3lLujXeA5fLOH+Z/G9WBc5pFVSMocI71I8bT8lIAzreg0WvkWg5V2WZsUMlnDL9 mpwIGFhlbM3gfDMs7MPMu8YQRFVdUvtSpaAs8OFfGQ0ia3LGZcjA6Ik2+xcqscEJzNH+qh8V m5jjp28yZgaqTaRbg3M/+MTbMpicpZuqF4rnB0AQD12/3BNWDR6bmh+EkYSMcEIpQmBM51qM EKYTQGybRCjpnKHGOxG0rfFY1085mBDZCH5Kx0cl0HVJuQKC+dV2ZY5AqjcKwAxpE75MLFkr wkkEGBECAAkFAlk3nEQCGwwACgkQoDSui/t3IH7nnwCfcJWUDUFKdCsBH/E5d+0ZnMQi+G0A nAuWpQkjM1ASeQwSHEeAWPgskBQL In-Reply-To: <0b57c6f0-9fd0-4f8a-93f8-dc494a1e952c@suse.com> Content-Transfer-Encoding: quoted-printable X-ZohoMail-DKIM: pass (identity @suse.com) X-ZM-MESSAGEID: 1751458703836116600 Content-Type: text/plain; charset="utf-8" The present clear_page_sse2() is useful in case a page isn't going to get touched again soon, or if we want to limit churn on the caches. Amend it by alternatively using CLZERO, which has been found to be quite a bit faster on Zen2 hardware at least. Note that to use CLZERO, we need to know the cache line size, and hence a feature dependency on CLFLUSH gets introduced. For cases where latency is the most important aspect, or when it is expected that sufficiently large parts of a page will get accessed again soon after the clearing, introduce a "hot" alternative. Again use alternatives patching to select between a "legacy" and an ERMS variant. Don't switch any callers just yet - this will be the subject of subsequent changes. Signed-off-by: Jan Beulich Reviewed-by: Jason Andryuk --- v5: Re-base. v3: Re-base. v2: New. --- Note: Ankur indicates that for ~L3-size or larger regions MOVNT/CLZERO is better even latency-wise. --- a/xen/arch/x86/clear_page.S +++ b/xen/arch/x86/clear_page.S @@ -5,7 +5,7 @@ #include #include =20 -FUNC(clear_page_sse2) + .macro clear_page_sse2 mov $PAGE_SIZE/32, %ecx xor %eax,%eax =20 @@ -19,4 +19,42 @@ FUNC(clear_page_sse2) =20 sfence RET -END(clear_page_sse2) + .endm + + .macro clear_page_clzero + mov %rdi, %rax + mov $PAGE_SIZE/64, %ecx + .globl clear_page_clzero_post_count +clear_page_clzero_post_count: + +0: clzero + sub $-64, %rax + .globl clear_page_clzero_post_neg_size +clear_page_clzero_post_neg_size: + sub $1, %ecx + jnz 0b + + sfence + RET + .endm + +FUNC(clear_page_cold) + ALTERNATIVE clear_page_sse2, clear_page_clzero, X86_FEATURE_CLZERO +END(clear_page_cold) + + .macro clear_page_stosb + mov $PAGE_SIZE, %ecx + xor %eax,%eax + rep stosb + .endm + + .macro clear_page_stosq + mov $PAGE_SIZE/8, %ecx + xor %eax, %eax + rep stosq + .endm + +FUNC(clear_page_hot) + ALTERNATIVE clear_page_stosq, clear_page_stosb, X86_FEATURE_ERMS + RET +END(clear_page_hot) --- a/xen/arch/x86/cpu/common.c +++ b/xen/arch/x86/cpu/common.c @@ -60,6 +60,9 @@ DEFINE_PER_CPU(bool, full_gdt_loaded); =20 DEFINE_PER_CPU(uint32_t, pkrs); =20 +extern uint32_t clear_page_clzero_post_count[]; +extern int8_t clear_page_clzero_post_neg_size[]; + void __init setup_clear_cpu_cap(unsigned int cap) { const uint32_t *dfs; @@ -357,8 +360,38 @@ void __init early_cpu_init(bool verbose) =20 edx &=3D ~cleared_caps[FEATURESET_1d]; ecx &=3D ~cleared_caps[FEATURESET_1c]; - if (edx & cpufeat_mask(X86_FEATURE_CLFLUSH)) - c->x86_cache_alignment =3D ((ebx >> 8) & 0xff) * 8; + if (edx & cpufeat_mask(X86_FEATURE_CLFLUSH)) { + unsigned int size =3D ((ebx >> 8) & 0xff) * 8; + + c->x86_cache_alignment =3D size; + + /* + * Patch in parameters of clear_page_cold()'s CLZERO + * alternative. Note that for now we cap this at 128 bytes. + * Larger cache line sizes would still be dealt with + * correctly, but would cause redundant work done. + */ + if (size > 128) + size =3D 128; + if (size && !(size & (size - 1))) { + /* + * Need to play some games to keep the compiler from + * recognizing the negative array index as being out + * of bounds. The labels in assembler code really are + * _after_ the locations to be patched, so the + * negative index is intentional. + */ + uint32_t *pcount =3D clear_page_clzero_post_count; + int8_t *neg_size =3D clear_page_clzero_post_neg_size; + + OPTIMIZER_HIDE_VAR(pcount); + OPTIMIZER_HIDE_VAR(neg_size); + pcount[-1] =3D PAGE_SIZE / size; + neg_size[-1] =3D -size; + } + else + setup_clear_cpu_cap(X86_FEATURE_CLZERO); + } /* Leaf 0x1 capabilities filled in early for Xen. */ c->x86_capability[FEATURESET_1d] =3D edx; c->x86_capability[FEATURESET_1c] =3D ecx; --- a/xen/arch/x86/include/asm/asm-defns.h +++ b/xen/arch/x86/include/asm/asm-defns.h @@ -1,5 +1,9 @@ #include =20 +.macro clzero + .byte 0x0f, 0x01, 0xfc +.endm + /* * Call a noreturn function. This could be JMP, but CALL results in a more * helpful backtrace. BUG is to catch functions which do decide to return= ... --- a/xen/arch/x86/include/asm/page.h +++ b/xen/arch/x86/include/asm/page.h @@ -219,10 +219,11 @@ typedef struct { u64 pfn; } pagetable_t; #define pagetable_from_paddr(p) pagetable_from_pfn((p)>>PAGE_SHIFT) #define pagetable_null() pagetable_from_pfn(0) =20 -void clear_page_sse2(void *pg); +void clear_page_hot(void *pg); +void clear_page_cold(void *pg); void copy_page_sse2(void *to, const void *from); =20 -#define clear_page(_p) clear_page_sse2(_p) +#define clear_page(_p) clear_page_cold(_p) #define copy_page(_t, _f) copy_page_sse2(_t, _f) =20 /* Convert between Xen-heap virtual addresses and machine addresses. */ --- a/xen/tools/gen-cpuid.py +++ b/xen/tools/gen-cpuid.py @@ -212,6 +212,10 @@ def crunch_numbers(state): # the first place. APIC: [X2APIC, TSC_DEADLINE, EXTAPIC], =20 + # The CLZERO insn requires a means to determine the cache line siz= e, + # which is tied to the CLFLUSH insn. + CLFLUSH: [CLZERO], + # AMD built MMXExtentions and 3DNow as extentions to MMX. MMX: [MMXEXT, _3DNOW], From nobody Sun Feb 8 21:37:16 2026 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) client-ip=192.237.175.120; envelope-from=xen-devel-bounces@lists.xenproject.org; helo=lists.xenproject.org; Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=pass(p=quarantine dis=none) header.from=suse.com ARC-Seal: i=1; a=rsa-sha256; t=1751458789; cv=none; d=zohomail.com; s=zohoarc; b=RPs7sLnbnCpcsyBPAiguE8M4vlf9G5uKZajCxfmPRLPWaaY3m1M6PyEug9OcPnvHwhl8eQdAOT+0Ngvzqnl6uDsBoeQoCq32r9sckx6bNM1Y0Tb+TghVGmLi1bCY/WTUniCqEcDmY5PKJOeE9hkbDndyRuaA5RRwt744CZNODaY= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1751458789; h=Content-Type:Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=RrahAouvwtmbuX3ET80MXFLODL8AuIbd/jGTYB0L7n8=; b=baTi6kHZK/P1svveII1aheuf3acpE6IahzycJWDeuoY9QJExES2R07uuvHVhw4DFlu6DE5rIZT/ch2m/6NSzt9lMjil4PFQ8bjoCokbqQKYknIk/5GLZncQcSlGDGQDm+j8UMInnvhK6HfC8/KHX98JA51Pk9bOwjuHESRhaTSw= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=pass header.from= (p=quarantine dis=none) Return-Path: Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) by mx.zohomail.com with SMTPS id 1751458789220396.97383674055664; Wed, 2 Jul 2025 05:19:49 -0700 (PDT) Received: from list by lists.xenproject.org with outflank-mailman.1031036.1404728 (Exim 4.92) (envelope-from ) id 1uWwR5-0002vk-E5; Wed, 02 Jul 2025 12:19:35 +0000 Received: by outflank-mailman (output) from mailman id 1031036.1404728; Wed, 02 Jul 2025 12:19:35 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1uWwR5-0002vd-An; Wed, 02 Jul 2025 12:19:35 +0000 Received: by outflank-mailman (input) for mailman id 1031036; Wed, 02 Jul 2025 12:19:34 +0000 Received: from se1-gles-sth1-in.inumbo.com ([159.253.27.254] helo=se1-gles-sth1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1uWwR4-0002vT-Ir for xen-devel@lists.xenproject.org; Wed, 02 Jul 2025 12:19:34 +0000 Received: from mail-wr1-x42a.google.com (mail-wr1-x42a.google.com [2a00:1450:4864:20::42a]) by se1-gles-sth1.inumbo.com (Halon) with ESMTPS id cfcbbb5d-573e-11f0-a313-13f23c93f187; Wed, 02 Jul 2025 14:19:33 +0200 (CEST) Received: by mail-wr1-x42a.google.com with SMTP id ffacd0b85a97d-3a54690d369so4122075f8f.3 for ; Wed, 02 Jul 2025 05:19:33 -0700 (PDT) Received: from [10.156.60.236] (ip-037-024-206-209.um08.pools.vodafone-ip.de. [37.24.206.209]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-23acb3c3985sm129064455ad.209.2025.07.02.05.19.26 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Wed, 02 Jul 2025 05:19:32 -0700 (PDT) X-Outflank-Mailman: Message body and most headers restored to incoming version X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: cfcbbb5d-573e-11f0-a313-13f23c93f187 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.com; s=google; t=1751458773; x=1752063573; darn=lists.xenproject.org; h=content-transfer-encoding:in-reply-to:autocrypt:content-language :references:cc:to:from:subject:user-agent:mime-version:date :message-id:from:to:cc:subject:date:message-id:reply-to; bh=RrahAouvwtmbuX3ET80MXFLODL8AuIbd/jGTYB0L7n8=; b=NV/5pySqoJ5dUI0U9IBi+AnebXMA2Wmihy2Sqwk6IBGwd5MVPCfmPwiAGrbuR6SEL+ q5X624LJfuEAP5qOSPY56cL3DsCnzVa0MNJh7lO15HfkdYf0hl01Rx5VDnKvg6zwx1g4 bwi3bOY5DeZ2k4DzoVomrI55ugmrlOFkf/lqZ0pqH0Lw5hSnKCxNmoEWmPueIm/yACUU V7npSuMS1fBw9M6p5w2NJbY3A3QnoXaPGrxpxYAukY0A+LC+8b8KiZ1nytaIxJT7qAcT 3rMmxffTvC/V0rw14ImMAN6JCuj2SLrMneEvXmxyfGdFBdGn0e7Dr0sSusdNGDf8ZqrW fTLg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1751458773; x=1752063573; h=content-transfer-encoding:in-reply-to:autocrypt:content-language :references:cc:to:from:subject:user-agent:mime-version:date :message-id:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=RrahAouvwtmbuX3ET80MXFLODL8AuIbd/jGTYB0L7n8=; b=C3kpQcn+3XJH2UcumxApVhykadPm8hptbuiPUQKLa5KTBLsKdGKRI+QT6HK/ntPd71 xUso0yGvdqbiRGPJ79T8ff34/DTNVTJ8XYfHPvBBy3qYvBxhq8kx4uArWdawSNnN+6d4 WSxpSNGCqJafebgQS8L/mvAxFjh2QTaa8tDrrvjMTBYF8f+ZnbIlqPQ99NLxXmw8Ban6 Jat6cIdJ+dWIXucod9C+75xhZ5ZeTeudicHbu/wLK1a+YIFhPZZ3BTkn/hMxTb5MAjzo FT+uYvnjGNnGfiF6psTeoECS7Cg6IJe90jMncCQhBCH2vh9/88DjLr4DCu+eGdg8N/+1 VS+A== X-Gm-Message-State: AOJu0YxWuxGi1zvXoW21tCWPQpS9DYt6VGD3jrfE5z7krmfZ3LZcc+Jj VYqe3n/K8RRqg0Kn+HALJAXr9hC3DVMBSvjISkKzE+fcxHV36XKxnSoZtRA8llO71qrXmZi12lv W5HI= X-Gm-Gg: ASbGnctidtv9yaq8WqnMnVrY+Zxx/S+zeHpAgKKKDNx6WrN2WQNaYv1wgJTBcMeC0ob JyOr7I5B9ozjtr4xEWKQ/YXIMr7jwCoC16ufji40Fbn+H92AIozpGZcFJk+08P45D+6sXtqh7mr PZaf38yoJP8/6o4EuYNDQjmM3HTEUizuJ0rxWMdFsXiM8vAWoO936Yajy91Z/CuFEh+piTr9o4Q 9y5SR0cP+0B+7uVWxa/Znp720wAvUYV4s45SjcMmeOHWMT+yWHZrFL86WKOrzyJeYQTh0vsJjI9 8+6SV4QWMavV4ZzB5tM+GcqdPwMgXmWB8wJIQtgfpCcUyrxkASNBdAq5GieDV0c3jcLCB1r2zEj kfWTN+Osb41nhEwl3I+A9YZRUl0xHLtJuPUYRn9sNK4fDEXY= X-Google-Smtp-Source: AGHT+IH7rozcdz17kb3Xn1PW5bFFhPLyNnKxaYok9nFakiHgjkLbRkRM6Ym/ThwdGFkChIzA5tYTGw== X-Received: by 2002:a5d:64c4:0:b0:3a4:e502:81e1 with SMTP id ffacd0b85a97d-3b2018c241bmr2165129f8f.52.1751458772834; Wed, 02 Jul 2025 05:19:32 -0700 (PDT) Message-ID: Date: Wed, 2 Jul 2025 14:19:21 +0200 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: [PATCH v7 7/7] mm: allow page scrubbing routine(s) to be arch controlled From: Jan Beulich To: "xen-devel@lists.xenproject.org" Cc: Andrew Cooper , Julien Grall , Stefano Stabellini , Anthony PERARD , Michal Orzel , =?UTF-8?Q?Roger_Pau_Monn=C3=A9?= , Bertrand Marquis , Volodymyr Babchuk , Oleksii Kurochko , Shawn Anastasio References: <0b57c6f0-9fd0-4f8a-93f8-dc494a1e952c@suse.com> Content-Language: en-US Autocrypt: addr=jbeulich@suse.com; keydata= xsDiBFk3nEQRBADAEaSw6zC/EJkiwGPXbWtPxl2xCdSoeepS07jW8UgcHNurfHvUzogEq5xk hu507c3BarVjyWCJOylMNR98Yd8VqD9UfmX0Hb8/BrA+Hl6/DB/eqGptrf4BSRwcZQM32aZK 7Pj2XbGWIUrZrd70x1eAP9QE3P79Y2oLrsCgbZJfEwCgvz9JjGmQqQkRiTVzlZVCJYcyGGsD /0tbFCzD2h20ahe8rC1gbb3K3qk+LpBtvjBu1RY9drYk0NymiGbJWZgab6t1jM7sk2vuf0Py O9Hf9XBmK0uE9IgMaiCpc32XV9oASz6UJebwkX+zF2jG5I1BfnO9g7KlotcA/v5ClMjgo6Gl MDY4HxoSRu3i1cqqSDtVlt+AOVBJBACrZcnHAUSuCXBPy0jOlBhxPqRWv6ND4c9PH1xjQ3NP nxJuMBS8rnNg22uyfAgmBKNLpLgAGVRMZGaGoJObGf72s6TeIqKJo/LtggAS9qAUiuKVnygo 3wjfkS9A3DRO+SpU7JqWdsveeIQyeyEJ/8PTowmSQLakF+3fote9ybzd880fSmFuIEJldWxp Y2ggPGpiZXVsaWNoQHN1c2UuY29tPsJgBBMRAgAgBQJZN5xEAhsDBgsJCAcDAgQVAggDBBYC AwECHgECF4AACgkQoDSui/t3IH4J+wCfQ5jHdEjCRHj23O/5ttg9r9OIruwAn3103WUITZee e7Sbg12UgcQ5lv7SzsFNBFk3nEQQCACCuTjCjFOUdi5Nm244F+78kLghRcin/awv+IrTcIWF hUpSs1Y91iQQ7KItirz5uwCPlwejSJDQJLIS+QtJHaXDXeV6NI0Uef1hP20+y8qydDiVkv6l IreXjTb7DvksRgJNvCkWtYnlS3mYvQ9NzS9PhyALWbXnH6sIJd2O9lKS1Mrfq+y0IXCP10eS FFGg+Av3IQeFatkJAyju0PPthyTqxSI4lZYuJVPknzgaeuJv/2NccrPvmeDg6Coe7ZIeQ8Yj t0ARxu2xytAkkLCel1Lz1WLmwLstV30g80nkgZf/wr+/BXJW/oIvRlonUkxv+IbBM3dX2OV8 AmRv1ySWPTP7AAMFB/9PQK/VtlNUJvg8GXj9ootzrteGfVZVVT4XBJkfwBcpC/XcPzldjv+3 HYudvpdNK3lLujXeA5fLOH+Z/G9WBc5pFVSMocI71I8bT8lIAzreg0WvkWg5V2WZsUMlnDL9 mpwIGFhlbM3gfDMs7MPMu8YQRFVdUvtSpaAs8OFfGQ0ia3LGZcjA6Ik2+xcqscEJzNH+qh8V m5jjp28yZgaqTaRbg3M/+MTbMpicpZuqF4rnB0AQD12/3BNWDR6bmh+EkYSMcEIpQmBM51qM EKYTQGybRCjpnKHGOxG0rfFY1085mBDZCH5Kx0cl0HVJuQKC+dV2ZY5AqjcKwAxpE75MLFkr wkkEGBECAAkFAlk3nEQCGwwACgkQoDSui/t3IH7nnwCfcJWUDUFKdCsBH/E5d+0ZnMQi+G0A nAuWpQkjM1ASeQwSHEeAWPgskBQL In-Reply-To: <0b57c6f0-9fd0-4f8a-93f8-dc494a1e952c@suse.com> Content-Transfer-Encoding: quoted-printable X-ZohoMail-DKIM: pass (identity @suse.com) X-ZM-MESSAGEID: 1751458791002116600 Content-Type: text/plain; charset="utf-8" Especially when dealing with large amounts of memory, memset() may not be very efficient; this can be bad enough that even for debug builds a custom function is warranted. We additionally want to distinguish "hot" and "cold" cases (with, as initial heuristic, "hot" being for any allocations a domain does for itself, assuming that in all other cases the page wouldn't be accessed [again] soon). The goal is for accesses of "cold" pages to not disturb caches (albeit finding a good balance between this and the higher latency looks to be difficult). Keep the default fallback to clear_page_*() in common code; this may want to be revisited down the road. Signed-off-by: Jan Beulich Acked-by: Julien Grall Reviewed-by: Oleksii Kurochko Reviewed-by: Jason Andryuk --- v7: ret -> RET. Underscore -> dash in new file's name. v6: Re-base. v4: Re-base. v3: Re-base. v2: New. --- The choice between hot and cold in scrub_one_page()'s callers is certainly up for discussion / improvement. --- a/xen/arch/arm/include/asm/page.h +++ b/xen/arch/arm/include/asm/page.h @@ -144,6 +144,12 @@ extern size_t dcache_line_bytes; =20 #define copy_page(dp, sp) memcpy(dp, sp, PAGE_SIZE) =20 +#define clear_page_hot clear_page +#define clear_page_cold clear_page + +#define scrub_page_hot(page) memset(page, SCRUB_BYTE_PATTERN, PAGE_SIZE) +#define scrub_page_cold scrub_page_hot + static inline size_t read_dcache_line_bytes(void) { register_t ctr; --- a/xen/arch/ppc/include/asm/page.h +++ b/xen/arch/ppc/include/asm/page.h @@ -188,6 +188,12 @@ static inline void invalidate_icache(voi #define clear_page(page) memset(page, 0, PAGE_SIZE) #define copy_page(dp, sp) memcpy(dp, sp, PAGE_SIZE) =20 +#define clear_page_hot clear_page +#define clear_page_cold clear_page + +#define scrub_page_hot(page) memset(page, SCRUB_BYTE_PATTERN, PAGE_SIZE) +#define scrub_page_cold scrub_page_hot + /* TODO: Flush the dcache for an entire page. */ static inline void flush_page_to_ram(unsigned long mfn, bool sync_icache) { --- a/xen/arch/riscv/include/asm/page.h +++ b/xen/arch/riscv/include/asm/page.h @@ -185,6 +185,12 @@ static inline void invalidate_icache(voi #define clear_page(page) memset((void *)(page), 0, PAGE_SIZE) #define copy_page(dp, sp) memcpy(dp, sp, PAGE_SIZE) =20 +#define clear_page_hot clear_page +#define clear_page_cold clear_page + +#define scrub_page_hot(page) memset(page, SCRUB_BYTE_PATTERN, PAGE_SIZE) +#define scrub_page_cold scrub_page_hot + void flush_page_to_ram(unsigned long mfn, bool sync_icache); =20 /* Write a pagetable entry. */ --- a/xen/arch/x86/Makefile +++ b/xen/arch/x86/Makefile @@ -58,6 +58,7 @@ obj-y +=3D pci.o obj-y +=3D physdev.o obj-$(CONFIG_COMPAT) +=3D x86_64/physdev.o obj-$(CONFIG_X86_PSR) +=3D psr.o +obj-bin-$(CONFIG_DEBUG) +=3D scrub-page.o obj-y +=3D setup.o obj-y +=3D shutdown.o obj-y +=3D smp.o --- a/xen/arch/x86/include/asm/page.h +++ b/xen/arch/x86/include/asm/page.h @@ -226,6 +226,11 @@ void copy_page_sse2(void *to, const void #define clear_page(_p) clear_page_cold(_p) #define copy_page(_t, _f) copy_page_sse2(_t, _f) =20 +#ifdef CONFIG_DEBUG +void scrub_page_hot(void *); +void scrub_page_cold(void *); +#endif + /* Convert between Xen-heap virtual addresses and machine addresses. */ #define __pa(x) (virt_to_maddr(x)) #define __va(x) (maddr_to_virt(x)) --- /dev/null +++ b/xen/arch/x86/scrub-page.S @@ -0,0 +1,38 @@ + .file __FILE__ + +#include +#include +#include + +FUNC(scrub_page_cold) + mov $PAGE_SIZE/32, %ecx + mov $SCRUB_PATTERN, %rax + +0: movnti %rax, (%rdi) + movnti %rax, 8(%rdi) + movnti %rax, 16(%rdi) + movnti %rax, 24(%rdi) + add $32, %rdi + sub $1, %ecx + jnz 0b + + sfence + RET +END(scrub_page_cold) + + .macro scrub_page_stosb + mov $PAGE_SIZE, %ecx + mov $SCRUB_BYTE_PATTERN, %eax + rep stosb + .endm + + .macro scrub_page_stosq + mov $PAGE_SIZE/8, %ecx + mov $SCRUB_PATTERN, %rax + rep stosq + .endm + +FUNC(scrub_page_hot) + ALTERNATIVE scrub_page_stosq, scrub_page_stosb, X86_FEATURE_ERMS + RET +END(scrub_page_hot) --- a/xen/common/page_alloc.c +++ b/xen/common/page_alloc.c @@ -135,6 +135,7 @@ #include #include #include +#include #include #include #include @@ -802,27 +803,31 @@ static void page_list_add_scrub(struct p page_list_add(pg, &heap(node, zone, order)); } =20 -/* SCRUB_PATTERN needs to be a repeating series of bytes. */ -#ifndef NDEBUG -#define SCRUB_PATTERN 0xc2c2c2c2c2c2c2c2ULL -#else -#define SCRUB_PATTERN 0ULL +/* + * While in debug builds we want callers to avoid relying on allocations + * returning zeroed pages, for a production build, clear_page_*() is the + * fastest way to scrub. + */ +#ifndef CONFIG_DEBUG +# undef scrub_page_hot +# define scrub_page_hot clear_page_hot +# undef scrub_page_cold +# define scrub_page_cold clear_page_cold #endif -#define SCRUB_BYTE_PATTERN (SCRUB_PATTERN & 0xff) =20 -static void scrub_one_page(const struct page_info *pg) +static void scrub_one_page(const struct page_info *pg, bool cold) { + void *ptr; + if ( unlikely(pg->count_info & PGC_broken) ) return; =20 -#ifndef NDEBUG - /* Avoid callers relying on allocations returning zeroed pages. */ - unmap_domain_page(memset(__map_domain_page(pg), - SCRUB_BYTE_PATTERN, PAGE_SIZE)); -#else - /* For a production build, clear_page() is the fastest way to scrub. */ - clear_domain_page(_mfn(page_to_mfn(pg))); -#endif + ptr =3D __map_domain_page(pg); + if ( cold ) + scrub_page_cold(ptr); + else + scrub_page_hot(ptr); + unmap_domain_page(ptr); } =20 static void poison_one_page(struct page_info *pg) @@ -1102,12 +1107,14 @@ static struct page_info *alloc_heap_page if ( first_dirty !=3D INVALID_DIRTY_IDX || (scrub_debug && !(memflags & MEMF_no_scrub)) ) { + bool cold =3D d && d !=3D current->domain; + for ( i =3D 0; i < (1U << order); i++ ) { if ( test_and_clear_bit(_PGC_need_scrub, &pg[i].count_info) ) { if ( !(memflags & MEMF_no_scrub) ) - scrub_one_page(&pg[i]); + scrub_one_page(&pg[i], cold); =20 dirty_cnt++; } @@ -1372,7 +1379,7 @@ bool scrub_free_pages(void) { if ( test_bit(_PGC_need_scrub, &pg[i].count_info) ) { - scrub_one_page(&pg[i]); + scrub_one_page(&pg[i], true); /* * We can modify count_info without holding heap * lock since we effectively locked this buddy by @@ -2076,7 +2083,7 @@ static struct page_info *alloc_color_hea if ( !(memflags & MEMF_no_scrub) ) { if ( need_scrub ) - scrub_one_page(pg); + scrub_one_page(pg, d !=3D current->domain); else check_one_page(pg); } @@ -2227,7 +2234,7 @@ static void __init cf_check smp_scrub_he if ( !mfn_valid(_mfn(mfn)) || !page_state_is(pg, free) ) continue; =20 - scrub_one_page(pg); + scrub_one_page(pg, true); } } =20 @@ -2925,7 +2932,7 @@ void unprepare_staticmem_pages(struct pa if ( need_scrub ) { /* TODO: asynchronous scrubbing for pages of static memory. */ - scrub_one_page(pg); + scrub_one_page(pg, true); } =20 pg[i].count_info |=3D PGC_static; --- /dev/null +++ b/xen/include/xen/scrub.h @@ -0,0 +1,24 @@ +#ifndef __XEN_SCRUB_H__ +#define __XEN_SCRUB_H__ + +#include + +/* SCRUB_PATTERN needs to be a repeating series of bytes. */ +#ifdef CONFIG_DEBUG +# define SCRUB_PATTERN _AC(0xc2c2c2c2c2c2c2c2,ULL) +#else +# define SCRUB_PATTERN _AC(0,ULL) +#endif +#define SCRUB_BYTE_PATTERN (SCRUB_PATTERN & 0xff) + +#endif /* __XEN_SCRUB_H__ */ + +/* + * Local variables: + * mode: C + * c-file-style: "BSD" + * c-basic-offset: 4 + * tab-width: 4 + * indent-tabs-mode: nil + * End: + */