From nobody Fri Oct 31 03:42:51 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) client-ip=192.237.175.120; envelope-from=xen-devel-bounces@lists.xenproject.org; helo=lists.xenproject.org; Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=pass(p=quarantine dis=none) header.from=suse.com ARC-Seal: i=1; a=rsa-sha256; t=1750078792; cv=none; d=zohomail.com; s=zohoarc; b=EaC2RqC561c/vZMQrU9gDY5isUH5HAtEdFztRpoQGo3U6W0rs4X39idUQX6wAcC68JjFFBgJpkU6j8aLsy8ZAk65RNW3+yRfDmqqAH+PYI0kX0oBig/rdk+BgqHPz6m6k0K9GUfxJ6/0A/cDujNTf/CJSoswV3rg3rxZgHQje80= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1750078792; h=Content-Type:Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=CCGJJ+18Hx71qZ1BImD3aJBlrhNSiedNN3y6oo5smPU=; b=HGt+PVrDcv/BF2gazx241BKZvS5nXKMnAatO+WhXQS5JV8u3S9fvUxgXW3QxM5eILDA+SRyUocNHBZG2GiHLxqu0YUSs4Q9eoblYdrrpo9FQwm06D4zYPSQLjtOvwoIWOT+bMtfHEHgyo3yaFsjfc820dv5mUqOiD6Q+38iwprg= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=pass header.from= (p=quarantine dis=none) Return-Path: Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) by mx.zohomail.com with SMTPS id 1750078792537427.98028022905316; Mon, 16 Jun 2025 05:59:52 -0700 (PDT) Received: from list by lists.xenproject.org with outflank-mailman.1017205.1394168 (Exim 4.92) (envelope-from ) id 1uR9R6-0003px-No; Mon, 16 Jun 2025 12:59:40 +0000 Received: by outflank-mailman (output) from mailman id 1017205.1394168; Mon, 16 Jun 2025 12:59:40 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1uR9R6-0003pq-LD; Mon, 16 Jun 2025 12:59:40 +0000 Received: by outflank-mailman (input) for mailman id 1017205; Mon, 16 Jun 2025 12:59:39 +0000 Received: from se1-gles-flk1-in.inumbo.com ([94.247.172.50] helo=se1-gles-flk1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1uR9R5-0003pW-FL for xen-devel@lists.xenproject.org; Mon, 16 Jun 2025 12:59:39 +0000 Received: from mail-wr1-x432.google.com (mail-wr1-x432.google.com [2a00:1450:4864:20::432]) by se1-gles-flk1.inumbo.com (Halon) with ESMTPS id c1f4b323-4ab1-11f0-b894-0df219b8e170; Mon, 16 Jun 2025 14:59:37 +0200 (CEST) Received: by mail-wr1-x432.google.com with SMTP id ffacd0b85a97d-3a57ae5cb17so873011f8f.0 for ; Mon, 16 Jun 2025 05:59:37 -0700 (PDT) Received: from [10.156.60.236] (ip-037-024-206-209.um08.pools.vodafone-ip.de. [37.24.206.209]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-748900062dasm6979838b3a.66.2025.06.16.05.59.33 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Mon, 16 Jun 2025 05:59:36 -0700 (PDT) X-Outflank-Mailman: Message body and most headers restored to incoming version X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: c1f4b323-4ab1-11f0-b894-0df219b8e170 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.com; s=google; t=1750078777; x=1750683577; darn=lists.xenproject.org; h=content-transfer-encoding:in-reply-to:autocrypt:content-language :references:cc:to:from:subject:user-agent:mime-version:date :message-id:from:to:cc:subject:date:message-id:reply-to; bh=CCGJJ+18Hx71qZ1BImD3aJBlrhNSiedNN3y6oo5smPU=; b=d41sfRt1bRz3vh2iqAmbc2CMDYQUXhz3k1jIRkYdwBJ7+RG4JkFuIuTfXWAOxSOHo+ rBhI45W+IY+sNefO1Goh0Qvt3MJlWfSXmNaZOSiV52B4B1inw9Hk2RVCcfC2yi+Rof4P GYki66At2+oM18rBiBxImN2LqoZUFe8zy4JW5945MvGhRJIuA/jMFA8QvAhNGz3C3H0X LZD/pt1EmwfYVfybV3LmU5UrDbua5v+YzL+ExLkguHTYsNrQE+FuZbga8wmCii4QEY/E I4jwPSDxWcdTeAfCSZjdDwYZBqf74rPWyGsb+V8UgLb7VLd2rM6SciqjfsH5S5IdA8Js 9JZg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1750078777; x=1750683577; h=content-transfer-encoding:in-reply-to:autocrypt:content-language :references:cc:to:from:subject:user-agent:mime-version:date :message-id:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=CCGJJ+18Hx71qZ1BImD3aJBlrhNSiedNN3y6oo5smPU=; b=BPaQ86XtkU1gbFytnhOaVja8Nakxtcv3wJtrso2FUtUU3E8+WEhnD982g5fyD0mAnr MItkb3oGpTsNDx+VIlvgAMV1Cc3I55O+qjbfxjY64iSdjLSiPlG+5mrZAbJeQhIKER3C ZmxZXuggzupSImtFzE7hWOP9QFf2t3GL2ZUDgNt2bDADYPXMPk2gY7zQ+PqdOlImMGHr tmZCbMB3V69AMkesa0R1vU5TcBpDYibnY2LELzBT+t7f5HFWaf8UQ0HPam2yP7p67+al 84FKE2flh1TQgb3pjfEq11ZdfyouR5hQvdxo6WolMv+6HMJ6x0m8OrYb2H0JxCRRPiKu SEXw== X-Gm-Message-State: AOJu0YxqbdFVboxGy7uzWpWbyyw2unUrZ5Yqzzf3+ELeDIjXWijIyQyI KQtWlFB3kwsXXN2U2sEIl2Ba6sJ6IwPeEiN7o09ULIv2NkMlaADtROndnJu/dbj3JsI4TSypYNr S26w= X-Gm-Gg: ASbGncv/J2IQOW2ip+xQO2g5GEVFk0OKslV/ED72nIgiCG5KY+BszzSArPYFd65SrNu woqjaHP0h7QbUWGvDB0n0Whw3hNbDVyN07HyeyLBtQxNdleepSBhZG+pQs27NhtPFRl/8/5mGbd 2rWh9QbS5jwSMYRWR3yhvZQEe7pKSHlhHgOdFkoEe2EAEM8uKGxVHniwr9zrENZEqAvg5VTdqpZ nqIFqBelv5cDFbGNCsEOqwRGttxTnARHbrI6sS1dQqiiXWpk/DF2Q2G8jNyAO56CpHaO6y/CrAZ EqdHbQ/xeDQ/DqybqZx0Isqg/M3vEoxJ3DlF38WLPAY8aR25eBQtk7GkfVALTDsXWOk8FhNi/KJ blmWqdv6RMwrgAzSdyKXQacYPB7Sbn8zIlgzWPafFait6cMk= X-Google-Smtp-Source: AGHT+IGQhIvVILD/mLsyDr/e8gz5x5RpAZerphy1lrk8w1UAU+KzQbiDkF+IA9zWhP++hZpQROFQYg== X-Received: by 2002:a05:6000:2313:b0:3a5:2d42:aa17 with SMTP id ffacd0b85a97d-3a572e79c14mr8398070f8f.31.1750078776746; Mon, 16 Jun 2025 05:59:36 -0700 (PDT) Message-ID: Date: Mon, 16 Jun 2025 14:59:30 +0200 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: [PATCH v6 1/7] x86: suppress ERMS for internal use when MISC_ENABLE.FAST_STRING is clear From: Jan Beulich To: "xen-devel@lists.xenproject.org" Cc: Andrew Cooper , =?UTF-8?Q?Roger_Pau_Monn=C3=A9?= References: Content-Language: en-US Autocrypt: addr=jbeulich@suse.com; keydata= xsDiBFk3nEQRBADAEaSw6zC/EJkiwGPXbWtPxl2xCdSoeepS07jW8UgcHNurfHvUzogEq5xk hu507c3BarVjyWCJOylMNR98Yd8VqD9UfmX0Hb8/BrA+Hl6/DB/eqGptrf4BSRwcZQM32aZK 7Pj2XbGWIUrZrd70x1eAP9QE3P79Y2oLrsCgbZJfEwCgvz9JjGmQqQkRiTVzlZVCJYcyGGsD /0tbFCzD2h20ahe8rC1gbb3K3qk+LpBtvjBu1RY9drYk0NymiGbJWZgab6t1jM7sk2vuf0Py O9Hf9XBmK0uE9IgMaiCpc32XV9oASz6UJebwkX+zF2jG5I1BfnO9g7KlotcA/v5ClMjgo6Gl MDY4HxoSRu3i1cqqSDtVlt+AOVBJBACrZcnHAUSuCXBPy0jOlBhxPqRWv6ND4c9PH1xjQ3NP nxJuMBS8rnNg22uyfAgmBKNLpLgAGVRMZGaGoJObGf72s6TeIqKJo/LtggAS9qAUiuKVnygo 3wjfkS9A3DRO+SpU7JqWdsveeIQyeyEJ/8PTowmSQLakF+3fote9ybzd880fSmFuIEJldWxp Y2ggPGpiZXVsaWNoQHN1c2UuY29tPsJgBBMRAgAgBQJZN5xEAhsDBgsJCAcDAgQVAggDBBYC AwECHgECF4AACgkQoDSui/t3IH4J+wCfQ5jHdEjCRHj23O/5ttg9r9OIruwAn3103WUITZee e7Sbg12UgcQ5lv7SzsFNBFk3nEQQCACCuTjCjFOUdi5Nm244F+78kLghRcin/awv+IrTcIWF hUpSs1Y91iQQ7KItirz5uwCPlwejSJDQJLIS+QtJHaXDXeV6NI0Uef1hP20+y8qydDiVkv6l IreXjTb7DvksRgJNvCkWtYnlS3mYvQ9NzS9PhyALWbXnH6sIJd2O9lKS1Mrfq+y0IXCP10eS FFGg+Av3IQeFatkJAyju0PPthyTqxSI4lZYuJVPknzgaeuJv/2NccrPvmeDg6Coe7ZIeQ8Yj t0ARxu2xytAkkLCel1Lz1WLmwLstV30g80nkgZf/wr+/BXJW/oIvRlonUkxv+IbBM3dX2OV8 AmRv1ySWPTP7AAMFB/9PQK/VtlNUJvg8GXj9ootzrteGfVZVVT4XBJkfwBcpC/XcPzldjv+3 HYudvpdNK3lLujXeA5fLOH+Z/G9WBc5pFVSMocI71I8bT8lIAzreg0WvkWg5V2WZsUMlnDL9 mpwIGFhlbM3gfDMs7MPMu8YQRFVdUvtSpaAs8OFfGQ0ia3LGZcjA6Ik2+xcqscEJzNH+qh8V m5jjp28yZgaqTaRbg3M/+MTbMpicpZuqF4rnB0AQD12/3BNWDR6bmh+EkYSMcEIpQmBM51qM EKYTQGybRCjpnKHGOxG0rfFY1085mBDZCH5Kx0cl0HVJuQKC+dV2ZY5AqjcKwAxpE75MLFkr wkkEGBECAAkFAlk3nEQCGwwACgkQoDSui/t3IH7nnwCfcJWUDUFKdCsBH/E5d+0ZnMQi+G0A nAuWpQkjM1ASeQwSHEeAWPgskBQL In-Reply-To: Content-Transfer-Encoding: quoted-printable X-ZohoMail-DKIM: pass (identity @suse.com) X-ZM-MESSAGEID: 1750078793362116600 Content-Type: text/plain; charset="utf-8" Before we start actually adjusting behavior when ERMS is available, follow Linux commit 161ec53c702c ("x86, mem, intel: Initialize Enhanced REP MOVSB/STOSB") and zap the CPUID-derived feature flag when the MSR bit is clear. Don't extend the artificial clearing to guest view, though: Guests can take their own decision in this regard, as they can read (most of) MISC_ENABLE. Signed-off-by: Jan Beulich Reviewed-by: Jason Andryuk --- TBD: Would be nice if "cpuid=3Dno-erms" propagated to guest view (for "cpuid=3D" generally meaning to affect guests as well as Xen), but since both disabling paths use setup_clear_cpu_cap() they're indistinguishable in guest_common_feature_adjustments(). A separate boolean could take care of this, but would look clumsy to me. --- v5: Correct guest_common_max_feature_adjustments() addition. v4: Also adjust guest_common_max_feature_adjustments(). v3: New. --- a/xen/arch/x86/cpu/intel.c +++ b/xen/arch/x86/cpu/intel.c @@ -366,8 +366,18 @@ static void cf_check early_init_intel(st paddr_bits =3D 36; =20 if (c =3D=3D &boot_cpu_data) { + uint64_t misc_enable; + check_memory_type_self_snoop_errata(); =20 + /* + * If fast string is not enabled in IA32_MISC_ENABLE for any reason, + * clear the enhanced fast string CPU capability. + */ + rdmsrl(MSR_IA32_MISC_ENABLE, misc_enable); + if (!(misc_enable & MSR_IA32_MISC_ENABLE_FAST_STRING)) + setup_clear_cpu_cap(X86_FEATURE_ERMS); + intel_init_levelling(); } =20 --- a/xen/arch/x86/cpu-policy.c +++ b/xen/arch/x86/cpu-policy.c @@ -487,6 +487,12 @@ static void __init guest_common_max_feat */ if ( test_bit(X86_FEATURE_RTM, fs) ) __set_bit(X86_FEATURE_RTM_ALWAYS_ABORT, fs); + + /* + * We expose MISC_ENABLE to guests, so our internal clearing of ERMS w= hen + * FAST_STRING is not set should not affect the view of migrating-in g= uests. + */ + __set_bit(X86_FEATURE_ERMS, fs); } =20 static void __init guest_common_default_feature_adjustments(uint32_t *fs) @@ -567,6 +573,16 @@ static void __init guest_common_default_ __clear_bit(X86_FEATURE_RTM, fs); __set_bit(X86_FEATURE_RTM_ALWAYS_ABORT, fs); } + + /* + * We expose MISC_ENABLE to guests, so our internal clearing of ERMS w= hen + * FAST_STRING is not set should not propagate to guest view. Guests = can + * judge on their own whether to ignore the CPUID bit when the MSR bit= is + * clear. The bit being uniformly set in the max policies, we only ne= ed + * to clear it here (if hardware doesn't have it). + */ + if ( !raw_cpu_policy.feat.erms ) + __clear_bit(X86_FEATURE_ERMS, fs); } =20 static void __init guest_common_feature_adjustments(uint32_t *fs) --- a/xen/arch/x86/include/asm/msr-index.h +++ b/xen/arch/x86/include/asm/msr-index.h @@ -493,6 +493,7 @@ #define MSR_IA32_THERM_INTERRUPT 0x0000019b #define MSR_IA32_THERM_STATUS 0x0000019c #define MSR_IA32_MISC_ENABLE 0x000001a0 +#define MSR_IA32_MISC_ENABLE_FAST_STRING (1<<0) #define MSR_IA32_MISC_ENABLE_PERF_AVAIL (1<<7) #define MSR_IA32_MISC_ENABLE_BTS_UNAVAIL (1<<11) #define MSR_IA32_MISC_ENABLE_PEBS_UNAVAIL (1<<12) From nobody Fri Oct 31 03:42:51 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) client-ip=192.237.175.120; envelope-from=xen-devel-bounces@lists.xenproject.org; helo=lists.xenproject.org; Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=pass(p=quarantine dis=none) header.from=suse.com ARC-Seal: i=1; a=rsa-sha256; t=1750078816; cv=none; d=zohomail.com; s=zohoarc; b=DhS4r8DNnyZZpt+ra5vlIDxqmnvsxLc1zPqwIi1v3RutTKcKp3RBdWYxRldmnEbPhboT8NK/EHewe8tqXl51nj6TAkNMALYsjdPxSvdq8qeU72K3vpO33k+0NZkNGYjxBE8m6ujoFaxaR+nTgM0p0zMHWKBvWKlzj3xKLWuh8pk= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1750078816; h=Content-Type:Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=+5Soi2kJcSJ+g49AA3Ctrt1tH15NUClp6ljEBqxL0MA=; b=UqXnc4yLkJRNgwKEOs4A/9rX+Ff23hVVEncvpXR2Fb+KIIk02uefpQ+voPNd4pOMQR7qvsc1DR4uC05zw/QskgccDk3lB8JGyi2YoyhscBbN3fyyfHwCgisJnT94Zu+d845nskFakKZi1mU9ELLcOe2mPKh+UI+y91OxXAHMdlg= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=pass header.from= (p=quarantine dis=none) Return-Path: Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) by mx.zohomail.com with SMTPS id 1750078816744734.9192044579289; Mon, 16 Jun 2025 06:00:16 -0700 (PDT) Received: from list by lists.xenproject.org with outflank-mailman.1017206.1394178 (Exim 4.92) (envelope-from ) id 1uR9RS-0004Zc-Vg; Mon, 16 Jun 2025 13:00:02 +0000 Received: by outflank-mailman (output) from mailman id 1017206.1394178; Mon, 16 Jun 2025 13:00:02 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1uR9RS-0004Z5-Sk; Mon, 16 Jun 2025 13:00:02 +0000 Received: by outflank-mailman (input) for mailman id 1017206; Mon, 16 Jun 2025 13:00:01 +0000 Received: from se1-gles-flk1-in.inumbo.com ([94.247.172.50] helo=se1-gles-flk1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1uR9RR-0003pW-RX for xen-devel@lists.xenproject.org; Mon, 16 Jun 2025 13:00:01 +0000 Received: from mail-wr1-x434.google.com (mail-wr1-x434.google.com [2a00:1450:4864:20::434]) by se1-gles-flk1.inumbo.com (Halon) with ESMTPS id cf83a78b-4ab1-11f0-b894-0df219b8e170; Mon, 16 Jun 2025 15:00:00 +0200 (CEST) Received: by mail-wr1-x434.google.com with SMTP id ffacd0b85a97d-3a510432236so3313320f8f.0 for ; Mon, 16 Jun 2025 06:00:00 -0700 (PDT) Received: from [10.156.60.236] (ip-037-024-206-209.um08.pools.vodafone-ip.de. [37.24.206.209]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-2365d88f447sm60051525ad.45.2025.06.16.05.59.56 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Mon, 16 Jun 2025 05:59:59 -0700 (PDT) X-Outflank-Mailman: Message body and most headers restored to incoming version X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: cf83a78b-4ab1-11f0-b894-0df219b8e170 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.com; s=google; t=1750078799; x=1750683599; darn=lists.xenproject.org; h=content-transfer-encoding:in-reply-to:autocrypt:content-language :references:cc:to:from:subject:user-agent:mime-version:date :message-id:from:to:cc:subject:date:message-id:reply-to; bh=+5Soi2kJcSJ+g49AA3Ctrt1tH15NUClp6ljEBqxL0MA=; b=CJ6AGN+AdNR7G/ELf5OxfU33EakJzDn6LwM1ZTbPHJpUUWW4PxNf2+0ViZAClRnGfS mD5na5OD8xFvYnL4f03GNO5msueXNppRgHujQ4oc4V6Qg0PCBKstg57BGO97l7RgdYWD ajvRg4b2R3YHIh5sseUAh3HiUfliYTudWNs+ryeloq9itqUYWCyrPARNFe4Z9IxRmX5n 6qxbAYHCRZ7mFBtmP/pf5Hiu3n51l2a4gjc60X7AcwePo+y0kvV2Xvh+qZpT7yTqkQjt 51SP3xY6ven9wCi1LCRlHXPY0xZ9rZ6WTFIrialpOCp2OsitEZEGHdbpeJ5Zl20dCc71 IOVg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1750078799; x=1750683599; h=content-transfer-encoding:in-reply-to:autocrypt:content-language :references:cc:to:from:subject:user-agent:mime-version:date :message-id:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=+5Soi2kJcSJ+g49AA3Ctrt1tH15NUClp6ljEBqxL0MA=; b=gmi3SEgKBWNhCtHYjKaFI2xGzqvVMQDyI41zl2+16cNL+s3gmi/WaWe5yK9B8NwkkM X4NWWXw0OpeOOuGZjGFBJDuSflz9ydmEb9tCjAWFquHon6vY4dQPns3zkiBZAWQcz5W1 xgBQRhDgAI0RwmRQ3QMPhJnO/SPvpoIT4Gde+CC54k5B4suIwuppPJJEISjRNnCziPOV /XpDYFKgjhARmpGP5hUsK36yaHM/P/J9DkLYeLYJZR1K4d6xtTAdzlOkY+J4NorAnc+Z c9W0D70jSR5Xvt585rNGzqaqJXY5+lxoNmPtaxv5FzWnX0QQH/0VxpjE35yQ+R50sJKT rZmg== X-Gm-Message-State: AOJu0YwEcsRVnCEg9FlCOJe0EzjP4Mp2WsvTkq2FMUwDG3OMGRhm3kwK Zn2XK/Dear0CtukE0DfpLUeLqtKOPExIGQUGNHKvkr20KWMZmK+rxXLyVbKQB4PGFBwKwpSNOL0 dOUM= X-Gm-Gg: ASbGncsn9aZ9LH/58uBOU+Foj9F0EiNd2tur/YYCzhpataPLDpJKAJ9wrabq87DZu/z ohKrejbeox0sH8NHjDy+S87NqXkzkiJPNkTilc1mJAnnJP6tgCmefcgElnQCEpQVXv62v3IKpVG q2omZYuqCrvnDZEJ2xtybNUPNTHldOxmab/EW96Kxpw5fbzQ4eFOfQDig2KGe2r7mCmir2B6j2q TPtghu8sONpbRutzW3KOksfknKCnRirQO9Kiq59AUAzT1MROb6DfUE5LyYbdhVacVvO5PFUwPo7 sQoyrmpXim+SiQVNaXSiYH7tGAnpNoq88w+Ql0ajSpLNzcMUUrWzvPG4b43phuAObc/7vghMaMq IL0m+3fEs9Rd7kVHWeokZrHTaT2v3qhin9Fov3CaiXfE1dfg= X-Google-Smtp-Source: AGHT+IHZNO5ywfqqom9NEqGjAoqenIhMq3NeePncRuDPT2gLtO+3sY5htYXdBhtw100vKrKUt+p1pw== X-Received: by 2002:a05:6000:3112:b0:3a4:e480:b5df with SMTP id ffacd0b85a97d-3a572e79749mr8135800f8f.44.1750078799533; Mon, 16 Jun 2025 05:59:59 -0700 (PDT) Message-ID: <658802b4-1847-455d-98ad-244d97fddc5e@suse.com> Date: Mon, 16 Jun 2025 14:59:52 +0200 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: [PATCH v6 2/7] x86/alternatives: serialize after (self-)modifying code From: Jan Beulich To: "xen-devel@lists.xenproject.org" Cc: Andrew Cooper , =?UTF-8?Q?Roger_Pau_Monn=C3=A9?= References: Content-Language: en-US Autocrypt: addr=jbeulich@suse.com; keydata= xsDiBFk3nEQRBADAEaSw6zC/EJkiwGPXbWtPxl2xCdSoeepS07jW8UgcHNurfHvUzogEq5xk hu507c3BarVjyWCJOylMNR98Yd8VqD9UfmX0Hb8/BrA+Hl6/DB/eqGptrf4BSRwcZQM32aZK 7Pj2XbGWIUrZrd70x1eAP9QE3P79Y2oLrsCgbZJfEwCgvz9JjGmQqQkRiTVzlZVCJYcyGGsD /0tbFCzD2h20ahe8rC1gbb3K3qk+LpBtvjBu1RY9drYk0NymiGbJWZgab6t1jM7sk2vuf0Py O9Hf9XBmK0uE9IgMaiCpc32XV9oASz6UJebwkX+zF2jG5I1BfnO9g7KlotcA/v5ClMjgo6Gl MDY4HxoSRu3i1cqqSDtVlt+AOVBJBACrZcnHAUSuCXBPy0jOlBhxPqRWv6ND4c9PH1xjQ3NP nxJuMBS8rnNg22uyfAgmBKNLpLgAGVRMZGaGoJObGf72s6TeIqKJo/LtggAS9qAUiuKVnygo 3wjfkS9A3DRO+SpU7JqWdsveeIQyeyEJ/8PTowmSQLakF+3fote9ybzd880fSmFuIEJldWxp Y2ggPGpiZXVsaWNoQHN1c2UuY29tPsJgBBMRAgAgBQJZN5xEAhsDBgsJCAcDAgQVAggDBBYC AwECHgECF4AACgkQoDSui/t3IH4J+wCfQ5jHdEjCRHj23O/5ttg9r9OIruwAn3103WUITZee e7Sbg12UgcQ5lv7SzsFNBFk3nEQQCACCuTjCjFOUdi5Nm244F+78kLghRcin/awv+IrTcIWF hUpSs1Y91iQQ7KItirz5uwCPlwejSJDQJLIS+QtJHaXDXeV6NI0Uef1hP20+y8qydDiVkv6l IreXjTb7DvksRgJNvCkWtYnlS3mYvQ9NzS9PhyALWbXnH6sIJd2O9lKS1Mrfq+y0IXCP10eS FFGg+Av3IQeFatkJAyju0PPthyTqxSI4lZYuJVPknzgaeuJv/2NccrPvmeDg6Coe7ZIeQ8Yj t0ARxu2xytAkkLCel1Lz1WLmwLstV30g80nkgZf/wr+/BXJW/oIvRlonUkxv+IbBM3dX2OV8 AmRv1ySWPTP7AAMFB/9PQK/VtlNUJvg8GXj9ootzrteGfVZVVT4XBJkfwBcpC/XcPzldjv+3 HYudvpdNK3lLujXeA5fLOH+Z/G9WBc5pFVSMocI71I8bT8lIAzreg0WvkWg5V2WZsUMlnDL9 mpwIGFhlbM3gfDMs7MPMu8YQRFVdUvtSpaAs8OFfGQ0ia3LGZcjA6Ik2+xcqscEJzNH+qh8V m5jjp28yZgaqTaRbg3M/+MTbMpicpZuqF4rnB0AQD12/3BNWDR6bmh+EkYSMcEIpQmBM51qM EKYTQGybRCjpnKHGOxG0rfFY1085mBDZCH5Kx0cl0HVJuQKC+dV2ZY5AqjcKwAxpE75MLFkr wkkEGBECAAkFAlk3nEQCGwwACgkQoDSui/t3IH7nnwCfcJWUDUFKdCsBH/E5d+0ZnMQi+G0A nAuWpQkjM1ASeQwSHEeAWPgskBQL In-Reply-To: Content-Transfer-Encoding: quoted-printable X-ZohoMail-DKIM: pass (identity @suse.com) X-ZM-MESSAGEID: 1750078817965116600 Content-Type: text/plain; charset="utf-8" While supposedly safe via enforcing a control flow change when modifying already prefetched code, it may not really be. Afaik a request is pending to drop the first of the two options in the SDM's "Handling Self- and Cross-Modifying Code" section (still present there as of version 087). Insert a serializing instruction there, and remove the "noinline" in exchange. Signed-off-by: Jan Beulich Reviewed-by: Jason Andryuk --- Strictly speaking with LIVEPATCH=3Dn the if() (but not its body) is dead code. Adding IS_ENABLED(CONFIG_LIVEPATCH) would feel quite a bit like extra clutter, though. Plus Misra rule 2.2 is globally deviated anyway. --- v6: Split off. Extend commentary. Load deterministic value into CR2. Drop "noinline". --- a/xen/arch/x86/alternative.c +++ b/xen/arch/x86/alternative.c @@ -194,13 +194,18 @@ void *place_ret(void *ptr) * You should run this with interrupts disabled or on code that is not * executing. * - * "noinline" to cause control flow change and thus invalidate I$ and - * cause refetch after modification. + * While the SDM continues to suggest using "noinline" would be sufficient= , it + * may not be, e.g. due to errata. Issue a serializing insn afterwards, u= nless + * this is for live-patching, where we modify code before it goes live. I= ssue + * a serializing insn which is unlikely to be intercepted by a hypervisor,= in + * case we run virtualized ourselves. */ -static void init_or_livepatch noinline +static void init_or_livepatch text_poke(void *addr, const void *opcode, size_t len) { memcpy(addr, opcode, len); + if ( system_state < SYS_STATE_active ) + asm volatile ( "mov %0, %%cr2" :: "r" (0L) : "memory" ); } =20 extern void *const __initdata_cf_clobber_start[]; From nobody Fri Oct 31 03:42:51 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) client-ip=192.237.175.120; envelope-from=xen-devel-bounces@lists.xenproject.org; helo=lists.xenproject.org; Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=pass(p=quarantine dis=none) header.from=suse.com ARC-Seal: i=1; a=rsa-sha256; t=1750078848; cv=none; d=zohomail.com; s=zohoarc; b=Za0TXgvc8wue6LD1IuuEuQdyQouTsPT0KwawiWMOvxpGvE4rCrzVN0SEynRPzlRZA75yYL2KazouFfJKTnswwGnTj7TTL8tZP91kdUbZ/6YmL3ylH0GQlMO9mQcMdZPFNn9UVvon06pvz7xsKFpbpeT9X3sAvfHvdx7kFfF2y8U= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1750078848; h=Content-Type:Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=pl2MdiB+kJYOREm+k9B0Yw8rIzN4HAkJgHoxKayTGVI=; b=nYmD/vFV2lRvnh+COWkVVED4tfFPclmECeEO8GR/L2REq+2bI7txOCCRw/VplU8NGdVyIll7NeSi0kHn7+kh7FVqwmgHyXhqw2d5WdYcAEjVC9nn/+opGanSAOGI6xFy7Fpc9y8X4WWz6/ul1+2iUs9lvhgvkBWNIEIp36iQDwA= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=pass header.from= (p=quarantine dis=none) Return-Path: Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) by mx.zohomail.com with SMTPS id 1750078848120432.7264778675027; Mon, 16 Jun 2025 06:00:48 -0700 (PDT) Received: from list by lists.xenproject.org with outflank-mailman.1017217.1394189 (Exim 4.92) (envelope-from ) id 1uR9Rw-0005j8-CE; Mon, 16 Jun 2025 13:00:32 +0000 Received: by outflank-mailman (output) from mailman id 1017217.1394189; Mon, 16 Jun 2025 13:00:32 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1uR9Rw-0005j1-9O; Mon, 16 Jun 2025 13:00:32 +0000 Received: by outflank-mailman (input) for mailman id 1017217; Mon, 16 Jun 2025 13:00:31 +0000 Received: from se1-gles-sth1-in.inumbo.com ([159.253.27.254] helo=se1-gles-sth1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1uR9Rv-0005HG-2k for xen-devel@lists.xenproject.org; Mon, 16 Jun 2025 13:00:31 +0000 Received: from mail-wr1-x433.google.com (mail-wr1-x433.google.com [2a00:1450:4864:20::433]) by se1-gles-sth1.inumbo.com (Halon) with ESMTPS id e1a5f503-4ab1-11f0-a309-13f23c93f187; Mon, 16 Jun 2025 15:00:30 +0200 (CEST) Received: by mail-wr1-x433.google.com with SMTP id ffacd0b85a97d-3a575a988f9so1326015f8f.0 for ; Mon, 16 Jun 2025 06:00:30 -0700 (PDT) Received: from [10.156.60.236] (ip-037-024-206-209.um08.pools.vodafone-ip.de. [37.24.206.209]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-b2fe164ca5fsm6787021a12.34.2025.06.16.06.00.26 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Mon, 16 Jun 2025 06:00:29 -0700 (PDT) X-Outflank-Mailman: Message body and most headers restored to incoming version X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: e1a5f503-4ab1-11f0-a309-13f23c93f187 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.com; s=google; t=1750078830; x=1750683630; darn=lists.xenproject.org; h=content-transfer-encoding:in-reply-to:autocrypt:content-language :references:cc:to:from:subject:user-agent:mime-version:date :message-id:from:to:cc:subject:date:message-id:reply-to; bh=pl2MdiB+kJYOREm+k9B0Yw8rIzN4HAkJgHoxKayTGVI=; b=NOlU31PpwkAnjBx12zZ7FxFoJ5lxRy345s0PPceqfm7A7hpIODhSdF+AztBnFN+DzZ 5KEbYvDgNyMBHc//w7OO9Ao4QEDSDPER69Ux5wbAXcP+5ZimofXQ1OLt20ERCZnz6Zg3 O5a8+zUqGRQhtgyCQllF7LMfGm8cH6vHjhAOh3i5jUDmmzqQzHe9i7GjRh2beXC4s8rE ajWpQ9k3W+BGj1pAPUzJVJYi2yvOFlfSSUaPq6HT2RB/dBXAvWHIpkvTKBICpA0v+QEr Bi8NQvMNK3oBohNs2Hi/Dvyb0HX1VCvI4pAo1NLUES/fBPv9mICFcyeTOZHskeln7/lT YfXw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1750078830; x=1750683630; h=content-transfer-encoding:in-reply-to:autocrypt:content-language :references:cc:to:from:subject:user-agent:mime-version:date :message-id:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=pl2MdiB+kJYOREm+k9B0Yw8rIzN4HAkJgHoxKayTGVI=; b=TNHnCKT2HKhZOWTB+IMQaeCejnm174V4co9oHOScw/iHdV08bsqCKbg/fsJ+SeP3IJ IhAxD1c0/F9ECG+vgTNLmw7XdTJTajG9wwvuf6G+ML72lK7MrdMZborcGn/9zen/paXK Mo3yBxHlvJ6lBDG8H0q0igVID+RjwOtMigguJtDdCQgSC1TbERY9Mxntw9XRZpM+WbbO EFU/eSKkGUdyPAqPJNMC1Wble4bUMTaoPifsN1tKeEuJ/fA5dgP1IY5EkxEFBs2BxJQ+ uIRMhX7lyCwgldUeVTdrvNgNYu1xMYtIxIpWSmYvRJlkWVvsRMSt1jgKBsA0XJAPaInU JWZg== X-Gm-Message-State: AOJu0YwbdUjxUnKWG4EXMuxVVdqV0Tt8lF/EFq7gC7MkpIFAIs8m469O v4YW3/6wGbTVfdb7kb9eVEr1NH9hWxrPYsqtF8C/QpLCRaOnnIFOmyu7+73/qwfq0k9QrFRuG2o 5Kzc= X-Gm-Gg: ASbGncsCqIPwgxInVnqWR4OAbxZczu2/ty935OfvezKOOtqDL9uaSVzKzo4kiY5C1aV nallxkXEvG6Xgbfz5/2uuyTrwh7J7lEVsI++qyF+opRd7xjE/jRXv3Iz4XklwJNSc9XHdawPvXE /dnaUrAAWVf+a8mxO5DnqmCqWHDuZrc/wu/lzPyj5+tRJbDbKybm9UNm3oSnxj8GnZQHLdQZmW6 8R6fgb1fsOgMpFBWSlbyP5H4ttSsakepggj06bWdL6qpaulBsSWxQqKm6AlwQMwb+iu4MitmXT9 KwmIT1HXyUQlJfWflLrUd4ze9xB4pNCb6adY+QURuAhXGYneMEauwfoJPknliErmxaYgJSkYib/ tI8YcaTrj++UKJcSrkhL1oRsBraNRSnD2x9Ycj/U0hGM7ktU= X-Google-Smtp-Source: AGHT+IGq3KMYoJsXLu0qEqM3HhiyH8Fd/f20KFa779MiA8R/Ui7xgF7YCvdIS6kLCWefSfxrnkPBjg== X-Received: by 2002:a05:6000:2881:b0:3a4:eef5:ae26 with SMTP id ffacd0b85a97d-3a572e9e42dmr5544178f8f.55.1750078829855; Mon, 16 Jun 2025 06:00:29 -0700 (PDT) Message-ID: Date: Mon, 16 Jun 2025 15:00:23 +0200 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: [PATCH v6 3/7] x86: re-work memset() From: Jan Beulich To: "xen-devel@lists.xenproject.org" Cc: Andrew Cooper , =?UTF-8?Q?Roger_Pau_Monn=C3=A9?= References: Content-Language: en-US Autocrypt: addr=jbeulich@suse.com; keydata= xsDiBFk3nEQRBADAEaSw6zC/EJkiwGPXbWtPxl2xCdSoeepS07jW8UgcHNurfHvUzogEq5xk hu507c3BarVjyWCJOylMNR98Yd8VqD9UfmX0Hb8/BrA+Hl6/DB/eqGptrf4BSRwcZQM32aZK 7Pj2XbGWIUrZrd70x1eAP9QE3P79Y2oLrsCgbZJfEwCgvz9JjGmQqQkRiTVzlZVCJYcyGGsD /0tbFCzD2h20ahe8rC1gbb3K3qk+LpBtvjBu1RY9drYk0NymiGbJWZgab6t1jM7sk2vuf0Py O9Hf9XBmK0uE9IgMaiCpc32XV9oASz6UJebwkX+zF2jG5I1BfnO9g7KlotcA/v5ClMjgo6Gl MDY4HxoSRu3i1cqqSDtVlt+AOVBJBACrZcnHAUSuCXBPy0jOlBhxPqRWv6ND4c9PH1xjQ3NP nxJuMBS8rnNg22uyfAgmBKNLpLgAGVRMZGaGoJObGf72s6TeIqKJo/LtggAS9qAUiuKVnygo 3wjfkS9A3DRO+SpU7JqWdsveeIQyeyEJ/8PTowmSQLakF+3fote9ybzd880fSmFuIEJldWxp Y2ggPGpiZXVsaWNoQHN1c2UuY29tPsJgBBMRAgAgBQJZN5xEAhsDBgsJCAcDAgQVAggDBBYC AwECHgECF4AACgkQoDSui/t3IH4J+wCfQ5jHdEjCRHj23O/5ttg9r9OIruwAn3103WUITZee e7Sbg12UgcQ5lv7SzsFNBFk3nEQQCACCuTjCjFOUdi5Nm244F+78kLghRcin/awv+IrTcIWF hUpSs1Y91iQQ7KItirz5uwCPlwejSJDQJLIS+QtJHaXDXeV6NI0Uef1hP20+y8qydDiVkv6l IreXjTb7DvksRgJNvCkWtYnlS3mYvQ9NzS9PhyALWbXnH6sIJd2O9lKS1Mrfq+y0IXCP10eS FFGg+Av3IQeFatkJAyju0PPthyTqxSI4lZYuJVPknzgaeuJv/2NccrPvmeDg6Coe7ZIeQ8Yj t0ARxu2xytAkkLCel1Lz1WLmwLstV30g80nkgZf/wr+/BXJW/oIvRlonUkxv+IbBM3dX2OV8 AmRv1ySWPTP7AAMFB/9PQK/VtlNUJvg8GXj9ootzrteGfVZVVT4XBJkfwBcpC/XcPzldjv+3 HYudvpdNK3lLujXeA5fLOH+Z/G9WBc5pFVSMocI71I8bT8lIAzreg0WvkWg5V2WZsUMlnDL9 mpwIGFhlbM3gfDMs7MPMu8YQRFVdUvtSpaAs8OFfGQ0ia3LGZcjA6Ik2+xcqscEJzNH+qh8V m5jjp28yZgaqTaRbg3M/+MTbMpicpZuqF4rnB0AQD12/3BNWDR6bmh+EkYSMcEIpQmBM51qM EKYTQGybRCjpnKHGOxG0rfFY1085mBDZCH5Kx0cl0HVJuQKC+dV2ZY5AqjcKwAxpE75MLFkr wkkEGBECAAkFAlk3nEQCGwwACgkQoDSui/t3IH7nnwCfcJWUDUFKdCsBH/E5d+0ZnMQi+G0A nAuWpQkjM1ASeQwSHEeAWPgskBQL In-Reply-To: Content-Transfer-Encoding: quoted-printable X-ZohoMail-DKIM: pass (identity @suse.com) X-ZM-MESSAGEID: 1750078849093116600 Content-Type: text/plain; charset="utf-8" Move the function to its own assembly file. Having it in C just for the entire body to be an asm() isn't really helpful. Then have two flavors: A "basic" version using qword steps for the bulk of the operation, and an ERMS version for modern hardware, to be substituted in via alternatives patching. For RET to be usable in an alternative's replacement code, extend the CALL/JMP patching to cover the case of "JMP __x86_return_thunk" coming last in replacement code. Signed-off-by: Jan Beulich Reviewed-by: Jason Andryuk --- We may want to consider branching over the REP STOSQ as well, if the number of qwords turns out to be zero. We may also want to consider using non-REP STOS{L,W,B} for the tail. --- v5: Re-base. v4: Use %r8 instead of %rsi in a few places. v3: Re-base. --- a/xen/arch/x86/Makefile +++ b/xen/arch/x86/Makefile @@ -47,6 +47,7 @@ obj-$(CONFIG_RETURN_THUNK) +=3D indirect-t obj-$(CONFIG_PV) +=3D ioport_emulate.o obj-y +=3D irq.o obj-$(CONFIG_KEXEC) +=3D machine_kexec.o +obj-y +=3D memset.o obj-y +=3D mm.o x86_64/mm.o obj-$(CONFIG_VM_EVENT) +=3D monitor.o obj-y +=3D mpparse.o --- a/xen/arch/x86/alternative.c +++ b/xen/arch/x86/alternative.c @@ -351,6 +351,12 @@ static int init_or_livepatch _apply_alte /* 0xe8/0xe9 are relative branches; fix the offset. */ if ( a->repl_len >=3D 5 && (*buf & 0xfe) =3D=3D 0xe8 ) *(int32_t *)(buf + 1) +=3D repl - orig; + else if ( IS_ENABLED(CONFIG_RETURN_THUNK) && + a->repl_len > 5 && buf[a->repl_len - 5] =3D=3D 0xe9 && + ((long)repl + a->repl_len + + *(int32_t *)(buf + a->repl_len - 4) =3D=3D + (long)__x86_return_thunk) ) + *(int32_t *)(buf + a->repl_len - 4) +=3D repl - orig; =20 a->priv =3D 1; =20 --- /dev/null +++ b/xen/arch/x86/memset.S @@ -0,0 +1,30 @@ +#include + +.macro memset + and $7, %edx + shr $3, %rcx + movzbl %sil, %esi + mov $0x0101010101010101, %rax + imul %rsi, %rax + mov %rdi, %r8 + rep stosq + or %edx, %ecx + jz 0f + rep stosb +0: + mov %r8, %rax + RET +.endm + +.macro memset_erms + mov %esi, %eax + mov %rdi, %r8 + rep stosb + mov %r8, %rax + RET +.endm + +FUNC(memset) + mov %rdx, %rcx + ALTERNATIVE memset, memset_erms, X86_FEATURE_ERMS +END(memset) --- a/xen/arch/x86/string.c +++ b/xen/arch/x86/string.c @@ -22,19 +22,6 @@ void *(memcpy)(void *dest, const void *s return dest; } =20 -void *(memset)(void *s, int c, size_t n) -{ - long d0, d1; - - asm volatile ( - "rep stosb" - : "=3D&c" (d0), "=3D&D" (d1) - : "a" (c), "1" (s), "0" (n) - : "memory"); - - return s; -} - void *(memmove)(void *dest, const void *src, size_t n) { long d0, d1, d2; From nobody Fri Oct 31 03:42:51 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) client-ip=192.237.175.120; envelope-from=xen-devel-bounces@lists.xenproject.org; helo=lists.xenproject.org; Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=pass(p=quarantine dis=none) header.from=suse.com ARC-Seal: i=1; a=rsa-sha256; t=1750078882; cv=none; d=zohomail.com; s=zohoarc; b=R3QXnh178D+f4bg8Rr3+L7dzxtdu3TiX5kEZWEiYKPFi0mr5SzhrhUXb+ghaJACJGS2EsAo/DveacG02a02UZBqxzz9GrKyc5W6F8UDz1gXYW4v1j6nUKtU829FzgfsAF9dNb/MKMWXry/LNKIip1c7EoHNv6NMrvgOB3sZ6NjE= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1750078882; h=Content-Type:Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=WQcA0jk9C0hbnGwgMgyk6/kBDd3xLT1I9c+U8NiS7QA=; b=WyzTrvArse0qVGKaPooqUjHnKd8Wkzf4hB03oFx72niAmMYpyvZReBj7vbMz/MK2B62JU2rkpS1vn6X4rSKdaypGj+WCD//IvQHhOhvzuoEs+QXl9LeM86MSPpH9XVe8o/10Y9M/nhppYTMfAYV6fPtYJA15XSDxL67DxwKhKe4= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=pass header.from= (p=quarantine dis=none) Return-Path: Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) by mx.zohomail.com with SMTPS id 1750078882773598.4924079914747; Mon, 16 Jun 2025 06:01:22 -0700 (PDT) Received: from list by lists.xenproject.org with outflank-mailman.1017228.1394200 (Exim 4.92) (envelope-from ) id 1uR9SU-0006PK-LN; Mon, 16 Jun 2025 13:01:06 +0000 Received: by outflank-mailman (output) from mailman id 1017228.1394200; Mon, 16 Jun 2025 13:01:06 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1uR9SU-0006PD-Gf; Mon, 16 Jun 2025 13:01:06 +0000 Received: by outflank-mailman (input) for mailman id 1017228; Mon, 16 Jun 2025 13:01:06 +0000 Received: from se1-gles-flk1-in.inumbo.com ([94.247.172.50] helo=se1-gles-flk1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1uR9SU-0003pW-7t for xen-devel@lists.xenproject.org; Mon, 16 Jun 2025 13:01:06 +0000 Received: from mail-wr1-x429.google.com (mail-wr1-x429.google.com [2a00:1450:4864:20::429]) by se1-gles-flk1.inumbo.com (Halon) with ESMTPS id f5f27c6f-4ab1-11f0-b894-0df219b8e170; Mon, 16 Jun 2025 15:01:04 +0200 (CEST) Received: by mail-wr1-x429.google.com with SMTP id ffacd0b85a97d-3a365a6804eso3189285f8f.3 for ; Mon, 16 Jun 2025 06:01:04 -0700 (PDT) Received: from [10.156.60.236] (ip-037-024-206-209.um08.pools.vodafone-ip.de. [37.24.206.209]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-748900d7e0asm6939967b3a.174.2025.06.16.06.01.00 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Mon, 16 Jun 2025 06:01:03 -0700 (PDT) X-Outflank-Mailman: Message body and most headers restored to incoming version X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: f5f27c6f-4ab1-11f0-b894-0df219b8e170 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.com; s=google; t=1750078864; x=1750683664; darn=lists.xenproject.org; h=content-transfer-encoding:in-reply-to:autocrypt:content-language :references:cc:to:from:subject:user-agent:mime-version:date :message-id:from:to:cc:subject:date:message-id:reply-to; bh=WQcA0jk9C0hbnGwgMgyk6/kBDd3xLT1I9c+U8NiS7QA=; b=EnFl/Lo0Dsq2zXASpFbbOcqQE7FWxD3vEprjfbu910qv3pzyDrXPENfEQg025sbWME dw/7C5hmV/a8630+Et0F162h2zMoOl11xp8v5DwR/cjYA8oly+wbKndBrFaGu7U1EjvH HeTud35xSWx+zHvPW47mBC9ukQFsMw2P29Iolwacxd+B26vFTR5T1DqoswaNnNRXtk0c 3Pf0bAi/ATh1gOEjegBOLXq7NaY8z3NNalzIAI2nsHMKC4hMwk81fMY8FdRdB3QBdDLT x2Dkh6Z36fJw0cwupXA8qejtwFdMapuibYslTzjv6SIa2EqAsgeqsYClblKMKG/geuri ibDw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1750078864; x=1750683664; h=content-transfer-encoding:in-reply-to:autocrypt:content-language :references:cc:to:from:subject:user-agent:mime-version:date :message-id:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=WQcA0jk9C0hbnGwgMgyk6/kBDd3xLT1I9c+U8NiS7QA=; b=Na4jqUljAxMDCdy5lNlieASPD89SIghyYMzTTzSiiwbgtGE3pDDiag26ESyP1g7Fj+ TwZTNA7rORDC13mnIWnD5ZMyABSjgMpsHUui/vGyuBJQCu96NoaszZr8oSbja/s8A440 fpTKLrlx4N7//eEYaWNAB2dz4xcJ1bJX0RRnO+ef81bok+QJDjev9PVLcsedydNl2yIO dA62ZZVrUAnHW2c0Tz8tWXZw0he7EBH0t9s3htisebBthToJvA/pW0gN4HlGZGTQGdvr yCyIIXU5zvEt7u0G6aeff1F/BZsipewavs4QpnG/5egs4o6qLjv2vBSvsW5IGf70hX44 mEzQ== X-Gm-Message-State: AOJu0Yy2uAlgtVmAGZtuGHggx72MztRt8k0+0NTs5lcnkWKSg+RAmOmy vl2OAOzrwT8X7Xt+hRuji62I0OLETyqtdo/U8/8AC0IzFcd2faKPyoYRgnN00815AoyTYRSRanR ikQ4= X-Gm-Gg: ASbGnctQcwgFJvVdt3cyKgYzrwjbCxuujbedFllGjpUd3qJhmxicWUhebcII0Xi2ajL RZs1MqOhA4I8QDFYf55mazbzsC62KDugdgzPLVtbevgzZXO+DhJkcEGxqafu5a2jE5zazXkL7BM OIGQG6WM7tzdhZUGyRv5ScfBvDf1et+ffuxV5e83JNM5KtWDKiZwXzkuC4Lb4Zr7NbfELlLi+zI WUI8wJk66w7Wwy8hmGFAehz9qROJGUmC1oVf9VUbfSL1ds7JZtgQVL1BARlvlU+QLGR61S5AXAi fuvQVdvN7Imf/AojWd3l7mbs9wHLgQSkBjVvFdQ92Y6tWNbM9ioTZUepZWHE6vGif0cEyrvCnGK CGRcXccz8xiyXbvl8vSJsxmSxhbhxY82vc7uKmhhNpUxuBVY= X-Google-Smtp-Source: AGHT+IFf29AbfGPH26yutrwPeNXyRBv8TOMvSaOgP0uOAz18ppyA5lsUOZBFuNdhP7jBbzTRijBxzQ== X-Received: by 2002:a05:6000:290a:b0:3a4:f520:8bfc with SMTP id ffacd0b85a97d-3a5723a3b03mr6780273f8f.36.1750078863555; Mon, 16 Jun 2025 06:01:03 -0700 (PDT) Message-ID: <991c9caf-c7b6-4e1f-9855-5b2930b7989c@suse.com> Date: Mon, 16 Jun 2025 15:00:56 +0200 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: [PATCH v6 4/7] x86: re-work memcpy() From: Jan Beulich To: "xen-devel@lists.xenproject.org" Cc: Andrew Cooper , =?UTF-8?Q?Roger_Pau_Monn=C3=A9?= References: Content-Language: en-US Autocrypt: addr=jbeulich@suse.com; keydata= xsDiBFk3nEQRBADAEaSw6zC/EJkiwGPXbWtPxl2xCdSoeepS07jW8UgcHNurfHvUzogEq5xk hu507c3BarVjyWCJOylMNR98Yd8VqD9UfmX0Hb8/BrA+Hl6/DB/eqGptrf4BSRwcZQM32aZK 7Pj2XbGWIUrZrd70x1eAP9QE3P79Y2oLrsCgbZJfEwCgvz9JjGmQqQkRiTVzlZVCJYcyGGsD /0tbFCzD2h20ahe8rC1gbb3K3qk+LpBtvjBu1RY9drYk0NymiGbJWZgab6t1jM7sk2vuf0Py O9Hf9XBmK0uE9IgMaiCpc32XV9oASz6UJebwkX+zF2jG5I1BfnO9g7KlotcA/v5ClMjgo6Gl MDY4HxoSRu3i1cqqSDtVlt+AOVBJBACrZcnHAUSuCXBPy0jOlBhxPqRWv6ND4c9PH1xjQ3NP nxJuMBS8rnNg22uyfAgmBKNLpLgAGVRMZGaGoJObGf72s6TeIqKJo/LtggAS9qAUiuKVnygo 3wjfkS9A3DRO+SpU7JqWdsveeIQyeyEJ/8PTowmSQLakF+3fote9ybzd880fSmFuIEJldWxp Y2ggPGpiZXVsaWNoQHN1c2UuY29tPsJgBBMRAgAgBQJZN5xEAhsDBgsJCAcDAgQVAggDBBYC AwECHgECF4AACgkQoDSui/t3IH4J+wCfQ5jHdEjCRHj23O/5ttg9r9OIruwAn3103WUITZee e7Sbg12UgcQ5lv7SzsFNBFk3nEQQCACCuTjCjFOUdi5Nm244F+78kLghRcin/awv+IrTcIWF hUpSs1Y91iQQ7KItirz5uwCPlwejSJDQJLIS+QtJHaXDXeV6NI0Uef1hP20+y8qydDiVkv6l IreXjTb7DvksRgJNvCkWtYnlS3mYvQ9NzS9PhyALWbXnH6sIJd2O9lKS1Mrfq+y0IXCP10eS FFGg+Av3IQeFatkJAyju0PPthyTqxSI4lZYuJVPknzgaeuJv/2NccrPvmeDg6Coe7ZIeQ8Yj t0ARxu2xytAkkLCel1Lz1WLmwLstV30g80nkgZf/wr+/BXJW/oIvRlonUkxv+IbBM3dX2OV8 AmRv1ySWPTP7AAMFB/9PQK/VtlNUJvg8GXj9ootzrteGfVZVVT4XBJkfwBcpC/XcPzldjv+3 HYudvpdNK3lLujXeA5fLOH+Z/G9WBc5pFVSMocI71I8bT8lIAzreg0WvkWg5V2WZsUMlnDL9 mpwIGFhlbM3gfDMs7MPMu8YQRFVdUvtSpaAs8OFfGQ0ia3LGZcjA6Ik2+xcqscEJzNH+qh8V m5jjp28yZgaqTaRbg3M/+MTbMpicpZuqF4rnB0AQD12/3BNWDR6bmh+EkYSMcEIpQmBM51qM EKYTQGybRCjpnKHGOxG0rfFY1085mBDZCH5Kx0cl0HVJuQKC+dV2ZY5AqjcKwAxpE75MLFkr wkkEGBECAAkFAlk3nEQCGwwACgkQoDSui/t3IH7nnwCfcJWUDUFKdCsBH/E5d+0ZnMQi+G0A nAuWpQkjM1ASeQwSHEeAWPgskBQL In-Reply-To: Content-Transfer-Encoding: quoted-printable X-ZohoMail-DKIM: pass (identity @suse.com) X-ZM-MESSAGEID: 1750078886512116600 Content-Type: text/plain; charset="utf-8" Move the function to its own assembly file. Having it in C just for the entire body to be an asm() isn't really helpful. Then have two flavors: A "basic" version using qword steps for the bulk of the operation, and an ERMS version for modern hardware, to be substituted in via alternatives patching. Alternatives patching, however, requires an extra precaution: It uses memcpy() itself, and hence the function may patch itself. Luckily the patched-in code only replaces the prolog of the original function. Make sure this remains this way. Signed-off-by: Jan Beulich Reviewed-by: Teddy Astie Reviewed-by: Jason Andryuk --- We may want to consider branching over the REP MOVSQ as well, if the number of qwords turns out to be zero. We may also want to consider using non-REP MOVS{L,W,B} for the tail. TBD: We may further need a workaround similar to Linux'es 8ca97812c3c8 ("x86/mce: Work around an erratum on fast string copy instructions"). TBD: Some older AMD CPUs have an issue with REP MOVS when source and destination are misaligned with one another (modulo 32?), which may require a separate memcpy() flavor. --- v6: Move alternatives.c change to a separate patch. v5: Re-base. v4: Use CR2 write as serializing insn, and limit its use to boot time. v3: Re-base. --- a/xen/arch/x86/Makefile +++ b/xen/arch/x86/Makefile @@ -47,6 +47,7 @@ obj-$(CONFIG_RETURN_THUNK) +=3D indirect-t obj-$(CONFIG_PV) +=3D ioport_emulate.o obj-y +=3D irq.o obj-$(CONFIG_KEXEC) +=3D machine_kexec.o +obj-y +=3D memcpy.o obj-y +=3D memset.o obj-y +=3D mm.o x86_64/mm.o obj-$(CONFIG_VM_EVENT) +=3D monitor.o --- /dev/null +++ b/xen/arch/x86/memcpy.S @@ -0,0 +1,20 @@ +#include + +FUNC(memcpy) + mov %rdx, %rcx + mov %rdi, %rax + /* + * We need to be careful here: memcpy() is involved in alternatives + * patching, so the code doing the actual copying (i.e. past setti= ng + * up registers) may not be subject to patching (unless further + * precautions were taken). + */ + ALTERNATIVE "and $7, %edx; shr $3, %rcx", \ + STR(rep movsb; RET), X86_FEATURE_ERMS + rep movsq + or %edx, %ecx + jz 1f + rep movsb +1: + RET +END(memcpy) --- a/xen/arch/x86/string.c +++ b/xen/arch/x86/string.c @@ -7,21 +7,6 @@ =20 #include =20 -void *(memcpy)(void *dest, const void *src, size_t n) -{ - long d0, d1, d2; - - asm volatile ( - " rep ; movs"__OS" ; " - " mov %k4,%k3 ; " - " rep ; movsb " - : "=3D&c" (d0), "=3D&D" (d1), "=3D&S" (d2) - : "0" (n/BYTES_PER_LONG), "r" (n%BYTES_PER_LONG), "1" (dest), "2" = (src) - : "memory" ); - - return dest; -} - void *(memmove)(void *dest, const void *src, size_t n) { long d0, d1, d2; From nobody Fri Oct 31 03:42:51 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) client-ip=192.237.175.120; envelope-from=xen-devel-bounces@lists.xenproject.org; helo=lists.xenproject.org; Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=pass(p=quarantine dis=none) header.from=suse.com ARC-Seal: i=1; a=rsa-sha256; t=1750079493; cv=none; d=zohomail.com; s=zohoarc; b=OonQkpMO7hdOcdj3gR9NwKlDQIlwpEG1STDU+6IP890qWP1bk5LvxWGa81WqLWyWmxzCrwMidQwzsv+ZaRB/FSS4fZWf3xFR3fl5ipuHMCau0INhF4lUD5W0Q2jrQbKt1bdKwhZ7XN4WdIUWmU4rAkE5cpkEPdChResVoMZuQVw= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1750079493; h=Content-Type:Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=/Qgc++fc93ux80i5RmqliRbRkUUe9ZSRv/76bEt+jaE=; b=Gw1nVwjUnhDPxPGNQDJXnzZuYVqb20CIyaixHBXCTMN6XCWg54Ww61fcDxnDGO2OyOLaeLjqsb3T6I/WkUesqCbCg0gGdlF9Rg7QYmtFSB0qpjp7gmq9vepgOYjnR9FsA77U7Ki+cdfdr13NBto3GKGgLJict8aumpMWEgcKPuM= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=pass header.from= (p=quarantine dis=none) Return-Path: Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) by mx.zohomail.com with SMTPS id 1750079493196939.8988042039238; Mon, 16 Jun 2025 06:11:33 -0700 (PDT) Received: from list by lists.xenproject.org with outflank-mailman.1017255.1394228 (Exim 4.92) (envelope-from ) id 1uR9c8-0000xA-2O; Mon, 16 Jun 2025 13:11:04 +0000 Received: by outflank-mailman (output) from mailman id 1017255.1394228; Mon, 16 Jun 2025 13:11:04 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1uR9c7-0000x3-W6; Mon, 16 Jun 2025 13:11:03 +0000 Received: by outflank-mailman (input) for mailman id 1017255; Mon, 16 Jun 2025 13:11:02 +0000 Received: from se1-gles-flk1-in.inumbo.com ([94.247.172.50] helo=se1-gles-flk1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1uR9Sv-0003pW-TN for xen-devel@lists.xenproject.org; Mon, 16 Jun 2025 13:01:33 +0000 Received: from mail-wr1-x42d.google.com (mail-wr1-x42d.google.com [2a00:1450:4864:20::42d]) by se1-gles-flk1.inumbo.com (Halon) with ESMTPS id 067907d3-4ab2-11f0-b894-0df219b8e170; Mon, 16 Jun 2025 15:01:32 +0200 (CEST) Received: by mail-wr1-x42d.google.com with SMTP id ffacd0b85a97d-3a4f72cba73so3799122f8f.1 for ; Mon, 16 Jun 2025 06:01:32 -0700 (PDT) Received: from [10.156.60.236] (ip-037-024-206-209.um08.pools.vodafone-ip.de. [37.24.206.209]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-2365dea8fd6sm60146695ad.145.2025.06.16.06.01.27 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Mon, 16 Jun 2025 06:01:30 -0700 (PDT) X-Outflank-Mailman: Message body and most headers restored to incoming version X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: 067907d3-4ab2-11f0-b894-0df219b8e170 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.com; s=google; t=1750078892; x=1750683692; darn=lists.xenproject.org; h=content-transfer-encoding:in-reply-to:autocrypt:content-language :references:cc:to:from:subject:user-agent:mime-version:date :message-id:from:to:cc:subject:date:message-id:reply-to; bh=/Qgc++fc93ux80i5RmqliRbRkUUe9ZSRv/76bEt+jaE=; b=Fvd92rhX/gAPMsa0EmEbLm/snAK+YAuvMOcGc+F4C9qGFP9a48R8me16HREG6w2x4q uG6m0f0MywfucW1CmdpWS8M7/hRxRaDXgslMwk3xYzQevpts6EoqYRpN6sblifzs4qWd NiatMNclheC+3J4dhJg3cSVjs/FcF4WR+q38b+HNwkKXY8XB1s/ltOl7iaP2EWHjHvM0 2GXoEZysUpnmNV7P8F9q8rnO7wzW4IludakIz/aGJPmI21zxHtxsL6nufT13CSqNQ4fa M80DZbkqPOQ0YrFbyLIMfau92FaHwnlO5JQKS8pgRQFF59+Ajq+juOkLvmDR4VjVIkIN DnLw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1750078892; x=1750683692; h=content-transfer-encoding:in-reply-to:autocrypt:content-language :references:cc:to:from:subject:user-agent:mime-version:date :message-id:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=/Qgc++fc93ux80i5RmqliRbRkUUe9ZSRv/76bEt+jaE=; b=P5GeYDflz5zQyQZ+eSYMzc+vpxZN9TTBHwOBYxMgy2Vu5az9m/yj6PSZo/NXtoi6Eg c9OTQcw7wdN4MdEUeh/lErNCcGz4ya/iX3mEIMhlscbjA37cBICuKOXu94EIpuz+lKOZ myiFjmolrT8ETBTDR8UnPWis8Leg+AABqRscgo7Wkolpibt3WynZ/f2VQZihCQStsZTP /8IJdDVx54JBM4ZhzIyK4Lg1IdhBDAdFm4FdK+n2hAL9oPlq+Y77uARy2LEuED+tEUA/ xuYKrgPRN3A/bPVcYRCepFtPUekVizj+ypG+krk1VmBtpT5ZdRXtMcp5ZqapU7sChk8k d1GA== X-Gm-Message-State: AOJu0YyJ8EvXAPlMf7NoIpCLcvgtOkZcStwCCq4yIoxgNuvsAKbBLiL2 1UCp/aloHSlCEFo31cA1RaMzdAeC8g34OnlUlE/+8jlucqO62Nps/IK70PdqyDgWD23zgfHTrh5 rCDs= X-Gm-Gg: ASbGncsW5qZAzWKzGWBAf1uv3EsJnOMfk9mve5KJY9itoSGLF6e6F9bCxoC0B3IVWgk g9289I/XBQnvpAKUB1Gg6XitRYOjk7gK64Exp8QHFNvwwH8X7vrF3X1TnTPI30NVghZS/TD5yya Gs2JS2rWW5Pez5uoZaOaoES6qB6ujw8DEn+XMIs9uH4Rf/CuLcOrXQckLd4UuAvT5WPkoROzjZX fZNQzCdiwGWiPMN6HeBYA1oGY5nLM/OXO7MwQcFHQ10S6RrL2Sn4MdCqt0pBLa4jolrMfadU2Rt HsKej17trJZ7LOpSF36EXDvfAcLoVfbZfVFkdyh98KHwSn2F51njqSGs5L2NHfPSjzvFIP9SJqC xwOvgae7j7DG2Sr/CDd+wnFAP1EO17zFTc9GPslypb7CAnsk= X-Google-Smtp-Source: AGHT+IF1/BiEdagP71nQb6AOqDYr1F9/K/R9nw/EWajmZrQN1TpdqMHOI4v1akSjxyuOoUWijBDTNg== X-Received: by 2002:a05:6000:2f88:b0:3a5:1360:6b34 with SMTP id ffacd0b85a97d-3a56d7bacfdmr8843603f8f.2.1750078890953; Mon, 16 Jun 2025 06:01:30 -0700 (PDT) Message-ID: Date: Mon, 16 Jun 2025 15:01:24 +0200 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: [PATCH v6 5/7] x86: control memset() and memcpy() inlining From: Jan Beulich To: "xen-devel@lists.xenproject.org" Cc: Andrew Cooper , =?UTF-8?Q?Roger_Pau_Monn=C3=A9?= References: Content-Language: en-US Autocrypt: addr=jbeulich@suse.com; keydata= xsDiBFk3nEQRBADAEaSw6zC/EJkiwGPXbWtPxl2xCdSoeepS07jW8UgcHNurfHvUzogEq5xk hu507c3BarVjyWCJOylMNR98Yd8VqD9UfmX0Hb8/BrA+Hl6/DB/eqGptrf4BSRwcZQM32aZK 7Pj2XbGWIUrZrd70x1eAP9QE3P79Y2oLrsCgbZJfEwCgvz9JjGmQqQkRiTVzlZVCJYcyGGsD /0tbFCzD2h20ahe8rC1gbb3K3qk+LpBtvjBu1RY9drYk0NymiGbJWZgab6t1jM7sk2vuf0Py O9Hf9XBmK0uE9IgMaiCpc32XV9oASz6UJebwkX+zF2jG5I1BfnO9g7KlotcA/v5ClMjgo6Gl MDY4HxoSRu3i1cqqSDtVlt+AOVBJBACrZcnHAUSuCXBPy0jOlBhxPqRWv6ND4c9PH1xjQ3NP nxJuMBS8rnNg22uyfAgmBKNLpLgAGVRMZGaGoJObGf72s6TeIqKJo/LtggAS9qAUiuKVnygo 3wjfkS9A3DRO+SpU7JqWdsveeIQyeyEJ/8PTowmSQLakF+3fote9ybzd880fSmFuIEJldWxp Y2ggPGpiZXVsaWNoQHN1c2UuY29tPsJgBBMRAgAgBQJZN5xEAhsDBgsJCAcDAgQVAggDBBYC AwECHgECF4AACgkQoDSui/t3IH4J+wCfQ5jHdEjCRHj23O/5ttg9r9OIruwAn3103WUITZee e7Sbg12UgcQ5lv7SzsFNBFk3nEQQCACCuTjCjFOUdi5Nm244F+78kLghRcin/awv+IrTcIWF hUpSs1Y91iQQ7KItirz5uwCPlwejSJDQJLIS+QtJHaXDXeV6NI0Uef1hP20+y8qydDiVkv6l IreXjTb7DvksRgJNvCkWtYnlS3mYvQ9NzS9PhyALWbXnH6sIJd2O9lKS1Mrfq+y0IXCP10eS FFGg+Av3IQeFatkJAyju0PPthyTqxSI4lZYuJVPknzgaeuJv/2NccrPvmeDg6Coe7ZIeQ8Yj t0ARxu2xytAkkLCel1Lz1WLmwLstV30g80nkgZf/wr+/BXJW/oIvRlonUkxv+IbBM3dX2OV8 AmRv1ySWPTP7AAMFB/9PQK/VtlNUJvg8GXj9ootzrteGfVZVVT4XBJkfwBcpC/XcPzldjv+3 HYudvpdNK3lLujXeA5fLOH+Z/G9WBc5pFVSMocI71I8bT8lIAzreg0WvkWg5V2WZsUMlnDL9 mpwIGFhlbM3gfDMs7MPMu8YQRFVdUvtSpaAs8OFfGQ0ia3LGZcjA6Ik2+xcqscEJzNH+qh8V m5jjp28yZgaqTaRbg3M/+MTbMpicpZuqF4rnB0AQD12/3BNWDR6bmh+EkYSMcEIpQmBM51qM EKYTQGybRCjpnKHGOxG0rfFY1085mBDZCH5Kx0cl0HVJuQKC+dV2ZY5AqjcKwAxpE75MLFkr wkkEGBECAAkFAlk3nEQCGwwACgkQoDSui/t3IH7nnwCfcJWUDUFKdCsBH/E5d+0ZnMQi+G0A nAuWpQkjM1ASeQwSHEeAWPgskBQL In-Reply-To: Content-Transfer-Encoding: quoted-printable X-ZohoMail-DKIM: pass (identity @suse.com) X-ZM-MESSAGEID: 1750079495255124100 Content-Type: text/plain; charset="utf-8" Stop the compiler from inlining non-trivial memset() and memcpy() (for memset() see e.g. map_vcpu_info() or kimage_load_segments() for examples). This way we even keep the compiler from using REP STOSQ / REP MOVSQ when we'd prefer REP STOSB / REP MOVSB (when ERMS is available). With gcc10 this yields a modest .text size reduction (release build) of around 2k. Unfortunately these options aren't understood by the clang versions I have readily available for testing with; I'm unaware of equivalents. Note also that using cc-option-add is not an option here, or at least I couldn't make things work with it (in case the option was not supported by the compiler): The embedded comma in the option looks to be getting in the way. Requested-by: Andrew Cooper Signed-off-by: Jan Beulich Reviewed-by: Jason Andryuk --- v3: Re-base. v2: New. --- The boundary values are of course up for discussion - I wasn't really certain whether to use 16 or 32; I'd be less certain about using yet larger values. A respective Linux patch [1] uses 256 ... Similarly whether to permit the compiler to emit REP STOSQ / REP MOVSQ for known size, properly aligned blocks is up for discussion. [1] https://lore.kernel.org/lkml/20250605164733.737543-1-mjguzik@gmail.com/= T/#u --- a/xen/arch/x86/arch.mk +++ b/xen/arch/x86/arch.mk @@ -58,6 +58,9 @@ endif $(call cc-option-add,CFLAGS_stack_boundary,CC,-mpreferred-stack-boundary= =3D3) export CFLAGS_stack_boundary =20 +CFLAGS +=3D $(call cc-option,$(CC),-mmemcpy-strategy=3Dunrolled_loop:16:no= align$(comma)libcall:-1:noalign) +CFLAGS +=3D $(call cc-option,$(CC),-mmemset-strategy=3Dunrolled_loop:16:no= align$(comma)libcall:-1:noalign) + ifeq ($(CONFIG_UBSAN),y) # Don't enable alignment sanitisation. x86 has efficient unaligned access= es, # and various things (ACPI tables, hypercall pages, stubs, etc) are wont-f= ix. From nobody Fri Oct 31 03:42:51 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) client-ip=192.237.175.120; envelope-from=xen-devel-bounces@lists.xenproject.org; helo=lists.xenproject.org; Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=pass(p=quarantine dis=none) header.from=suse.com ARC-Seal: i=1; a=rsa-sha256; t=1750078930; cv=none; d=zohomail.com; s=zohoarc; b=beWLLSYV1t1DZrkPnE6rtwlRTY7/55irC1XL3fkrOKP14HwZS0ATOEHfCfz0tX6fews9LvVhfrgrEe1/Rzgrohrz6yvXlO9sDgFw/GXUbVuDH2oyjXugVD/xnBfMFIkTKm1UAdKc3l4f9dZ8AvOsm90CZ7gA+5ySQA3ZJ6Mf3Pc= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1750078930; h=Content-Type:Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=uAx+h4vdQMnFmuPeDTCptQ0FqB53/bxqCnS75yppj5g=; b=F1oxQL229Nlk/YK3rUqJBVlMfsC/s7Z82vL9xbg0SE06msKgzGpNvpVHnZjZWGulgn9is+HJUitpowjGYjvYwH8ePfUmlMYbkH3BirzBJKBual30xD4KLGRi0Ekttbp7Q3M1nGaBSsuYOXkLV6n0w1y6Cb/XrmgREFMrXLYufdQ= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=pass header.from= (p=quarantine dis=none) Return-Path: Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) by mx.zohomail.com with SMTPS id 1750078930369319.4119611520972; Mon, 16 Jun 2025 06:02:10 -0700 (PDT) Received: from list by lists.xenproject.org with outflank-mailman.1017235.1394210 (Exim 4.92) (envelope-from ) id 1uR9TK-00073I-Tv; Mon, 16 Jun 2025 13:01:58 +0000 Received: by outflank-mailman (output) from mailman id 1017235.1394210; Mon, 16 Jun 2025 13:01:58 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1uR9TK-00073B-Pg; Mon, 16 Jun 2025 13:01:58 +0000 Received: by outflank-mailman (input) for mailman id 1017235; Mon, 16 Jun 2025 13:01:58 +0000 Received: from se1-gles-sth1-in.inumbo.com ([159.253.27.254] helo=se1-gles-sth1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1uR9TK-00072w-9k for xen-devel@lists.xenproject.org; Mon, 16 Jun 2025 13:01:58 +0000 Received: from mail-lf1-x130.google.com (mail-lf1-x130.google.com [2a00:1450:4864:20::130]) by se1-gles-sth1.inumbo.com (Halon) with ESMTPS id 1578a124-4ab2-11f0-a309-13f23c93f187; Mon, 16 Jun 2025 15:01:57 +0200 (CEST) Received: by mail-lf1-x130.google.com with SMTP id 2adb3069b0e04-553be4d2fbfso1375852e87.0 for ; Mon, 16 Jun 2025 06:01:57 -0700 (PDT) Received: from [10.156.60.236] (ip-037-024-206-209.um08.pools.vodafone-ip.de. [37.24.206.209]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-748900b0d07sm6718883b3a.114.2025.06.16.06.01.51 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Mon, 16 Jun 2025 06:01:53 -0700 (PDT) X-Outflank-Mailman: Message body and most headers restored to incoming version X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: 1578a124-4ab2-11f0-a309-13f23c93f187 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.com; s=google; t=1750078917; x=1750683717; darn=lists.xenproject.org; h=content-transfer-encoding:in-reply-to:autocrypt:content-language :references:cc:to:from:subject:user-agent:mime-version:date :message-id:from:to:cc:subject:date:message-id:reply-to; bh=uAx+h4vdQMnFmuPeDTCptQ0FqB53/bxqCnS75yppj5g=; b=WZ0SPab/uQmIKxX5ZZRbfrI/v6cqQ1TP2w15L+7hmDvYdhSGCy+XTmOaAWH/hQ2ue7 w+Iftj5wgLNCZQgnr8LJhA14RW7fNWLUvC56RTOtNayx3gEbDxhu8rOHw4TAqpNWUVvS lJL5CmcyE08DczrxF4VOcqmSZBEwMXww1PLFqOw5kOpqpTSmSqOTPw6vKByra4ym8p+z qkhxb2L3NJmqbOGPP3zKW/RnvEzFppeBYvFfoqRxYj5Vn9pV0R+V1D6+g29XJU1CuBTS dNfxUON7tpl+Bw/+MCzyEXuYpOdT3YXlDjwvMrWu4IXZQq8T5Q4tsAmEuY1jdzieX84S KcRQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1750078917; x=1750683717; h=content-transfer-encoding:in-reply-to:autocrypt:content-language :references:cc:to:from:subject:user-agent:mime-version:date :message-id:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=uAx+h4vdQMnFmuPeDTCptQ0FqB53/bxqCnS75yppj5g=; b=C8yyh7NXQQ7AyapY7r4pHMapzWIjD8zas0VL+lbBwvOuvh0Vt38p6uvMDU5cIaajNf +v+lEt7rvDS4sTeLy6BRp8FhsGV0f5cxVBmC8MtIOiq03icTnZCjN+ewOSyP/n3/Auge POnV57ByLrI6sndkZ2SBuky+Bh2cs0UZt8WCdL+O3+yE0GTLqIITzgrI5PJapFaU+nWP B8VIH/Q6+Sz81U/JksFS1Wg1XTeaNbtASE9yu8yhlPBMzA4+ZdRRgti9Uz+fFvef24k2 ILIgdiL9QaTTMYhPnw4kqUOM1V9Mh0ec5e+VT9x8yNaDjrWZ6C/1f+YI3uSPz9VsU4/1 ShmA== X-Gm-Message-State: AOJu0Yzto8meqCdu/EUiXrOIHEwGrbVA7PgmnHzUpHXUZG4/wVyxeW71 ZeP/VDJlAMgOWuSZMWup4ZbNevhKEAKTjIUPVLNJlnh8yDrU4S0BWwJggRSLVG91Fqd5TtbXZQO lAQI= X-Gm-Gg: ASbGnctVF8CVO3oImWFrfybn9HS3uHI04NqmGnoXjiRSTORcyuHPtWYHgWOjUbjyJk7 6XjihcFjic96kuWrSYK+uXuxhz7ostqqSHUUqAPlc1PBLFTpXGnX9TDpfSr0o3pMeRDWc3tsO8v XZZRmcu5B3x/joj2cXUTk88uMy8D1+nRrIEL5yGv7DhwsK17IFvawnOyC+G5wq94La7I2uYZJJ7 jgSfkgyKUbhgE1Mle9Ei/9PDnG90R8Ud/iwh3NhHfB8zfnCbjGxWYzRSfSkKV/ebU21+SbNd1YZ 0yhHUnDcTBOH6+cp8QOJfbif78UF9ON/SyNnx4hYSdOqe9T2GC3lRDyrxgeS1xvNuUHoHcj0v7C 1kTnh6TT82Iii6t459Hv8wJe0HlZMi6+OdKMZiSowupZ5WctwB21F7fA12w== X-Google-Smtp-Source: AGHT+IH/LQefGMLIyKc9oY2HjcKqXbVrCil1vf/R5BUo8lps01W8HAeW7Hf3ZcEI6+G7CvfD74aofA== X-Received: by 2002:a05:6512:308b:b0:553:2438:8d02 with SMTP id 2adb3069b0e04-553b6f32e7emr2019776e87.47.1750078914340; Mon, 16 Jun 2025 06:01:54 -0700 (PDT) Message-ID: Date: Mon, 16 Jun 2025 15:01:47 +0200 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: [PATCH v6 6/7] x86: introduce "hot" and "cold" page clearing functions From: Jan Beulich To: "xen-devel@lists.xenproject.org" Cc: Andrew Cooper , =?UTF-8?Q?Roger_Pau_Monn=C3=A9?= References: Content-Language: en-US Autocrypt: addr=jbeulich@suse.com; keydata= xsDiBFk3nEQRBADAEaSw6zC/EJkiwGPXbWtPxl2xCdSoeepS07jW8UgcHNurfHvUzogEq5xk hu507c3BarVjyWCJOylMNR98Yd8VqD9UfmX0Hb8/BrA+Hl6/DB/eqGptrf4BSRwcZQM32aZK 7Pj2XbGWIUrZrd70x1eAP9QE3P79Y2oLrsCgbZJfEwCgvz9JjGmQqQkRiTVzlZVCJYcyGGsD /0tbFCzD2h20ahe8rC1gbb3K3qk+LpBtvjBu1RY9drYk0NymiGbJWZgab6t1jM7sk2vuf0Py O9Hf9XBmK0uE9IgMaiCpc32XV9oASz6UJebwkX+zF2jG5I1BfnO9g7KlotcA/v5ClMjgo6Gl MDY4HxoSRu3i1cqqSDtVlt+AOVBJBACrZcnHAUSuCXBPy0jOlBhxPqRWv6ND4c9PH1xjQ3NP nxJuMBS8rnNg22uyfAgmBKNLpLgAGVRMZGaGoJObGf72s6TeIqKJo/LtggAS9qAUiuKVnygo 3wjfkS9A3DRO+SpU7JqWdsveeIQyeyEJ/8PTowmSQLakF+3fote9ybzd880fSmFuIEJldWxp Y2ggPGpiZXVsaWNoQHN1c2UuY29tPsJgBBMRAgAgBQJZN5xEAhsDBgsJCAcDAgQVAggDBBYC AwECHgECF4AACgkQoDSui/t3IH4J+wCfQ5jHdEjCRHj23O/5ttg9r9OIruwAn3103WUITZee e7Sbg12UgcQ5lv7SzsFNBFk3nEQQCACCuTjCjFOUdi5Nm244F+78kLghRcin/awv+IrTcIWF hUpSs1Y91iQQ7KItirz5uwCPlwejSJDQJLIS+QtJHaXDXeV6NI0Uef1hP20+y8qydDiVkv6l IreXjTb7DvksRgJNvCkWtYnlS3mYvQ9NzS9PhyALWbXnH6sIJd2O9lKS1Mrfq+y0IXCP10eS FFGg+Av3IQeFatkJAyju0PPthyTqxSI4lZYuJVPknzgaeuJv/2NccrPvmeDg6Coe7ZIeQ8Yj t0ARxu2xytAkkLCel1Lz1WLmwLstV30g80nkgZf/wr+/BXJW/oIvRlonUkxv+IbBM3dX2OV8 AmRv1ySWPTP7AAMFB/9PQK/VtlNUJvg8GXj9ootzrteGfVZVVT4XBJkfwBcpC/XcPzldjv+3 HYudvpdNK3lLujXeA5fLOH+Z/G9WBc5pFVSMocI71I8bT8lIAzreg0WvkWg5V2WZsUMlnDL9 mpwIGFhlbM3gfDMs7MPMu8YQRFVdUvtSpaAs8OFfGQ0ia3LGZcjA6Ik2+xcqscEJzNH+qh8V m5jjp28yZgaqTaRbg3M/+MTbMpicpZuqF4rnB0AQD12/3BNWDR6bmh+EkYSMcEIpQmBM51qM EKYTQGybRCjpnKHGOxG0rfFY1085mBDZCH5Kx0cl0HVJuQKC+dV2ZY5AqjcKwAxpE75MLFkr wkkEGBECAAkFAlk3nEQCGwwACgkQoDSui/t3IH7nnwCfcJWUDUFKdCsBH/E5d+0ZnMQi+G0A nAuWpQkjM1ASeQwSHEeAWPgskBQL In-Reply-To: Content-Transfer-Encoding: quoted-printable X-ZohoMail-DKIM: pass (identity @suse.com) X-ZM-MESSAGEID: 1750078932968116600 Content-Type: text/plain; charset="utf-8" The present clear_page_sse2() is useful in case a page isn't going to get touched again soon, or if we want to limit churn on the caches. Amend it by alternatively using CLZERO, which has been found to be quite a bit faster on Zen2 hardware at least. Note that to use CLZERO, we need to know the cache line size, and hence a feature dependency on CLFLUSH gets introduced. For cases where latency is the most important aspect, or when it is expected that sufficiently large parts of a page will get accessed again soon after the clearing, introduce a "hot" alternative. Again use alternatives patching to select between a "legacy" and an ERMS variant. Don't switch any callers just yet - this will be the subject of subsequent changes. Signed-off-by: Jan Beulich Reviewed-by: Jason Andryuk --- v5: Re-base. v3: Re-base. v2: New. --- Note: Ankur indicates that for ~L3-size or larger regions MOVNT/CLZERO is better even latency-wise. --- a/xen/arch/x86/clear_page.S +++ b/xen/arch/x86/clear_page.S @@ -5,7 +5,7 @@ #include #include =20 -FUNC(clear_page_sse2) + .macro clear_page_sse2 mov $PAGE_SIZE/32, %ecx xor %eax,%eax =20 @@ -19,4 +19,42 @@ FUNC(clear_page_sse2) =20 sfence RET -END(clear_page_sse2) + .endm + + .macro clear_page_clzero + mov %rdi, %rax + mov $PAGE_SIZE/64, %ecx + .globl clear_page_clzero_post_count +clear_page_clzero_post_count: + +0: clzero + sub $-64, %rax + .globl clear_page_clzero_post_neg_size +clear_page_clzero_post_neg_size: + sub $1, %ecx + jnz 0b + + sfence + RET + .endm + +FUNC(clear_page_cold) + ALTERNATIVE clear_page_sse2, clear_page_clzero, X86_FEATURE_CLZERO +END(clear_page_cold) + + .macro clear_page_stosb + mov $PAGE_SIZE, %ecx + xor %eax,%eax + rep stosb + .endm + + .macro clear_page_stosq + mov $PAGE_SIZE/8, %ecx + xor %eax, %eax + rep stosq + .endm + +FUNC(clear_page_hot) + ALTERNATIVE clear_page_stosq, clear_page_stosb, X86_FEATURE_ERMS + RET +END(clear_page_hot) --- a/xen/arch/x86/cpu/common.c +++ b/xen/arch/x86/cpu/common.c @@ -60,6 +60,9 @@ DEFINE_PER_CPU(bool, full_gdt_loaded); =20 DEFINE_PER_CPU(uint32_t, pkrs); =20 +extern uint32_t clear_page_clzero_post_count[]; +extern int8_t clear_page_clzero_post_neg_size[]; + void __init setup_clear_cpu_cap(unsigned int cap) { const uint32_t *dfs; @@ -357,8 +360,38 @@ void __init early_cpu_init(bool verbose) =20 edx &=3D ~cleared_caps[FEATURESET_1d]; ecx &=3D ~cleared_caps[FEATURESET_1c]; - if (edx & cpufeat_mask(X86_FEATURE_CLFLUSH)) - c->x86_cache_alignment =3D ((ebx >> 8) & 0xff) * 8; + if (edx & cpufeat_mask(X86_FEATURE_CLFLUSH)) { + unsigned int size =3D ((ebx >> 8) & 0xff) * 8; + + c->x86_cache_alignment =3D size; + + /* + * Patch in parameters of clear_page_cold()'s CLZERO + * alternative. Note that for now we cap this at 128 bytes. + * Larger cache line sizes would still be dealt with + * correctly, but would cause redundant work done. + */ + if (size > 128) + size =3D 128; + if (size && !(size & (size - 1))) { + /* + * Need to play some games to keep the compiler from + * recognizing the negative array index as being out + * of bounds. The labels in assembler code really are + * _after_ the locations to be patched, so the + * negative index is intentional. + */ + uint32_t *pcount =3D clear_page_clzero_post_count; + int8_t *neg_size =3D clear_page_clzero_post_neg_size; + + OPTIMIZER_HIDE_VAR(pcount); + OPTIMIZER_HIDE_VAR(neg_size); + pcount[-1] =3D PAGE_SIZE / size; + neg_size[-1] =3D -size; + } + else + setup_clear_cpu_cap(X86_FEATURE_CLZERO); + } /* Leaf 0x1 capabilities filled in early for Xen. */ c->x86_capability[FEATURESET_1d] =3D edx; c->x86_capability[FEATURESET_1c] =3D ecx; --- a/xen/arch/x86/include/asm/asm-defns.h +++ b/xen/arch/x86/include/asm/asm-defns.h @@ -1,5 +1,9 @@ #include =20 +.macro clzero + .byte 0x0f, 0x01, 0xfc +.endm + /* * Call a noreturn function. This could be JMP, but CALL results in a more * helpful backtrace. BUG is to catch functions which do decide to return= ... --- a/xen/arch/x86/include/asm/page.h +++ b/xen/arch/x86/include/asm/page.h @@ -219,10 +219,11 @@ typedef struct { u64 pfn; } pagetable_t; #define pagetable_from_paddr(p) pagetable_from_pfn((p)>>PAGE_SHIFT) #define pagetable_null() pagetable_from_pfn(0) =20 -void clear_page_sse2(void *pg); +void clear_page_hot(void *pg); +void clear_page_cold(void *pg); void copy_page_sse2(void *to, const void *from); =20 -#define clear_page(_p) clear_page_sse2(_p) +#define clear_page(_p) clear_page_cold(_p) #define copy_page(_t, _f) copy_page_sse2(_t, _f) =20 /* Convert between Xen-heap virtual addresses and machine addresses. */ --- a/xen/tools/gen-cpuid.py +++ b/xen/tools/gen-cpuid.py @@ -212,6 +212,10 @@ def crunch_numbers(state): # the first place. APIC: [X2APIC, TSC_DEADLINE, EXTAPIC], =20 + # The CLZERO insn requires a means to determine the cache line siz= e, + # which is tied to the CLFLUSH insn. + CLFLUSH: [CLZERO], + # AMD built MMXExtentions and 3DNow as extentions to MMX. MMX: [MMXEXT, _3DNOW], From nobody Fri Oct 31 03:42:51 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) client-ip=192.237.175.120; envelope-from=xen-devel-bounces@lists.xenproject.org; helo=lists.xenproject.org; Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=pass(p=quarantine dis=none) header.from=suse.com ARC-Seal: i=1; a=rsa-sha256; t=1750078996; cv=none; d=zohomail.com; s=zohoarc; b=jNircenZ6HUku6rfhG4CxdYUnQ/uFyp4oCWQtySTVcBZv7eTgYz0lfBp0Z0a9tgot5U+Xz2uTJfvbk8CFQkAl8VPgJyFxrcOZJHyxrGvPBLW+xynpKHeEshrnd63WA7Zt78w3NeVnW5XzU1SSsAg2W5F7Oz83yolGFVv3lfN1eY= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1750078996; h=Content-Type:Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=+UiMK0XqUdP3w4VQXOM4KTrdMDDbozq0m8l9m7I8RdI=; b=gsUEJWLkkEHHEWaJrShPcRhC91YdZKvlYUBz7q+OOljRwm5qLiGn8b1JRewmzvYfriLgIb5mR87tQ7EXkwTCYxf+ZrQ2tpnZhOwO387IQkBOTD/Hgu6fo3xJaO7S+KwnEFqBennD3JIW4BWNGWjFvJKVIt20Mlk0Rb9IYUXCxCw= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=pass header.from= (p=quarantine dis=none) Return-Path: Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) by mx.zohomail.com with SMTPS id 1750078996026694.4079795170629; Mon, 16 Jun 2025 06:03:16 -0700 (PDT) Received: from list by lists.xenproject.org with outflank-mailman.1017243.1394218 (Exim 4.92) (envelope-from ) id 1uR9UH-0007Z2-5V; Mon, 16 Jun 2025 13:02:57 +0000 Received: by outflank-mailman (output) from mailman id 1017243.1394218; Mon, 16 Jun 2025 13:02:57 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1uR9UH-0007Yv-2i; Mon, 16 Jun 2025 13:02:57 +0000 Received: by outflank-mailman (input) for mailman id 1017243; Mon, 16 Jun 2025 13:02:55 +0000 Received: from se1-gles-flk1-in.inumbo.com ([94.247.172.50] helo=se1-gles-flk1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1uR9UF-0007Yl-Cd for xen-devel@lists.xenproject.org; Mon, 16 Jun 2025 13:02:55 +0000 Received: from mail-wr1-x429.google.com (mail-wr1-x429.google.com [2a00:1450:4864:20::429]) by se1-gles-flk1.inumbo.com (Halon) with ESMTPS id 3628b433-4ab2-11f0-b894-0df219b8e170; Mon, 16 Jun 2025 15:02:52 +0200 (CEST) Received: by mail-wr1-x429.google.com with SMTP id ffacd0b85a97d-3a4ef2c2ef3so3731840f8f.2 for ; Mon, 16 Jun 2025 06:02:52 -0700 (PDT) Received: from [10.156.60.236] (ip-037-024-206-209.um08.pools.vodafone-ip.de. [37.24.206.209]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-74890006232sm6936111b3a.53.2025.06.16.06.02.42 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Mon, 16 Jun 2025 06:02:49 -0700 (PDT) X-Outflank-Mailman: Message body and most headers restored to incoming version X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: 3628b433-4ab2-11f0-b894-0df219b8e170 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.com; s=google; t=1750078972; x=1750683772; darn=lists.xenproject.org; h=content-transfer-encoding:in-reply-to:autocrypt:content-language :references:cc:to:from:subject:user-agent:mime-version:date :message-id:from:to:cc:subject:date:message-id:reply-to; bh=+UiMK0XqUdP3w4VQXOM4KTrdMDDbozq0m8l9m7I8RdI=; b=J4pCGXalIPg3SFmTWpsbw5oey+n6asJ0AHNapQQMS1lG3tgUfcDfWx1nbPWFeOZoUW sQgqIJqS8WGd0mEhwb03CG3k+V7pl1rSNl6RjcTLQxgIF5XFj3FM8xLU6sultJJWy2br qhg1pXEHbrkZrJBXpdZnL7+GmNESbPqtdM/hmffwgfTXmqBAW8wwA0uO6aRAtCPQQB1m /PyH7bIe24H3SEniL2X7oIP08r6yGZGIEH9JzJ2oDkvnTddqEOYfINcvIz12m0frcafP x9K2NjzAo5VBpDCJPFVZzd2uiQYivU6cZz0HCRsrCTYa/lxvWdH5nv9+1kqzjYsCFmUq PUiw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1750078972; x=1750683772; h=content-transfer-encoding:in-reply-to:autocrypt:content-language :references:cc:to:from:subject:user-agent:mime-version:date :message-id:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=+UiMK0XqUdP3w4VQXOM4KTrdMDDbozq0m8l9m7I8RdI=; b=AQ+epXnXZbkohB4XMJA1ypoeAToupyJe2asSFJIuMr0vvJI1WtV+c7UY0WaUzRef1S D9B09cdl2nGY6ZIWl6cXmHmbgwN+05Mp7LKax8M9Yq2rSbicWQslCGsa0Q3v2j2F44lx QlZJbS3qoMEZvQWE2CDGt6GlUClF0FY/ycgR6rkdvYRBqtQOs9d/arUUbV1P1VyEPciy isYix7xrV7L5rmB+iQSpLxqIxiCfqTLkGE8lnH26V1+yYTuKGhsHXM1RKnkbTIpn8KRM o8Kd4gmHBNauMEQb54ON/3iOw5fbbdFO9AXmpVy44XaBSuqbUlw7E4kQ5/7Tsqr7kVg/ tG9Q== X-Gm-Message-State: AOJu0YyCEqNOoEmVmWgB7pj8hlLTsb2WfyARsRXG+oZG3A1PszsZKvZL ATURFvseX7YTpjPgPVEPPKLaqWeSUAyYVX85EAxZMEnIr+KV4Na4THUR3fWAI5gkXGLdXhKLfan gaKs= X-Gm-Gg: ASbGnct7GURDMl2QrCb9NQJDM1A+awKma1qqx1SOxz3VQyrG5XrMhV+fs8KJOPKUzU3 U2M1KC0WAggyqvquQbTiLcd9RTL7pnLzqyUkrhy2kdlzQZTog6TrlWr9qxCJ8l9VR/1dNRnAlwt S4iqX1PEzahCjTn8XOyH3VlhpukHEMZ3GDdTN2uNoPwEOp7v7CHEpGcXHDwxjFmPyCviXDwXU0G ttOJjkVo3kZcVMJ8hcOvEeSaFYZ2psG+2jWKCiBwYtpDQjX3aziB8euXr7Ipz8seB4ra/rllVcN l8pmDAQWhlCtN7NThuJkWkHxAI8pI6PY/x6iqNyWsTkQeEh3jfHAjkB00DZFhySrb6msYJ9eWlf hfU9ElBXvgnm2WGoF1whqyDC6a/QlAt8iNUdd08+ZxubEm10= X-Google-Smtp-Source: AGHT+IFIbky8FQo0JyQ9zkDb7pEz7pPbS+l5iuXqXV5EiNEBpskkZOf9XQK9TMnLuBufJjZUcy9pGg== X-Received: by 2002:a5d:5f90:0:b0:3a4:e6e7:3acd with SMTP id ffacd0b85a97d-3a572371f31mr7318412f8f.18.1750078969953; Mon, 16 Jun 2025 06:02:49 -0700 (PDT) Message-ID: <0d2b44e3-bf6c-40ae-be4f-d0ad2845e925@suse.com> Date: Mon, 16 Jun 2025 15:02:39 +0200 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: [PATCH v6 7/7] mm: allow page scrubbing routine(s) to be arch controlled From: Jan Beulich To: "xen-devel@lists.xenproject.org" Cc: Andrew Cooper , Julien Grall , Stefano Stabellini , Anthony PERARD , Michal Orzel , =?UTF-8?Q?Roger_Pau_Monn=C3=A9?= , Bertrand Marquis , Volodymyr Babchuk , Oleksii Kurochko , Shawn Anastasio References: Content-Language: en-US Autocrypt: addr=jbeulich@suse.com; keydata= xsDiBFk3nEQRBADAEaSw6zC/EJkiwGPXbWtPxl2xCdSoeepS07jW8UgcHNurfHvUzogEq5xk hu507c3BarVjyWCJOylMNR98Yd8VqD9UfmX0Hb8/BrA+Hl6/DB/eqGptrf4BSRwcZQM32aZK 7Pj2XbGWIUrZrd70x1eAP9QE3P79Y2oLrsCgbZJfEwCgvz9JjGmQqQkRiTVzlZVCJYcyGGsD /0tbFCzD2h20ahe8rC1gbb3K3qk+LpBtvjBu1RY9drYk0NymiGbJWZgab6t1jM7sk2vuf0Py O9Hf9XBmK0uE9IgMaiCpc32XV9oASz6UJebwkX+zF2jG5I1BfnO9g7KlotcA/v5ClMjgo6Gl MDY4HxoSRu3i1cqqSDtVlt+AOVBJBACrZcnHAUSuCXBPy0jOlBhxPqRWv6ND4c9PH1xjQ3NP nxJuMBS8rnNg22uyfAgmBKNLpLgAGVRMZGaGoJObGf72s6TeIqKJo/LtggAS9qAUiuKVnygo 3wjfkS9A3DRO+SpU7JqWdsveeIQyeyEJ/8PTowmSQLakF+3fote9ybzd880fSmFuIEJldWxp Y2ggPGpiZXVsaWNoQHN1c2UuY29tPsJgBBMRAgAgBQJZN5xEAhsDBgsJCAcDAgQVAggDBBYC AwECHgECF4AACgkQoDSui/t3IH4J+wCfQ5jHdEjCRHj23O/5ttg9r9OIruwAn3103WUITZee e7Sbg12UgcQ5lv7SzsFNBFk3nEQQCACCuTjCjFOUdi5Nm244F+78kLghRcin/awv+IrTcIWF hUpSs1Y91iQQ7KItirz5uwCPlwejSJDQJLIS+QtJHaXDXeV6NI0Uef1hP20+y8qydDiVkv6l IreXjTb7DvksRgJNvCkWtYnlS3mYvQ9NzS9PhyALWbXnH6sIJd2O9lKS1Mrfq+y0IXCP10eS FFGg+Av3IQeFatkJAyju0PPthyTqxSI4lZYuJVPknzgaeuJv/2NccrPvmeDg6Coe7ZIeQ8Yj t0ARxu2xytAkkLCel1Lz1WLmwLstV30g80nkgZf/wr+/BXJW/oIvRlonUkxv+IbBM3dX2OV8 AmRv1ySWPTP7AAMFB/9PQK/VtlNUJvg8GXj9ootzrteGfVZVVT4XBJkfwBcpC/XcPzldjv+3 HYudvpdNK3lLujXeA5fLOH+Z/G9WBc5pFVSMocI71I8bT8lIAzreg0WvkWg5V2WZsUMlnDL9 mpwIGFhlbM3gfDMs7MPMu8YQRFVdUvtSpaAs8OFfGQ0ia3LGZcjA6Ik2+xcqscEJzNH+qh8V m5jjp28yZgaqTaRbg3M/+MTbMpicpZuqF4rnB0AQD12/3BNWDR6bmh+EkYSMcEIpQmBM51qM EKYTQGybRCjpnKHGOxG0rfFY1085mBDZCH5Kx0cl0HVJuQKC+dV2ZY5AqjcKwAxpE75MLFkr wkkEGBECAAkFAlk3nEQCGwwACgkQoDSui/t3IH7nnwCfcJWUDUFKdCsBH/E5d+0ZnMQi+G0A nAuWpQkjM1ASeQwSHEeAWPgskBQL In-Reply-To: Content-Transfer-Encoding: quoted-printable X-ZohoMail-DKIM: pass (identity @suse.com) X-ZM-MESSAGEID: 1750078997381124100 Content-Type: text/plain; charset="utf-8" Especially when dealing with large amounts of memory, memset() may not be very efficient; this can be bad enough that even for debug builds a custom function is warranted. We additionally want to distinguish "hot" and "cold" cases (with, as initial heuristic, "hot" being for any allocations a domain does for itself, assuming that in all other cases the page wouldn't be accessed [again] soon). The goal is for accesses of "cold" pages to not disturb caches (albeit finding a good balance between this and the higher latency looks to be difficult). Keep the default fallback to clear_page_*() in common code; this may want to be revisited down the road. Signed-off-by: Jan Beulich Acked-by: Julien Grall Reviewed-by: Oleksii Kurochko Reviewed-by: Jason Andryuk --- v6: Re-base. v4: Re-base. v3: Re-base. v2: New. --- The choice between hot and cold in scrub_one_page()'s callers is certainly up for discussion / improvement. --- a/xen/arch/arm/include/asm/page.h +++ b/xen/arch/arm/include/asm/page.h @@ -144,6 +144,12 @@ extern size_t dcache_line_bytes; =20 #define copy_page(dp, sp) memcpy(dp, sp, PAGE_SIZE) =20 +#define clear_page_hot clear_page +#define clear_page_cold clear_page + +#define scrub_page_hot(page) memset(page, SCRUB_BYTE_PATTERN, PAGE_SIZE) +#define scrub_page_cold scrub_page_hot + static inline size_t read_dcache_line_bytes(void) { register_t ctr; --- a/xen/arch/ppc/include/asm/page.h +++ b/xen/arch/ppc/include/asm/page.h @@ -188,6 +188,12 @@ static inline void invalidate_icache(voi #define clear_page(page) memset(page, 0, PAGE_SIZE) #define copy_page(dp, sp) memcpy(dp, sp, PAGE_SIZE) =20 +#define clear_page_hot clear_page +#define clear_page_cold clear_page + +#define scrub_page_hot(page) memset(page, SCRUB_BYTE_PATTERN, PAGE_SIZE) +#define scrub_page_cold scrub_page_hot + /* TODO: Flush the dcache for an entire page. */ static inline void flush_page_to_ram(unsigned long mfn, bool sync_icache) { --- a/xen/arch/riscv/include/asm/page.h +++ b/xen/arch/riscv/include/asm/page.h @@ -185,6 +185,12 @@ static inline void invalidate_icache(voi #define clear_page(page) memset((void *)(page), 0, PAGE_SIZE) #define copy_page(dp, sp) memcpy(dp, sp, PAGE_SIZE) =20 +#define clear_page_hot clear_page +#define clear_page_cold clear_page + +#define scrub_page_hot(page) memset(page, SCRUB_BYTE_PATTERN, PAGE_SIZE) +#define scrub_page_cold scrub_page_hot + void flush_page_to_ram(unsigned long mfn, bool sync_icache); =20 /* Write a pagetable entry. */ --- a/xen/arch/x86/Makefile +++ b/xen/arch/x86/Makefile @@ -58,6 +58,7 @@ obj-y +=3D pci.o obj-y +=3D physdev.o obj-$(CONFIG_COMPAT) +=3D x86_64/physdev.o obj-$(CONFIG_X86_PSR) +=3D psr.o +obj-bin-$(CONFIG_DEBUG) +=3D scrub_page.o obj-y +=3D setup.o obj-y +=3D shutdown.o obj-y +=3D smp.o --- a/xen/arch/x86/include/asm/page.h +++ b/xen/arch/x86/include/asm/page.h @@ -226,6 +226,11 @@ void copy_page_sse2(void *to, const void #define clear_page(_p) clear_page_cold(_p) #define copy_page(_t, _f) copy_page_sse2(_t, _f) =20 +#ifdef CONFIG_DEBUG +void scrub_page_hot(void *); +void scrub_page_cold(void *); +#endif + /* Convert between Xen-heap virtual addresses and machine addresses. */ #define __pa(x) (virt_to_maddr(x)) #define __va(x) (maddr_to_virt(x)) --- /dev/null +++ b/xen/arch/x86/scrub_page.S @@ -0,0 +1,39 @@ + .file __FILE__ + +#include +#include +#include + +FUNC(scrub_page_cold) + mov $PAGE_SIZE/32, %ecx + mov $SCRUB_PATTERN, %rax + +0: movnti %rax, (%rdi) + movnti %rax, 8(%rdi) + movnti %rax, 16(%rdi) + movnti %rax, 24(%rdi) + add $32, %rdi + sub $1, %ecx + jnz 0b + + sfence + ret +END(scrub_page_cold) + + .macro scrub_page_stosb + mov $PAGE_SIZE, %ecx + mov $SCRUB_BYTE_PATTERN, %eax + rep stosb + ret + .endm + + .macro scrub_page_stosq + mov $PAGE_SIZE/8, %ecx + mov $SCRUB_PATTERN, %rax + rep stosq + ret + .endm + +FUNC(scrub_page_hot) + ALTERNATIVE scrub_page_stosq, scrub_page_stosb, X86_FEATURE_ERMS +END(scrub_page_hot) --- a/xen/common/page_alloc.c +++ b/xen/common/page_alloc.c @@ -135,6 +135,7 @@ #include #include #include +#include #include #include #include @@ -800,27 +801,31 @@ static void page_list_add_scrub(struct p page_list_add(pg, &heap(node, zone, order)); } =20 -/* SCRUB_PATTERN needs to be a repeating series of bytes. */ -#ifndef NDEBUG -#define SCRUB_PATTERN 0xc2c2c2c2c2c2c2c2ULL -#else -#define SCRUB_PATTERN 0ULL +/* + * While in debug builds we want callers to avoid relying on allocations + * returning zeroed pages, for a production build, clear_page_*() is the + * fastest way to scrub. + */ +#ifndef CONFIG_DEBUG +# undef scrub_page_hot +# define scrub_page_hot clear_page_hot +# undef scrub_page_cold +# define scrub_page_cold clear_page_cold #endif -#define SCRUB_BYTE_PATTERN (SCRUB_PATTERN & 0xff) =20 -static void scrub_one_page(const struct page_info *pg) +static void scrub_one_page(const struct page_info *pg, bool cold) { + void *ptr; + if ( unlikely(pg->count_info & PGC_broken) ) return; =20 -#ifndef NDEBUG - /* Avoid callers relying on allocations returning zeroed pages. */ - unmap_domain_page(memset(__map_domain_page(pg), - SCRUB_BYTE_PATTERN, PAGE_SIZE)); -#else - /* For a production build, clear_page() is the fastest way to scrub. */ - clear_domain_page(_mfn(page_to_mfn(pg))); -#endif + ptr =3D __map_domain_page(pg); + if ( cold ) + scrub_page_cold(ptr); + else + scrub_page_hot(ptr); + unmap_domain_page(ptr); } =20 static void poison_one_page(struct page_info *pg) @@ -1100,12 +1105,14 @@ static struct page_info *alloc_heap_page if ( first_dirty !=3D INVALID_DIRTY_IDX || (scrub_debug && !(memflags & MEMF_no_scrub)) ) { + bool cold =3D d && d !=3D current->domain; + for ( i =3D 0; i < (1U << order); i++ ) { if ( test_and_clear_bit(_PGC_need_scrub, &pg[i].count_info) ) { if ( !(memflags & MEMF_no_scrub) ) - scrub_one_page(&pg[i]); + scrub_one_page(&pg[i], cold); =20 dirty_cnt++; } @@ -1370,7 +1377,7 @@ bool scrub_free_pages(void) { if ( test_bit(_PGC_need_scrub, &pg[i].count_info) ) { - scrub_one_page(&pg[i]); + scrub_one_page(&pg[i], true); /* * We can modify count_info without holding heap * lock since we effectively locked this buddy by @@ -2074,7 +2081,7 @@ static struct page_info *alloc_color_hea if ( !(memflags & MEMF_no_scrub) ) { if ( need_scrub ) - scrub_one_page(pg); + scrub_one_page(pg, d !=3D current->domain); else check_one_page(pg); } @@ -2225,7 +2232,7 @@ static void __init cf_check smp_scrub_he if ( !mfn_valid(_mfn(mfn)) || !page_state_is(pg, free) ) continue; =20 - scrub_one_page(pg); + scrub_one_page(pg, true); } } =20 @@ -2923,7 +2930,7 @@ void unprepare_staticmem_pages(struct pa if ( need_scrub ) { /* TODO: asynchronous scrubbing for pages of static memory. */ - scrub_one_page(pg); + scrub_one_page(pg, true); } =20 pg[i].count_info |=3D PGC_static; --- /dev/null +++ b/xen/include/xen/scrub.h @@ -0,0 +1,24 @@ +#ifndef __XEN_SCRUB_H__ +#define __XEN_SCRUB_H__ + +#include + +/* SCRUB_PATTERN needs to be a repeating series of bytes. */ +#ifdef CONFIG_DEBUG +# define SCRUB_PATTERN _AC(0xc2c2c2c2c2c2c2c2,ULL) +#else +# define SCRUB_PATTERN _AC(0,ULL) +#endif +#define SCRUB_BYTE_PATTERN (SCRUB_PATTERN & 0xff) + +#endif /* __XEN_SCRUB_H__ */ + +/* + * Local variables: + * mode: C + * c-file-style: "BSD" + * c-basic-offset: 4 + * tab-width: 4 + * indent-tabs-mode: nil + * End: + */