From nobody Fri Dec 12 19:30:14 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) client-ip=192.237.175.120; envelope-from=xen-devel-bounces@lists.xenproject.org; helo=lists.xenproject.org; Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=pass(p=none dis=none) header.from=gmail.com ARC-Seal: i=1; a=rsa-sha256; t=1765202153; cv=none; d=zohomail.com; s=zohoarc; b=kS0a0hlg4DMAX9dovRpDSJ5sxSas7b1Lk/h2QX/eUhAK5Z/Wlrw3+7dXqzgY1N3gKavZY8fDF9Rbkez4RXQNWxyISmuygJv5wuxEX39GSqu/eEXNEm28G2tbv9vm2e2NEhmKYXYxVSZgJ99lflgN2o0Vci7iLyoHfznm7en5ZFA= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1765202153; h=Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=uPY8I9p89Z49IT6xTWbbfssEr8lHB6MKrNil8DAb3PU=; b=jmT9LDIvreLG+xAmIun2cxGpuSxVzzUl8xWQ+mBa0n+Yx5UwXELrSh21qbl3Ch7P/X3M8WXaD8efRqEn61/sVPn05qDpItLppyvVPPjHBVhr9cE4utS+5z4MrLcF6n6QA0Q9NOz6PlZo0uD4PW8giV6Gsc0GhGpB3oDxsBSugdg= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) by mx.zohomail.com with SMTPS id 1765202153989828.2309303400162; Mon, 8 Dec 2025 05:55:53 -0800 (PST) Received: from list by lists.xenproject.org with outflank-mailman.1180637.1503793 (Exim 4.92) (envelope-from ) id 1vSbiC-0005Lv-LQ; Mon, 08 Dec 2025 13:55:36 +0000 Received: by outflank-mailman (output) from mailman id 1180637.1503793; Mon, 08 Dec 2025 13:55:36 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1vSbiC-0005Lm-HP; Mon, 08 Dec 2025 13:55:36 +0000 Received: by outflank-mailman (input) for mailman id 1180637; Mon, 08 Dec 2025 13:55:35 +0000 Received: from se1-gles-flk1-in.inumbo.com ([94.247.172.50] helo=se1-gles-flk1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1vSbiB-0005KM-Ni for xen-devel@lists.xenproject.org; Mon, 08 Dec 2025 13:55:35 +0000 Received: from mail-wm1-x32b.google.com (mail-wm1-x32b.google.com [2a00:1450:4864:20::32b]) by se1-gles-flk1.inumbo.com (Halon) with ESMTPS id 90c39b8c-d43d-11f0-980a-7dc792cee155; Mon, 08 Dec 2025 14:55:33 +0100 (CET) Received: by mail-wm1-x32b.google.com with SMTP id 5b1f17b1804b1-4779cc419b2so51066095e9.3 for ; Mon, 08 Dec 2025 05:55:33 -0800 (PST) Received: from PKL-HASEEBA-LT.. ([39.36.89.17]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-4792afe32cbsm144746585e9.0.2025.12.08.05.55.27 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 08 Dec 2025 05:55:32 -0800 (PST) X-Outflank-Mailman: Message body and most headers restored to incoming version X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: 90c39b8c-d43d-11f0-980a-7dc792cee155 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1765202133; x=1765806933; darn=lists.xenproject.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=uPY8I9p89Z49IT6xTWbbfssEr8lHB6MKrNil8DAb3PU=; b=B+6daUTjitQs6mQRROeawnJnY1RYRuoUWmakmIj/zPd7TD0zGHoBWwvKqwcJdM3GKq eNu9JegtYlP9BeHXMuKp1kd+0NM1c7huvh9/aGj3cs+fxOXtrq4sDvnS/TwpO+43MF6V 6P14krpEu6iGJhE2gtn/4gdZOF25wmdK847voLKx+yF1xOtwv96lk/+i152tB9kr5uGZ RkoUwm8132DcsA+45BXAJBEwGXFgkUw2DfRVu0VcbFClELIYG2tNIuX/dYX9CQHTD28H 91v6+Ku5N1OKE3TgLxauxDzqBvycKIE9L0MTvGHCFonVA4U3J4rho1nwtq7A490yWJwS svDg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1765202133; x=1765806933; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=uPY8I9p89Z49IT6xTWbbfssEr8lHB6MKrNil8DAb3PU=; b=Dnp3a0thcmnT6v0ykTj56QfozdxVJC9H31tKTe7zOJsbzBmsvvtmtX2XHN8Ztl3ijQ Ekc+qPPwHdwUMDpu7YCpk9wRJXlMwWEF+kLEhhAA0Hf5DfMyjgDJj4+MunGowz0Die2q J49w6PGLhtK0BwjqEOEQKIN2i2cGQrfO8SBXEBZVpW5CwDVeOcJ5sLuyEIpiJ8/aMu3K 40xHFadJ23ZaPzWgM8En0qtIIw9si47sA+KNvsoottHygdLHiBvSoAB3QLLzr5upwes4 iO+vfZSpqQnVuhskaJ2KZHlZMgckDjBJiHPZWYk8ekfAPU75sM96sPUKeJlqCKF+tR4B v7CQ== X-Gm-Message-State: AOJu0YwGmzjwudb9FfBK77ob2undNoCCeCHKFINJqhQt4DbMNxKf0ZIK v3dI9AWR9x4+CeTSrGKIvw7jNxYNdF6IJJLWcli7IATKJtPEgdwQ+UNZmKIaQ9bb X-Gm-Gg: ASbGnctUcsn8z0LqUoUg5v32v+P6VTDsEQLkxX3E9YfjZ8TEDpwqvXTAvEFzasWA8b8 Q8oAzlkq23X4i2+vKw9w3epTq/DRrxXi4IbChTButN6Keh0S0SNd7WP4CKJmLd7aI6gWHg+FsBL FDou9VJWvWDvLdktl+TFXSuBwSJFa2nfEd+reCmHKAMKIaWaUxfA6N9uzj9Rb4pmqGgLLw45nBQ u6vX2zWGy7zz7Ah5Ttegigbfte0JzpzX0zcBGWOXm6DuOEC50GYJdeNDoYOJtFBwPsVDR/w64Sh zCypyWwV7PnttAlEGvksst5/E2pyI9QPeuHfv0o0wMC4RdW8wAFmZxsSQlX4LjHuddQAM73+myM mRh6JE4MP066BW3d7QfbC6BgP+VrwwnmhFxhTHy0/MhEOTTYXF1gaKWUxUNTZQUSjaOumL6yYHI rOQj1wJIQDpLZLQnLRugwJQ+TDeMet1CAieXvt617yjf2FIWB1 X-Google-Smtp-Source: AGHT+IFROT6R+E7EsRYwySNUKn1MFwIMxmn+zDuNcR6BsCHwzxL82BBKdgXgO2S5B43m42Q+sdWJqA== X-Received: by 2002:a05:600c:1ca4:b0:477:a978:3a7b with SMTP id 5b1f17b1804b1-47939e27832mr96055955e9.22.1765202132629; Mon, 08 Dec 2025 05:55:32 -0800 (PST) From: Haseeb Ashraf To: xen-devel@lists.xenproject.org Cc: Haseeb Ashraf , Stefano Stabellini , Julien Grall , Bertrand Marquis , Michal Orzel , Volodymyr Babchuk , Dan Driscoll , Noor Ahsan Khawaja , Fahad Arslan , Andrew Bachtel Subject: [PATCH 1/3] xen/arm/p2m: perform IPA-based TLBI when IPA is known Date: Mon, 8 Dec 2025 18:55:14 +0500 Message-ID: X-Mailer: git-send-email 2.43.0 In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ZohoMail-DKIM: pass (identity @gmail.com) X-ZM-MESSAGEID: 1765202155115158500 Content-Type: text/plain; charset="utf-8" From: Haseeb Ashraf This commit addresses a major issue for running Xen on KVM i.e. costly emulation of VMALLS12E1IS which becomes worse when this TLBI is invoked too many times. There are mainly two places where this is problematic: (a) When vCPUs switch on a pCPU or pCPUs (b) When domu mapped pages onto dom0, are to be unmapped, then each page being removed by XENMEM_remove_from_physmap has its TLBs invalidated by VMALLS12E1IS. The first one is addressed by relaxing VMALLS12E1 -> VMALLE1 as the stage-2 is common between all the vCPUs of a VM. Since each CPU has its own private TLBs, so flush between vCPU of the same domains is still required to avoid translations from vCPUx to "leak" to the vCPUy which can be achieved by using VMALLE1. The second one is addressed by using IPA-based TLBI (IPAS2E1) in combination with VMALLE1 whenever the IPA range is known instead of using VMALLS12E1. There is an upper cap placed on number of IPA-based TLBI. This factor for execution time of VMALLS12E1 vs IPAS2E1 is found to be 70K on Graviton4 in Xen on KVM virtualization. So, 64K * 4KB =3D 256MB is set as the threshold. For arm32, TLBIALL instruction can invalidate both stage-1 and stage-2 entries, so using IPA-based TLBI would be redundant as TLBIALL is required in any case to invalidate corresponding cached entries from stage-1. Suggested-by: Julien Grall Signed-off-by: Haseeb Ashraf Changes in v3: - Updated IPA-based TLBI sequence to apply ARM64 repeat TLBI workaround to only final TLBI and DSB of the sequence. - Removed TLB_HELPER_IPA and instead directly used the TLBI instruction where needed as that was the only instance where it is being used. - Removed flush_guest_tlb_range_ipa_local() as it was not being used. - Updated comments as per feedback in v2 about holding lock before p2m_load_vttbr. - Updated references of ARM ARM to use newer version DDI 0487L.b instead of older version DDI 0487A.e. Changes in v2: - This commit implements the basline implementation to address the problem at hand. Removed the FEAT_nTLBPA implementation from this commit which will be implemented in following commit using CPU capability. - Moved ARM32 and ARM64 specific implementations of TLBIs to architecture specific flushtlb.h. - Added references of ARM ARM in code comments. - Evaluated and added a threshold to select between IPA-based TLB invalidation vs fallback to full stage TLB invalidation above the threshold. --- xen/arch/arm/include/asm/arm32/flushtlb.h | 53 +++++++++++++ xen/arch/arm/include/asm/arm64/flushtlb.h | 46 ++++++++++++ xen/arch/arm/include/asm/mmu/p2m.h | 2 + xen/arch/arm/mmu/p2m.c | 92 +++++++++++++++++------ 4 files changed, 168 insertions(+), 25 deletions(-) diff --git a/xen/arch/arm/include/asm/arm32/flushtlb.h b/xen/arch/arm/inclu= de/asm/arm32/flushtlb.h index 61c25a3189..3c0c2123d4 100644 --- a/xen/arch/arm/include/asm/arm32/flushtlb.h +++ b/xen/arch/arm/include/asm/arm32/flushtlb.h @@ -45,6 +45,43 @@ TLB_HELPER(flush_xen_tlb_local, TLBIALLH, nsh) =20 #undef TLB_HELPER =20 +/* + * Flush TLB of local processor. Use when flush for only stage-1 is intend= ed. + * + * The following function should be used where intention is to clear only + * stage-1 TLBs. This would be helpful in future in identifying which stag= e-1 + * TLB flushes can be skipped such as in present of FEAT_nTLBPA. + */ +static inline void flush_guest_tlb_s1_local(void) +{ + /* + * Same instruction can invalidate both stage-1 and stage-2 TLBs depen= ding + * upon the execution context. + * + * See ARMv8 (DDI 0487L.b): G5-11698 Table G5-23. + */ + return flush_guest_tlb_local(); +} + +/* + * Flush TLB of inner-shareable processor domain. Use when flush for only + * stage-1 is intended. + * + * The following function should be used where intention is to clear only + * stage-1 TLBs. This would be helpful in future in identifying which stag= e-1 + * TLB flushes can be skipped such as in present of FEAT_nTLBPA. + */ +static inline void flush_guest_tlb_s1(void) +{ + /* + * Same instruction can invalidate both stage-1 and stage-2 TLBs depen= ding + * upon the execution context. + * + * See ARMv8 (DDI 0487L.b): G5-11698 Table G5-23. + */ + return flush_guest_tlb(); +} + /* Flush TLB of local processor for address va. */ static inline void __flush_xen_tlb_one_local(vaddr_t va) { @@ -57,6 +94,22 @@ static inline void __flush_xen_tlb_one(vaddr_t va) asm volatile(STORE_CP32(0, TLBIMVAHIS) : : "r" (va) : "memory"); } =20 +/* + * Flush a range of IPA's mappings from the TLB of all processors in the + * inner-shareable domain. + */ +static inline void flush_guest_tlb_range_ipa(paddr_t ipa, + unsigned long size) +{ + /* + * Following can invalidate both stage-1 and stage-2 TLBs depending up= on + * the execution mode. + * + * See ARMv8 (DDI 0487L.b): G5-11698 Table G5-23. + */ + flush_guest_tlb(); +} + #endif /* __ASM_ARM_ARM32_FLUSHTLB_H__ */ /* * Local variables: diff --git a/xen/arch/arm/include/asm/arm64/flushtlb.h b/xen/arch/arm/inclu= de/asm/arm64/flushtlb.h index 3b99c11b50..67ae616993 100644 --- a/xen/arch/arm/include/asm/arm64/flushtlb.h +++ b/xen/arch/arm/include/asm/arm64/flushtlb.h @@ -1,6 +1,8 @@ #ifndef __ASM_ARM_ARM64_FLUSHTLB_H__ #define __ASM_ARM_ARM64_FLUSHTLB_H__ =20 +#include /* For SZ_* macros. */ + /* * Every invalidation operation use the following patterns: * @@ -72,6 +74,12 @@ TLB_HELPER(flush_guest_tlb_local, vmalls12e1, nsh) /* Flush innershareable TLBs, current VMID only */ TLB_HELPER(flush_guest_tlb, vmalls12e1is, ish) =20 +/* Flush local TLBs, current VMID, stage-1 only */ +TLB_HELPER(flush_guest_tlb_s1_local, vmalle1, nsh) + +/* Flush innershareable TLBs, current VMID, stage-1 only */ +TLB_HELPER(flush_guest_tlb_s1, vmalle1is, ish) + /* Flush local TLBs, all VMIDs, non-hypervisor mode */ TLB_HELPER(flush_all_guests_tlb_local, alle1, nsh) =20 @@ -90,6 +98,44 @@ TLB_HELPER_VA(__flush_xen_tlb_one, vae2is) #undef TLB_HELPER #undef TLB_HELPER_VA =20 +/* + * Flush a range of IPA's mappings from the TLB of all processors in the + * inner-shareable domain. + */ +static inline void flush_guest_tlb_range_ipa(paddr_t ipa, unsigned long si= ze) +{ + paddr_t end; + + /* + * If IPA range is too big (empirically found to be 256M), then fallba= ck to + * full TLB flush. + */ + if ( size > SZ_256M ) + return flush_guest_tlb(); + + end =3D ipa + size; + + /* + * See ARM ARM DDI 0487L.b D8.17.6.1 (Invalidating TLB entries from st= age 2 + * translations) for details of TLBI sequence. + */ + dsb(ishst); /* Ensure prior page-tables updates have completed */ + while ( ipa < end ) + { + /* Flush stage-2 TLBs for ipa address */ + asm_inline volatile ( + "tlbi ipas2e1is, %0;" : : "r" (ipa >> PAGE_SHIFT) : "memory" ); + ipa +=3D PAGE_SIZE; + } + /* + * As ARM64_WORKAROUND_REPEAT_TLBI is required to be applied to last T= LBI + * of the sequence, it is only needed to be handled in the following + * invocation. Final dsb() and isb() are also applied in the following + * invocation. + */ + flush_guest_tlb_s1(); +} + #endif /* __ASM_ARM_ARM64_FLUSHTLB_H__ */ /* * Local variables: diff --git a/xen/arch/arm/include/asm/mmu/p2m.h b/xen/arch/arm/include/asm/= mmu/p2m.h index 58496c0b09..8a16722b82 100644 --- a/xen/arch/arm/include/asm/mmu/p2m.h +++ b/xen/arch/arm/include/asm/mmu/p2m.h @@ -10,6 +10,8 @@ extern unsigned int p2m_root_level; =20 struct p2m_domain; void p2m_force_tlb_flush_sync(struct p2m_domain *p2m); +void p2m_force_tlb_flush_range_sync(struct p2m_domain *p2m, uint64_t start= _ipa, + uint64_t page_count); void p2m_tlb_flush_sync(struct p2m_domain *p2m); =20 void p2m_clear_root_pages(struct p2m_domain *p2m); diff --git a/xen/arch/arm/mmu/p2m.c b/xen/arch/arm/mmu/p2m.c index 51abf3504f..eec59056fa 100644 --- a/xen/arch/arm/mmu/p2m.c +++ b/xen/arch/arm/mmu/p2m.c @@ -235,33 +235,28 @@ void p2m_restore_state(struct vcpu *n) * when running multiple vCPU of the same domain on a single pCPU. */ if ( *last_vcpu_ran !=3D INVALID_VCPU_ID && *last_vcpu_ran !=3D n->vcp= u_id ) - flush_guest_tlb_local(); + flush_guest_tlb_s1_local(); =20 *last_vcpu_ran =3D n->vcpu_id; } =20 /* - * Force a synchronous P2M TLB flush. + * Loads VTTBR from given P2M. * * Must be called with the p2m lock held. + * + * This returns switched out VTTBR. */ -void p2m_force_tlb_flush_sync(struct p2m_domain *p2m) +static uint64_t p2m_load_vttbr(struct p2m_domain *p2m, unsigned long *flag= s) { - unsigned long flags =3D 0; uint64_t ovttbr; =20 - ASSERT(p2m_is_write_locked(p2m)); - - /* - * ARM only provides an instruction to flush TLBs for the current - * VMID. So switch to the VTTBR of a given P2M if different. - */ ovttbr =3D READ_SYSREG64(VTTBR_EL2); if ( ovttbr !=3D p2m->vttbr ) { uint64_t vttbr; =20 - local_irq_save(flags); + local_irq_save(*flags); =20 /* * ARM64_WORKAROUND_AT_SPECULATE: We need to stop AT to allocate @@ -280,8 +275,14 @@ void p2m_force_tlb_flush_sync(struct p2m_domain *p2m) isb(); } =20 - flush_guest_tlb(); + return ovttbr; +} =20 +/* + * Restores VTTBR which was switched out as a result of p2m_load_vttbr(). + */ +static void p2m_restore_vttbr(uint64_t ovttbr, unsigned long flags) +{ if ( ovttbr !=3D READ_SYSREG64(VTTBR_EL2) ) { WRITE_SYSREG64(ovttbr, VTTBR_EL2); @@ -289,10 +290,58 @@ void p2m_force_tlb_flush_sync(struct p2m_domain *p2m) isb(); local_irq_restore(flags); } +} + +/* + * Force a synchronous P2M TLB flush. + * + * Must be called with the p2m lock held. + */ +void p2m_force_tlb_flush_sync(struct p2m_domain *p2m) +{ + unsigned long flags =3D 0; + uint64_t ovttbr; + + ASSERT(p2m_is_write_locked(p2m)); + + /* + * ARM only provides an instruction to flush TLBs for the current + * VMID. So switch to the VTTBR of a given P2M if different. + */ + ovttbr =3D p2m_load_vttbr(p2m, &flags); + + flush_guest_tlb(); + + p2m_restore_vttbr(ovttbr, flags); =20 p2m->need_flush =3D false; } =20 +/* + * Force a synchronous P2M TLB flush on a range of addresses. + * + * Must be called with the p2m lock held. + */ +void p2m_force_tlb_flush_range_sync(struct p2m_domain *p2m, uint64_t start= _ipa, + uint64_t page_count) +{ + unsigned long flags =3D 0; + uint64_t ovttbr; + + ASSERT(p2m_is_write_locked(p2m)); + + /* + * ARM only provides an instruction to flush TLBs for the current + * VMID. So switch to the VTTBR of a given P2M if different. + */ + ovttbr =3D p2m_load_vttbr(p2m, &flags); + + /* Invalidate TLB entries by IPA range */ + flush_guest_tlb_range_ipa(start_ipa, PAGE_SIZE * page_count); + + p2m_restore_vttbr(ovttbr, flags); +} + void p2m_tlb_flush_sync(struct p2m_domain *p2m) { if ( p2m->need_flush ) @@ -1034,7 +1083,8 @@ static int __p2m_set_entry(struct p2m_domain *p2m, * For more details see (D4.7.1 in ARM DDI 0487A.j). */ p2m_remove_pte(entry, p2m->clean_pte); - p2m_force_tlb_flush_sync(p2m); + p2m_force_tlb_flush_range_sync(p2m, gfn_x(sgfn) << PAGE_SHIFT, + 1UL << page_order); =20 p2m_write_pte(entry, split_pte, p2m->clean_pte); =20 @@ -1090,8 +1140,8 @@ static int __p2m_set_entry(struct p2m_domain *p2m, p2m_remove_pte(entry, p2m->clean_pte); =20 if ( removing_mapping ) - /* Flush can be deferred if the entry is removed */ - p2m->need_flush |=3D !!lpae_is_valid(orig_pte); + p2m_force_tlb_flush_range_sync(p2m, gfn_x(sgfn) << PAGE_SHIFT, + 1UL << page_order); else { lpae_t pte =3D mfn_to_p2m_entry(smfn, t, a); @@ -1102,18 +1152,10 @@ static int __p2m_set_entry(struct p2m_domain *p2m, /* * It is necessary to flush the TLB before writing the new entry * to keep coherency when the previous entry was valid. - * - * Although, it could be defered when only the permissions are - * changed (e.g in case of memaccess). */ if ( lpae_is_valid(orig_pte) ) - { - if ( likely(!p2m->mem_access_enabled) || - P2M_CLEAR_PERM(pte) !=3D P2M_CLEAR_PERM(orig_pte) ) - p2m_force_tlb_flush_sync(p2m); - else - p2m->need_flush =3D true; - } + p2m_force_tlb_flush_range_sync(p2m, gfn_x(sgfn) << PAGE_SHIFT, + 1UL << page_order); else if ( !p2m_is_valid(orig_pte) ) /* new mapping */ p2m->stats.mappings[level]++; =20 --=20 2.43.0 From nobody Fri Dec 12 19:30:14 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) client-ip=192.237.175.120; envelope-from=xen-devel-bounces@lists.xenproject.org; helo=lists.xenproject.org; Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=pass(p=none dis=none) header.from=gmail.com ARC-Seal: i=1; a=rsa-sha256; t=1765202155; cv=none; d=zohomail.com; s=zohoarc; b=VRyzIq+3gCw121ainP7laXNL5SMqNKi0WYIzxRH69sdxg9/WeXaKQaMgf+M6J3S5TaD1bjZ5tLGJcH20jZpgSbnDjYJ5Er/oVJvg/NNP8gWUuUxUADE9zH+4fTijEfwDlUK+vFcsHoRDMo2+Ml31QtZkxOfKxE3Km/2ln3dbZLA= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1765202155; h=Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=89jrADcfTNLToqirJ07l6LVFLAbwHAWxMw55ZpCsj8k=; b=mCP9/dQhprTVmJE/o0WF7UJUfzMWo0eM1aTRGMTcQC/BfrCJ0vMxKCVCyESK3kDRua7iwWwdk8NqQah0/zFdzRR/MJQ0F/lKsJwBz8/O6twr8PEqZbIKHSk8B19wECRqaCX2TRkVWGpI0O35OQEnKD2F3nJW76sCwSbJat5gaqU= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) by mx.zohomail.com with SMTPS id 1765202155140223.54548858933651; Mon, 8 Dec 2025 05:55:55 -0800 (PST) Received: from list by lists.xenproject.org with outflank-mailman.1180638.1503801 (Exim 4.92) (envelope-from ) id 1vSbiJ-0005eF-S1; Mon, 08 Dec 2025 13:55:43 +0000 Received: by outflank-mailman (output) from mailman id 1180638.1503801; Mon, 08 Dec 2025 13:55:43 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1vSbiJ-0005e6-Om; Mon, 08 Dec 2025 13:55:43 +0000 Received: by outflank-mailman (input) for mailman id 1180638; Mon, 08 Dec 2025 13:55:43 +0000 Received: from se1-gles-sth1-in.inumbo.com ([159.253.27.254] helo=se1-gles-sth1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1vSbiI-00055Z-Sj for xen-devel@lists.xenproject.org; Mon, 08 Dec 2025 13:55:43 +0000 Received: from mail-wm1-x335.google.com (mail-wm1-x335.google.com [2a00:1450:4864:20::335]) by se1-gles-sth1.inumbo.com (Halon) with ESMTPS id 95264e19-d43d-11f0-9d1b-b5c5bf9af7f9; Mon, 08 Dec 2025 14:55:41 +0100 (CET) Received: by mail-wm1-x335.google.com with SMTP id 5b1f17b1804b1-47774d3536dso42487505e9.0 for ; Mon, 08 Dec 2025 05:55:41 -0800 (PST) Received: from PKL-HASEEBA-LT.. ([39.36.89.17]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-4792afe32cbsm144746585e9.0.2025.12.08.05.55.34 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 08 Dec 2025 05:55:38 -0800 (PST) X-Outflank-Mailman: Message body and most headers restored to incoming version X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: 95264e19-d43d-11f0-9d1b-b5c5bf9af7f9 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1765202140; x=1765806940; darn=lists.xenproject.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=89jrADcfTNLToqirJ07l6LVFLAbwHAWxMw55ZpCsj8k=; b=R7kOkrVKTZWHzlEqEkbplbZIhCsrLOv68BdOznIcfKCdt9TRx/5bRu4dYzwWINSQ9i Ukdonohwb8edMkgezbTC2RUrhlpDDTgsDE33y0C0yALQWTw8fAS9mYg2VAG81FTf084A DW52rixlqDid93HYVonlHknB8BbKldaTcAbo4S/YT16uYAsgK/phqKown7HKzxMOcQJV pPFNKLbjpLjYSDdxJnOgSwtNtrXMh0GNzb9SQarCZnA7/DrLAd8NOFzQD16tmF47Zj6X nLe4lAwaXBOmenbpBWU+cjD4hpYdQntEpWuMy1AhwaQRuhxVBKVDtxj1HybuHd6N0y7C 8a0Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1765202140; x=1765806940; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=89jrADcfTNLToqirJ07l6LVFLAbwHAWxMw55ZpCsj8k=; b=jgX/oMZyYVE1gJZyj6U9HNq29vIWy9e4veAp0V+X7KVcgQMh5oHV7FCtuWJ/AhrIFy hpwdcYYHh2r1Ljc97vphEvkf9b0IxudR3yEeGKQ9oZ3lnYUytQRdpioEYT6MYEzK5pc0 AHJbdauLUaSgzWWw1evoiaer3cQEwdgnOfmYFt+CdVpCraSlm7b8WzWjE7V2YCAeXk6l yWrNvcyqpjl08VN2TwOhaq6ULTmA0s0tnZo4VUX15+bE1sk8OvbzmY8aLtlt1UZG9JKe hgT52nrs9xixsOaiChOVRksIlSxHygBr0ufcD42/0sCdaAGRm+z8sOrPGrxw9k8EnTIh c2mg== X-Gm-Message-State: AOJu0YzSKyjmsWky77lJti6UFgMi3vS+H3RwHy4/bBBPei616Qi5RWnE IyAaIDoFt2+t3Ogi2vm9wSjIUmr1bBLnqJxEH+TVIS9Z1QNcoWGr+iw6Qka5/E7i X-Gm-Gg: ASbGnct4hRzhgim/epibP3p7OQ+/mrP6xV6HBVQlEyFJR1TC7kw8aUbdjNbLTFWsB9L eIMnU1ZrhlESPE/GW9Cx15l703/XbH3BCbiCZkWHyf3wWuGqxIIhyB2UkTPZUFolehvfNyzCWsU WZwHHxu+6CfvgvRUBVA6mROh6UmeYlNK4rQ7szfLcV3NcKLHojUQaXyAwA5NU4EpG45b/DLT3OH qZGB5nibi5J5WG55MOqrOtqW5I7ZcTMPlAA0yF97kdZI5qBmzmi6V+pIEbtuKnG4FcjaCpQ0Jw0 6ZXpU5VcfQyjACTbiAWd/oSHgzgb2WN+tQffEvcDnqtWYYx04KpdQzc8+irx088ToF0QzyB6RDU 094OZHfCOWhemnTczmxDXq1mBkTQqEOoPIVyDlpK6Muyg2A5dtgs5eY5uxiNTALwnrU8ZpdiYqd Bk1KA4zqZkADV/Ocg8X2DsgLxTCj+7tQb5EQt0arfc+uNFg9nm X-Google-Smtp-Source: AGHT+IGpJ9lI7qlZKtyZlSbU9CKaEyhK+/QVNIKFw96TPH9dDQojcnNuoPmt6VzvmTXcTyZ7xin53w== X-Received: by 2002:a05:600c:3513:b0:475:d9de:952e with SMTP id 5b1f17b1804b1-47939deff25mr89472025e9.1.1765202139699; Mon, 08 Dec 2025 05:55:39 -0800 (PST) From: Haseeb Ashraf To: xen-devel@lists.xenproject.org Cc: Haseeb Ashraf , Stefano Stabellini , Julien Grall , Bertrand Marquis , Michal Orzel , Volodymyr Babchuk , Dan Driscoll , Noor Ahsan Khawaja , Fahad Arslan , Andrew Bachtel , Mohamed Mediouni Subject: [PATCH 2/3] xen/arm: optimize stage-1,2 combined TLBI in presence of FEAT_nTLBPA Date: Mon, 8 Dec 2025 18:55:15 +0500 Message-ID: <025aae317ebfbb234554be7621fd38fcad08a0c7.1765197209.git.haseeb.ashraf@siemens.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ZohoMail-DKIM: pass (identity @gmail.com) X-ZM-MESSAGEID: 1765202156961158500 Content-Type: text/plain; charset="utf-8" From: Haseeb Ashraf FEAT_nTLBPA (quoting definition) introduces a mechanism to identify if the intermediate caching of translation table walks does not include non-coherent caches of previous valid translation table entries since the last completed TLBI applicable to the PE. As there won't be any non-coherent caches since the last completed TLBI, stage-1 TLBI won't be required while performing stage-2 TLBI. This feature is optionally available in both arm32 and arm64. Suggested-by: Mohamed Mediouni Signed-off-by: Haseeb Ashraf Changes in v3: - This commit has no functional change in v3, only rebasing changes due to updates in commit-1. Changes in v2: - This commit is implemented in v2 and is splitted from commit-1 in v1. This is implemented by using CPU capability. --- xen/arch/arm/cpufeature.c | 19 ++++++ xen/arch/arm/include/asm/arm32/flushtlb.h | 14 +++-- xen/arch/arm/include/asm/arm64/flushtlb.h | 77 ++++++++++++++++------- xen/arch/arm/include/asm/cpufeature.h | 24 ++++++- xen/arch/arm/include/asm/processor.h | 7 +++ 5 files changed, 109 insertions(+), 32 deletions(-) diff --git a/xen/arch/arm/cpufeature.c b/xen/arch/arm/cpufeature.c index 1a80738571..9fa1c45869 100644 --- a/xen/arch/arm/cpufeature.c +++ b/xen/arch/arm/cpufeature.c @@ -17,7 +17,19 @@ DECLARE_BITMAP(cpu_hwcaps, ARM_NCAPS); =20 struct cpuinfo_arm __read_mostly domain_cpuinfo; =20 +#ifdef CONFIG_ARM_32 +static bool has_ntlbpa(const struct arm_cpu_capabilities *entry) +{ + return system_cpuinfo.mm32.ntlbpa =3D=3D MM32_NTLBPA_SUPPORT_IMP; +} +#endif + #ifdef CONFIG_ARM_64 +static bool has_ntlbpa(const struct arm_cpu_capabilities *entry) +{ + return system_cpuinfo.mm64.ntlbpa =3D=3D MM64_NTLBPA_SUPPORT_IMP; +} + static bool has_sb_instruction(const struct arm_cpu_capabilities *entry) { return system_cpuinfo.isa64.sb; @@ -25,6 +37,13 @@ static bool has_sb_instruction(const struct arm_cpu_capa= bilities *entry) #endif =20 static const struct arm_cpu_capabilities arm_features[] =3D { +#if defined(CONFIG_ARM_32) || defined(CONFIG_ARM_64) + { + .desc =3D "Intermediate caching of translation table walks (nTLBPA= )", + .capability =3D ARM_HAS_NTLBPA, + .matches =3D has_ntlbpa, + }, +#endif #ifdef CONFIG_ARM_64 { .desc =3D "Speculation barrier instruction (SB)", diff --git a/xen/arch/arm/include/asm/arm32/flushtlb.h b/xen/arch/arm/inclu= de/asm/arm32/flushtlb.h index 3c0c2123d4..7cff042508 100644 --- a/xen/arch/arm/include/asm/arm32/flushtlb.h +++ b/xen/arch/arm/include/asm/arm32/flushtlb.h @@ -49,8 +49,8 @@ TLB_HELPER(flush_xen_tlb_local, TLBIALLH, nsh) * Flush TLB of local processor. Use when flush for only stage-1 is intend= ed. * * The following function should be used where intention is to clear only - * stage-1 TLBs. This would be helpful in future in identifying which stag= e-1 - * TLB flushes can be skipped such as in present of FEAT_nTLBPA. + * stage-1 TLBs. This would be helpful in identifying which stage-1 TLB fl= ushes + * can be skipped such as in present of FEAT_nTLBPA. */ static inline void flush_guest_tlb_s1_local(void) { @@ -60,7 +60,8 @@ static inline void flush_guest_tlb_s1_local(void) * * See ARMv8 (DDI 0487L.b): G5-11698 Table G5-23. */ - return flush_guest_tlb_local(); + if ( !cpus_have_const_cap(ARM_HAS_NTLBPA) ) + flush_guest_tlb_local(); } =20 /* @@ -68,8 +69,8 @@ static inline void flush_guest_tlb_s1_local(void) * stage-1 is intended. * * The following function should be used where intention is to clear only - * stage-1 TLBs. This would be helpful in future in identifying which stag= e-1 - * TLB flushes can be skipped such as in present of FEAT_nTLBPA. + * stage-1 TLBs. This would be helpful in identifying which stage-1 TLB fl= ushes + * can be skipped such as in present of FEAT_nTLBPA. */ static inline void flush_guest_tlb_s1(void) { @@ -79,7 +80,8 @@ static inline void flush_guest_tlb_s1(void) * * See ARMv8 (DDI 0487L.b): G5-11698 Table G5-23. */ - return flush_guest_tlb(); + if ( !cpus_have_const_cap(ARM_HAS_NTLBPA) ) + flush_guest_tlb(); } =20 /* Flush TLB of local processor for address va. */ diff --git a/xen/arch/arm/include/asm/arm64/flushtlb.h b/xen/arch/arm/inclu= de/asm/arm64/flushtlb.h index 67ae616993..0f0d5050e5 100644 --- a/xen/arch/arm/include/asm/arm64/flushtlb.h +++ b/xen/arch/arm/include/asm/arm64/flushtlb.h @@ -47,6 +47,24 @@ static inline void name(void) \ : : : "memory"); \ } =20 +#define TLB_HELPER_NTLBPA(name, tlbop, sh) \ +static inline void name(void) \ +{ \ + if ( !cpus_have_const_cap(ARM_HAS_NTLBPA) ) \ + asm_inline volatile ( \ + "dsb " # sh "st;" \ + "tlbi " # tlbop ";" \ + ALTERNATIVE( \ + "nop; nop;", \ + "dsb ish;" \ + "tlbi " # tlbop ";", \ + ARM64_WORKAROUND_REPEAT_TLBI, \ + CONFIG_ARM64_WORKAROUND_REPEAT_TLBI) \ + "dsb " # sh ";" \ + "isb;" \ + : : : "memory"); \ +} + /* * FLush TLB by VA. This will likely be used in a loop, so the caller * is responsible to use the appropriate memory barriers before/after @@ -75,10 +93,10 @@ TLB_HELPER(flush_guest_tlb_local, vmalls12e1, nsh) TLB_HELPER(flush_guest_tlb, vmalls12e1is, ish) =20 /* Flush local TLBs, current VMID, stage-1 only */ -TLB_HELPER(flush_guest_tlb_s1_local, vmalle1, nsh) +TLB_HELPER_NTLBPA(flush_guest_tlb_s1_local, vmalle1, nsh) =20 /* Flush innershareable TLBs, current VMID, stage-1 only */ -TLB_HELPER(flush_guest_tlb_s1, vmalle1is, ish) +TLB_HELPER_NTLBPA(flush_guest_tlb_s1, vmalle1is, ish) =20 /* Flush local TLBs, all VMIDs, non-hypervisor mode */ TLB_HELPER(flush_all_guests_tlb_local, alle1, nsh) @@ -104,8 +122,6 @@ TLB_HELPER_VA(__flush_xen_tlb_one, vae2is) */ static inline void flush_guest_tlb_range_ipa(paddr_t ipa, unsigned long si= ze) { - paddr_t end; - /* * If IPA range is too big (empirically found to be 256M), then fallba= ck to * full TLB flush. @@ -113,27 +129,42 @@ static inline void flush_guest_tlb_range_ipa(paddr_t = ipa, unsigned long size) if ( size > SZ_256M ) return flush_guest_tlb(); =20 - end =3D ipa + size; - - /* - * See ARM ARM DDI 0487L.b D8.17.6.1 (Invalidating TLB entries from st= age 2 - * translations) for details of TLBI sequence. - */ - dsb(ishst); /* Ensure prior page-tables updates have completed */ - while ( ipa < end ) + else if ( size > 0 ) { - /* Flush stage-2 TLBs for ipa address */ - asm_inline volatile ( - "tlbi ipas2e1is, %0;" : : "r" (ipa >> PAGE_SHIFT) : "memory" ); - ipa +=3D PAGE_SIZE; + paddr_t end =3D ipa + size; + + /* + * See ARM ARM DDI 0487L.b D8.17.6.1 (Invalidating TLB entries from + * stage 2 translations) for details on TLBI sequence. + */ + dsb(ishst); /* Ensure prior page-tables updates have completed */ + while ( ipa < end ) + { + /* Flush stage-2 TLBs for ipa address */ + asm_inline volatile ( + "tlbi ipas2e1is, %0;" : : "r" (ipa >> PAGE_SHIFT) : "memor= y" ); + ipa +=3D PAGE_SIZE; + } + if ( cpus_have_const_cap(ARM_HAS_NTLBPA) ) + asm_inline volatile ( + ALTERNATIVE( + "nop; nop;", + "dsb ish;" + "tlbi ipas2e1is, %0;", + ARM64_WORKAROUND_REPEAT_TLBI, + CONFIG_ARM64_WORKAROUND_REPEAT_TLBI) + "dsb ish;" + "isb;" + : : "r" ((ipa - PAGE_SIZE) >> PAGE_SHIFT) : "memory" ); + else + /* + * As ARM64_WORKAROUND_REPEAT_TLBI is required to be applied to + * last TLBI of the sequence, it is only needed to be handled = in + * the following invocation. Final dsb() and isb() are also ap= plied + * in the following invocation. + */ + flush_guest_tlb_s1(); } - /* - * As ARM64_WORKAROUND_REPEAT_TLBI is required to be applied to last T= LBI - * of the sequence, it is only needed to be handled in the following - * invocation. Final dsb() and isb() are also applied in the following - * invocation. - */ - flush_guest_tlb_s1(); } =20 #endif /* __ASM_ARM_ARM64_FLUSHTLB_H__ */ diff --git a/xen/arch/arm/include/asm/cpufeature.h b/xen/arch/arm/include/a= sm/cpufeature.h index 13353c8e1a..9f796ed4c1 100644 --- a/xen/arch/arm/include/asm/cpufeature.h +++ b/xen/arch/arm/include/asm/cpufeature.h @@ -76,8 +76,9 @@ #define ARM_WORKAROUND_BHB_SMCC_3 15 #define ARM_HAS_SB 16 #define ARM64_WORKAROUND_1508412 17 +#define ARM_HAS_NTLBPA 18 =20 -#define ARM_NCAPS 18 +#define ARM_NCAPS 19 =20 #ifndef __ASSEMBLER__ =20 @@ -269,7 +270,8 @@ struct cpuinfo_arm { unsigned long ets:4; unsigned long __res1:4; unsigned long afp:4; - unsigned long __res2:12; + unsigned long ntlbpa:4; + unsigned long __res2:8; unsigned long ecbhb:4; =20 /* MMFR2 */ @@ -430,8 +432,24 @@ struct cpuinfo_arm { register_t bits[1]; } aux32; =20 - struct { + union { register_t bits[6]; + struct { + /* MMFR0 */ + unsigned long __res0:32; + /* MMFR1 */ + unsigned long __res1:32; + /* MMFR2 */ + unsigned long __res2:32; + /* MMFR3 */ + unsigned long __res3:32; + /* MMFR4 */ + unsigned long __res4:32; + /* MMFR5 */ + unsigned long __res5:4; + unsigned long ntlbpa:4; + unsigned long __res6:24; + }; } mm32; =20 struct { diff --git a/xen/arch/arm/include/asm/processor.h b/xen/arch/arm/include/as= m/processor.h index 1a48c9ff3b..85f3b643a0 100644 --- a/xen/arch/arm/include/asm/processor.h +++ b/xen/arch/arm/include/asm/processor.h @@ -459,9 +459,16 @@ /* FSR long format */ #define FSRL_STATUS_DEBUG (_AC(0x22,UL)<<0) =20 +#ifdef CONFIG_ARM_32 +#define MM32_NTLBPA_SUPPORT_NI 0x0 +#define MM32_NTLBPA_SUPPORT_IMP 0x1 +#endif + #ifdef CONFIG_ARM_64 #define MM64_VMID_8_BITS_SUPPORT 0x0 #define MM64_VMID_16_BITS_SUPPORT 0x2 +#define MM64_NTLBPA_SUPPORT_NI 0x0 +#define MM64_NTLBPA_SUPPORT_IMP 0x1 #endif =20 #ifndef __ASSEMBLER__ --=20 2.43.0 From nobody Fri Dec 12 19:30:14 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) client-ip=192.237.175.120; envelope-from=xen-devel-bounces@lists.xenproject.org; helo=lists.xenproject.org; Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=pass(p=none dis=none) header.from=gmail.com ARC-Seal: i=1; a=rsa-sha256; t=1765202163; cv=none; d=zohomail.com; s=zohoarc; b=Cz+LxViQwktRumNv1cu0D5sEn8CkoT0p38UbfxyQInp6rAMk7ybDVrXyHu/Pjk3In4qYVyAlB4pskibfrzq4j9E/VWviBiEwr9WI2Fqtk8E+IDUfERpV5qwjtusPIQTDPTqQiQx1Xwd/HcvhQya4hiCvtsaDqL74tqg96gB2WOs= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1765202163; h=Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=MPopwKkyNB/YxawSKEN75nDXwr6v2rb4f3Mwy9XTqs4=; b=XYfU0S7+BQ59xHBePlKZYMYDj6QOrWDHRAZLZDngFLMvIwDhamADpO9DizV8qGxQ6r6vRqIzY70AgmcLoyBtqr+H+RHVbQ0Wwwb60qmUyOh5kjsVWOXccUyoofASq7nAwxHqTBQUUb8S4TLLY/MYvLMq/XTqMfhzngidlgykJ1U= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) by mx.zohomail.com with SMTPS id 1765202163181366.25367415546486; Mon, 8 Dec 2025 05:56:03 -0800 (PST) Received: from list by lists.xenproject.org with outflank-mailman.1180641.1503813 (Exim 4.92) (envelope-from ) id 1vSbiO-0005xr-8z; Mon, 08 Dec 2025 13:55:48 +0000 Received: by outflank-mailman (output) from mailman id 1180641.1503813; Mon, 08 Dec 2025 13:55:48 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1vSbiO-0005xi-3d; Mon, 08 Dec 2025 13:55:48 +0000 Received: by outflank-mailman (input) for mailman id 1180641; Mon, 08 Dec 2025 13:55:46 +0000 Received: from se1-gles-sth1-in.inumbo.com ([159.253.27.254] helo=se1-gles-sth1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1vSbiM-00055Z-SP for xen-devel@lists.xenproject.org; Mon, 08 Dec 2025 13:55:46 +0000 Received: from mail-wm1-x32d.google.com (mail-wm1-x32d.google.com [2a00:1450:4864:20::32d]) by se1-gles-sth1.inumbo.com (Halon) with ESMTPS id 98332bf2-d43d-11f0-9d1b-b5c5bf9af7f9; Mon, 08 Dec 2025 14:55:46 +0100 (CET) Received: by mail-wm1-x32d.google.com with SMTP id 5b1f17b1804b1-4775e891b5eso22220355e9.2 for ; Mon, 08 Dec 2025 05:55:46 -0800 (PST) Received: from PKL-HASEEBA-LT.. ([39.36.89.17]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-4792afe32cbsm144746585e9.0.2025.12.08.05.55.41 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 08 Dec 2025 05:55:44 -0800 (PST) X-Outflank-Mailman: Message body and most headers restored to incoming version X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: 98332bf2-d43d-11f0-9d1b-b5c5bf9af7f9 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1765202145; x=1765806945; darn=lists.xenproject.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=MPopwKkyNB/YxawSKEN75nDXwr6v2rb4f3Mwy9XTqs4=; b=LaWRiAWH3U9p2RIolrxorVorRddIpUL9KEB4qmQO3ep2pEvJKXhfLOSTmTHsqABJs9 WgiRp31Po7IXsXe/f6DWKKYIrN1Ki1vwc0+lq2tGfQGcub6iU7ep3WWj+mlytms/2M54 0HSpMEhGCwPVGqf688jmiEHCmMoZGHVNyb+5h9Iav3Bdu+Pcjzti1AaAusFiy8wgr6+Z zBKlqwtQ+pBTndaaMkwYlSZMk6YnSQa0Ql3r4L2XX6s0NAJoXO9CebaomAZoSWpINS0/ KYTiIFbv2SCjnRl/3RcARj5LoaFfuzwyRL0J1nmJLdFwfWL1eW5MqPyDk3KBS5zeodl4 MZtg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1765202145; x=1765806945; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=MPopwKkyNB/YxawSKEN75nDXwr6v2rb4f3Mwy9XTqs4=; b=M6L7RUGwLfWWH54Z83MIpHf1LqljWqp6syF/plFznR8fcyHWXxoq7FGkYB/RfWXBHk vkL29rO1ZyNNbZU6VQgD4OnLC/fvU4sn0Q1PwESGx2oL79itHpwI3AHhVO4gEJPrWpta X3Tw70DJXDqEYlHFeeOMnyDxCX8rJ4uDi8yGsiIpLQb/2lsetrbzEwtvLspBZprPd7Io uquk28i/5C7XHtL4gtbZtk1NliJPxNA9wgOzVcBzCM+AktnpYVrqF33l8zNHk3NZu/OV 5KXD7KYkeU5a80VOSAxA2I8D2yiw1KIN+TI1EslUBgsl6NDjnLbDPj0Em11UknGWgr3E qFEQ== X-Gm-Message-State: AOJu0YyZJTCjhnaDEjFWyBHKlVCRlWvSfPDRmg1D0ioNP/E+ozotSmUa wcbForT1pHwk0qKfuBVCfCH7jHXsldhFVNNMEsCYorWCvQWD+tBKq27lTWGZ+LiN X-Gm-Gg: ASbGncsAfp2XKwc0auVl9Pxaeo4v6VMQwD1VcAxi1nWYxTcY5zJtbc/huJemPVaLyJX HKc1c81Vi9baV3D39TiN+eoXFEMB79lh+anYFz0IGkq00YUCdlDZpVIUVNq49SJZkrrWVK03Y9m evTu4hkPaWWO6JSIqtZXW/EVWhzaOEi4ndvsx+oHMMzmoAZcDt+NDJPopR3UPnbLquXn50AcfSa tMF8igiw7nhCKEhQ2Ogmkmg/u1S24giVE/NE4RL4e9H+BVCB2sLAiMaupDm0Q8/44tAnor1QQZd /VKo8M7lgEQj2YbMm/3jmstQqzhlmcWTpasmsINxgln2D+d3voyhqx/uGMKWWZtoMJEgsmpqF5S QoLHpkehchB4oWHtnjl8KBxZWkTRcqjwAk4LIvj166alwYkazRiT59iNyiBGwX5D3GXs76g1BAb lfl2W4rKR5sPb5du3gKL/V+w6wNZM1MJ1ML2IF4CVM+Gs0Php3 X-Google-Smtp-Source: AGHT+IFKsQ6KVKizbOCvWoWILsKDhV/jKkHa/MC30/90rsJ7rWCeFy9yjpUbWDLrWUqCwyT+/BK/SQ== X-Received: by 2002:a05:600c:a43:b0:47a:75b6:32c with SMTP id 5b1f17b1804b1-47a7b17cfdfmr4106165e9.2.1765202145199; Mon, 08 Dec 2025 05:55:45 -0800 (PST) From: Haseeb Ashraf To: xen-devel@lists.xenproject.org Cc: Haseeb Ashraf , Stefano Stabellini , Julien Grall , Bertrand Marquis , Michal Orzel , Volodymyr Babchuk , Dan Driscoll , Noor Ahsan Khawaja , Fahad Arslan , Andrew Bachtel Subject: [PATCH 3/3] xen/arm32: add CPU capability for IPA-based TLBI Date: Mon, 8 Dec 2025 18:55:16 +0500 Message-ID: <68ad0721305814f6d7081223df4039b71627ae1f.1765197209.git.haseeb.ashraf@siemens.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ZohoMail-DKIM: pass (identity @gmail.com) X-ZM-MESSAGEID: 1765202165016158500 Content-Type: text/plain; charset="utf-8" From: Haseeb Ashraf This feature is available since armv8 and can be used to perform IPA-based TLBI for arm32. XENMEM_remove_from_physmap performs this invalidation in each hypercall so this code path will be optimized, instead of performing a TLBIALL each time in presence of nTLBPA. Suggested-by: Julien Grall Signed-off-by: Haseeb Ashraf Changes in v3: - There are no functional changes in this version. There are minor code updates and comment updates as per the feedback on v2. - The cpregs are defined in order as per Coprocessor-> CRn-> Opcode 1 -> CRm-> Opcode 2. - Added comment to explain why IPA-based TLBI is added only in presence of FEAT_nTLBPA. - Replaced `goto default_tlbi` with if...else. - Removed extra definitions of MM32_UNITLB_* macros which were not being used. Changes in v2: - This commit is implemented in v2 as per the feedback to implement IPA-based TLBI for Arm32 in addition to Arm64. --- xen/arch/arm/cpufeature.c | 12 +++++++ xen/arch/arm/include/asm/arm32/flushtlb.h | 42 ++++++++++++++++++++--- xen/arch/arm/include/asm/cpregs.h | 4 +++ xen/arch/arm/include/asm/cpufeature.h | 15 ++++---- xen/arch/arm/include/asm/processor.h | 3 ++ 5 files changed, 65 insertions(+), 11 deletions(-) diff --git a/xen/arch/arm/cpufeature.c b/xen/arch/arm/cpufeature.c index 9fa1c45869..d18c6449c6 100644 --- a/xen/arch/arm/cpufeature.c +++ b/xen/arch/arm/cpufeature.c @@ -18,6 +18,11 @@ DECLARE_BITMAP(cpu_hwcaps, ARM_NCAPS); struct cpuinfo_arm __read_mostly domain_cpuinfo; =20 #ifdef CONFIG_ARM_32 +static bool has_tlb_ipa_instruction(const struct arm_cpu_capabilities *ent= ry) +{ + return system_cpuinfo.mm32.unitlb =3D=3D MM32_UNITLB_BY_IPA; +} + static bool has_ntlbpa(const struct arm_cpu_capabilities *entry) { return system_cpuinfo.mm32.ntlbpa =3D=3D MM32_NTLBPA_SUPPORT_IMP; @@ -37,6 +42,13 @@ static bool has_sb_instruction(const struct arm_cpu_capa= bilities *entry) #endif =20 static const struct arm_cpu_capabilities arm_features[] =3D { +#ifdef CONFIG_ARM_32 + { + .desc =3D "IPA-based TLB Invalidation", + .capability =3D ARM32_HAS_TLB_IPA, + .matches =3D has_tlb_ipa_instruction, + }, +#endif #if defined(CONFIG_ARM_32) || defined(CONFIG_ARM_64) { .desc =3D "Intermediate caching of translation table walks (nTLBPA= )", diff --git a/xen/arch/arm/include/asm/arm32/flushtlb.h b/xen/arch/arm/inclu= de/asm/arm32/flushtlb.h index 7cff042508..3e6f86f6d2 100644 --- a/xen/arch/arm/include/asm/arm32/flushtlb.h +++ b/xen/arch/arm/include/asm/arm32/flushtlb.h @@ -1,6 +1,8 @@ #ifndef __ASM_ARM_ARM32_FLUSHTLB_H__ #define __ASM_ARM_ARM32_FLUSHTLB_H__ =20 +#include /* For SZ_* macros. */ + /* * Every invalidation operation use the following patterns: * @@ -104,12 +106,42 @@ static inline void flush_guest_tlb_range_ipa(paddr_t = ipa, unsigned long size) { /* - * Following can invalidate both stage-1 and stage-2 TLBs depending up= on - * the execution mode. - * - * See ARMv8 (DDI 0487L.b): G5-11698 Table G5-23. + * IPA-based TLBI is used only in presence of nTLBPA, otherwise, stage= -1 + * invalidation would still be required and there is no separate TLBI = for + * stage-1 on Arm32. So in absence of nTLBPA, it is pointless to flush= by + * IPA. */ - flush_guest_tlb(); + if ( cpus_have_const_cap(ARM_HAS_NTLBPA) && + cpus_have_const_cap(ARM32_HAS_TLB_IPA) ) + { + /* + * If IPA range is too big (empirically found to be 256M), then + * fallback to full TLB flush + */ + if ( size > SZ_256M ) + /* + * Following can invalidate both stage-1 and stage-2 TLBs depe= nding + * upon the execution mode. + * + * See ARMv8 (DDI 0487L.b): G5-11698 Table G5-23. + */ + flush_guest_tlb(); + else + { + paddr_t end =3D ipa + size; + + dsb(ishst); /* Ensure prior page-tables updates have completed= */ + while ( ipa < end ) + { + /* Flush stage-2 TLBs for ipa address. */ + asm volatile(STORE_CP32(0, TLBIIPAS2IS) + : : "r" (ipa >> PAGE_SHIFT) : "memory"); + ipa +=3D PAGE_SIZE; + } + dsb(ish); + isb(); + } + } } =20 #endif /* __ASM_ARM_ARM32_FLUSHTLB_H__ */ diff --git a/xen/arch/arm/include/asm/cpregs.h b/xen/arch/arm/include/asm/c= pregs.h index a7503a190f..51f091dace 100644 --- a/xen/arch/arm/include/asm/cpregs.h +++ b/xen/arch/arm/include/asm/cpregs.h @@ -223,9 +223,13 @@ #define TLBIMVA p15,0,c8,c7,1 /* invalidate unified TLB entry by= MVA */ #define TLBIASID p15,0,c8,c7,2 /* invalid unified TLB by ASID mat= ch */ #define TLBIMVAA p15,0,c8,c7,3 /* invalidate unified TLB entries = by MVA all ASID */ +#define TLBIIPAS2IS p15,4,c8,c0,1 /* Invalidate unified TLB entry fo= r stage 2 by IPA inner shareable */ +#define TLBIIPAS2LIS p15,4,c8,c0,5 /* Invalidate unified TLB entry fo= r stage 2 last level by IPA inner shareable */ #define TLBIALLHIS p15,4,c8,c3,0 /* Invalidate Entire Hyp. Unified = TLB inner shareable */ #define TLBIMVAHIS p15,4,c8,c3,1 /* Invalidate Unified Hyp. TLB by = MVA inner shareable */ #define TLBIALLNSNHIS p15,4,c8,c3,4 /* Invalidate Entire Non-Secure No= n-Hyp. Unified TLB inner shareable */ +#define TLBIIPAS2 p15,4,c8,c4,1 /* Invalidate unified TLB entry fo= r stage 2 by IPA */ +#define TLBIIPAS2L p15,4,c8,c4,5 /* Invalidate unified TLB entry fo= r stage 2 last level by IPA */ #define TLBIALLH p15,4,c8,c7,0 /* Invalidate Entire Hyp. Unified = TLB */ #define TLBIMVAH p15,4,c8,c7,1 /* Invalidate Unified Hyp. TLB by = MVA */ #define TLBIALLNSNH p15,4,c8,c7,4 /* Invalidate Entire Non-Secure No= n-Hyp. Unified TLB */ diff --git a/xen/arch/arm/include/asm/cpufeature.h b/xen/arch/arm/include/a= sm/cpufeature.h index 9f796ed4c1..07f1d770b3 100644 --- a/xen/arch/arm/include/asm/cpufeature.h +++ b/xen/arch/arm/include/asm/cpufeature.h @@ -77,8 +77,9 @@ #define ARM_HAS_SB 16 #define ARM64_WORKAROUND_1508412 17 #define ARM_HAS_NTLBPA 18 +#define ARM32_HAS_TLB_IPA 19 =20 -#define ARM_NCAPS 19 +#define ARM_NCAPS 20 =20 #ifndef __ASSEMBLER__ =20 @@ -440,15 +441,17 @@ struct cpuinfo_arm { /* MMFR1 */ unsigned long __res1:32; /* MMFR2 */ - unsigned long __res2:32; + unsigned long __res2:16; + unsigned long unitlb:4; + unsigned long __res3:12; /* MMFR3 */ - unsigned long __res3:32; - /* MMFR4 */ unsigned long __res4:32; + /* MMFR4 */ + unsigned long __res5:32; /* MMFR5 */ - unsigned long __res5:4; + unsigned long __res6:4; unsigned long ntlbpa:4; - unsigned long __res6:24; + unsigned long __res7:24; }; } mm32; =20 diff --git a/xen/arch/arm/include/asm/processor.h b/xen/arch/arm/include/as= m/processor.h index 85f3b643a0..eda39566e1 100644 --- a/xen/arch/arm/include/asm/processor.h +++ b/xen/arch/arm/include/asm/processor.h @@ -460,6 +460,9 @@ #define FSRL_STATUS_DEBUG (_AC(0x22,UL)<<0) =20 #ifdef CONFIG_ARM_32 +#define MM32_UNITLB_NI 0x0 +#define MM32_UNITLB_BY_IPA 0x6 + #define MM32_NTLBPA_SUPPORT_NI 0x0 #define MM32_NTLBPA_SUPPORT_IMP 0x1 #endif --=20 2.43.0