From nobody Tue Dec 2 01:26:24 2025 Received: from shelob.surriel.com (shelob.surriel.com [96.67.55.147]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E55272E22BF for ; Fri, 21 Nov 2025 18:55:39 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=96.67.55.147 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1763751342; cv=none; b=lYnQ92grfXBVoVYvIKtW0grvXx2XmaDEfGN4X0T7KY55exkEeNbZHBaHPD1LKDvmOy547K5YsZpKvfyFc27YKTCBAVGmI3nP5v5yi75F/fR0aS5Oear0hi43R2Kyu2hpBPWMjqqz2uzzLLE+auNd/Iyg9wqrqfJ4eHdNBa2v+LA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1763751342; c=relaxed/simple; bh=Cd3YiqfWeJk791JefxPprm5/Wn769yeHF/2YpqSsR1I=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=L+J5w0V3TbrGZZMf4hT9stBvq5S6Jv07ZAo3ocPHEaEGLDgQ6/xvKK1+i4xHGUl7pJpRVhiQl5FAYXEfrLL1EeMKVuhNHfn7jfZxRvujhk1lVBQ1ryZ7jexieEBo3jkPyLG7M0QXg2iRibPeDHUzjNr8J4ddICxrfiv6MJGf0J8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=surriel.com; spf=pass smtp.mailfrom=surriel.com; dkim=pass (2048-bit key) header.d=surriel.com header.i=@surriel.com header.b=NA5z7vlf; arc=none smtp.client-ip=96.67.55.147 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=surriel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=surriel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=surriel.com header.i=@surriel.com header.b="NA5z7vlf" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=surriel.com ; s=mail; h=Content-Transfer-Encoding:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:Cc:To:From:Sender:Reply-To:Content-Type:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Id:List-Help:List-Unsubscribe:List-Subscribe: List-Post:List-Owner:List-Archive; bh=lFtNXw38IE3HRdDCdMoM1V9qz9lJ/AVCWnMaz6ZoLPI=; b=NA5z7vlf/bsATWv9tvXtLCcgfw hYmnIk5IUs5Qp+EEs8zNJYhVYqbJ9TX6F636UFk538EEkuW5qkDhjdlOJM+azQhNl4bErHZSVkjrN 9wHW3G2j/+HK0EJDL/QvzmF7ePEkA2/Al318LvauERbHC5bwkqcmZjqVMzdYx30UWHD4ugIzx8mNP BIkUitADajkzZO4JShqJ/hESdl/dT8351dE/iqrnUgI/wYCCVzwTslNYZir/HptpGijk/xuAHu8z7 cuj0LFaDzVL3oWXMLuWdVeloAVxeYZkex0xulc4I9sCvyZ6Id6m8x3N2lHdH4oyKaOeqo1piZWmzK Bgpr4WvQ==; Received: from fangorn.home.surriel.com ([10.0.13.7]) by shelob.surriel.com with esmtpsa (TLS1.2) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.97.1) (envelope-from ) id 1vMWI9-000000003i4-1keH; Fri, 21 Nov 2025 13:55:34 -0500 From: Rik van Riel To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, x86@kernel.org, dave.hansen@linux.intel.com, peterz@infradead.org, kernel-team@meta.com, bp@alien8.de, Rik van Riel , Rik van Riel Subject: [RFC v5 6/8] x86/mm: use RAR for kernel TLB flushes Date: Fri, 21 Nov 2025 13:54:27 -0500 Message-ID: <20251121185530.21876-7-riel@surriel.com> X-Mailer: git-send-email 2.51.1 In-Reply-To: <20251121185530.21876-1-riel@surriel.com> References: <20251121185530.21876-1-riel@surriel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Rik van Riel Use Intel RAR for kernel TLB flushes, when enabled. Pass in PCID 0 to smp_call_rar_many() to flush the specified addresses, regardless of which PCID they might be cached under in any destination CPU. Signed-off-by: Rik van Riel --- arch/x86/mm/tlb.c | 36 ++++++++++++++++++++++++++++++++++++ 1 file changed, 36 insertions(+) diff --git a/arch/x86/mm/tlb.c b/arch/x86/mm/tlb.c index f5b93e01e347..19c28386d8de 100644 --- a/arch/x86/mm/tlb.c +++ b/arch/x86/mm/tlb.c @@ -22,6 +22,7 @@ #include #include #include +#include #include =20 #include "mm_internal.h" @@ -1489,6 +1490,17 @@ static void do_flush_tlb_all(void *info) __flush_tlb_all(); } =20 +static void rar_full_flush(const cpumask_t *cpumask) +{ + guard(preempt)(); + smp_call_rar_many(cpumask, 0, 0, TLB_FLUSH_ALL); +} + +static void rar_flush_all(void) +{ + rar_full_flush(cpu_online_mask); +} + void flush_tlb_all(void) { count_vm_tlb_event(NR_TLB_REMOTE_FLUSH); @@ -1496,6 +1508,8 @@ void flush_tlb_all(void) /* First try (faster) hardware-assisted TLB invalidation. */ if (cpu_feature_enabled(X86_FEATURE_INVLPGB)) invlpgb_flush_all(); + else if (cpu_feature_enabled(X86_FEATURE_RAR)) + rar_flush_all(); else /* Fall back to the IPI-based invalidation. */ on_each_cpu(do_flush_tlb_all, NULL, 1); @@ -1525,15 +1539,35 @@ static void do_kernel_range_flush(void *info) struct flush_tlb_info *f =3D info; unsigned long addr; =20 + /* + * With PTI kernel TLB entries in all PCIDs need to be flushed. + * With RAR the PCID space becomes so large, we might as well flush it al= l. + * + * Either of the two by itself works with targeted flushes. + */ + if (cpu_feature_enabled(X86_FEATURE_RAR) && + cpu_feature_enabled(X86_FEATURE_PTI)) { + invpcid_flush_all(); + return; + } + /* flush range by one by one 'invlpg' */ for (addr =3D f->start; addr < f->end; addr +=3D PAGE_SIZE) flush_tlb_one_kernel(addr); } =20 +static void rar_kernel_range_flush(struct flush_tlb_info *info) +{ + guard(preempt)(); + smp_call_rar_many(cpu_online_mask, 0, info->start, info->end); +} + static void kernel_tlb_flush_all(struct flush_tlb_info *info) { if (cpu_feature_enabled(X86_FEATURE_INVLPGB)) invlpgb_flush_all(); + else if (cpu_feature_enabled(X86_FEATURE_RAR)) + rar_flush_all(); else on_each_cpu(do_flush_tlb_all, NULL, 1); } @@ -1542,6 +1576,8 @@ static void kernel_tlb_flush_range(struct flush_tlb_i= nfo *info) { if (cpu_feature_enabled(X86_FEATURE_INVLPGB)) invlpgb_kernel_range_flush(info); + else if (cpu_feature_enabled(X86_FEATURE_RAR)) + rar_kernel_range_flush(info); else on_each_cpu(do_kernel_range_flush, info, 1); } --=20 2.51.1