From nobody Fri Dec 26 17:23:07 2025 Received: from mail-pj1-f43.google.com (mail-pj1-f43.google.com [209.85.216.43]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9F0E718AF4 for ; Tue, 2 Jan 2024 22:01:38 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=sifive.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=sifive.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=sifive.com header.i=@sifive.com header.b="k0R4KwlT" Received: by mail-pj1-f43.google.com with SMTP id 98e67ed59e1d1-28ce6eeb4easo310245a91.0 for ; Tue, 02 Jan 2024 14:01:38 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1704232898; x=1704837698; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Sx1CrYCYqcy0I3lbTucfKtzktUZzOavzb19InQrqEBM=; b=k0R4KwlTr46S55jHvhzg/o8eollZhzK6pzTHwCz1T8zqKvHQU3BsmWUf4lurfepP/m 4IYn3Ua468pGBIZwAcdXb7+rO3nCvvl5Mo+1W07ES0fRyOjfjm+mdUibpgGnEQwBP3Lk KG8T359hvMvcuU+36tIUT3hc5ov0asYY1zDGJCd1lionGf4/sWZukLGPJMu/z+58IDKZ cyBWCM+M5oJuwwiyQUyt2b0NTyRL1VMYFPrsy49tlGm73PNeqHK+SKg3qKrTUjYsyPVN 4tDrj+rh/CkyVAaHtD8EtNz7AbCQ0olPZ4JKDFmAroinso3n64OkV1H/D/bP3ugOARAb iihA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1704232898; x=1704837698; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Sx1CrYCYqcy0I3lbTucfKtzktUZzOavzb19InQrqEBM=; b=IsuWJSh15s3LHoOn1AQhByGZtMasWaQ73rHj40PKhqnHkHHXv2TRMuOIIpj32faYg6 e1BzBw5LjF9P1M9vA2jzJQ6nKlcQg/glC2L0uLTCFqkpBoed0EW88FpI3qcmNLOiuTOl KXXH9lugPKe6Iu5xkDZmzMuD+k3soM0BwkujappzW7oAgw6hmZrJneO6H9u8y7SFHhMt xfnipmCfPk0rNKGbR7prBUG0Wmee6ydvihD6vPHjI8kbNri5BLIvJSVm8vd+nIXnYeAe Qcx6AYYUi+Z/GUhrja5lyVaFnSkbb0HTpmIkopwhU5zU4t/o0pounIjNKaU3V92ZXxUw AnVQ== X-Gm-Message-State: AOJu0Yz5p8zt1366m442V/I2bCojlU/qfD+jghYPfzClYjZ6g2l2NYoM i0z9h63+TMWNYKXd/J3RdA++XL8Ozaxr+g== X-Google-Smtp-Source: AGHT+IHSNOWtd16BNBpr13f63JilpUsY7HXImYHsivjSF6nbDu6T9MOgsg44oEC59WRv4BX5AKGF0w== X-Received: by 2002:a17:90a:728e:b0:28c:a9d0:33ff with SMTP id e14-20020a17090a728e00b0028ca9d033ffmr2151486pjg.62.1704232897945; Tue, 02 Jan 2024 14:01:37 -0800 (PST) Received: from sw06.internal.sifive.com ([4.53.31.132]) by smtp.gmail.com with ESMTPSA id r59-20020a17090a43c100b0028ce507cd7dsm101724pjg.55.2024.01.02.14.01.37 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 02 Jan 2024 14:01:37 -0800 (PST) From: Samuel Holland To: Palmer Dabbelt , linux-riscv@lists.infradead.org Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, Alexandre Ghiti , Samuel Holland Subject: [PATCH v4 01/12] riscv: Flush the instruction cache during SMP bringup Date: Tue, 2 Jan 2024 14:00:38 -0800 Message-ID: <20240102220134.3229156-2-samuel.holland@sifive.com> X-Mailer: git-send-email 2.42.0 In-Reply-To: <20240102220134.3229156-1-samuel.holland@sifive.com> References: <20240102220134.3229156-1-samuel.holland@sifive.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Instruction cache flush IPIs are sent only to CPUs in cpu_online_mask, so they will not target a CPU until it calls set_cpu_online() earlier in smp_callin(). As a result, if instruction memory is modified between the CPU coming out of reset and that point, then its instruction cache may contain stale data. Therefore, the instruction cache must be flushed after the set_cpu_online() synchronization point. Fixes: 08f051eda33b ("RISC-V: Flush I$ when making a dirty page executable") Signed-off-by: Samuel Holland Reviewed-by: Alexandre Ghiti --- Changes in v4: - New patch for v4 arch/riscv/kernel/smpboot.c | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/arch/riscv/kernel/smpboot.c b/arch/riscv/kernel/smpboot.c index d162bf339beb..48af5bd3ec30 100644 --- a/arch/riscv/kernel/smpboot.c +++ b/arch/riscv/kernel/smpboot.c @@ -26,7 +26,7 @@ #include #include =20 -#include +#include #include #include #include @@ -257,9 +257,10 @@ asmlinkage __visible void smp_callin(void) riscv_user_isa_enable(); =20 /* - * Remote TLB flushes are ignored while the CPU is offline, so emit - * a local TLB flush right now just in case. + * Remote cache and TLB flushes are ignored while the CPU is offline, + * so flush them both right now just in case. */ + local_flush_icache_all(); local_flush_tlb_all(); complete(&cpu_running); /* --=20 2.42.0 From nobody Fri Dec 26 17:23:07 2025 Received: from mail-pl1-f170.google.com (mail-pl1-f170.google.com [209.85.214.170]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id EEE5518B0D for ; Tue, 2 Jan 2024 22:01:39 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=sifive.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=sifive.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=sifive.com header.i=@sifive.com header.b="FfjNMhrQ" Received: by mail-pl1-f170.google.com with SMTP id d9443c01a7336-1d3ed1ca402so75965005ad.2 for ; Tue, 02 Jan 2024 14:01:39 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1704232899; x=1704837699; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=pynuwdwrgTDQTs9EZPEkGjuJgTxII1R0VKB5FEKmh8A=; b=FfjNMhrQ2J03UobrzaKUakTrVdNaNEJx8tGD521XPYAekzCF2+wKX4MPKhMGSK/gN8 A5/UK5wGRYTrb/y2VqWNWIu4O3FctlSLMghABai1+vBkqQpMhWeUEqi8Cx4hlPiMawDK Ue+oDPUDovmJDplqE74hieMY66whud/qCvV79FjxMmYmgUFpzbO0/Is1G9/Kh0qBC1B4 D2x2AIWsdIH/0+odvOSP1jCiv1pzomJNDCMaPfWN6GUpbsCTPCmOqMfuxrV1+YgrW6Of +wD1ZAlTyTwaTgY/vibR/mTHIEfhQ2DnV39lrz3a9ieOzH4FkQseQ1GRck7cWoCnTXDH ZMKg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1704232899; x=1704837699; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=pynuwdwrgTDQTs9EZPEkGjuJgTxII1R0VKB5FEKmh8A=; b=Y+xBmar7GaC1K6P573Uitx2y5jRbOMgVHRzBR4Jn4CaX8n3gU6FJXDXXfs8Ixf8wc6 lBK5E/oh4aAaf5osgzwfWnRdJAf+V0TTRN+LGvp3EVAKG9Mb/jY573AZanhSstidt/eG GW9pQy059geDkfn6qrhwTeRPs7tMZqzbmBAisG62MocvraD+dlJBybLBT9bbZzaXBPck rN95CUcUWzvcB4fN+6zWvY55BBgY+l+Tx8kuYwyNQzjAMJ6ba6fRKf+fHxxBsEc+waq2 4iT8CCHXOTNxpnH2v1+l9c80FFX4iNnjzT1p6/qMtItsC7ncSA15oRK4Bqour2pQEonl feSg== X-Gm-Message-State: AOJu0YwsTFIvZqyVxYhNLDz+U58ssYma+zISkNNmWiD8gj9H/Y5I1nSP +/Cg0IB206daqf/+UaO/OLrejuLX4Saa0bttroWrHUsSLdY= X-Google-Smtp-Source: AGHT+IEYPya8PYA19IRLK0MnT0T3isXxWKz5hXZR0znVEsgxQFvYhbUCwOfyfHQYZwqQ+xa53Aj2UQ== X-Received: by 2002:a17:90b:3ec2:b0:28c:1b98:e6e7 with SMTP id rm2-20020a17090b3ec200b0028c1b98e6e7mr8644807pjb.47.1704232899208; Tue, 02 Jan 2024 14:01:39 -0800 (PST) Received: from sw06.internal.sifive.com ([4.53.31.132]) by smtp.gmail.com with ESMTPSA id r59-20020a17090a43c100b0028ce507cd7dsm101724pjg.55.2024.01.02.14.01.38 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 02 Jan 2024 14:01:38 -0800 (PST) From: Samuel Holland To: Palmer Dabbelt , linux-riscv@lists.infradead.org Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, Alexandre Ghiti , Samuel Holland Subject: [PATCH v4 02/12] riscv: Use IPIs for remote cache/TLB flushes by default Date: Tue, 2 Jan 2024 14:00:39 -0800 Message-ID: <20240102220134.3229156-3-samuel.holland@sifive.com> X-Mailer: git-send-email 2.42.0 In-Reply-To: <20240102220134.3229156-1-samuel.holland@sifive.com> References: <20240102220134.3229156-1-samuel.holland@sifive.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" An IPI backend is always required in an SMP configuration, but an SBI implementation is not. For example, SBI will be unavailable when the kernel runs in M mode. Generally, IPIs are assumed to be faster than SBI calls due to the SBI context switch overhead. However, when SBI is used as the IPI backend, then the context switch cost must be paid anyway, and performing the cache/TLB flush directly in the SBI implementation is more efficient than inserting an interrupt to the kernel. This is the only scenario where riscv_ipi_set_virq_range()'s use_for_rfence parameter is false. Thus, it makes sense for remote fences to use IPIs by default, and make the SBI remote fence extension the special case. sbi_ipi_init() already checks riscv_ipi_have_virq_range(), so it only calls riscv_ipi_set_virq_range() when no other IPI device is available. So we can move the static key and drop the use_for_rfence parameter. Furthermore, the static branch only makes sense when CONFIG_RISCV_SBI is enabled. Optherwise, IPIs must be used. Add a fallback definition of riscv_use_sbi_for_rfence() which handles this case and removes the need to check CONFIG_RISCV_SBI elsewhere, such as in cacheflush.c. Signed-off-by: Samuel Holland Reviewed-by: Alexandre Ghiti --- Changes in v4: - New patch for v4 arch/riscv/include/asm/sbi.h | 4 ++++ arch/riscv/include/asm/smp.h | 15 ++------------- arch/riscv/kernel/sbi-ipi.c | 11 ++++++++++- arch/riscv/kernel/smp.c | 11 +---------- arch/riscv/mm/cacheflush.c | 5 ++--- arch/riscv/mm/tlbflush.c | 31 ++++++++++++++----------------- drivers/clocksource/timer-clint.c | 2 +- 7 files changed, 34 insertions(+), 45 deletions(-) diff --git a/arch/riscv/include/asm/sbi.h b/arch/riscv/include/asm/sbi.h index 0892f4421bc4..aeee0127df76 100644 --- a/arch/riscv/include/asm/sbi.h +++ b/arch/riscv/include/asm/sbi.h @@ -339,8 +339,12 @@ unsigned long riscv_cached_marchid(unsigned int cpu_id= ); unsigned long riscv_cached_mimpid(unsigned int cpu_id); =20 #if IS_ENABLED(CONFIG_SMP) && IS_ENABLED(CONFIG_RISCV_SBI) +DECLARE_STATIC_KEY_FALSE(riscv_sbi_for_rfence); +#define riscv_use_sbi_for_rfence() \ + static_branch_unlikely(&riscv_sbi_for_rfence) void sbi_ipi_init(void); #else +static inline bool riscv_use_sbi_for_rfence(void) { return false; } static inline void sbi_ipi_init(void) { } #endif =20 diff --git a/arch/riscv/include/asm/smp.h b/arch/riscv/include/asm/smp.h index 0d555847cde6..7ac80e9f2288 100644 --- a/arch/riscv/include/asm/smp.h +++ b/arch/riscv/include/asm/smp.h @@ -49,12 +49,7 @@ void riscv_ipi_disable(void); bool riscv_ipi_have_virq_range(void); =20 /* Set the IPI interrupt numbers for arch (called by irqchip drivers) */ -void riscv_ipi_set_virq_range(int virq, int nr, bool use_for_rfence); - -/* Check if we can use IPIs for remote FENCEs */ -DECLARE_STATIC_KEY_FALSE(riscv_ipi_for_rfence); -#define riscv_use_ipi_for_rfence() \ - static_branch_unlikely(&riscv_ipi_for_rfence) +void riscv_ipi_set_virq_range(int virq, int nr); =20 /* Check other CPUs stop or not */ bool smp_crash_stop_failed(void); @@ -104,16 +99,10 @@ static inline bool riscv_ipi_have_virq_range(void) return false; } =20 -static inline void riscv_ipi_set_virq_range(int virq, int nr, - bool use_for_rfence) +static inline void riscv_ipi_set_virq_range(int virq, int nr) { } =20 -static inline bool riscv_use_ipi_for_rfence(void) -{ - return false; -} - #endif /* CONFIG_SMP */ =20 #if defined(CONFIG_HOTPLUG_CPU) && (CONFIG_SMP) diff --git a/arch/riscv/kernel/sbi-ipi.c b/arch/riscv/kernel/sbi-ipi.c index a4559695ce62..1026e22955cc 100644 --- a/arch/riscv/kernel/sbi-ipi.c +++ b/arch/riscv/kernel/sbi-ipi.c @@ -13,6 +13,9 @@ #include #include =20 +DEFINE_STATIC_KEY_FALSE(riscv_sbi_for_rfence); +EXPORT_SYMBOL_GPL(riscv_sbi_for_rfence); + static int sbi_ipi_virq; =20 static void sbi_ipi_handle(struct irq_desc *desc) @@ -72,6 +75,12 @@ void __init sbi_ipi_init(void) "irqchip/sbi-ipi:starting", sbi_ipi_starting_cpu, NULL); =20 - riscv_ipi_set_virq_range(virq, BITS_PER_BYTE, false); + riscv_ipi_set_virq_range(virq, BITS_PER_BYTE); pr_info("providing IPIs using SBI IPI extension\n"); + + /* + * Use the SBI remote fence extension to avoid + * the extra context switch needed to handle IPIs. + */ + static_branch_enable(&riscv_sbi_for_rfence); } diff --git a/arch/riscv/kernel/smp.c b/arch/riscv/kernel/smp.c index 40420afbb1a0..1d06df04eb71 100644 --- a/arch/riscv/kernel/smp.c +++ b/arch/riscv/kernel/smp.c @@ -171,10 +171,7 @@ bool riscv_ipi_have_virq_range(void) return (ipi_virq_base) ? true : false; } =20 -DEFINE_STATIC_KEY_FALSE(riscv_ipi_for_rfence); -EXPORT_SYMBOL_GPL(riscv_ipi_for_rfence); - -void riscv_ipi_set_virq_range(int virq, int nr, bool use_for_rfence) +void riscv_ipi_set_virq_range(int virq, int nr) { int i, err; =20 @@ -197,12 +194,6 @@ void riscv_ipi_set_virq_range(int virq, int nr, bool u= se_for_rfence) =20 /* Enabled IPIs for boot CPU immediately */ riscv_ipi_enable(); - - /* Update RFENCE static key */ - if (use_for_rfence) - static_branch_enable(&riscv_ipi_for_rfence); - else - static_branch_disable(&riscv_ipi_for_rfence); } =20 static const char * const ipi_names[] =3D { diff --git a/arch/riscv/mm/cacheflush.c b/arch/riscv/mm/cacheflush.c index 55a34f2020a8..47c485bc7df0 100644 --- a/arch/riscv/mm/cacheflush.c +++ b/arch/riscv/mm/cacheflush.c @@ -21,7 +21,7 @@ void flush_icache_all(void) { local_flush_icache_all(); =20 - if (IS_ENABLED(CONFIG_RISCV_SBI) && !riscv_use_ipi_for_rfence()) + if (riscv_use_sbi_for_rfence()) sbi_remote_fence_i(NULL); else on_each_cpu(ipi_remote_fence_i, NULL, 1); @@ -69,8 +69,7 @@ void flush_icache_mm(struct mm_struct *mm, bool local) * with flush_icache_deferred(). */ smp_mb(); - } else if (IS_ENABLED(CONFIG_RISCV_SBI) && - !riscv_use_ipi_for_rfence()) { + } else if (riscv_use_sbi_for_rfence()) { sbi_remote_fence_i(&others); } else { on_each_cpu_mask(&others, ipi_remote_fence_i, NULL, 1); diff --git a/arch/riscv/mm/tlbflush.c b/arch/riscv/mm/tlbflush.c index e6659d7368b3..09b03bf71e6a 100644 --- a/arch/riscv/mm/tlbflush.c +++ b/arch/riscv/mm/tlbflush.c @@ -73,10 +73,10 @@ static void __ipi_flush_tlb_all(void *info) =20 void flush_tlb_all(void) { - if (riscv_use_ipi_for_rfence()) - on_each_cpu(__ipi_flush_tlb_all, NULL, 1); - else + if (riscv_use_sbi_for_rfence()) sbi_remote_sfence_vma_asid(NULL, 0, FLUSH_TLB_MAX_SIZE, FLUSH_TLB_NO_ASI= D); + else + on_each_cpu(__ipi_flush_tlb_all, NULL, 1); } =20 struct flush_tlb_range_data { @@ -96,7 +96,6 @@ static void __ipi_flush_tlb_range_asid(void *info) static void __flush_tlb_range(struct mm_struct *mm, unsigned long start, unsigned long size, unsigned long stride) { - struct flush_tlb_range_data ftd; const struct cpumask *cmask; unsigned long asid =3D FLUSH_TLB_NO_ASID; bool broadcast; @@ -119,20 +118,18 @@ static void __flush_tlb_range(struct mm_struct *mm, u= nsigned long start, broadcast =3D true; } =20 - if (broadcast) { - if (riscv_use_ipi_for_rfence()) { - ftd.asid =3D asid; - ftd.start =3D start; - ftd.size =3D size; - ftd.stride =3D stride; - on_each_cpu_mask(cmask, - __ipi_flush_tlb_range_asid, - &ftd, 1); - } else - sbi_remote_sfence_vma_asid(cmask, - start, size, asid); - } else { + if (!broadcast) { local_flush_tlb_range_asid(start, size, stride, asid); + } else if (riscv_use_sbi_for_rfence()) { + sbi_remote_sfence_vma_asid(cmask, start, size, asid); + } else { + struct flush_tlb_range_data ftd; + + ftd.asid =3D asid; + ftd.start =3D start; + ftd.size =3D size; + ftd.stride =3D stride; + on_each_cpu_mask(cmask, __ipi_flush_tlb_range_asid, &ftd, 1); } =20 if (mm) diff --git a/drivers/clocksource/timer-clint.c b/drivers/clocksource/timer-= clint.c index 9a55e733ae99..7ccc16dd6a76 100644 --- a/drivers/clocksource/timer-clint.c +++ b/drivers/clocksource/timer-clint.c @@ -251,7 +251,7 @@ static int __init clint_timer_init_dt(struct device_nod= e *np) } =20 irq_set_chained_handler(clint_ipi_irq, clint_ipi_interrupt); - riscv_ipi_set_virq_range(rc, BITS_PER_BYTE, true); + riscv_ipi_set_virq_range(rc, BITS_PER_BYTE); clint_clear_ipi(); #endif =20 --=20 2.42.0 From nobody Fri Dec 26 17:23:07 2025 Received: from mail-pj1-f48.google.com (mail-pj1-f48.google.com [209.85.216.48]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1B04C18C13 for ; Tue, 2 Jan 2024 22:01:40 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=sifive.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=sifive.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=sifive.com header.i=@sifive.com header.b="AGqNsbmM" Received: by mail-pj1-f48.google.com with SMTP id 98e67ed59e1d1-28c7c9b19f1so2514434a91.1 for ; Tue, 02 Jan 2024 14:01:40 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1704232900; x=1704837700; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=XA8TOHyyeKTc95PFXJRpYatfRR5oPHMuobG+iziT3HE=; b=AGqNsbmMnp9vjyebsFChQjy7nWz6SWLDLP0QAiuNu27kO49irRUptsDDwu9E5V9Ket xvodU/gGJFv9vP5cV6Q8RA8C4/ZF7/VZCDCtJQBuiyuuiJ4ia418Kc4zam429n8/ofuS Uo4UFXB4eRCJsditxh58ECPJ53zpK0LjtIA1qftThpO0uq2LhqaLAgJjWjzzkUlH9NmS 3hbJTa8CZsVqH4yce8yvbbnq0iYoVoCBRNG3qqBh9fSl22ovicWKevmCdZu8bx5sSLCU OmkkQy20Jb0DsOTi5CKI0zsCgdq/J983tSNp8xKiwyEAaignZWNbTGCJ2zZ1fv9T5cu3 jclQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1704232900; x=1704837700; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=XA8TOHyyeKTc95PFXJRpYatfRR5oPHMuobG+iziT3HE=; b=V4td4ENHyyYtcM+wxiSEEbRhkVd9kzd4wISlCEKoeyA8vlg5iLFOFrreiGA8lLZ2LE 3O4S/e+3Em/tuoUbqJ5SHZKPIk7iOeV55uiRa0zwdoD6TbSl2xNY5MBf/Ny4tjDHJ/43 qpAiTu5mdeL89kXl556Zg38xKoTVpQWOa7E7kX6YkCvC6yezt2zvubyeSbuEQfkn7wA6 udWT1JyiiF63FG99pVHwz4tfAdm828O3c7JV7QXgG/sxvFj6J6GHFxx6CCXbJTsYshFq jy5G75HSowy9vKM+omj61B0LtBoOuWOBMSwhJO7jfb7WAMxtWXxWFVRZXEf8rmsHDVH0 8N2A== X-Gm-Message-State: AOJu0YyDVgS/6Foj946cMGA17geeGxh7FA1lraLhsxHPWaPJukZcZ0JT b78N1xOwbYvLuELqWHf99NR/48hHHDkNRQ== X-Google-Smtp-Source: AGHT+IGXoPWHR5PafwwvCfsK03qnPYeIgtaQPhGwO0vZvHcHemW0jssRCcGF3figFCmbRtQxmnrgGQ== X-Received: by 2002:a17:90a:974c:b0:28c:accc:b1db with SMTP id i12-20020a17090a974c00b0028cacccb1dbmr2272863pjw.44.1704232900428; Tue, 02 Jan 2024 14:01:40 -0800 (PST) Received: from sw06.internal.sifive.com ([4.53.31.132]) by smtp.gmail.com with ESMTPSA id r59-20020a17090a43c100b0028ce507cd7dsm101724pjg.55.2024.01.02.14.01.39 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 02 Jan 2024 14:01:40 -0800 (PST) From: Samuel Holland To: Palmer Dabbelt , linux-riscv@lists.infradead.org Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, Alexandre Ghiti , Samuel Holland Subject: [PATCH v4 03/12] riscv: mm: Broadcast kernel TLB flushes only when needed Date: Tue, 2 Jan 2024 14:00:40 -0800 Message-ID: <20240102220134.3229156-4-samuel.holland@sifive.com> X-Mailer: git-send-email 2.42.0 In-Reply-To: <20240102220134.3229156-1-samuel.holland@sifive.com> References: <20240102220134.3229156-1-samuel.holland@sifive.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" __flush_tlb_range() avoids broadcasting TLB flushes when an mm context is only active on the local CPU. Apply this same optimization to TLB flushes of kernel memory when only one CPU is online. This check can be constant-folded when SMP is disabled. Signed-off-by: Samuel Holland Reviewed-by: Alexandre Ghiti --- Changes in v4: - New patch for v4 arch/riscv/mm/tlbflush.c | 17 ++++++----------- 1 file changed, 6 insertions(+), 11 deletions(-) diff --git a/arch/riscv/mm/tlbflush.c b/arch/riscv/mm/tlbflush.c index 09b03bf71e6a..2f18fe6fc4f3 100644 --- a/arch/riscv/mm/tlbflush.c +++ b/arch/riscv/mm/tlbflush.c @@ -98,27 +98,23 @@ static void __flush_tlb_range(struct mm_struct *mm, uns= igned long start, { const struct cpumask *cmask; unsigned long asid =3D FLUSH_TLB_NO_ASID; - bool broadcast; + unsigned int cpu; =20 if (mm) { - unsigned int cpuid; - cmask =3D mm_cpumask(mm); if (cpumask_empty(cmask)) return; =20 - cpuid =3D get_cpu(); - /* check if the tlbflush needs to be sent to other CPUs */ - broadcast =3D cpumask_any_but(cmask, cpuid) < nr_cpu_ids; - if (static_branch_unlikely(&use_asid_allocator)) asid =3D atomic_long_read(&mm->context.id) & asid_mask; } else { cmask =3D cpu_online_mask; - broadcast =3D true; } =20 - if (!broadcast) { + cpu =3D get_cpu(); + + /* Check if the TLB flush needs to be sent to other CPUs. */ + if (cpumask_any_but(cmask, cpu) >=3D nr_cpu_ids) { local_flush_tlb_range_asid(start, size, stride, asid); } else if (riscv_use_sbi_for_rfence()) { sbi_remote_sfence_vma_asid(cmask, start, size, asid); @@ -132,8 +128,7 @@ static void __flush_tlb_range(struct mm_struct *mm, uns= igned long start, on_each_cpu_mask(cmask, __ipi_flush_tlb_range_asid, &ftd, 1); } =20 - if (mm) - put_cpu(); + put_cpu(); } =20 void flush_tlb_mm(struct mm_struct *mm) --=20 2.42.0 From nobody Fri Dec 26 17:23:07 2025 Received: from mail-pg1-f170.google.com (mail-pg1-f170.google.com [209.85.215.170]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 15FB418C35 for ; Tue, 2 Jan 2024 22:01:41 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=sifive.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=sifive.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=sifive.com header.i=@sifive.com header.b="DgRqkJoN" Received: by mail-pg1-f170.google.com with SMTP id 41be03b00d2f7-5c229dabbb6so2569289a12.0 for ; Tue, 02 Jan 2024 14:01:41 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1704232901; x=1704837701; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=YDpWqQ4mc+jX/Roh6ocibeJ1LmLlK7pqWIKc4/L4AE8=; b=DgRqkJoN7jmq4C8Mete+CRT6XrLFu/qo41dxk46PW9v6kz+g8QgJvnk6qO7xoHR+JN vm3DhCNlDOJ+DzdkuZPjd3T1BzPLejafjTfOLF9j0KNqj5ghgmvqV1dIWAECjHALDlXz LoMnfwTR1mPwilWyEifFn4whkLsUqDNqlxffLPsZvoNUXWF0ZBs/8p2SUuuj6DPLkbid W26tySgW6sKFChUQYQsKU8fT2ZkvAJFy4sgT6sK/1jihvVJEIOuLwvfcp4/JDOke5ggL ws9zS7AzG4voaTu2urPb1f2YSLkfXuO1DNTURKWFMuR5yrKzBI4NmEtEdoEZq8f0cxZp 0VjQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1704232901; x=1704837701; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=YDpWqQ4mc+jX/Roh6ocibeJ1LmLlK7pqWIKc4/L4AE8=; b=WKqC4CvQFhUZbVjhiOCu5QGJDxhePG1vwLIRK771A9cfW8OiP8/JuwDnbWKuFCx00j UPJVyoaOTyMsZKyXEwC6Pm8evkOSTUAazrFypPTP0qP8N3Aw1IYSOGjX75juwUTqkyaR B/DAIcDDs8SjMVx43FY+uFprarfiOFN2Go1ni0KXQQ7sFvlZZOfykeleVj4QFxlP+7AQ IolhiN891P55EcsCgeccsgSxr/rvD7FyTEv9pQdy5SPqzeuo2azW2/sL6OwZRctgSE/r Sn9c8oXuNn4TlbR4215G0MyFsCZMFDzQWnnncK1E8tZFkUQNR2iFegVe1JNTtYRQP+7j HwCA== X-Gm-Message-State: AOJu0YzkEUSkws0sJRqQAp4MihPl0FEzcQ41BcVUjEZNk0gClqldWGLt Rzabx5d94U8tpAvOu+ice3MOP9xdkfsENCZS4vkccVSGlE0= X-Google-Smtp-Source: AGHT+IGhgK6idMxI6CkoCNiy9Y4cs5Xn9ml4r4H30wFrLDF3Z3r2uMsg0CAIjypKVmYj0k77Mk3d6A== X-Received: by 2002:a17:90a:6ba6:b0:28c:640b:ce0b with SMTP id w35-20020a17090a6ba600b0028c640bce0bmr4536054pjj.33.1704232901387; Tue, 02 Jan 2024 14:01:41 -0800 (PST) Received: from sw06.internal.sifive.com ([4.53.31.132]) by smtp.gmail.com with ESMTPSA id r59-20020a17090a43c100b0028ce507cd7dsm101724pjg.55.2024.01.02.14.01.40 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 02 Jan 2024 14:01:41 -0800 (PST) From: Samuel Holland To: Palmer Dabbelt , linux-riscv@lists.infradead.org Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, Alexandre Ghiti , Samuel Holland Subject: [PATCH v4 04/12] riscv: Only send remote fences when some other CPU is online Date: Tue, 2 Jan 2024 14:00:41 -0800 Message-ID: <20240102220134.3229156-5-samuel.holland@sifive.com> X-Mailer: git-send-email 2.42.0 In-Reply-To: <20240102220134.3229156-1-samuel.holland@sifive.com> References: <20240102220134.3229156-1-samuel.holland@sifive.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" If no other CPU is online, a local cache or TLB flush is sufficient. These checks can be constant-folded when SMP is disabled. Signed-off-by: Samuel Holland --- Changes in v4: - New patch for v4 arch/riscv/mm/cacheflush.c | 4 +++- arch/riscv/mm/tlbflush.c | 4 +++- 2 files changed, 6 insertions(+), 2 deletions(-) diff --git a/arch/riscv/mm/cacheflush.c b/arch/riscv/mm/cacheflush.c index 47c485bc7df0..f7933ae88a55 100644 --- a/arch/riscv/mm/cacheflush.c +++ b/arch/riscv/mm/cacheflush.c @@ -21,7 +21,9 @@ void flush_icache_all(void) { local_flush_icache_all(); =20 - if (riscv_use_sbi_for_rfence()) + if (num_online_cpus() < 2) + return; + else if (riscv_use_sbi_for_rfence()) sbi_remote_fence_i(NULL); else on_each_cpu(ipi_remote_fence_i, NULL, 1); diff --git a/arch/riscv/mm/tlbflush.c b/arch/riscv/mm/tlbflush.c index 2f18fe6fc4f3..37b3c93e3c30 100644 --- a/arch/riscv/mm/tlbflush.c +++ b/arch/riscv/mm/tlbflush.c @@ -73,7 +73,9 @@ static void __ipi_flush_tlb_all(void *info) =20 void flush_tlb_all(void) { - if (riscv_use_sbi_for_rfence()) + if (num_online_cpus() < 2) + local_flush_tlb_all(); + else if (riscv_use_sbi_for_rfence()) sbi_remote_sfence_vma_asid(NULL, 0, FLUSH_TLB_MAX_SIZE, FLUSH_TLB_NO_ASI= D); else on_each_cpu(__ipi_flush_tlb_all, NULL, 1); --=20 2.42.0 From nobody Fri Dec 26 17:23:07 2025 Received: from mail-pj1-f42.google.com (mail-pj1-f42.google.com [209.85.216.42]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E1AEE18E1C for ; Tue, 2 Jan 2024 22:01:42 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=sifive.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=sifive.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=sifive.com header.i=@sifive.com header.b="DAIC4PEH" Received: by mail-pj1-f42.google.com with SMTP id 98e67ed59e1d1-28c0d8dd88bso4065989a91.2 for ; Tue, 02 Jan 2024 14:01:42 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1704232902; x=1704837702; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=LtzHkWRF5gALofCSRDnlYgSzHUyaPaF++esfIcBOkBw=; b=DAIC4PEHXK6zN41yGxbL8WtnEyXB7nYGu49/aORPHuv2xTX+4si3aH4of6n+waXs+n QRpUA/BGpBPkwGoGWxZ5S9NRdjoXPFJNTiGuzXbsM+k4srZHnZZ4pj0iLwfUdSsFcDb6 PGPtfbLTo8KNJ0NVUQiJBf6s7tYRjqyobBZ8SCazTKU4VkIMM1wSSov7ZMAnJVXxqG1q OleZkGQ+YCQ0EkF1344I0kMlE1SjqEsrWAToUpADyIXqPMxJf4vXa6E0By2FNGttP7Zn yUcBaJgG8E4Pe8Hec4TCWqq+sPfy8ngGJ2wXfbAS6M+NeLPZbSJRuw1H728uJlnpPdHI +eyQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1704232902; x=1704837702; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=LtzHkWRF5gALofCSRDnlYgSzHUyaPaF++esfIcBOkBw=; b=I3nbBbO8vE2U/gaedSgg4TVm15v3A5EOz7z75h8f/gxHjBJno+gl5qp7YVXmYl8tU7 HktOFiCu1c80wp6VekZC1cKEdLlyaxq/AkVnDqljQfg0gKAFMKL8x5MEatWOxIPaGgHu w3alK/e75gRQgPe3tw9+Eggc4mrgaWTx2tVxKz18hV1ZkibDRCMW8FQY7Cn74a7z8MxS yp5OXLw0k6IaN/OE+Cnrju1T3zufnuivT5q5yRZrk50/2BKSh10+MLlvqzcj8ONT2aYN pjjE7bp0PaivvNV/VtpcxKLuzRxikUT9jkm0FeGuGglIKN2ezaZWFuX8yrIjk/kXRILP CpcQ== X-Gm-Message-State: AOJu0YxWQW8edkcra5Kaikg6tJZZrybPCprPgyD/msjNaMpJdjOO6ydm XytC8chEGN/AcRb1Dfy0Ueqgv04u/Ch4TA== X-Google-Smtp-Source: AGHT+IExkJSZEsbRdHGUNd8kiI0pICSa1vMynIFUPWzzc2fqMiijvWM2jfH6XllBTjKhdLtvimrjxg== X-Received: by 2002:a17:90a:740a:b0:286:7f0d:6254 with SMTP id a10-20020a17090a740a00b002867f0d6254mr6006468pjg.63.1704232902374; Tue, 02 Jan 2024 14:01:42 -0800 (PST) Received: from sw06.internal.sifive.com ([4.53.31.132]) by smtp.gmail.com with ESMTPSA id r59-20020a17090a43c100b0028ce507cd7dsm101724pjg.55.2024.01.02.14.01.41 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 02 Jan 2024 14:01:42 -0800 (PST) From: Samuel Holland To: Palmer Dabbelt , linux-riscv@lists.infradead.org Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, Alexandre Ghiti , Samuel Holland Subject: [PATCH v4 05/12] riscv: mm: Combine the SMP and UP TLB flush code Date: Tue, 2 Jan 2024 14:00:42 -0800 Message-ID: <20240102220134.3229156-6-samuel.holland@sifive.com> X-Mailer: git-send-email 2.42.0 In-Reply-To: <20240102220134.3229156-1-samuel.holland@sifive.com> References: <20240102220134.3229156-1-samuel.holland@sifive.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" In SMP configurations, all TLB flushing narrower than flush_tlb_all() goes through __flush_tlb_range(). Do the same in UP configurations. This allows UP configurations to take advantage of recent improvements to the code in tlbflush.c, such as support for huge pages and flushing multiple-page ranges. Signed-off-by: Samuel Holland Reviewed-by: Alexandre Ghiti --- Changes in v4: - Merge the two copies of __flush_tlb_range() and rely on the compiler to optimize out the broadcast path (both clang and gcc do this) - Merge the two copies of flush_tlb_all() and rely on constant folding Changes in v2: - Move the SMP/UP merge earlier in the series to avoid build issues - Make a copy of __flush_tlb_range() instead of adding ifdefs inside - local_flush_tlb_all() is the only function used on !MMU (smpboot.c) arch/riscv/include/asm/tlbflush.h | 29 +++-------------------------- arch/riscv/mm/Makefile | 5 +---- 2 files changed, 4 insertions(+), 30 deletions(-) diff --git a/arch/riscv/include/asm/tlbflush.h b/arch/riscv/include/asm/tlb= flush.h index 8f3418c5f172..7712ffe2f6c4 100644 --- a/arch/riscv/include/asm/tlbflush.h +++ b/arch/riscv/include/asm/tlbflush.h @@ -27,12 +27,7 @@ static inline void local_flush_tlb_page(unsigned long ad= dr) { ALT_FLUSH_TLB_PAGE(__asm__ __volatile__ ("sfence.vma %0" : : "r" (addr) := "memory")); } -#else /* CONFIG_MMU */ -#define local_flush_tlb_all() do { } while (0) -#define local_flush_tlb_page(addr) do { } while (0) -#endif /* CONFIG_MMU */ =20 -#if defined(CONFIG_SMP) && defined(CONFIG_MMU) void flush_tlb_all(void); void flush_tlb_mm(struct mm_struct *mm); void flush_tlb_mm_range(struct mm_struct *mm, unsigned long start, @@ -46,26 +41,8 @@ void flush_tlb_kernel_range(unsigned long start, unsigne= d long end); void flush_pmd_tlb_range(struct vm_area_struct *vma, unsigned long start, unsigned long end); #endif -#else /* CONFIG_SMP && CONFIG_MMU */ - -#define flush_tlb_all() local_flush_tlb_all() -#define flush_tlb_page(vma, addr) local_flush_tlb_page(addr) - -static inline void flush_tlb_range(struct vm_area_struct *vma, - unsigned long start, unsigned long end) -{ - local_flush_tlb_all(); -} - -/* Flush a range of kernel pages */ -static inline void flush_tlb_kernel_range(unsigned long start, - unsigned long end) -{ - local_flush_tlb_all(); -} - -#define flush_tlb_mm(mm) flush_tlb_all() -#define flush_tlb_mm_range(mm, start, end, page_size) flush_tlb_all() -#endif /* !CONFIG_SMP || !CONFIG_MMU */ +#else /* CONFIG_MMU */ +#define local_flush_tlb_all() do { } while (0) +#endif /* CONFIG_MMU */ =20 #endif /* _ASM_RISCV_TLBFLUSH_H */ diff --git a/arch/riscv/mm/Makefile b/arch/riscv/mm/Makefile index 3a4dfc8babcf..96e65c571ce8 100644 --- a/arch/riscv/mm/Makefile +++ b/arch/riscv/mm/Makefile @@ -13,15 +13,12 @@ endif KCOV_INSTRUMENT_init.o :=3D n =20 obj-y +=3D init.o -obj-$(CONFIG_MMU) +=3D extable.o fault.o pageattr.o +obj-$(CONFIG_MMU) +=3D extable.o fault.o pageattr.o tlbflush.o obj-y +=3D cacheflush.o obj-y +=3D context.o obj-y +=3D pgtable.o obj-y +=3D pmem.o =20 -ifeq ($(CONFIG_MMU),y) -obj-$(CONFIG_SMP) +=3D tlbflush.o -endif obj-$(CONFIG_HUGETLB_PAGE) +=3D hugetlbpage.o obj-$(CONFIG_PTDUMP_CORE) +=3D ptdump.o obj-$(CONFIG_KASAN) +=3D kasan_init.o --=20 2.42.0 From nobody Fri Dec 26 17:23:07 2025 Received: from mail-pg1-f178.google.com (mail-pg1-f178.google.com [209.85.215.178]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 07A7418EA1 for ; Tue, 2 Jan 2024 22:01:43 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=sifive.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=sifive.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=sifive.com header.i=@sifive.com header.b="HP1wacbR" Received: by mail-pg1-f178.google.com with SMTP id 41be03b00d2f7-53fa455cd94so2562418a12.2 for ; Tue, 02 Jan 2024 14:01:43 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1704232903; x=1704837703; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=6c+wYTacIgINHvokNTIzxqzbFxSPL5G6VDrdebR45nU=; b=HP1wacbRDgJCzBv7AGUbFOLF9b1VWCqm1Dfy4j8RUI11g4HsIFZaPeN2dsxBhW64Oh HMp2yVVjhGcS3SVOsqJn6mI9NV+3gzBTAlo9QY9MowPlpIM0mq4/ft/JxoFh3xZ5Mrf7 EiR0pvh1HashwVTzQI5rlrk8FQVzG56V9eVwCjuMN2hsAS4Eqab5Rq+0S67UoBSD3tKr krCs+qnAjt16JeI5rs0ovGOpNiE5LlV/wNvluwlCzV78Ara8ksKlyKMDKmS/MuC/E7KS 6Y6vpwcTtP08CLaQ0EYhOk4N1nhM8ZGxMa1NLVETUNdPSEvD0WUXaaalr0VOCo88aPgU 5NZQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1704232903; x=1704837703; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=6c+wYTacIgINHvokNTIzxqzbFxSPL5G6VDrdebR45nU=; b=bElJ4hXujPMdCM+Feux5OoLSe3nZ7Ed12G60a4UM/o3+Dgh8uKVCPFoRRkWDtr3toh xul84wcIDyVZxhTRbv50gxomQFc2T0NV655IY2dRGfT0oY2pHJ1roErFcoFOw/Ne/7+L 30sctMeUCSqmPtj47GtWUSvqxJCcqoK0bGb5feOldS19cE78HmkqBAAHydX6Kxxjy4LC lPPKDow7FedXQTvwIObgiCAcpupLy/150T4GfV62WdU3s+XYHd8lrcbbcUhmghGALJRz JWi/JQ+q6cPgd5xR/9lt0ksA293Utr46Ve3kMop52cN6ryenzgNaqXP8WcuOexH5PKCE UZnA== X-Gm-Message-State: AOJu0YxMRd9rULtoF14/eQKo2By0JS3q9kmyNnKDW0vfaCkUFi3CzXj7 r3oIJiAECOY7d/9xfRJH+jKvGByx+eH1qQ== X-Google-Smtp-Source: AGHT+IE2GdHLRUcW4ROKdXldqRiWcx2M7fdl7/RqRT5d/m4h5bCpV21xGk3IklfQQqw3IhSZ5sLqag== X-Received: by 2002:a17:902:d584:b0:1d4:5f9:d3d7 with SMTP id k4-20020a170902d58400b001d405f9d3d7mr8348439plh.127.1704232903384; Tue, 02 Jan 2024 14:01:43 -0800 (PST) Received: from sw06.internal.sifive.com ([4.53.31.132]) by smtp.gmail.com with ESMTPSA id r59-20020a17090a43c100b0028ce507cd7dsm101724pjg.55.2024.01.02.14.01.42 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 02 Jan 2024 14:01:43 -0800 (PST) From: Samuel Holland To: Palmer Dabbelt , linux-riscv@lists.infradead.org Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, Alexandre Ghiti , Samuel Holland Subject: [PATCH v4 06/12] riscv: Apply SiFive CIP-1200 workaround to single-ASID sfence.vma Date: Tue, 2 Jan 2024 14:00:43 -0800 Message-ID: <20240102220134.3229156-7-samuel.holland@sifive.com> X-Mailer: git-send-email 2.42.0 In-Reply-To: <20240102220134.3229156-1-samuel.holland@sifive.com> References: <20240102220134.3229156-1-samuel.holland@sifive.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" commit 3f1e782998cd ("riscv: add ASID-based tlbflushing methods") added calls to the sfence.vma instruction with rs2 !=3D x0. These single-ASID instruction variants are also affected by SiFive errata CIP-1200. Until now, the errata workaround was not needed for the single-ASID sfence.vma variants, because they were only used when the ASID allocator was enabled, and the affected SiFive platforms do not support multiple ASIDs. However, we are going to start using those sfence.vma variants regardless of ASID support, so now we need alternatives covering them. Signed-off-by: Samuel Holland --- (no changes since v2) Changes in v2: - Rebase on Alexandre's "riscv: tlb flush improvements" series v5 arch/riscv/include/asm/errata_list.h | 12 +++++++++++- arch/riscv/include/asm/tlbflush.h | 19 ++++++++++++++++++- arch/riscv/mm/tlbflush.c | 23 ----------------------- 3 files changed, 29 insertions(+), 25 deletions(-) diff --git a/arch/riscv/include/asm/errata_list.h b/arch/riscv/include/asm/= errata_list.h index 83ed25e43553..6781460ae564 100644 --- a/arch/riscv/include/asm/errata_list.h +++ b/arch/riscv/include/asm/errata_list.h @@ -44,11 +44,21 @@ ALTERNATIVE(__stringify(RISCV_PTR do_page_fault), \ CONFIG_ERRATA_SIFIVE_CIP_453) #else /* !__ASSEMBLY__ */ =20 -#define ALT_FLUSH_TLB_PAGE(x) \ +#define ALT_SFENCE_VMA_ASID(asid) \ +asm(ALTERNATIVE("sfence.vma x0, %0", "sfence.vma", SIFIVE_VENDOR_ID, \ + ERRATA_SIFIVE_CIP_1200, CONFIG_ERRATA_SIFIVE_CIP_1200) \ + : : "r" (asid) : "memory") + +#define ALT_SFENCE_VMA_ADDR(addr) \ asm(ALTERNATIVE("sfence.vma %0", "sfence.vma", SIFIVE_VENDOR_ID, \ ERRATA_SIFIVE_CIP_1200, CONFIG_ERRATA_SIFIVE_CIP_1200) \ : : "r" (addr) : "memory") =20 +#define ALT_SFENCE_VMA_ADDR_ASID(addr, asid) \ +asm(ALTERNATIVE("sfence.vma %0, %1", "sfence.vma", SIFIVE_VENDOR_ID, \ + ERRATA_SIFIVE_CIP_1200, CONFIG_ERRATA_SIFIVE_CIP_1200) \ + : : "r" (addr), "r" (asid) : "memory") + /* * _val is marked as "will be overwritten", so need to set it to 0 * in the default case. diff --git a/arch/riscv/include/asm/tlbflush.h b/arch/riscv/include/asm/tlb= flush.h index 7712ffe2f6c4..002c4c2620f3 100644 --- a/arch/riscv/include/asm/tlbflush.h +++ b/arch/riscv/include/asm/tlbflush.h @@ -22,10 +22,27 @@ static inline void local_flush_tlb_all(void) __asm__ __volatile__ ("sfence.vma" : : : "memory"); } =20 +static inline void local_flush_tlb_all_asid(unsigned long asid) +{ + if (asid !=3D FLUSH_TLB_NO_ASID) + ALT_SFENCE_VMA_ASID(asid); + else + local_flush_tlb_all(); +} + /* Flush one page from local TLB */ static inline void local_flush_tlb_page(unsigned long addr) { - ALT_FLUSH_TLB_PAGE(__asm__ __volatile__ ("sfence.vma %0" : : "r" (addr) := "memory")); + ALT_SFENCE_VMA_ADDR(addr); +} + +static inline void local_flush_tlb_page_asid(unsigned long addr, + unsigned long asid) +{ + if (asid !=3D FLUSH_TLB_NO_ASID) + ALT_SFENCE_VMA_ADDR_ASID(addr, asid); + else + local_flush_tlb_page(addr); } =20 void flush_tlb_all(void); diff --git a/arch/riscv/mm/tlbflush.c b/arch/riscv/mm/tlbflush.c index 37b3c93e3c30..292d7cf3c4f6 100644 --- a/arch/riscv/mm/tlbflush.c +++ b/arch/riscv/mm/tlbflush.c @@ -7,29 +7,6 @@ #include #include =20 -static inline void local_flush_tlb_all_asid(unsigned long asid) -{ - if (asid !=3D FLUSH_TLB_NO_ASID) - __asm__ __volatile__ ("sfence.vma x0, %0" - : - : "r" (asid) - : "memory"); - else - local_flush_tlb_all(); -} - -static inline void local_flush_tlb_page_asid(unsigned long addr, - unsigned long asid) -{ - if (asid !=3D FLUSH_TLB_NO_ASID) - __asm__ __volatile__ ("sfence.vma %0, %1" - : - : "r" (addr), "r" (asid) - : "memory"); - else - local_flush_tlb_page(addr); -} - /* * Flush entire TLB if number of entries to be flushed is greater * than the threshold below. --=20 2.42.0 From nobody Fri Dec 26 17:23:07 2025 Received: from mail-pj1-f50.google.com (mail-pj1-f50.google.com [209.85.216.50]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E460D18EC7 for ; Tue, 2 Jan 2024 22:01:44 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=sifive.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=sifive.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=sifive.com header.i=@sifive.com header.b="TP2bfssn" Received: by mail-pj1-f50.google.com with SMTP id 98e67ed59e1d1-28c0df4b42eso5243380a91.1 for ; Tue, 02 Jan 2024 14:01:44 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1704232904; x=1704837704; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=1tIOva8KFaIwh3i+98q+ysPtJyT8vpZl2kFy86xHGzc=; b=TP2bfssnGxqJPMtzP4cLt7lxFe+FsXfD55VmnK823h/myUpbpnj3vf8GqR5ZQ0yfqI 6dnsPojh9MFPy8oWLimIWNgwCwX8DLcAZWmnHp5zdTI+jBQJHMFTBB7kgxMunmrYGCrr Rtu9tCidaSaWo7IRhQ5giGKYdxsnxoFFYRwOGG/2CmjILC910XSkSXvUDNE8t8tALJfN l7vUcJyOsFKIls+6W1a9xmQkpLt2lRR+zzPjypvne2gz24wELloosE3mIuyHI6Al+DJd VQqWeY0Bc4PDtMOHsnZ8TR01/wQ5Oy50fDyWDo6aGWbbt9IAha9EjAgawjrWtTKglPsN Ik9g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1704232904; x=1704837704; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=1tIOva8KFaIwh3i+98q+ysPtJyT8vpZl2kFy86xHGzc=; b=vqq2oxBRzE+LVNo7nej4fgOORmZ8IKBiuyGkRMYOJy4zlilKaOOI4lr7I5OlOvQH5U FheWZSIw8X8n0gRsx3Dy72+9F5gllC3OcsFlWw6RjC/HVv4vVEti+dEcPWnGOlItNlhl BxJ/dB0lOYqy18FmCqaU3WJJ7+FDfMf96oz2V72IVh1tJ59u2m60qhl5sYx3sTczroJF av5MzzrV3bFop9WReXqLI2brp6R/htpapKqaxJBRoXkwGWHJR2fjANy4xPiDMAtSskTP uk1FiWJa7lX9JGC0H3lmrulAh1mXFRRwegKd3uWG7c1ZeZgZQMqvV2Ad2Rmx3jYb6tuR r44w== X-Gm-Message-State: AOJu0Yy1hJb7pv2KXMqF95R8vRUFT4AprjeQihesJIQCo9gRq8cAgfB1 YS5PzqhN8Bgx6Hzo9ehY9nVMFI8xYNmXyQ== X-Google-Smtp-Source: AGHT+IERqoXqHCzKpOJ5RqmrnOEOyFJpME6yqKz2yoKuSVbxPLF5GZg8UdvpqpiYsRMOZbO3DMheoA== X-Received: by 2002:a17:90a:d90b:b0:28c:300d:d858 with SMTP id c11-20020a17090ad90b00b0028c300dd858mr127429pjv.20.1704232904383; Tue, 02 Jan 2024 14:01:44 -0800 (PST) Received: from sw06.internal.sifive.com ([4.53.31.132]) by smtp.gmail.com with ESMTPSA id r59-20020a17090a43c100b0028ce507cd7dsm101724pjg.55.2024.01.02.14.01.43 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 02 Jan 2024 14:01:44 -0800 (PST) From: Samuel Holland To: Palmer Dabbelt , linux-riscv@lists.infradead.org Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, Alexandre Ghiti , Samuel Holland Subject: [PATCH v4 07/12] riscv: Avoid TLB flush loops when affected by SiFive CIP-1200 Date: Tue, 2 Jan 2024 14:00:44 -0800 Message-ID: <20240102220134.3229156-8-samuel.holland@sifive.com> X-Mailer: git-send-email 2.42.0 In-Reply-To: <20240102220134.3229156-1-samuel.holland@sifive.com> References: <20240102220134.3229156-1-samuel.holland@sifive.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Since implementations affected by SiFive errata CIP-1200 always use the global variant of the sfence.vma instruction, they only need to execute the instruction once. The range-based loop only hurts performance. Signed-off-by: Samuel Holland --- Changes in v4: - Only set tlb_flush_all_threshold when CONFIG_MMU=3Dy. Changes in v3: - New patch for v3 arch/riscv/errata/sifive/errata.c | 5 +++++ arch/riscv/include/asm/tlbflush.h | 2 ++ arch/riscv/mm/tlbflush.c | 2 +- 3 files changed, 8 insertions(+), 1 deletion(-) diff --git a/arch/riscv/errata/sifive/errata.c b/arch/riscv/errata/sifive/e= rrata.c index 3d9a32d791f7..716cfedad3a2 100644 --- a/arch/riscv/errata/sifive/errata.c +++ b/arch/riscv/errata/sifive/errata.c @@ -42,6 +42,11 @@ static bool errata_cip_1200_check_func(unsigned long ar= ch_id, unsigned long imp return false; if ((impid & 0xffffff) > 0x200630 || impid =3D=3D 0x1200626) return false; + +#ifdef CONFIG_MMU + tlb_flush_all_threshold =3D 0; +#endif + return true; } =20 diff --git a/arch/riscv/include/asm/tlbflush.h b/arch/riscv/include/asm/tlb= flush.h index 002c4c2620f3..d9913590f82e 100644 --- a/arch/riscv/include/asm/tlbflush.h +++ b/arch/riscv/include/asm/tlbflush.h @@ -58,6 +58,8 @@ void flush_tlb_kernel_range(unsigned long start, unsigned= long end); void flush_pmd_tlb_range(struct vm_area_struct *vma, unsigned long start, unsigned long end); #endif + +extern unsigned long tlb_flush_all_threshold; #else /* CONFIG_MMU */ #define local_flush_tlb_all() do { } while (0) #endif /* CONFIG_MMU */ diff --git a/arch/riscv/mm/tlbflush.c b/arch/riscv/mm/tlbflush.c index 292d7cf3c4f6..76b24d4ed4ab 100644 --- a/arch/riscv/mm/tlbflush.c +++ b/arch/riscv/mm/tlbflush.c @@ -11,7 +11,7 @@ * Flush entire TLB if number of entries to be flushed is greater * than the threshold below. */ -static unsigned long tlb_flush_all_threshold __read_mostly =3D 64; +unsigned long tlb_flush_all_threshold __read_mostly =3D 64; =20 static void local_flush_tlb_range_threshold_asid(unsigned long start, unsigned long size, --=20 2.42.0 From nobody Fri Dec 26 17:23:07 2025 Received: from mail-pg1-f171.google.com (mail-pg1-f171.google.com [209.85.215.171]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E9E8619471 for ; Tue, 2 Jan 2024 22:01:45 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=sifive.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=sifive.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=sifive.com header.i=@sifive.com header.b="bfZ/XZFv" Received: by mail-pg1-f171.google.com with SMTP id 41be03b00d2f7-5ce2170b716so4730641a12.1 for ; Tue, 02 Jan 2024 14:01:45 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1704232905; x=1704837705; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=d7sRColwM16ZDpkHsKhbx/H1UjX1V2ZP/gk6hKZVk/k=; b=bfZ/XZFvnuIlbRaPhiMl3TU27P/xztx2cS1Kj3kl7/uEFDQZZurqnQ7s5bw2wKOnts EnnRARcDRCwsp1t0lPh/w3kOVUQ4wm3Hx6w2VbuBDZLSO1tJqkqgm65OVRk3jsXb4J5K 3bp3foZ24OKfppgCAsSRyyj0zx1SMWtrD/sdH7EgHQdwKNgI+AKhKQDeL0lBLjBnALX8 05mhfIeLGkoP519SliEsfudgx3id0R2pYdMK8dumUCCU9u33jNZGrvKewxWtDJ6vx4Mt 9KI+Cy0mY9j6YvYVbRnNgty3tyzrf75iSVY6xQwb+4cbzVDAlWs9nh5890ZpAKZSIvSB Fx9w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1704232905; x=1704837705; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=d7sRColwM16ZDpkHsKhbx/H1UjX1V2ZP/gk6hKZVk/k=; b=GYHt4aHlJpW+vD6HieHAqP160JSB2b8xLci43pfadxe4Exd/2lxcVaIaoFei2kxqAO earJhVHLT8SVGYWzpytPjOPlAdk50o0QLI/EE91sV93SGHhUZKlUQ+3CVZ0WtPgPjnPI kTqw8t2SRxMLoJ4SFe3ichrk1qipcvOsOgUksQelpRH8xYTEEu3s2ebv0zvnr6c3zY9I mHJFpNuH6l7qQi1XHMeUTXNJ6qXi2PcdzbYuc2m51ViA/b3kYS8aOrAe8b15cvl4FSr2 ZXOrz1LbVZ5H3dp0f4swlEwCvJ7cmNYMDWPmoBjtksDNvJtdql80zISiUrK60IUmFSWT t4OA== X-Gm-Message-State: AOJu0YwHMJpX88ccFdCAwTO9ta2UntQQsaIisRTbZFwMcwc8EOw4NlTp LcLMyK+VQetkz2ZH5ShSo4flF+QvkfrT7ft940nhVjaDtoU= X-Google-Smtp-Source: AGHT+IEo3bZM2aLuSMOGwMaVus9GR4XwWujHLJvE5N9ecURPZ+yptZqxQ7nXUllBGYAXlfndapu+WA== X-Received: by 2002:a05:6a20:a107:b0:197:2fc0:7e25 with SMTP id q7-20020a056a20a10700b001972fc07e25mr2866566pzk.3.1704232905452; Tue, 02 Jan 2024 14:01:45 -0800 (PST) Received: from sw06.internal.sifive.com ([4.53.31.132]) by smtp.gmail.com with ESMTPSA id r59-20020a17090a43c100b0028ce507cd7dsm101724pjg.55.2024.01.02.14.01.44 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 02 Jan 2024 14:01:45 -0800 (PST) From: Samuel Holland To: Palmer Dabbelt , linux-riscv@lists.infradead.org Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, Alexandre Ghiti , Samuel Holland Subject: [PATCH v4 08/12] riscv: mm: Introduce cntx2asid/cntx2version helper macros Date: Tue, 2 Jan 2024 14:00:45 -0800 Message-ID: <20240102220134.3229156-9-samuel.holland@sifive.com> X-Mailer: git-send-email 2.42.0 In-Reply-To: <20240102220134.3229156-1-samuel.holland@sifive.com> References: <20240102220134.3229156-1-samuel.holland@sifive.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" When using the ASID allocator, the MM context ID contains two values: the ASID in the lower bits, and the allocator version number in the remaining bits. Use macros to make this separation more obvious. Signed-off-by: Samuel Holland Reviewed-by: Alexandre Ghiti --- (no changes since v1) arch/riscv/include/asm/mmu.h | 3 +++ arch/riscv/mm/context.c | 12 ++++++------ arch/riscv/mm/tlbflush.c | 2 +- 3 files changed, 10 insertions(+), 7 deletions(-) diff --git a/arch/riscv/include/asm/mmu.h b/arch/riscv/include/asm/mmu.h index 355504b37f8e..a550fbf770be 100644 --- a/arch/riscv/include/asm/mmu.h +++ b/arch/riscv/include/asm/mmu.h @@ -26,6 +26,9 @@ typedef struct { #endif } mm_context_t; =20 +#define cntx2asid(cntx) ((cntx) & asid_mask) +#define cntx2version(cntx) ((cntx) & ~asid_mask) + void __init create_pgd_mapping(pgd_t *pgdp, uintptr_t va, phys_addr_t pa, phys_addr_t sz, pgprot_t prot); #endif /* __ASSEMBLY__ */ diff --git a/arch/riscv/mm/context.c b/arch/riscv/mm/context.c index 217fd4de6134..43d005f63253 100644 --- a/arch/riscv/mm/context.c +++ b/arch/riscv/mm/context.c @@ -81,7 +81,7 @@ static void __flush_context(void) if (cntx =3D=3D 0) cntx =3D per_cpu(reserved_context, i); =20 - __set_bit(cntx & asid_mask, context_asid_map); + __set_bit(cntx2asid(cntx), context_asid_map); per_cpu(reserved_context, i) =3D cntx; } =20 @@ -102,7 +102,7 @@ static unsigned long __new_context(struct mm_struct *mm) lockdep_assert_held(&context_lock); =20 if (cntx !=3D 0) { - unsigned long newcntx =3D ver | (cntx & asid_mask); + unsigned long newcntx =3D ver | cntx2asid(cntx); =20 /* * If our current CONTEXT was active during a rollover, we @@ -115,7 +115,7 @@ static unsigned long __new_context(struct mm_struct *mm) * We had a valid CONTEXT in a previous life, so try to * re-use it if possible. */ - if (!__test_and_set_bit(cntx & asid_mask, context_asid_map)) + if (!__test_and_set_bit(cntx2asid(cntx), context_asid_map)) return newcntx; } =20 @@ -168,7 +168,7 @@ static void set_mm_asid(struct mm_struct *mm, unsigned = int cpu) */ old_active_cntx =3D atomic_long_read(&per_cpu(active_context, cpu)); if (old_active_cntx && - ((cntx & ~asid_mask) =3D=3D atomic_long_read(¤t_version)) && + (cntx2version(cntx) =3D=3D atomic_long_read(¤t_version)) && atomic_long_cmpxchg_relaxed(&per_cpu(active_context, cpu), old_active_cntx, cntx)) goto switch_mm_fast; @@ -177,7 +177,7 @@ static void set_mm_asid(struct mm_struct *mm, unsigned = int cpu) =20 /* Check that our ASID belongs to the current_version. */ cntx =3D atomic_long_read(&mm->context.id); - if ((cntx & ~asid_mask) !=3D atomic_long_read(¤t_version)) { + if (cntx2version(cntx) !=3D atomic_long_read(¤t_version)) { cntx =3D __new_context(mm); atomic_long_set(&mm->context.id, cntx); } @@ -191,7 +191,7 @@ static void set_mm_asid(struct mm_struct *mm, unsigned = int cpu) =20 switch_mm_fast: csr_write(CSR_SATP, virt_to_pfn(mm->pgd) | - ((cntx & asid_mask) << SATP_ASID_SHIFT) | + (cntx2asid(cntx) << SATP_ASID_SHIFT) | satp_mode); =20 if (need_flush_tlb) diff --git a/arch/riscv/mm/tlbflush.c b/arch/riscv/mm/tlbflush.c index 76b24d4ed4ab..5ec621545c69 100644 --- a/arch/riscv/mm/tlbflush.c +++ b/arch/riscv/mm/tlbflush.c @@ -85,7 +85,7 @@ static void __flush_tlb_range(struct mm_struct *mm, unsig= ned long start, return; =20 if (static_branch_unlikely(&use_asid_allocator)) - asid =3D atomic_long_read(&mm->context.id) & asid_mask; + asid =3D cntx2asid(atomic_long_read(&mm->context.id)); } else { cmask =3D cpu_online_mask; } --=20 2.42.0 From nobody Fri Dec 26 17:23:07 2025 Received: from mail-pj1-f49.google.com (mail-pj1-f49.google.com [209.85.216.49]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1AB461947A for ; Tue, 2 Jan 2024 22:01:46 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=sifive.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=sifive.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=sifive.com header.i=@sifive.com header.b="NKqqWxPa" Received: by mail-pj1-f49.google.com with SMTP id 98e67ed59e1d1-28bc7155755so3952296a91.2 for ; Tue, 02 Jan 2024 14:01:46 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1704232906; x=1704837706; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=g2cmoIK/tf9yPohv/YvO5QQvGunkgXtONjIGDQZEX58=; b=NKqqWxPaj9rcV1IWAEDEU5QgBO1QwiNOgePEN9zSk7nzNbpWa44amkz2NbAO6NLdHJ MLMvcUQ4n3FWjNLww45EzULW9lgASED1bsOKehFf9/sLOsXYlIsRRPLlMWQPAzz3pndL N3I9aLOTxJ+0sXdLCvpQlk6IuLCik+2236voMxhb8/Yz1qGbOIdAK34fcs8SFsBFW5VT xtncr81K2RPu27HOCTYsAS1XUumIxuNDtA78pj/zp/T45YydmbEu2OfyEOxCJ9S1a79i BdTjkNPY9HL72mdjFvJq4nMloRpUIfk37237cZfmt+vARzPch6FZxki2j+JN9RFiJmL8 u8+Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1704232906; x=1704837706; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=g2cmoIK/tf9yPohv/YvO5QQvGunkgXtONjIGDQZEX58=; b=g8Px5+hBA5tqkHUcGnjR9fbo46CrvGxSqphTEixYCDDbrzjj1hsJa1bgQ9y8syRDEA oEABHh3/UEyX/6+MwMmKuNEfTUxXY8ipUSMnjAL1O9lne2uke6DAP9LOx0B6ITyoGr6R 3v8WWMfjCEpCkiVtjKJaARVaS8BDZwSxaOmelm1urKlqrDYnNxgH8H0/0sLaPIgnMoaJ s4T/+agPjWY+r+ne2+whoeHY6znjjNMwQzmo+BLT9YD0ad8FbpvWsTPNKI1+G1LZlvlc kzppI4BuycbVXGDfyXe1IC9CrmeW80VaipE2QThMweFL0hTSOcUgUGMEBEIiBKL9zq1C 4wAw== X-Gm-Message-State: AOJu0YzeX87//xDNABoytTYrZxNjDKgMmTXNCFCH+rxPpa3QuEjqfGJW ueRBvoI4y/AzGxHov0W/lfjSDaeBUU/EOw== X-Google-Smtp-Source: AGHT+IG8w/0THQ/uqOPivBsiw22hWysSlz1giatwLj9QOMmwVtysp2KaHolkndd/R0ziY6DtdT9KEQ== X-Received: by 2002:a17:90a:134f:b0:28c:b1bd:3c42 with SMTP id y15-20020a17090a134f00b0028cb1bd3c42mr2130687pjf.55.1704232906433; Tue, 02 Jan 2024 14:01:46 -0800 (PST) Received: from sw06.internal.sifive.com ([4.53.31.132]) by smtp.gmail.com with ESMTPSA id r59-20020a17090a43c100b0028ce507cd7dsm101724pjg.55.2024.01.02.14.01.45 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 02 Jan 2024 14:01:46 -0800 (PST) From: Samuel Holland To: Palmer Dabbelt , linux-riscv@lists.infradead.org Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, Alexandre Ghiti , Samuel Holland Subject: [PATCH v4 09/12] riscv: mm: Use a fixed layout for the MM context ID Date: Tue, 2 Jan 2024 14:00:46 -0800 Message-ID: <20240102220134.3229156-10-samuel.holland@sifive.com> X-Mailer: git-send-email 2.42.0 In-Reply-To: <20240102220134.3229156-1-samuel.holland@sifive.com> References: <20240102220134.3229156-1-samuel.holland@sifive.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Currently, the size of the ASID field in the MM context ID dynamically depends on the number of hardware-supported ASID bits. This requires reading a global variable to extract either field from the context ID. Instead, allocate the maximum possible number of bits to the ASID field, so the layout of the context ID is known at compile-time. Signed-off-by: Samuel Holland Reviewed-by: Alexandre Ghiti --- (no changes since v1) arch/riscv/include/asm/mmu.h | 4 ++-- arch/riscv/include/asm/tlbflush.h | 2 -- arch/riscv/mm/context.c | 6 ++---- 3 files changed, 4 insertions(+), 8 deletions(-) diff --git a/arch/riscv/include/asm/mmu.h b/arch/riscv/include/asm/mmu.h index a550fbf770be..dc0273f7905f 100644 --- a/arch/riscv/include/asm/mmu.h +++ b/arch/riscv/include/asm/mmu.h @@ -26,8 +26,8 @@ typedef struct { #endif } mm_context_t; =20 -#define cntx2asid(cntx) ((cntx) & asid_mask) -#define cntx2version(cntx) ((cntx) & ~asid_mask) +#define cntx2asid(cntx) ((cntx) & SATP_ASID_MASK) +#define cntx2version(cntx) ((cntx) & ~SATP_ASID_MASK) =20 void __init create_pgd_mapping(pgd_t *pgdp, uintptr_t va, phys_addr_t pa, phys_addr_t sz, pgprot_t prot); diff --git a/arch/riscv/include/asm/tlbflush.h b/arch/riscv/include/asm/tlb= flush.h index d9913590f82e..5bfd37cfd8c3 100644 --- a/arch/riscv/include/asm/tlbflush.h +++ b/arch/riscv/include/asm/tlbflush.h @@ -15,8 +15,6 @@ #define FLUSH_TLB_NO_ASID ((unsigned long)-1) =20 #ifdef CONFIG_MMU -extern unsigned long asid_mask; - static inline void local_flush_tlb_all(void) { __asm__ __volatile__ ("sfence.vma" : : : "memory"); diff --git a/arch/riscv/mm/context.c b/arch/riscv/mm/context.c index 43d005f63253..b5170ac1b742 100644 --- a/arch/riscv/mm/context.c +++ b/arch/riscv/mm/context.c @@ -22,7 +22,6 @@ DEFINE_STATIC_KEY_FALSE(use_asid_allocator); =20 static unsigned long asid_bits; static unsigned long num_asids; -unsigned long asid_mask; =20 static atomic_long_t current_version; =20 @@ -128,7 +127,7 @@ static unsigned long __new_context(struct mm_struct *mm) goto set_asid; =20 /* We're out of ASIDs, so increment current_version */ - ver =3D atomic_long_add_return_relaxed(num_asids, ¤t_version); + ver =3D atomic_long_add_return_relaxed(BIT(SATP_ASID_BITS), ¤t_vers= ion); =20 /* Flush everything */ __flush_context(); @@ -247,7 +246,6 @@ static int __init asids_init(void) /* Pre-compute ASID details */ if (asid_bits) { num_asids =3D 1 << asid_bits; - asid_mask =3D num_asids - 1; } =20 /* @@ -255,7 +253,7 @@ static int __init asids_init(void) * at-least twice more than CPUs */ if (num_asids > (2 * num_possible_cpus())) { - atomic_long_set(¤t_version, num_asids); + atomic_long_set(¤t_version, BIT(SATP_ASID_BITS)); =20 context_asid_map =3D bitmap_zalloc(num_asids, GFP_KERNEL); if (!context_asid_map) --=20 2.42.0 From nobody Fri Dec 26 17:23:07 2025 Received: from mail-pj1-f53.google.com (mail-pj1-f53.google.com [209.85.216.53]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E7B6F199B3 for ; Tue, 2 Jan 2024 22:01:47 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=sifive.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=sifive.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=sifive.com header.i=@sifive.com header.b="Fq3hbOFI" Received: by mail-pj1-f53.google.com with SMTP id 98e67ed59e1d1-28bc7155755so3952305a91.2 for ; Tue, 02 Jan 2024 14:01:47 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1704232907; x=1704837707; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=O8uQuAx5HbeGrqX4Ip/qQY7nv6lrxw4qacCJFbpKbko=; b=Fq3hbOFITwhrCF1xo3JKafxvDiY2hTS9MG3k7e4U6FsfOrftpP2sgWuYYotBePj36L r7sCj50iDICiKIuU1qwmr09yvMpY20VBxLYXG6yxBqvnvwQPjeC0V7wW6LpAEulpXfiE v3Rz8ObmvE59Bu3mCceu1lmwgUwSHxmcM3KheDBO2Bmz2pZ00PTqqqC/NBcNH2AGHXrt ND4lYHzancP0+KLfkwpx7ENF8Yp1DWN4O9hCWx2N3pse78IR1KJZszpuBAtog0GqQ1M9 IIHzSvnEP9FlF50k0LH9O7g8vqLSyjk/7xHMrW4TFcp5v9AsH5gaJ73h2GIzwZb6UHoj vzfw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1704232907; x=1704837707; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=O8uQuAx5HbeGrqX4Ip/qQY7nv6lrxw4qacCJFbpKbko=; b=aiYRk/sAsIbXlwN2X6acgr4GekvqfPejipnlsTraEbESE0JsuRhKjP0ZfgNPH2pLId JrkrBroK4KvwCSNpSBrQnZqYA87pR8VldWoNc2fLvmcSHwbjwZT1UgOxWYlOhCg55zpc AKFnPTLXuCopmurQVyTePWO49C6GkqsNWQteuJtntoRB3gwVHzoHQOGgmytlwe5LIx9w H2RfUP+3iZuc/eXehSonT98k13J/ezCNWZRavWpv+KLnYDePJO5TKZ6LNI+xtte113Gx XxtlAe7n53zjVhIauTU7UpQUwcuOt+AXAWMTpScjBtNX07uE8Vq4H3cpnQruOiE/CzDe b5oA== X-Gm-Message-State: AOJu0YwC0s7Hbc9S/XXeGyZTNRlbMZC3fMjmmPwJn6M40U5leNSMeRdm zPbu7CAwxVoD1RAW4bNWdS1sVjbfi1EiUkgbFEq0z2pD/Zs= X-Google-Smtp-Source: AGHT+IEU78MnSVmG+tTKoY5SsP+KmgJHfZS8cB0cAVh744mS+tR8J0bWreXJl70FWHY5n2NbigpZXg== X-Received: by 2002:a17:90b:33cc:b0:28a:f2cc:e008 with SMTP id lk12-20020a17090b33cc00b0028af2cce008mr6619129pjb.54.1704232907387; Tue, 02 Jan 2024 14:01:47 -0800 (PST) Received: from sw06.internal.sifive.com ([4.53.31.132]) by smtp.gmail.com with ESMTPSA id r59-20020a17090a43c100b0028ce507cd7dsm101724pjg.55.2024.01.02.14.01.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 02 Jan 2024 14:01:47 -0800 (PST) From: Samuel Holland To: Palmer Dabbelt , linux-riscv@lists.infradead.org Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, Alexandre Ghiti , Samuel Holland Subject: [PATCH v4 10/12] riscv: mm: Make asid_bits a local variable Date: Tue, 2 Jan 2024 14:00:47 -0800 Message-ID: <20240102220134.3229156-11-samuel.holland@sifive.com> X-Mailer: git-send-email 2.42.0 In-Reply-To: <20240102220134.3229156-1-samuel.holland@sifive.com> References: <20240102220134.3229156-1-samuel.holland@sifive.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" This variable is only used inside asids_init(). Signed-off-by: Samuel Holland Reviewed-by: Alexandre Ghiti --- (no changes since v1) arch/riscv/mm/context.c | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/arch/riscv/mm/context.c b/arch/riscv/mm/context.c index b5170ac1b742..43a8bc2d5af4 100644 --- a/arch/riscv/mm/context.c +++ b/arch/riscv/mm/context.c @@ -20,7 +20,6 @@ =20 DEFINE_STATIC_KEY_FALSE(use_asid_allocator); =20 -static unsigned long asid_bits; static unsigned long num_asids; =20 static atomic_long_t current_version; @@ -226,7 +225,7 @@ static inline void set_mm(struct mm_struct *prev, =20 static int __init asids_init(void) { - unsigned long old; + unsigned long asid_bits, old; =20 /* Figure-out number of ASID bits in HW */ old =3D csr_read(CSR_SATP); --=20 2.42.0 From nobody Fri Dec 26 17:23:07 2025 Received: from mail-pj1-f52.google.com (mail-pj1-f52.google.com [209.85.216.52]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D7DCF18B0D for ; Tue, 2 Jan 2024 22:01:48 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=sifive.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=sifive.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=sifive.com header.i=@sifive.com header.b="NX5xwfDX" Received: by mail-pj1-f52.google.com with SMTP id 98e67ed59e1d1-28ca63fd071so1666557a91.3 for ; Tue, 02 Jan 2024 14:01:48 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1704232908; x=1704837708; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=MROdnZxFJAn7brM6ye/jIKPJcvVCbRqlgsjUSxtP0PY=; b=NX5xwfDXegsIZdZeD3oMwQwMHll3o+dryBRf6MwNR6F6IX4VNOGzcoupwUi0hInOhr h6sF53qbIWdDZj6Qgu4O+pdeZ+C9CECw8Y7lwEU4TONlqJ9Q8wa8OzdNURP/c/bf2QQc ro0tzJ7vCnxgAUvDgSgLW9K4RFISSOKH43jgvpkaFvYEEtN4TMa+QSZhoNyyTWW+sgc8 YNS5kZFkfxv2lcIiMzufz2EQ8x+nGWpgKxnoG87ULrnUExvhQiFHPka+bgs7NfgAkuLe bQe19ylYJbvD8Xl448n8aB/05Vt5ui5sGcmfLmpAonKKDwtmm37SxcSLb0u68c9NTOFx rVWA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1704232908; x=1704837708; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=MROdnZxFJAn7brM6ye/jIKPJcvVCbRqlgsjUSxtP0PY=; b=UFqMNXlOfX78Q7InhSbI15FSi2sm+mJwX/taC0GkxQbM/6aP6Luk1Qahjrx3lf/gjN iMZtGkeiFeyrb3rHzlzsbkYkXCBnhlAM+L+/i8GMr2/SDhM7WofFLxSilR/+ppoKo5EA wGDCJO76jeWKxdE/eRmi8RLrvb/j8zgYiltkFIHCg/DAEc5ZY5Mkjwm/nEkr5nNFztUR PdzJbSSVkuQP+FRwXP5rl2rLhizl2qxr2j6kJBslYmh2LRqnhT8tvhLliBEbIem75xfE ZvUe37x2Ic5j9F2DtxdqjW3MROZ+ORGrujKeAmmRa3iP6CWMiflPQEpDePC2Iv/vUn0g iNGA== X-Gm-Message-State: AOJu0YxOUlChG7ngwFnuj78SiRsgv46pNcq0Z+H4pVIa8552e+AVby6V tE9EAKHBb7LD00JnNaxcApNF3hvAC3N93g== X-Google-Smtp-Source: AGHT+IFQ4MdRufOELZ2z13xg7yZ/1JoYbB5tE+2va7NZu/tN1uOevH1SKPh+nsWqBiMa4TjGPndiZg== X-Received: by 2002:a17:90a:5c82:b0:28c:2e1f:8bf2 with SMTP id r2-20020a17090a5c8200b0028c2e1f8bf2mr4930306pji.84.1704232908323; Tue, 02 Jan 2024 14:01:48 -0800 (PST) Received: from sw06.internal.sifive.com ([4.53.31.132]) by smtp.gmail.com with ESMTPSA id r59-20020a17090a43c100b0028ce507cd7dsm101724pjg.55.2024.01.02.14.01.47 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 02 Jan 2024 14:01:48 -0800 (PST) From: Samuel Holland To: Palmer Dabbelt , linux-riscv@lists.infradead.org Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, Alexandre Ghiti , Samuel Holland Subject: [PATCH v4 11/12] riscv: mm: Preserve global TLB entries when switching contexts Date: Tue, 2 Jan 2024 14:00:48 -0800 Message-ID: <20240102220134.3229156-12-samuel.holland@sifive.com> X-Mailer: git-send-email 2.42.0 In-Reply-To: <20240102220134.3229156-1-samuel.holland@sifive.com> References: <20240102220134.3229156-1-samuel.holland@sifive.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" If the CPU does not support multiple ASIDs, all MM contexts use ASID 0. In this case, it is still beneficial to flush the TLB by ASID, as the single-ASID variant of the sfence.vma instruction preserves TLB entries for global (kernel) pages. This optimization is recommended by the RISC-V privileged specification: If the implementation does not provide ASIDs, or software chooses to always use ASID 0, then after every satp write, software should execute SFENCE.VMA with rs1=3Dx0. In the common case that no global translations have been modified, rs2 should be set to a register other than x0 but which contains the value zero, so that global translations are not flushed. It is not possible to apply this optimization when using the ASID allocator, because that code must flush the TLB for all ASIDs at once when incrementing the version number. Signed-off-by: Samuel Holland Reviewed-by: Alexandre Ghiti --- (no changes since v1) arch/riscv/mm/context.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/arch/riscv/mm/context.c b/arch/riscv/mm/context.c index 43a8bc2d5af4..3ca9b653df7d 100644 --- a/arch/riscv/mm/context.c +++ b/arch/riscv/mm/context.c @@ -200,7 +200,7 @@ static void set_mm_noasid(struct mm_struct *mm) { /* Switch the page table and blindly nuke entire local TLB */ csr_write(CSR_SATP, virt_to_pfn(mm->pgd) | satp_mode); - local_flush_tlb_all(); + local_flush_tlb_all_asid(0); } =20 static inline void set_mm(struct mm_struct *prev, --=20 2.42.0 From nobody Fri Dec 26 17:23:07 2025 Received: from mail-pj1-f44.google.com (mail-pj1-f44.google.com [209.85.216.44]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id F0070199DF for ; Tue, 2 Jan 2024 22:01:49 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=sifive.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=sifive.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=sifive.com header.i=@sifive.com header.b="S+xnwffw" Received: by mail-pj1-f44.google.com with SMTP id 98e67ed59e1d1-28ce6eeb4easo310365a91.0 for ; Tue, 02 Jan 2024 14:01:49 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1704232909; x=1704837709; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=ntXY1BdgwMpYCVbC/qFvmg3dz3t/ZzKIeqKl93B+ekg=; b=S+xnwffwYxZ2FpgRDUEM5aG5lh6apSSIe8QE1YEuvg5HNZl2uvXS5xB3I4h68PXJtp gDoasIZL+E6vJDxsAlaAz2qiuZGnidTBKhLbKUVB9MCqyezHINS1vk9XB1P8QhCeQ/Gk 1F3vkO36JKmC6wu3aDyGwRtHq/T4uIGYQI8W7lwNy99bF6j1+kb6RN8jomX5tdXoDy0z vv5Imvd5qeKtPNXykUn7nqE3fbpEknvZSukERFlYHljwch8UPjc169Yq0S62vKZRE1+2 +sL6iybOEqjROZIlxUaOrgJTurzWQkqGwTWUSYMSkJZ+grzlapuwMTGlafSeAc4TDLZ5 juuw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1704232909; x=1704837709; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ntXY1BdgwMpYCVbC/qFvmg3dz3t/ZzKIeqKl93B+ekg=; b=wUFzXaOKgPQ6bpVNRdd3sXVSphJIgLT4XAs1Hg6wuk8igZ47iWWsJfoBFkPc2rw7HU X6YbehGy/PwEp08QENOlEm7ESgzXRzWS1JJeNE1CZaOwAhGYpB6+bnR7bXzt46K+FwRp LjFjU8YANW+/tBaQqBl3m9hziCm5Qn6SZooUnlwIeIiSRqXnIpRwHCaYnVbXv3toXfVv mTyRX/e6VWaw2o5Yju0ChSbfl4ZCqIHPoLZgVcO/oI3bta1WQHrxooAabJjpIJNm1bg3 ADqN3DVVKGxgOJnxIx7nssthW7NIKhsH1Z6AhZeaCqssk69jT27puFZ1AOz2WwKD0tLg bENA== X-Gm-Message-State: AOJu0Yxjvt/cT1av97KCBOP0oJOF7Cfc0U7dZzR3wCqymxwaSWrUni1I yuHESkk5hUG5ZAfGWnlSFpzncjCxxtwGQw== X-Google-Smtp-Source: AGHT+IE6m2P6rWzQ+Gmcf0Ghh7A4crRVy8Bq5ePskuN3Kp4kY6ZC52OLPfUohVtqOdFfc+/yJvJ6pQ== X-Received: by 2002:a17:90a:bc46:b0:28b:31f5:9668 with SMTP id t6-20020a17090abc4600b0028b31f59668mr5241043pjv.30.1704232909372; Tue, 02 Jan 2024 14:01:49 -0800 (PST) Received: from sw06.internal.sifive.com ([4.53.31.132]) by smtp.gmail.com with ESMTPSA id r59-20020a17090a43c100b0028ce507cd7dsm101724pjg.55.2024.01.02.14.01.48 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 02 Jan 2024 14:01:49 -0800 (PST) From: Samuel Holland To: Palmer Dabbelt , linux-riscv@lists.infradead.org Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, Alexandre Ghiti , Samuel Holland Subject: [PATCH v4 12/12] riscv: mm: Always use an ASID to flush mm contexts Date: Tue, 2 Jan 2024 14:00:49 -0800 Message-ID: <20240102220134.3229156-13-samuel.holland@sifive.com> X-Mailer: git-send-email 2.42.0 In-Reply-To: <20240102220134.3229156-1-samuel.holland@sifive.com> References: <20240102220134.3229156-1-samuel.holland@sifive.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Even if multiple ASIDs are not supported, using the single-ASID variant of the sfence.vma instruction preserves TLB entries for global (kernel) pages. So it is always more efficient to use the single-ASID code path. Signed-off-by: Samuel Holland Reviewed-by: Alexandre Ghiti --- Changes in v4: - There is now only one copy of __flush_tlb_range() Changes in v2: - Update both copies of __flush_tlb_range() arch/riscv/include/asm/mmu_context.h | 2 -- arch/riscv/mm/context.c | 3 +-- arch/riscv/mm/tlbflush.c | 3 +-- 3 files changed, 2 insertions(+), 6 deletions(-) diff --git a/arch/riscv/include/asm/mmu_context.h b/arch/riscv/include/asm/= mmu_context.h index 7030837adc1a..b0659413a080 100644 --- a/arch/riscv/include/asm/mmu_context.h +++ b/arch/riscv/include/asm/mmu_context.h @@ -33,8 +33,6 @@ static inline int init_new_context(struct task_struct *ts= k, return 0; } =20 -DECLARE_STATIC_KEY_FALSE(use_asid_allocator); - #include =20 #endif /* _ASM_RISCV_MMU_CONTEXT_H */ diff --git a/arch/riscv/mm/context.c b/arch/riscv/mm/context.c index 3ca9b653df7d..20057085ab8a 100644 --- a/arch/riscv/mm/context.c +++ b/arch/riscv/mm/context.c @@ -18,8 +18,7 @@ =20 #ifdef CONFIG_MMU =20 -DEFINE_STATIC_KEY_FALSE(use_asid_allocator); - +static DEFINE_STATIC_KEY_FALSE(use_asid_allocator); static unsigned long num_asids; =20 static atomic_long_t current_version; diff --git a/arch/riscv/mm/tlbflush.c b/arch/riscv/mm/tlbflush.c index 5ec621545c69..39d80f56d292 100644 --- a/arch/riscv/mm/tlbflush.c +++ b/arch/riscv/mm/tlbflush.c @@ -84,8 +84,7 @@ static void __flush_tlb_range(struct mm_struct *mm, unsig= ned long start, if (cpumask_empty(cmask)) return; =20 - if (static_branch_unlikely(&use_asid_allocator)) - asid =3D cntx2asid(atomic_long_read(&mm->context.id)); + asid =3D cntx2asid(atomic_long_read(&mm->context.id)); } else { cmask =3D cpu_online_mask; } --=20 2.42.0