From nobody Sun Feb 8 20:22:57 2026 Received: from mail-pf1-f174.google.com (mail-pf1-f174.google.com [209.85.210.174]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E56096D52F for ; Thu, 29 Feb 2024 23:22:14 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.174 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709248936; cv=none; b=oYPlq4hwaJliz6rXS/nVyOpKDMrq1shS77rVv47Z1BJPwMTQkOb15p5RPhEkNr+ECPOILJCiOCVHBtIc2c48EqvyLcGodAlg/DvjibBhOGdP285KVdUfxH9/ZE5I05wbu3lGUB1tekjF/BwAnAxlmeP2RogHFnG24ez1S/j2Y0Q= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709248936; c=relaxed/simple; bh=KWtoavmsQ9Zyjmm584UX//N/CxD8kDI7cE1TWuh589Y=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=pB+GrbU7Z/ANJ2XHjcS/wplP/vefrBl8g+umekIhXshnhAYEobHraveMFs7YDFM05u5Wxl/ww6DoWdXXpnjqS0fSHC1bYddI/ucLxlDojeyJK2Hr9k/v1JYxwdAxtqODd9dyQXQWrlq9rO2frxcM39JdYnsARqz/OGBLEpB77nc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=sifive.com; spf=pass smtp.mailfrom=sifive.com; dkim=pass (2048-bit key) header.d=sifive.com header.i=@sifive.com header.b=gmTxm6T0; arc=none smtp.client-ip=209.85.210.174 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=sifive.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=sifive.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=sifive.com header.i=@sifive.com header.b="gmTxm6T0" Received: by mail-pf1-f174.google.com with SMTP id d2e1a72fcca58-6e5896846f2so1080168b3a.2 for ; Thu, 29 Feb 2024 15:22:14 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1709248934; x=1709853734; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=O37VWqG4iCvkiZDgqVtP8oElF9rd1vMtEBQWx77Nmx0=; b=gmTxm6T0avUpX9VJ7qCwLt1Rfkqoxx8cFMtmIMvsDRBBrKpf0SbuHte/6vrcxvgbE9 W6okPqhJfltLijsznF+BcrEfSMXPXPDFSulZeKD5MD2VaUefw4qWCuyjvQqD4/atesfq 98lhyxshteYBc2moJwwnBYpXZngNGPecE1AyivgB/XFeAlb/+XssbU4kBeajC2GxKNlS nOXf1GDd7b/1bdWQHJfuKcw5+dUYE4Y88G1Mz6VlNW+LOhh+jrHY4BdUCQnFMZGYBjHK uTlhnTvKOSnD+Trg8ZRa+FkTyhvGMTknAmhPoCriPfL3H0LH0HjUqCK0u3E5jEdhC5iG TS6A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709248934; x=1709853734; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=O37VWqG4iCvkiZDgqVtP8oElF9rd1vMtEBQWx77Nmx0=; b=BRkEcEDuFubf6GxAqSckhMV7WrI2peULZr6yMPRyiWeS9BEjwDnug9G0MXfl5hT/2T uwnRRKvrPEHvEMgCJ1QMh3VcsHFvMcLD+DBaORtWr5wlfCvUonh1/csLIkoj1YPmKIT/ Scqm0UNPWv/greuiayFbwhlS8f7cfXEkw0zy99aG483iHiUqbzLiXgwpIMajN2vyn/sa qfGS/cwa0EpClcCfhwVlrcfwSfi5ayhruOk6yBnIfgulxoF8AT1dPghAcMuG53/emH8L 5BkRqM2Dc+PPeVKE73WFmI2fTQuUgQuhOL04GXcLahZ/MMq6RkXVfIFkHGbGtzvr77fk g94Q== X-Gm-Message-State: AOJu0YyQFxl32DCFu67chu4ojaIrBvIe/08J7dcEA8O5aNMSWaa81Xz+ NhDwl4TZFIC0UrvNT0HB46oltwoz/ZuJCjGohLOYpjW4ytU0fj3xDNqhSRyhCuo= X-Google-Smtp-Source: AGHT+IEXcsW9CGPGmNlysP87xM+KqyMeGQDT/2i2zpW2/RfnrP1VyWuH3lyJldn2H0bz6i0E+Cfqiw== X-Received: by 2002:a05:6a00:2d1c:b0:6e3:fa52:2f26 with SMTP id fa28-20020a056a002d1c00b006e3fa522f26mr168419pfb.8.1709248934208; Thu, 29 Feb 2024 15:22:14 -0800 (PST) Received: from sw06.internal.sifive.com ([4.53.31.132]) by smtp.gmail.com with ESMTPSA id c6-20020aa78806000000b006e55aa75d6csm1779719pfo.122.2024.02.29.15.22.13 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 29 Feb 2024 15:22:13 -0800 (PST) From: Samuel Holland To: Palmer Dabbelt , linux-riscv@lists.infradead.org Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, Alexandre Ghiti , Jisheng Zhang , Yunhui Cui , Samuel Holland Subject: [PATCH v5 01/13] riscv: Flush the instruction cache during SMP bringup Date: Thu, 29 Feb 2024 15:21:42 -0800 Message-ID: <20240229232211.161961-2-samuel.holland@sifive.com> X-Mailer: git-send-email 2.43.1 In-Reply-To: <20240229232211.161961-1-samuel.holland@sifive.com> References: <20240229232211.161961-1-samuel.holland@sifive.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Instruction cache flush IPIs are sent only to CPUs in cpu_online_mask, so they will not target a CPU until it calls set_cpu_online() earlier in smp_callin(). As a result, if instruction memory is modified between the CPU coming out of reset and that point, then its instruction cache may contain stale data. Therefore, the instruction cache must be flushed after the set_cpu_online() synchronization point. Fixes: 08f051eda33b ("RISC-V: Flush I$ when making a dirty page executable") Reviewed-by: Alexandre Ghiti Signed-off-by: Samuel Holland --- (no changes since v4) Changes in v4: - New patch for v4 arch/riscv/kernel/smpboot.c | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/arch/riscv/kernel/smpboot.c b/arch/riscv/kernel/smpboot.c index cfbe4b840d42..e1b612f37dd9 100644 --- a/arch/riscv/kernel/smpboot.c +++ b/arch/riscv/kernel/smpboot.c @@ -26,7 +26,7 @@ #include #include =20 -#include +#include #include #include #include @@ -238,9 +238,10 @@ asmlinkage __visible void smp_callin(void) riscv_user_isa_enable(); =20 /* - * Remote TLB flushes are ignored while the CPU is offline, so emit - * a local TLB flush right now just in case. + * Remote cache and TLB flushes are ignored while the CPU is offline, + * so flush them both right now just in case. */ + local_flush_icache_all(); local_flush_tlb_all(); complete(&cpu_running); /* --=20 2.43.1 From nobody Sun Feb 8 20:22:57 2026 Received: from mail-pf1-f174.google.com (mail-pf1-f174.google.com [209.85.210.174]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C67BA6D535 for ; Thu, 29 Feb 2024 23:22:15 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.174 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709248937; cv=none; b=fFkenk3+4PJIV/YHYe99bJJqbG+9KFsMD92uHtxJsqcCPn3RUCwYBA2DFV112hWdVFv1m3+1Y99AgEAeLrS/1NdcSKFvceVhm71CqAvG74x8cKnjvPBS/ztWw1wYJb4qkaUx8vtB4B/zHfn13uj/Ww3s360jWT1MDFtu4RFgJVc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709248937; c=relaxed/simple; bh=AZIiAOFhR7ivzwD+1gH5VFDhXubgORx9r0fNtPydx90=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=iSGfx67MWk/P+fNbt0+2/pr5VIBlSednVuj1g4/khSZbj9yNtmgE8vWi0dVGSENh0IyqRhYJQ6/BjK4vrBplqHwTfZf6ctfsZpCeWzN3B4LJSOuyRW4ahP+3a4AKhnfWrrP0XmC0a75AHVT2cWfcMZFYVij2dg8xY4yd2+RGh/E= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=sifive.com; spf=pass smtp.mailfrom=sifive.com; dkim=pass (2048-bit key) header.d=sifive.com header.i=@sifive.com header.b=h/MyfBgN; arc=none smtp.client-ip=209.85.210.174 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=sifive.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=sifive.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=sifive.com header.i=@sifive.com header.b="h/MyfBgN" Received: by mail-pf1-f174.google.com with SMTP id d2e1a72fcca58-6e581ba25e1so1204857b3a.1 for ; Thu, 29 Feb 2024 15:22:15 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1709248935; x=1709853735; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=XEa0bZw6X2PF+fvXkyrnyWuZOURSNgi5dGIyeM9k7ic=; b=h/MyfBgNZmwR2myHs7E6KXTLfy/zy1gfa3oxhsXQYIegsrJqrp4hzpBgLA+YjMO/RJ 9lva7lenbVWleD/pau1Jvi9JO2ZQIr0o4SkAaDhT01LWrn20n0pVuRv3qIwdmrmQYOEI AV44QE89HoHSIa7tTRgg7pQIcpntzZ+3wq4NrvzR4Fw7HUoIiLCoZYDGLlXa4DvmQng6 rcGfi6KVOeYbtFg+ySwZtfR7nwaWuQXirx6iv2NmpWMKsDjJ7OPylpJQZLwO8eUHRmOv gzUqsM2AYEilcIvQM1nKITXsQcmmXAi5kG9PTVVxRWDmDGK9Yzm1aPAfLXWvD06v9rvW Zcfw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709248935; x=1709853735; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=XEa0bZw6X2PF+fvXkyrnyWuZOURSNgi5dGIyeM9k7ic=; b=Hrg7qfypbqJSJODLlxLpbivMyPlKysyucR5hEZ03zgPoaZ7IaENg8QUxLMdphm7wCm OV8Cni8EhIh8DtVqvn6FDBEy7Uhs+iwmGeJh2/4udtbDd0XjBumOG2O9e52WVBQifSHc FPvv+snM0PJMmcR42YvN2cC2G/941Bv1nfsCcGR7ljUM5d6WeAoV2MLKH0UlehXwxjoa J4dCgY+FA3YZVL4C1NgWNYvXP00Km6Is3E3OEhli8sCEEJkPcQwDlQC7d8E8Jyjo0ZdY AFvmO2WXiY6JV1Cyb54rcR8AeNNFtEK9Ojuzr5ziamPdDb00EV7tIXuFjeRr4dZaaoPL fPSw== X-Gm-Message-State: AOJu0YwIdF0Mubo5zP6Ou4ke4dLiDg2HQ6V7S0UsHirKBc5C/c9XkqHV 7NaPvOZmQQmXeQHGaaKXicNk5bdSFlYMS2/9byKNssQqKTcz/lPZAAuoN3sSq98= X-Google-Smtp-Source: AGHT+IFdgcTr06rQ4rVKFKQwHiCzNjuzvF9/zHzBzMpbTCozBAylfcba6wC7aAVCdn9otyCllPR9tg== X-Received: by 2002:a05:6a00:2293:b0:6e5:599a:5252 with SMTP id f19-20020a056a00229300b006e5599a5252mr606918pfe.15.1709248935218; Thu, 29 Feb 2024 15:22:15 -0800 (PST) Received: from sw06.internal.sifive.com ([4.53.31.132]) by smtp.gmail.com with ESMTPSA id c6-20020aa78806000000b006e55aa75d6csm1779719pfo.122.2024.02.29.15.22.14 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 29 Feb 2024 15:22:14 -0800 (PST) From: Samuel Holland To: Palmer Dabbelt , linux-riscv@lists.infradead.org Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, Alexandre Ghiti , Jisheng Zhang , Yunhui Cui , Samuel Holland Subject: [PATCH v5 02/13] riscv: Factor out page table TLB synchronization Date: Thu, 29 Feb 2024 15:21:43 -0800 Message-ID: <20240229232211.161961-3-samuel.holland@sifive.com> X-Mailer: git-send-email 2.43.1 In-Reply-To: <20240229232211.161961-1-samuel.holland@sifive.com> References: <20240229232211.161961-1-samuel.holland@sifive.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" The logic is the same for all page table levels. See commit 69be3fb111e7 ("riscv: enable MMU_GATHER_RCU_TABLE_FREE for SMP && MMU"). Signed-off-by: Samuel Holland --- Changes in v5: - New patch for v5 arch/riscv/include/asm/pgalloc.h | 31 +++++++++++++------------------ 1 file changed, 13 insertions(+), 18 deletions(-) diff --git a/arch/riscv/include/asm/pgalloc.h b/arch/riscv/include/asm/pgal= loc.h index deaf971253a2..87468f67951a 100644 --- a/arch/riscv/include/asm/pgalloc.h +++ b/arch/riscv/include/asm/pgalloc.h @@ -88,6 +88,14 @@ static inline pud_t *pud_alloc_one(struct mm_struct *mm,= unsigned long addr) return NULL; } =20 +static inline void riscv_tlb_remove_ptdesc(struct mmu_gather *tlb, void *p= t) +{ + if (riscv_use_ipi_for_rfence()) + tlb_remove_page_ptdesc(tlb, pt); + else + tlb_remove_ptdesc(tlb, pt); +} + #define pud_free pud_free static inline void pud_free(struct mm_struct *mm, pud_t *pud) { @@ -102,10 +110,7 @@ static inline void __pud_free_tlb(struct mmu_gather *t= lb, pud_t *pud, struct ptdesc *ptdesc =3D virt_to_ptdesc(pud); =20 pagetable_pud_dtor(ptdesc); - if (riscv_use_ipi_for_rfence()) - tlb_remove_page_ptdesc(tlb, ptdesc); - else - tlb_remove_ptdesc(tlb, ptdesc); + riscv_tlb_remove_ptdesc(tlb, ptdesc); } } =20 @@ -139,12 +144,8 @@ static inline void p4d_free(struct mm_struct *mm, p4d_= t *p4d) static inline void __p4d_free_tlb(struct mmu_gather *tlb, p4d_t *p4d, unsigned long addr) { - if (pgtable_l5_enabled) { - if (riscv_use_ipi_for_rfence()) - tlb_remove_page_ptdesc(tlb, virt_to_ptdesc(p4d)); - else - tlb_remove_ptdesc(tlb, virt_to_ptdesc(p4d)); - } + if (pgtable_l5_enabled) + riscv_tlb_remove_ptdesc(tlb, virt_to_ptdesc(p4d)); } #endif /* __PAGETABLE_PMD_FOLDED */ =20 @@ -176,10 +177,7 @@ static inline void __pmd_free_tlb(struct mmu_gather *t= lb, pmd_t *pmd, struct ptdesc *ptdesc =3D virt_to_ptdesc(pmd); =20 pagetable_pmd_dtor(ptdesc); - if (riscv_use_ipi_for_rfence()) - tlb_remove_page_ptdesc(tlb, ptdesc); - else - tlb_remove_ptdesc(tlb, ptdesc); + riscv_tlb_remove_ptdesc(tlb, ptdesc); } =20 #endif /* __PAGETABLE_PMD_FOLDED */ @@ -190,10 +188,7 @@ static inline void __pte_free_tlb(struct mmu_gather *t= lb, pgtable_t pte, struct ptdesc *ptdesc =3D page_ptdesc(pte); =20 pagetable_pte_dtor(ptdesc); - if (riscv_use_ipi_for_rfence()) - tlb_remove_page_ptdesc(tlb, ptdesc); - else - tlb_remove_ptdesc(tlb, ptdesc); + riscv_tlb_remove_ptdesc(tlb, ptdesc); } #endif /* CONFIG_MMU */ =20 --=20 2.43.1 From nobody Sun Feb 8 20:22:57 2026 Received: from mail-il1-f169.google.com (mail-il1-f169.google.com [209.85.166.169]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 48A9975815 for ; Thu, 29 Feb 2024 23:22:16 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.166.169 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709248939; cv=none; b=ePreVcJUeErL1qfVm3KKUD/2z2NgaAFzGq9kTF19sSM17+dEdFXpT8xydZSaQag0hwu3Io5rWW/kjy6ixzs9lMH2QUFrde2tPwjXEnUD3OIdwhUWt+U8R5bEhMpNshaharfdcHOCXj+EiLKLvKspKeTEy6rgKRiiSROS6IRwnlc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709248939; c=relaxed/simple; bh=POwNoYcpvng6qPt5oFG0srd43nGliYjpqXmrjwOe5C0=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=go70LVeRiE84tAr16x1WokLo1WzuKBnv8LQW8zL9nPi51PZ1pYy3YtRJfASdkb6w6Pc3VnCoQ+9ptseCGYdU09awMmcQVK+r4gWAGTHPb+nPctVHfsoN4DomysU/6+smYYQgZKuq+Q8ikxGcYPyI66n+Kn0h8gMwN48ye0TMzIQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=sifive.com; spf=pass smtp.mailfrom=sifive.com; dkim=pass (2048-bit key) header.d=sifive.com header.i=@sifive.com header.b=ZVsYrktr; arc=none smtp.client-ip=209.85.166.169 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=sifive.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=sifive.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=sifive.com header.i=@sifive.com header.b="ZVsYrktr" Received: by mail-il1-f169.google.com with SMTP id e9e14a558f8ab-3652731b2b7so6019635ab.2 for ; Thu, 29 Feb 2024 15:22:16 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1709248936; x=1709853736; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=nL3CU8jW9vVfZeENVgIQs2Z0lmYtCUXWU5kIlhxMA+M=; b=ZVsYrktrfxOF4ow9ec/phqqw8bbpPoKv0ww/8/GpeepsbGmuxuzaOUMPNURIFn/8r2 fo/IIDHsqAdnNejCAdsRioXiD4YJjy0cD0wG/89HIXTwEPgzfggWLb2YcgAnJ+ntFlPZ MbLYJ7cCfeaFyD/x4YJbkIjPfilnFeQQUztBoyS8XTA2vvTUAdqGK3G5CRrbC94kpA2M +9ZfKRy12PUnvejRO7nl96dIPkaphzmJr4CjwpbVd/yQlU5bYBJl/UhEaO1rmLBv8Uv7 8i4VN3RNQ0X+Fc6YNywhNnRKtF8UWt+pNKG8l9pvmwANttliGdH0EpJhuZXfqYscWlXA Y4bg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709248936; x=1709853736; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=nL3CU8jW9vVfZeENVgIQs2Z0lmYtCUXWU5kIlhxMA+M=; b=hOtQ/qYO6OyQ33eabhYs1DWMhs8f2pHBpGul/sediv538yA8pSNXmmEHIBKMRUKxGk zi4T6fX2y9FuyzTyks9sI+qMKah6sLqGCfwRaCZA/CXK31QWdUy8isJ5/PMs6U3QELqZ 8GHUxex+9B0IzavOiOpZ4qxVbTQ6WV0RmfD70LV+rAsA3/3EyW0pnN/leP0sdp3Dvztt WXRCc1FX2KYW/aFt0Rfvt0yyPg1XRiOf7WcaEtdhoavFexlr3nJuZnxbwhmN6v6+SOpt iWUGnBf3QlWbzw9y2YxP8J0oNsmKAEMcVVmKWPf+NROZcl+6+17hkV8U6YOtjOXzJsEj xjjA== X-Gm-Message-State: AOJu0Yx5lHbzJ2p6Jzu6Q08mJhB+mTEl1zh7lQmQGAIMKmXmXYycSq/i rIQibwJtlVUtlSRK6DA0hsHBWG/rSqUnoUhVsQUkZcpDTvM0FVD3/ZFteBhHbLI= X-Google-Smtp-Source: AGHT+IEuQOuQ50R8qoZP0RbIoJTNSdVCKrs4lRWLzJ+znFdHc+eSLoYj/Ypw4BmFNiQ++vK2RkPm8w== X-Received: by 2002:a05:6e02:1b8f:b0:365:ae26:5827 with SMTP id h15-20020a056e021b8f00b00365ae265827mr159548ili.5.1709248936255; Thu, 29 Feb 2024 15:22:16 -0800 (PST) Received: from sw06.internal.sifive.com ([4.53.31.132]) by smtp.gmail.com with ESMTPSA id c6-20020aa78806000000b006e55aa75d6csm1779719pfo.122.2024.02.29.15.22.15 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 29 Feb 2024 15:22:15 -0800 (PST) From: Samuel Holland To: Palmer Dabbelt , linux-riscv@lists.infradead.org Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, Alexandre Ghiti , Jisheng Zhang , Yunhui Cui , Samuel Holland Subject: [PATCH v5 03/13] riscv: Use IPIs for remote cache/TLB flushes by default Date: Thu, 29 Feb 2024 15:21:44 -0800 Message-ID: <20240229232211.161961-4-samuel.holland@sifive.com> X-Mailer: git-send-email 2.43.1 In-Reply-To: <20240229232211.161961-1-samuel.holland@sifive.com> References: <20240229232211.161961-1-samuel.holland@sifive.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" An IPI backend is always required in an SMP configuration, but an SBI implementation is not. For example, SBI will be unavailable when the kernel runs in M mode. Generally, IPIs are assumed to be faster than SBI calls due to the SBI context switch overhead. However, when SBI is used as the IPI backend, then the context switch cost must be paid anyway, and performing the cache/TLB flush directly in the SBI implementation is more efficient than inserting an interrupt to the kernel. This is the only scenario where riscv_ipi_set_virq_range()'s use_for_rfence parameter is false. Thus, it makes sense for remote fences to use IPIs by default, and make the SBI remote fence extension the special case. sbi_ipi_init() already checks riscv_ipi_have_virq_range(), so it only calls riscv_ipi_set_virq_range() when no other IPI device is available. So we can move the static key and drop the use_for_rfence parameter. Furthermore, the static branch only makes sense when CONFIG_RISCV_SBI is enabled. Optherwise, IPIs must be used. Add a fallback definition of riscv_use_sbi_for_rfence() which handles this case and removes the need to check CONFIG_RISCV_SBI elsewhere, such as in cacheflush.c. Signed-off-by: Samuel Holland Reviewed-by: Anup Patel --- Changes in v5: - Also switch to riscv_use_sbi_for_rfence() in asm/pgalloc.h Changes in v4: - New patch for v4 arch/riscv/include/asm/pgalloc.h | 7 ++++--- arch/riscv/include/asm/sbi.h | 4 ++++ arch/riscv/include/asm/smp.h | 15 ++------------- arch/riscv/kernel/sbi-ipi.c | 11 ++++++++++- arch/riscv/kernel/smp.c | 11 +---------- arch/riscv/mm/cacheflush.c | 5 ++--- arch/riscv/mm/tlbflush.c | 31 ++++++++++++++----------------- drivers/clocksource/timer-clint.c | 2 +- 8 files changed, 38 insertions(+), 48 deletions(-) diff --git a/arch/riscv/include/asm/pgalloc.h b/arch/riscv/include/asm/pgal= loc.h index 87468f67951a..6578054977ef 100644 --- a/arch/riscv/include/asm/pgalloc.h +++ b/arch/riscv/include/asm/pgalloc.h @@ -8,6 +8,7 @@ #define _ASM_RISCV_PGALLOC_H =20 #include +#include #include =20 #ifdef CONFIG_MMU @@ -90,10 +91,10 @@ static inline pud_t *pud_alloc_one(struct mm_struct *mm= , unsigned long addr) =20 static inline void riscv_tlb_remove_ptdesc(struct mmu_gather *tlb, void *p= t) { - if (riscv_use_ipi_for_rfence()) - tlb_remove_page_ptdesc(tlb, pt); - else + if (riscv_use_sbi_for_rfence()) tlb_remove_ptdesc(tlb, pt); + else + tlb_remove_page_ptdesc(tlb, pt); } =20 #define pud_free pud_free diff --git a/arch/riscv/include/asm/sbi.h b/arch/riscv/include/asm/sbi.h index 6e68f8dff76b..ea84392ca9d7 100644 --- a/arch/riscv/include/asm/sbi.h +++ b/arch/riscv/include/asm/sbi.h @@ -375,8 +375,12 @@ unsigned long riscv_cached_marchid(unsigned int cpu_id= ); unsigned long riscv_cached_mimpid(unsigned int cpu_id); =20 #if IS_ENABLED(CONFIG_SMP) && IS_ENABLED(CONFIG_RISCV_SBI) +DECLARE_STATIC_KEY_FALSE(riscv_sbi_for_rfence); +#define riscv_use_sbi_for_rfence() \ + static_branch_unlikely(&riscv_sbi_for_rfence) void sbi_ipi_init(void); #else +static inline bool riscv_use_sbi_for_rfence(void) { return false; } static inline void sbi_ipi_init(void) { } #endif =20 diff --git a/arch/riscv/include/asm/smp.h b/arch/riscv/include/asm/smp.h index 0d555847cde6..7ac80e9f2288 100644 --- a/arch/riscv/include/asm/smp.h +++ b/arch/riscv/include/asm/smp.h @@ -49,12 +49,7 @@ void riscv_ipi_disable(void); bool riscv_ipi_have_virq_range(void); =20 /* Set the IPI interrupt numbers for arch (called by irqchip drivers) */ -void riscv_ipi_set_virq_range(int virq, int nr, bool use_for_rfence); - -/* Check if we can use IPIs for remote FENCEs */ -DECLARE_STATIC_KEY_FALSE(riscv_ipi_for_rfence); -#define riscv_use_ipi_for_rfence() \ - static_branch_unlikely(&riscv_ipi_for_rfence) +void riscv_ipi_set_virq_range(int virq, int nr); =20 /* Check other CPUs stop or not */ bool smp_crash_stop_failed(void); @@ -104,16 +99,10 @@ static inline bool riscv_ipi_have_virq_range(void) return false; } =20 -static inline void riscv_ipi_set_virq_range(int virq, int nr, - bool use_for_rfence) +static inline void riscv_ipi_set_virq_range(int virq, int nr) { } =20 -static inline bool riscv_use_ipi_for_rfence(void) -{ - return false; -} - #endif /* CONFIG_SMP */ =20 #if defined(CONFIG_HOTPLUG_CPU) && (CONFIG_SMP) diff --git a/arch/riscv/kernel/sbi-ipi.c b/arch/riscv/kernel/sbi-ipi.c index a4559695ce62..1026e22955cc 100644 --- a/arch/riscv/kernel/sbi-ipi.c +++ b/arch/riscv/kernel/sbi-ipi.c @@ -13,6 +13,9 @@ #include #include =20 +DEFINE_STATIC_KEY_FALSE(riscv_sbi_for_rfence); +EXPORT_SYMBOL_GPL(riscv_sbi_for_rfence); + static int sbi_ipi_virq; =20 static void sbi_ipi_handle(struct irq_desc *desc) @@ -72,6 +75,12 @@ void __init sbi_ipi_init(void) "irqchip/sbi-ipi:starting", sbi_ipi_starting_cpu, NULL); =20 - riscv_ipi_set_virq_range(virq, BITS_PER_BYTE, false); + riscv_ipi_set_virq_range(virq, BITS_PER_BYTE); pr_info("providing IPIs using SBI IPI extension\n"); + + /* + * Use the SBI remote fence extension to avoid + * the extra context switch needed to handle IPIs. + */ + static_branch_enable(&riscv_sbi_for_rfence); } diff --git a/arch/riscv/kernel/smp.c b/arch/riscv/kernel/smp.c index 45dd4035416e..8e6eb64459af 100644 --- a/arch/riscv/kernel/smp.c +++ b/arch/riscv/kernel/smp.c @@ -171,10 +171,7 @@ bool riscv_ipi_have_virq_range(void) return (ipi_virq_base) ? true : false; } =20 -DEFINE_STATIC_KEY_FALSE(riscv_ipi_for_rfence); -EXPORT_SYMBOL_GPL(riscv_ipi_for_rfence); - -void riscv_ipi_set_virq_range(int virq, int nr, bool use_for_rfence) +void riscv_ipi_set_virq_range(int virq, int nr) { int i, err; =20 @@ -197,12 +194,6 @@ void riscv_ipi_set_virq_range(int virq, int nr, bool u= se_for_rfence) =20 /* Enabled IPIs for boot CPU immediately */ riscv_ipi_enable(); - - /* Update RFENCE static key */ - if (use_for_rfence) - static_branch_enable(&riscv_ipi_for_rfence); - else - static_branch_disable(&riscv_ipi_for_rfence); } =20 static const char * const ipi_names[] =3D { diff --git a/arch/riscv/mm/cacheflush.c b/arch/riscv/mm/cacheflush.c index 55a34f2020a8..47c485bc7df0 100644 --- a/arch/riscv/mm/cacheflush.c +++ b/arch/riscv/mm/cacheflush.c @@ -21,7 +21,7 @@ void flush_icache_all(void) { local_flush_icache_all(); =20 - if (IS_ENABLED(CONFIG_RISCV_SBI) && !riscv_use_ipi_for_rfence()) + if (riscv_use_sbi_for_rfence()) sbi_remote_fence_i(NULL); else on_each_cpu(ipi_remote_fence_i, NULL, 1); @@ -69,8 +69,7 @@ void flush_icache_mm(struct mm_struct *mm, bool local) * with flush_icache_deferred(). */ smp_mb(); - } else if (IS_ENABLED(CONFIG_RISCV_SBI) && - !riscv_use_ipi_for_rfence()) { + } else if (riscv_use_sbi_for_rfence()) { sbi_remote_fence_i(&others); } else { on_each_cpu_mask(&others, ipi_remote_fence_i, NULL, 1); diff --git a/arch/riscv/mm/tlbflush.c b/arch/riscv/mm/tlbflush.c index 8d12b26f5ac3..0373661bd1c4 100644 --- a/arch/riscv/mm/tlbflush.c +++ b/arch/riscv/mm/tlbflush.c @@ -78,10 +78,10 @@ static void __ipi_flush_tlb_all(void *info) =20 void flush_tlb_all(void) { - if (riscv_use_ipi_for_rfence()) - on_each_cpu(__ipi_flush_tlb_all, NULL, 1); - else + if (riscv_use_sbi_for_rfence()) sbi_remote_sfence_vma_asid(NULL, 0, FLUSH_TLB_MAX_SIZE, FLUSH_TLB_NO_ASI= D); + else + on_each_cpu(__ipi_flush_tlb_all, NULL, 1); } =20 struct flush_tlb_range_data { @@ -102,7 +102,6 @@ static void __flush_tlb_range(struct cpumask *cmask, un= signed long asid, unsigned long start, unsigned long size, unsigned long stride) { - struct flush_tlb_range_data ftd; bool broadcast; =20 if (cpumask_empty(cmask)) @@ -118,20 +117,18 @@ static void __flush_tlb_range(struct cpumask *cmask, = unsigned long asid, broadcast =3D true; } =20 - if (broadcast) { - if (riscv_use_ipi_for_rfence()) { - ftd.asid =3D asid; - ftd.start =3D start; - ftd.size =3D size; - ftd.stride =3D stride; - on_each_cpu_mask(cmask, - __ipi_flush_tlb_range_asid, - &ftd, 1); - } else - sbi_remote_sfence_vma_asid(cmask, - start, size, asid); - } else { + if (!broadcast) { local_flush_tlb_range_asid(start, size, stride, asid); + } else if (riscv_use_sbi_for_rfence()) { + sbi_remote_sfence_vma_asid(cmask, start, size, asid); + } else { + struct flush_tlb_range_data ftd; + + ftd.asid =3D asid; + ftd.start =3D start; + ftd.size =3D size; + ftd.stride =3D stride; + on_each_cpu_mask(cmask, __ipi_flush_tlb_range_asid, &ftd, 1); } =20 if (cmask !=3D cpu_online_mask) diff --git a/drivers/clocksource/timer-clint.c b/drivers/clocksource/timer-= clint.c index 09fd292eb83d..0bdd9d7ec545 100644 --- a/drivers/clocksource/timer-clint.c +++ b/drivers/clocksource/timer-clint.c @@ -251,7 +251,7 @@ static int __init clint_timer_init_dt(struct device_nod= e *np) } =20 irq_set_chained_handler(clint_ipi_irq, clint_ipi_interrupt); - riscv_ipi_set_virq_range(rc, BITS_PER_BYTE, true); + riscv_ipi_set_virq_range(rc, BITS_PER_BYTE); clint_clear_ipi(); #endif =20 --=20 2.43.1 From nobody Sun Feb 8 20:22:57 2026 Received: from mail-pf1-f182.google.com (mail-pf1-f182.google.com [209.85.210.182]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 04BE0134419 for ; Thu, 29 Feb 2024 23:22:17 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.182 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709248939; cv=none; b=Wa1WgqfhA6Vaq9QTDppjnhXSjIqyYRrkCHMRyB3+xyDhraN/iVby9tnYupJ30ClTgQfPG2dYt2BtqKBu2kQoCV2fQos69o8g6yAF136fxeQZpWpBd5slq7u52b26LUHr3uZ6d4jhpFoUKIA+yQuzRrv6e61bWk+yn/LFcS6MB/E= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709248939; c=relaxed/simple; bh=+FVwMSfaHr0vNyET+wjjwD1Kp9smtCmocHiX0IEwmbE=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=MJokqbcm9pH5w3H3hYJ2QUGY88hW5Mj7YM/A6EJqgDAUg7IOWtzrRfFM/HFZZtF8jOfLZCFjU7JIHE/LczCp3pLLArLEU27MB2KxXwLonHd+zpYvGhtMDbR50jJ/DXBhi8wAgSe8Yp0ef1+vsirCrzWrYr6VExI6vuDSEOW4QrA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=sifive.com; spf=pass smtp.mailfrom=sifive.com; dkim=pass (2048-bit key) header.d=sifive.com header.i=@sifive.com header.b=Y5PTzjo+; arc=none smtp.client-ip=209.85.210.182 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=sifive.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=sifive.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=sifive.com header.i=@sifive.com header.b="Y5PTzjo+" Received: by mail-pf1-f182.google.com with SMTP id d2e1a72fcca58-6e571666829so1039247b3a.3 for ; Thu, 29 Feb 2024 15:22:17 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1709248937; x=1709853737; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=1/3iO4NfkALka4Buc8Img0QDYdUTnf5/4SNG0fcN5rQ=; b=Y5PTzjo+xBUgRjntIDSAgAbd24KT29iEbqMGCCLRMTr35b1b3rLtcOwWzYC1SUMGMI X6vxlEWAr2F+jpllNu/1xe2pqgPqZQc3lYoTEGfTj/ocdYencX2QG9so2pRU3rXDGly8 UJsBav/WBvbvCwFFUs+d/2J4TOT5MADKnsqkGN4XjmnmrDYmE1q8xxBJ360dAZsxXcLg D2HX4OmSFrfQthSrs7V6Hf4ChzrL75U2EuNPBX2lfoE8xKaagefK+YFeK+v+MLCSrOfq nXFDSnclrkt3oJ+YFhyA1wGvpwSjzDD9B+wQNpK/36sMScXN1C7leNHfz1dpUxzCIzJ5 dKSQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709248937; x=1709853737; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=1/3iO4NfkALka4Buc8Img0QDYdUTnf5/4SNG0fcN5rQ=; b=beDnSaIJslE0SpK6II225n00JqQp3ChkrtBOCRPIRYNvV5uF3eFHCSdOp/QK9W8GA5 xNyFDpHmmATMIMEZF0nIV+2lAAiqnnI3ckALXZXz9XMFwWG1Xa4txSZZbiaa2qEHLS9T zM/rem/a5t+ZqU58K8Pf2ryeseHAHB1+dF+b5nD4fLNJRyQ+z7Ia3II4YKRBgZe4VGeh 3HdEJjkcwOteYxMNO7bstMYixoBhgtS/xhD+3aSlCfRx2Vhi6H3Z3pI0hvhH6nKJNtD9 sZgvXl0BXZn3zQV4z1srig/RJ81p5BCBxS8/y+x2c48rENauIew7gBTBU6UoaSwGQRoI 8RXQ== X-Gm-Message-State: AOJu0Yz1RtM8T4iojcH541GjTslcQI2AJUM0VHQwAH8F+SsSJ/548Jjo bYTuqY+Z2bYoPg7/svz6ZmJWYYWfC5dHESz/0r2nmdRX9YU8a2NcHRIciy8NCgo= X-Google-Smtp-Source: AGHT+IHEWuHXVZ/LoNCuQjq9ikFwqlpYM8ivEnjNJqMVh5lQ2hbDE0NnP51hssPDjMbEUOy86+sdSg== X-Received: by 2002:a05:6a00:14cf:b0:6e5:a996:5201 with SMTP id w15-20020a056a0014cf00b006e5a9965201mr172471pfu.10.1709248937282; Thu, 29 Feb 2024 15:22:17 -0800 (PST) Received: from sw06.internal.sifive.com ([4.53.31.132]) by smtp.gmail.com with ESMTPSA id c6-20020aa78806000000b006e55aa75d6csm1779719pfo.122.2024.02.29.15.22.16 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 29 Feb 2024 15:22:16 -0800 (PST) From: Samuel Holland To: Palmer Dabbelt , linux-riscv@lists.infradead.org Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, Alexandre Ghiti , Jisheng Zhang , Yunhui Cui , Samuel Holland Subject: [PATCH v5 04/13] riscv: mm: Broadcast kernel TLB flushes only when needed Date: Thu, 29 Feb 2024 15:21:45 -0800 Message-ID: <20240229232211.161961-5-samuel.holland@sifive.com> X-Mailer: git-send-email 2.43.1 In-Reply-To: <20240229232211.161961-1-samuel.holland@sifive.com> References: <20240229232211.161961-1-samuel.holland@sifive.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" __flush_tlb_range() avoids broadcasting TLB flushes when an mm context is only active on the local CPU. Apply this same optimization to TLB flushes of kernel memory when only one CPU is online. This check can be constant-folded when SMP is disabled. Reviewed-by: Alexandre Ghiti Signed-off-by: Samuel Holland --- (no changes since v4) Changes in v4: - New patch for v4 arch/riscv/mm/tlbflush.c | 18 +++++------------- 1 file changed, 5 insertions(+), 13 deletions(-) diff --git a/arch/riscv/mm/tlbflush.c b/arch/riscv/mm/tlbflush.c index 0373661bd1c4..8cdb082f00ca 100644 --- a/arch/riscv/mm/tlbflush.c +++ b/arch/riscv/mm/tlbflush.c @@ -102,22 +102,15 @@ static void __flush_tlb_range(struct cpumask *cmask, = unsigned long asid, unsigned long start, unsigned long size, unsigned long stride) { - bool broadcast; + unsigned int cpu; =20 if (cpumask_empty(cmask)) return; =20 - if (cmask !=3D cpu_online_mask) { - unsigned int cpuid; + cpu =3D get_cpu(); =20 - cpuid =3D get_cpu(); - /* check if the tlbflush needs to be sent to other CPUs */ - broadcast =3D cpumask_any_but(cmask, cpuid) < nr_cpu_ids; - } else { - broadcast =3D true; - } - - if (!broadcast) { + /* Check if the TLB flush needs to be sent to other CPUs. */ + if (cpumask_any_but(cmask, cpu) >=3D nr_cpu_ids) { local_flush_tlb_range_asid(start, size, stride, asid); } else if (riscv_use_sbi_for_rfence()) { sbi_remote_sfence_vma_asid(cmask, start, size, asid); @@ -131,8 +124,7 @@ static void __flush_tlb_range(struct cpumask *cmask, un= signed long asid, on_each_cpu_mask(cmask, __ipi_flush_tlb_range_asid, &ftd, 1); } =20 - if (cmask !=3D cpu_online_mask) - put_cpu(); + put_cpu(); } =20 static inline unsigned long get_mm_asid(struct mm_struct *mm) --=20 2.43.1 From nobody Sun Feb 8 20:22:57 2026 Received: from mail-oi1-f180.google.com (mail-oi1-f180.google.com [209.85.167.180]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 616D713E7DC for ; Thu, 29 Feb 2024 23:22:19 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.167.180 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709248940; cv=none; b=M78XxQ+325ihQdQ90g+8gBZo5wiA+wANf8GCPsNliuLAZifhhBX+XiTO+sidDd+ucbcBCkyOMfrnkS4lHcY/g9a6gkoo57XHsoNbUfISB3GijbAEOJ6nMU6/4+YZzht09APRjnsHqbCzaD+kcOvT8xT++8RLwB/CZTjmURXOgJ8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709248940; c=relaxed/simple; bh=a0/O/6KXaeNjvwFlcxHlcnSxOyEoC+55OmJ2xLa9M+Y=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=MiywI4aHJUicxdwDw9xi5QxPzx/mEHdM7PyHWQNBPArlbTkjHWBGXg5nKbHHpCPzjKbodsCaTZwMXppL0G9Y6kzR3CQ1s3NiIkQ8MUukcnvMKafAvEzJ/9s/dkYUx0nihPHa5NMgoKWlbpu61gbJn/aNyp8sXBg5hABWau2EXak= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=sifive.com; spf=pass smtp.mailfrom=sifive.com; dkim=pass (2048-bit key) header.d=sifive.com header.i=@sifive.com header.b=R4zqQOQx; arc=none smtp.client-ip=209.85.167.180 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=sifive.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=sifive.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=sifive.com header.i=@sifive.com header.b="R4zqQOQx" Received: by mail-oi1-f180.google.com with SMTP id 5614622812f47-3bbbc6bcc78so1082695b6e.1 for ; Thu, 29 Feb 2024 15:22:19 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1709248938; x=1709853738; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=9YO6O944kqWb4oNKtEMwGtpxpVk1nDXsHdzKgYw/NyE=; b=R4zqQOQxA6YWW/2eNFx52MOxRUF7TQJLI3sZyM+IQyqZRzS7REEFcHh9dk6lFH6MA4 ijyUzDrxTfTfJswzq5E39MyjcK2Op4YmBvp0he30ds6rJgkHNMfE9qjFzmf5mJoYdFkA vD594eHOfPd+SYN4rNvn9eEIjOrSqOHjMq2A1uAMQ3r+ZjxNCYyrWD8nrnDBdX7kzBfX riS8VwaZRZ71UibaobFzpHksyB7OX+FbFygcuTx1nX7z70FBIQRwKpBlXkPFTrSZiz4o K4UVsRf6elLXWnwqf2cVY3K8RJun/kXoqKnsheqV7uSCQDV3bBjmNlNr2gL92hoqJgC6 MaZw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709248938; x=1709853738; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=9YO6O944kqWb4oNKtEMwGtpxpVk1nDXsHdzKgYw/NyE=; b=szyYA0EenaxhV2SxejIepnW9SAHLfmgUAIdQNTvnBF9Resj7zhEnNvHJQPAnJ3rEC1 Drd+8HVj3l8x7z20Ai/It+3b2OnGQaYNIUR5wkKLDarkdPotA2gjWkmWjPf9wDZmgtUv Y8eVXJflbdiBlDaHg3lIwGfPYXpmQPWaSh0UrPTHS6AoG8gtz2eCEIsw/cHd/5Xkf2F2 vuRKQOMW/ZpVR00Oeu7eQs+gjQCwXPl2Owaa1v0YqV3uH576xFPNnnMMqMqQ6OVnvIqa +AGQZ3vWOCnPMnz0JZI5MQdbAVXq/wFOo6Dwvc8ZyrDh1mdAi/Zj5Mv8xty32Q3pwtgA hpxw== X-Gm-Message-State: AOJu0YyHhq8oUtuCStsuajzFuTxE7f4nnYwGJ87eLdxMfF+Sfh3oYCrt b8HtGNPWnHIzU3/CfsFnZKTNUX4KFfCBhsQsYmmTqK9cXb+jqpyZSgled41hEKQ3MODl2wb1hvY v X-Google-Smtp-Source: AGHT+IF/+PdA1OOJxMn23/EE1kPZ70u92KMkSqj19wYtg5glHTCWIaPOP2OtxcWfToq1z4V1DP9b6Q== X-Received: by 2002:a05:6808:8f2:b0:3c1:c37b:1a71 with SMTP id d18-20020a05680808f200b003c1c37b1a71mr62346oic.57.1709248938472; Thu, 29 Feb 2024 15:22:18 -0800 (PST) Received: from sw06.internal.sifive.com ([4.53.31.132]) by smtp.gmail.com with ESMTPSA id c6-20020aa78806000000b006e55aa75d6csm1779719pfo.122.2024.02.29.15.22.17 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 29 Feb 2024 15:22:17 -0800 (PST) From: Samuel Holland To: Palmer Dabbelt , linux-riscv@lists.infradead.org Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, Alexandre Ghiti , Jisheng Zhang , Yunhui Cui , Samuel Holland Subject: [PATCH v5 05/13] riscv: Only send remote fences when some other CPU is online Date: Thu, 29 Feb 2024 15:21:46 -0800 Message-ID: <20240229232211.161961-6-samuel.holland@sifive.com> X-Mailer: git-send-email 2.43.1 In-Reply-To: <20240229232211.161961-1-samuel.holland@sifive.com> References: <20240229232211.161961-1-samuel.holland@sifive.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" If no other CPU is online, a local cache or TLB flush is sufficient. These checks can be constant-folded when SMP is disabled. Signed-off-by: Samuel Holland --- (no changes since v4) Changes in v4: - New patch for v4 arch/riscv/mm/cacheflush.c | 4 +++- arch/riscv/mm/tlbflush.c | 4 +++- 2 files changed, 6 insertions(+), 2 deletions(-) diff --git a/arch/riscv/mm/cacheflush.c b/arch/riscv/mm/cacheflush.c index 47c485bc7df0..f7933ae88a55 100644 --- a/arch/riscv/mm/cacheflush.c +++ b/arch/riscv/mm/cacheflush.c @@ -21,7 +21,9 @@ void flush_icache_all(void) { local_flush_icache_all(); =20 - if (riscv_use_sbi_for_rfence()) + if (num_online_cpus() < 2) + return; + else if (riscv_use_sbi_for_rfence()) sbi_remote_fence_i(NULL); else on_each_cpu(ipi_remote_fence_i, NULL, 1); diff --git a/arch/riscv/mm/tlbflush.c b/arch/riscv/mm/tlbflush.c index 8cdb082f00ca..69402c260a89 100644 --- a/arch/riscv/mm/tlbflush.c +++ b/arch/riscv/mm/tlbflush.c @@ -78,7 +78,9 @@ static void __ipi_flush_tlb_all(void *info) =20 void flush_tlb_all(void) { - if (riscv_use_sbi_for_rfence()) + if (num_online_cpus() < 2) + local_flush_tlb_all(); + else if (riscv_use_sbi_for_rfence()) sbi_remote_sfence_vma_asid(NULL, 0, FLUSH_TLB_MAX_SIZE, FLUSH_TLB_NO_ASI= D); else on_each_cpu(__ipi_flush_tlb_all, NULL, 1); --=20 2.43.1 From nobody Sun Feb 8 20:22:57 2026 Received: from mail-pf1-f182.google.com (mail-pf1-f182.google.com [209.85.210.182]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 2CD4213E7FD for ; Thu, 29 Feb 2024 23:22:19 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.182 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709248941; cv=none; b=Fb7kQiW6/87WMaBk7Rn52vVM77ZNXfCG5kjiYpJniqYzqNenWydmsBSn4o7hwup7wKssUTDaIc3kx8J5JC/4F3q+PoF7zRboLU6uAdMcO1MTJ4+lxo37JQJuKERzFD/hvUIC56mlSh23iMP7n8vKKP8iNIr2TpOO5Av4QT1G2JE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709248941; c=relaxed/simple; bh=a4goP/+lcARBu1PHrcIVBscDySvDXJckpo1nEsaZBxk=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=cZSNeBvbrxx4Pm8PRz7meBacG8pmFqlrWtb0c6DFaiHZh72zDtC7waFryFdcC2q3tE6TFNWPedGY2mXgy+EoiAb+PHNoOiRWWLXBj1FCmaILo/PyUCQvtUndet7HIGU5SQN5McuhRGfCjucCIwK/cYXQsn7eBl2SWLso4NldTPU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=sifive.com; spf=pass smtp.mailfrom=sifive.com; dkim=pass (2048-bit key) header.d=sifive.com header.i=@sifive.com header.b=T/6413Kz; arc=none smtp.client-ip=209.85.210.182 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=sifive.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=sifive.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=sifive.com header.i=@sifive.com header.b="T/6413Kz" Received: by mail-pf1-f182.google.com with SMTP id d2e1a72fcca58-6e558a67f70so1504583b3a.0 for ; Thu, 29 Feb 2024 15:22:19 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1709248939; x=1709853739; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=uLB9vlYe9JfwQMhDb14nNgv57tMyr8X6yVrB4e0zcQU=; b=T/6413KzX+8h2XvpIH5v6dAupdJp9nBlSmldX5ofq7vIevhizsmcZcHduKAW5H3ahf 6Oyf9Sb2ae4Crq6iEPW33X9w1sr/hDMAIlkdqIu67a9OKb4Ao7RWVKlxRqO1emaGmxeM zjJpdNQwp1NdazZI8idHbGY1Auv5AqybsrkC5NyzbC2P2csGYNOWZZR5Lvnq8cqlg5im 9tY8BKKufyGRGcB9XZSk/ENFLD5a9pbTzx7kIiI5qJD3Vpoq4P9NCkubUc+rfVopLG/B +DO20kKFjrOaGOAgjxbdayLO4Mye5aheHHdcGvFa1DFyO6kvedlRT3tgC3/6OKIQh1Ai ADig== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709248939; x=1709853739; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=uLB9vlYe9JfwQMhDb14nNgv57tMyr8X6yVrB4e0zcQU=; b=m7XaQcPRpKWkcsufoJlTxY6FyUO62TC1fL2L5QRhx08rJJn1oscxNYA7vYmGt8H0WV XbnejA8cofNoZhQyCItSa8cOga1rQ5MIfyclTGcPDR+MuODFwgvhWZKVuClsrqSA1ch/ dzbm3sN/F1aPJS9BvzFARZZ1b/tvJ3Fr1vqJYPGzo5L6c13Io1Gntzk1cczbNrOe6X5+ yxGKUntYZSrS+I9eEXFdpw+6a55lkTqgS5SmJD7CqHRQ2Ts9sh1BGkq0n/JNXTxCfqZP 16DmLaaSI8yWhs84dYWlBgaueGahUybNC/5HcrN6KdOtM8NmeIX02aBGnEQDuC7T9JrN vqfA== X-Gm-Message-State: AOJu0YxLAALlXQSZlJ4v4wqxLMSCRIBAGeOT7lOBW28W/iXsL0rbNzpT i2B2ENa4Xu3OlkQtJUlmCDUPtlHbQg0WXKTNq0cUn2+Ap3UZRwCybmBoUkGi5YI= X-Google-Smtp-Source: AGHT+IFNMqThgNRh63sPPm1+a+RZC7oBqAsJKeQJ9ncYhwlwuRqR+WMAL4ZrZQk20TebpNdIS4mveg== X-Received: by 2002:a05:6a00:2354:b0:6e5:80a4:2ff2 with SMTP id j20-20020a056a00235400b006e580a42ff2mr152068pfj.30.1709248939492; Thu, 29 Feb 2024 15:22:19 -0800 (PST) Received: from sw06.internal.sifive.com ([4.53.31.132]) by smtp.gmail.com with ESMTPSA id c6-20020aa78806000000b006e55aa75d6csm1779719pfo.122.2024.02.29.15.22.18 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 29 Feb 2024 15:22:19 -0800 (PST) From: Samuel Holland To: Palmer Dabbelt , linux-riscv@lists.infradead.org Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, Alexandre Ghiti , Jisheng Zhang , Yunhui Cui , Samuel Holland Subject: [PATCH v5 06/13] riscv: mm: Combine the SMP and UP TLB flush code Date: Thu, 29 Feb 2024 15:21:47 -0800 Message-ID: <20240229232211.161961-7-samuel.holland@sifive.com> X-Mailer: git-send-email 2.43.1 In-Reply-To: <20240229232211.161961-1-samuel.holland@sifive.com> References: <20240229232211.161961-1-samuel.holland@sifive.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" In SMP configurations, all TLB flushing narrower than flush_tlb_all() goes through __flush_tlb_range(). Do the same in UP configurations. This allows UP configurations to take advantage of recent improvements to the code in tlbflush.c, such as support for huge pages and flushing multiple-page ranges. Reviewed-by: Alexandre Ghiti Signed-off-by: Samuel Holland --- (no changes since v4) Changes in v4: - Merge the two copies of __flush_tlb_range() and rely on the compiler to optimize out the broadcast path (both clang and gcc do this) - Merge the two copies of flush_tlb_all() and rely on constant folding Changes in v2: - Move the SMP/UP merge earlier in the series to avoid build issues - Make a copy of __flush_tlb_range() instead of adding ifdefs inside - local_flush_tlb_all() is the only function used on !MMU (smpboot.c) arch/riscv/Kconfig | 2 +- arch/riscv/include/asm/tlbflush.h | 30 +++--------------------------- arch/riscv/mm/Makefile | 5 +---- 3 files changed, 5 insertions(+), 32 deletions(-) diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig index 0bfcfec67ed5..de9b6f2279ff 100644 --- a/arch/riscv/Kconfig +++ b/arch/riscv/Kconfig @@ -60,7 +60,7 @@ config RISCV select ARCH_USE_MEMTEST select ARCH_USE_QUEUED_RWLOCKS select ARCH_USES_CFI_TRAPS if CFI_CLANG - select ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH if SMP && MMU + select ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH if MMU select ARCH_WANT_DEFAULT_TOPDOWN_MMAP_LAYOUT if MMU select ARCH_WANT_FRAME_POINTERS select ARCH_WANT_GENERAL_HUGETLB if !RISCV_ISA_SVNAPOT diff --git a/arch/riscv/include/asm/tlbflush.h b/arch/riscv/include/asm/tlb= flush.h index 928f096dca21..4f86424b1ba5 100644 --- a/arch/riscv/include/asm/tlbflush.h +++ b/arch/riscv/include/asm/tlbflush.h @@ -27,12 +27,7 @@ static inline void local_flush_tlb_page(unsigned long ad= dr) { ALT_FLUSH_TLB_PAGE(__asm__ __volatile__ ("sfence.vma %0" : : "r" (addr) := "memory")); } -#else /* CONFIG_MMU */ -#define local_flush_tlb_all() do { } while (0) -#define local_flush_tlb_page(addr) do { } while (0) -#endif /* CONFIG_MMU */ =20 -#if defined(CONFIG_SMP) && defined(CONFIG_MMU) void flush_tlb_all(void); void flush_tlb_mm(struct mm_struct *mm); void flush_tlb_mm_range(struct mm_struct *mm, unsigned long start, @@ -54,27 +49,8 @@ void arch_tlbbatch_add_pending(struct arch_tlbflush_unma= p_batch *batch, unsigned long uaddr); void arch_flush_tlb_batched_pending(struct mm_struct *mm); void arch_tlbbatch_flush(struct arch_tlbflush_unmap_batch *batch); - -#else /* CONFIG_SMP && CONFIG_MMU */ - -#define flush_tlb_all() local_flush_tlb_all() -#define flush_tlb_page(vma, addr) local_flush_tlb_page(addr) - -static inline void flush_tlb_range(struct vm_area_struct *vma, - unsigned long start, unsigned long end) -{ - local_flush_tlb_all(); -} - -/* Flush a range of kernel pages */ -static inline void flush_tlb_kernel_range(unsigned long start, - unsigned long end) -{ - local_flush_tlb_all(); -} - -#define flush_tlb_mm(mm) flush_tlb_all() -#define flush_tlb_mm_range(mm, start, end, page_size) flush_tlb_all() -#endif /* !CONFIG_SMP || !CONFIG_MMU */ +#else /* CONFIG_MMU */ +#define local_flush_tlb_all() do { } while (0) +#endif /* CONFIG_MMU */ =20 #endif /* _ASM_RISCV_TLBFLUSH_H */ diff --git a/arch/riscv/mm/Makefile b/arch/riscv/mm/Makefile index 2c869f8026a8..cbe4d775ef56 100644 --- a/arch/riscv/mm/Makefile +++ b/arch/riscv/mm/Makefile @@ -13,14 +13,11 @@ endif KCOV_INSTRUMENT_init.o :=3D n =20 obj-y +=3D init.o -obj-$(CONFIG_MMU) +=3D extable.o fault.o pageattr.o pgtable.o +obj-$(CONFIG_MMU) +=3D extable.o fault.o pageattr.o pgtable.o tlbflush.o obj-y +=3D cacheflush.o obj-y +=3D context.o obj-y +=3D pmem.o =20 -ifeq ($(CONFIG_MMU),y) -obj-$(CONFIG_SMP) +=3D tlbflush.o -endif obj-$(CONFIG_HUGETLB_PAGE) +=3D hugetlbpage.o obj-$(CONFIG_PTDUMP_CORE) +=3D ptdump.o obj-$(CONFIG_KASAN) +=3D kasan_init.o --=20 2.43.1 From nobody Sun Feb 8 20:22:57 2026 Received: from mail-ot1-f51.google.com (mail-ot1-f51.google.com [209.85.210.51]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7766B143720 for ; Thu, 29 Feb 2024 23:22:21 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.51 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709248943; cv=none; b=s5QBXL9PgzljNNAjLlL6RkCPmuniqwz/0Ze+tGqJ3rqy40wzB79A2lEeXOCpx9wxjMFXibUKC5cfS3A/VXtoI2dVjWepzlswSOKYiWzOiBEqCG7rPXrPh0jxdlpo289LJE6OEgxVZvOwICc6hGhb1bjh9KY/vZk8xAqU+mYR1Z0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709248943; c=relaxed/simple; bh=mxWZWiyysmkBFHfNnHhHfYVQO5Mh4OcNRTXlF0LxGbA=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=t2P+EoqVJrYGq6yDFbSNJNYUxnbRlXu/y9q6F5JPWuy/2j2AJFQ5IA5WuAd+5haam40ueXJ7HwR2cF6CsqKz1iv+O0NkS+4jOChaY5rVoCNlq5Mp8Xrd/z35ev4JtItJV40lwWpjX/d3h0vCUL7eGI5z4PcBpsjhLrcrbyHKMZM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=sifive.com; spf=pass smtp.mailfrom=sifive.com; dkim=pass (2048-bit key) header.d=sifive.com header.i=@sifive.com header.b=k6LDVrnp; arc=none smtp.client-ip=209.85.210.51 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=sifive.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=sifive.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=sifive.com header.i=@sifive.com header.b="k6LDVrnp" Received: by mail-ot1-f51.google.com with SMTP id 46e09a7af769-6e49812ba77so722927a34.2 for ; Thu, 29 Feb 2024 15:22:21 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1709248940; x=1709853740; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=4jEC1X5CZdJa4ryFg9hWmhSsIovSPBi+ZbblNH/6mwk=; b=k6LDVrnpThZ8kscB1/zh8r7zr/HU+vP/rlweyGAIXTA+x6BUdHnjrbbqh3Fg0TRtbt 4U0JGHQlI7Q1yv9VsJ/Xjn0KK4NIA3o3x6IXPLwhUf3mFyj5scNI3EmajkuGgWHX6xU8 jy1XOcrTDFHK+Ma0fG+gNobfVutCHNMcaUyBTBJUpqzTBtrquZXZIzMeumznZZZurM6d ISK+cjUeFvHEgjCV6oISSwnznccB6ovGfNcMgVKPLdOTFwqvoMa1ql4PpfMxC+lixQiP VpVjYNfHcm+TuakQlNtv6n6che7Y1vjG9nkH7zrKrTxBn7PmjxbUPB8wF+pcXYoCHhIn q22g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709248940; x=1709853740; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=4jEC1X5CZdJa4ryFg9hWmhSsIovSPBi+ZbblNH/6mwk=; b=EPPtPxhAvBA+lTAFirFw+VIOFxwAS6BrFp3lzAWPItbTUCsjog07P6f7SQD60hctED 1sV53uMTPy/PErDH9QJ3qQeOdJ4MpXNtQgCHDkRBZJkCwOLR74UAzsemh9OhA6L29WWr RsvdoIfDioMk1FJ9LVl6m4EyKyvKcWw6FrftQgeg0YXXvkGNc7h2d7hb/j4dV6w7xKzp EWD474IyJT2ocLq/uZkcm1jJ9IWwB7EjX7pDNIB+Fdh0yTTxDregPGckzzk6amf6iCyW oPE/CdmO48GsIFkFgPuwiCt59c7R1v+IyieFc+/Ohhy7vr/IZfWtoz5k4zqjgBr2ZvZA dLsQ== X-Gm-Message-State: AOJu0YwEB6dY0OyVoGSVBDEoyq++Lpv8//lIdB/7dAPy9mKclg/QPQfL 2ZX9pROTUBDJcckafLKTgNECVT6O+hxVj7IX627dPA5uj4AWDc3xti8RAdZUvDo= X-Google-Smtp-Source: AGHT+IGX4/HV5W58fetZh8Yw57gYr75gd/4aFWyaezvIp/BsV5LakI2yn+mW+sbVnuOSeLnU1moubA== X-Received: by 2002:a05:6870:ac22:b0:21e:dd7a:2d3e with SMTP id kw34-20020a056870ac2200b0021edd7a2d3emr4104469oab.22.1709248940592; Thu, 29 Feb 2024 15:22:20 -0800 (PST) Received: from sw06.internal.sifive.com ([4.53.31.132]) by smtp.gmail.com with ESMTPSA id c6-20020aa78806000000b006e55aa75d6csm1779719pfo.122.2024.02.29.15.22.19 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 29 Feb 2024 15:22:20 -0800 (PST) From: Samuel Holland To: Palmer Dabbelt , linux-riscv@lists.infradead.org Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, Alexandre Ghiti , Jisheng Zhang , Yunhui Cui , Samuel Holland Subject: [PATCH v5 07/13] riscv: Apply SiFive CIP-1200 workaround to single-ASID sfence.vma Date: Thu, 29 Feb 2024 15:21:48 -0800 Message-ID: <20240229232211.161961-8-samuel.holland@sifive.com> X-Mailer: git-send-email 2.43.1 In-Reply-To: <20240229232211.161961-1-samuel.holland@sifive.com> References: <20240229232211.161961-1-samuel.holland@sifive.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" commit 3f1e782998cd ("riscv: add ASID-based tlbflushing methods") added calls to the sfence.vma instruction with rs2 !=3D x0. These single-ASID instruction variants are also affected by SiFive errata CIP-1200. Until now, the errata workaround was not needed for the single-ASID sfence.vma variants, because they were only used when the ASID allocator was enabled, and the affected SiFive platforms do not support multiple ASIDs. However, we are going to start using those sfence.vma variants regardless of ASID support, so now we need alternatives covering them. Signed-off-by: Samuel Holland --- (no changes since v2) Changes in v2: - Rebase on Alexandre's "riscv: tlb flush improvements" series v5 arch/riscv/include/asm/errata_list.h | 12 +++++++++++- arch/riscv/include/asm/tlbflush.h | 19 ++++++++++++++++++- arch/riscv/mm/tlbflush.c | 23 ----------------------- 3 files changed, 29 insertions(+), 25 deletions(-) diff --git a/arch/riscv/include/asm/errata_list.h b/arch/riscv/include/asm/= errata_list.h index ea33288f8a25..6e2762101968 100644 --- a/arch/riscv/include/asm/errata_list.h +++ b/arch/riscv/include/asm/errata_list.h @@ -43,11 +43,21 @@ ALTERNATIVE(__stringify(RISCV_PTR do_page_fault), \ CONFIG_ERRATA_SIFIVE_CIP_453) #else /* !__ASSEMBLY__ */ =20 -#define ALT_FLUSH_TLB_PAGE(x) \ +#define ALT_SFENCE_VMA_ASID(asid) \ +asm(ALTERNATIVE("sfence.vma x0, %0", "sfence.vma", SIFIVE_VENDOR_ID, \ + ERRATA_SIFIVE_CIP_1200, CONFIG_ERRATA_SIFIVE_CIP_1200) \ + : : "r" (asid) : "memory") + +#define ALT_SFENCE_VMA_ADDR(addr) \ asm(ALTERNATIVE("sfence.vma %0", "sfence.vma", SIFIVE_VENDOR_ID, \ ERRATA_SIFIVE_CIP_1200, CONFIG_ERRATA_SIFIVE_CIP_1200) \ : : "r" (addr) : "memory") =20 +#define ALT_SFENCE_VMA_ADDR_ASID(addr, asid) \ +asm(ALTERNATIVE("sfence.vma %0, %1", "sfence.vma", SIFIVE_VENDOR_ID, \ + ERRATA_SIFIVE_CIP_1200, CONFIG_ERRATA_SIFIVE_CIP_1200) \ + : : "r" (addr), "r" (asid) : "memory") + /* * _val is marked as "will be overwritten", so need to set it to 0 * in the default case. diff --git a/arch/riscv/include/asm/tlbflush.h b/arch/riscv/include/asm/tlb= flush.h index 4f86424b1ba5..463b615d7728 100644 --- a/arch/riscv/include/asm/tlbflush.h +++ b/arch/riscv/include/asm/tlbflush.h @@ -22,10 +22,27 @@ static inline void local_flush_tlb_all(void) __asm__ __volatile__ ("sfence.vma" : : : "memory"); } =20 +static inline void local_flush_tlb_all_asid(unsigned long asid) +{ + if (asid !=3D FLUSH_TLB_NO_ASID) + ALT_SFENCE_VMA_ASID(asid); + else + local_flush_tlb_all(); +} + /* Flush one page from local TLB */ static inline void local_flush_tlb_page(unsigned long addr) { - ALT_FLUSH_TLB_PAGE(__asm__ __volatile__ ("sfence.vma %0" : : "r" (addr) := "memory")); + ALT_SFENCE_VMA_ADDR(addr); +} + +static inline void local_flush_tlb_page_asid(unsigned long addr, + unsigned long asid) +{ + if (asid !=3D FLUSH_TLB_NO_ASID) + ALT_SFENCE_VMA_ADDR_ASID(addr, asid); + else + local_flush_tlb_page(addr); } =20 void flush_tlb_all(void); diff --git a/arch/riscv/mm/tlbflush.c b/arch/riscv/mm/tlbflush.c index 69402c260a89..365e0a0e4725 100644 --- a/arch/riscv/mm/tlbflush.c +++ b/arch/riscv/mm/tlbflush.c @@ -7,29 +7,6 @@ #include #include =20 -static inline void local_flush_tlb_all_asid(unsigned long asid) -{ - if (asid !=3D FLUSH_TLB_NO_ASID) - __asm__ __volatile__ ("sfence.vma x0, %0" - : - : "r" (asid) - : "memory"); - else - local_flush_tlb_all(); -} - -static inline void local_flush_tlb_page_asid(unsigned long addr, - unsigned long asid) -{ - if (asid !=3D FLUSH_TLB_NO_ASID) - __asm__ __volatile__ ("sfence.vma %0, %1" - : - : "r" (addr), "r" (asid) - : "memory"); - else - local_flush_tlb_page(addr); -} - /* * Flush entire TLB if number of entries to be flushed is greater * than the threshold below. --=20 2.43.1 From nobody Sun Feb 8 20:22:57 2026 Received: from mail-pf1-f178.google.com (mail-pf1-f178.google.com [209.85.210.178]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4883C14373B for ; Thu, 29 Feb 2024 23:22:22 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.178 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709248943; cv=none; b=AMmtoKVpn2ZrS94o/+gH6D2FXUeSyVMGapQ6Tb2SIcHdY4Q/JPwolEmZrDvUf545bahtru8ZGcO2lBW4FOAay/VYwcLlpSjUDb66u0fhUtWENRshSwGoPRwc3wAwVOE5mMXql0IUpIxB6twWSbsR9gnjbXLp6RweY/4Vt71TAP8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709248943; c=relaxed/simple; bh=jkqIBDvnCBgBGk4k5FZq/mUWCLaMf0fOgLBnB4HUugQ=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=bRnNmwiKAtb6EWjj3ws0iEtL3okN0omImDgmcJ87Oh+MqbGbTEMEyyws/2bLuRHlSAI83+1wG/1+k+0xlAzvYFR9zFjakefpidBkXfUrR+18xRvF6de3Jl9SGwiHJMcodSI317Xqkjz3Obb3Rwx6VYpcso4GVCOUH8ClCJg+lMM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=sifive.com; spf=pass smtp.mailfrom=sifive.com; dkim=pass (2048-bit key) header.d=sifive.com header.i=@sifive.com header.b=Mk6byHHF; arc=none smtp.client-ip=209.85.210.178 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=sifive.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=sifive.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=sifive.com header.i=@sifive.com header.b="Mk6byHHF" Received: by mail-pf1-f178.google.com with SMTP id d2e1a72fcca58-6e5675f2ca2so977775b3a.0 for ; Thu, 29 Feb 2024 15:22:22 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1709248942; x=1709853742; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=wHMbuRyYmPRZ8L34XsarFB0wBhmLHt6PApeEi+FQBPY=; b=Mk6byHHFJBfEZXYBz3yaFeDsYmhz51BLgR3FLsX2wkqi9auwAdRoNcOz4xF2PHrJqv xpWBYapxHBuxwdwbv5akbzUDAUGoJfkzhIj3XsbYQwZXuWaSgOVG09gQ8a6LItOl2xzK dYfdaLp9AqFTyOH2mKXHQG0vxaVPG53NTPp/nE1jIF6/R07qNph5W1vRYqGaGGX8XB8c DbQl6t7ayGZnf58pIQ2dPCgG20zvHSOGAyBClPTMgwpkBHSjpyrGRSsMsPvgORGsTbaM yORJYsEf6I9qaWMPUYERuZuUerMIkity5L+Qr5CDL12kVBfYTiAAw/8gFtt76NuCOimc VDkw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709248942; x=1709853742; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=wHMbuRyYmPRZ8L34XsarFB0wBhmLHt6PApeEi+FQBPY=; b=RNtprg5gYgEGTMm+1odHO43NhOs8q+JpancBHY1l9h4cnqskjPfwtQm2/SFH+nNXVh DSVXN1dQBqJjTMu5jCy6K+KtZ2JfzNU2nC0wW2BTf//85P2XRP3U0FRuwoiq/QxNh3I3 IQbjljyO0AB0Mbw6mV0/wZ39wgFVAsrhWx964sODVDJS/2YjCDdMUcdHfom8jW9GxGbh lJ/gkC9TJG96XhjiJbR9SGpgMYHELBLyxLbyMpohliG6Kn1SVJs90X7rPGOEJ9NDhdGP IUJ/RYOpa6RwU+aYycHFbBHu95K9KCdbKgeCNv/W2SlpaBmBme0tXi1EQx92XzNWcM1A cfaA== X-Gm-Message-State: AOJu0YxDAlUp3bvIQWrpRLJMxoO43jvf4JCb+mq/IkHKoEmGxk4c0eBe yoak3kmr4/pZJMQBZMH752XJTewC4GssEjyh2w842YNL/pu2bKlowj7myy37NGI= X-Google-Smtp-Source: AGHT+IE0KT3RuIOZoiwK4WAulPfaiFPRjhF12PREr/S+YlSI1LgFKc32EbVKWDUbhqJfm1349m8bOA== X-Received: by 2002:a05:6a00:1955:b0:6e4:eb29:357 with SMTP id s21-20020a056a00195500b006e4eb290357mr165166pfk.12.1709248941678; Thu, 29 Feb 2024 15:22:21 -0800 (PST) Received: from sw06.internal.sifive.com ([4.53.31.132]) by smtp.gmail.com with ESMTPSA id c6-20020aa78806000000b006e55aa75d6csm1779719pfo.122.2024.02.29.15.22.20 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 29 Feb 2024 15:22:21 -0800 (PST) From: Samuel Holland To: Palmer Dabbelt , linux-riscv@lists.infradead.org Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, Alexandre Ghiti , Jisheng Zhang , Yunhui Cui , Samuel Holland Subject: [PATCH v5 08/13] riscv: Avoid TLB flush loops when affected by SiFive CIP-1200 Date: Thu, 29 Feb 2024 15:21:49 -0800 Message-ID: <20240229232211.161961-9-samuel.holland@sifive.com> X-Mailer: git-send-email 2.43.1 In-Reply-To: <20240229232211.161961-1-samuel.holland@sifive.com> References: <20240229232211.161961-1-samuel.holland@sifive.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Since implementations affected by SiFive errata CIP-1200 always use the global variant of the sfence.vma instruction, they only need to execute the instruction once. The range-based loop only hurts performance. Signed-off-by: Samuel Holland --- (no changes since v4) Changes in v4: - Only set tlb_flush_all_threshold when CONFIG_MMU=3Dy. Changes in v3: - New patch for v3 arch/riscv/errata/sifive/errata.c | 5 +++++ arch/riscv/include/asm/tlbflush.h | 2 ++ arch/riscv/mm/tlbflush.c | 2 +- 3 files changed, 8 insertions(+), 1 deletion(-) diff --git a/arch/riscv/errata/sifive/errata.c b/arch/riscv/errata/sifive/e= rrata.c index 3d9a32d791f7..716cfedad3a2 100644 --- a/arch/riscv/errata/sifive/errata.c +++ b/arch/riscv/errata/sifive/errata.c @@ -42,6 +42,11 @@ static bool errata_cip_1200_check_func(unsigned long ar= ch_id, unsigned long imp return false; if ((impid & 0xffffff) > 0x200630 || impid =3D=3D 0x1200626) return false; + +#ifdef CONFIG_MMU + tlb_flush_all_threshold =3D 0; +#endif + return true; } =20 diff --git a/arch/riscv/include/asm/tlbflush.h b/arch/riscv/include/asm/tlb= flush.h index 463b615d7728..8e329721375b 100644 --- a/arch/riscv/include/asm/tlbflush.h +++ b/arch/riscv/include/asm/tlbflush.h @@ -66,6 +66,8 @@ void arch_tlbbatch_add_pending(struct arch_tlbflush_unmap= _batch *batch, unsigned long uaddr); void arch_flush_tlb_batched_pending(struct mm_struct *mm); void arch_tlbbatch_flush(struct arch_tlbflush_unmap_batch *batch); + +extern unsigned long tlb_flush_all_threshold; #else /* CONFIG_MMU */ #define local_flush_tlb_all() do { } while (0) #endif /* CONFIG_MMU */ diff --git a/arch/riscv/mm/tlbflush.c b/arch/riscv/mm/tlbflush.c index 365e0a0e4725..22870f213188 100644 --- a/arch/riscv/mm/tlbflush.c +++ b/arch/riscv/mm/tlbflush.c @@ -11,7 +11,7 @@ * Flush entire TLB if number of entries to be flushed is greater * than the threshold below. */ -static unsigned long tlb_flush_all_threshold __read_mostly =3D 64; +unsigned long tlb_flush_all_threshold __read_mostly =3D 64; =20 static void local_flush_tlb_range_threshold_asid(unsigned long start, unsigned long size, --=20 2.43.1 From nobody Sun Feb 8 20:22:57 2026 Received: from mail-pf1-f179.google.com (mail-pf1-f179.google.com [209.85.210.179]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 47AD014375E for ; Thu, 29 Feb 2024 23:22:23 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.179 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709248944; cv=none; b=QExdE5RHg8Zg4i/1yZSK7Ad+/HuBpNTOJ7ioKYbDavJsYupVEHBPTzKtx9rgEr7PhpQvije6K+GXZAqV1cIxF0CkISQaYPq5zh/fTGzFzKvHCE2S+Stab7zTmwOBUv7qfoSK4mGeMOinxxwb4t7W0rpVPKmFULeedN44wssXYW4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709248944; c=relaxed/simple; bh=efpj5keG/fa3VmpEzeoIkOhTduHo/5YrxpRo1zbxoZM=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=idMUxC4E5AYA/Nlmhq8MuRK0aLkeBcFbP3u8lvvOP97OgRCm6bFsEvHEkG647ex9jX4J5jl+B5ipFWtvyRPGb+29iRWUI2WP1XLcyENPcMiUPnlHPMKY0V8MDQJbDodHU4vgc8qKnMuIU10v5r6DMW6bUgZ3ACmVoxVjNPDGXDk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=sifive.com; spf=pass smtp.mailfrom=sifive.com; dkim=pass (2048-bit key) header.d=sifive.com header.i=@sifive.com header.b=nDD9n6Tf; arc=none smtp.client-ip=209.85.210.179 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=sifive.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=sifive.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=sifive.com header.i=@sifive.com header.b="nDD9n6Tf" Received: by mail-pf1-f179.google.com with SMTP id d2e1a72fcca58-6e4d48a5823so1324360b3a.1 for ; Thu, 29 Feb 2024 15:22:23 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1709248943; x=1709853743; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=lukLV3IzObY+fmX3TTuFURPx1t0s+LwQl+LTSkieoBQ=; b=nDD9n6TfJphnkUNaSmOrfdujaHPypiGX9aGZA6i6t64AlqbTkdAi6yucnNRRIHi4VY GnM5p/KerJQ1HyA0erX8qOslXOsO+qQAswlP14ztm07//zR15Gcv61PQ5OypDT51mQs8 UBfvTEjHzOL+BdenBN4zeKwS5bCRL02tiYaffqbAj12jc650Sbtnuj61Dv5TvXVl3uYn uenS/EYm32EadfZZk/86XdeIqkHCGN+gDiA0O7ALZRTHElahTcSmKRbLqKqCyRIwjovt JRkIQ+i0zryWBWyRFMU6PeWmZQeIAA5+jhRQoF3513oh+mziVCNEOUgv0RsSKcYwjdpm DbjA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709248943; x=1709853743; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=lukLV3IzObY+fmX3TTuFURPx1t0s+LwQl+LTSkieoBQ=; b=JX4ZTrTAQArnyeVJdfE+R63Nn+ftKNtb5WI3cLWM2mkiPj+90q0+NjM9SNCQRrq8K8 eNJMFKDCQWdWBb9/BtCIWQRQWXx+NpZr7pzfM5nvMQamGvGXR5AqSHLGVO8Drex04eiC ZISWGkKHF1VGGObD06KaZfRYiT5ACzBksOCBD7S1vEqHMgmSPDizAEnKtx86Yn4RHLPV nuvQKyNyP43DU9JnCf/rMxgbNRLeNX4Lu2H9WslzCiXGkJ0VBDse7webw4Y5TRUCBNPF PsKz5PpXES4TIiRmvbb+dK+6AR4sCLTSEkeZcfx/RquqCRioBqjbzgpxytHv0qUkWnst dWjw== X-Gm-Message-State: AOJu0YwWvl6S3gCeMXMKC3y7WQK1OtmNKmD/0HhdgAlM5kEv6g/wEA55 8eQ6CLLBw8NVXd0LvCS37x1gkPds2ShNhy1CzeJW0u2Xri0r50r9tsOoKp1meZI= X-Google-Smtp-Source: AGHT+IFn2rYTQCqOuHVpuGhcF7BC7asS4ROedWcgIAd4N2h6ux4X4gjzLV3woCw85UyqGNmjtKX//g== X-Received: by 2002:a05:6a00:1955:b0:6e5:5116:2cb5 with SMTP id s21-20020a056a00195500b006e551162cb5mr172054pfk.15.1709248942678; Thu, 29 Feb 2024 15:22:22 -0800 (PST) Received: from sw06.internal.sifive.com ([4.53.31.132]) by smtp.gmail.com with ESMTPSA id c6-20020aa78806000000b006e55aa75d6csm1779719pfo.122.2024.02.29.15.22.22 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 29 Feb 2024 15:22:22 -0800 (PST) From: Samuel Holland To: Palmer Dabbelt , linux-riscv@lists.infradead.org Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, Alexandre Ghiti , Jisheng Zhang , Yunhui Cui , Samuel Holland Subject: [PATCH v5 09/13] riscv: mm: Introduce cntx2asid/cntx2version helper macros Date: Thu, 29 Feb 2024 15:21:50 -0800 Message-ID: <20240229232211.161961-10-samuel.holland@sifive.com> X-Mailer: git-send-email 2.43.1 In-Reply-To: <20240229232211.161961-1-samuel.holland@sifive.com> References: <20240229232211.161961-1-samuel.holland@sifive.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" When using the ASID allocator, the MM context ID contains two values: the ASID in the lower bits, and the allocator version number in the remaining bits. Use macros to make this separation more obvious. Reviewed-by: Alexandre Ghiti Signed-off-by: Samuel Holland --- (no changes since v1) arch/riscv/include/asm/mmu.h | 3 +++ arch/riscv/mm/context.c | 12 ++++++------ arch/riscv/mm/tlbflush.c | 2 +- 3 files changed, 10 insertions(+), 7 deletions(-) diff --git a/arch/riscv/include/asm/mmu.h b/arch/riscv/include/asm/mmu.h index 355504b37f8e..a550fbf770be 100644 --- a/arch/riscv/include/asm/mmu.h +++ b/arch/riscv/include/asm/mmu.h @@ -26,6 +26,9 @@ typedef struct { #endif } mm_context_t; =20 +#define cntx2asid(cntx) ((cntx) & asid_mask) +#define cntx2version(cntx) ((cntx) & ~asid_mask) + void __init create_pgd_mapping(pgd_t *pgdp, uintptr_t va, phys_addr_t pa, phys_addr_t sz, pgprot_t prot); #endif /* __ASSEMBLY__ */ diff --git a/arch/riscv/mm/context.c b/arch/riscv/mm/context.c index ba8eb3944687..b562b3c44487 100644 --- a/arch/riscv/mm/context.c +++ b/arch/riscv/mm/context.c @@ -81,7 +81,7 @@ static void __flush_context(void) if (cntx =3D=3D 0) cntx =3D per_cpu(reserved_context, i); =20 - __set_bit(cntx & asid_mask, context_asid_map); + __set_bit(cntx2asid(cntx), context_asid_map); per_cpu(reserved_context, i) =3D cntx; } =20 @@ -102,7 +102,7 @@ static unsigned long __new_context(struct mm_struct *mm) lockdep_assert_held(&context_lock); =20 if (cntx !=3D 0) { - unsigned long newcntx =3D ver | (cntx & asid_mask); + unsigned long newcntx =3D ver | cntx2asid(cntx); =20 /* * If our current CONTEXT was active during a rollover, we @@ -115,7 +115,7 @@ static unsigned long __new_context(struct mm_struct *mm) * We had a valid CONTEXT in a previous life, so try to * re-use it if possible. */ - if (!__test_and_set_bit(cntx & asid_mask, context_asid_map)) + if (!__test_and_set_bit(cntx2asid(cntx), context_asid_map)) return newcntx; } =20 @@ -168,7 +168,7 @@ static void set_mm_asid(struct mm_struct *mm, unsigned = int cpu) */ old_active_cntx =3D atomic_long_read(&per_cpu(active_context, cpu)); if (old_active_cntx && - ((cntx & ~asid_mask) =3D=3D atomic_long_read(¤t_version)) && + (cntx2version(cntx) =3D=3D atomic_long_read(¤t_version)) && atomic_long_cmpxchg_relaxed(&per_cpu(active_context, cpu), old_active_cntx, cntx)) goto switch_mm_fast; @@ -177,7 +177,7 @@ static void set_mm_asid(struct mm_struct *mm, unsigned = int cpu) =20 /* Check that our ASID belongs to the current_version. */ cntx =3D atomic_long_read(&mm->context.id); - if ((cntx & ~asid_mask) !=3D atomic_long_read(¤t_version)) { + if (cntx2version(cntx) !=3D atomic_long_read(¤t_version)) { cntx =3D __new_context(mm); atomic_long_set(&mm->context.id, cntx); } @@ -191,7 +191,7 @@ static void set_mm_asid(struct mm_struct *mm, unsigned = int cpu) =20 switch_mm_fast: csr_write(CSR_SATP, virt_to_pfn(mm->pgd) | - ((cntx & asid_mask) << SATP_ASID_SHIFT) | + (cntx2asid(cntx) << SATP_ASID_SHIFT) | satp_mode); =20 if (need_flush_tlb) diff --git a/arch/riscv/mm/tlbflush.c b/arch/riscv/mm/tlbflush.c index 22870f213188..e194e14e5b2b 100644 --- a/arch/riscv/mm/tlbflush.c +++ b/arch/riscv/mm/tlbflush.c @@ -109,7 +109,7 @@ static void __flush_tlb_range(struct cpumask *cmask, un= signed long asid, static inline unsigned long get_mm_asid(struct mm_struct *mm) { return static_branch_unlikely(&use_asid_allocator) ? - atomic_long_read(&mm->context.id) & asid_mask : FLUSH_TLB_NO_ASID; + cntx2asid(atomic_long_read(&mm->context.id)) : FLUSH_TLB_NO_ASID; } =20 void flush_tlb_mm(struct mm_struct *mm) --=20 2.43.1 From nobody Sun Feb 8 20:22:57 2026 Received: from mail-oi1-f171.google.com (mail-oi1-f171.google.com [209.85.167.171]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8E0BA14564F for ; Thu, 29 Feb 2024 23:22:24 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.167.171 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709248947; cv=none; b=akFc1ntTgNf+HI55plev8ptCxGvtxUtS8bxIk+rgZSahyq0Y7oBZSTAipvdUDu1xC8/ZlJN2XCilLqJhTF9iGbizluBmCN9l2MWONsUtVkqZvC8bhqwS6LzwFpmXfMaSyftr/NSCMsReK2KbFuFyB9skbEsSefTwANJ7ZSJqhOs= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709248947; c=relaxed/simple; bh=ftmlZUm13WxGdRUTJnLym9NaWym8BpVy8g2b1tMP78U=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=X9wWxID3hapdWtOnlE2pe4F7WPNi+AMXHkcI+bUjb+/xuslsZMIx8oMjFafDjOKvY43Q67ctOEoovRlKzPx4n6EVIuZ/7QvUaYcxorJKbP2iAevkVutFuE+pkkH7X/fOKIv0CS7Yx7fGcTHL8fBd0jhVcdmc4bF5T9isQfjKvBQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=sifive.com; spf=pass smtp.mailfrom=sifive.com; dkim=pass (2048-bit key) header.d=sifive.com header.i=@sifive.com header.b=L/K1eENv; arc=none smtp.client-ip=209.85.167.171 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=sifive.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=sifive.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=sifive.com header.i=@sifive.com header.b="L/K1eENv" Received: by mail-oi1-f171.google.com with SMTP id 5614622812f47-3bbb4806f67so1099947b6e.3 for ; Thu, 29 Feb 2024 15:22:24 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1709248943; x=1709853743; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=A+Ju3+5NsQ7zqLi78mVdBh82QtKzTZZzrfFOA+rFDFs=; b=L/K1eENvtww414ZLEnzDtPnjNRpwoUx8rzv082ES4ghMYTt0wArVxwWYwV9LFyhNOY 0Yv1RLyPpE4A9oYBi1R3kWeUCCPp+CwknA69YGrsM0rq0BlyDH22s4nlOvsuRzD5R7JE osCrnmUUiOhHzydd+gGWiju5iJSQZUbnmETb8AARXOWQySCPFBeLauTKBX+M4/2is/r6 a3jDMMUcuwZDG9NHN9jkf5DbSMP1gyvmb4OIgtPb3dsrnE6N5Zso0K+FLMsDexGzsLWz 3EaCeKUV7FM9Y4j+e9VUlxp5EF72ldTHsS20NmVNvE0thm7dvHKn2H66IoYzVNK9X3AD RomQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709248943; x=1709853743; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=A+Ju3+5NsQ7zqLi78mVdBh82QtKzTZZzrfFOA+rFDFs=; b=hBtBjwZmcGlTE0iXrP5BIjztf5cpojlhRTx1x/3qRGtBx9MXcLoQmJ/gDXiXBPpREd SESz3CE39RYt52TlND0ez0URTgNr/buf161IuHKuZmkPz0bxpd32NbOtYOf1mVX1v42w 6ZPU6tKWbBCvoijcJbCht7zZ+kOVYPPtiMnZA/3GIz7VNEpkqLDnZhXKvBeXMrtyr9xb seKx4WRc5ofq/F99XH53r2FvOeoc2v9jMrFxGzOLGP7ZxOkFgEbSwItzKrsljhJDKNgO 3+vj1NAbApan2Q23kSb4yGoUtSOgp1C3xwRddC4mh/c0GCbvw/eca6ZseyEw10n0jiyf nsqw== X-Gm-Message-State: AOJu0YwwX1QkmeHTFGjO5vavN2FUDXGX6MY/QIwqKpGoT//np7ECkIVI fQir83lDj4EyrkZRFYbYm4ph5VLb27WAAZgcQzWoOzi/fgAO0A7V50FkCRJdqPqgxxsug0/yrCf T X-Google-Smtp-Source: AGHT+IHzbXWkwDmlfZ/eL4jDsB3ccQ2hOxgMwKxCvl8MiEGc4S4tbeeGWY8zmXhmQ74ouvbz4gXZvw== X-Received: by 2002:a05:6808:1447:b0:3c1:b28b:76f0 with SMTP id x7-20020a056808144700b003c1b28b76f0mr91074oiv.9.1709248943714; Thu, 29 Feb 2024 15:22:23 -0800 (PST) Received: from sw06.internal.sifive.com ([4.53.31.132]) by smtp.gmail.com with ESMTPSA id c6-20020aa78806000000b006e55aa75d6csm1779719pfo.122.2024.02.29.15.22.23 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 29 Feb 2024 15:22:23 -0800 (PST) From: Samuel Holland To: Palmer Dabbelt , linux-riscv@lists.infradead.org Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, Alexandre Ghiti , Jisheng Zhang , Yunhui Cui , Samuel Holland Subject: [PATCH v5 10/13] riscv: mm: Use a fixed layout for the MM context ID Date: Thu, 29 Feb 2024 15:21:51 -0800 Message-ID: <20240229232211.161961-11-samuel.holland@sifive.com> X-Mailer: git-send-email 2.43.1 In-Reply-To: <20240229232211.161961-1-samuel.holland@sifive.com> References: <20240229232211.161961-1-samuel.holland@sifive.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Currently, the size of the ASID field in the MM context ID dynamically depends on the number of hardware-supported ASID bits. This requires reading a global variable to extract either field from the context ID. Instead, allocate the maximum possible number of bits to the ASID field, so the layout of the context ID is known at compile-time. Reviewed-by: Alexandre Ghiti Signed-off-by: Samuel Holland --- (no changes since v1) arch/riscv/include/asm/mmu.h | 4 ++-- arch/riscv/include/asm/tlbflush.h | 2 -- arch/riscv/mm/context.c | 6 ++---- 3 files changed, 4 insertions(+), 8 deletions(-) diff --git a/arch/riscv/include/asm/mmu.h b/arch/riscv/include/asm/mmu.h index a550fbf770be..dc0273f7905f 100644 --- a/arch/riscv/include/asm/mmu.h +++ b/arch/riscv/include/asm/mmu.h @@ -26,8 +26,8 @@ typedef struct { #endif } mm_context_t; =20 -#define cntx2asid(cntx) ((cntx) & asid_mask) -#define cntx2version(cntx) ((cntx) & ~asid_mask) +#define cntx2asid(cntx) ((cntx) & SATP_ASID_MASK) +#define cntx2version(cntx) ((cntx) & ~SATP_ASID_MASK) =20 void __init create_pgd_mapping(pgd_t *pgdp, uintptr_t va, phys_addr_t pa, phys_addr_t sz, pgprot_t prot); diff --git a/arch/riscv/include/asm/tlbflush.h b/arch/riscv/include/asm/tlb= flush.h index 8e329721375b..72e559934952 100644 --- a/arch/riscv/include/asm/tlbflush.h +++ b/arch/riscv/include/asm/tlbflush.h @@ -15,8 +15,6 @@ #define FLUSH_TLB_NO_ASID ((unsigned long)-1) =20 #ifdef CONFIG_MMU -extern unsigned long asid_mask; - static inline void local_flush_tlb_all(void) { __asm__ __volatile__ ("sfence.vma" : : : "memory"); diff --git a/arch/riscv/mm/context.c b/arch/riscv/mm/context.c index b562b3c44487..5315af06cd4d 100644 --- a/arch/riscv/mm/context.c +++ b/arch/riscv/mm/context.c @@ -22,7 +22,6 @@ DEFINE_STATIC_KEY_FALSE(use_asid_allocator); =20 static unsigned long asid_bits; static unsigned long num_asids; -unsigned long asid_mask; =20 static atomic_long_t current_version; =20 @@ -128,7 +127,7 @@ static unsigned long __new_context(struct mm_struct *mm) goto set_asid; =20 /* We're out of ASIDs, so increment current_version */ - ver =3D atomic_long_add_return_relaxed(num_asids, ¤t_version); + ver =3D atomic_long_add_return_relaxed(BIT(SATP_ASID_BITS), ¤t_vers= ion); =20 /* Flush everything */ __flush_context(); @@ -247,7 +246,6 @@ static int __init asids_init(void) /* Pre-compute ASID details */ if (asid_bits) { num_asids =3D 1 << asid_bits; - asid_mask =3D num_asids - 1; } =20 /* @@ -255,7 +253,7 @@ static int __init asids_init(void) * at-least twice more than CPUs */ if (num_asids > (2 * num_possible_cpus())) { - atomic_long_set(¤t_version, num_asids); + atomic_long_set(¤t_version, BIT(SATP_ASID_BITS)); =20 context_asid_map =3D bitmap_zalloc(num_asids, GFP_KERNEL); if (!context_asid_map) --=20 2.43.1 From nobody Sun Feb 8 20:22:57 2026 Received: from mail-pf1-f170.google.com (mail-pf1-f170.google.com [209.85.210.170]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 768BC74BFE for ; Thu, 29 Feb 2024 23:22:25 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.170 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709248946; cv=none; b=RrUJKFnjCQTjklwy7TH3bBJCJjy01Vkpie6jHcDiF/8Y0K2gXIWNyrFQuB/pwdFDDgFduBXS3Y9Eur0nXnxAgQ8SJzsZd4KbhFc13Vszf2eMZMuR46DAzdVZ+GgNfRGAfLuD5eB1a5hz7YxpL4sL5DRVtXU70VNfTlou90jSODg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709248946; c=relaxed/simple; bh=G1BXM2z6rQBc5K4wZw4N6I/e+joRY7yH/Z0Jdaxm+no=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=DgxFmdpuIO+eFqXlFlrSUm9UHNSJex7e9vw4juT0QICltwsr7BDjKs5yNoYS5OLz1IqdKv0xKYpVpT4pQi1tehIPcNF42xDeymVtnVz2y2+FMVjxvA/OjhpysmOlAV0pTGC3rInMrGshY3K5D/T2bas2w7cg/8LUKca1JV8CRzg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=sifive.com; spf=pass smtp.mailfrom=sifive.com; dkim=pass (2048-bit key) header.d=sifive.com header.i=@sifive.com header.b=gLPBsPMW; arc=none smtp.client-ip=209.85.210.170 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=sifive.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=sifive.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=sifive.com header.i=@sifive.com header.b="gLPBsPMW" Received: by mail-pf1-f170.google.com with SMTP id d2e1a72fcca58-6e55bb75c9eso1336260b3a.3 for ; Thu, 29 Feb 2024 15:22:25 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1709248945; x=1709853745; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=4wFLc5gcnBs285NRnicKVmmQVV53EnA/2rANcUM2uCA=; b=gLPBsPMW5tbj3NNZa7yyp9h+BFyIBw958P3oU791RdPlMGHM6i9bHaDXg9MjJQwH8h B+9Um0TXzck7yUGL2G6fMNG+GXID+epbq1SDC5PlO0DQpG3scIxABxZIVUI6c90mUd/I HFJ67HfX1QSZgxgDASlbHuVyj6tE6TTpLkhT0KJ9a71tmeLcn2vz5H3xrxj/7Xj/3NBR sON03IgHbXlNJUtrZk+aWESptb5RIbQkPObsJ1tAQHejb2VjaKjO7BV3Swaz8xW2ZjmD YqFkf6jY/GqZDGHyB4r9/9NuwfPKM0ZB0mL84hxVWbZDCgE3nMeIqn8RPA6l6cgIBNm5 jJvA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709248945; x=1709853745; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=4wFLc5gcnBs285NRnicKVmmQVV53EnA/2rANcUM2uCA=; b=wkfv1j3MkB1ZL3M/nxe2es6ls7VD90/PI56q17aESviCqa3i0mchRBshxw9UbR+2qs XzY1gBCcrbDHjaP7pgySo+1sv5Oi5ZqnGBDg9RsprUlBymVdLgO9RkxTyhS42nXILaQh aFtwpOURs1Nxc4flXQxRUsReoUomnHAb+l+NtDWBm8W5zY4/gnTszQVwK36dgZa5JYpJ 0tom+AWvAK+0u239eiA8X3Zv+tloS4XvFNrZqG2kyeDIQhqROJTbmtuRLrg1+fVzhLrk a5LoUyH7dBgrxENo/FB8bw6Fgl66LgaQ3TirZLckkYkJlCC+3WI4MYf6D3LDkbj7DiUk 7Z1Q== X-Gm-Message-State: AOJu0Yw7QqCzI91rJM+y/6gSXh9HXdGJK4IL4o5Uq6xv2NS+ExRBrt7l 83SLi3+LbsBiS2KcYVxM5aVQ/dYn7jT3ngiE9sdtq9o+0Ggp79Xn2eFtRa9pOHA= X-Google-Smtp-Source: AGHT+IHQJB+ROn7vFDGIQNWMDCSQS4WpsKXvgO9ytwgYp+s+V0Upk9jrJAXqE3yMlwAjPHhZUdgCcg== X-Received: by 2002:a05:6a00:3d42:b0:6e4:eee0:5798 with SMTP id lp2-20020a056a003d4200b006e4eee05798mr179957pfb.14.1709248944849; Thu, 29 Feb 2024 15:22:24 -0800 (PST) Received: from sw06.internal.sifive.com ([4.53.31.132]) by smtp.gmail.com with ESMTPSA id c6-20020aa78806000000b006e55aa75d6csm1779719pfo.122.2024.02.29.15.22.24 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 29 Feb 2024 15:22:24 -0800 (PST) From: Samuel Holland To: Palmer Dabbelt , linux-riscv@lists.infradead.org Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, Alexandre Ghiti , Jisheng Zhang , Yunhui Cui , Samuel Holland Subject: [PATCH v5 11/13] riscv: mm: Make asid_bits a local variable Date: Thu, 29 Feb 2024 15:21:52 -0800 Message-ID: <20240229232211.161961-12-samuel.holland@sifive.com> X-Mailer: git-send-email 2.43.1 In-Reply-To: <20240229232211.161961-1-samuel.holland@sifive.com> References: <20240229232211.161961-1-samuel.holland@sifive.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" This variable is only used inside asids_init(). Reviewed-by: Alexandre Ghiti Signed-off-by: Samuel Holland --- (no changes since v1) arch/riscv/mm/context.c | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/arch/riscv/mm/context.c b/arch/riscv/mm/context.c index 5315af06cd4d..0bf6d0070a14 100644 --- a/arch/riscv/mm/context.c +++ b/arch/riscv/mm/context.c @@ -20,7 +20,6 @@ =20 DEFINE_STATIC_KEY_FALSE(use_asid_allocator); =20 -static unsigned long asid_bits; static unsigned long num_asids; =20 static atomic_long_t current_version; @@ -226,7 +225,7 @@ static inline void set_mm(struct mm_struct *prev, =20 static int __init asids_init(void) { - unsigned long old; + unsigned long asid_bits, old; =20 /* Figure-out number of ASID bits in HW */ old =3D csr_read(CSR_SATP); --=20 2.43.1 From nobody Sun Feb 8 20:22:57 2026 Received: from mail-oi1-f170.google.com (mail-oi1-f170.google.com [209.85.167.170]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 94168145668 for ; Thu, 29 Feb 2024 23:22:26 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.167.170 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709248948; cv=none; b=GpIUqyhwn9sJmdsEICsn57css4NLH71GWWKZ3+7Uns4HDkDzvCNRrwY+uJcDOrwyH4frePlmaZpyEXRoIkwTNEjkdW8ltOmUJWqaJve3Pu63XuGSUqse2ZvBRqAvRN6TrLl5HMDDBZL1A1p2UXo+izpaWW5WUvzwkzvD9QnvYok= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709248948; c=relaxed/simple; bh=MRE4XT0P9lLUUCp1Ujsa6SQk4prpBDb9nMVRKvQ6tEU=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=E4GAX3Xv+UI+98BgTuHZlqCVwwEnsm2gYONN4Xv2SfQbR+SejtyfS/lsvxpcBK9uJMORDX3H7Gm0NgIQXrL8ebwxxSa7AyLVn9ZDBS3nhroaLkcaYaTzApXJWD3/SDW6yYq2VcVBlklXI4xstQAWmtd7V2QvI45QRu/c8CFDTBg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=sifive.com; spf=pass smtp.mailfrom=sifive.com; dkim=pass (2048-bit key) header.d=sifive.com header.i=@sifive.com header.b=XSvn8kij; arc=none smtp.client-ip=209.85.167.170 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=sifive.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=sifive.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=sifive.com header.i=@sifive.com header.b="XSvn8kij" Received: by mail-oi1-f170.google.com with SMTP id 5614622812f47-3bb9b28acb4so1094686b6e.2 for ; Thu, 29 Feb 2024 15:22:26 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1709248946; x=1709853746; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=hwpkv2OuO7YHUplKH98oFVfOa+CFbclxXNu0ysmduyU=; b=XSvn8kijH43lMHYjQhBT0h1zp2LObu8ygYFKiDSr3mS/HXqIzckAiAmvpc+COu0a2e 3jOeUVSnSk+Onb1pHstwBjfqUS5KGf8RXtO6tAtJm6eaCHkXou/JQiWDojnRBlolaYHM kBUDXsvMVoJVYzEmE1oL5rZfj/ljFQ26OMJ1meUy3YBJ8i3hlPYk/OSLOkUHA5KGTZrU jbxkqEEbCl5g12LXsmxXoSIRe3rVDZFQSFjX0o0s0dTO28YG2KbXa7laBLDx5j4GGwn9 lK4IDJko3KLj3es+Gy7PODJy5cb4nvbV9LZxJ5ACeZHpQCJ4Y1SwtE00qMWrJT4JaNoG LPfw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709248946; x=1709853746; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=hwpkv2OuO7YHUplKH98oFVfOa+CFbclxXNu0ysmduyU=; b=DRvGW7+TLapQHMFqVqMOK1dSanEk6P6tYCcCvhqciRup2ORjyzFt/inCea4qikuUOX Rey35XDu1R5AsX152gsfy+P4lzW9bIAcWdbPnDrwzenKqQrl4zzHfEAmOiex8DSHVkCt YWVxa130ohLvXktkz0oy2uLlPQBel5WebQxkQKlwAlerAV7opgjFbidGZVH6beG8M1U+ 4slINWcQScDEGlfBhfjOJivfDaNHfEXkT7LKE56feHiCoiNVwg3HvPDGstlMX3oykRbD XmZGSSIFT014P7r0nnsk429vS23OSuFgyOHH2m1SNQ9YM7j3xkTGxu0voYl0Ha080pjK tUVQ== X-Gm-Message-State: AOJu0YyPLL4gG6leb9aGuFLI4a76KCLB3ZvrLpUanB0cPLhOB2Ecssx+ C4tOi1U9Oty7EbrjYNZEudr5KI21ZTtRIdV7H0drvTpYqKtrWCApNp1qRW9yBNk= X-Google-Smtp-Source: AGHT+IHmmt9lzZLUCIFAbm5GqyPJaePE9QFXJ4x/3NAo+fPZmowQvwtei5P759brqGkqtPXCOnb8SQ== X-Received: by 2002:a05:6808:17a7:b0:3c1:af49:222b with SMTP id bg39-20020a05680817a700b003c1af49222bmr47209oib.37.1709248945851; Thu, 29 Feb 2024 15:22:25 -0800 (PST) Received: from sw06.internal.sifive.com ([4.53.31.132]) by smtp.gmail.com with ESMTPSA id c6-20020aa78806000000b006e55aa75d6csm1779719pfo.122.2024.02.29.15.22.25 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 29 Feb 2024 15:22:25 -0800 (PST) From: Samuel Holland To: Palmer Dabbelt , linux-riscv@lists.infradead.org Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, Alexandre Ghiti , Jisheng Zhang , Yunhui Cui , Samuel Holland Subject: [PATCH v5 12/13] riscv: mm: Preserve global TLB entries when switching contexts Date: Thu, 29 Feb 2024 15:21:53 -0800 Message-ID: <20240229232211.161961-13-samuel.holland@sifive.com> X-Mailer: git-send-email 2.43.1 In-Reply-To: <20240229232211.161961-1-samuel.holland@sifive.com> References: <20240229232211.161961-1-samuel.holland@sifive.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" If the CPU does not support multiple ASIDs, all MM contexts use ASID 0. In this case, it is still beneficial to flush the TLB by ASID, as the single-ASID variant of the sfence.vma instruction preserves TLB entries for global (kernel) pages. This optimization is recommended by the RISC-V privileged specification: If the implementation does not provide ASIDs, or software chooses to always use ASID 0, then after every satp write, software should execute SFENCE.VMA with rs1=3Dx0. In the common case that no global translations have been modified, rs2 should be set to a register other than x0 but which contains the value zero, so that global translations are not flushed. It is not possible to apply this optimization when using the ASID allocator, because that code must flush the TLB for all ASIDs at once when incrementing the version number. Reviewed-by: Alexandre Ghiti Signed-off-by: Samuel Holland --- (no changes since v1) arch/riscv/mm/context.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/arch/riscv/mm/context.c b/arch/riscv/mm/context.c index 0bf6d0070a14..60cb0b82240e 100644 --- a/arch/riscv/mm/context.c +++ b/arch/riscv/mm/context.c @@ -200,7 +200,7 @@ static void set_mm_noasid(struct mm_struct *mm) { /* Switch the page table and blindly nuke entire local TLB */ csr_write(CSR_SATP, virt_to_pfn(mm->pgd) | satp_mode); - local_flush_tlb_all(); + local_flush_tlb_all_asid(0); } =20 static inline void set_mm(struct mm_struct *prev, --=20 2.43.1 From nobody Sun Feb 8 20:22:57 2026 Received: from mail-pf1-f177.google.com (mail-pf1-f177.google.com [209.85.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 71C86145679 for ; Thu, 29 Feb 2024 23:22:27 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.177 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709248948; cv=none; b=Ux4QAhlHH5KYasppMubXsEr/CSnwVUOH7Is/ROQwm8ocKfFVKsSK5iedN1EOw7qvEITTeS2Lhefx9cfPTkf5Ub8nVyaumVOudax+dMgK8uDfRuDupatIkqwBxZyAyCZlc+OzmVKNw+Xeddsxa1sy8pqLR9gRurh0APNz2mWgBwk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709248948; c=relaxed/simple; bh=iR1wf5c+BMYzI7Dk7MIElBjHFZAAvj9ByRRGMTXW5IA=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=AhXNJqvUQtp5NTnVOX4R5APkVJb/zANiHAVK5NSu3RGD0JNXT5yZ7WEcWDiNqDBOrUTG5rL7k/Vl04/oKdkfrDVIdhrFIQiF0zm/A1zRPuXsVrttmYA317KeoHgQ+3GpTyyQTqx5UgYuntEq+MybzqqdApucU2xWFt9mkZToqAA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=sifive.com; spf=pass smtp.mailfrom=sifive.com; dkim=pass (2048-bit key) header.d=sifive.com header.i=@sifive.com header.b=e+Pa+A3E; arc=none smtp.client-ip=209.85.210.177 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=sifive.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=sifive.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=sifive.com header.i=@sifive.com header.b="e+Pa+A3E" Received: by mail-pf1-f177.google.com with SMTP id d2e1a72fcca58-6e5a232fe80so701516b3a.0 for ; Thu, 29 Feb 2024 15:22:27 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1709248947; x=1709853747; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=cJvv9umwOkXOwc0iOvLk+NQYD7Mm5kACvwQaK1YGZP0=; b=e+Pa+A3EB/w6kupfp9BgaUqZShW6EPMpsncIOTv/l1dVOxtBqjGKlDsQBm7MtnR1JN 4SnYfrzns3hITSjfawED+HzdC3tP9fKxIm5ZueckFyxVNHvZRjZOJMakDv7iJfyG3qZ3 Qh9q5QeXaJYeWlqTI+yVedi05TUrmreisshxgKOtl1CQ7fJTZu3o88tEunmZDptSRhrX +E4LgwjC0O7SBMY5bEf+4imc8HWHDz9HUx1jiD9IIQxxHixE0rmHa5tGHAxVBHdKwPun 1ndp8tpgfVlzt4dRo+6bprH5MRxPu5n6GCbiL71RqlY8+GSGc+y/9BTIm0+hG06Cf0Zo xnGQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709248947; x=1709853747; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=cJvv9umwOkXOwc0iOvLk+NQYD7Mm5kACvwQaK1YGZP0=; b=pFmcP/27HCLSFFPYW8C405ju4v+p4GkhfFlh0+i+hXrqtsiKcgpno4gu6GfGtHbJik ehEJRzbM4PLKt7FFzSgfK8OvrlejNU+24KGTKseEZnzBQxbSyXbTaAJKvp3y4SrqFqCZ nr3j4pfSm7NkdikHM9HjJ/Af4AIhNDbMGLmNdqhjeKtbygmMl96FRbRrswgIoTyPNM74 zyE+j+mKCVwuDBJjjF9VgV5jog0K7ouegYdcBR+OUSazihlT9/IRddtsQoJ1M1WfuQ9V omT/PUSqxSWZJh2Rv3L613yxwDz9Av3FzL+URfKH74fDHI6vZTwxPg+cLdYEG1MYlWQb Iz1w== X-Gm-Message-State: AOJu0YwEvz5XD8MHpgwF7NE6PHrGj2IKXANt4L0g8NaqL/IAAmXFyMzJ D4rSFuQpXUSbUYHvMUnZIQDC8sJSMML+98nnWQCfa4bNM1m8Zf8jl+VY3dXTh4M= X-Google-Smtp-Source: AGHT+IG000bMn11OdL3LH1zGovTy9Qg+UUgYE8CMrW4h6bNv2nzitQBp1dzsEHx4shrcmNE8zaAVpw== X-Received: by 2002:a05:6a00:27a3:b0:6e5:3dea:bf51 with SMTP id bd35-20020a056a0027a300b006e53deabf51mr562992pfb.7.1709248946831; Thu, 29 Feb 2024 15:22:26 -0800 (PST) Received: from sw06.internal.sifive.com ([4.53.31.132]) by smtp.gmail.com with ESMTPSA id c6-20020aa78806000000b006e55aa75d6csm1779719pfo.122.2024.02.29.15.22.26 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 29 Feb 2024 15:22:26 -0800 (PST) From: Samuel Holland To: Palmer Dabbelt , linux-riscv@lists.infradead.org Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, Alexandre Ghiti , Jisheng Zhang , Yunhui Cui , Samuel Holland Subject: [PATCH v5 13/13] riscv: mm: Always use an ASID to flush mm contexts Date: Thu, 29 Feb 2024 15:21:54 -0800 Message-ID: <20240229232211.161961-14-samuel.holland@sifive.com> X-Mailer: git-send-email 2.43.1 In-Reply-To: <20240229232211.161961-1-samuel.holland@sifive.com> References: <20240229232211.161961-1-samuel.holland@sifive.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Even if multiple ASIDs are not supported, using the single-ASID variant of the sfence.vma instruction preserves TLB entries for global (kernel) pages. So it is always more efficient to use the single-ASID code path. Reviewed-by: Alexandre Ghiti Signed-off-by: Samuel Holland --- Changes in v5: - Leave use_asid_allocator declared in asm/mmu_context.h Changes in v4: - There is now only one copy of __flush_tlb_range() Changes in v2: - Update both copies of __flush_tlb_range() arch/riscv/mm/tlbflush.c | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/arch/riscv/mm/tlbflush.c b/arch/riscv/mm/tlbflush.c index e194e14e5b2b..5b473588a985 100644 --- a/arch/riscv/mm/tlbflush.c +++ b/arch/riscv/mm/tlbflush.c @@ -108,8 +108,7 @@ static void __flush_tlb_range(struct cpumask *cmask, un= signed long asid, =20 static inline unsigned long get_mm_asid(struct mm_struct *mm) { - return static_branch_unlikely(&use_asid_allocator) ? - cntx2asid(atomic_long_read(&mm->context.id)) : FLUSH_TLB_NO_ASID; + return cntx2asid(atomic_long_read(&mm->context.id)); } =20 void flush_tlb_mm(struct mm_struct *mm) --=20 2.43.1