From nobody Tue Dec 16 20:00:28 2025 Received: from mail-pg1-f178.google.com (mail-pg1-f178.google.com [209.85.215.178]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4702A3DABF8 for ; Wed, 27 Mar 2024 04:50:40 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.215.178 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711515041; cv=none; b=ObFu/xYAB2e3bL/meua773NWMYVzEwrccnxWj9++2gQNnEVZ07ETUgQu7XO/TYiBKLSGqrqXT3yJBZEgAkfuQAXmNbDpyix28qEPsaC719vUhRO8WdMeMbaaIGFcX5w25dsF+5e5ObcRl649+LJx8LY62VbNebtLq4W/3R97QRs= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711515041; c=relaxed/simple; bh=9pe7R/eOPtpKV/p0oqNPsZLIbMKVAFsvdDFGxpx/rcs=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=kII1mVzD6n+3gH6xAwOt0MlIx0LUE/EAHWSVma4Xvk8MPQZqPAzd5Lrx1X3whMBWlSWh7Do2vKYKotEzp3jtRaXTMzXpMsPgNe6MNpp4tznf5lggQiI3ApVZRC+oFjpmzJeBZMJuh3Z7h0qYJvl8+qrySNZji8IlEONe3lxVuo4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=sifive.com; spf=pass smtp.mailfrom=sifive.com; dkim=pass (2048-bit key) header.d=sifive.com header.i=@sifive.com header.b=Z5t5F8WX; arc=none smtp.client-ip=209.85.215.178 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=sifive.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=sifive.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=sifive.com header.i=@sifive.com header.b="Z5t5F8WX" Received: by mail-pg1-f178.google.com with SMTP id 41be03b00d2f7-5d8b887bb0cso4480264a12.2 for ; Tue, 26 Mar 2024 21:50:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1711515039; x=1712119839; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=shQKeKcQeNJUU/JQ5KWmpXdfmdrhkhe16Q2MMRo9oOg=; b=Z5t5F8WX+DWGeHMb2BK0DpVQhHlDuvFvHMB6cN8v5REGRr7Kji79Ns2Ohley9bdxIG 0f9xU7t9pY8bZlVWszzPBe4qJ5me1WSbrXnliwhrlcjnbHZrChEiKQRf3tOy1wUT6/B7 cJsQdcVFgZJBCUnf2V94UhoUsL7pLZCspTA+vdQmmBz0qMVG1KdLhXKDS94GZBzjxPG4 IzkpEymuMhkGls9GcVgtwlXFEs8LYn/mvmLdvlEYqHRhJxfy/adiApW/pZCi7rga9LgD x+4u4FTv+BfwKpDxVxQl5GUP8irCwI986ZYvUS+vGtTihQvlR7CxqGl4jU32wgqAD4pX f24w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711515039; x=1712119839; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=shQKeKcQeNJUU/JQ5KWmpXdfmdrhkhe16Q2MMRo9oOg=; b=WQUe6/Dj8HOvNfKpQuhe4TXbRAZgi6ZSKuxVfG+np4rQ38kb9nPhHPyoK5om459OLl mhYEKuL9raTUBTJ3Vhkji+kcCsiGj23CEglyCNgmG3BtIGjWLndjPtGPPDhe7BNogrUk +P1hfyQNoyff89pRJicnqfG/EGDtSFLwlX5/MAXAerzLRX1fMxfZAVkMzHpxRahXnBLd 1KJFLFyGRqGmufBPLdEl4NgkXxOsESUrcrX4eNKlw6zMBosT7JxL7mgUr5e3F9A7+ve8 3pbIFgrWW/wCoQQ67P9Iur9Vrub5kiwEK/i1rRZXaJffI2uvVdOUzVXL6NhBuyddc0eM PtnA== X-Gm-Message-State: AOJu0Ywhy2B0J9hpvASzl4/caukb3Zmzx+j90D06jsZzZMLw1bXRfmH1 MnMh/rPPaGKbCLhV63tRd+vcwXTTBUX+YYomV4DBRuEikIHN0QtkaKALc2n+Q+M= X-Google-Smtp-Source: AGHT+IHpHyQ5wkbbDg1/9QOIdZT7U8XEUGKwcMibXVBUL9WTi2GIa5kveh+VvtS/wwBFRvp4M1UHpQ== X-Received: by 2002:a05:6a21:164e:b0:1a3:4979:f25e with SMTP id no14-20020a056a21164e00b001a34979f25emr128528pzb.59.1711515039607; Tue, 26 Mar 2024 21:50:39 -0700 (PDT) Received: from sw06.internal.sifive.com ([4.53.31.132]) by smtp.gmail.com with ESMTPSA id e31-20020a63501f000000b005e4666261besm8351500pgb.50.2024.03.26.21.50.38 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 26 Mar 2024 21:50:38 -0700 (PDT) From: Samuel Holland To: Palmer Dabbelt , linux-riscv@lists.infradead.org Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, Alexandre Ghiti , Jisheng Zhang , Yunhui Cui , Samuel Holland Subject: [PATCH v6 01/13] riscv: Flush the instruction cache during SMP bringup Date: Tue, 26 Mar 2024 21:49:42 -0700 Message-ID: <20240327045035.368512-2-samuel.holland@sifive.com> X-Mailer: git-send-email 2.43.1 In-Reply-To: <20240327045035.368512-1-samuel.holland@sifive.com> References: <20240327045035.368512-1-samuel.holland@sifive.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Instruction cache flush IPIs are sent only to CPUs in cpu_online_mask, so they will not target a CPU until it calls set_cpu_online() earlier in smp_callin(). As a result, if instruction memory is modified between the CPU coming out of reset and that point, then its instruction cache may contain stale data. Therefore, the instruction cache must be flushed after the set_cpu_online() synchronization point. Fixes: 08f051eda33b ("RISC-V: Flush I$ when making a dirty page executable") Reviewed-by: Alexandre Ghiti Signed-off-by: Samuel Holland --- (no changes since v4) Changes in v4: - New patch for v4 arch/riscv/kernel/smpboot.c | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/arch/riscv/kernel/smpboot.c b/arch/riscv/kernel/smpboot.c index d41090fc3203..4b3c50da48ba 100644 --- a/arch/riscv/kernel/smpboot.c +++ b/arch/riscv/kernel/smpboot.c @@ -26,7 +26,7 @@ #include #include =20 -#include +#include #include #include #include @@ -234,9 +234,10 @@ asmlinkage __visible void smp_callin(void) riscv_user_isa_enable(); =20 /* - * Remote TLB flushes are ignored while the CPU is offline, so emit - * a local TLB flush right now just in case. + * Remote cache and TLB flushes are ignored while the CPU is offline, + * so flush them both right now just in case. */ + local_flush_icache_all(); local_flush_tlb_all(); complete(&cpu_running); /* --=20 2.43.1 From nobody Tue Dec 16 20:00:28 2025 Received: from mail-ot1-f53.google.com (mail-ot1-f53.google.com [209.85.210.53]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E0EA5B67E for ; Wed, 27 Mar 2024 04:50:43 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.53 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711515045; cv=none; b=RHU5mYl94/tklzMG3cXtdXTWMNKmTaUiU1jV4/e71gsKZ9s1+JbEmQAPj61R4CG7OslO0mYo0KeYEXvGypxfGIDvBulFi395gbsl5xS1YHvjtRBzLSEir1yzr1nnY0OPd26acHTLl4kuLataET72VgUhvKFaOPxBVXweBCwCpK8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711515045; c=relaxed/simple; bh=PNo9fUUIIt9SGGQDqI5YlJUfPkSDmqOPYa4cRVAr/j8=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=js/L+gOHhxFwShxfl0xk8zfL1IZYMWv6mgTaeLy/ElRzd2TAGadV6snS7s/i2PCyiRfqQlK2dKKiCrCcK1hOZ3irvxqRVbtx4YMC60zqHgfMsW8v1Iuf29823IyH9cNbuE/SdC8W0G6NaAz2cnLQPWY3LaMuBXoLmUdO1cx249Q= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=sifive.com; spf=pass smtp.mailfrom=sifive.com; dkim=pass (2048-bit key) header.d=sifive.com header.i=@sifive.com header.b=FzfRSZqK; arc=none smtp.client-ip=209.85.210.53 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=sifive.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=sifive.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=sifive.com header.i=@sifive.com header.b="FzfRSZqK" Received: by mail-ot1-f53.google.com with SMTP id 46e09a7af769-6e50e8dcc72so2629172a34.3 for ; Tue, 26 Mar 2024 21:50:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1711515043; x=1712119843; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Tmp/TOSalTk1E9Om53lqcGFTwB3eFzfKle7hmp+7bao=; b=FzfRSZqKT+avHlzxgaXIOJ7UUuwHeo6s8whPqrsSsSxvLXFf8geB+CKvZReY1Y3rdt GRLh8KgXYGMjTs9I+cO1mQyEE8z/J2rPCv7r4R1nRen8gPflEX/fdyuZ36y7yJQc8LHu 7LyEEKGYcDch9khG3sdYWwcI90mG8NwYfuWstbevpeKhZga0db2UoZikFcrlmBU/RE1F rperXo4BRTfdyTRhD1EI6aRLivTiWlEdPKL1wq0hKdvURo+15NtwX/l4VEtf/FO04xKk ewcoq2UpHdc7d33F6p6Qx8NSGCnoWK72KlS7HeKnWe0R/FhZsoPnHLiY+T9GE2Snx4W8 ygsw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711515043; x=1712119843; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Tmp/TOSalTk1E9Om53lqcGFTwB3eFzfKle7hmp+7bao=; b=JmqjEfLuWqB7dqBBKICO0f3VPPhQNCAXxmoRSoPS/cjHYP3b8Mi56VQ3TRQVSC6J9b Db+aepbGMTzvnhzH7zeiH90u4OEp2C7Hfe7Vi+0NXTopzpVXv7KK6eI+2711LGkwK8w1 w7zQNMrswm+53wUnwsqnthehyvlL0mkqUvKgcqx5mwXhcAmSEWdPYIGWYzZVdQ9a/DIU JXCaAtuWboPPkEhPqYvWgYw07mzw2F1yy+0OusgKotxkhONLRQ61yZubxKHcgvbwfVvO tJQrbqYUsknSEMDSfVC4tgVMu/Ts2hy75FnIGSv558tnntF2ElRCBBHpVZ9HBoa0eO0s ntTA== X-Gm-Message-State: AOJu0YwVeOk58dXO7gzwbe0aiBrEgmugyLR9BcGQcGVgmz6j+wILqLjl e7qXnOCdXQiI3ELV5PMg2wx0jHjUM5nKzm1tjJGUPUBfX/WV27NAm8ncbC5o/Ss= X-Google-Smtp-Source: AGHT+IFl3z8IwkcLePPXROsXhoQYUv/uXEkbIlJCNYH90sGzXXkCB+mcXoO2kI0LDgwvd8XvzM/1fA== X-Received: by 2002:a05:6830:1bd5:b0:6e6:99da:8944 with SMTP id v21-20020a0568301bd500b006e699da8944mr3176474ota.26.1711515043004; Tue, 26 Mar 2024 21:50:43 -0700 (PDT) Received: from sw06.internal.sifive.com ([4.53.31.132]) by smtp.gmail.com with ESMTPSA id e31-20020a63501f000000b005e4666261besm8351500pgb.50.2024.03.26.21.50.40 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 26 Mar 2024 21:50:42 -0700 (PDT) From: Samuel Holland To: Palmer Dabbelt , linux-riscv@lists.infradead.org Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, Alexandre Ghiti , Jisheng Zhang , Yunhui Cui , Samuel Holland Subject: [PATCH v6 02/13] riscv: Factor out page table TLB synchronization Date: Tue, 26 Mar 2024 21:49:43 -0700 Message-ID: <20240327045035.368512-3-samuel.holland@sifive.com> X-Mailer: git-send-email 2.43.1 In-Reply-To: <20240327045035.368512-1-samuel.holland@sifive.com> References: <20240327045035.368512-1-samuel.holland@sifive.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" The logic is the same for all page table levels. See commit 69be3fb111e7 ("riscv: enable MMU_GATHER_RCU_TABLE_FREE for SMP && MMU"). Signed-off-by: Samuel Holland Reviewed-by: Alexandre Ghiti --- Changes in v6: - Move riscv_tlb_remove_ptdesc() definition to fix 32-bit build Changes in v5: - New patch for v5 arch/riscv/include/asm/pgalloc.h | 31 +++++++++++++------------------ 1 file changed, 13 insertions(+), 18 deletions(-) diff --git a/arch/riscv/include/asm/pgalloc.h b/arch/riscv/include/asm/pgal= loc.h index deaf971253a2..b34587da8882 100644 --- a/arch/riscv/include/asm/pgalloc.h +++ b/arch/riscv/include/asm/pgalloc.h @@ -15,6 +15,14 @@ #define __HAVE_ARCH_PUD_FREE #include =20 +static inline void riscv_tlb_remove_ptdesc(struct mmu_gather *tlb, void *p= t) +{ + if (riscv_use_ipi_for_rfence()) + tlb_remove_page_ptdesc(tlb, pt); + else + tlb_remove_ptdesc(tlb, pt); +} + static inline void pmd_populate_kernel(struct mm_struct *mm, pmd_t *pmd, pte_t *pte) { @@ -102,10 +110,7 @@ static inline void __pud_free_tlb(struct mmu_gather *t= lb, pud_t *pud, struct ptdesc *ptdesc =3D virt_to_ptdesc(pud); =20 pagetable_pud_dtor(ptdesc); - if (riscv_use_ipi_for_rfence()) - tlb_remove_page_ptdesc(tlb, ptdesc); - else - tlb_remove_ptdesc(tlb, ptdesc); + riscv_tlb_remove_ptdesc(tlb, ptdesc); } } =20 @@ -139,12 +144,8 @@ static inline void p4d_free(struct mm_struct *mm, p4d_= t *p4d) static inline void __p4d_free_tlb(struct mmu_gather *tlb, p4d_t *p4d, unsigned long addr) { - if (pgtable_l5_enabled) { - if (riscv_use_ipi_for_rfence()) - tlb_remove_page_ptdesc(tlb, virt_to_ptdesc(p4d)); - else - tlb_remove_ptdesc(tlb, virt_to_ptdesc(p4d)); - } + if (pgtable_l5_enabled) + riscv_tlb_remove_ptdesc(tlb, virt_to_ptdesc(p4d)); } #endif /* __PAGETABLE_PMD_FOLDED */ =20 @@ -176,10 +177,7 @@ static inline void __pmd_free_tlb(struct mmu_gather *t= lb, pmd_t *pmd, struct ptdesc *ptdesc =3D virt_to_ptdesc(pmd); =20 pagetable_pmd_dtor(ptdesc); - if (riscv_use_ipi_for_rfence()) - tlb_remove_page_ptdesc(tlb, ptdesc); - else - tlb_remove_ptdesc(tlb, ptdesc); + riscv_tlb_remove_ptdesc(tlb, ptdesc); } =20 #endif /* __PAGETABLE_PMD_FOLDED */ @@ -190,10 +188,7 @@ static inline void __pte_free_tlb(struct mmu_gather *t= lb, pgtable_t pte, struct ptdesc *ptdesc =3D page_ptdesc(pte); =20 pagetable_pte_dtor(ptdesc); - if (riscv_use_ipi_for_rfence()) - tlb_remove_page_ptdesc(tlb, ptdesc); - else - tlb_remove_ptdesc(tlb, ptdesc); + riscv_tlb_remove_ptdesc(tlb, ptdesc); } #endif /* CONFIG_MMU */ =20 --=20 2.43.1 From nobody Tue Dec 16 20:00:28 2025 Received: from mail-yw1-f178.google.com (mail-yw1-f178.google.com [209.85.128.178]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C309C3DAC13 for ; Wed, 27 Mar 2024 04:50:45 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.178 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711515047; cv=none; b=nkYBa+H+SV3WzCr8tpyNqmjFluxFIP5kkMdVqCQ6V31b210g+l+scviOcXVKnMqXmLCERsLkKw9BKT9Vts8ntk+Uj8K4SqjNGc0/Rg6XhV1ByA9AfAmK1MaeLzvdwgtfoxONX27iNylBIYuMiMmHJwt+ELaKcEbQqQkcBP+GSbY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711515047; c=relaxed/simple; bh=9+kHQMgLvVcAI1DoTqT+nRSSAlZswillYYHPJoS2Qmc=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=gf3N9ClwKwXznSyOkHERArSnqQA8n9O7WkW2zzkeYrvi1TAsbZiBx79kTrxWURDZagdJf0QFmnp3TCpVjXAtEheDtdGup1LXfYPXb9/gwwFFJU0rYHVAFg45ImdSssDWENWcxurs2IFIHZsUzKDa7X/Ofx2GHsXogsI7K2P9q08= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=sifive.com; spf=pass smtp.mailfrom=sifive.com; dkim=pass (2048-bit key) header.d=sifive.com header.i=@sifive.com header.b=aRcCHUkY; arc=none smtp.client-ip=209.85.128.178 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=sifive.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=sifive.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=sifive.com header.i=@sifive.com header.b="aRcCHUkY" Received: by mail-yw1-f178.google.com with SMTP id 00721157ae682-60a0599f631so55694567b3.2 for ; Tue, 26 Mar 2024 21:50:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1711515045; x=1712119845; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=G6C+MSH7tzUf74TudLETMF5DBzv2rpp8bzfwEKTodUs=; b=aRcCHUkYDjv3SzmzA6Mx58hlY3XspN7E7LNvZG+ikjWwzvSsutzTVA90FzeQCHdWl0 a6MVO8/T9Z4QaQQpjIFXrWWQJg6wPw2xbhfyg46PmZ1g8KPYj5MwC187jfq+Q/z85Z55 Gik7L+COuIf6XOBHE51wmgTMbckOe9VVr7IyCUkpjG/r+nYsQC3Zb1ESMeLDKZ1RducT OsZM0teZyvyWLIgVKyhRhmFVUVI9gRqGSG9Az5HQ5qTwY0wIuYC7p/O2MmePN1CedxbA WeZKnt6YKGYTQD0iEPErQpqPJzedrHl9UwRoSuIEMb7lE81QZ+il0TXmSUASNsG4zQ5n Xhgg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711515045; x=1712119845; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=G6C+MSH7tzUf74TudLETMF5DBzv2rpp8bzfwEKTodUs=; b=TxNgWl0qFD8RGfgkmHdTqe4UQH0uWCwUNv2Jg7jguyOlxEkABrjXY+RzdfUyAPoSYC DRRTiTEGjSentUm/HWe78QyaLm1BXohKFcs1ygkHSNi+yl3msiku+uKVhRLOpqrzhaX3 zqSzs7J9ayb1cOiSznKVJM8Lj0OSICdh2z6p88CW8P+QFW0Xr3vE9TNbbrcxlULpT+EA RVivPa5jNlxqjtMx16xosJJ10Ezp0q9eoqLQhlh/Tmobej1QxrR1j1uVgqlR20BafePt 2o67B+qkZJBJ3giaWcpC8m/4tVQPC+qe1N9ISwW9uO6JpeZtvE6QRpdTx8Ru69EWw/cy fjAA== X-Gm-Message-State: AOJu0Yxn1kW8m7lPVtEpxTMkzH+69sj1ZxXuht3mCFj/SHomr3Gk2m67 xHYHhkgGMSbqpduJhvuLa8HqJCsEjg4ROA4ru/7ddqwNtOA75F1Tpc7wjaNarmA= X-Google-Smtp-Source: AGHT+IH+jSoiPAnTh406w99fZLlqeG+exqQi31XclYwr8IOVXBgedOOFyirc6G8eGQn9SYpg/+yAIQ== X-Received: by 2002:a81:8084:0:b0:611:9be5:4e17 with SMTP id q126-20020a818084000000b006119be54e17mr18863ywf.51.1711515044780; Tue, 26 Mar 2024 21:50:44 -0700 (PDT) Received: from sw06.internal.sifive.com ([4.53.31.132]) by smtp.gmail.com with ESMTPSA id e31-20020a63501f000000b005e4666261besm8351500pgb.50.2024.03.26.21.50.43 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 26 Mar 2024 21:50:43 -0700 (PDT) From: Samuel Holland To: Palmer Dabbelt , linux-riscv@lists.infradead.org Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, Alexandre Ghiti , Jisheng Zhang , Yunhui Cui , Samuel Holland , Anup Patel Subject: [PATCH v6 03/13] riscv: Use IPIs for remote cache/TLB flushes by default Date: Tue, 26 Mar 2024 21:49:44 -0700 Message-ID: <20240327045035.368512-4-samuel.holland@sifive.com> X-Mailer: git-send-email 2.43.1 In-Reply-To: <20240327045035.368512-1-samuel.holland@sifive.com> References: <20240327045035.368512-1-samuel.holland@sifive.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" An IPI backend is always required in an SMP configuration, but an SBI implementation is not. For example, SBI will be unavailable when the kernel runs in M mode. For this reason, consider IPI delivery of cache and TLB flushes to be the base case, and any other implementation (such as the SBI remote fence extension) to be an optimization. Generally, if IPIs can be delivered without firmware assistance, they are assumed to be faster than SBI calls due to the SBI context switch overhead. However, when SBI is used as the IPI backend, then the context switch cost must be paid anyway, and performing the cache/TLB flush directly in the SBI implementation is more efficient than injecting an interrupt to S-mode. This is the only existing scenario where riscv_ipi_set_virq_range() is called with use_for_rfence set to false. sbi_ipi_init() already checks riscv_ipi_have_virq_range(), so it only calls riscv_ipi_set_virq_range() when no other IPI device is available. This allows moving the static key and dropping the use_for_rfence parameter. This decouples the static key from the irqchip driver probe order. Furthermore, the static branch only makes sense when CONFIG_RISCV_SBI is enabled. Optherwise, IPIs must be used. Add a fallback definition of riscv_use_sbi_for_rfence() which handles this case and removes the need to check CONFIG_RISCV_SBI elsewhere, such as in cacheflush.c. Reviewed-by: Anup Patel Signed-off-by: Samuel Holland Reviewed-by: Alexandre Ghiti --- Changes in v6: - Clarify the commit message for patch 3 based on ML discussion Changes in v5: - Also switch to riscv_use_sbi_for_rfence() in asm/pgalloc.h Changes in v4: - New patch for v4 arch/riscv/include/asm/pgalloc.h | 7 ++++--- arch/riscv/include/asm/sbi.h | 4 ++++ arch/riscv/include/asm/smp.h | 15 ++------------- arch/riscv/kernel/sbi-ipi.c | 11 ++++++++++- arch/riscv/kernel/smp.c | 11 +---------- arch/riscv/mm/cacheflush.c | 5 ++--- arch/riscv/mm/tlbflush.c | 31 ++++++++++++++----------------- drivers/clocksource/timer-clint.c | 2 +- 8 files changed, 38 insertions(+), 48 deletions(-) diff --git a/arch/riscv/include/asm/pgalloc.h b/arch/riscv/include/asm/pgal= loc.h index b34587da8882..f52264304f77 100644 --- a/arch/riscv/include/asm/pgalloc.h +++ b/arch/riscv/include/asm/pgalloc.h @@ -8,6 +8,7 @@ #define _ASM_RISCV_PGALLOC_H =20 #include +#include #include =20 #ifdef CONFIG_MMU @@ -17,10 +18,10 @@ =20 static inline void riscv_tlb_remove_ptdesc(struct mmu_gather *tlb, void *p= t) { - if (riscv_use_ipi_for_rfence()) - tlb_remove_page_ptdesc(tlb, pt); - else + if (riscv_use_sbi_for_rfence()) tlb_remove_ptdesc(tlb, pt); + else + tlb_remove_page_ptdesc(tlb, pt); } =20 static inline void pmd_populate_kernel(struct mm_struct *mm, diff --git a/arch/riscv/include/asm/sbi.h b/arch/riscv/include/asm/sbi.h index 6e68f8dff76b..ea84392ca9d7 100644 --- a/arch/riscv/include/asm/sbi.h +++ b/arch/riscv/include/asm/sbi.h @@ -375,8 +375,12 @@ unsigned long riscv_cached_marchid(unsigned int cpu_id= ); unsigned long riscv_cached_mimpid(unsigned int cpu_id); =20 #if IS_ENABLED(CONFIG_SMP) && IS_ENABLED(CONFIG_RISCV_SBI) +DECLARE_STATIC_KEY_FALSE(riscv_sbi_for_rfence); +#define riscv_use_sbi_for_rfence() \ + static_branch_unlikely(&riscv_sbi_for_rfence) void sbi_ipi_init(void); #else +static inline bool riscv_use_sbi_for_rfence(void) { return false; } static inline void sbi_ipi_init(void) { } #endif =20 diff --git a/arch/riscv/include/asm/smp.h b/arch/riscv/include/asm/smp.h index 0d555847cde6..7ac80e9f2288 100644 --- a/arch/riscv/include/asm/smp.h +++ b/arch/riscv/include/asm/smp.h @@ -49,12 +49,7 @@ void riscv_ipi_disable(void); bool riscv_ipi_have_virq_range(void); =20 /* Set the IPI interrupt numbers for arch (called by irqchip drivers) */ -void riscv_ipi_set_virq_range(int virq, int nr, bool use_for_rfence); - -/* Check if we can use IPIs for remote FENCEs */ -DECLARE_STATIC_KEY_FALSE(riscv_ipi_for_rfence); -#define riscv_use_ipi_for_rfence() \ - static_branch_unlikely(&riscv_ipi_for_rfence) +void riscv_ipi_set_virq_range(int virq, int nr); =20 /* Check other CPUs stop or not */ bool smp_crash_stop_failed(void); @@ -104,16 +99,10 @@ static inline bool riscv_ipi_have_virq_range(void) return false; } =20 -static inline void riscv_ipi_set_virq_range(int virq, int nr, - bool use_for_rfence) +static inline void riscv_ipi_set_virq_range(int virq, int nr) { } =20 -static inline bool riscv_use_ipi_for_rfence(void) -{ - return false; -} - #endif /* CONFIG_SMP */ =20 #if defined(CONFIG_HOTPLUG_CPU) && (CONFIG_SMP) diff --git a/arch/riscv/kernel/sbi-ipi.c b/arch/riscv/kernel/sbi-ipi.c index a4559695ce62..1026e22955cc 100644 --- a/arch/riscv/kernel/sbi-ipi.c +++ b/arch/riscv/kernel/sbi-ipi.c @@ -13,6 +13,9 @@ #include #include =20 +DEFINE_STATIC_KEY_FALSE(riscv_sbi_for_rfence); +EXPORT_SYMBOL_GPL(riscv_sbi_for_rfence); + static int sbi_ipi_virq; =20 static void sbi_ipi_handle(struct irq_desc *desc) @@ -72,6 +75,12 @@ void __init sbi_ipi_init(void) "irqchip/sbi-ipi:starting", sbi_ipi_starting_cpu, NULL); =20 - riscv_ipi_set_virq_range(virq, BITS_PER_BYTE, false); + riscv_ipi_set_virq_range(virq, BITS_PER_BYTE); pr_info("providing IPIs using SBI IPI extension\n"); + + /* + * Use the SBI remote fence extension to avoid + * the extra context switch needed to handle IPIs. + */ + static_branch_enable(&riscv_sbi_for_rfence); } diff --git a/arch/riscv/kernel/smp.c b/arch/riscv/kernel/smp.c index 45dd4035416e..8e6eb64459af 100644 --- a/arch/riscv/kernel/smp.c +++ b/arch/riscv/kernel/smp.c @@ -171,10 +171,7 @@ bool riscv_ipi_have_virq_range(void) return (ipi_virq_base) ? true : false; } =20 -DEFINE_STATIC_KEY_FALSE(riscv_ipi_for_rfence); -EXPORT_SYMBOL_GPL(riscv_ipi_for_rfence); - -void riscv_ipi_set_virq_range(int virq, int nr, bool use_for_rfence) +void riscv_ipi_set_virq_range(int virq, int nr) { int i, err; =20 @@ -197,12 +194,6 @@ void riscv_ipi_set_virq_range(int virq, int nr, bool u= se_for_rfence) =20 /* Enabled IPIs for boot CPU immediately */ riscv_ipi_enable(); - - /* Update RFENCE static key */ - if (use_for_rfence) - static_branch_enable(&riscv_ipi_for_rfence); - else - static_branch_disable(&riscv_ipi_for_rfence); } =20 static const char * const ipi_names[] =3D { diff --git a/arch/riscv/mm/cacheflush.c b/arch/riscv/mm/cacheflush.c index bc61ee5975e4..d76fc73e594b 100644 --- a/arch/riscv/mm/cacheflush.c +++ b/arch/riscv/mm/cacheflush.c @@ -21,7 +21,7 @@ void flush_icache_all(void) { local_flush_icache_all(); =20 - if (IS_ENABLED(CONFIG_RISCV_SBI) && !riscv_use_ipi_for_rfence()) + if (riscv_use_sbi_for_rfence()) sbi_remote_fence_i(NULL); else on_each_cpu(ipi_remote_fence_i, NULL, 1); @@ -69,8 +69,7 @@ void flush_icache_mm(struct mm_struct *mm, bool local) * with flush_icache_deferred(). */ smp_mb(); - } else if (IS_ENABLED(CONFIG_RISCV_SBI) && - !riscv_use_ipi_for_rfence()) { + } else if (riscv_use_sbi_for_rfence()) { sbi_remote_fence_i(&others); } else { on_each_cpu_mask(&others, ipi_remote_fence_i, NULL, 1); diff --git a/arch/riscv/mm/tlbflush.c b/arch/riscv/mm/tlbflush.c index 893566e004b7..0435605b07d0 100644 --- a/arch/riscv/mm/tlbflush.c +++ b/arch/riscv/mm/tlbflush.c @@ -79,10 +79,10 @@ static void __ipi_flush_tlb_all(void *info) =20 void flush_tlb_all(void) { - if (riscv_use_ipi_for_rfence()) - on_each_cpu(__ipi_flush_tlb_all, NULL, 1); - else + if (riscv_use_sbi_for_rfence()) sbi_remote_sfence_vma_asid(NULL, 0, FLUSH_TLB_MAX_SIZE, FLUSH_TLB_NO_ASI= D); + else + on_each_cpu(__ipi_flush_tlb_all, NULL, 1); } =20 struct flush_tlb_range_data { @@ -103,7 +103,6 @@ static void __flush_tlb_range(struct cpumask *cmask, un= signed long asid, unsigned long start, unsigned long size, unsigned long stride) { - struct flush_tlb_range_data ftd; bool broadcast; =20 if (cpumask_empty(cmask)) @@ -119,20 +118,18 @@ static void __flush_tlb_range(struct cpumask *cmask, = unsigned long asid, broadcast =3D true; } =20 - if (broadcast) { - if (riscv_use_ipi_for_rfence()) { - ftd.asid =3D asid; - ftd.start =3D start; - ftd.size =3D size; - ftd.stride =3D stride; - on_each_cpu_mask(cmask, - __ipi_flush_tlb_range_asid, - &ftd, 1); - } else - sbi_remote_sfence_vma_asid(cmask, - start, size, asid); - } else { + if (!broadcast) { local_flush_tlb_range_asid(start, size, stride, asid); + } else if (riscv_use_sbi_for_rfence()) { + sbi_remote_sfence_vma_asid(cmask, start, size, asid); + } else { + struct flush_tlb_range_data ftd; + + ftd.asid =3D asid; + ftd.start =3D start; + ftd.size =3D size; + ftd.stride =3D stride; + on_each_cpu_mask(cmask, __ipi_flush_tlb_range_asid, &ftd, 1); } =20 if (cmask !=3D cpu_online_mask) diff --git a/drivers/clocksource/timer-clint.c b/drivers/clocksource/timer-= clint.c index 09fd292eb83d..0bdd9d7ec545 100644 --- a/drivers/clocksource/timer-clint.c +++ b/drivers/clocksource/timer-clint.c @@ -251,7 +251,7 @@ static int __init clint_timer_init_dt(struct device_nod= e *np) } =20 irq_set_chained_handler(clint_ipi_irq, clint_ipi_interrupt); - riscv_ipi_set_virq_range(rc, BITS_PER_BYTE, true); + riscv_ipi_set_virq_range(rc, BITS_PER_BYTE); clint_clear_ipi(); #endif =20 --=20 2.43.1 From nobody Tue Dec 16 20:00:28 2025 Received: from mail-pf1-f182.google.com (mail-pf1-f182.google.com [209.85.210.182]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 193DD1172C for ; Wed, 27 Mar 2024 04:50:46 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.182 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711515048; cv=none; b=KctcYKumxOnH+084tZ1DneRkJ258XFsnWeqmC94kHJMBtKoDahgk9xdbYgNU9nN7DNp/kZiMxqVK5bPkT4+1XzTWqPwBnb8n4dsOaMzR0aJk9SeSEYy1PKkU1EN0CZF8/SdUxrkBH5DIegadf6gMNK+4AjKE9jrIXodaDlBx0PE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711515048; c=relaxed/simple; bh=R5LG+o542v8AgH1AF2U8/WvJmQDWsmGeEqymqdAXiWA=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=egTOt/aKAoRKlrbK9wrIXMBppY+J1cKNHZvsDLmGL11t8Ar8RWyKdWv1anPk1NeZsd92p/jgoe6YR6CXkIi5mm0pkB93fYL2IwzU8Hj1X4vGe/yE7XcbRwvmmHOt/aJP+CXK9kJ6L8GS701DzpJBypG3sN+BGYW4UFzBQQySfkE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=sifive.com; spf=pass smtp.mailfrom=sifive.com; dkim=pass (2048-bit key) header.d=sifive.com header.i=@sifive.com header.b=ivDqwdIa; arc=none smtp.client-ip=209.85.210.182 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=sifive.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=sifive.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=sifive.com header.i=@sifive.com header.b="ivDqwdIa" Received: by mail-pf1-f182.google.com with SMTP id d2e1a72fcca58-6ea9a605ca7so381411b3a.0 for ; Tue, 26 Mar 2024 21:50:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1711515046; x=1712119846; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=2fwmOl0CxtBfQxlaR9NQyd7IGBnsfsK+MLDyKM1aSvU=; b=ivDqwdIaeIS4u4I3npdDBQdffMhJJQeh20WWT6pm0Q4PdS9Cs0oJ48UhUZlmj7Rd3P fKN8HAdCH7xJAl7EqUivq5maekwcBzMV1CRmt0h9/A2i694iV6m6i59gotVv+LZQerJ+ 8fGlwFvAm0JuGVnzxRsF57Bunob2b5RhdAdQyvyat2qcU1pfgsc6tFdMaYkp5Tg7JyiC 6QFK+X+qzhSF+8KQa998g8UCBymSwfGaDUw3zK2kVJQHsmE3nc9yeY5EAfeVhGPIFV/6 pjxqFhY4fCy1HOd5HBduDHIbFGROdvqKjb6dt4YN5KxgBbLv9Yz3qTsKM2HdmOGvdeV3 h9Vw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711515046; x=1712119846; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=2fwmOl0CxtBfQxlaR9NQyd7IGBnsfsK+MLDyKM1aSvU=; b=HnBirnpU2Xf2gTZC6udruUNWc36K2zI3M646JOIYn82hF0oVLPECJDQrBX56/+Xyfb ZK07Ck9qfAkRZI9llM42jdnE+/s6L7YQBPLWf0R3ktRyajqRXfAjX5Tu1DXdDxhtXY4s zE0s4JYIkTId3Uhq/pK5G3jzzy+AsFlu1mYCS3laCeYu3j9jfjLI1n+IMNTrttvTisUF B+V648mebULeznRKw9iKXea8NYt6GbazCZ3CWTgOLaITVQ/aVrCxlvPTstoSzOe3FV1S oPz3dEjF1IpQlDPEQl9tgB1sDSKivPw+4z/MKYHQ3WEWCdocn5MAUxOFVXjaVax+S20a Dhqw== X-Gm-Message-State: AOJu0Yzd2Tej93u/CjKWksCgy5neNBh6FE8eS3m3DJ61vx/4QwpeohZA gb1R9NU5f+TEV49N8Y/OmwL8HDQ24OYMigQqUJ89/sxZQ0+HIEk48lc70nnMhTA= X-Google-Smtp-Source: AGHT+IGHAmpkwghveoQCxqBYVJp1RVR15SSZ2ypXvD8LXUp94lm0FkTd5l29fFm2kWdFZwCld1auyw== X-Received: by 2002:a05:6a20:3d11:b0:1a3:6a11:4898 with SMTP id y17-20020a056a203d1100b001a36a114898mr4924839pzi.3.1711515046438; Tue, 26 Mar 2024 21:50:46 -0700 (PDT) Received: from sw06.internal.sifive.com ([4.53.31.132]) by smtp.gmail.com with ESMTPSA id e31-20020a63501f000000b005e4666261besm8351500pgb.50.2024.03.26.21.50.45 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 26 Mar 2024 21:50:45 -0700 (PDT) From: Samuel Holland To: Palmer Dabbelt , linux-riscv@lists.infradead.org Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, Alexandre Ghiti , Jisheng Zhang , Yunhui Cui , Samuel Holland Subject: [PATCH v6 04/13] riscv: mm: Broadcast kernel TLB flushes only when needed Date: Tue, 26 Mar 2024 21:49:45 -0700 Message-ID: <20240327045035.368512-5-samuel.holland@sifive.com> X-Mailer: git-send-email 2.43.1 In-Reply-To: <20240327045035.368512-1-samuel.holland@sifive.com> References: <20240327045035.368512-1-samuel.holland@sifive.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" __flush_tlb_range() avoids broadcasting TLB flushes when an mm context is only active on the local CPU. Apply this same optimization to TLB flushes of kernel memory when only one CPU is online. This check can be constant-folded when SMP is disabled. Reviewed-by: Alexandre Ghiti Signed-off-by: Samuel Holland --- (no changes since v4) Changes in v4: - New patch for v4 arch/riscv/mm/tlbflush.c | 18 +++++------------- 1 file changed, 5 insertions(+), 13 deletions(-) diff --git a/arch/riscv/mm/tlbflush.c b/arch/riscv/mm/tlbflush.c index 0435605b07d0..da821315d43e 100644 --- a/arch/riscv/mm/tlbflush.c +++ b/arch/riscv/mm/tlbflush.c @@ -103,22 +103,15 @@ static void __flush_tlb_range(struct cpumask *cmask, = unsigned long asid, unsigned long start, unsigned long size, unsigned long stride) { - bool broadcast; + unsigned int cpu; =20 if (cpumask_empty(cmask)) return; =20 - if (cmask !=3D cpu_online_mask) { - unsigned int cpuid; + cpu =3D get_cpu(); =20 - cpuid =3D get_cpu(); - /* check if the tlbflush needs to be sent to other CPUs */ - broadcast =3D cpumask_any_but(cmask, cpuid) < nr_cpu_ids; - } else { - broadcast =3D true; - } - - if (!broadcast) { + /* Check if the TLB flush needs to be sent to other CPUs. */ + if (cpumask_any_but(cmask, cpu) >=3D nr_cpu_ids) { local_flush_tlb_range_asid(start, size, stride, asid); } else if (riscv_use_sbi_for_rfence()) { sbi_remote_sfence_vma_asid(cmask, start, size, asid); @@ -132,8 +125,7 @@ static void __flush_tlb_range(struct cpumask *cmask, un= signed long asid, on_each_cpu_mask(cmask, __ipi_flush_tlb_range_asid, &ftd, 1); } =20 - if (cmask !=3D cpu_online_mask) - put_cpu(); + put_cpu(); } =20 static inline unsigned long get_mm_asid(struct mm_struct *mm) --=20 2.43.1 From nobody Tue Dec 16 20:00:28 2025 Received: from mail-ot1-f44.google.com (mail-ot1-f44.google.com [209.85.210.44]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 30CDD17565 for ; Wed, 27 Mar 2024 04:50:48 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.44 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711515050; cv=none; b=mRpoa/do7sKCb5cgvitWt/RuTpU7xvubwqyW7zz9IQ4yd50T4L02GoL6g7NJXEQv0/zm67LjPojrYG4e/E1IA3qAYv3+9N6GCjB1pHjo4IT33k0m9/eZWM6uHQUJu/ZJUXwY+q2attUCS0QFD8cqfTE/FmuuYS14Ql+Ly+6ghZU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711515050; c=relaxed/simple; bh=ya97YfXpoRK/xYSjnVMkP8sXm9+Mh9TCgaVbVD32L2Y=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=Jxmp1W/fJrYjummRNqhLBMiZ6AmMwLtdtBrbyhZCm5NNA+LJaeaN2SsNk+WU38JWDNgHwJTOfkQ72zansm4XAk5Q1ij6L44bIDbOTGXSeKkM5JghDVkgKMvyOKK/gkMi+q+YsQo/Ki2NP7reihJeKQzEYSDPPNsMM9/7LAlDStA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=sifive.com; spf=pass smtp.mailfrom=sifive.com; dkim=pass (2048-bit key) header.d=sifive.com header.i=@sifive.com header.b=nwXZug69; arc=none smtp.client-ip=209.85.210.44 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=sifive.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=sifive.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=sifive.com header.i=@sifive.com header.b="nwXZug69" Received: by mail-ot1-f44.google.com with SMTP id 46e09a7af769-6e67b5d6dd8so3499275a34.2 for ; Tue, 26 Mar 2024 21:50:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1711515048; x=1712119848; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=8r90DmKfTzo12+b9oHF/xBbsXVRSBOTytxxA308aMe8=; b=nwXZug69+IWYGlBdBnH+/S0tnz8xTjrFivzLBhkfZu916ZIO7k8frGmvIrXU6F2BEB TP7b3LdkOt7LhwjHG88NAsgbK7Eg4byv2iaCrOM2j1gXbiXT5d/hjq0oeG3+n8ySCUcw PRaMjJSsIolgo1DuhZI/p0RG+VCfaEe75TVpuBTJkDEDdQ5vl8sN60j3rDaCEtCq1sph IHM7ajqVmrmigUnMAkt88+9F/xhu1vul6dUp5Rjl5Pqgdt0Z3rtyZArpNT25fqDX6gI7 pVP2hDuBBSvnOobpHnjkORXkP7mSMiiqXLV7e9JMw46t0B4ou0VarDBYl53xrACdz2X5 Dy8w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711515048; x=1712119848; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=8r90DmKfTzo12+b9oHF/xBbsXVRSBOTytxxA308aMe8=; b=g7gai5jqF5m4sEJiabBU8umyFK2lWIOAVudSkl+5OYq2kuo/8AuQIPBsQNFxwUrOXg rZQ9IX33CdPw0/zzPnM3uB1OEm75cfNWzCujxeB36uRSwyDBIPXzuJN6Om7I9BOyrEiT /db/YzvNRjJ1F+4UlRZCN4yMO7nFRRmrcnWoV8ED0oMou9g00EwGJYnUmd97qsOK4TAf O/ultPR+4hKeBAbKyyrc64plDkV/0Lc7MkcY9nO7BPdQLGpvVzYEbmu/ebo6xtMg36Xh zPdFJFd95LPy1OOv2gbKZSVtuQcyYBhQV2aku7YYNQhIfqJ0Vag+N6Ymwhq7hqiuWGYZ 9o1w== X-Gm-Message-State: AOJu0YyCS/D762bV8UEOsSi/metpt5jPv4msbVWf0mt911In6SNBfdW7 27L73H041F8LHMxt8fRIl4T6Q5i6/1WbEjmLMagjhGnlA9R7y2wnxLlf/zEOSEs= X-Google-Smtp-Source: AGHT+IE/gtW/a2EQvN+qhLpDTcQ1EZrF+9VNRUlZn60iPj1iarim6P+gGyjDapgmLmO07m/XYQ3rgg== X-Received: by 2002:a05:6830:1e72:b0:6e6:7db1:b444 with SMTP id m18-20020a0568301e7200b006e67db1b444mr154078otr.36.1711515048291; Tue, 26 Mar 2024 21:50:48 -0700 (PDT) Received: from sw06.internal.sifive.com ([4.53.31.132]) by smtp.gmail.com with ESMTPSA id e31-20020a63501f000000b005e4666261besm8351500pgb.50.2024.03.26.21.50.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 26 Mar 2024 21:50:47 -0700 (PDT) From: Samuel Holland To: Palmer Dabbelt , linux-riscv@lists.infradead.org Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, Alexandre Ghiti , Jisheng Zhang , Yunhui Cui , Samuel Holland Subject: [PATCH v6 05/13] riscv: Only send remote fences when some other CPU is online Date: Tue, 26 Mar 2024 21:49:46 -0700 Message-ID: <20240327045035.368512-6-samuel.holland@sifive.com> X-Mailer: git-send-email 2.43.1 In-Reply-To: <20240327045035.368512-1-samuel.holland@sifive.com> References: <20240327045035.368512-1-samuel.holland@sifive.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" If no other CPU is online, a local cache or TLB flush is sufficient. These checks can be constant-folded when SMP is disabled. Signed-off-by: Samuel Holland Reviewed-by: Alexandre Ghiti --- (no changes since v4) Changes in v4: - New patch for v4 arch/riscv/mm/cacheflush.c | 4 +++- arch/riscv/mm/tlbflush.c | 4 +++- 2 files changed, 6 insertions(+), 2 deletions(-) diff --git a/arch/riscv/mm/cacheflush.c b/arch/riscv/mm/cacheflush.c index d76fc73e594b..f5be1fec8191 100644 --- a/arch/riscv/mm/cacheflush.c +++ b/arch/riscv/mm/cacheflush.c @@ -21,7 +21,9 @@ void flush_icache_all(void) { local_flush_icache_all(); =20 - if (riscv_use_sbi_for_rfence()) + if (num_online_cpus() < 2) + return; + else if (riscv_use_sbi_for_rfence()) sbi_remote_fence_i(NULL); else on_each_cpu(ipi_remote_fence_i, NULL, 1); diff --git a/arch/riscv/mm/tlbflush.c b/arch/riscv/mm/tlbflush.c index da821315d43e..0901aa47b58f 100644 --- a/arch/riscv/mm/tlbflush.c +++ b/arch/riscv/mm/tlbflush.c @@ -79,7 +79,9 @@ static void __ipi_flush_tlb_all(void *info) =20 void flush_tlb_all(void) { - if (riscv_use_sbi_for_rfence()) + if (num_online_cpus() < 2) + local_flush_tlb_all(); + else if (riscv_use_sbi_for_rfence()) sbi_remote_sfence_vma_asid(NULL, 0, FLUSH_TLB_MAX_SIZE, FLUSH_TLB_NO_ASI= D); else on_each_cpu(__ipi_flush_tlb_all, NULL, 1); --=20 2.43.1 From nobody Tue Dec 16 20:00:28 2025 Received: from mail-pf1-f172.google.com (mail-pf1-f172.google.com [209.85.210.172]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B120C1DA21 for ; Wed, 27 Mar 2024 04:50:51 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711515053; cv=none; b=rO9Xl1OxmN4hDUbLR+SsD8R6tiDhDOoM/9BAZvuqwRSJKg07I3vgLlTrPa8fA7LvhREOOvIAmZ4/R1/afc0ZwmCiY/tbcrLXOOCWT4gNY0zqtbEsVEcboB4UcuI2xRUDl79j6wA1RVoouYdBRDqB/w7texDW/I+eVcMoFMKMWQ4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711515053; c=relaxed/simple; bh=mRY/IQDj/i6mPg/qYecdh9eDZwLkEM+YLCus0ff2B10=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=EeDvHZUnGQDhyeVtj9FUG16WMCDiIThHWTRdL8oh1PAn3yrp6ORBvmlMbWbeSc+ewN0KikATQ9MvVQiJ2bxAPy8URmP0RSKstSj1tvNVEEEWzSYXpXJL9u1pLbmP9g3EySpeOM2S0xXONok8JALkS7RCOzhdeC79oQa9XpFRErw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=sifive.com; spf=pass smtp.mailfrom=sifive.com; dkim=pass (2048-bit key) header.d=sifive.com header.i=@sifive.com header.b=gjmlZvCC; arc=none smtp.client-ip=209.85.210.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=sifive.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=sifive.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=sifive.com header.i=@sifive.com header.b="gjmlZvCC" Received: by mail-pf1-f172.google.com with SMTP id d2e1a72fcca58-6e73e8bdea2so5252904b3a.0 for ; Tue, 26 Mar 2024 21:50:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1711515051; x=1712119851; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=7nL2m+hLarkGQvecSEd0KAq0QtYwqUItfZBnVAx3yeI=; b=gjmlZvCCdiOp8Uq0kVfrm8sXXwopEDQ5DxYHvzKX15vXD9+27lQ3PwPQ4488t6MGyG v5H3Vwv0z2cD1uGKEXdzxblbXYcdVg84u6uGSU9pfyLhtkaSuFk8F/TLhCqo3nxRutH4 XXyofF+dLT38TDTf9H+5fCOyGS0j+fdT1KPeU4eTSRkym6fa3hswfsg14+MkWOIFStUm CoohVQkX+zDbR3RKs+gag9rC9Y+eE3PQ1aT43TIp43PbwD9WdwFksBFGP2caeKighcuA tjstoXutVPYqbpIUQtwL0VaaRJGetJ7JylnNvp2Zf8b4uAmdFLOZN0mpdDSIXvS/eilo fzcg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711515051; x=1712119851; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=7nL2m+hLarkGQvecSEd0KAq0QtYwqUItfZBnVAx3yeI=; b=m1I33sm+3lEfRR8nF6FzysqJTofTm0o6QQVFawD70YwvQOoZ8mWJ6QVGm1GU0FPYlU /rghyf3uI4uQ8Of2/VCZjNonvKZsNA1JZsyWOKY5JTolIEevQt19leOOiyfbWFFwyhXM Ibt5Ms3R4R3suwnok1ZFAeI60LCkrkuzcRF4vYVuweA/NIoXgKRfc2D+N/3wCYmFvaBW B4wgFXTH5FZYeUHwkHFRMrhm7wx/RgJ8KlXxJ09HHTcM/T84AhkPurT/IAiiylHIfQYU uhCT3gDcIZ/Endzz17kNypuB2bJzdNzyTCA2l1JOpl+Ag6YLUg9PV9mYTf9AGsTX1ggr SgIg== X-Gm-Message-State: AOJu0Yx+kry2es7gVqHOn9C9XJZspdUXUO4PUlpniGunt7MlucnFmNyN /oXyQtIjR7cXXECmMRlm3BKIYUygMZ39ir5t44vZVRrYhP5zGJ1n99+W5gN9lF8= X-Google-Smtp-Source: AGHT+IGyz4xXUqNq5aXCdK4lWIBgBxDZJLK6nH/Xm7w+Te1ce++GEK9gbqC/aCbf24OJ2Eh82pVCjA== X-Received: by 2002:a05:6a00:13a2:b0:6ea:8e89:7eca with SMTP id t34-20020a056a0013a200b006ea8e897ecamr1940335pfg.32.1711515050263; Tue, 26 Mar 2024 21:50:50 -0700 (PDT) Received: from sw06.internal.sifive.com ([4.53.31.132]) by smtp.gmail.com with ESMTPSA id e31-20020a63501f000000b005e4666261besm8351500pgb.50.2024.03.26.21.50.48 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 26 Mar 2024 21:50:49 -0700 (PDT) From: Samuel Holland To: Palmer Dabbelt , linux-riscv@lists.infradead.org Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, Alexandre Ghiti , Jisheng Zhang , Yunhui Cui , Samuel Holland Subject: [PATCH v6 06/13] riscv: mm: Combine the SMP and UP TLB flush code Date: Tue, 26 Mar 2024 21:49:47 -0700 Message-ID: <20240327045035.368512-7-samuel.holland@sifive.com> X-Mailer: git-send-email 2.43.1 In-Reply-To: <20240327045035.368512-1-samuel.holland@sifive.com> References: <20240327045035.368512-1-samuel.holland@sifive.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" In SMP configurations, all TLB flushing narrower than flush_tlb_all() goes through __flush_tlb_range(). Do the same in UP configurations. This allows UP configurations to take advantage of recent improvements to the code in tlbflush.c, such as support for huge pages and flushing multiple-page ranges. Reviewed-by: Alexandre Ghiti Signed-off-by: Samuel Holland Reviewed-by: Yunhui Cui --- (no changes since v4) Changes in v4: - Merge the two copies of __flush_tlb_range() and rely on the compiler to optimize out the broadcast path (both clang and gcc do this) - Merge the two copies of flush_tlb_all() and rely on constant folding Changes in v2: - Move the SMP/UP merge earlier in the series to avoid build issues - Make a copy of __flush_tlb_range() instead of adding ifdefs inside - local_flush_tlb_all() is the only function used on !MMU (smpboot.c) arch/riscv/Kconfig | 2 +- arch/riscv/include/asm/tlbflush.h | 31 +++---------------------------- arch/riscv/mm/Makefile | 5 +---- 3 files changed, 5 insertions(+), 33 deletions(-) diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig index be09c8836d56..442532819a44 100644 --- a/arch/riscv/Kconfig +++ b/arch/riscv/Kconfig @@ -60,7 +60,7 @@ config RISCV select ARCH_USE_MEMTEST select ARCH_USE_QUEUED_RWLOCKS select ARCH_USES_CFI_TRAPS if CFI_CLANG - select ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH if SMP && MMU + select ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH if MMU select ARCH_WANT_DEFAULT_TOPDOWN_MMAP_LAYOUT if MMU select ARCH_WANT_FRAME_POINTERS select ARCH_WANT_GENERAL_HUGETLB if !RISCV_ISA_SVNAPOT diff --git a/arch/riscv/include/asm/tlbflush.h b/arch/riscv/include/asm/tlb= flush.h index 4112cc8d1d69..4f86424b1ba5 100644 --- a/arch/riscv/include/asm/tlbflush.h +++ b/arch/riscv/include/asm/tlbflush.h @@ -27,12 +27,7 @@ static inline void local_flush_tlb_page(unsigned long ad= dr) { ALT_FLUSH_TLB_PAGE(__asm__ __volatile__ ("sfence.vma %0" : : "r" (addr) := "memory")); } -#else /* CONFIG_MMU */ -#define local_flush_tlb_all() do { } while (0) -#define local_flush_tlb_page(addr) do { } while (0) -#endif /* CONFIG_MMU */ =20 -#if defined(CONFIG_SMP) && defined(CONFIG_MMU) void flush_tlb_all(void); void flush_tlb_mm(struct mm_struct *mm); void flush_tlb_mm_range(struct mm_struct *mm, unsigned long start, @@ -54,28 +49,8 @@ void arch_tlbbatch_add_pending(struct arch_tlbflush_unma= p_batch *batch, unsigned long uaddr); void arch_flush_tlb_batched_pending(struct mm_struct *mm); void arch_tlbbatch_flush(struct arch_tlbflush_unmap_batch *batch); - -#else /* CONFIG_SMP && CONFIG_MMU */ - -#define flush_tlb_all() local_flush_tlb_all() -#define flush_tlb_page(vma, addr) local_flush_tlb_page(addr) - -static inline void flush_tlb_range(struct vm_area_struct *vma, - unsigned long start, unsigned long end) -{ - local_flush_tlb_all(); -} - -/* Flush a range of kernel pages */ -static inline void flush_tlb_kernel_range(unsigned long start, - unsigned long end) -{ - local_flush_tlb_all(); -} - -#define flush_tlb_mm(mm) flush_tlb_all() -#define flush_tlb_mm_range(mm, start, end, page_size) flush_tlb_all() -#define local_flush_tlb_kernel_range(start, end) flush_tlb_all() -#endif /* !CONFIG_SMP || !CONFIG_MMU */ +#else /* CONFIG_MMU */ +#define local_flush_tlb_all() do { } while (0) +#endif /* CONFIG_MMU */ =20 #endif /* _ASM_RISCV_TLBFLUSH_H */ diff --git a/arch/riscv/mm/Makefile b/arch/riscv/mm/Makefile index 2c869f8026a8..cbe4d775ef56 100644 --- a/arch/riscv/mm/Makefile +++ b/arch/riscv/mm/Makefile @@ -13,14 +13,11 @@ endif KCOV_INSTRUMENT_init.o :=3D n =20 obj-y +=3D init.o -obj-$(CONFIG_MMU) +=3D extable.o fault.o pageattr.o pgtable.o +obj-$(CONFIG_MMU) +=3D extable.o fault.o pageattr.o pgtable.o tlbflush.o obj-y +=3D cacheflush.o obj-y +=3D context.o obj-y +=3D pmem.o =20 -ifeq ($(CONFIG_MMU),y) -obj-$(CONFIG_SMP) +=3D tlbflush.o -endif obj-$(CONFIG_HUGETLB_PAGE) +=3D hugetlbpage.o obj-$(CONFIG_PTDUMP_CORE) +=3D ptdump.o obj-$(CONFIG_KASAN) +=3D kasan_init.o --=20 2.43.1 From nobody Tue Dec 16 20:00:28 2025 Received: from mail-ot1-f45.google.com (mail-ot1-f45.google.com [209.85.210.45]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E032D1EB5E for ; Wed, 27 Mar 2024 04:50:52 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.45 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711515054; cv=none; b=sjPA7Ts771YXyFlyWFHzLzmBI94q37Gz8+AcwDrM2jMDKNyskExHadnvI6Rz0+Yy6vOQ2eTnb63zbYxFzkzEUiGJfqppaBIt8cAp2xvTvl5eXSaAYd/+/6XoDropqo4XkPihq5fsxdb8z8UF7gXEdDWGKEf7SZ4n0QGP3vN6lMw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711515054; c=relaxed/simple; bh=rWKHETzcUwKRgItYuayevEp2FgvSWRc2ekB7V6Se14w=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=OmCyIPh1vS9VRHnrbqceF3x86o7TZ6L6qRVdCv8VX4EevrcK1hmhEPK6EVxuHFnRVk/V0QBitkVgSZ4/X5iBmlHc2yReMnZC1cNU+rNdGanJzi4/1uP+o2APbNTZfSWE9x+VWkEji3baXGw+en1OYPefafVMMtMrCtJAhm9R774= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=sifive.com; spf=pass smtp.mailfrom=sifive.com; dkim=pass (2048-bit key) header.d=sifive.com header.i=@sifive.com header.b=fETlCPHt; arc=none smtp.client-ip=209.85.210.45 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=sifive.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=sifive.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=sifive.com header.i=@sifive.com header.b="fETlCPHt" Received: by mail-ot1-f45.google.com with SMTP id 46e09a7af769-6e6a5bd015dso2946637a34.2 for ; Tue, 26 Mar 2024 21:50:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1711515052; x=1712119852; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=kHzKH2DTKf6Vc5BlA0YNBJTz56l+llAyGgYGMd65KTs=; b=fETlCPHtdNJxR8pqGTFAtCaLtDfQQP1DtouZ0HjzmOrJlp27fQMOR0oanT+WvNQz5i f9YCJL2OScRlPZ/VEerkJJZ/IaiO7c+Q7YB1uylVKphjf2jV+mYMSYynusHXeCwnwVs8 gXaeWgVzu07vi2KXUX5GTKeXM/fUErVVFLroCKPbJa4CkgRG22U3cXu1FmH5/02pAwGQ CHlSdYOIEqbvkVPsqdjmejJ2iyCiP8SPIM+GVzPGECNBd0fSUFlVCEhuqFC3rpXh+t1T Y+z3vPr9femLY1r43tkz8JXOZiZuv1OQv3LYYOdC28hyog7HGKviwwxz6LxgoMjnAxv+ aPag== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711515052; x=1712119852; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=kHzKH2DTKf6Vc5BlA0YNBJTz56l+llAyGgYGMd65KTs=; b=u5jKf05QAOsj9TqCuo1n5YJSiyjov87BA8fid3k4cD4/Z2ESE6vcOiPo3s7UuOR/MX OYjkpP8JmihMywzZCH84RbH1qSRz4wT/Heuge8BJptnYqmGvUrCL0j+GHkwcMlySjBHL SsUvz2lhO6T/5u1f8zsNzPr6pHDCxlecaqf592zzcPtwVFYMQlfFHN4ryet7dPqeq7G1 Ay5c3mw1L5ZHWcVhQPs0kKXUlHAN2wfzsJyyeaYspqwyovxhQqM3WedHvY6HWGYLi2GA IguolVdndSJ2vZH+OgUyHIQ1Xfle3V5ZKU9dVMk3Xu21TYHGrVJa5AJlgia7AslJoCUY kDgg== X-Gm-Message-State: AOJu0YyHhnfYKmqp0x+4obKcJ0m/3Hoe0/owATFCPA1gPjkFaZVryOHm 9x4WHJNmh9kahTgtLgGiw8YrdUXaZC8kduygKQTY70549C4N+jhnzNvqc0+Oyps= X-Google-Smtp-Source: AGHT+IFNZ9hbtEq1CmgyYTIu4vUTxuEiR33JD4yc+xyNKXepJuE48avCWWfj7WZgftYdmFVpAMAO4A== X-Received: by 2002:a9d:6e0c:0:b0:6e6:b2ed:f12a with SMTP id e12-20020a9d6e0c000000b006e6b2edf12amr1764968otr.30.1711515052040; Tue, 26 Mar 2024 21:50:52 -0700 (PDT) Received: from sw06.internal.sifive.com ([4.53.31.132]) by smtp.gmail.com with ESMTPSA id e31-20020a63501f000000b005e4666261besm8351500pgb.50.2024.03.26.21.50.50 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 26 Mar 2024 21:50:51 -0700 (PDT) From: Samuel Holland To: Palmer Dabbelt , linux-riscv@lists.infradead.org Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, Alexandre Ghiti , Jisheng Zhang , Yunhui Cui , Samuel Holland Subject: [PATCH v6 07/13] riscv: Apply SiFive CIP-1200 workaround to single-ASID sfence.vma Date: Tue, 26 Mar 2024 21:49:48 -0700 Message-ID: <20240327045035.368512-8-samuel.holland@sifive.com> X-Mailer: git-send-email 2.43.1 In-Reply-To: <20240327045035.368512-1-samuel.holland@sifive.com> References: <20240327045035.368512-1-samuel.holland@sifive.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" commit 3f1e782998cd ("riscv: add ASID-based tlbflushing methods") added calls to the sfence.vma instruction with rs2 !=3D x0. These single-ASID instruction variants are also affected by SiFive errata CIP-1200. Until now, the errata workaround was not needed for the single-ASID sfence.vma variants, because they were only used when the ASID allocator was enabled, and the affected SiFive platforms do not support multiple ASIDs. However, we are going to start using those sfence.vma variants regardless of ASID support, so now we need alternatives covering them. Signed-off-by: Samuel Holland --- (no changes since v2) Changes in v2: - Rebase on Alexandre's "riscv: tlb flush improvements" series v5 arch/riscv/include/asm/errata_list.h | 12 +++++++++++- arch/riscv/include/asm/tlbflush.h | 19 ++++++++++++++++++- arch/riscv/mm/tlbflush.c | 23 ----------------------- 3 files changed, 29 insertions(+), 25 deletions(-) diff --git a/arch/riscv/include/asm/errata_list.h b/arch/riscv/include/asm/= errata_list.h index 1f2dbfb8a8bf..35ce26899960 100644 --- a/arch/riscv/include/asm/errata_list.h +++ b/arch/riscv/include/asm/errata_list.h @@ -43,11 +43,21 @@ ALTERNATIVE(__stringify(RISCV_PTR do_page_fault), \ CONFIG_ERRATA_SIFIVE_CIP_453) #else /* !__ASSEMBLY__ */ =20 -#define ALT_FLUSH_TLB_PAGE(x) \ +#define ALT_SFENCE_VMA_ASID(asid) \ +asm(ALTERNATIVE("sfence.vma x0, %0", "sfence.vma", SIFIVE_VENDOR_ID, \ + ERRATA_SIFIVE_CIP_1200, CONFIG_ERRATA_SIFIVE_CIP_1200) \ + : : "r" (asid) : "memory") + +#define ALT_SFENCE_VMA_ADDR(addr) \ asm(ALTERNATIVE("sfence.vma %0", "sfence.vma", SIFIVE_VENDOR_ID, \ ERRATA_SIFIVE_CIP_1200, CONFIG_ERRATA_SIFIVE_CIP_1200) \ : : "r" (addr) : "memory") =20 +#define ALT_SFENCE_VMA_ADDR_ASID(addr, asid) \ +asm(ALTERNATIVE("sfence.vma %0, %1", "sfence.vma", SIFIVE_VENDOR_ID, \ + ERRATA_SIFIVE_CIP_1200, CONFIG_ERRATA_SIFIVE_CIP_1200) \ + : : "r" (addr), "r" (asid) : "memory") + /* * _val is marked as "will be overwritten", so need to set it to 0 * in the default case. diff --git a/arch/riscv/include/asm/tlbflush.h b/arch/riscv/include/asm/tlb= flush.h index 4f86424b1ba5..463b615d7728 100644 --- a/arch/riscv/include/asm/tlbflush.h +++ b/arch/riscv/include/asm/tlbflush.h @@ -22,10 +22,27 @@ static inline void local_flush_tlb_all(void) __asm__ __volatile__ ("sfence.vma" : : : "memory"); } =20 +static inline void local_flush_tlb_all_asid(unsigned long asid) +{ + if (asid !=3D FLUSH_TLB_NO_ASID) + ALT_SFENCE_VMA_ASID(asid); + else + local_flush_tlb_all(); +} + /* Flush one page from local TLB */ static inline void local_flush_tlb_page(unsigned long addr) { - ALT_FLUSH_TLB_PAGE(__asm__ __volatile__ ("sfence.vma %0" : : "r" (addr) := "memory")); + ALT_SFENCE_VMA_ADDR(addr); +} + +static inline void local_flush_tlb_page_asid(unsigned long addr, + unsigned long asid) +{ + if (asid !=3D FLUSH_TLB_NO_ASID) + ALT_SFENCE_VMA_ADDR_ASID(addr, asid); + else + local_flush_tlb_page(addr); } =20 void flush_tlb_all(void); diff --git a/arch/riscv/mm/tlbflush.c b/arch/riscv/mm/tlbflush.c index 0901aa47b58f..ad7bdcfcc219 100644 --- a/arch/riscv/mm/tlbflush.c +++ b/arch/riscv/mm/tlbflush.c @@ -7,29 +7,6 @@ #include #include =20 -static inline void local_flush_tlb_all_asid(unsigned long asid) -{ - if (asid !=3D FLUSH_TLB_NO_ASID) - __asm__ __volatile__ ("sfence.vma x0, %0" - : - : "r" (asid) - : "memory"); - else - local_flush_tlb_all(); -} - -static inline void local_flush_tlb_page_asid(unsigned long addr, - unsigned long asid) -{ - if (asid !=3D FLUSH_TLB_NO_ASID) - __asm__ __volatile__ ("sfence.vma %0, %1" - : - : "r" (addr), "r" (asid) - : "memory"); - else - local_flush_tlb_page(addr); -} - /* * Flush entire TLB if number of entries to be flushed is greater * than the threshold below. --=20 2.43.1 From nobody Tue Dec 16 20:00:28 2025 Received: from mail-ot1-f54.google.com (mail-ot1-f54.google.com [209.85.210.54]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C73EB1C68F for ; Wed, 27 Mar 2024 04:50:54 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.54 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711515056; cv=none; b=or4mIshcemuFySr/OwMEaR7BcZuPwFvJ6w2StSzvHY4qO8kRgYpbnD7Oy2j7gj/om9cQ7maSl2xuqpoadEeGFv29wlYf9OpZFrAiQRktQvGLsLNjZtzydWPq13ZvSaIJXZ2ut5IsjR2mCYlXxRMYff677WyTu0jFJx54Bx/5FzA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711515056; c=relaxed/simple; bh=Iif7UTJ12URNnNmgEZh/g3vOOkBXUamO89prNktl0Fg=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=LU9WZCHXevtNuMOA3/l51sB8m265C2OF1r9/JfEZdtrwUuqTRd9JiQ3m/WOq53rP6iaf1vHMdG44N9gyVHFRoEeSj4MwTjgV8lvKARyaatqQOHTEastSUakSlzGEDkXOH0YuZcAxAphTmcxXL4hlCKn8Cq1lubsOADtuRrz/yro= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=sifive.com; spf=pass smtp.mailfrom=sifive.com; dkim=pass (2048-bit key) header.d=sifive.com header.i=@sifive.com header.b=cMSxOfqG; arc=none smtp.client-ip=209.85.210.54 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=sifive.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=sifive.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=sifive.com header.i=@sifive.com header.b="cMSxOfqG" Received: by mail-ot1-f54.google.com with SMTP id 46e09a7af769-6e6e37d33c5so992083a34.2 for ; Tue, 26 Mar 2024 21:50:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1711515054; x=1712119854; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=4DMxbhiunBS0lKLURYVBJG3nyObzwUO8ZrXza9ckh2c=; b=cMSxOfqGKSsa7JMEOtNfyE0WC1EZgJeCD74yAJ3guvzcHJejioBByzrzpjIwR6n1Y2 SkFsT6xh+y9jXXHtWdZSn7xE68jAH7Ou4tybR0KYqnJTiag64Zo1HCORkzOr9dO8f8ZL /QKtDOY1XgdHTieiblQAZqN5G8Xfpbx4vjtTzIw4D04YOzOuJZ6AenRcYK76YF4L6zWw gprcbQG/nufiP0qh6ahE+EW0w2kuOMKBX8BKVnXw2sDoczvlWv+uMf3bkfOcAIjEfAT6 7fP/qGcJuVjFhQAIA6M29bGHpIXNoJIrxUaGe/QggZwH2hC5NxP0MvBsXZqtgFPyT23C IPIw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711515054; x=1712119854; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=4DMxbhiunBS0lKLURYVBJG3nyObzwUO8ZrXza9ckh2c=; b=YB1hvlkDL5fZJ4AeVrfF5GJ6qL2Fd16OQhhwSbeWbBI9UYvbjiXLHg2t3jKIb9rQkU KP4QYcrTagTV8qD72B+PaGhcHU5Iln7a0dLnMUwvoW0VGmcQG+SmEkSKy9K7j1InCW5W moKjMhmBZR+H/Yqmaz2yH8B54hD3gFOs+YkB9j1eO7cysZYrdou+JmQaFnI1X+b83oLY r+mPrSJ9nVTWGAaYpr26EwlKvsQDIzpAR6enouJlZTrkfUeVPxts9RxSac1gzScAjS2/ 7fSaLCaK20R6MuySg3rkTzRLQhKoQkMbooP/UfWZvNXdwat8uauhwxgdCeR+DAOsFUkm Je6Q== X-Gm-Message-State: AOJu0YxPc2OXunFssFZWVfLILlQhlk+HlR2UFGwpx0jznKZz7AX0Xede +d+rYjjNg43afU/zvefo4CPg4BqU8LvJ8ZpZyCHncceCp4ecZGDt92xIH1lqU9Y= X-Google-Smtp-Source: AGHT+IEKVJUxYPHGmGQ7jZ/KC0+y46rPbmkW2DQ4LqVeHQ3jmUBdRhZe5PVjdoDThc5iPwQM0jPRUg== X-Received: by 2002:a9d:61d8:0:b0:6e6:9af5:752b with SMTP id h24-20020a9d61d8000000b006e69af5752bmr1730488otk.0.1711515053696; Tue, 26 Mar 2024 21:50:53 -0700 (PDT) Received: from sw06.internal.sifive.com ([4.53.31.132]) by smtp.gmail.com with ESMTPSA id e31-20020a63501f000000b005e4666261besm8351500pgb.50.2024.03.26.21.50.52 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 26 Mar 2024 21:50:52 -0700 (PDT) From: Samuel Holland To: Palmer Dabbelt , linux-riscv@lists.infradead.org Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, Alexandre Ghiti , Jisheng Zhang , Yunhui Cui , Samuel Holland Subject: [PATCH v6 08/13] riscv: Avoid TLB flush loops when affected by SiFive CIP-1200 Date: Tue, 26 Mar 2024 21:49:49 -0700 Message-ID: <20240327045035.368512-9-samuel.holland@sifive.com> X-Mailer: git-send-email 2.43.1 In-Reply-To: <20240327045035.368512-1-samuel.holland@sifive.com> References: <20240327045035.368512-1-samuel.holland@sifive.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Implementations affected by SiFive errata CIP-1200 have a bug which forces the kernel to always use the global variant of the sfence.vma instruction. When affected by this errata, do not attempt to flush a range of addresses; each iteration of the loop would actually flush the whole TLB instead. Instead, minimize the overall number of sfence.vma instructions. Signed-off-by: Samuel Holland Reviewed-by: Yunhui Cui --- Changes in v6: - Clarify the commit message for patch 8 based on ML discussion Changes in v4: - Only set tlb_flush_all_threshold when CONFIG_MMU=3Dy. Changes in v3: - New patch for v3 arch/riscv/errata/sifive/errata.c | 5 +++++ arch/riscv/include/asm/tlbflush.h | 2 ++ arch/riscv/mm/tlbflush.c | 2 +- 3 files changed, 8 insertions(+), 1 deletion(-) diff --git a/arch/riscv/errata/sifive/errata.c b/arch/riscv/errata/sifive/e= rrata.c index 3d9a32d791f7..716cfedad3a2 100644 --- a/arch/riscv/errata/sifive/errata.c +++ b/arch/riscv/errata/sifive/errata.c @@ -42,6 +42,11 @@ static bool errata_cip_1200_check_func(unsigned long ar= ch_id, unsigned long imp return false; if ((impid & 0xffffff) > 0x200630 || impid =3D=3D 0x1200626) return false; + +#ifdef CONFIG_MMU + tlb_flush_all_threshold =3D 0; +#endif + return true; } =20 diff --git a/arch/riscv/include/asm/tlbflush.h b/arch/riscv/include/asm/tlb= flush.h index 463b615d7728..8e329721375b 100644 --- a/arch/riscv/include/asm/tlbflush.h +++ b/arch/riscv/include/asm/tlbflush.h @@ -66,6 +66,8 @@ void arch_tlbbatch_add_pending(struct arch_tlbflush_unmap= _batch *batch, unsigned long uaddr); void arch_flush_tlb_batched_pending(struct mm_struct *mm); void arch_tlbbatch_flush(struct arch_tlbflush_unmap_batch *batch); + +extern unsigned long tlb_flush_all_threshold; #else /* CONFIG_MMU */ #define local_flush_tlb_all() do { } while (0) #endif /* CONFIG_MMU */ diff --git a/arch/riscv/mm/tlbflush.c b/arch/riscv/mm/tlbflush.c index ad7bdcfcc219..18af7b5053af 100644 --- a/arch/riscv/mm/tlbflush.c +++ b/arch/riscv/mm/tlbflush.c @@ -11,7 +11,7 @@ * Flush entire TLB if number of entries to be flushed is greater * than the threshold below. */ -static unsigned long tlb_flush_all_threshold __read_mostly =3D 64; +unsigned long tlb_flush_all_threshold __read_mostly =3D 64; =20 static void local_flush_tlb_range_threshold_asid(unsigned long start, unsigned long size, --=20 2.43.1 From nobody Tue Dec 16 20:00:28 2025 Received: from mail-oo1-f47.google.com (mail-oo1-f47.google.com [209.85.161.47]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D9D4063B3 for ; Wed, 27 Mar 2024 04:50:56 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.161.47 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711515058; cv=none; b=VA0waLZXC5mDWRt+WzusppGbsZUomczkqGpkArSzwGULAleFGeYEi1ACZOKBecmH0ABKizQfKZS3/c2Nn3DtuvPJipJ0zLkqw0WAm/NdaNcIyv98CivNME7BR3WJyNtRsynom5Gs6vArtqTZ1hY1dTg3c5JQzXc7y1mjEq30WsA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711515058; c=relaxed/simple; bh=h6ENFi8wwoz/1RsosVsupi85G/Ck90qRb4MFtlbEcLM=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=p0gZ2yHkJxcte5syc9rYvb+uJWH4Ho493j5IX9EiFja6hDpJWA55BdaFBw8oJdqinjR66SA1wiXNHVBTzzJo9BS/tq5RM83wgRAY+JYcoBpQfqdetHW0580rsGwDEG3TVJ4ijPrsnPWYxNb1IgrEzeW24ZIHYaE8znfwFzvNI0Y= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=sifive.com; spf=pass smtp.mailfrom=sifive.com; dkim=pass (2048-bit key) header.d=sifive.com header.i=@sifive.com header.b=CjqznVHU; arc=none smtp.client-ip=209.85.161.47 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=sifive.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=sifive.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=sifive.com header.i=@sifive.com header.b="CjqznVHU" Received: by mail-oo1-f47.google.com with SMTP id 006d021491bc7-5a58009fe88so511981eaf.0 for ; Tue, 26 Mar 2024 21:50:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1711515056; x=1712119856; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=2OOp6ZEyIwMU5OnG6xTEWcirae/pY2nPqfU9WqSousc=; b=CjqznVHUKLdFJUT+AwhDsONLhja7y0P0+BDwOtNIs1xPyrDfT23/l6GTKBsD1Dj2pr VibCA9hZWDThdIbnx/63M9eMII4NBaD0WDRBPXdNVtGZTp6xyxKFInLKPVasxW7+XxqO +ZaEyqWi5yc2IirGZt80XbcJRQvXovpEtmDVMwD0Pt/M17zxsOx1MAwOUwoTxeqniy6+ Rq46gw7oF5oVSFo8vmOrvWRbeeuYDerNSbPW6sZiFuTBoCkfh2NST6P6+DkDhjIiBp8P vw1YYcIhjIBGpAKTAe7bjgwBLrIyGsOA9o8fP3yghkcNLQYIZu9wIinrQxKAfqLxb5K8 e1Sw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711515056; x=1712119856; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=2OOp6ZEyIwMU5OnG6xTEWcirae/pY2nPqfU9WqSousc=; b=HAFTE2DxBqwe3YifbsLAwTPI4autudNkQWCcszVCir/QzRoZoDb6Z5coY82lz1qJ51 cv2wG+kUuxb7FHYtKSYf05v6Fv7UfU/9/b0RDvn5iltJ6bobqe2sYGlZCCyLB8midT4e TLxzJEtAv3LeHgZF5JEFsq+/3Iu0w+xKh0XXZBjlLSjv3PrdNkDO6QTxpGjZ9+2WtjCC dE4sClLV99hn4/hS5ofrTzadhCKzQxKc8dIBtCPQzTlywCntjH9GMEA68fo3STUN7ZJQ X3iSPXoLYyZomn7NN8nsN7re5s/lwIzgFoF/eQq3d/purDO/jeLi6pX+J08LqH1IIvHc mSkg== X-Gm-Message-State: AOJu0YzxvpbWL9H/U3E4Mq6UYsxd8X6XsKGbCzvhmhZScVlPppTIZr4y AO6bwXvl02kHqVTTNKGh6rJzJrLWKwjQkcAvwxvxA6CZNnBJohMVAjcGLdZWyp8i1Muk76IDtCT s X-Google-Smtp-Source: AGHT+IH2eQoyDSJi6zqBjW5QAslrBjHf9N1FFSb9F+mmhA++Fk/xLCVtnw/qNQbwZOYB8ND1BkvynQ== X-Received: by 2002:a05:6358:7584:b0:17c:1e6c:14c0 with SMTP id x4-20020a056358758400b0017c1e6c14c0mr311341rwf.16.1711515055737; Tue, 26 Mar 2024 21:50:55 -0700 (PDT) Received: from sw06.internal.sifive.com ([4.53.31.132]) by smtp.gmail.com with ESMTPSA id e31-20020a63501f000000b005e4666261besm8351500pgb.50.2024.03.26.21.50.54 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 26 Mar 2024 21:50:54 -0700 (PDT) From: Samuel Holland To: Palmer Dabbelt , linux-riscv@lists.infradead.org Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, Alexandre Ghiti , Jisheng Zhang , Yunhui Cui , Samuel Holland Subject: [PATCH v6 09/13] riscv: mm: Introduce cntx2asid/cntx2version helper macros Date: Tue, 26 Mar 2024 21:49:50 -0700 Message-ID: <20240327045035.368512-10-samuel.holland@sifive.com> X-Mailer: git-send-email 2.43.1 In-Reply-To: <20240327045035.368512-1-samuel.holland@sifive.com> References: <20240327045035.368512-1-samuel.holland@sifive.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" When using the ASID allocator, the MM context ID contains two values: the ASID in the lower bits, and the allocator version number in the remaining bits. Use macros to make this separation more obvious. Reviewed-by: Alexandre Ghiti Signed-off-by: Samuel Holland --- (no changes since v1) arch/riscv/include/asm/mmu.h | 3 +++ arch/riscv/mm/context.c | 12 ++++++------ arch/riscv/mm/tlbflush.c | 2 +- 3 files changed, 10 insertions(+), 7 deletions(-) diff --git a/arch/riscv/include/asm/mmu.h b/arch/riscv/include/asm/mmu.h index 355504b37f8e..a550fbf770be 100644 --- a/arch/riscv/include/asm/mmu.h +++ b/arch/riscv/include/asm/mmu.h @@ -26,6 +26,9 @@ typedef struct { #endif } mm_context_t; =20 +#define cntx2asid(cntx) ((cntx) & asid_mask) +#define cntx2version(cntx) ((cntx) & ~asid_mask) + void __init create_pgd_mapping(pgd_t *pgdp, uintptr_t va, phys_addr_t pa, phys_addr_t sz, pgprot_t prot); #endif /* __ASSEMBLY__ */ diff --git a/arch/riscv/mm/context.c b/arch/riscv/mm/context.c index ba8eb3944687..b562b3c44487 100644 --- a/arch/riscv/mm/context.c +++ b/arch/riscv/mm/context.c @@ -81,7 +81,7 @@ static void __flush_context(void) if (cntx =3D=3D 0) cntx =3D per_cpu(reserved_context, i); =20 - __set_bit(cntx & asid_mask, context_asid_map); + __set_bit(cntx2asid(cntx), context_asid_map); per_cpu(reserved_context, i) =3D cntx; } =20 @@ -102,7 +102,7 @@ static unsigned long __new_context(struct mm_struct *mm) lockdep_assert_held(&context_lock); =20 if (cntx !=3D 0) { - unsigned long newcntx =3D ver | (cntx & asid_mask); + unsigned long newcntx =3D ver | cntx2asid(cntx); =20 /* * If our current CONTEXT was active during a rollover, we @@ -115,7 +115,7 @@ static unsigned long __new_context(struct mm_struct *mm) * We had a valid CONTEXT in a previous life, so try to * re-use it if possible. */ - if (!__test_and_set_bit(cntx & asid_mask, context_asid_map)) + if (!__test_and_set_bit(cntx2asid(cntx), context_asid_map)) return newcntx; } =20 @@ -168,7 +168,7 @@ static void set_mm_asid(struct mm_struct *mm, unsigned = int cpu) */ old_active_cntx =3D atomic_long_read(&per_cpu(active_context, cpu)); if (old_active_cntx && - ((cntx & ~asid_mask) =3D=3D atomic_long_read(¤t_version)) && + (cntx2version(cntx) =3D=3D atomic_long_read(¤t_version)) && atomic_long_cmpxchg_relaxed(&per_cpu(active_context, cpu), old_active_cntx, cntx)) goto switch_mm_fast; @@ -177,7 +177,7 @@ static void set_mm_asid(struct mm_struct *mm, unsigned = int cpu) =20 /* Check that our ASID belongs to the current_version. */ cntx =3D atomic_long_read(&mm->context.id); - if ((cntx & ~asid_mask) !=3D atomic_long_read(¤t_version)) { + if (cntx2version(cntx) !=3D atomic_long_read(¤t_version)) { cntx =3D __new_context(mm); atomic_long_set(&mm->context.id, cntx); } @@ -191,7 +191,7 @@ static void set_mm_asid(struct mm_struct *mm, unsigned = int cpu) =20 switch_mm_fast: csr_write(CSR_SATP, virt_to_pfn(mm->pgd) | - ((cntx & asid_mask) << SATP_ASID_SHIFT) | + (cntx2asid(cntx) << SATP_ASID_SHIFT) | satp_mode); =20 if (need_flush_tlb) diff --git a/arch/riscv/mm/tlbflush.c b/arch/riscv/mm/tlbflush.c index 18af7b5053af..35266dd9a9a2 100644 --- a/arch/riscv/mm/tlbflush.c +++ b/arch/riscv/mm/tlbflush.c @@ -110,7 +110,7 @@ static void __flush_tlb_range(struct cpumask *cmask, un= signed long asid, static inline unsigned long get_mm_asid(struct mm_struct *mm) { return static_branch_unlikely(&use_asid_allocator) ? - atomic_long_read(&mm->context.id) & asid_mask : FLUSH_TLB_NO_ASID; + cntx2asid(atomic_long_read(&mm->context.id)) : FLUSH_TLB_NO_ASID; } =20 void flush_tlb_mm(struct mm_struct *mm) --=20 2.43.1 From nobody Tue Dec 16 20:00:28 2025 Received: from mail-oo1-f45.google.com (mail-oo1-f45.google.com [209.85.161.45]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4DB202868D for ; Wed, 27 Mar 2024 04:50:58 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.161.45 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711515059; cv=none; b=QUnaRwUZ56XeQp63hxZM2L4EuEsbj+8KAYoTxtZU/ajmZZngrpyn9dCPeF2Og1nmTJ80DCoKyzIea6fIzEwrh85Zku4ClylfVpvXwDpoQ0K7zng1mQlnReRrI6q34TItSWevymhZhZ2UN3mgOZ7KgToZc7kAY9+Hy/m4cm+CLe4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711515059; c=relaxed/simple; bh=ftmlZUm13WxGdRUTJnLym9NaWym8BpVy8g2b1tMP78U=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=I1gg9iqvLZyRGRpwMa7pDCxaT66rtl2ta9bezM+K93TE+AaYfm6eu6X1MuZlxG06b47jgsVi18LUqnYR4BO5jB3BComeWltCo6yj1/Ka4tE2HbMLZ7CgB44Xg4cR+6dDF+1AKKQ12mvgelAwNzyLC4nF5f9X+SlSII7ziLwZ67I= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=sifive.com; spf=pass smtp.mailfrom=sifive.com; dkim=pass (2048-bit key) header.d=sifive.com header.i=@sifive.com header.b=GqPplVQL; arc=none smtp.client-ip=209.85.161.45 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=sifive.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=sifive.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=sifive.com header.i=@sifive.com header.b="GqPplVQL" Received: by mail-oo1-f45.google.com with SMTP id 006d021491bc7-5a56710cdccso790123eaf.3 for ; Tue, 26 Mar 2024 21:50:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1711515057; x=1712119857; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=A+Ju3+5NsQ7zqLi78mVdBh82QtKzTZZzrfFOA+rFDFs=; b=GqPplVQLAFBRhF6X3RmN4Cn19Ru6zRkrkkJb+Vr0EchPxzvuYVzG71t8eLMqkN7ykw NL01Ccv4tYgzle7oZ5JEFCu4kHa2OOHhKmLUfz1EnLftqFqN0eVSAC4LfujuYCbteDux GtV0zaX3rRsSm0VIJ0VkDPfdQjmWCNNB/GRSzuTvbWUISNKXsJ4HLzhZGSetqvcAxzNB BiJF0LbuHkyEpjarGyaxB5z6SZxVr3iDck8jo4vL64j8DbSmgVmQAEIdTbo5DdksYIRq GMnx6oWQYQoHWbrZqCA4Sg5K2Ii8+gqnd9hdFUYDwhQ12LfTGWoT8xxlv2hRHbtK5UAt Wtyw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711515057; x=1712119857; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=A+Ju3+5NsQ7zqLi78mVdBh82QtKzTZZzrfFOA+rFDFs=; b=xGKW1LxB52GgKsX7/imWYm4k9faHXpoQ+1D6h0chvYWv+SoIXXO0bh0SZy0+dzlLr3 7YmeSfTYS8JS3SLb7Fwol4eyeR2MXp4zZwRav813vUEdzysdV7asJUrG4USjKfM3w17Q XYd2S+gc0UlwA3arJc4GXsWMPqqM9beAwxRkudgDZ8iaGLdLcFWNoTbWLmwUG7PQxG5a lDWFVPshT6LwftTcPm3bVihAmAxfEKSlBIfxphfhaDmBmtaBe2Os6Fhiju3FvUQb90GQ JriuvrQ/4CRNxTI0fXvBMiOy0SvRpGx3nSCabdvR0H8RNoi7oN3r0NYka8l2jDg6XShu Wfcw== X-Gm-Message-State: AOJu0Yxr6+x2pD14AhnjFr2ncpd7yPaMq6tLXpPYj070Zr8jeW3Q2oig XU/BSF7TrVwqWmYYq1HWLpICCjarRVs8D45RMPn822AQIHF3VGAqxOuTWV8DPPI= X-Google-Smtp-Source: AGHT+IE4clbeICIX6LaCSNcjSNKRNzkyOLoTskHKcKR+t8ek8fOhgjRruwacqNmtd1HR0FedGGJk+g== X-Received: by 2002:a05:6358:6397:b0:17e:c8d9:328 with SMTP id k23-20020a056358639700b0017ec8d90328mr5244771rwh.11.1711515057489; Tue, 26 Mar 2024 21:50:57 -0700 (PDT) Received: from sw06.internal.sifive.com ([4.53.31.132]) by smtp.gmail.com with ESMTPSA id e31-20020a63501f000000b005e4666261besm8351500pgb.50.2024.03.26.21.50.56 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 26 Mar 2024 21:50:56 -0700 (PDT) From: Samuel Holland To: Palmer Dabbelt , linux-riscv@lists.infradead.org Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, Alexandre Ghiti , Jisheng Zhang , Yunhui Cui , Samuel Holland Subject: [PATCH v6 10/13] riscv: mm: Use a fixed layout for the MM context ID Date: Tue, 26 Mar 2024 21:49:51 -0700 Message-ID: <20240327045035.368512-11-samuel.holland@sifive.com> X-Mailer: git-send-email 2.43.1 In-Reply-To: <20240327045035.368512-1-samuel.holland@sifive.com> References: <20240327045035.368512-1-samuel.holland@sifive.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Currently, the size of the ASID field in the MM context ID dynamically depends on the number of hardware-supported ASID bits. This requires reading a global variable to extract either field from the context ID. Instead, allocate the maximum possible number of bits to the ASID field, so the layout of the context ID is known at compile-time. Reviewed-by: Alexandre Ghiti Signed-off-by: Samuel Holland --- (no changes since v1) arch/riscv/include/asm/mmu.h | 4 ++-- arch/riscv/include/asm/tlbflush.h | 2 -- arch/riscv/mm/context.c | 6 ++---- 3 files changed, 4 insertions(+), 8 deletions(-) diff --git a/arch/riscv/include/asm/mmu.h b/arch/riscv/include/asm/mmu.h index a550fbf770be..dc0273f7905f 100644 --- a/arch/riscv/include/asm/mmu.h +++ b/arch/riscv/include/asm/mmu.h @@ -26,8 +26,8 @@ typedef struct { #endif } mm_context_t; =20 -#define cntx2asid(cntx) ((cntx) & asid_mask) -#define cntx2version(cntx) ((cntx) & ~asid_mask) +#define cntx2asid(cntx) ((cntx) & SATP_ASID_MASK) +#define cntx2version(cntx) ((cntx) & ~SATP_ASID_MASK) =20 void __init create_pgd_mapping(pgd_t *pgdp, uintptr_t va, phys_addr_t pa, phys_addr_t sz, pgprot_t prot); diff --git a/arch/riscv/include/asm/tlbflush.h b/arch/riscv/include/asm/tlb= flush.h index 8e329721375b..72e559934952 100644 --- a/arch/riscv/include/asm/tlbflush.h +++ b/arch/riscv/include/asm/tlbflush.h @@ -15,8 +15,6 @@ #define FLUSH_TLB_NO_ASID ((unsigned long)-1) =20 #ifdef CONFIG_MMU -extern unsigned long asid_mask; - static inline void local_flush_tlb_all(void) { __asm__ __volatile__ ("sfence.vma" : : : "memory"); diff --git a/arch/riscv/mm/context.c b/arch/riscv/mm/context.c index b562b3c44487..5315af06cd4d 100644 --- a/arch/riscv/mm/context.c +++ b/arch/riscv/mm/context.c @@ -22,7 +22,6 @@ DEFINE_STATIC_KEY_FALSE(use_asid_allocator); =20 static unsigned long asid_bits; static unsigned long num_asids; -unsigned long asid_mask; =20 static atomic_long_t current_version; =20 @@ -128,7 +127,7 @@ static unsigned long __new_context(struct mm_struct *mm) goto set_asid; =20 /* We're out of ASIDs, so increment current_version */ - ver =3D atomic_long_add_return_relaxed(num_asids, ¤t_version); + ver =3D atomic_long_add_return_relaxed(BIT(SATP_ASID_BITS), ¤t_vers= ion); =20 /* Flush everything */ __flush_context(); @@ -247,7 +246,6 @@ static int __init asids_init(void) /* Pre-compute ASID details */ if (asid_bits) { num_asids =3D 1 << asid_bits; - asid_mask =3D num_asids - 1; } =20 /* @@ -255,7 +253,7 @@ static int __init asids_init(void) * at-least twice more than CPUs */ if (num_asids > (2 * num_possible_cpus())) { - atomic_long_set(¤t_version, num_asids); + atomic_long_set(¤t_version, BIT(SATP_ASID_BITS)); =20 context_asid_map =3D bitmap_zalloc(num_asids, GFP_KERNEL); if (!context_asid_map) --=20 2.43.1 From nobody Tue Dec 16 20:00:28 2025 Received: from mail-oo1-f50.google.com (mail-oo1-f50.google.com [209.85.161.50]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 143A88836 for ; Wed, 27 Mar 2024 04:50:59 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.161.50 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711515061; cv=none; b=ZJpbDji8sK4U3HbBYg2tr3I1SmvzOsrsUnLmCPg8fC2Lsqfb8mKQumBc+xglUQK+RszlkuKX+XGgNltFdpUjShEmfzelabaeDNQvSiqkrg73w3o1KNNdl4wMfGym3Q7axxKgg4rZ0alJbenX9YazfAQNIUzp9DPNTqnzLLUpUrw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711515061; c=relaxed/simple; bh=G1BXM2z6rQBc5K4wZw4N6I/e+joRY7yH/Z0Jdaxm+no=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=QlHl7lyQJdNKb4vTzuk0qNylAC8tUaArTX4+btn/+feoOUHBQeemTuMxw0i+twEMmM+qLvypKWlYaIWB02iZE9VMJPtccIWS/vUGtpHyvc3lo1B7+dAGeTBizS4uR2j7oz621cSvnlFeM2bRVIvLb6puxv8KXWYtRGRQ4azQXe0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=sifive.com; spf=pass smtp.mailfrom=sifive.com; dkim=pass (2048-bit key) header.d=sifive.com header.i=@sifive.com header.b=GFoncGkd; arc=none smtp.client-ip=209.85.161.50 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=sifive.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=sifive.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=sifive.com header.i=@sifive.com header.b="GFoncGkd" Received: by mail-oo1-f50.google.com with SMTP id 006d021491bc7-5a1b6800ba8so2230312eaf.0 for ; Tue, 26 Mar 2024 21:50:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1711515059; x=1712119859; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=4wFLc5gcnBs285NRnicKVmmQVV53EnA/2rANcUM2uCA=; b=GFoncGkd4CnXjWpR6dZgceUdNTns+EkWUxgW1bzfHBUz6IYFilVgPq8ZuRAsiKohEN QKK9q8ehr/sXYPBNqXHwAsEm9BEuYFFBLxrN1Fe5HOM9OeKdt3MmGY0R0JmM93f6nxfx 0AE0eQi0+UrnJa+TRSxIpKXNlrU4XSTw9n1lYCZkw2nDrP2ZDWAU+4UTwYn7ybN1PWKF JS98FZhB+JkRiKeUc2fGOK2VtNm+x896G3z5KQmYGUjFKzFJvW3w9/RMOxwOk53mneoS 2sG8yhZDtds9lhwB0Kb8yBB3XLqurR+N2y3wVtUQnuV4In3RYGeFu+7X/5evOtOB8r/a 5mTA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711515059; x=1712119859; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=4wFLc5gcnBs285NRnicKVmmQVV53EnA/2rANcUM2uCA=; b=qA+e7v4NelTjykLAvg2w6HzscsH0D+VjpfFMUdqlL7V37G2lu01JZU0OFKDtIx5uS8 fMjfr41zXqGsse+Fylso+7P17CiPtEYVxAMZ6bC1qZhZfemrW0lar2xWt8M9/t9T7zDp JF8+tVIyRlv692wZhjMtuj+GF61uhWkb0Qdu9jLHsHspOhx7P7wUeJrsOoY4Pktcbfl6 7DCD9VHgAc6h7NGlnc0T9WrWXdBfGt1Rsog/dGZqI5Mrc0zm1QzndycEKy9NPnehkxEY EXiyAiG3AJD1YwoGsyv+NuvcTaEjK2f2waYMWcXwQw0ujFiPAt8f376W0/1oes4aM2LA da1Q== X-Gm-Message-State: AOJu0YybpLHMQ3xR/XbLdojXz13dQrX/VuQ724eooraRMhxXUT2sWQHP OK+feMLAIYkopULyauVPGJ0qGBUOoHjU7/gsNqhGDGa1pftAJOsixfWE3RnpzDc= X-Google-Smtp-Source: AGHT+IFmC6K+A8yipt+L5BT3VYhAZMjFopo36khv4UzVVH+BHr/EbytiKBlOS5fSbYkXOITYtqGpvA== X-Received: by 2002:a05:6358:320c:b0:17e:bb52:927f with SMTP id a12-20020a056358320c00b0017ebb52927fmr1922247rwe.1.1711515059146; Tue, 26 Mar 2024 21:50:59 -0700 (PDT) Received: from sw06.internal.sifive.com ([4.53.31.132]) by smtp.gmail.com with ESMTPSA id e31-20020a63501f000000b005e4666261besm8351500pgb.50.2024.03.26.21.50.57 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 26 Mar 2024 21:50:58 -0700 (PDT) From: Samuel Holland To: Palmer Dabbelt , linux-riscv@lists.infradead.org Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, Alexandre Ghiti , Jisheng Zhang , Yunhui Cui , Samuel Holland Subject: [PATCH v6 11/13] riscv: mm: Make asid_bits a local variable Date: Tue, 26 Mar 2024 21:49:52 -0700 Message-ID: <20240327045035.368512-12-samuel.holland@sifive.com> X-Mailer: git-send-email 2.43.1 In-Reply-To: <20240327045035.368512-1-samuel.holland@sifive.com> References: <20240327045035.368512-1-samuel.holland@sifive.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" This variable is only used inside asids_init(). Reviewed-by: Alexandre Ghiti Signed-off-by: Samuel Holland --- (no changes since v1) arch/riscv/mm/context.c | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/arch/riscv/mm/context.c b/arch/riscv/mm/context.c index 5315af06cd4d..0bf6d0070a14 100644 --- a/arch/riscv/mm/context.c +++ b/arch/riscv/mm/context.c @@ -20,7 +20,6 @@ =20 DEFINE_STATIC_KEY_FALSE(use_asid_allocator); =20 -static unsigned long asid_bits; static unsigned long num_asids; =20 static atomic_long_t current_version; @@ -226,7 +225,7 @@ static inline void set_mm(struct mm_struct *prev, =20 static int __init asids_init(void) { - unsigned long old; + unsigned long asid_bits, old; =20 /* Figure-out number of ASID bits in HW */ old =3D csr_read(CSR_SATP); --=20 2.43.1 From nobody Tue Dec 16 20:00:28 2025 Received: from mail-pf1-f180.google.com (mail-pf1-f180.google.com [209.85.210.180]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1636B2C1A8 for ; Wed, 27 Mar 2024 04:51:01 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.180 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711515063; cv=none; b=Bomg9+YJaYzuSD0fZ/ac0G5KgFTUYhq80DmzlIH2Ak7R8caXjcgJ/YxYKS/YHlEJ5H0XSne5H4A6ZOHtdNbtgz68ZKs3OCRiAd9eKv/yw7td7VcremJhdstJeD3+fd33/cKnRgmk7DaI7JKFBl4X6wys6O7XhhfsdSCDFjHGPN8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711515063; c=relaxed/simple; bh=MRE4XT0P9lLUUCp1Ujsa6SQk4prpBDb9nMVRKvQ6tEU=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=rn0iBRL0HDdmal9SYwn+n9cUgtwUGY4DJngnmlICmK1ICS0l2utNXHUIhJhmY7kOWIxXmBxM4O2YE79ChRFCo2AlVUFT9FT5UXUQziPW/ZKpRegmoCJxrRzxycFy76oKdw8qDX9m92p6hiHLl/2zgbHtpY3rV/QpSkqhwG/5nxw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=sifive.com; spf=pass smtp.mailfrom=sifive.com; dkim=pass (2048-bit key) header.d=sifive.com header.i=@sifive.com header.b=jM+qYYj5; arc=none smtp.client-ip=209.85.210.180 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=sifive.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=sifive.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=sifive.com header.i=@sifive.com header.b="jM+qYYj5" Received: by mail-pf1-f180.google.com with SMTP id d2e1a72fcca58-6ea9a605ca7so381507b3a.0 for ; Tue, 26 Mar 2024 21:51:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1711515061; x=1712119861; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=hwpkv2OuO7YHUplKH98oFVfOa+CFbclxXNu0ysmduyU=; b=jM+qYYj5QTJ+dGUdyDeBKHcRLJZk81elChhqV+twofbmJhbnMwqqn/XqUpoaRJTDoL mXcSyDUZ9e+L9AY0zJK+HiOqTQjrPZelmz1av7t6yvXS/3Yzud7PKBhTBzzxSdBr8nlH jqyFQXPYbPqycIGgCyv/qolFRyfsUbOq2iJFs/t96dT2uHoMBNEF1840ayYmGL8EWTQR PuCevHEZLgb1FjLdDFwfqVGk/o4NrwOlIY56z/018WgVWuQss3+CKVWLNBP9fvCu9n7G bmOX3usTWNd3PqbM/JC3ACpqbetuooUGhAupq8LIAiQTBS2Cho7monHkEoD/nvohZFoV mWbA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711515061; x=1712119861; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=hwpkv2OuO7YHUplKH98oFVfOa+CFbclxXNu0ysmduyU=; b=Iobp5+X6iEofl1BR0kqhhWoE3bifSGoPN1cdQ2IGvoSRCTKn2n2YF+xx7mW/Yzbf6Q SX+2gYt5y/z07gzrrlm3YK113GI8BX5IP2q+BNczgTV/TuL1huEcaUAKNubbFLYxpeSE VTc1LNEhuncKLH2pfAaEkZFWw994Jk9bi5LbBg3zym6gP/S+sTjCoieOmKGOweOZKsXi fbPHUXFNAIo/Cxh9uLr/wGxvxm8+bGlMdd23vfzfwuUXBm8ozhAnOEq2LU0j+a1wqvp2 O/8mYLWSYhbGxyjswrmZIVRMR27c/CBMeQ5ZC+rnhhxJnUYm2jH6BtFE0tszQ0LsumzO e4pA== X-Gm-Message-State: AOJu0Ywsz1lmO+5mpcORnKNFGG6h96VBas+ymVOxEAgEpTGGXsczyZdm elT5d3lBesWUgNbiHnxtDf+4aQH1fwWA9YnZzK+/TwsqwGU0g1Ov9hj8hKljcoI= X-Google-Smtp-Source: AGHT+IFIatp35KNpgDUDmJycAxTP3nvYaAVXQURqJcQtx3/AD7qiy3AeB8dChiHf0CcNcubYBhb1CQ== X-Received: by 2002:a05:6a20:7345:b0:1a1:67c0:c751 with SMTP id v5-20020a056a20734500b001a167c0c751mr5157753pzc.17.1711515061281; Tue, 26 Mar 2024 21:51:01 -0700 (PDT) Received: from sw06.internal.sifive.com ([4.53.31.132]) by smtp.gmail.com with ESMTPSA id e31-20020a63501f000000b005e4666261besm8351500pgb.50.2024.03.26.21.50.59 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 26 Mar 2024 21:51:00 -0700 (PDT) From: Samuel Holland To: Palmer Dabbelt , linux-riscv@lists.infradead.org Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, Alexandre Ghiti , Jisheng Zhang , Yunhui Cui , Samuel Holland Subject: [PATCH v6 12/13] riscv: mm: Preserve global TLB entries when switching contexts Date: Tue, 26 Mar 2024 21:49:53 -0700 Message-ID: <20240327045035.368512-13-samuel.holland@sifive.com> X-Mailer: git-send-email 2.43.1 In-Reply-To: <20240327045035.368512-1-samuel.holland@sifive.com> References: <20240327045035.368512-1-samuel.holland@sifive.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" If the CPU does not support multiple ASIDs, all MM contexts use ASID 0. In this case, it is still beneficial to flush the TLB by ASID, as the single-ASID variant of the sfence.vma instruction preserves TLB entries for global (kernel) pages. This optimization is recommended by the RISC-V privileged specification: If the implementation does not provide ASIDs, or software chooses to always use ASID 0, then after every satp write, software should execute SFENCE.VMA with rs1=3Dx0. In the common case that no global translations have been modified, rs2 should be set to a register other than x0 but which contains the value zero, so that global translations are not flushed. It is not possible to apply this optimization when using the ASID allocator, because that code must flush the TLB for all ASIDs at once when incrementing the version number. Reviewed-by: Alexandre Ghiti Signed-off-by: Samuel Holland --- (no changes since v1) arch/riscv/mm/context.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/arch/riscv/mm/context.c b/arch/riscv/mm/context.c index 0bf6d0070a14..60cb0b82240e 100644 --- a/arch/riscv/mm/context.c +++ b/arch/riscv/mm/context.c @@ -200,7 +200,7 @@ static void set_mm_noasid(struct mm_struct *mm) { /* Switch the page table and blindly nuke entire local TLB */ csr_write(CSR_SATP, virt_to_pfn(mm->pgd) | satp_mode); - local_flush_tlb_all(); + local_flush_tlb_all_asid(0); } =20 static inline void set_mm(struct mm_struct *prev, --=20 2.43.1 From nobody Tue Dec 16 20:00:28 2025 Received: from mail-oo1-f47.google.com (mail-oo1-f47.google.com [209.85.161.47]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8FEF42C6B6 for ; Wed, 27 Mar 2024 04:51:04 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.161.47 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711515066; cv=none; b=JEIUrz/VbfVhQiePiXKzoa0bPn7k1dAg27jUQbZMoBk3U0W/55/LPRFGb7ew+eRpxwAgADNkhOmwYKq0RL2coN55+oK6XiMuIqljYlLdBL2APE+HCTbvqtSh/TxW5M9ijP8Fl2Mam4yUSkfuouIJxUftL1dOqC3UR54/AK5hCK4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711515066; c=relaxed/simple; bh=D/kvOoscoO9GAvES/r3sKJAilm9YD07kx5uQNIOnQFY=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=WvRROqVdim0BNAGA8LmqV8MlgzG+0SLRygNDEjS1t1zUxjAdZXacI/veU4vnP/aQrnF0ukNCYuddWyMZ5mqrPgj8NHeaQ1SFRumIpKGwdXccwtA8gWZYGF66kIzvgQJZ8n14wSvjA5JnnTLGct3igbSCz73jM9mljX77bcY9Zv4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=sifive.com; spf=pass smtp.mailfrom=sifive.com; dkim=pass (2048-bit key) header.d=sifive.com header.i=@sifive.com header.b=ceoRPsJR; arc=none smtp.client-ip=209.85.161.47 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=sifive.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=sifive.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=sifive.com header.i=@sifive.com header.b="ceoRPsJR" Received: by mail-oo1-f47.google.com with SMTP id 006d021491bc7-5a5362ceb7bso1591058eaf.1 for ; Tue, 26 Mar 2024 21:51:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1711515063; x=1712119863; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=DStjBcAn1SHrL/HpDMugRz0dC8r4hdt1hrlyujsEs2s=; b=ceoRPsJRxo9+qc2acSFLkXlaPq0aHu0TpsMyFUXrAst8Ila4TWwUAOz59yOedpyG4U +20etI4OOwpZbOXlp8yOXtW2pQoRe2Eb9/P/0DbuAkAm7eeWbJ8CMPOfhHgfuoAmkaR9 MIILW1QN/PFdcrjsR6C1/vfM+Oj8UJXlC05WlseTV5EKudyD5A6zOUAh4nzvYyBD3XCe cOwmaqQUC0vI2+m3FSFUX0/nyacCIlDaVF15wNhx4ITpA0aNWOaTnmwtsoWLX6oDCI5t 6GMGIIhKvzHXPzXqU7960N4m05YHLVMRrZHUoRw7SEH0/ix6YUfM145pNZvdm74hlXC0 cOVg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711515063; x=1712119863; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=DStjBcAn1SHrL/HpDMugRz0dC8r4hdt1hrlyujsEs2s=; b=GDHhhaS2UKZO5COGCMM+z5/h+mM/yKXtn7hMv+XbF4hk1H3MmM5VF67IDFHydsJTUx 3lbfUziRAosgUAqz4uvWVJfOmlkYyTng01xwrQxngyE9BliNkqO+iIJyytvgZOHmKVEP CKZIIlYoP42Zqwq6XVwCRdW3dGGW8Go3o9vktxgFzG5IP8CGjBy0MamgLoe7TzTcY1H2 wczenAAo7MawvYVAZSZghU6FQHuYPdCz31bZhO2uBl0QHsyYKLX52neKgH+BWkOTQ8h3 50kHltHI7uqxxyKAFU7529rG1WJ8NHCVUBrS7R89ujM+/JnRNAACdenTMF2+oRONyt6p 7Lvg== X-Gm-Message-State: AOJu0YyKSmdsVktDBoSWuuJoB9Nk4Um4t/l+AZuaUu9MGjVb7aR7/6bd c9d0UrxsXBJkdTI4wbRq9dDcMf4UUe4O5F+0ju4ji1ohGNuZaGbJW0aEYu7J9hE= X-Google-Smtp-Source: AGHT+IElUFan01DMTcKq7mpLuQEXv/M6ZjDhhp1pW7v7cPBwp502ELftTLKlJV+xiDuinHxlNWw0zA== X-Received: by 2002:a05:6358:520d:b0:17f:1d34:6253 with SMTP id b13-20020a056358520d00b0017f1d346253mr268628rwa.15.1711515063614; Tue, 26 Mar 2024 21:51:03 -0700 (PDT) Received: from sw06.internal.sifive.com ([4.53.31.132]) by smtp.gmail.com with ESMTPSA id e31-20020a63501f000000b005e4666261besm8351500pgb.50.2024.03.26.21.51.01 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 26 Mar 2024 21:51:02 -0700 (PDT) From: Samuel Holland To: Palmer Dabbelt , linux-riscv@lists.infradead.org Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, Alexandre Ghiti , Jisheng Zhang , Yunhui Cui , Samuel Holland Subject: [PATCH v6 13/13] riscv: mm: Always use an ASID to flush mm contexts Date: Tue, 26 Mar 2024 21:49:54 -0700 Message-ID: <20240327045035.368512-14-samuel.holland@sifive.com> X-Mailer: git-send-email 2.43.1 In-Reply-To: <20240327045035.368512-1-samuel.holland@sifive.com> References: <20240327045035.368512-1-samuel.holland@sifive.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Even if multiple ASIDs are not supported, using the single-ASID variant of the sfence.vma instruction preserves TLB entries for global (kernel) pages. So it is always more efficient to use the single-ASID code path. Reviewed-by: Alexandre Ghiti Signed-off-by: Samuel Holland --- (no changes since v5) Changes in v5: - Leave use_asid_allocator declared in asm/mmu_context.h Changes in v4: - There is now only one copy of __flush_tlb_range() Changes in v2: - Update both copies of __flush_tlb_range() arch/riscv/mm/tlbflush.c | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/arch/riscv/mm/tlbflush.c b/arch/riscv/mm/tlbflush.c index 35266dd9a9a2..44e7ed4e194f 100644 --- a/arch/riscv/mm/tlbflush.c +++ b/arch/riscv/mm/tlbflush.c @@ -109,8 +109,7 @@ static void __flush_tlb_range(struct cpumask *cmask, un= signed long asid, =20 static inline unsigned long get_mm_asid(struct mm_struct *mm) { - return static_branch_unlikely(&use_asid_allocator) ? - cntx2asid(atomic_long_read(&mm->context.id)) : FLUSH_TLB_NO_ASID; + return cntx2asid(atomic_long_read(&mm->context.id)); } =20 void flush_tlb_mm(struct mm_struct *mm) --=20 2.43.1