From nobody Tue Dec 16 20:00:44 2025 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 51736C001B0 for ; Sat, 28 Oct 2023 23:14:01 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229880AbjJ1XNt (ORCPT ); Sat, 28 Oct 2023 19:13:49 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42598 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229805AbjJ1XNp (ORCPT ); Sat, 28 Oct 2023 19:13:45 -0400 Received: from mail-pl1-x62a.google.com (mail-pl1-x62a.google.com [IPv6:2607:f8b0:4864:20::62a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 95E0DD9 for ; Sat, 28 Oct 2023 16:13:43 -0700 (PDT) Received: by mail-pl1-x62a.google.com with SMTP id d9443c01a7336-1cc3bb32b5dso3315715ad.3 for ; Sat, 28 Oct 2023 16:13:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1698534823; x=1699139623; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=2EsAJ3DvynnbB6jUfsinRvKaNQwCusxxXujfrC8RtXI=; b=Kz6ZFZekc1mo2dxoD2XHRtugETvR+zuoZtRIdZksTNRbflSk8B7ZN2wIgQZZGlwBbj xF8dG1IkQNxLs9sOOl1X1iXWslQ0Cy/MLeGmALbn2HMB9VqVAmwgVnZ7ryhetm1LnqDs 3wjjYfw1kMPlVLl1f7SbHFnW0WOQZhPCIUOkuGjfCUM5udP/E5VJw0XVOTz/gYX2uUsF I4/eFWiVRH50KqxjRqNcgKC18uMidcJalcHGNJvcATE0SIfE8uxuZzUXsOp/Y3OWkyDn FEufHtOLRn80DOUDi57m+q4E+gs8HivYW6RmepUGHsQFvAhqU1+3Ujs1mgbyHtzmOfLW YXkQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1698534823; x=1699139623; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=2EsAJ3DvynnbB6jUfsinRvKaNQwCusxxXujfrC8RtXI=; b=VRNIO5besVdn7IvAsZ/NsrI5TJSOS7hQ2FcMGY2C4v9dBrbRGMTMcRRw7YDMmk7AUS 8S0zeTgZskERzg22EFso9kGlHlKhMP+ucTEFaeHI/xiomHOgZ4HHWrATkSBIKR6VCkKO ETk264D69oq82oimqQnCLCFGUAn8E8WZulxgXT8qoNhtneqSflFqsS0VdMjk0Vta6m6y 20+1BFAMbCogkaioOyQefB8JGqRMg/c2NcZSGez1mxg0QJCoBVRJJ2xTekaJDx45Q7PA Stcq01JfY7WQ1sD02Kwk1CNOWFvjbceGSa4fqahqN9qTo0SpT8Sy18pBOmcqfA38v1N/ IhZA== X-Gm-Message-State: AOJu0YwPqX7Apedp2BFjxWA8ZPyB8b2GYt3YJhX1gtO1twCBTnGU24b2 fzMSmMkXi5Buibj/FQsXjaqxOw== X-Google-Smtp-Source: AGHT+IFw/0gO2jty39BfEl53e0TAm8jThBxaHB3Pi6HPPc0H5aL70GOaasoPy4Ex/mKBNftI1RCeRA== X-Received: by 2002:a17:902:cf41:b0:1ca:8252:a91 with SMTP id e1-20020a170902cf4100b001ca82520a91mr7454451plg.41.1698534823149; Sat, 28 Oct 2023 16:13:43 -0700 (PDT) Received: from sw06.internal.sifive.com ([4.53.31.132]) by smtp.gmail.com with ESMTPSA id u17-20020a17090341d100b001b8622c1ad2sm3679345ple.130.2023.10.28.16.13.42 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 28 Oct 2023 16:13:42 -0700 (PDT) From: Samuel Holland To: Palmer Dabbelt , Alexandre Ghiti , linux-riscv@lists.infradead.org Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, Andrew Jones , Lad Prabhakar , Samuel Holland Subject: [PATCH v2 01/11] riscv: Improve tlb_flush() Date: Sat, 28 Oct 2023 16:11:59 -0700 Message-ID: <20231028231339.3116618-2-samuel.holland@sifive.com> X-Mailer: git-send-email 2.42.0 In-Reply-To: <20231028231339.3116618-1-samuel.holland@sifive.com> References: <20231028231339.3116618-1-samuel.holland@sifive.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Alexandre Ghiti For now, tlb_flush() simply calls flush_tlb_mm() which results in a flush of the whole TLB. So let's use mmu_gather fields to provide a more fine-grained flush of the TLB. Signed-off-by: Alexandre Ghiti Reviewed-by: Andrew Jones Tested-by: Lad Prabhakar # On RZ/= Five SMARC Signed-off-by: Samuel Holland --- Changes in v2: - Rebase on Alexandre's "riscv: tlb flush improvements" series v5 arch/riscv/include/asm/tlb.h | 8 +++++++- arch/riscv/include/asm/tlbflush.h | 3 +++ arch/riscv/mm/tlbflush.c | 7 +++++++ 3 files changed, 17 insertions(+), 1 deletion(-) diff --git a/arch/riscv/include/asm/tlb.h b/arch/riscv/include/asm/tlb.h index 120bcf2ed8a8..1eb5682b2af6 100644 --- a/arch/riscv/include/asm/tlb.h +++ b/arch/riscv/include/asm/tlb.h @@ -15,7 +15,13 @@ static void tlb_flush(struct mmu_gather *tlb); =20 static inline void tlb_flush(struct mmu_gather *tlb) { - flush_tlb_mm(tlb->mm); +#ifdef CONFIG_MMU + if (tlb->fullmm || tlb->need_flush_all) + flush_tlb_mm(tlb->mm); + else + flush_tlb_mm_range(tlb->mm, tlb->start, tlb->end, + tlb_get_unmap_size(tlb)); +#endif } =20 #endif /* _ASM_RISCV_TLB_H */ diff --git a/arch/riscv/include/asm/tlbflush.h b/arch/riscv/include/asm/tlb= flush.h index a09196f8de68..f5c4fb0ae642 100644 --- a/arch/riscv/include/asm/tlbflush.h +++ b/arch/riscv/include/asm/tlbflush.h @@ -32,6 +32,8 @@ static inline void local_flush_tlb_page(unsigned long add= r) #if defined(CONFIG_SMP) && defined(CONFIG_MMU) void flush_tlb_all(void); void flush_tlb_mm(struct mm_struct *mm); +void flush_tlb_mm_range(struct mm_struct *mm, unsigned long start, + unsigned long end, unsigned int page_size); void flush_tlb_page(struct vm_area_struct *vma, unsigned long addr); void flush_tlb_range(struct vm_area_struct *vma, unsigned long start, unsigned long end); @@ -52,6 +54,7 @@ static inline void flush_tlb_range(struct vm_area_struct = *vma, } =20 #define flush_tlb_mm(mm) flush_tlb_all() +#define flush_tlb_mm_range(mm, start, end, page_size) flush_tlb_all() #endif /* !CONFIG_SMP || !CONFIG_MMU */ =20 /* Flush a range of kernel pages */ diff --git a/arch/riscv/mm/tlbflush.c b/arch/riscv/mm/tlbflush.c index 77be59aadc73..fa03289853d8 100644 --- a/arch/riscv/mm/tlbflush.c +++ b/arch/riscv/mm/tlbflush.c @@ -132,6 +132,13 @@ void flush_tlb_mm(struct mm_struct *mm) __flush_tlb_range(mm, 0, -1, PAGE_SIZE); } =20 +void flush_tlb_mm_range(struct mm_struct *mm, + unsigned long start, unsigned long end, + unsigned int page_size) +{ + __flush_tlb_range(mm, start, end - start, page_size); +} + void flush_tlb_page(struct vm_area_struct *vma, unsigned long addr) { __flush_tlb_range(vma->vm_mm, addr, PAGE_SIZE, PAGE_SIZE); --=20 2.42.0 From nobody Tue Dec 16 20:00:44 2025 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 858A4C4167D for ; Sat, 28 Oct 2023 23:14:01 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229916AbjJ1XNv (ORCPT ); Sat, 28 Oct 2023 19:13:51 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59562 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229846AbjJ1XNr (ORCPT ); Sat, 28 Oct 2023 19:13:47 -0400 Received: from mail-pf1-x435.google.com (mail-pf1-x435.google.com [IPv6:2607:f8b0:4864:20::435]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 357D2CF for ; Sat, 28 Oct 2023 16:13:45 -0700 (PDT) Received: by mail-pf1-x435.google.com with SMTP id d2e1a72fcca58-6be0277c05bso2898763b3a.0 for ; Sat, 28 Oct 2023 16:13:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1698534824; x=1699139624; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=h4NRNhpDELMiOf33wYuzLfhwPwCzaIpnJue9EEzIQdY=; b=cYexk+t4MDdaGS9C/G7GEcnAquNdYT4h4DJmHyfN939k5hS/3v5peyIsTse7ItTDSN 5cWTIjiFsIm23ptts5Lgkq3tYwc3QPfqB+rlm2QlYkLpYd50euh1FI5bLNAUZEKOqKQz g+VQJEeVs6c/2xakEfffs9xVHllio+LN8xZg1LXXcq13pSLHnG6WRXFq0IbJC+f8OSDR PuF9cu8ivJXUaojwfrjg6r/CWH5AF6SebaZ53xIsePzRyPr3ETdf5XZx684Vorh5nD3O 694by1kg720Sz99E7eVFl7WwwwJyGM/rPIeikiGpJXD1raNdsZuNQpEBeXirFExd7alQ Jdsw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1698534824; x=1699139624; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=h4NRNhpDELMiOf33wYuzLfhwPwCzaIpnJue9EEzIQdY=; b=QiiTr8WfYK0032zI16vHpynfApRDKZnq4qmYluFbg4XzSmFSLAi0QcalTQoQWJnZJy bVSEzAEN1L0EbSl5AO8LU5hyIR/gALANo+al/+xTlpxORn7vASrYw4uEJntIkDJBnzKU rrzYacQQzaxXWR7dzyM7n16ZxPH3VDWgArgFD/itnaGmH74PRCiWgPeCT9LQ9Xtw+4wi jnJdL52k0qA2aacG+txyIGCgMwZFmTPI8AY2S0hMdThB5cI8g31YvwZJmriOuljp/XTu /8/NB0mjFDZ4TD/O6fujg05KtxJaVlLiKDZIAXaN8it3qruQH8mJZSUfDmqpnm3gzHGS wIZg== X-Gm-Message-State: AOJu0YxGTn4MDkKwcd6wd1tP3gsA16oyJeCV7OPuA32+O9Vp+witCvJ3 +8L9EchMKnBNTS1r0uw/YIit0A== X-Google-Smtp-Source: AGHT+IHFtwqkvMLL8vTXw7lsMTaEWiNW/nYYuBPFSSXqkCgtp1/aoWCoS2LcxRJlFYSU2eWn0N2URQ== X-Received: by 2002:a05:6a21:3b45:b0:16b:a5fb:eee5 with SMTP id zy5-20020a056a213b4500b0016ba5fbeee5mr5984191pzb.28.1698534824609; Sat, 28 Oct 2023 16:13:44 -0700 (PDT) Received: from sw06.internal.sifive.com ([4.53.31.132]) by smtp.gmail.com with ESMTPSA id u17-20020a17090341d100b001b8622c1ad2sm3679345ple.130.2023.10.28.16.13.43 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 28 Oct 2023 16:13:43 -0700 (PDT) From: Samuel Holland To: Palmer Dabbelt , Alexandre Ghiti , linux-riscv@lists.infradead.org Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, Samuel Holland Subject: [PATCH v2 02/11] riscv: Improve flush_tlb_range() for hugetlb pages Date: Sat, 28 Oct 2023 16:12:00 -0700 Message-ID: <20231028231339.3116618-3-samuel.holland@sifive.com> X-Mailer: git-send-email 2.42.0 In-Reply-To: <20231028231339.3116618-1-samuel.holland@sifive.com> References: <20231028231339.3116618-1-samuel.holland@sifive.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Alexandre Ghiti flush_tlb_range() uses a fixed stride of PAGE_SIZE and in its current form, when a hugetlb mapping needs to be flushed, flush_tlb_range() flushes the whole tlb: so set a stride of the size of the hugetlb mapping in order to only flush the hugetlb mapping. However, if the hugepage is a NAPOT region, all PTEs that constitute this mapping must be invalidated, so the stride size must actually be the size of the PTE. Note that THPs are directly handled by flush_pmd_tlb_range(). Signed-off-by: Alexandre Ghiti [Samuel: Removed CONFIG_RISCV_ISA_SVNAPOT check] Signed-off-by: Samuel Holland --- Changes in v2: - Rebase on Alexandre's "riscv: tlb flush improvements" series v5 arch/riscv/mm/tlbflush.c | 29 ++++++++++++++++++++++++++++- 1 file changed, 28 insertions(+), 1 deletion(-) diff --git a/arch/riscv/mm/tlbflush.c b/arch/riscv/mm/tlbflush.c index fa03289853d8..b6d712a82306 100644 --- a/arch/riscv/mm/tlbflush.c +++ b/arch/riscv/mm/tlbflush.c @@ -3,6 +3,7 @@ #include #include #include +#include #include #include =20 @@ -147,7 +148,33 @@ void flush_tlb_page(struct vm_area_struct *vma, unsign= ed long addr) void flush_tlb_range(struct vm_area_struct *vma, unsigned long start, unsigned long end) { - __flush_tlb_range(vma->vm_mm, start, end - start, PAGE_SIZE); + unsigned long stride_size; + + if (!is_vm_hugetlb_page(vma)) { + stride_size =3D PAGE_SIZE; + } else { + stride_size =3D huge_page_size(hstate_vma(vma)); + + /* + * As stated in the privileged specification, every PTE in a + * NAPOT region must be invalidated, so reset the stride in that + * case. + */ + if (has_svnapot()) { + if (stride_size >=3D PGDIR_SIZE) + stride_size =3D PGDIR_SIZE; + else if (stride_size >=3D P4D_SIZE) + stride_size =3D P4D_SIZE; + else if (stride_size >=3D PUD_SIZE) + stride_size =3D PUD_SIZE; + else if (stride_size >=3D PMD_SIZE) + stride_size =3D PMD_SIZE; + else + stride_size =3D PAGE_SIZE; + } + } + + __flush_tlb_range(vma->vm_mm, start, end - start, stride_size); } #ifdef CONFIG_TRANSPARENT_HUGEPAGE void flush_pmd_tlb_range(struct vm_area_struct *vma, unsigned long start, --=20 2.42.0 From nobody Tue Dec 16 20:00:44 2025 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 99F53C4708E for ; Sat, 28 Oct 2023 23:14:01 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229946AbjJ1XNw (ORCPT ); Sat, 28 Oct 2023 19:13:52 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59578 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229900AbjJ1XNt (ORCPT ); Sat, 28 Oct 2023 19:13:49 -0400 Received: from mail-pl1-x633.google.com (mail-pl1-x633.google.com [IPv6:2607:f8b0:4864:20::633]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6D104CF for ; Sat, 28 Oct 2023 16:13:46 -0700 (PDT) Received: by mail-pl1-x633.google.com with SMTP id d9443c01a7336-1cc1ee2d8dfso17081865ad.3 for ; Sat, 28 Oct 2023 16:13:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1698534826; x=1699139626; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=KwX4pskItPHUM4maFDdS5O3Er1CtrhdQQ55njchpgeM=; b=KCI56l7+q406XimDYx0gHIBqVZ7c1ibFjbheys2NVrylNMm2YNQibd2+ZgE+f8P5AV 5H147MHBlgZJg/iASpblI4y1l+ESFSioxHF8JEl2xCRTwRLQJzzlHwPK5JbyT3duQqpY 796veG/Tpm846iniT/lRWF2KT2WPED3/sXaTXnasfBl5MeW0QoQZ0nBilKFAAiJRb6hu eJ/lOonaRcy3i57mpLH2KTYeEvPhDFmOQikCRgnwuiOnEgQ0oaCfhkN9GWaOz3yKCCAx p+2IzgYHZ6O6NECKFBsTeYsmgF+uSu6+8y7XhnneRzA9fZfEaoLLVJvyCtv2XI+XRewe S9qQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1698534826; x=1699139626; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=KwX4pskItPHUM4maFDdS5O3Er1CtrhdQQ55njchpgeM=; b=RfEKWuIdW6B3P5RUHpIq9WderDs/5yjTR8bx/JKA3a2nRm67ZIX3tsXBSFAXuDrA2C 2NYQhsobyz33cozLqlrQI41lUfHdYA1QTZOZjd383u0axHhqG5hFr2WvcqlAarFCx6Yo DAy5zd5UE5Om2LTRfAxb8Rrc2kAk9srKkoAeRumdpcd2QvtpRvvlzFtchbrErqwTZQPS SYHXHXRKgexIT2Wm1db2u+3rB2i1auDqXXd3q811nc5igi4xoRBovcytWd1ELO22tlI8 3UbcDdAFXWWzatTIyQzcRwX0B7gih6C+U4ADHVDlOA9AeL+3G1M3CKSaUDe/8YfXE0R7 aXtw== X-Gm-Message-State: AOJu0Yy5B6qvzM7vI5YeNAFZYVcTym+RLzRJIgkvPuSdDTOwmZt03gH4 svygVpHLw9rYoPmoaYNZybRb3A== X-Google-Smtp-Source: AGHT+IEP1RLLbzkV6SDyhDMWypu6mUWTgp+BfL6QnGjrDAvgGu3DpZZT/BgKJhnh9HCZ+gZpZ5q05A== X-Received: by 2002:a17:902:8c85:b0:1c9:c0fa:dfb7 with SMTP id t5-20020a1709028c8500b001c9c0fadfb7mr6367498plo.57.1698534825930; Sat, 28 Oct 2023 16:13:45 -0700 (PDT) Received: from sw06.internal.sifive.com ([4.53.31.132]) by smtp.gmail.com with ESMTPSA id u17-20020a17090341d100b001b8622c1ad2sm3679345ple.130.2023.10.28.16.13.45 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 28 Oct 2023 16:13:45 -0700 (PDT) From: Samuel Holland To: Palmer Dabbelt , Alexandre Ghiti , linux-riscv@lists.infradead.org Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, Mayuresh Chitale , Andrew Jones , Lad Prabhakar , Samuel Holland Subject: [PATCH v2 03/11] riscv: Make __flush_tlb_range() loop over pte instead of flushing the whole tlb Date: Sat, 28 Oct 2023 16:12:01 -0700 Message-ID: <20231028231339.3116618-4-samuel.holland@sifive.com> X-Mailer: git-send-email 2.42.0 In-Reply-To: <20231028231339.3116618-1-samuel.holland@sifive.com> References: <20231028231339.3116618-1-samuel.holland@sifive.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Alexandre Ghiti Currently, when the range to flush covers more than one page (a 4K page or a hugepage), __flush_tlb_range() flushes the whole tlb. Flushing the whole tlb comes with a greater cost than flushing a single entry so we should flush single entries up to a certain threshold so that: threshold * cost of flushing a single entry < cost of flushing the whole tlb. Co-developed-by: Mayuresh Chitale Signed-off-by: Mayuresh Chitale Signed-off-by: Alexandre Ghiti Reviewed-by: Andrew Jones Tested-by: Lad Prabhakar # On RZ/= Five SMARC [Samuel: Fixed type of nr_ptes_in_range] Signed-off-by: Samuel Holland --- Changes in v2: - Rebase on Alexandre's "riscv: tlb flush improvements" series v5 arch/riscv/include/asm/sbi.h | 3 - arch/riscv/include/asm/tlbflush.h | 3 + arch/riscv/kernel/sbi.c | 32 +++------ arch/riscv/mm/tlbflush.c | 115 +++++++++++++++--------------- 4 files changed, 72 insertions(+), 81 deletions(-) diff --git a/arch/riscv/include/asm/sbi.h b/arch/riscv/include/asm/sbi.h index 5b4a1bf5f439..b79d0228144f 100644 --- a/arch/riscv/include/asm/sbi.h +++ b/arch/riscv/include/asm/sbi.h @@ -273,9 +273,6 @@ void sbi_set_timer(uint64_t stime_value); void sbi_shutdown(void); void sbi_send_ipi(unsigned int cpu); int sbi_remote_fence_i(const struct cpumask *cpu_mask); -int sbi_remote_sfence_vma(const struct cpumask *cpu_mask, - unsigned long start, - unsigned long size); =20 int sbi_remote_sfence_vma_asid(const struct cpumask *cpu_mask, unsigned long start, diff --git a/arch/riscv/include/asm/tlbflush.h b/arch/riscv/include/asm/tlb= flush.h index f5c4fb0ae642..170a49c531c6 100644 --- a/arch/riscv/include/asm/tlbflush.h +++ b/arch/riscv/include/asm/tlbflush.h @@ -11,6 +11,9 @@ #include #include =20 +#define FLUSH_TLB_MAX_SIZE ((unsigned long)-1) +#define FLUSH_TLB_NO_ASID ((unsigned long)-1) + #ifdef CONFIG_MMU extern unsigned long asid_mask; =20 diff --git a/arch/riscv/kernel/sbi.c b/arch/riscv/kernel/sbi.c index c672c8ba9a2a..5a62ed1da453 100644 --- a/arch/riscv/kernel/sbi.c +++ b/arch/riscv/kernel/sbi.c @@ -11,6 +11,7 @@ #include #include #include +#include =20 /* default SBI version is 0.1 */ unsigned long sbi_spec_version __ro_after_init =3D SBI_SPEC_VERSION_DEFAUL= T; @@ -376,32 +377,15 @@ int sbi_remote_fence_i(const struct cpumask *cpu_mask) } EXPORT_SYMBOL(sbi_remote_fence_i); =20 -/** - * sbi_remote_sfence_vma() - Execute SFENCE.VMA instructions on given remo= te - * harts for the specified virtual address range. - * @cpu_mask: A cpu mask containing all the target harts. - * @start: Start of the virtual address - * @size: Total size of the virtual address range. - * - * Return: 0 on success, appropriate linux error code otherwise. - */ -int sbi_remote_sfence_vma(const struct cpumask *cpu_mask, - unsigned long start, - unsigned long size) -{ - return __sbi_rfence(SBI_EXT_RFENCE_REMOTE_SFENCE_VMA, - cpu_mask, start, size, 0, 0); -} -EXPORT_SYMBOL(sbi_remote_sfence_vma); - /** * sbi_remote_sfence_vma_asid() - Execute SFENCE.VMA instructions on given - * remote harts for a virtual address range belonging to a specific ASID. + * remote harts for a virtual address range belonging to a specific ASID o= r not. * * @cpu_mask: A cpu mask containing all the target harts. * @start: Start of the virtual address * @size: Total size of the virtual address range. - * @asid: The value of address space identifier (ASID). + * @asid: The value of address space identifier (ASID), or FLUSH_TLB_NO_AS= ID + * for flushing all address spaces. * * Return: 0 on success, appropriate linux error code otherwise. */ @@ -410,8 +394,12 @@ int sbi_remote_sfence_vma_asid(const struct cpumask *c= pu_mask, unsigned long size, unsigned long asid) { - return __sbi_rfence(SBI_EXT_RFENCE_REMOTE_SFENCE_VMA_ASID, - cpu_mask, start, size, asid, 0); + if (asid =3D=3D FLUSH_TLB_NO_ASID) + return __sbi_rfence(SBI_EXT_RFENCE_REMOTE_SFENCE_VMA, + cpu_mask, start, size, 0, 0); + else + return __sbi_rfence(SBI_EXT_RFENCE_REMOTE_SFENCE_VMA_ASID, + cpu_mask, start, size, asid, 0); } EXPORT_SYMBOL(sbi_remote_sfence_vma_asid); =20 diff --git a/arch/riscv/mm/tlbflush.c b/arch/riscv/mm/tlbflush.c index b6d712a82306..e46fefc70927 100644 --- a/arch/riscv/mm/tlbflush.c +++ b/arch/riscv/mm/tlbflush.c @@ -9,28 +9,50 @@ =20 static inline void local_flush_tlb_all_asid(unsigned long asid) { - __asm__ __volatile__ ("sfence.vma x0, %0" - : - : "r" (asid) - : "memory"); + if (asid !=3D FLUSH_TLB_NO_ASID) + __asm__ __volatile__ ("sfence.vma x0, %0" + : + : "r" (asid) + : "memory"); + else + local_flush_tlb_all(); } =20 static inline void local_flush_tlb_page_asid(unsigned long addr, unsigned long asid) { - __asm__ __volatile__ ("sfence.vma %0, %1" - : - : "r" (addr), "r" (asid) - : "memory"); + if (asid !=3D FLUSH_TLB_NO_ASID) + __asm__ __volatile__ ("sfence.vma %0, %1" + : + : "r" (addr), "r" (asid) + : "memory"); + else + local_flush_tlb_page(addr); } =20 -static inline void local_flush_tlb_range(unsigned long start, - unsigned long size, unsigned long stride) +/* + * Flush entire TLB if number of entries to be flushed is greater + * than the threshold below. + */ +static unsigned long tlb_flush_all_threshold __read_mostly =3D 64; + +static void local_flush_tlb_range_threshold_asid(unsigned long start, + unsigned long size, + unsigned long stride, + unsigned long asid) { - if (size <=3D stride) - local_flush_tlb_page(start); - else - local_flush_tlb_all(); + unsigned long nr_ptes_in_range =3D DIV_ROUND_UP(size, stride); + int i; + + if (nr_ptes_in_range > tlb_flush_all_threshold) { + local_flush_tlb_all_asid(asid); + return; + } + + for (i =3D 0; i < nr_ptes_in_range; ++i) { + local_flush_tlb_page_asid(start, asid); + start +=3D stride; + } } =20 static inline void local_flush_tlb_range_asid(unsigned long start, @@ -38,8 +60,10 @@ static inline void local_flush_tlb_range_asid(unsigned l= ong start, { if (size <=3D stride) local_flush_tlb_page_asid(start, asid); - else + else if (size =3D=3D FLUSH_TLB_MAX_SIZE) local_flush_tlb_all_asid(asid); + else + local_flush_tlb_range_threshold_asid(start, size, stride, asid); } =20 static void __ipi_flush_tlb_all(void *info) @@ -52,7 +76,7 @@ void flush_tlb_all(void) if (riscv_use_ipi_for_rfence()) on_each_cpu(__ipi_flush_tlb_all, NULL, 1); else - sbi_remote_sfence_vma(NULL, 0, -1); + sbi_remote_sfence_vma_asid(NULL, 0, FLUSH_TLB_MAX_SIZE, FLUSH_TLB_NO_ASI= D); } =20 struct flush_tlb_range_data { @@ -69,18 +93,12 @@ static void __ipi_flush_tlb_range_asid(void *info) local_flush_tlb_range_asid(d->start, d->size, d->stride, d->asid); } =20 -static void __ipi_flush_tlb_range(void *info) -{ - struct flush_tlb_range_data *d =3D info; - - local_flush_tlb_range(d->start, d->size, d->stride); -} - static void __flush_tlb_range(struct mm_struct *mm, unsigned long start, unsigned long size, unsigned long stride) { struct flush_tlb_range_data ftd; struct cpumask *cmask =3D mm_cpumask(mm); + unsigned long asid =3D FLUSH_TLB_NO_ASID; unsigned int cpuid; bool broadcast; =20 @@ -90,39 +108,24 @@ static void __flush_tlb_range(struct mm_struct *mm, un= signed long start, cpuid =3D get_cpu(); /* check if the tlbflush needs to be sent to other CPUs */ broadcast =3D cpumask_any_but(cmask, cpuid) < nr_cpu_ids; - if (static_branch_unlikely(&use_asid_allocator)) { - unsigned long asid =3D atomic_long_read(&mm->context.id) & asid_mask; - - if (broadcast) { - if (riscv_use_ipi_for_rfence()) { - ftd.asid =3D asid; - ftd.start =3D start; - ftd.size =3D size; - ftd.stride =3D stride; - on_each_cpu_mask(cmask, - __ipi_flush_tlb_range_asid, - &ftd, 1); - } else - sbi_remote_sfence_vma_asid(cmask, - start, size, asid); - } else { - local_flush_tlb_range_asid(start, size, stride, asid); - } + + if (static_branch_unlikely(&use_asid_allocator)) + asid =3D atomic_long_read(&mm->context.id) & asid_mask; + + if (broadcast) { + if (riscv_use_ipi_for_rfence()) { + ftd.asid =3D asid; + ftd.start =3D start; + ftd.size =3D size; + ftd.stride =3D stride; + on_each_cpu_mask(cmask, + __ipi_flush_tlb_range_asid, + &ftd, 1); + } else + sbi_remote_sfence_vma_asid(cmask, + start, size, asid); } else { - if (broadcast) { - if (riscv_use_ipi_for_rfence()) { - ftd.asid =3D 0; - ftd.start =3D start; - ftd.size =3D size; - ftd.stride =3D stride; - on_each_cpu_mask(cmask, - __ipi_flush_tlb_range, - &ftd, 1); - } else - sbi_remote_sfence_vma(cmask, start, size); - } else { - local_flush_tlb_range(start, size, stride); - } + local_flush_tlb_range_asid(start, size, stride, asid); } =20 put_cpu(); @@ -130,7 +133,7 @@ static void __flush_tlb_range(struct mm_struct *mm, uns= igned long start, =20 void flush_tlb_mm(struct mm_struct *mm) { - __flush_tlb_range(mm, 0, -1, PAGE_SIZE); + __flush_tlb_range(mm, 0, FLUSH_TLB_MAX_SIZE, PAGE_SIZE); } =20 void flush_tlb_mm_range(struct mm_struct *mm, --=20 2.42.0 From nobody Tue Dec 16 20:00:44 2025 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id CC3ACC4167B for ; Sat, 28 Oct 2023 23:14:04 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229994AbjJ1XOD (ORCPT ); Sat, 28 Oct 2023 19:14:03 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59600 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229907AbjJ1XNu (ORCPT ); Sat, 28 Oct 2023 19:13:50 -0400 Received: from mail-oo1-xc31.google.com (mail-oo1-xc31.google.com [IPv6:2607:f8b0:4864:20::c31]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 57BAECF for ; Sat, 28 Oct 2023 16:13:48 -0700 (PDT) Received: by mail-oo1-xc31.google.com with SMTP id 006d021491bc7-5842ea6f4d5so2056534eaf.2 for ; Sat, 28 Oct 2023 16:13:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1698534827; x=1699139627; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=56YUmquglDtdI2AgsFp8Dfqen0eRNcOUaE781b7rud8=; b=ltVGMWN501Ht3FyBhCMSAqOhTivfwTdhkwxCzbmDH56Uw4Wu9iSpbYjdgTnTyalDrh 4hLbUQOqIyiM3ffGlkR6nHhxy4y7KNqaFEX96gaoV5GliS2T2CPJGYdaqqotWD2tbptT /Aq9ofZi1dhTGuq42OVU8sExodm3w9xVfwtFFpmaFUOU59wx+uuEoBtCC7v+1Hw9wuF7 s6svYum0TR5sC3ESGxkUb0WEBbzP3o6jsxJdGCNiT4MNDbEEU3Yp/C6KRLLqbdrgOG0i M2xHGfAvMVhqNoEOPuPQoIgfArpnwGKXr0ChZc09GWA/rVmHR5EXGFpSP2hE/jemWYGj w/4A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1698534827; x=1699139627; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=56YUmquglDtdI2AgsFp8Dfqen0eRNcOUaE781b7rud8=; b=pnxOXzXx7DQ45p2PZrmATTIP1qdXT+5kcY6R16b7NIA80mHkOQdTm10nhkOXAo+abP h6gEy+PWuBC7IVlTN98tk1WbbuQ5barXGsQudJOqmSchwua8nCd+9RljDDXzOsAlPf0C y2uxQT3v9vkPgoQ73gJfph4qCpCKqN6wwG42yFOSYE7xV0f+gXHp4h+UNm8stRaesWSu fLzDdrPASNmTHHYjPSjsZT1CDtw8Ir+o8qGCbHE6o0oFMBZyX1qJVtsZs5dFmQFKDEsF clTPh81bOKe+k1HOKqI0MrrGcWBL2ChbAHeT9ziixrkN8fYgJj4kAlsP7aKS5I9d/1tY JiRQ== X-Gm-Message-State: AOJu0YxoPnM+sMcDPEhIoZJbJaFLyx3g7FLXORRP8ClsbFUmSTHX7ZwZ RrOs03z/uw8REkcuLpFxhlByoA== X-Google-Smtp-Source: AGHT+IGq5Mcxzf3tN07mJTIgv1Wxotaz2Uaine2ZZiRfV+MdLhpoDUKwMV8rRW6x4wnAP4zHnD4SGA== X-Received: by 2002:a05:6358:903:b0:168:dc03:5b90 with SMTP id r3-20020a056358090300b00168dc035b90mr9086199rwi.23.1698534827455; Sat, 28 Oct 2023 16:13:47 -0700 (PDT) Received: from sw06.internal.sifive.com ([4.53.31.132]) by smtp.gmail.com with ESMTPSA id u17-20020a17090341d100b001b8622c1ad2sm3679345ple.130.2023.10.28.16.13.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 28 Oct 2023 16:13:46 -0700 (PDT) From: Samuel Holland To: Palmer Dabbelt , Alexandre Ghiti , linux-riscv@lists.infradead.org Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, Andrew Jones , Lad Prabhakar , Samuel Holland Subject: [PATCH v2 04/11] riscv: Improve flush_tlb_kernel_range() Date: Sat, 28 Oct 2023 16:12:02 -0700 Message-ID: <20231028231339.3116618-5-samuel.holland@sifive.com> X-Mailer: git-send-email 2.42.0 In-Reply-To: <20231028231339.3116618-1-samuel.holland@sifive.com> References: <20231028231339.3116618-1-samuel.holland@sifive.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Alexandre Ghiti This function used to simply flush the whole tlb of all harts, be more subtile and try to only flush the range. The problem is that we can only use PAGE_SIZE as stride since we don't know the size of the underlying mapping and then this function will be improved only if the size of the region to flush is < threshold * PAGE_SIZE. Signed-off-by: Alexandre Ghiti Reviewed-by: Andrew Jones Tested-by: Lad Prabhakar # On RZ/= Five SMARC [Samuel: Use cpu_online_mask and merge if statements] Signed-off-by: Samuel Holland --- Changes in v2: - Rebase on Alexandre's "riscv: tlb flush improvements" series v5 arch/riscv/include/asm/tlbflush.h | 11 +++++----- arch/riscv/mm/tlbflush.c | 34 ++++++++++++++++++++++--------- 2 files changed, 30 insertions(+), 15 deletions(-) diff --git a/arch/riscv/include/asm/tlbflush.h b/arch/riscv/include/asm/tlb= flush.h index 170a49c531c6..8f3418c5f172 100644 --- a/arch/riscv/include/asm/tlbflush.h +++ b/arch/riscv/include/asm/tlbflush.h @@ -40,6 +40,7 @@ void flush_tlb_mm_range(struct mm_struct *mm, unsigned lo= ng start, void flush_tlb_page(struct vm_area_struct *vma, unsigned long addr); void flush_tlb_range(struct vm_area_struct *vma, unsigned long start, unsigned long end); +void flush_tlb_kernel_range(unsigned long start, unsigned long end); #ifdef CONFIG_TRANSPARENT_HUGEPAGE #define __HAVE_ARCH_FLUSH_PMD_TLB_RANGE void flush_pmd_tlb_range(struct vm_area_struct *vma, unsigned long start, @@ -56,15 +57,15 @@ static inline void flush_tlb_range(struct vm_area_struc= t *vma, local_flush_tlb_all(); } =20 -#define flush_tlb_mm(mm) flush_tlb_all() -#define flush_tlb_mm_range(mm, start, end, page_size) flush_tlb_all() -#endif /* !CONFIG_SMP || !CONFIG_MMU */ - /* Flush a range of kernel pages */ static inline void flush_tlb_kernel_range(unsigned long start, unsigned long end) { - flush_tlb_all(); + local_flush_tlb_all(); } =20 +#define flush_tlb_mm(mm) flush_tlb_all() +#define flush_tlb_mm_range(mm, start, end, page_size) flush_tlb_all() +#endif /* !CONFIG_SMP || !CONFIG_MMU */ + #endif /* _ASM_RISCV_TLBFLUSH_H */ diff --git a/arch/riscv/mm/tlbflush.c b/arch/riscv/mm/tlbflush.c index e46fefc70927..e6659d7368b3 100644 --- a/arch/riscv/mm/tlbflush.c +++ b/arch/riscv/mm/tlbflush.c @@ -97,20 +97,27 @@ static void __flush_tlb_range(struct mm_struct *mm, uns= igned long start, unsigned long size, unsigned long stride) { struct flush_tlb_range_data ftd; - struct cpumask *cmask =3D mm_cpumask(mm); + const struct cpumask *cmask; unsigned long asid =3D FLUSH_TLB_NO_ASID; - unsigned int cpuid; bool broadcast; =20 - if (cpumask_empty(cmask)) - return; + if (mm) { + unsigned int cpuid; + + cmask =3D mm_cpumask(mm); + if (cpumask_empty(cmask)) + return; =20 - cpuid =3D get_cpu(); - /* check if the tlbflush needs to be sent to other CPUs */ - broadcast =3D cpumask_any_but(cmask, cpuid) < nr_cpu_ids; + cpuid =3D get_cpu(); + /* check if the tlbflush needs to be sent to other CPUs */ + broadcast =3D cpumask_any_but(cmask, cpuid) < nr_cpu_ids; =20 - if (static_branch_unlikely(&use_asid_allocator)) - asid =3D atomic_long_read(&mm->context.id) & asid_mask; + if (static_branch_unlikely(&use_asid_allocator)) + asid =3D atomic_long_read(&mm->context.id) & asid_mask; + } else { + cmask =3D cpu_online_mask; + broadcast =3D true; + } =20 if (broadcast) { if (riscv_use_ipi_for_rfence()) { @@ -128,7 +135,8 @@ static void __flush_tlb_range(struct mm_struct *mm, uns= igned long start, local_flush_tlb_range_asid(start, size, stride, asid); } =20 - put_cpu(); + if (mm) + put_cpu(); } =20 void flush_tlb_mm(struct mm_struct *mm) @@ -179,6 +187,12 @@ void flush_tlb_range(struct vm_area_struct *vma, unsig= ned long start, =20 __flush_tlb_range(vma->vm_mm, start, end - start, stride_size); } + +void flush_tlb_kernel_range(unsigned long start, unsigned long end) +{ + __flush_tlb_range(NULL, start, end - start, PAGE_SIZE); +} + #ifdef CONFIG_TRANSPARENT_HUGEPAGE void flush_pmd_tlb_range(struct vm_area_struct *vma, unsigned long start, unsigned long end) --=20 2.42.0 From nobody Tue Dec 16 20:00:44 2025 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 42361C0018C for ; Sat, 28 Oct 2023 23:14:01 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230064AbjJ1XOA (ORCPT ); Sat, 28 Oct 2023 19:14:00 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59602 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229930AbjJ1XNw (ORCPT ); Sat, 28 Oct 2023 19:13:52 -0400 Received: from mail-pl1-x62a.google.com (mail-pl1-x62a.google.com [IPv6:2607:f8b0:4864:20::62a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 434BAD6 for ; Sat, 28 Oct 2023 16:13:49 -0700 (PDT) Received: by mail-pl1-x62a.google.com with SMTP id d9443c01a7336-1cc20df5187so12487385ad.0 for ; Sat, 28 Oct 2023 16:13:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1698534829; x=1699139629; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=7iD5kVeuyzksz9qFDj4nk3gJEMMhm0SiL+jEgfo9gAo=; b=N0Riwa/miVN0GdyvyQAo5JXhlyamqaUNRyieLkclAEVbUF0t+IaKzLi0qcgVJbQAnI 8UBcy7M/LFJa5UUH0V9KRqbpswk00GSmI+OgXiODEImnkc8R217odLBl9uM4/guxZfBZ V4e8rT1W5O9E9u7DCaGBloMBDH3Q6iTLAWaRmMAU3lfOUCsVlrQYWa5wLglvLmkFS4+C DyEvVh3xBxVwJXagt4WRfXdzOlCQ9wOJH6qgZ1gl2kSn0VY0NANWLkFbKVyX0BZeNkYj uKbKaqc0bzt4mRPf3HTVVGgWLNQuK/91q0e9VEZ/a8whg/xLO/ptCjMG2+9bfTdketH0 ye3Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1698534829; x=1699139629; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=7iD5kVeuyzksz9qFDj4nk3gJEMMhm0SiL+jEgfo9gAo=; b=EkFsgmZjX/yeUahXR86RbbZa/Q80V7oTIzqbjg6aSKZ8IgLePmAXbNHGNC7LOVTzna dMUV7q6CCr548BsBCd95JK2t0m8rN5oQN7kszfRRT5ch1joUa5SchEXPYrom8lnMfKow ZBVy0Jgt3ld3Ll+31IZ9892oFXYelFj7zhpUaH6KmUmnGfmF4HHcSRnvGTqWHwU6wRnp vGVxSsYyLWYxfIKJUW6u2P/7vPlTkPhgR3cgLul4/5tM7EKG7Zzm4oQjWi9H3VA5c/eO fi66hrt2VbmwRmpuh2s6j7er2SoNn6dfSnUGjVQGwbMqlC+UKh6mP+3GAhVDo6Qhz7T1 MvwA== X-Gm-Message-State: AOJu0YzmOmDWxFixcNkmuOh2t25z6GszEj+f2VncdHVzIGC6kwFCU40B cvz0s+cMGmqcBQe+lfzV1azLhQ== X-Google-Smtp-Source: AGHT+IGZfjM6oWIducpDDQ7hM02X3ekD2Mrbs3+cVpCJdtVQrXLcSNbo2i119+LD0HVTSm0uB+XkgQ== X-Received: by 2002:a17:903:11d1:b0:1cc:f41:8f82 with SMTP id q17-20020a17090311d100b001cc0f418f82mr8777913plh.16.1698534828754; Sat, 28 Oct 2023 16:13:48 -0700 (PDT) Received: from sw06.internal.sifive.com ([4.53.31.132]) by smtp.gmail.com with ESMTPSA id u17-20020a17090341d100b001b8622c1ad2sm3679345ple.130.2023.10.28.16.13.47 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 28 Oct 2023 16:13:48 -0700 (PDT) From: Samuel Holland To: Palmer Dabbelt , Alexandre Ghiti , linux-riscv@lists.infradead.org Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, Samuel Holland Subject: [PATCH v2 05/11] riscv: mm: Combine the SMP and UP TLB flush code Date: Sat, 28 Oct 2023 16:12:03 -0700 Message-ID: <20231028231339.3116618-6-samuel.holland@sifive.com> X-Mailer: git-send-email 2.42.0 In-Reply-To: <20231028231339.3116618-1-samuel.holland@sifive.com> References: <20231028231339.3116618-1-samuel.holland@sifive.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" In SMP configurations, all TLB flushing narrower than flush_tlb_all() goes through __flush_tlb_range(). Do the same in UP configurations. This allows UP configurations to take advantage of recent improvements to the code in tlbflush.c, such as support for huge pages and flushing multiple-page ranges. Signed-off-by: Samuel Holland --- Changes in v2: - Move the SMP/UP merge earlier in the series to avoid build issues - Make a copy of __flush_tlb_range() instead of adding ifdefs inside - local_flush_tlb_all() is the only function used on !MMU (smpboot.c) arch/riscv/include/asm/tlbflush.h | 33 +++++++------------------------ arch/riscv/mm/Makefile | 5 +---- arch/riscv/mm/tlbflush.c | 13 ++++++++++++ 3 files changed, 21 insertions(+), 30 deletions(-) diff --git a/arch/riscv/include/asm/tlbflush.h b/arch/riscv/include/asm/tlb= flush.h index 8f3418c5f172..317a1811aa51 100644 --- a/arch/riscv/include/asm/tlbflush.h +++ b/arch/riscv/include/asm/tlbflush.h @@ -27,13 +27,12 @@ static inline void local_flush_tlb_page(unsigned long a= ddr) { ALT_FLUSH_TLB_PAGE(__asm__ __volatile__ ("sfence.vma %0" : : "r" (addr) := "memory")); } -#else /* CONFIG_MMU */ -#define local_flush_tlb_all() do { } while (0) -#define local_flush_tlb_page(addr) do { } while (0) -#endif /* CONFIG_MMU */ =20 -#if defined(CONFIG_SMP) && defined(CONFIG_MMU) +#ifdef CONFIG_SMP void flush_tlb_all(void); +#else +#define flush_tlb_all() local_flush_tlb_all() +#endif void flush_tlb_mm(struct mm_struct *mm); void flush_tlb_mm_range(struct mm_struct *mm, unsigned long start, unsigned long end, unsigned int page_size); @@ -46,26 +45,8 @@ void flush_tlb_kernel_range(unsigned long start, unsigne= d long end); void flush_pmd_tlb_range(struct vm_area_struct *vma, unsigned long start, unsigned long end); #endif -#else /* CONFIG_SMP && CONFIG_MMU */ - -#define flush_tlb_all() local_flush_tlb_all() -#define flush_tlb_page(vma, addr) local_flush_tlb_page(addr) - -static inline void flush_tlb_range(struct vm_area_struct *vma, - unsigned long start, unsigned long end) -{ - local_flush_tlb_all(); -} - -/* Flush a range of kernel pages */ -static inline void flush_tlb_kernel_range(unsigned long start, - unsigned long end) -{ - local_flush_tlb_all(); -} - -#define flush_tlb_mm(mm) flush_tlb_all() -#define flush_tlb_mm_range(mm, start, end, page_size) flush_tlb_all() -#endif /* !CONFIG_SMP || !CONFIG_MMU */ +#else /* CONFIG_MMU */ +#define local_flush_tlb_all() do { } while (0) +#endif /* CONFIG_MMU */ =20 #endif /* _ASM_RISCV_TLBFLUSH_H */ diff --git a/arch/riscv/mm/Makefile b/arch/riscv/mm/Makefile index 9c454f90fd3d..64f901674e35 100644 --- a/arch/riscv/mm/Makefile +++ b/arch/riscv/mm/Makefile @@ -13,15 +13,12 @@ endif KCOV_INSTRUMENT_init.o :=3D n =20 obj-y +=3D init.o -obj-$(CONFIG_MMU) +=3D extable.o fault.o pageattr.o +obj-$(CONFIG_MMU) +=3D extable.o fault.o pageattr.o tlbflush.o obj-y +=3D cacheflush.o obj-y +=3D context.o obj-y +=3D pgtable.o obj-y +=3D pmem.o =20 -ifeq ($(CONFIG_MMU),y) -obj-$(CONFIG_SMP) +=3D tlbflush.o -endif obj-$(CONFIG_HUGETLB_PAGE) +=3D hugetlbpage.o obj-$(CONFIG_PTDUMP_CORE) +=3D ptdump.o obj-$(CONFIG_KASAN) +=3D kasan_init.o diff --git a/arch/riscv/mm/tlbflush.c b/arch/riscv/mm/tlbflush.c index e6659d7368b3..22d7ed5abf8e 100644 --- a/arch/riscv/mm/tlbflush.c +++ b/arch/riscv/mm/tlbflush.c @@ -66,6 +66,7 @@ static inline void local_flush_tlb_range_asid(unsigned lo= ng start, local_flush_tlb_range_threshold_asid(start, size, stride, asid); } =20 +#ifdef CONFIG_SMP static void __ipi_flush_tlb_all(void *info) { local_flush_tlb_all(); @@ -138,6 +139,18 @@ static void __flush_tlb_range(struct mm_struct *mm, un= signed long start, if (mm) put_cpu(); } +#else +static void __flush_tlb_range(struct mm_struct *mm, unsigned long start, + unsigned long size, unsigned long stride) +{ + unsigned long asid =3D FLUSH_TLB_NO_ASID; + + if (mm && static_branch_unlikely(&use_asid_allocator)) + asid =3D atomic_long_read(&mm->context.id) & asid_mask; + + local_flush_tlb_range_asid(start, size, stride, asid); +} +#endif =20 void flush_tlb_mm(struct mm_struct *mm) { --=20 2.42.0 From nobody Tue Dec 16 20:00:44 2025 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 67A11C41535 for ; Sat, 28 Oct 2023 23:14:01 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229988AbjJ1XN4 (ORCPT ); Sat, 28 Oct 2023 19:13:56 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59612 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229950AbjJ1XNw (ORCPT ); Sat, 28 Oct 2023 19:13:52 -0400 Received: from mail-pj1-x1029.google.com (mail-pj1-x1029.google.com [IPv6:2607:f8b0:4864:20::1029]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8CDAAE4 for ; Sat, 28 Oct 2023 16:13:50 -0700 (PDT) Received: by mail-pj1-x1029.google.com with SMTP id 98e67ed59e1d1-2802b744e52so398849a91.0 for ; Sat, 28 Oct 2023 16:13:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1698534830; x=1699139630; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=UP8DHxv66mJzZKknfJMFXXbwrunx+Hl9Ii6vLteV5ng=; b=KZQvbv2t9KMJVwfFhixZqRILepp27m0ZVrsGf5MWiIeMNtNBTETPT6v845X4WvC6l0 58EZuLrqaScyacFKVDop2B6F7peZd3JKhnF9yEAQJrJQ2s8GgngZ69G2El/RTER080Jk J0HWwmS5ZEK7fETEUBmhsSJ3xSEJP7K5XQUOqA0s6nnSmZY9fDPgg5JYvxYUF2i30WSE oJKIzHm+QEVwn8fkHSnajlMALS2aOzcoQOVuejdUG+FpUJ6MbpXkh4jAob0WEC5+XNCm LlqZCtbfWcZGCJjOxlYq9d1W3oiHvbhqUHGsTEf4/tgQOYKiWpdqhPHdEG7ULrZt8wt2 miCA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1698534830; x=1699139630; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=UP8DHxv66mJzZKknfJMFXXbwrunx+Hl9Ii6vLteV5ng=; b=kiAR9Tp4tnpnXrqk7Q4ti4TMecvxDFm1jXdAzGcCuuefe3sP4T6WOoigFAxuzNQtnr E2k0WsreVhRiuCsxQO0qfGl5+dM8ecQUN6xJCIkOK/ycHdhmTDw+CCqJ5D1sCVaQGIVM o0t5Ez3vWsR/5R2q3R9dzPHS2FGtHEL8Wu3vM6gIITJx6dGkwLZSECDYJCGMTgG0F/Y2 ZIuiRuNaRDo5mTHYDg8d033QfSVaR9WpskNjBrm4kD0sQqOE9iFLtJn1AE/0/9D/kWkt LUnUZJ66cDWjQHWduGMht4GAfhmW2H94fx0VL7lm3dL8d8RQ+jQ//LLgOLs55VSGxvdM s3hA== X-Gm-Message-State: AOJu0YyN4f6Sdo+iR5M43hTkG+epgAstgteH+R6f7b/FwYnlbIVmWoKE 7H9b5VhKYhmfwRdeQ7BStzG7x7cQctsdjMnDphQ= X-Google-Smtp-Source: AGHT+IGnkbRXw2gehdqQUYovjjCjlvZO0f8kuIzIEbkVVgufqh5Fzfqz26Y3uN2f14MpzFpb+9CFvQ== X-Received: by 2002:a17:90a:1506:b0:280:2a3f:9938 with SMTP id l6-20020a17090a150600b002802a3f9938mr2800689pja.12.1698534830039; Sat, 28 Oct 2023 16:13:50 -0700 (PDT) Received: from sw06.internal.sifive.com ([4.53.31.132]) by smtp.gmail.com with ESMTPSA id u17-20020a17090341d100b001b8622c1ad2sm3679345ple.130.2023.10.28.16.13.49 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 28 Oct 2023 16:13:49 -0700 (PDT) From: Samuel Holland To: Palmer Dabbelt , Alexandre Ghiti , linux-riscv@lists.infradead.org Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, Samuel Holland Subject: [PATCH v2 06/11] riscv: Apply SiFive CIP-1200 workaround to single-ASID sfence.vma Date: Sat, 28 Oct 2023 16:12:04 -0700 Message-ID: <20231028231339.3116618-7-samuel.holland@sifive.com> X-Mailer: git-send-email 2.42.0 In-Reply-To: <20231028231339.3116618-1-samuel.holland@sifive.com> References: <20231028231339.3116618-1-samuel.holland@sifive.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" commit 3f1e782998cd ("riscv: add ASID-based tlbflushing methods") added calls to the sfence.vma instruction with rs2 !=3D x0. These single-ASID instruction variants are also affected by SiFive errata CIP-1200. Until now, the errata workaround was not needed for the single-ASID sfence.vma variants, because they were only used when the ASID allocator was enabled, and the affected SiFive platforms do not support multiple ASIDs. However, we are going to start using those sfence.vma variants regardless of ASID support, so now we need alternatives covering them. Signed-off-by: Samuel Holland --- Changes in v2: - Rebase on Alexandre's "riscv: tlb flush improvements" series v5 arch/riscv/include/asm/errata_list.h | 12 +++++++++++- arch/riscv/include/asm/tlbflush.h | 19 ++++++++++++++++++- arch/riscv/mm/tlbflush.c | 23 ----------------------- 3 files changed, 29 insertions(+), 25 deletions(-) diff --git a/arch/riscv/include/asm/errata_list.h b/arch/riscv/include/asm/= errata_list.h index b55b434f0059..d3f3c237adad 100644 --- a/arch/riscv/include/asm/errata_list.h +++ b/arch/riscv/include/asm/errata_list.h @@ -44,11 +44,21 @@ ALTERNATIVE(__stringify(RISCV_PTR do_page_fault), \ CONFIG_ERRATA_SIFIVE_CIP_453) #else /* !__ASSEMBLY__ */ =20 -#define ALT_FLUSH_TLB_PAGE(x) \ +#define ALT_SFENCE_VMA_ASID(asid) \ +asm(ALTERNATIVE("sfence.vma x0, %0", "sfence.vma", SIFIVE_VENDOR_ID, \ + ERRATA_SIFIVE_CIP_1200, CONFIG_ERRATA_SIFIVE_CIP_1200) \ + : : "r" (asid) : "memory") + +#define ALT_SFENCE_VMA_ADDR(addr) \ asm(ALTERNATIVE("sfence.vma %0", "sfence.vma", SIFIVE_VENDOR_ID, \ ERRATA_SIFIVE_CIP_1200, CONFIG_ERRATA_SIFIVE_CIP_1200) \ : : "r" (addr) : "memory") =20 +#define ALT_SFENCE_VMA_ADDR_ASID(addr, asid) \ +asm(ALTERNATIVE("sfence.vma %0, %1", "sfence.vma", SIFIVE_VENDOR_ID, \ + ERRATA_SIFIVE_CIP_1200, CONFIG_ERRATA_SIFIVE_CIP_1200) \ + : : "r" (addr), "r" (asid) : "memory") + /* * _val is marked as "will be overwritten", so need to set it to 0 * in the default case. diff --git a/arch/riscv/include/asm/tlbflush.h b/arch/riscv/include/asm/tlb= flush.h index 317a1811aa51..e529a643be17 100644 --- a/arch/riscv/include/asm/tlbflush.h +++ b/arch/riscv/include/asm/tlbflush.h @@ -22,10 +22,27 @@ static inline void local_flush_tlb_all(void) __asm__ __volatile__ ("sfence.vma" : : : "memory"); } =20 +static inline void local_flush_tlb_all_asid(unsigned long asid) +{ + if (asid !=3D FLUSH_TLB_NO_ASID) + ALT_SFENCE_VMA_ASID(asid); + else + local_flush_tlb_all(); +} + /* Flush one page from local TLB */ static inline void local_flush_tlb_page(unsigned long addr) { - ALT_FLUSH_TLB_PAGE(__asm__ __volatile__ ("sfence.vma %0" : : "r" (addr) := "memory")); + ALT_SFENCE_VMA_ADDR(addr); +} + +static inline void local_flush_tlb_page_asid(unsigned long addr, + unsigned long asid) +{ + if (asid !=3D FLUSH_TLB_NO_ASID) + ALT_SFENCE_VMA_ADDR_ASID(addr, asid); + else + local_flush_tlb_page(addr); } =20 #ifdef CONFIG_SMP diff --git a/arch/riscv/mm/tlbflush.c b/arch/riscv/mm/tlbflush.c index 22d7ed5abf8e..0feccb8932d2 100644 --- a/arch/riscv/mm/tlbflush.c +++ b/arch/riscv/mm/tlbflush.c @@ -7,29 +7,6 @@ #include #include =20 -static inline void local_flush_tlb_all_asid(unsigned long asid) -{ - if (asid !=3D FLUSH_TLB_NO_ASID) - __asm__ __volatile__ ("sfence.vma x0, %0" - : - : "r" (asid) - : "memory"); - else - local_flush_tlb_all(); -} - -static inline void local_flush_tlb_page_asid(unsigned long addr, - unsigned long asid) -{ - if (asid !=3D FLUSH_TLB_NO_ASID) - __asm__ __volatile__ ("sfence.vma %0, %1" - : - : "r" (addr), "r" (asid) - : "memory"); - else - local_flush_tlb_page(addr); -} - /* * Flush entire TLB if number of entries to be flushed is greater * than the threshold below. --=20 2.42.0 From nobody Tue Dec 16 20:00:44 2025 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id D7F5CC0032E for ; Sat, 28 Oct 2023 23:14:06 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230041AbjJ1XOG (ORCPT ); Sat, 28 Oct 2023 19:14:06 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59648 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229900AbjJ1XNy (ORCPT ); Sat, 28 Oct 2023 19:13:54 -0400 Received: from mail-pl1-x62c.google.com (mail-pl1-x62c.google.com [IPv6:2607:f8b0:4864:20::62c]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1042EEB for ; Sat, 28 Oct 2023 16:13:52 -0700 (PDT) Received: by mail-pl1-x62c.google.com with SMTP id d9443c01a7336-1cc3bb32b5dso3316135ad.3 for ; Sat, 28 Oct 2023 16:13:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1698534831; x=1699139631; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=uq0QdLYJN1f+TxJhAAAn9k5Zt34COvQfV/ogec/O0wM=; b=RLNomWrxly9VYMzEM4GBRfylLuRbEIJ8Ho4PcimecPGIdPZhk86Nw+X/aCUAXOeq9W DmG2IogMsnFv1U+D9Qm+fst/JKxmGkmUPs4COe7N2JbLZK6lnNgCgNuvhYTTlus/tGsk nt7PnBVaiNLsi9VUmLsMtv+8gwoA++eQns8583KdyUjEEOMXrdvUb7vvpGlAqIR/Q13A l7wJHRqR/aFPXmmsL8hqZsfoWGoGYj0sdWYmVQRK4STzq1UH/pXetiUrgg77OF21+EHW guSp+TYYItzM0PJCMM9c+ZeLQgobX7NbVPpWVJy/4/zDIMGeqAw6JmCux59sDpHKDlSr uy/A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1698534831; x=1699139631; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=uq0QdLYJN1f+TxJhAAAn9k5Zt34COvQfV/ogec/O0wM=; b=iH9vkOO/ztzj8IwJYzzwL1R8KYql3EzG02rea8eRYgqmxWcl7HB5pO4IRbqFNPKtSN Iu2JmOKaloOhv2v8BIV0ZtoiBud248R8QZdp4fpdN99Vo+v7zCHyojHs2PPl7LkCFniP qV8fN9ihnHTmNMVIxgl8p8CQX40zMQ6qUwhwhVFzxapj56ojGvciY9bWBsGm+bBBwkpM zum/l6uYz1NUy03eys/qQSnFCjc2N98io1kewH9T1VSra0pjjVgyFAAxkFnJSI5VT7JW XkMGctRKrTcb9NNKYdrdWTRJiWFPNxjRp7M1d8Y/hYRmc/R+TX5kmtU3yYj/y8pRHNzD cBQQ== X-Gm-Message-State: AOJu0YxcVt376+Kw25x2x5CzZzGGaNIHbQKxFppjh3JhzDd7TwGqQiin 8cibMafuZPNs/alnsrctyRsKQypBaTt7uIOgB6w= X-Google-Smtp-Source: AGHT+IGqJfNl+X6UXX6MsfY+UkH1b50Ad4rlkkQrptVRUTaYY/ZhDSmvjBpzXH88ANRSqVScovaqNA== X-Received: by 2002:a17:902:6b8c:b0:1c9:bf14:4f94 with SMTP id p12-20020a1709026b8c00b001c9bf144f94mr6530418plk.44.1698534831309; Sat, 28 Oct 2023 16:13:51 -0700 (PDT) Received: from sw06.internal.sifive.com ([4.53.31.132]) by smtp.gmail.com with ESMTPSA id u17-20020a17090341d100b001b8622c1ad2sm3679345ple.130.2023.10.28.16.13.50 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 28 Oct 2023 16:13:50 -0700 (PDT) From: Samuel Holland To: Palmer Dabbelt , Alexandre Ghiti , linux-riscv@lists.infradead.org Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, Samuel Holland Subject: [PATCH v2 07/11] riscv: mm: Introduce cntx2asid/cntx2version helper macros Date: Sat, 28 Oct 2023 16:12:05 -0700 Message-ID: <20231028231339.3116618-8-samuel.holland@sifive.com> X-Mailer: git-send-email 2.42.0 In-Reply-To: <20231028231339.3116618-1-samuel.holland@sifive.com> References: <20231028231339.3116618-1-samuel.holland@sifive.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" When using the ASID allocator, the MM context ID contains two values: the ASID in the lower bits, and the allocator version number in the remaining bits. Use macros to make this separation more obvious. Signed-off-by: Samuel Holland --- (no changes since v1) arch/riscv/include/asm/mmu.h | 3 +++ arch/riscv/mm/context.c | 12 ++++++------ arch/riscv/mm/tlbflush.c | 4 ++-- 3 files changed, 11 insertions(+), 8 deletions(-) diff --git a/arch/riscv/include/asm/mmu.h b/arch/riscv/include/asm/mmu.h index 355504b37f8e..a550fbf770be 100644 --- a/arch/riscv/include/asm/mmu.h +++ b/arch/riscv/include/asm/mmu.h @@ -26,6 +26,9 @@ typedef struct { #endif } mm_context_t; =20 +#define cntx2asid(cntx) ((cntx) & asid_mask) +#define cntx2version(cntx) ((cntx) & ~asid_mask) + void __init create_pgd_mapping(pgd_t *pgdp, uintptr_t va, phys_addr_t pa, phys_addr_t sz, pgprot_t prot); #endif /* __ASSEMBLY__ */ diff --git a/arch/riscv/mm/context.c b/arch/riscv/mm/context.c index 217fd4de6134..43d005f63253 100644 --- a/arch/riscv/mm/context.c +++ b/arch/riscv/mm/context.c @@ -81,7 +81,7 @@ static void __flush_context(void) if (cntx =3D=3D 0) cntx =3D per_cpu(reserved_context, i); =20 - __set_bit(cntx & asid_mask, context_asid_map); + __set_bit(cntx2asid(cntx), context_asid_map); per_cpu(reserved_context, i) =3D cntx; } =20 @@ -102,7 +102,7 @@ static unsigned long __new_context(struct mm_struct *mm) lockdep_assert_held(&context_lock); =20 if (cntx !=3D 0) { - unsigned long newcntx =3D ver | (cntx & asid_mask); + unsigned long newcntx =3D ver | cntx2asid(cntx); =20 /* * If our current CONTEXT was active during a rollover, we @@ -115,7 +115,7 @@ static unsigned long __new_context(struct mm_struct *mm) * We had a valid CONTEXT in a previous life, so try to * re-use it if possible. */ - if (!__test_and_set_bit(cntx & asid_mask, context_asid_map)) + if (!__test_and_set_bit(cntx2asid(cntx), context_asid_map)) return newcntx; } =20 @@ -168,7 +168,7 @@ static void set_mm_asid(struct mm_struct *mm, unsigned = int cpu) */ old_active_cntx =3D atomic_long_read(&per_cpu(active_context, cpu)); if (old_active_cntx && - ((cntx & ~asid_mask) =3D=3D atomic_long_read(¤t_version)) && + (cntx2version(cntx) =3D=3D atomic_long_read(¤t_version)) && atomic_long_cmpxchg_relaxed(&per_cpu(active_context, cpu), old_active_cntx, cntx)) goto switch_mm_fast; @@ -177,7 +177,7 @@ static void set_mm_asid(struct mm_struct *mm, unsigned = int cpu) =20 /* Check that our ASID belongs to the current_version. */ cntx =3D atomic_long_read(&mm->context.id); - if ((cntx & ~asid_mask) !=3D atomic_long_read(¤t_version)) { + if (cntx2version(cntx) !=3D atomic_long_read(¤t_version)) { cntx =3D __new_context(mm); atomic_long_set(&mm->context.id, cntx); } @@ -191,7 +191,7 @@ static void set_mm_asid(struct mm_struct *mm, unsigned = int cpu) =20 switch_mm_fast: csr_write(CSR_SATP, virt_to_pfn(mm->pgd) | - ((cntx & asid_mask) << SATP_ASID_SHIFT) | + (cntx2asid(cntx) << SATP_ASID_SHIFT) | satp_mode); =20 if (need_flush_tlb) diff --git a/arch/riscv/mm/tlbflush.c b/arch/riscv/mm/tlbflush.c index 0feccb8932d2..1cfac683bda4 100644 --- a/arch/riscv/mm/tlbflush.c +++ b/arch/riscv/mm/tlbflush.c @@ -91,7 +91,7 @@ static void __flush_tlb_range(struct mm_struct *mm, unsig= ned long start, broadcast =3D cpumask_any_but(cmask, cpuid) < nr_cpu_ids; =20 if (static_branch_unlikely(&use_asid_allocator)) - asid =3D atomic_long_read(&mm->context.id) & asid_mask; + asid =3D cntx2asid(atomic_long_read(&mm->context.id)); } else { cmask =3D cpu_online_mask; broadcast =3D true; @@ -123,7 +123,7 @@ static void __flush_tlb_range(struct mm_struct *mm, uns= igned long start, unsigned long asid =3D FLUSH_TLB_NO_ASID; =20 if (mm && static_branch_unlikely(&use_asid_allocator)) - asid =3D atomic_long_read(&mm->context.id) & asid_mask; + asid =3D cntx2asid(atomic_long_read(&mm->context.id)); =20 local_flush_tlb_range_asid(start, size, stride, asid); } --=20 2.42.0 From nobody Tue Dec 16 20:00:44 2025 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 69342C4167D for ; Sat, 28 Oct 2023 23:14:09 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230102AbjJ1XOJ (ORCPT ); Sat, 28 Oct 2023 19:14:09 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42480 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230003AbjJ1XN6 (ORCPT ); Sat, 28 Oct 2023 19:13:58 -0400 Received: from mail-ot1-x32b.google.com (mail-ot1-x32b.google.com [IPv6:2607:f8b0:4864:20::32b]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 692EEED for ; Sat, 28 Oct 2023 16:13:53 -0700 (PDT) Received: by mail-ot1-x32b.google.com with SMTP id 46e09a7af769-6c4cbab83aaso2241624a34.1 for ; Sat, 28 Oct 2023 16:13:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1698534832; x=1699139632; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=nfPkHia9ku1mIe8MIuwK1ts/4i7vxeqzbyNHZF4YvJg=; b=iHcRbx+mh/is2zOiLO1C8565EhG3yX4UN8thHJRx9oleH6+7wXkj7cW0KEpxZregGH BJWAGCmSvyFf+2K/Q2VbBh8Gy3o6ln4PljiE+FPSM4p2mSDjc+iP1EvFd56h09x7KeHV MLcjs/pStT9tR6EJmp6mWJvFWx4e1qr5ji9/SilbgdvDHhvfgr9VczOjTgm5eCP4t72K oYzpZpS/IRCZyIVQvjvHcv35icPLcYm2V7EaztsrKz9S9afogzMO9tgVfLrNc4Ftjlmo IuhahHPfFAIytTAjOFx/uugbSsuP4fn68zFHM0PI2RXFBPdqpfW6PsTlsLxYq0oh0gWw wJTA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1698534832; x=1699139632; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=nfPkHia9ku1mIe8MIuwK1ts/4i7vxeqzbyNHZF4YvJg=; b=e3zCOaSbtB9oQHhEi+pBGz5XQu/qtfKGVLrJYWN+geDetcrxKL9WNH+PEfxLWm8S8D 92pPr0eY0z+DwbKazIMCz9Is5UoxRGKzd1Ti42kGgyBomnR41vASjXrquMZO1REszbn5 sYYb5yqd4NjCwwcdYDC0dLfBMhqjEr66jHyuB7UWc3Qfs42nYPjov07RTli2rdmGIRV0 2WuOILfVyD6jNyqGxmwT6G0sYNhkFgOg7T2ySB/QSnlI+OPlKn8nh8Jk/FfdOvgnZ2y/ 649COM+l/WVjkPsfEkt9Dl7AwAR0BK3w42ijSiSysEl9wGefa0iLGaI4SbHHz0yKXH68 g82A== X-Gm-Message-State: AOJu0YznLrojlpQb1sXr3WeuXi/3t3uYBdO1pP+1EDVcDovS0kso/kpx 5Eufn1gTaDimjBgaU6ccNOLPDUpx4GZhrxsEtbc= X-Google-Smtp-Source: AGHT+IERITWcnpZawWSuuFpc/cjj/Gs86LMrNqv2CdnNpOWafKnnt4wsBVVnxZL60abLMgaX5a0ZCA== X-Received: by 2002:a05:6830:1e30:b0:6bd:c7c3:aac2 with SMTP id t16-20020a0568301e3000b006bdc7c3aac2mr6567117otr.18.1698534832734; Sat, 28 Oct 2023 16:13:52 -0700 (PDT) Received: from sw06.internal.sifive.com ([4.53.31.132]) by smtp.gmail.com with ESMTPSA id u17-20020a17090341d100b001b8622c1ad2sm3679345ple.130.2023.10.28.16.13.51 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 28 Oct 2023 16:13:52 -0700 (PDT) From: Samuel Holland To: Palmer Dabbelt , Alexandre Ghiti , linux-riscv@lists.infradead.org Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, Samuel Holland Subject: [PATCH v2 08/11] riscv: mm: Use a fixed layout for the MM context ID Date: Sat, 28 Oct 2023 16:12:06 -0700 Message-ID: <20231028231339.3116618-9-samuel.holland@sifive.com> X-Mailer: git-send-email 2.42.0 In-Reply-To: <20231028231339.3116618-1-samuel.holland@sifive.com> References: <20231028231339.3116618-1-samuel.holland@sifive.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Currently, the size of the ASID field in the MM context ID dynamically depends on the number of hardware-supported ASID bits. This requires reading a global variable to extract either field from the context ID. Instead, allocate the maximum possible number of bits to the ASID field, so the layout of the context ID is known at compile-time. Signed-off-by: Samuel Holland --- (no changes since v1) arch/riscv/include/asm/mmu.h | 4 ++-- arch/riscv/include/asm/tlbflush.h | 2 -- arch/riscv/mm/context.c | 6 ++---- 3 files changed, 4 insertions(+), 8 deletions(-) diff --git a/arch/riscv/include/asm/mmu.h b/arch/riscv/include/asm/mmu.h index a550fbf770be..dc0273f7905f 100644 --- a/arch/riscv/include/asm/mmu.h +++ b/arch/riscv/include/asm/mmu.h @@ -26,8 +26,8 @@ typedef struct { #endif } mm_context_t; =20 -#define cntx2asid(cntx) ((cntx) & asid_mask) -#define cntx2version(cntx) ((cntx) & ~asid_mask) +#define cntx2asid(cntx) ((cntx) & SATP_ASID_MASK) +#define cntx2version(cntx) ((cntx) & ~SATP_ASID_MASK) =20 void __init create_pgd_mapping(pgd_t *pgdp, uintptr_t va, phys_addr_t pa, phys_addr_t sz, pgprot_t prot); diff --git a/arch/riscv/include/asm/tlbflush.h b/arch/riscv/include/asm/tlb= flush.h index e529a643be17..62d780037169 100644 --- a/arch/riscv/include/asm/tlbflush.h +++ b/arch/riscv/include/asm/tlbflush.h @@ -15,8 +15,6 @@ #define FLUSH_TLB_NO_ASID ((unsigned long)-1) =20 #ifdef CONFIG_MMU -extern unsigned long asid_mask; - static inline void local_flush_tlb_all(void) { __asm__ __volatile__ ("sfence.vma" : : : "memory"); diff --git a/arch/riscv/mm/context.c b/arch/riscv/mm/context.c index 43d005f63253..b5170ac1b742 100644 --- a/arch/riscv/mm/context.c +++ b/arch/riscv/mm/context.c @@ -22,7 +22,6 @@ DEFINE_STATIC_KEY_FALSE(use_asid_allocator); =20 static unsigned long asid_bits; static unsigned long num_asids; -unsigned long asid_mask; =20 static atomic_long_t current_version; =20 @@ -128,7 +127,7 @@ static unsigned long __new_context(struct mm_struct *mm) goto set_asid; =20 /* We're out of ASIDs, so increment current_version */ - ver =3D atomic_long_add_return_relaxed(num_asids, ¤t_version); + ver =3D atomic_long_add_return_relaxed(BIT(SATP_ASID_BITS), ¤t_vers= ion); =20 /* Flush everything */ __flush_context(); @@ -247,7 +246,6 @@ static int __init asids_init(void) /* Pre-compute ASID details */ if (asid_bits) { num_asids =3D 1 << asid_bits; - asid_mask =3D num_asids - 1; } =20 /* @@ -255,7 +253,7 @@ static int __init asids_init(void) * at-least twice more than CPUs */ if (num_asids > (2 * num_possible_cpus())) { - atomic_long_set(¤t_version, num_asids); + atomic_long_set(¤t_version, BIT(SATP_ASID_BITS)); =20 context_asid_map =3D bitmap_zalloc(num_asids, GFP_KERNEL); if (!context_asid_map) --=20 2.42.0 From nobody Tue Dec 16 20:00:44 2025 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id DDD0BC4167B for ; Sat, 28 Oct 2023 23:14:14 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230158AbjJ1XOO (ORCPT ); Sat, 28 Oct 2023 19:14:14 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42522 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229929AbjJ1XN7 (ORCPT ); Sat, 28 Oct 2023 19:13:59 -0400 Received: from mail-pl1-x62e.google.com (mail-pl1-x62e.google.com [IPv6:2607:f8b0:4864:20::62e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 97110FA for ; Sat, 28 Oct 2023 16:13:54 -0700 (PDT) Received: by mail-pl1-x62e.google.com with SMTP id d9443c01a7336-1cc3bb32b5dso3316305ad.3 for ; Sat, 28 Oct 2023 16:13:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1698534834; x=1699139634; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=O8uQuAx5HbeGrqX4Ip/qQY7nv6lrxw4qacCJFbpKbko=; b=MmvkdMknLmTQClsjK8ugUVtqsyQmg6tVO/1w4J4VNZatWQs1MmVKK64hnOkzO21W// E1VkGeV2KWBlSEHmofbKNvp45//7TViS/QOFU8H5bJuRGRHtA2qdwXG/PFPs8aUy6BZp DREM4SCUqKv75EY3biEfCXGGjsewZoUJtklvl5QDDm1TWNI/jKM/0Kj8ClaM7dIWaiyv bILXd/eBfWblnwFScA7JHDxP5fWB9AfykeYDUdl9oSqTECQXg7k6Lu3f5VH8zD9LA/MW i8R6PWwSbdIEv7yVww+JTfQVTtVuc+XVRHxTgwWySboz+uqLlZxgqgsShzJZAQMAXKWi NxkA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1698534834; x=1699139634; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=O8uQuAx5HbeGrqX4Ip/qQY7nv6lrxw4qacCJFbpKbko=; b=HycajA64lMZKoyh+vP7GxH0lhHF/5wHEoTsG4N3v2W3PaZlVMwCaLhu+5uF/g4Csn6 im1OjZMQMui8XJSJyDkhBw8jzAxnkyiCJw47NvS7qmwjdaXz/fPKPaNiGOV6knQcd/kk qbidbafSmyitq9fOUoUNnVwjJQ29AOL6K0LU3H2BjLg7iKae1cB4mpAWKvoTLOxWhDPC WnjjHRYuYtXmUbBraZ3cJIPYuDuvwerfk435DAzS5aICJ4oEv00NiIpEnkgA+Tvhcvuy 4+koszpuIyhpxVywSud1dehaXiDjvbcWXZDEPsrnOSxH4IMlmVmCk0qmla8FI5nw4449 0NoA== X-Gm-Message-State: AOJu0YydTsaNld1ZSrdOIPArStVyCLjC+nspry7YYMWfPkbaX63WaQq/ MA/dkDSwfSZnBUcqaOZhBHDeTQ== X-Google-Smtp-Source: AGHT+IEXxDKHTizsqyBXjr5oqt/mWPSp5qFgTjueov73uPEcJh4MjB79gaQaqVj5Le1ZHH1qO9h3pA== X-Received: by 2002:a17:902:c751:b0:1cc:29ef:df7d with SMTP id q17-20020a170902c75100b001cc29efdf7dmr3951181plq.65.1698534834011; Sat, 28 Oct 2023 16:13:54 -0700 (PDT) Received: from sw06.internal.sifive.com ([4.53.31.132]) by smtp.gmail.com with ESMTPSA id u17-20020a17090341d100b001b8622c1ad2sm3679345ple.130.2023.10.28.16.13.53 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 28 Oct 2023 16:13:53 -0700 (PDT) From: Samuel Holland To: Palmer Dabbelt , Alexandre Ghiti , linux-riscv@lists.infradead.org Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, Samuel Holland Subject: [PATCH v2 09/11] riscv: mm: Make asid_bits a local variable Date: Sat, 28 Oct 2023 16:12:07 -0700 Message-ID: <20231028231339.3116618-10-samuel.holland@sifive.com> X-Mailer: git-send-email 2.42.0 In-Reply-To: <20231028231339.3116618-1-samuel.holland@sifive.com> References: <20231028231339.3116618-1-samuel.holland@sifive.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" This variable is only used inside asids_init(). Signed-off-by: Samuel Holland --- (no changes since v1) arch/riscv/mm/context.c | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/arch/riscv/mm/context.c b/arch/riscv/mm/context.c index b5170ac1b742..43a8bc2d5af4 100644 --- a/arch/riscv/mm/context.c +++ b/arch/riscv/mm/context.c @@ -20,7 +20,6 @@ =20 DEFINE_STATIC_KEY_FALSE(use_asid_allocator); =20 -static unsigned long asid_bits; static unsigned long num_asids; =20 static atomic_long_t current_version; @@ -226,7 +225,7 @@ static inline void set_mm(struct mm_struct *prev, =20 static int __init asids_init(void) { - unsigned long old; + unsigned long asid_bits, old; =20 /* Figure-out number of ASID bits in HW */ old =3D csr_read(CSR_SATP); --=20 2.42.0 From nobody Tue Dec 16 20:00:44 2025 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 128C6C4332F for ; Sat, 28 Oct 2023 23:14:17 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230053AbjJ1XOQ (ORCPT ); Sat, 28 Oct 2023 19:14:16 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42576 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230081AbjJ1XOA (ORCPT ); Sat, 28 Oct 2023 19:14:00 -0400 Received: from mail-oa1-x2d.google.com (mail-oa1-x2d.google.com [IPv6:2001:4860:4864:20::2d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C89A8191 for ; Sat, 28 Oct 2023 16:13:56 -0700 (PDT) Received: by mail-oa1-x2d.google.com with SMTP id 586e51a60fabf-1ea4a3d0a44so2219336fac.1 for ; Sat, 28 Oct 2023 16:13:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1698534835; x=1699139635; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=MROdnZxFJAn7brM6ye/jIKPJcvVCbRqlgsjUSxtP0PY=; b=McM3n4oUqzaDTF5lv19KaayaUI9Y2y0FIeMtPE5+3B5VNGAmDxra6khHUcIZMLiWYx /cXpsmsAMrPSCGJP9u4T6ZhQ5ztdqDQ4AMcp0fVelgtBdUdqQaHxHU3IEJlymdJZt1qI B0QkLQFpNCu92n+pQ5RlbE+rk/mMuzQJJDzziF4YgST7qN5gmXCV3tDFZGl54VoW0G/r hx5UpLSUjbRswr/znEvQyoSiPnQ3OZhbA+K0IUfTJsTZWPgLfj8nJR0X+GOkl++t0DTK ud5KhNFzWfDdGiIm6fVXNmwZnQLlEhR7t90a66fai4CWu3aBKgm5n909vYvtu9QdVphs OLyg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1698534835; x=1699139635; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=MROdnZxFJAn7brM6ye/jIKPJcvVCbRqlgsjUSxtP0PY=; b=XHnOdRNGLELzd0IkZlTnOnxEMSh+l99/q+tqDv+kNsqHR5Z1YAaU5L4UdkPgNdZqQV xSHe+jcrBaxOkCyZCdSuAU/iei5YXpzazij4bbwdMH8bkW8ES2dFbuLtf2c0yrQ/QSXz DlmvV7oZxcTEHYgFyLVw5wAR4b8KSX52sZrVMqMiNSYUc0Zn2fkM7QhRGj8HuPE77CXW 0yFhuw998Pl97B9ldJwGVH8QZJyhEb23AuGi4C5tpXqgYnhQOFnfe/SN7/qqClDlVbLL XUDYTQ/pjsEQPxVjPlolCnerC128MHWEWtMy2CUwKmvuepcFA/uDaVwrlqubE31DXVJ4 CIIw== X-Gm-Message-State: AOJu0YwTAXJSGDQsIzyIu+a1yYqDn8onvtG9BZil1EcCH2k9OwvRjOKh f4xiRoun50rBzgJZcBqOQOaRdg== X-Google-Smtp-Source: AGHT+IEpjr2TCXm5am55hbUBmsnjcw94CHvXcnlavbn7wyDrZYS6wclRzO1RWsfOFMjAji60OjxQDg== X-Received: by 2002:a05:6358:5e08:b0:168:d6cd:7b2e with SMTP id q8-20020a0563585e0800b00168d6cd7b2emr6151237rwn.29.1698534835540; Sat, 28 Oct 2023 16:13:55 -0700 (PDT) Received: from sw06.internal.sifive.com ([4.53.31.132]) by smtp.gmail.com with ESMTPSA id u17-20020a17090341d100b001b8622c1ad2sm3679345ple.130.2023.10.28.16.13.54 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 28 Oct 2023 16:13:54 -0700 (PDT) From: Samuel Holland To: Palmer Dabbelt , Alexandre Ghiti , linux-riscv@lists.infradead.org Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, Samuel Holland Subject: [PATCH v2 10/11] riscv: mm: Preserve global TLB entries when switching contexts Date: Sat, 28 Oct 2023 16:12:08 -0700 Message-ID: <20231028231339.3116618-11-samuel.holland@sifive.com> X-Mailer: git-send-email 2.42.0 In-Reply-To: <20231028231339.3116618-1-samuel.holland@sifive.com> References: <20231028231339.3116618-1-samuel.holland@sifive.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" If the CPU does not support multiple ASIDs, all MM contexts use ASID 0. In this case, it is still beneficial to flush the TLB by ASID, as the single-ASID variant of the sfence.vma instruction preserves TLB entries for global (kernel) pages. This optimization is recommended by the RISC-V privileged specification: If the implementation does not provide ASIDs, or software chooses to always use ASID 0, then after every satp write, software should execute SFENCE.VMA with rs1=3Dx0. In the common case that no global translations have been modified, rs2 should be set to a register other than x0 but which contains the value zero, so that global translations are not flushed. It is not possible to apply this optimization when using the ASID allocator, because that code must flush the TLB for all ASIDs at once when incrementing the version number. Signed-off-by: Samuel Holland --- (no changes since v1) arch/riscv/mm/context.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/arch/riscv/mm/context.c b/arch/riscv/mm/context.c index 43a8bc2d5af4..3ca9b653df7d 100644 --- a/arch/riscv/mm/context.c +++ b/arch/riscv/mm/context.c @@ -200,7 +200,7 @@ static void set_mm_noasid(struct mm_struct *mm) { /* Switch the page table and blindly nuke entire local TLB */ csr_write(CSR_SATP, virt_to_pfn(mm->pgd) | satp_mode); - local_flush_tlb_all(); + local_flush_tlb_all_asid(0); } =20 static inline void set_mm(struct mm_struct *prev, --=20 2.42.0 From nobody Tue Dec 16 20:00:44 2025 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id C740BC4332F for ; Sat, 28 Oct 2023 23:14:20 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229750AbjJ1XOU (ORCPT ); Sat, 28 Oct 2023 19:14:20 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42530 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230122AbjJ1XON (ORCPT ); Sat, 28 Oct 2023 19:14:13 -0400 Received: from mail-pl1-x636.google.com (mail-pl1-x636.google.com [IPv6:2607:f8b0:4864:20::636]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 915571B1 for ; Sat, 28 Oct 2023 16:13:57 -0700 (PDT) Received: by mail-pl1-x636.google.com with SMTP id d9443c01a7336-1cc1ee2d8dfso17082325ad.3 for ; Sat, 28 Oct 2023 16:13:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1698534837; x=1699139637; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=RYq2JBnvfDkZMtFq+gFMcak9lyRAM53Oqx2LdQZI3vI=; b=gEZkrP2YiX4vVYOeeXB7ZtljVqPEl/maZqiQL1ssgjFayGJpuXCPYIAqziTMfgAUzS 3cBkzMHmea475AZSzmKa6imwcEd0EpMd+O/H0NS1DK2PP+WUYaIAfXDO1QHrBgtw6prn VxCIrw4o0+plAADTIUEwUv1J9UMKleFWxAhdm/S7tewX0RZSQ5PLZktGAhA1LSHuOyrN spKthjTJp1fwm/tZWO27TKIsK50Ffw4mjsyav8Q6AvPwxQY2OROMsfYXl3oaCQ+yrK/H 6BG3AJropXRvA3ApD5tgRK8sin7gfPYSm3RCZgQ4V5ZV+IB102mTSXGruKeaAt4ZCeuT i/ag== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1698534837; x=1699139637; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=RYq2JBnvfDkZMtFq+gFMcak9lyRAM53Oqx2LdQZI3vI=; b=jc3zzIS8B6/Yfyz7EihbRFYMlkBk0Uzk1toh8OWUyWbGU7iX9/u4KH7dFtOnJozK/4 bMGT1c3t7P6oBur2rexjpJAcJF8OaRVq99InDDOMRZzpO6Qn45IyiIxhz/krgOdNm80a 1PHShUxjbfGDmajtUCueV+d6PrYNBhsaDfT5Yy0UhU0W8RCJJVUrWpTSoyNAYT/50l6X fZl8OjriNSkvvHwpQNs/Au7ubchraRVDLFllKJFyBQDIO84YOuzL27CdeyQY//rJwf/D QojQBIwXsPq7mbmw70bTJxShB22iGGBlEX+mPJByaMzm2is6cQf1pilQ8wo6PNjUlI7E 5K6Q== X-Gm-Message-State: AOJu0Ywyio2vQL7ib9tlTaRBZ3m/IRRxFGFXtnPRlne8/gma1TSUC75A FsvUTu8tLnzIhjPA1JE7Q44A2g== X-Google-Smtp-Source: AGHT+IHR5gJkl+2hmALdgeC6Mspb2OIhkdCFo9rAzDT7R89li8y2sxi8zbkov4fwyI0EL6o8dg+K/w== X-Received: by 2002:a17:902:f542:b0:1c5:a7b7:291c with SMTP id h2-20020a170902f54200b001c5a7b7291cmr8203243plf.12.1698534836728; Sat, 28 Oct 2023 16:13:56 -0700 (PDT) Received: from sw06.internal.sifive.com ([4.53.31.132]) by smtp.gmail.com with ESMTPSA id u17-20020a17090341d100b001b8622c1ad2sm3679345ple.130.2023.10.28.16.13.55 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 28 Oct 2023 16:13:56 -0700 (PDT) From: Samuel Holland To: Palmer Dabbelt , Alexandre Ghiti , linux-riscv@lists.infradead.org Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, Samuel Holland Subject: [PATCH v2 11/11] riscv: mm: Always use ASID to flush MM contexts Date: Sat, 28 Oct 2023 16:12:09 -0700 Message-ID: <20231028231339.3116618-12-samuel.holland@sifive.com> X-Mailer: git-send-email 2.42.0 In-Reply-To: <20231028231339.3116618-1-samuel.holland@sifive.com> References: <20231028231339.3116618-1-samuel.holland@sifive.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Even if multiple ASIDs are not supported, using the single-ASID variant of the sfence.vma instruction preserves TLB entries for global (kernel) pages. So it is always most efficient to use the single-ASID code path. Signed-off-by: Samuel Holland --- Changes in v2: - Update both copies of __flush_tlb_range() arch/riscv/include/asm/mmu_context.h | 2 -- arch/riscv/mm/context.c | 3 +-- arch/riscv/mm/tlbflush.c | 5 ++--- 3 files changed, 3 insertions(+), 7 deletions(-) diff --git a/arch/riscv/include/asm/mmu_context.h b/arch/riscv/include/asm/= mmu_context.h index 7030837adc1a..b0659413a080 100644 --- a/arch/riscv/include/asm/mmu_context.h +++ b/arch/riscv/include/asm/mmu_context.h @@ -33,8 +33,6 @@ static inline int init_new_context(struct task_struct *ts= k, return 0; } =20 -DECLARE_STATIC_KEY_FALSE(use_asid_allocator); - #include =20 #endif /* _ASM_RISCV_MMU_CONTEXT_H */ diff --git a/arch/riscv/mm/context.c b/arch/riscv/mm/context.c index 3ca9b653df7d..20057085ab8a 100644 --- a/arch/riscv/mm/context.c +++ b/arch/riscv/mm/context.c @@ -18,8 +18,7 @@ =20 #ifdef CONFIG_MMU =20 -DEFINE_STATIC_KEY_FALSE(use_asid_allocator); - +static DEFINE_STATIC_KEY_FALSE(use_asid_allocator); static unsigned long num_asids; =20 static atomic_long_t current_version; diff --git a/arch/riscv/mm/tlbflush.c b/arch/riscv/mm/tlbflush.c index 1cfac683bda4..9d06a3e9d330 100644 --- a/arch/riscv/mm/tlbflush.c +++ b/arch/riscv/mm/tlbflush.c @@ -90,8 +90,7 @@ static void __flush_tlb_range(struct mm_struct *mm, unsig= ned long start, /* check if the tlbflush needs to be sent to other CPUs */ broadcast =3D cpumask_any_but(cmask, cpuid) < nr_cpu_ids; =20 - if (static_branch_unlikely(&use_asid_allocator)) - asid =3D cntx2asid(atomic_long_read(&mm->context.id)); + asid =3D cntx2asid(atomic_long_read(&mm->context.id)); } else { cmask =3D cpu_online_mask; broadcast =3D true; @@ -122,7 +121,7 @@ static void __flush_tlb_range(struct mm_struct *mm, uns= igned long start, { unsigned long asid =3D FLUSH_TLB_NO_ASID; =20 - if (mm && static_branch_unlikely(&use_asid_allocator)) + if (mm) asid =3D cntx2asid(atomic_long_read(&mm->context.id)); =20 local_flush_tlb_range_asid(start, size, stride, asid); --=20 2.42.0