From nobody Tue Dec 16 13:33:09 2025 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7778AC4167D for ; Fri, 10 Nov 2023 19:21:53 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1346058AbjKJTVx (ORCPT ); Fri, 10 Nov 2023 14:21:53 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36348 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236053AbjKJTVY (ORCPT ); Fri, 10 Nov 2023 14:21:24 -0500 Received: from mail-lf1-x12f.google.com (mail-lf1-x12f.google.com [IPv6:2a00:1450:4864:20::12f]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6EDA1387A6 for ; Fri, 10 Nov 2023 06:08:26 -0800 (PST) Received: by mail-lf1-x12f.google.com with SMTP id 2adb3069b0e04-50970c2115eso2768161e87.1 for ; Fri, 10 Nov 2023 06:08:26 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1699625304; x=1700230104; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=2AMKJOgOBU+36iIxGMD9Trqw8IVdjZaTyhAA3F26Egw=; b=eIA9nXW68Ua6zGvVP5U5NOXpaDjO2/YtcWXNVmMnWluJ5dinohTSQVYrvSVq/AERyJ VLLtZ6FSi5CrGaylCvMbBKm/xO2deR6pAgvMIc5DuUz+IQaO0xSM/YSELK98QHQc0phT Uq5NB5GGZCmvI92fNAB9bA2ARsPfbmcW35FIRKf58IoVCc4r+6iYuxTv+GxCaILALaj5 OqJ7FOoZEFGxlF59C3Py9QKoCIk1IRYae4u8XfpwuRE1FSrMVFmqLpltnQnwo46Gv6AU OMC/EVwoQ/vz84S89+wUJWTJk7wC+M8xg4T1ZEu4GdIx8jZNK/NgN1If8Pi/gdVJhxn2 q0Ag== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1699625304; x=1700230104; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=2AMKJOgOBU+36iIxGMD9Trqw8IVdjZaTyhAA3F26Egw=; b=qjaQ7f4sUQyi6HR7YD3/N4NvoDDLbIsACmsTw6mQtfaRCUJ8rvQkrW45dFKoZz4GTb FqQnUPd6HIzl8xmUGKHr1l2ZjIa9E5wr3PfxHqjWHNA9oWesK/5HA9gNHaXK6stI3ob/ 1lTtBsYnC8J3UZkvRjiGTCYWU3ViEK8Os7TZvb0NNcl/zRD3fy6UWtn5jSujI2xLVBkN qMSN8eOSC0fhJOe+n0GamxcT/KzvGmQyHvx+4XNFVrI+Vs8U1pXv2vzbpgYpk1dHMKTF zbjMRn7jLgP65jlAbbImTuYG74oZFZWmcncJeyzz91QE4U0yj0tOQgeIut2TPhvL7zkb QnjQ== X-Gm-Message-State: AOJu0YyWOm+Eoi8rrryybqgaqEeB39OEMjy0k1HAXWOc3JWfpz2mxI7B OAEWS9685Gd/i1XArN3m/bX2TA== X-Google-Smtp-Source: AGHT+IEGn96XLIR0ZFAsjYB2eJfL/m69l1crDYV4Ee9Qc9cnFj8KCnXUpx20enm/StvHsssXdR1xaw== X-Received: by 2002:ac2:5209:0:b0:503:1783:d5a9 with SMTP id a9-20020ac25209000000b005031783d5a9mr3895555lfl.3.1699625304483; Fri, 10 Nov 2023 06:08:24 -0800 (PST) Received: from alex-rivos.home (amontpellier-656-1-456-62.w92-145.abo.wanadoo.fr. [92.145.124.62]) by smtp.gmail.com with ESMTPSA id t18-20020a05600c451200b004076f522058sm5366438wmo.0.2023.11.10.06.08.23 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 10 Nov 2023 06:08:24 -0800 (PST) From: Alexandre Ghiti To: Paul Walmsley , Palmer Dabbelt , Albert Ou , Andrey Ryabinin , Alexander Potapenko , Andrey Konovalov , Dmitry Vyukov , Vincenzo Frascino , Arnd Bergmann , Dennis Zhou , Tejun Heo , Christoph Lameter , Andrew Morton , linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org, kasan-dev@googlegroups.com, linux-arch@vger.kernel.org, linux-mm@kvack.org Cc: Alexandre Ghiti Subject: [PATCH 1/2] mm: Introduce flush_cache_vmap_early() and its riscv implementation Date: Fri, 10 Nov 2023 15:07:20 +0100 Message-Id: <20231110140721.114235-2-alexghiti@rivosinc.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20231110140721.114235-1-alexghiti@rivosinc.com> References: <20231110140721.114235-1-alexghiti@rivosinc.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" The pcpu setup when using the page allocator sets up a new vmalloc mapping very early in the boot process, so early that it cannot use the flush_cache_vmap() function which may depend on structures not yet initialized (for example in riscv, we currently send an IPI to flush other cpus TLB). But on some architectures, we must call flush_cache_vmap(): for example, in riscv, some uarchs can cache invalid TLB entries so we need to flush the new established mapping to avoid taking an exception. So fix this by introducing a new function flush_cache_vmap_early() which is called right after setting the new page table entry and before accessing this new mapping. This new function implements a local flush tlb on riscv and is no-op for other architectures (same as today). Signed-off-by: Alexandre Ghiti --- arch/riscv/include/asm/cacheflush.h | 3 ++- arch/riscv/include/asm/tlbflush.h | 2 ++ arch/riscv/mm/tlbflush.c | 5 +++++ include/asm-generic/cacheflush.h | 6 ++++++ mm/percpu.c | 8 +------- 5 files changed, 16 insertions(+), 8 deletions(-) diff --git a/arch/riscv/include/asm/cacheflush.h b/arch/riscv/include/asm/c= acheflush.h index 3cb53c4df27c..a129dac4521d 100644 --- a/arch/riscv/include/asm/cacheflush.h +++ b/arch/riscv/include/asm/cacheflush.h @@ -37,7 +37,8 @@ static inline void flush_dcache_page(struct page *page) flush_icache_mm(vma->vm_mm, 0) =20 #ifdef CONFIG_64BIT -#define flush_cache_vmap(start, end) flush_tlb_kernel_range(start, end) +#define flush_cache_vmap(start, end) flush_tlb_kernel_range(start, end) +#define flush_cache_vmap_early(start, end) local_flush_tlb_kernel_range(st= art, end) #endif =20 #ifndef CONFIG_SMP diff --git a/arch/riscv/include/asm/tlbflush.h b/arch/riscv/include/asm/tlb= flush.h index 8f3418c5f172..f0d6328076b6 100644 --- a/arch/riscv/include/asm/tlbflush.h +++ b/arch/riscv/include/asm/tlbflush.h @@ -41,6 +41,7 @@ void flush_tlb_page(struct vm_area_struct *vma, unsigned = long addr); void flush_tlb_range(struct vm_area_struct *vma, unsigned long start, unsigned long end); void flush_tlb_kernel_range(unsigned long start, unsigned long end); +void local_flush_tlb_kernel_range(unsigned long start, unsigned long end); #ifdef CONFIG_TRANSPARENT_HUGEPAGE #define __HAVE_ARCH_FLUSH_PMD_TLB_RANGE void flush_pmd_tlb_range(struct vm_area_struct *vma, unsigned long start, @@ -64,6 +65,7 @@ static inline void flush_tlb_kernel_range(unsigned long s= tart, local_flush_tlb_all(); } =20 +#define local_flush_tlb_kernel_range(start, end) flush_tlb_kernel_range(st= art, end) #define flush_tlb_mm(mm) flush_tlb_all() #define flush_tlb_mm_range(mm, start, end, page_size) flush_tlb_all() #endif /* !CONFIG_SMP || !CONFIG_MMU */ diff --git a/arch/riscv/mm/tlbflush.c b/arch/riscv/mm/tlbflush.c index e6659d7368b3..8aadc5f71c93 100644 --- a/arch/riscv/mm/tlbflush.c +++ b/arch/riscv/mm/tlbflush.c @@ -66,6 +66,11 @@ static inline void local_flush_tlb_range_asid(unsigned l= ong start, local_flush_tlb_range_threshold_asid(start, size, stride, asid); } =20 +void local_flush_tlb_kernel_range(unsigned long start, unsigned long end) +{ + local_flush_tlb_range_asid(start, end, PAGE_SIZE, FLUSH_TLB_NO_ASID); +} + static void __ipi_flush_tlb_all(void *info) { local_flush_tlb_all(); diff --git a/include/asm-generic/cacheflush.h b/include/asm-generic/cachefl= ush.h index 84ec53ccc450..7ee8a179d103 100644 --- a/include/asm-generic/cacheflush.h +++ b/include/asm-generic/cacheflush.h @@ -91,6 +91,12 @@ static inline void flush_cache_vmap(unsigned long start,= unsigned long end) } #endif =20 +#ifndef flush_cache_vmap_early +static inline void flush_cache_vmap_early(unsigned long start, unsigned lo= ng end) +{ +} +#endif + #ifndef flush_cache_vunmap static inline void flush_cache_vunmap(unsigned long start, unsigned long e= nd) { diff --git a/mm/percpu.c b/mm/percpu.c index a7665de8485f..d287cebd58ca 100644 --- a/mm/percpu.c +++ b/mm/percpu.c @@ -3306,13 +3306,7 @@ int __init pcpu_page_first_chunk(size_t reserved_siz= e, pcpu_fc_cpu_to_node_fn_t if (rc < 0) panic("failed to map percpu area, err=3D%d\n", rc); =20 - /* - * FIXME: Archs with virtual cache should flush local - * cache for the linear mapping here - something - * equivalent to flush_cache_vmap() on the local cpu. - * flush_cache_vmap() can't be used as most supporting - * data structures are not set up yet. - */ + flush_cache_vmap_early(unit_addr, unit_addr + ai->unit_size); =20 /* copy static data */ memcpy((void *)unit_addr, __per_cpu_load, ai->static_size); --=20 2.39.2