From nobody Tue Dec 16 05:44:36 2025 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7778AC4167D for ; Fri, 10 Nov 2023 19:21:53 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1346058AbjKJTVx (ORCPT ); Fri, 10 Nov 2023 14:21:53 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36348 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236053AbjKJTVY (ORCPT ); Fri, 10 Nov 2023 14:21:24 -0500 Received: from mail-lf1-x12f.google.com (mail-lf1-x12f.google.com [IPv6:2a00:1450:4864:20::12f]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6EDA1387A6 for ; Fri, 10 Nov 2023 06:08:26 -0800 (PST) Received: by mail-lf1-x12f.google.com with SMTP id 2adb3069b0e04-50970c2115eso2768161e87.1 for ; Fri, 10 Nov 2023 06:08:26 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1699625304; x=1700230104; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=2AMKJOgOBU+36iIxGMD9Trqw8IVdjZaTyhAA3F26Egw=; b=eIA9nXW68Ua6zGvVP5U5NOXpaDjO2/YtcWXNVmMnWluJ5dinohTSQVYrvSVq/AERyJ VLLtZ6FSi5CrGaylCvMbBKm/xO2deR6pAgvMIc5DuUz+IQaO0xSM/YSELK98QHQc0phT Uq5NB5GGZCmvI92fNAB9bA2ARsPfbmcW35FIRKf58IoVCc4r+6iYuxTv+GxCaILALaj5 OqJ7FOoZEFGxlF59C3Py9QKoCIk1IRYae4u8XfpwuRE1FSrMVFmqLpltnQnwo46Gv6AU OMC/EVwoQ/vz84S89+wUJWTJk7wC+M8xg4T1ZEu4GdIx8jZNK/NgN1If8Pi/gdVJhxn2 q0Ag== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1699625304; x=1700230104; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=2AMKJOgOBU+36iIxGMD9Trqw8IVdjZaTyhAA3F26Egw=; b=qjaQ7f4sUQyi6HR7YD3/N4NvoDDLbIsACmsTw6mQtfaRCUJ8rvQkrW45dFKoZz4GTb FqQnUPd6HIzl8xmUGKHr1l2ZjIa9E5wr3PfxHqjWHNA9oWesK/5HA9gNHaXK6stI3ob/ 1lTtBsYnC8J3UZkvRjiGTCYWU3ViEK8Os7TZvb0NNcl/zRD3fy6UWtn5jSujI2xLVBkN qMSN8eOSC0fhJOe+n0GamxcT/KzvGmQyHvx+4XNFVrI+Vs8U1pXv2vzbpgYpk1dHMKTF zbjMRn7jLgP65jlAbbImTuYG74oZFZWmcncJeyzz91QE4U0yj0tOQgeIut2TPhvL7zkb QnjQ== X-Gm-Message-State: AOJu0YyWOm+Eoi8rrryybqgaqEeB39OEMjy0k1HAXWOc3JWfpz2mxI7B OAEWS9685Gd/i1XArN3m/bX2TA== X-Google-Smtp-Source: AGHT+IEGn96XLIR0ZFAsjYB2eJfL/m69l1crDYV4Ee9Qc9cnFj8KCnXUpx20enm/StvHsssXdR1xaw== X-Received: by 2002:ac2:5209:0:b0:503:1783:d5a9 with SMTP id a9-20020ac25209000000b005031783d5a9mr3895555lfl.3.1699625304483; Fri, 10 Nov 2023 06:08:24 -0800 (PST) Received: from alex-rivos.home (amontpellier-656-1-456-62.w92-145.abo.wanadoo.fr. [92.145.124.62]) by smtp.gmail.com with ESMTPSA id t18-20020a05600c451200b004076f522058sm5366438wmo.0.2023.11.10.06.08.23 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 10 Nov 2023 06:08:24 -0800 (PST) From: Alexandre Ghiti To: Paul Walmsley , Palmer Dabbelt , Albert Ou , Andrey Ryabinin , Alexander Potapenko , Andrey Konovalov , Dmitry Vyukov , Vincenzo Frascino , Arnd Bergmann , Dennis Zhou , Tejun Heo , Christoph Lameter , Andrew Morton , linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org, kasan-dev@googlegroups.com, linux-arch@vger.kernel.org, linux-mm@kvack.org Cc: Alexandre Ghiti Subject: [PATCH 1/2] mm: Introduce flush_cache_vmap_early() and its riscv implementation Date: Fri, 10 Nov 2023 15:07:20 +0100 Message-Id: <20231110140721.114235-2-alexghiti@rivosinc.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20231110140721.114235-1-alexghiti@rivosinc.com> References: <20231110140721.114235-1-alexghiti@rivosinc.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" The pcpu setup when using the page allocator sets up a new vmalloc mapping very early in the boot process, so early that it cannot use the flush_cache_vmap() function which may depend on structures not yet initialized (for example in riscv, we currently send an IPI to flush other cpus TLB). But on some architectures, we must call flush_cache_vmap(): for example, in riscv, some uarchs can cache invalid TLB entries so we need to flush the new established mapping to avoid taking an exception. So fix this by introducing a new function flush_cache_vmap_early() which is called right after setting the new page table entry and before accessing this new mapping. This new function implements a local flush tlb on riscv and is no-op for other architectures (same as today). Signed-off-by: Alexandre Ghiti --- arch/riscv/include/asm/cacheflush.h | 3 ++- arch/riscv/include/asm/tlbflush.h | 2 ++ arch/riscv/mm/tlbflush.c | 5 +++++ include/asm-generic/cacheflush.h | 6 ++++++ mm/percpu.c | 8 +------- 5 files changed, 16 insertions(+), 8 deletions(-) diff --git a/arch/riscv/include/asm/cacheflush.h b/arch/riscv/include/asm/c= acheflush.h index 3cb53c4df27c..a129dac4521d 100644 --- a/arch/riscv/include/asm/cacheflush.h +++ b/arch/riscv/include/asm/cacheflush.h @@ -37,7 +37,8 @@ static inline void flush_dcache_page(struct page *page) flush_icache_mm(vma->vm_mm, 0) =20 #ifdef CONFIG_64BIT -#define flush_cache_vmap(start, end) flush_tlb_kernel_range(start, end) +#define flush_cache_vmap(start, end) flush_tlb_kernel_range(start, end) +#define flush_cache_vmap_early(start, end) local_flush_tlb_kernel_range(st= art, end) #endif =20 #ifndef CONFIG_SMP diff --git a/arch/riscv/include/asm/tlbflush.h b/arch/riscv/include/asm/tlb= flush.h index 8f3418c5f172..f0d6328076b6 100644 --- a/arch/riscv/include/asm/tlbflush.h +++ b/arch/riscv/include/asm/tlbflush.h @@ -41,6 +41,7 @@ void flush_tlb_page(struct vm_area_struct *vma, unsigned = long addr); void flush_tlb_range(struct vm_area_struct *vma, unsigned long start, unsigned long end); void flush_tlb_kernel_range(unsigned long start, unsigned long end); +void local_flush_tlb_kernel_range(unsigned long start, unsigned long end); #ifdef CONFIG_TRANSPARENT_HUGEPAGE #define __HAVE_ARCH_FLUSH_PMD_TLB_RANGE void flush_pmd_tlb_range(struct vm_area_struct *vma, unsigned long start, @@ -64,6 +65,7 @@ static inline void flush_tlb_kernel_range(unsigned long s= tart, local_flush_tlb_all(); } =20 +#define local_flush_tlb_kernel_range(start, end) flush_tlb_kernel_range(st= art, end) #define flush_tlb_mm(mm) flush_tlb_all() #define flush_tlb_mm_range(mm, start, end, page_size) flush_tlb_all() #endif /* !CONFIG_SMP || !CONFIG_MMU */ diff --git a/arch/riscv/mm/tlbflush.c b/arch/riscv/mm/tlbflush.c index e6659d7368b3..8aadc5f71c93 100644 --- a/arch/riscv/mm/tlbflush.c +++ b/arch/riscv/mm/tlbflush.c @@ -66,6 +66,11 @@ static inline void local_flush_tlb_range_asid(unsigned l= ong start, local_flush_tlb_range_threshold_asid(start, size, stride, asid); } =20 +void local_flush_tlb_kernel_range(unsigned long start, unsigned long end) +{ + local_flush_tlb_range_asid(start, end, PAGE_SIZE, FLUSH_TLB_NO_ASID); +} + static void __ipi_flush_tlb_all(void *info) { local_flush_tlb_all(); diff --git a/include/asm-generic/cacheflush.h b/include/asm-generic/cachefl= ush.h index 84ec53ccc450..7ee8a179d103 100644 --- a/include/asm-generic/cacheflush.h +++ b/include/asm-generic/cacheflush.h @@ -91,6 +91,12 @@ static inline void flush_cache_vmap(unsigned long start,= unsigned long end) } #endif =20 +#ifndef flush_cache_vmap_early +static inline void flush_cache_vmap_early(unsigned long start, unsigned lo= ng end) +{ +} +#endif + #ifndef flush_cache_vunmap static inline void flush_cache_vunmap(unsigned long start, unsigned long e= nd) { diff --git a/mm/percpu.c b/mm/percpu.c index a7665de8485f..d287cebd58ca 100644 --- a/mm/percpu.c +++ b/mm/percpu.c @@ -3306,13 +3306,7 @@ int __init pcpu_page_first_chunk(size_t reserved_siz= e, pcpu_fc_cpu_to_node_fn_t if (rc < 0) panic("failed to map percpu area, err=3D%d\n", rc); =20 - /* - * FIXME: Archs with virtual cache should flush local - * cache for the linear mapping here - something - * equivalent to flush_cache_vmap() on the local cpu. - * flush_cache_vmap() can't be used as most supporting - * data structures are not set up yet. - */ + flush_cache_vmap_early(unit_addr, unit_addr + ai->unit_size); =20 /* copy static data */ memcpy((void *)unit_addr, __per_cpu_load, ai->static_size); --=20 2.39.2 From nobody Tue Dec 16 05:44:36 2025 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 130ADC4332F for ; Fri, 10 Nov 2023 18:09:29 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1344921AbjKJSJ3 (ORCPT ); Fri, 10 Nov 2023 13:09:29 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40482 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1344073AbjKJSFs (ORCPT ); Fri, 10 Nov 2023 13:05:48 -0500 Received: from mail-lf1-x131.google.com (mail-lf1-x131.google.com [IPv6:2a00:1450:4864:20::131]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 56471387AA for ; Fri, 10 Nov 2023 06:09:27 -0800 (PST) Received: by mail-lf1-x131.google.com with SMTP id 2adb3069b0e04-507a3b8b113so2685344e87.0 for ; Fri, 10 Nov 2023 06:09:27 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1699625365; x=1700230165; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=TNncknImGYd0G7MADnDhbFKl7Y37Hs80keyCtMwuCbk=; b=YY/K/GzRK0tHFcdeDuuvXZLWIU8pvr4LiFxNwxVaycV8AyG0/dQ73OldQoy5UfNvc+ PBzvKwKOLpDxF+kgbZPdFXBweB2Rne1tFx0oxa+SU7i8+N5RB8XJVijg+yVYlCnyMczp rYde4d2U8EXENC/VdFsV2G1w6RdDTcDlrlfIqyCThsDMj5DTfQBTwKkNxfMToXZQLapR dOtcYiJzmVXldzPsCcIHgJzIY1/SAswcUdbo5zynpwKYhNCEIbRcDdKbZxkNnTi8x5Mq zYfnNhyqPy9YzRSf8lJm31+LUg44huokCRrnagBXH6Ms0cXF7/tR4Sl6ldV5qX4TzWEv m7Ig== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1699625365; x=1700230165; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=TNncknImGYd0G7MADnDhbFKl7Y37Hs80keyCtMwuCbk=; b=o3mFa8BbI/wG258BJDHA7fNlje/nz4VJjaNyskPDdeXpZ6xipBHuYGFsSdWJ0Ve4Bf 3AQX+wi83c6iSj6ujzpYKXxO+pWA4uaKXe2itlJJPRcohUlelgRbPlI65dapfDpDNK1l vjpH2kHRWiJ5J41h8K8Z3BTG/0HY2ZovSmQl0mHVH8xsu3lV8L6Cd1nr4taPsnOozlJs iiiVhVVw7bk75j9n3YdskJbnm6VJxnlUC5qVIaqjy0HVe7kMu5XrasxXI50Q5zLJuZXb 2UsB4evUiVdU4XiMaGinma4QMGLkDQsvPu3iaBZvvtmzRsM+H9t23CIRgaUTO5sx5Bq4 OFEA== X-Gm-Message-State: AOJu0YxS/nkUtU1ssv31b4analwMtIx1rHUJhiYS5vPd5g9qYUKa5ITb ey92ph8AvdKWd2n7R97p0wwHFQ== X-Google-Smtp-Source: AGHT+IFnc5DDn6eDvAmwHfvD11R9Doa6XtjEu4mYLiLBs82TOIbZPRsbYbDAWR9qLMG/ZVUIcvzhnw== X-Received: by 2002:a2e:2c15:0:b0:2c6:f51f:c96d with SMTP id s21-20020a2e2c15000000b002c6f51fc96dmr6506404ljs.13.1699625365568; Fri, 10 Nov 2023 06:09:25 -0800 (PST) Received: from alex-rivos.home (amontpellier-656-1-456-62.w92-145.abo.wanadoo.fr. [92.145.124.62]) by smtp.gmail.com with ESMTPSA id b12-20020a05600c150c00b004083a105f27sm5173099wmg.26.2023.11.10.06.09.24 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 10 Nov 2023 06:09:25 -0800 (PST) From: Alexandre Ghiti To: Paul Walmsley , Palmer Dabbelt , Albert Ou , Andrey Ryabinin , Alexander Potapenko , Andrey Konovalov , Dmitry Vyukov , Vincenzo Frascino , Arnd Bergmann , Dennis Zhou , Tejun Heo , Christoph Lameter , Andrew Morton , linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org, kasan-dev@googlegroups.com, linux-arch@vger.kernel.org, linux-mm@kvack.org Cc: Alexandre Ghiti Subject: [PATCH 2/2] riscv: Enable pcpu page first chunk allocator Date: Fri, 10 Nov 2023 15:07:21 +0100 Message-Id: <20231110140721.114235-3-alexghiti@rivosinc.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20231110140721.114235-1-alexghiti@rivosinc.com> References: <20231110140721.114235-1-alexghiti@rivosinc.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" As explained in commit 6ea529a2037c ("percpu: make embedding first chunk allocator check vmalloc space size"), the embedding first chunk allocator needs the vmalloc space to be larger than the maximum distance between units which are grouped into NUMA nodes. On a very sparse NUMA configurations and a small vmalloc area (for example, it is 64GB in sv39), the allocation of dynamic percpu data in the vmalloc area could fail. So provide the pcpu page allocator as a fallback in case we fall into such a sparse configuration (which happened in arm64 as shown by commit 09cea6195073 ("arm64: support page mapping percpu first chunk allocator")). Signed-off-by: Alexandre Ghiti --- arch/riscv/Kconfig | 2 ++ arch/riscv/mm/kasan_init.c | 8 ++++++++ 2 files changed, 10 insertions(+) diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig index 5b1e61aca6cf..7b82d8301e42 100644 --- a/arch/riscv/Kconfig +++ b/arch/riscv/Kconfig @@ -416,7 +416,9 @@ config NUMA depends on SMP && MMU select ARCH_SUPPORTS_NUMA_BALANCING select GENERIC_ARCH_NUMA + select HAVE_SETUP_PER_CPU_AREA select NEED_PER_CPU_EMBED_FIRST_CHUNK + select NEED_PER_CPU_PAGE_FIRST_CHUNK select OF_NUMA select USE_PERCPU_NUMA_NODE_ID help diff --git a/arch/riscv/mm/kasan_init.c b/arch/riscv/mm/kasan_init.c index 5e39dcf23fdb..4c9a2c527f08 100644 --- a/arch/riscv/mm/kasan_init.c +++ b/arch/riscv/mm/kasan_init.c @@ -438,6 +438,14 @@ static void __init kasan_shallow_populate(void *start,= void *end) kasan_shallow_populate_pgd(vaddr, vend); } =20 +#ifdef CONFIG_KASAN_VMALLOC +void __init kasan_populate_early_vm_area_shadow(void *start, unsigned long= size) +{ + kasan_populate(kasan_mem_to_shadow(start), + kasan_mem_to_shadow(start + size)); +} +#endif + static void __init create_tmp_mapping(void) { void *ptr; --=20 2.39.2