From nobody Sun Apr 19 13:41:34 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7AF8CCCA47B for ; Thu, 30 Jun 2022 08:38:27 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233826AbiF3Ii0 (ORCPT ); Thu, 30 Jun 2022 04:38:26 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41398 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233434AbiF3IiY (ORCPT ); Thu, 30 Jun 2022 04:38:24 -0400 Received: from out30-57.freemail.mail.aliyun.com (out30-57.freemail.mail.aliyun.com [115.124.30.57]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6EE2F18397 for ; Thu, 30 Jun 2022 01:38:20 -0700 (PDT) X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R161e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=ay29a033018046051;MF=guanghuifeng@linux.alibaba.com;NM=1;PH=DS;RN=21;SR=0;TI=SMTPD_---0VHr2UPz_1656578288; Received: from VM20190228-102.tbsite.net(mailfrom:guanghuifeng@linux.alibaba.com fp:SMTPD_---0VHr2UPz_1656578288) by smtp.aliyun-inc.com; Thu, 30 Jun 2022 16:38:16 +0800 From: Guanghui Feng To: baolin.wang@linux.alibaba.com, catalin.marinas@arm.com, will@kernel.org, akpm@linux-foundation.org, david@redhat.com, jianyong.wu@arm.com, james.morse@arm.com, quic_qiancai@quicinc.com, christophe.leroy@csgroup.eu, jonathan@marek.ca, mark.rutland@arm.com, thunder.leizhen@huawei.com, anshuman.khandual@arm.com, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, rppt@kernel.org, geert+renesas@glider.be, ardb@kernel.org, linux-mm@kvack.org, yaohongbo@linux.alibaba.com Cc: alikernel-developer@linux.alibaba.com Subject: [PATCH v2] arm64: mm: fix linear mapping mem access performance degradation Date: Thu, 30 Jun 2022 16:38:08 +0800 Message-Id: <1656578288-11259-1-git-send-email-guanghuifeng@linux.alibaba.com> X-Mailer: git-send-email 1.8.3.1 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" The arm64 can build 2M/1G block/sectiion mapping. When using DMA/DMA32 zone (enable crashkernel, disable rodata full, disable kfence), the mem_map will use non block/section mapping(for crashkernel requires to shrink the region in page granularity). But it will degrade performance when doing larging continuous mem access in kernel(memcpy/memmove, etc). There are many changes and discussions: commit 031495635b46 ("arm64: Do not defer reserve_crashkernel() for platforms with no DMA memory zones") commit 0a30c53573b0 ("arm64: mm: Move reserve_crashkernel() into mem_init()") commit 2687275a5843 ("arm64: Force NO_BLOCK_MAPPINGS if crashkernel reservation is required") This patch changes mem_map to use block/section mapping with crashkernel. Firstly, do block/section mapping(normally 2M or 1G) for all avail mem at mem_map, reserve crashkernel memory. And then walking pagetable to split block/section mapping to non block/section mapping(normally 4K) [[[only]]] for crashkernel mem. So the linear mem mapping use block/section mapping as more as possible. We will reduce the cpu dTLB miss conspicuously, and accelerate mem access about 10-20% performance improvement. I have tested it with pft(Page Fault Test) and fio, obtained great performace improvement. For fio test: 1.prepare ramdisk modprobe -r brd modprobe brd rd_nr=3D1 rd_size=3D67108864 dmsetup remove_all wipefs -a --force /dev/ram0 mkfs -t ext4 -E lazy_itable_init=3D0,lazy_journal_init=3D0 -q -F /dev/ram0 mkdir -p /fs/ram0 mount -t ext4 /dev/ram0 /fs/ram0 2.prepare fio paremeter in x.fio file: [global] bs=3D4k ioengine=3Dpsync iodepth=3D128 size=3D32G direct=3D1 invalidate=3D1 group_reporting thread=3D1 rw=3Dread directory=3D/fs/ram0 numjobs=3D1 [task_0] cpus_allowed=3D16 stonewall=3D1 3.run testcase: perf stat -e dTLB-load-misses fio x.fio 4.contrast Reported-by: kernel test robot ------------------------ without patch with patch fio READ aggrb=3D1493.2MB/s aggrb=3D1775.3MB/s dTLB-load-misses 1,818,320,693 438,729,774 time elapsed(s) 70.500326434 62.877316408 user(s) 15.926332000 15.684721000 sys(s) 54.211939000 47.046165000 5.conclusion Using this patch will reduce dTLB misses and improve performace greatly. Signed-off-by: Guanghui Feng --- arch/arm64/include/asm/mmu.h | 1 + arch/arm64/mm/init.c | 8 +-- arch/arm64/mm/mmu.c | 163 ++++++++++++++++++++++++++++++---------= ---- 3 files changed, 116 insertions(+), 56 deletions(-) diff --git a/arch/arm64/include/asm/mmu.h b/arch/arm64/include/asm/mmu.h index 48f8466..1a46b81 100644 --- a/arch/arm64/include/asm/mmu.h +++ b/arch/arm64/include/asm/mmu.h @@ -63,6 +63,7 @@ static inline bool arm64_kernel_unmapped_at_el0(void) extern void arm64_memblock_init(void); extern void paging_init(void); extern void bootmem_init(void); +extern void map_crashkernel(void); extern void __iomem *early_io_map(phys_addr_t phys, unsigned long virt); extern void init_mem_pgprot(void); extern void create_pgd_mapping(struct mm_struct *mm, phys_addr_t phys, diff --git a/arch/arm64/mm/init.c b/arch/arm64/mm/init.c index 339ee84..241d27e 100644 --- a/arch/arm64/mm/init.c +++ b/arch/arm64/mm/init.c @@ -190,6 +190,7 @@ static void __init reserve_crashkernel(void) crashk_res.start =3D crash_base; crashk_res.end =3D crash_base + crash_size - 1; insert_resource(&iomem_resource, &crashk_res); + map_crashkernel(); } =20 /* @@ -388,10 +389,6 @@ void __init arm64_memblock_init(void) } =20 early_init_fdt_scan_reserved_mem(); - - if (!IS_ENABLED(CONFIG_ZONE_DMA) && !IS_ENABLED(CONFIG_ZONE_DMA32)) - reserve_crashkernel(); - high_memory =3D __va(memblock_end_of_DRAM() - 1) + 1; } =20 @@ -438,8 +435,7 @@ void __init bootmem_init(void) * request_standard_resources() depends on crashkernel's memory being * reserved, so do it here. */ - if (IS_ENABLED(CONFIG_ZONE_DMA) || IS_ENABLED(CONFIG_ZONE_DMA32)) - reserve_crashkernel(); + reserve_crashkernel(); =20 memblock_dump_all(); } diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index 626ec32..c832848 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -42,6 +42,7 @@ #define NO_BLOCK_MAPPINGS BIT(0) #define NO_CONT_MAPPINGS BIT(1) #define NO_EXEC_MAPPINGS BIT(2) /* assumes FEAT_HPDS is not used */ +#define NO_SEC_REMAPPINGS BIT(3) /* rebuild with non block/sec mapping*/ =20 u64 idmap_t0sz =3D TCR_T0SZ(VA_BITS_MIN); u64 idmap_ptrs_per_pgd =3D PTRS_PER_PGD; @@ -156,11 +157,12 @@ static bool pgattr_change_is_safe(u64 old, u64 new) } =20 static void init_pte(pmd_t *pmdp, unsigned long addr, unsigned long end, - phys_addr_t phys, pgprot_t prot) + phys_addr_t phys, pgprot_t prot, int flags) { pte_t *ptep; =20 - ptep =3D pte_set_fixmap_offset(pmdp, addr); + ptep =3D (flags & NO_SEC_REMAPPINGS) ? pte_offset_kernel(pmdp, addr) : + pte_set_fixmap_offset(pmdp, addr); do { pte_t old_pte =3D READ_ONCE(*ptep); =20 @@ -176,7 +178,8 @@ static void init_pte(pmd_t *pmdp, unsigned long addr, u= nsigned long end, phys +=3D PAGE_SIZE; } while (ptep++, addr +=3D PAGE_SIZE, addr !=3D end); =20 - pte_clear_fixmap(); + if (!(flags & NO_SEC_REMAPPINGS)) + pte_clear_fixmap(); } =20 static void alloc_init_cont_pte(pmd_t *pmdp, unsigned long addr, @@ -208,11 +211,12 @@ static void alloc_init_cont_pte(pmd_t *pmdp, unsigned= long addr, next =3D pte_cont_addr_end(addr, end); =20 /* use a contiguous mapping if the range is suitably aligned */ - if ((((addr | next | phys) & ~CONT_PTE_MASK) =3D=3D 0) && + if (!(flags & NO_SEC_REMAPPINGS) && + (((addr | next | phys) & ~CONT_PTE_MASK) =3D=3D 0) && (flags & NO_CONT_MAPPINGS) =3D=3D 0) __prot =3D __pgprot(pgprot_val(prot) | PTE_CONT); =20 - init_pte(pmdp, addr, next, phys, __prot); + init_pte(pmdp, addr, next, phys, __prot, flags); =20 phys +=3D next - addr; } while (addr =3D next, addr !=3D end); @@ -224,15 +228,44 @@ static void init_pmd(pud_t *pudp, unsigned long addr,= unsigned long end, { unsigned long next; pmd_t *pmdp; + phys_addr_t map_offset; + pmdval_t pmdval; =20 - pmdp =3D pmd_set_fixmap_offset(pudp, addr); + pmdp =3D (flags & NO_SEC_REMAPPINGS) ? pmd_offset(pudp, addr) : + pmd_set_fixmap_offset(pudp, addr); do { pmd_t old_pmd =3D READ_ONCE(*pmdp); =20 next =3D pmd_addr_end(addr, end); =20 - /* try section mapping first */ - if (((addr | next | phys) & ~PMD_MASK) =3D=3D 0 && + if (flags & NO_SEC_REMAPPINGS) { + if (!pmd_none(*pmdp) && pmd_sect(*pmdp)) { + phys_addr_t pte_phys =3D pgtable_alloc(PAGE_SHIFT); + pmd_clear(pmdp); + pmdval =3D PMD_TYPE_TABLE | PMD_TABLE_UXN; + if (flags & NO_EXEC_MAPPINGS) + pmdval |=3D PMD_TABLE_PXN; + __pmd_populate(pmdp, pte_phys, pmdval); + flush_tlb_kernel_range(addr, addr + PAGE_SIZE); + + map_offset =3D addr - (addr & PMD_MASK); + if (map_offset) + alloc_init_cont_pte(pmdp, addr & PMD_MASK, addr, + phys - map_offset, prot, + pgtable_alloc, + flags & (~NO_SEC_REMAPPINGS)); + + if (next < (addr & PMD_MASK) + PMD_SIZE) + alloc_init_cont_pte(pmdp, next, + (addr & PUD_MASK) + PUD_SIZE, + next - addr + phys, + prot, pgtable_alloc, + flags & (~NO_SEC_REMAPPINGS)); + } + alloc_init_cont_pte(pmdp, addr, next, phys, prot, + pgtable_alloc, flags); + }/* try section mapping first */ + else if (((addr | next | phys) & ~PMD_MASK) =3D=3D 0 && (flags & NO_BLOCK_MAPPINGS) =3D=3D 0) { pmd_set_huge(pmdp, phys, prot); =20 @@ -252,7 +285,8 @@ static void init_pmd(pud_t *pudp, unsigned long addr, u= nsigned long end, phys +=3D next - addr; } while (pmdp++, addr =3D next, addr !=3D end); =20 - pmd_clear_fixmap(); + if (!(flags & NO_SEC_REMAPPINGS)) + pmd_clear_fixmap(); } =20 static void alloc_init_cont_pmd(pud_t *pudp, unsigned long addr, @@ -286,7 +320,8 @@ static void alloc_init_cont_pmd(pud_t *pudp, unsigned l= ong addr, next =3D pmd_cont_addr_end(addr, end); =20 /* use a contiguous mapping if the range is suitably aligned */ - if ((((addr | next | phys) & ~CONT_PMD_MASK) =3D=3D 0) && + if (!(flags & NO_SEC_REMAPPINGS) && + (((addr | next | phys) & ~CONT_PMD_MASK) =3D=3D 0) && (flags & NO_CONT_MAPPINGS) =3D=3D 0) __prot =3D __pgprot(pgprot_val(prot) | PTE_CONT); =20 @@ -301,7 +336,9 @@ static void alloc_init_pud(pgd_t *pgdp, unsigned long a= ddr, unsigned long end, phys_addr_t (*pgtable_alloc)(int), int flags) { + phys_addr_t map_offset; unsigned long next; + pudval_t pudval; pud_t *pudp; p4d_t *p4dp =3D p4d_offset(pgdp, addr); p4d_t p4d =3D READ_ONCE(*p4dp); @@ -325,18 +362,48 @@ static void alloc_init_pud(pgd_t *pgdp, unsigned long= addr, unsigned long end, */ if (system_state !=3D SYSTEM_BOOTING) mutex_lock(&fixmap_lock); - pudp =3D pud_set_fixmap_offset(p4dp, addr); + + pudp =3D (flags & NO_SEC_REMAPPINGS) ? pud_offset(p4dp, addr) : + pud_set_fixmap_offset(p4dp, addr); do { pud_t old_pud =3D READ_ONCE(*pudp); - next =3D pud_addr_end(addr, end); =20 + if (flags & NO_SEC_REMAPPINGS) { + if (!pud_none(*pudp) && pud_sect(*pudp)) { + phys_addr_t pmd_phys =3D pgtable_alloc(PMD_SHIFT); + pud_clear(pudp); + pudval =3D PUD_TYPE_TABLE | PUD_TABLE_UXN; + if (flags & NO_EXEC_MAPPINGS) + pudval |=3D PUD_TABLE_PXN; + + __pud_populate(pudp, pmd_phys, pudval); + flush_tlb_kernel_range(addr, addr + PAGE_SIZE); + + map_offset =3D addr - (addr & PUD_MASK); + if (map_offset) + alloc_init_cont_pmd(pudp, addr & PUD_MASK, + addr, phys - map_offset, + prot, pgtable_alloc, + flags & (~NO_SEC_REMAPPINGS)); + + if (next < (addr & PUD_MASK) + PUD_SIZE) + alloc_init_cont_pmd(pudp, next, + (addr & PUD_MASK) + + PUD_SIZE, + next - addr + phys, + prot, pgtable_alloc, + flags & (~NO_SEC_REMAPPINGS)); + } + alloc_init_cont_pmd(pudp, addr, next, phys, prot, + pgtable_alloc, flags); + } /* * For 4K granule only, attempt to put down a 1GB block */ - if (pud_sect_supported() && - ((addr | next | phys) & ~PUD_MASK) =3D=3D 0 && - (flags & NO_BLOCK_MAPPINGS) =3D=3D 0) { + else if (pud_sect_supported() && + ((addr | next | phys) & ~PUD_MASK) =3D=3D 0 && + (flags & NO_BLOCK_MAPPINGS) =3D=3D 0) { pud_set_huge(pudp, phys, prot); =20 /* @@ -355,7 +422,8 @@ static void alloc_init_pud(pgd_t *pgdp, unsigned long a= ddr, unsigned long end, phys +=3D next - addr; } while (pudp++, addr =3D next, addr !=3D end); =20 - pud_clear_fixmap(); + if (!(flags & NO_SEC_REMAPPINGS)) + pud_clear_fixmap(); if (system_state !=3D SYSTEM_BOOTING) mutex_unlock(&fixmap_lock); } @@ -483,20 +551,39 @@ void __init mark_linear_text_alias_ro(void) PAGE_KERNEL_RO); } =20 -static bool crash_mem_map __initdata; +#ifdef CONFIG_KEXEC_CORE +static phys_addr_t __init early_crashkernel_pgtable_alloc(int shift) +{ + phys_addr_t phys; + void *ptr; =20 -static int __init enable_crash_mem_map(char *arg) + phys =3D memblock_phys_alloc_range(PAGE_SIZE, PAGE_SIZE, 0, + MEMBLOCK_ALLOC_NOLEAKTRACE); + if (!phys) + panic("Failed to allocate page table page\n"); + + ptr =3D (void *)__phys_to_virt(phys); + memset(ptr, 0, PAGE_SIZE); + return phys; +} + +void __init map_crashkernel(void) { - /* - * Proper parameter parsing is done by reserve_crashkernel(). We only - * need to know if the linear map has to avoid block mappings so that - * the crashkernel reservations can be unmapped later. - */ - crash_mem_map =3D true; + if (can_set_direct_map() || IS_ENABLED(CONFIG_KFENCE)) + return; =20 - return 0; + if (!crashk_res.end) + return; + + __create_pgd_mapping(swapper_pg_dir, crashk_res.start, + __phys_to_virt(crashk_res.start), + crashk_res.end + 1 - crashk_res.start, PAGE_KERNEL, + early_crashkernel_pgtable_alloc, + NO_EXEC_MAPPINGS | NO_SEC_REMAPPINGS); } -early_param("crashkernel", enable_crash_mem_map); +#else +void __init map_crashkernel(void) {} +#endif =20 static void __init map_mem(pgd_t *pgdp) { @@ -527,17 +614,6 @@ static void __init map_mem(pgd_t *pgdp) */ memblock_mark_nomap(kernel_start, kernel_end - kernel_start); =20 -#ifdef CONFIG_KEXEC_CORE - if (crash_mem_map) { - if (IS_ENABLED(CONFIG_ZONE_DMA) || - IS_ENABLED(CONFIG_ZONE_DMA32)) - flags |=3D NO_BLOCK_MAPPINGS | NO_CONT_MAPPINGS; - else if (crashk_res.end) - memblock_mark_nomap(crashk_res.start, - resource_size(&crashk_res)); - } -#endif - /* map all the memory banks */ for_each_mem_range(i, &start, &end) { if (start >=3D end) @@ -570,19 +646,6 @@ static void __init map_mem(pgd_t *pgdp) * in page granularity and put back unused memory to buddy system * through /sys/kernel/kexec_crash_size interface. */ -#ifdef CONFIG_KEXEC_CORE - if (crash_mem_map && - !IS_ENABLED(CONFIG_ZONE_DMA) && !IS_ENABLED(CONFIG_ZONE_DMA32)) { - if (crashk_res.end) { - __map_memblock(pgdp, crashk_res.start, - crashk_res.end + 1, - PAGE_KERNEL, - NO_BLOCK_MAPPINGS | NO_CONT_MAPPINGS); - memblock_clear_nomap(crashk_res.start, - resource_size(&crashk_res)); - } - } -#endif } =20 void mark_rodata_ro(void) --=20 1.8.3.1