From nobody Thu Dec 18 09:45:36 2025 Received: from mail-pg1-f177.google.com (mail-pg1-f177.google.com [209.85.215.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 559B21DB951 for ; Thu, 5 Dec 2024 10:38:28 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.215.177 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1733395110; cv=none; b=PDtOJpU19ONHPjI/iKqod7+tOVvAohEj4NUcPKFL2g98y3XCpYCdUbLA6SZ5BbaMXPBkara7v1R13W2ogxJCsHMzOLiBI4S8YCWCcoErA2+8tCXBaqfYAjuKpG9ALyqjiL5oZYuxaagt9n9An5PuokNpykO1cKKaS8UoUebcjCk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1733395110; c=relaxed/simple; bh=iaiXED/tqibRXcFpLigD3NvJz1CulAasWjUYazfFtPc=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=fzrfH+FfmlHQP5Kgg/tQCZtVQuQF5e54DPZuVUPVBaGYv0XoU2Pd7YbXinrNEJB0PD1cUO4ed8sHgauEthpOkW/B5fgSgV3HKMdDzizTqTpefosLZ4WFKLpBzkd3TEAzuNxeUkb+d8flw+xa9ntCuBctRrixzAF9X+57ztyB6SY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=aTWOpJLQ; arc=none smtp.client-ip=209.85.215.177 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="aTWOpJLQ" Received: by mail-pg1-f177.google.com with SMTP id 41be03b00d2f7-7fcfac22a30so832252a12.3 for ; Thu, 05 Dec 2024 02:38:28 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1733395108; x=1733999908; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=HFQIXohFMC+aCLCj7cIVe0JD4G758F8Hwu3svQYABEY=; b=aTWOpJLQpSF75/OhWUsAWdmRJmhklV2DlaxSg/3IT7WwdLbkTHdh3/zGg+CL+tyQjn xY0pHJZBTS802R0y3Q78XglE5c8HjkTQaNP/rzOomPcs7q+PbLn4zYuuRQGs9hTRGLam jKLAYBxNNICIvWdLQRGq7XIihkVpNCvYdKLmMFKHguHJBmxJAo5B5BOBi7QrWzbH6b1k fh05d+UEDyDlpbowKIdZCYH37rmYwJ3wqbLQez6DlR2xWiqwct8RGA1r7alBQ/DjuUJ6 su+/eyaAMo6Py+M4jFYz/IBKSo/lsZ95pRcEWmKLai40P64EE9BNVSsrH4o/sHt4jNB5 N8GQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1733395108; x=1733999908; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=HFQIXohFMC+aCLCj7cIVe0JD4G758F8Hwu3svQYABEY=; b=mBmzc93QZrMVlEe1TqqP4ic3OIjYiUZlAYInUCTETpuImheJTLPFwa2e0Y3BNlfDvZ fXMlzhcWaXNLekh4pnCMJvQXfMKqDMHbxMMpkZHhZljTb53fO6F6z+0wGpPf1J/V8zkz SdB8dzIBCQGxQqt9Vdr0fBTW1MyaHf4hlCwvem19zTZNz5NrZTQroQWeyWYjGnr862tA GHRZ9DCRLVaq40VkWtJJ82v2VXll2G4wgYuZIFPaD11HRsbFRw/7H9KjnlFZtN0yGQaQ eUMlUzyRGJs9CV6loCYjpxM7MhHSJn6F4EGiH/1dNS6WG60L5QfH985xhlcllDgIA8hm 2Ujg== X-Forwarded-Encrypted: i=1; AJvYcCUbo0oTx8M/PL10N7no69rjCgtsKdNF/ZrjwWoLKPcWycEtnhFuVX7fMdwXO8hPabczeC2HXcIxGveIPP4=@vger.kernel.org X-Gm-Message-State: AOJu0YzE0Ms6EZg8t9W2fXQXiWyg53H34YHGIDCV0vIkolRzMKjQUpN/ sxmGXKHBsEFFykwAqSYndflQ0LMLoxqSv8yAYczoZxeEt+UzK4Omc0iofarTG2Y= X-Gm-Gg: ASbGncsdfhXeqI3OBcH8vYzjQxwg+OGe8GX4p21V/sMLQRMk3JolY4RYpy22F+3uXnT 8JZxaq0mf3+x993VNdVvpRd6w5mt3a4+4jqXOM1b0XNxGRXdeBEn9hQ2djzWts6KCqWlRuIYpfY OpGz0zsDob9XkAGUChViW1RHHC0s0sdJQR84bS1iQKJXkRa61MU1x6Bnp6aWy/7mUi28K6a1tlt IiGr4lnrdTvgt1QEG/p3FYgN0Jp5gsvJYs7Ns4wOp5twZe9mjFNZ1IPIyqJ+oRypiiXPd1YvBH3 F/yvdHSFgn7HQOtMBeVvDIodEKey8xmY X-Google-Smtp-Source: AGHT+IEVSD+tgAqJx2WiLXY6iHldH8R0b3LHGoJ+robZaZ03Gbz5dxvb2ZG0GtuQ60x2xrNdVJqFOw== X-Received: by 2002:a05:6300:4041:b0:1e1:1659:82a4 with SMTP id adf61e73a8af0-1e16541341fmr16353259637.41.1733395107751; Thu, 05 Dec 2024 02:38:27 -0800 (PST) Received: from J9GPGXL7NT.bytedance.net ([61.213.176.56]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-7fd156f048csm886826a12.39.2024.12.05.02.38.24 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Thu, 05 Dec 2024 02:38:27 -0800 (PST) From: Xu Lu To: paul.walmsley@sifive.com, palmer@dabbelt.com, aou@eecs.berkeley.edu, ardb@kernel.org, anup@brainfault.org, atishp@atishpatra.org Cc: xieyongji@bytedance.com, lihangjing@bytedance.com, punit.agrawal@bytedance.com, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, Xu Lu Subject: [RFC PATCH v2 12/21] riscv: mm: Reimplement tlb flush function Date: Thu, 5 Dec 2024 18:37:20 +0800 Message-Id: <20241205103729.14798-13-luxu.kernel@bytedance.com> X-Mailer: git-send-email 2.39.5 (Apple Git-154) In-Reply-To: <20241205103729.14798-1-luxu.kernel@bytedance.com> References: <20241205103729.14798-1-luxu.kernel@bytedance.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" When tlb flushing a page correponding to a certain address, CPU actually only flushes tlb entries of the first 4K hardware page. This commit reimplements tlb flushing function to flush all tlb entries of hardware pag= es in the same software page. Signed-off-by: Xu Lu --- arch/riscv/include/asm/pgtable.h | 9 ++++++--- arch/riscv/include/asm/tlbflush.h | 26 ++++++++++++++++++++------ arch/riscv/mm/fault.c | 13 +++++++++---- arch/riscv/mm/init.c | 2 +- arch/riscv/mm/tlbflush.c | 31 +++++++++++++++++++++---------- 5 files changed, 57 insertions(+), 24 deletions(-) diff --git a/arch/riscv/include/asm/pgtable.h b/arch/riscv/include/asm/pgta= ble.h index c0f7442c8a9e..9fa16c0c20aa 100644 --- a/arch/riscv/include/asm/pgtable.h +++ b/arch/riscv/include/asm/pgtable.h @@ -701,7 +701,7 @@ static inline void update_mmu_cache_range(struct vm_fau= lt *vmf, * the extra traps reduce performance. So, eagerly SFENCE.VMA. */ while (nr--) - local_flush_tlb_page(address + nr * PAGE_SIZE); + local_flush_tlb_page(address + nr * PAGE_SIZE, PAGE_SIZE); =20 svvptc:; /* @@ -719,9 +719,12 @@ svvptc:; static inline void update_mmu_cache_pmd(struct vm_area_struct *vma, unsigned long address, pmd_t *pmdp) { - pte_t *ptep =3D (pte_t *)pmdp; + asm goto(ALTERNATIVE("nop", "j %l[svvptc]", 0, RISCV_ISA_EXT_SVVPTC, 1) + : : : : svvptc); =20 - update_mmu_cache(vma, address, ptep); + local_flush_tlb_page(address, PMD_SIZE); + +svvptc:; } =20 #define __HAVE_ARCH_PTE_SAME diff --git a/arch/riscv/include/asm/tlbflush.h b/arch/riscv/include/asm/tlb= flush.h index 72e559934952..25cc39ab84d5 100644 --- a/arch/riscv/include/asm/tlbflush.h +++ b/arch/riscv/include/asm/tlbflush.h @@ -29,18 +29,32 @@ static inline void local_flush_tlb_all_asid(unsigned lo= ng asid) } =20 /* Flush one page from local TLB */ -static inline void local_flush_tlb_page(unsigned long addr) +static inline void local_flush_tlb_page(unsigned long addr, + unsigned long page_size) { - ALT_SFENCE_VMA_ADDR(addr); + unsigned int i; + unsigned long hw_page_num =3D 1 << (PAGE_SHIFT - HW_PAGE_SHIFT); + unsigned long hw_page_size =3D page_size >> (PAGE_SHIFT - HW_PAGE_SHIFT); + + for (i =3D 0; i < hw_page_num; i++, addr +=3D hw_page_size) + ALT_SFENCE_VMA_ADDR(addr); } =20 static inline void local_flush_tlb_page_asid(unsigned long addr, + unsigned long page_size, unsigned long asid) { - if (asid !=3D FLUSH_TLB_NO_ASID) - ALT_SFENCE_VMA_ADDR_ASID(addr, asid); - else - local_flush_tlb_page(addr); + unsigned int i; + unsigned long hw_page_num, hw_page_size; + + if (asid !=3D FLUSH_TLB_NO_ASID) { + hw_page_num =3D 1 << (PAGE_SHIFT - HW_PAGE_SHIFT); + hw_page_size =3D page_size >> (PAGE_SHIFT - HW_PAGE_SHIFT); + + for (i =3D 0; i < hw_page_num; i++, addr +=3D hw_page_size) + ALT_SFENCE_VMA_ADDR_ASID(addr, asid); + } else + local_flush_tlb_page(addr, page_size); } =20 void flush_tlb_all(void); diff --git a/arch/riscv/mm/fault.c b/arch/riscv/mm/fault.c index 4772152be0f9..94524e5adc0b 100644 --- a/arch/riscv/mm/fault.c +++ b/arch/riscv/mm/fault.c @@ -118,7 +118,7 @@ static inline void vmalloc_fault(struct pt_regs *regs, = int code, unsigned long a pmd_t *pmd_k; pte_t *pte_k; int index; - unsigned long pfn; + unsigned long pfn, page_size; =20 /* User mode accesses just cause a SIGSEGV */ if (user_mode(regs)) @@ -154,8 +154,10 @@ static inline void vmalloc_fault(struct pt_regs *regs,= int code, unsigned long a no_context(regs, addr); return; } - if (pud_leaf(pudp_get(pud_k))) + if (pud_leaf(pudp_get(pud_k))) { + page_size =3D PUD_SIZE; goto flush_tlb; + } =20 /* * Since the vmalloc area is global, it is unnecessary @@ -166,8 +168,10 @@ static inline void vmalloc_fault(struct pt_regs *regs,= int code, unsigned long a no_context(regs, addr); return; } - if (pmd_leaf(pmdp_get(pmd_k))) + if (pmd_leaf(pmdp_get(pmd_k))) { + page_size =3D PMD_SIZE; goto flush_tlb; + } =20 /* * Make sure the actual PTE exists as well to @@ -180,6 +184,7 @@ static inline void vmalloc_fault(struct pt_regs *regs, = int code, unsigned long a no_context(regs, addr); return; } + page_size =3D PAGE_SIZE; =20 /* * The kernel assumes that TLBs don't cache invalid @@ -188,7 +193,7 @@ static inline void vmalloc_fault(struct pt_regs *regs, = int code, unsigned long a * necessary even after writing invalid entries. */ flush_tlb: - local_flush_tlb_page(addr); + local_flush_tlb_page(addr, page_size); } =20 static inline bool access_error(unsigned long cause, struct vm_area_struct= *vma) diff --git a/arch/riscv/mm/init.c b/arch/riscv/mm/init.c index f9334aab45a6..678b892b4ed8 100644 --- a/arch/riscv/mm/init.c +++ b/arch/riscv/mm/init.c @@ -356,7 +356,7 @@ void __set_fixmap(enum fixed_addresses idx, phys_addr_t= phys, pgprot_t prot) set_pte(ptep, pfn_pte(phys >> PAGE_SHIFT, prot)); else pte_clear(&init_mm, addr, ptep); - local_flush_tlb_page(addr); + local_flush_tlb_page(addr, PAGE_SIZE); } =20 static inline pte_t *__init get_pte_virt_early(phys_addr_t pa) diff --git a/arch/riscv/mm/tlbflush.c b/arch/riscv/mm/tlbflush.c index 9b6e86ce3867..d5036f2a8244 100644 --- a/arch/riscv/mm/tlbflush.c +++ b/arch/riscv/mm/tlbflush.c @@ -27,7 +27,7 @@ static void local_flush_tlb_range_threshold_asid(unsigned= long start, } =20 for (i =3D 0; i < nr_ptes_in_range; ++i) { - local_flush_tlb_page_asid(start, asid); + local_flush_tlb_page_asid(start, stride, asid); start +=3D stride; } } @@ -36,7 +36,7 @@ static inline void local_flush_tlb_range_asid(unsigned lo= ng start, unsigned long size, unsigned long stride, unsigned long asid) { if (size <=3D stride) - local_flush_tlb_page_asid(start, asid); + local_flush_tlb_page_asid(start, stride, asid); else if (size =3D=3D FLUSH_TLB_MAX_SIZE) local_flush_tlb_all_asid(asid); else @@ -126,14 +126,7 @@ void flush_tlb_mm_range(struct mm_struct *mm, start, end - start, page_size); } =20 -void flush_tlb_page(struct vm_area_struct *vma, unsigned long addr) -{ - __flush_tlb_range(mm_cpumask(vma->vm_mm), get_mm_asid(vma->vm_mm), - addr, PAGE_SIZE, PAGE_SIZE); -} - -void flush_tlb_range(struct vm_area_struct *vma, unsigned long start, - unsigned long end) +static inline unsigned long local_flush_tlb_page_size(struct vm_area_struc= t *vma) { unsigned long stride_size; =20 @@ -161,6 +154,24 @@ void flush_tlb_range(struct vm_area_struct *vma, unsig= ned long start, } } =20 + return stride_size; +} + +void flush_tlb_page(struct vm_area_struct *vma, unsigned long addr) +{ + unsigned long page_size; + + page_size =3D local_flush_tlb_page_size(vma); + __flush_tlb_range(mm_cpumask(vma->vm_mm), get_mm_asid(vma->vm_mm), + addr, page_size, page_size); +} + +void flush_tlb_range(struct vm_area_struct *vma, unsigned long start, + unsigned long end) +{ + unsigned long stride_size; + + stride_size =3D local_flush_tlb_page_size(vma); __flush_tlb_range(mm_cpumask(vma->vm_mm), get_mm_asid(vma->vm_mm), start, end - start, stride_size); } --=20 2.20.1