From nobody Mon Dec 1 22:36:54 2025 Received: from mail-pf1-f178.google.com (mail-pf1-f178.google.com [209.85.210.178]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E576433290F for ; Thu, 27 Nov 2025 14:12:37 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.178 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1764252759; cv=none; b=sXKJK9Xns8Czjz/Q5tmYkTaZo94tlI+sbb+v8zTe9A/i5CNxzuzfqwxyXIpJVnGsT7MvXIugoxVEtjVnBUgooxOf2gcgH6ypBwqb+3XPUr55t7gVPHxPagCbJQyWJuEsvkPXIMpGReoZmyjg4D7gRJyWn+zS+9xKHM3YS4Ip6Lo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1764252759; c=relaxed/simple; bh=K/lO+vh7mnLcg0Ur+IKYTaxb8wzEl8qPzjU81LLXgJA=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=W8D4j1Ca2DLfD8y06GJjL/1hu+yejp5tGxDN/810Ve5mQGI+/gZemDT4dcOCzb66Q7oXC3irHCNdjgwgEZ2i8YNLV/k2AyQWio2tJScwh0gTFgWxgqWxfMML4jK0xCK4EBXVgxJGqeUAGe0ahF/Hg/+WWYlz2Ov23tpzhwbzYPo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=ECp05fti; arc=none smtp.client-ip=209.85.210.178 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="ECp05fti" Received: by mail-pf1-f178.google.com with SMTP id d2e1a72fcca58-794e300e20dso1065443b3a.1 for ; Thu, 27 Nov 2025 06:12:37 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1764252757; x=1764857557; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Zl77PqgzhSQnbli+DMk7tCeC4XrJE4OWFtNvT9IO89M=; b=ECp05ftinIy9U4jB0J9TnaFwgiZVqkdPatM/RxnhIIl3j9k+WDctH3iRnmjUgyuz9K zmkyNfMvNOoDp1XdxcF7RRRgDgU0oh7UMkve9eo/ns1CATNHTIdWIpW33WnRexntOXb7 0bYH13sBtoNlkCSy9lREnwlqjUPxPiIq2DYSLwSlrjESuM1xFcPzj/WW6WLBmd/m0VPs I1nR5JyS+KvODUxgNhcqv9+IvKjQm1Bf6wRpBCSuJ01ftvJir2MrP7YugEH7bA3Z38BN jAcmL82mRoeFZ8iRRdxc8LTgT4tcOj+b2IcUjrHqw7kCSfYrunhdKiFZdpAkvq7ojjmk DUfA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1764252757; x=1764857557; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=Zl77PqgzhSQnbli+DMk7tCeC4XrJE4OWFtNvT9IO89M=; b=tOL7h50uGnMulOtsaoI0dNzicKI/Oa3AjwBDY26PMH5r1v3znqegPQa6YM/15SWfVN KLPsWH4gfZthMIWknrH+J5ZvNVeUO/p5wio2dgtZhWY9TyxuCPs1pzgG3NtYpm2ecodE miLPvs8B+Y7cXTW/kLFpMrxOG4vvSUFokB9NsZhaAw8d9g95tov7ZRWETXZsSIcCKijD mVLztoakPTxFjo0SHstiA4A365jbiUVHSSG8zbY0r4HygCWVozPk9CXsdDJgA7sD0Lxd qV+Gtpj+jZysStbVenh4jUjzsJz7zFJP2nH7IGlOPP4o6BXqFl3mU9svwHNAgk4Rf0kI t8VA== X-Forwarded-Encrypted: i=1; AJvYcCU1jLsx3ojkgHZJgH+TG/0j9k1/R6328nyCqTU46yXL1I4FIkMnnBgxC4/jMbyOCB9upDq4b6toPAOtzqE=@vger.kernel.org X-Gm-Message-State: AOJu0YyS1Nc3K+sYWe/6JzI9D9illiaj3hhuzV2vb2FZhWi6bejnXSmX AR1uuiUfm21skjfsTKwAj8//+KWvRXfkyUTM3D1QgNCIBXgcVgnCvn7BXENdTECiuLA= X-Gm-Gg: ASbGncvQ4JnKdmpgweI5/U5EGexIY5dwQ2LK4OtF1JNJcAc1VxA1lTYwFhgw0Oe+FsF iE4hZMu6LUwt89itv7tmtKXQTiaVhwV6rLm3uFzmtVSo3EfiCQxf04rrtdXM8rHsDi8bg2c2Dyz 5OWSjywwUOrcq20APKc2jrBnngaPe7rEgsKTvqmw9xMRXYetlMccUnKQ/WBN8ZYjjt1C+K8deGe XMzqLea4PBEwWPA2cfy8f7+/dCalzNuLf+0iKt4hWbqMPqNwZfcgKfNMvvp7Fh7r+5Wtotg1KW7 spPG2c52PtdIeN8VmCppd8KeYpOv1+hPo83S7YVeHv6e73mwlPg2ubPUSlpiE6SUEUfL0XSb/p6 GupudpbFfJr4EVPPpLWGGfFCs5fhwQzPduX3vz3LCEa80/FXqh3fXsximkZnFejy1H794E8yYId W1ZaTuC9MFseN/aRJ/Msx+awokpo92rg+qyeTLJcRLQxYFHs9LILg5nGpZyVOejHA5tTATE5W9T g== X-Google-Smtp-Source: AGHT+IFs53VTIZhomsFYAowx9J26TGi3f3ksQWsE5AJKE20aCMEW51a+o1FMLwfvpm4UU6skwP1mVg== X-Received: by 2002:a17:90b:38ca:b0:32e:23c9:6f41 with SMTP id 98e67ed59e1d1-347331be358mr21956360a91.5.1764252756999; Thu, 27 Nov 2025 06:12:36 -0800 (PST) Received: from J9GPGXL7NT.bytedance.net ([61.213.176.58]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-3477b7341d2sm2030249a91.11.2025.11.27.06.12.26 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Thu, 27 Nov 2025 06:12:36 -0800 (PST) From: Xu Lu To: pjw@kernel.org, palmer@dabbelt.com, aou@eecs.berkeley.edu, alex@ghiti.fr, kees@kernel.org, mingo@redhat.com, peterz@infradead.org, juri.lelli@redhat.com, vincent.guittot@linaro.org, akpm@linux-foundation.org, david@redhat.com, apatel@ventanamicro.com, guoren@kernel.org Cc: linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Xu Lu Subject: [RFC PATCH v2 6/9] riscv: mm: Introduce percpu TLB Flush queue Date: Thu, 27 Nov 2025 22:11:14 +0800 Message-ID: <20251127141117.87420-7-luxu.kernel@bytedance.com> X-Mailer: git-send-email 2.50.1 In-Reply-To: <20251127141117.87420-1-luxu.kernel@bytedance.com> References: <20251127141117.87420-1-luxu.kernel@bytedance.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" When memory mapping of a mm is modified, instead of sending IPI to all CPUs recorded in its mm_cpumask, we check whether each target CPU is using this mm right now. If not, we just store the TLB Flush information in target CPU's percpu TLB Flush queue, avoiding the IPI. Signed-off-by: Xu Lu --- arch/riscv/include/asm/tlbflush.h | 19 +++++++++ arch/riscv/mm/context.c | 2 + arch/riscv/mm/tlbflush.c | 71 ++++++++++++++++++++++++++++++- 3 files changed, 90 insertions(+), 2 deletions(-) diff --git a/arch/riscv/include/asm/tlbflush.h b/arch/riscv/include/asm/tlb= flush.h index e7365a53265a6..c9630267c58cd 100644 --- a/arch/riscv/include/asm/tlbflush.h +++ b/arch/riscv/include/asm/tlbflush.h @@ -75,23 +75,40 @@ extern unsigned long tlb_flush_all_threshold; #ifdef CONFIG_RISCV_LAZY_TLB_FLUSH =20 #define MAX_LOADED_MM 6 +#define MAX_TLB_FLUSH_TASK 32 +#define FLUSH_TLB_ALL_ASID 0x1 =20 struct tlb_context { struct mm_struct *mm; unsigned int gen; + bool need_flush; }; =20 +struct tlb_flush_task { + unsigned long start; + unsigned long size; + unsigned long stride; +}; + +struct tlb_flush_queue { + atomic_t len; + unsigned int flag; + struct tlb_flush_task tasks[MAX_TLB_FLUSH_TASK]; +} ____cacheline_aligned_in_smp; + struct tlb_info { rwlock_t rwlock; struct mm_struct *active_mm; unsigned int next_gen; struct tlb_context contexts[MAX_LOADED_MM]; + struct tlb_flush_queue *flush_queues; }; =20 DECLARE_PER_CPU_SHARED_ALIGNED(struct tlb_info, tlbinfo); =20 void local_load_tlb_mm(struct mm_struct *mm); void local_flush_tlb_mm(struct mm_struct *mm); +void __init lazy_tlb_flush_init(void); =20 #else /* CONFIG_RISCV_LAZY_TLB_FLUSH */ =20 @@ -102,6 +119,8 @@ static inline void local_flush_tlb_mm(struct mm_struct = *mm) local_flush_tlb_all_asid(get_mm_asid(mm)); } =20 +static inline void lazy_tlb_flush_init(void) {} + #endif /* CONFIG_RISCV_LAZY_TLB_FLUSH */ =20 #else /* CONFIG_MMU */ diff --git a/arch/riscv/mm/context.c b/arch/riscv/mm/context.c index 3335080e5f720..c381c4ed46bfb 100644 --- a/arch/riscv/mm/context.c +++ b/arch/riscv/mm/context.c @@ -263,6 +263,8 @@ static int __init asids_init(void) =20 __set_bit(0, context_asid_map); =20 + lazy_tlb_flush_init(); + static_branch_enable(&use_asid_allocator); =20 pr_info("ASID allocator using %lu bits (%lu entries)\n", diff --git a/arch/riscv/mm/tlbflush.c b/arch/riscv/mm/tlbflush.c index a47bacf5801ab..b5a2d9874d62b 100644 --- a/arch/riscv/mm/tlbflush.c +++ b/arch/riscv/mm/tlbflush.c @@ -97,6 +97,7 @@ void flush_tlb_all(void) } =20 struct flush_tlb_range_data { + struct mm_struct *mm; unsigned long asid; unsigned long start; unsigned long size; @@ -109,7 +110,8 @@ DEFINE_PER_CPU_SHARED_ALIGNED(struct tlb_info, tlbinfo)= =3D { .rwlock =3D __RW_LOCK_UNLOCKED(tlbinfo.rwlock), .active_mm =3D NULL, .next_gen =3D 1, - .contexts =3D { { NULL, 0, }, }, + .contexts =3D { { NULL, 0, false, }, }, + .next_gen =3D 0, }; =20 static DEFINE_PER_CPU(mm_context_t *, mmdrop_victims); @@ -155,6 +157,47 @@ static inline void mmdrop_lazy_mm(struct mm_struct *mm) } } =20 +static bool should_ipi_flush(int cpu, void *data) +{ + struct tlb_info *info =3D per_cpu_ptr(&tlbinfo, cpu); + struct tlb_context *contexts =3D info->contexts; + struct tlb_flush_queue *queue =3D NULL; + struct flush_tlb_range_data *ftd =3D data; + unsigned int i, index; + unsigned long flags; + + if (info->active_mm =3D=3D ftd->mm) + return true; + + read_lock_irqsave(&info->rwlock, flags); + + if (info->active_mm =3D=3D ftd->mm) { + read_unlock_irqrestore(&info->rwlock, flags); + return true; + } + + for (i =3D 0; i < MAX_LOADED_MM; i++) { + if (contexts[i].mm !=3D ftd->mm) + continue; + + queue =3D &info->flush_queues[i]; + index =3D atomic_fetch_add_unless(&queue->len, 1, MAX_TLB_FLUSH_TASK); + if (index < MAX_TLB_FLUSH_TASK) { + queue->tasks[index].start =3D ftd->start; + queue->tasks[index].stride =3D ftd->stride; + queue->tasks[index].size =3D ftd->size; + } else { + queue->flag |=3D FLUSH_TLB_ALL_ASID; + } + contexts[i].need_flush =3D true; + break; + } + + read_unlock_irqrestore(&info->rwlock, flags); + + return false; +} + #endif /* CONFIG_RISCV_LAZY_TLB_FLUSH */ =20 static void __ipi_flush_tlb_range_asid(void *info) @@ -185,11 +228,20 @@ static void __flush_tlb_range(struct mm_struct *mm, } else { struct flush_tlb_range_data ftd; =20 + ftd.mm =3D mm; ftd.asid =3D asid; ftd.start =3D start; ftd.size =3D size; ftd.stride =3D stride; - on_each_cpu_mask(cmask, __ipi_flush_tlb_range_asid, &ftd, 1); +#ifdef CONFIG_RISCV_LAZY_TLB_FLUSH + if (static_branch_unlikely(&use_asid_allocator) && mm) + on_each_cpu_cond_mask(should_ipi_flush, + __ipi_flush_tlb_range_asid, + &ftd, 1, cmask); + else +#endif + on_each_cpu_mask(cmask, __ipi_flush_tlb_range_asid, + &ftd, 1); } =20 put_cpu(); @@ -376,4 +428,19 @@ void local_flush_tlb_mm(struct mm_struct *mm) local_flush_tlb_all_asid(asid); } =20 +void __init lazy_tlb_flush_init(void) +{ + struct tlb_flush_queue *queue; + unsigned int cpu, size; + + size =3D MAX_LOADED_MM * sizeof(struct tlb_flush_queue); + for_each_possible_cpu(cpu) { + queue =3D kzalloc_node(size, GFP_KERNEL, cpu_to_node(cpu)); + if (!queue) + panic("Failed to alloc per cpu tlb flush queue\n"); + + per_cpu(tlbinfo, cpu).flush_queues =3D queue; + } +} + #endif /* CONFIG_RISCV_LAZY_TLB_FLUSH */ --=20 2.20.1