From nobody Mon Dec 1 22:36:56 2025 Received: from mail-pj1-f53.google.com (mail-pj1-f53.google.com [209.85.216.53]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 98795332EC0 for ; Thu, 27 Nov 2025 14:11:45 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.53 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1764252707; cv=none; b=cRXW8I+Nz00QMA8QDqVKdvX4NCCW9NLVCI8XLymkWbrin6huMaIouBdHOq2/IqgW7tact++G5CfgoIy4SugWrhVAPakhm/bybQcy/3UwMsWAJcXReQBkkOGlWUuLHW7WdNbuMNw+LIFGoIJafNcy3Y8mRt8FNSFiR8PH16AMLH0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1764252707; c=relaxed/simple; bh=okM0GCRyHBc9yD3vJjpVMACcPIWCzIvckcEqbsl4dEI=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=sC1g+hQlMuu8bJTYGP1dPUX6VMkmhb7r5iWxpj/0JBg+wx62MCmZFvk8rSPaVQ8jzUcKOZs2Pwf4re5flu7UTqIRZCtA8UohlypUn94QlStn/UvvVQCRAp9mkxnrgUK1uH/5JigDunLmwWA3RJ09xhohrMVt5pfHi2cGhLdzqAw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=dwzDy9kR; arc=none smtp.client-ip=209.85.216.53 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="dwzDy9kR" Received: by mail-pj1-f53.google.com with SMTP id 98e67ed59e1d1-34585428e33so941963a91.3 for ; Thu, 27 Nov 2025 06:11:45 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1764252705; x=1764857505; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=FPuxDa0NXAL0s5rvqqEK7cssEYmk0gHTIqKy/7wy1IU=; b=dwzDy9kRxjAfm/1vNlSj4U5v89AnOn0GFQIViuoHQ3+YAQMtcwud4h5LtZfCG8ca0c QhLXpRroFJy1aAi+c9jTPzx/1qxvWaT2fifD+bPL8JKOVqg0JwW0FhNiWxgD66IE+SIS i58zKOgPlpbzBYj9Lx/FEqwpBWQIkaKNq6yCeIq6avOePXkv4FpvEEZGUZjrCJy2oggJ Fy6jUuzOBU6cGkdhgff3JPwU/k+ElVvcfuH5o+Lm4CJZo05zv22GJ1tC8U4iSwYOakRl mF5yXEd0tt84yB9RxU63TG7/lQlUe2cbYDOBPADK1bIiQF/M6Zmo8j+GLIkLKkm12xyX DcHg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1764252705; x=1764857505; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=FPuxDa0NXAL0s5rvqqEK7cssEYmk0gHTIqKy/7wy1IU=; b=DaWAvlWIoOfx49UYJ/k+YambNdDsi6+O32gtpFABT/RbMP9CbI0ze0L2DggVuq4mCh IdKF6ARSkQl7QnHmarlqkw+6aGJw7CtnXpjpZ25AKJW3xqfxmgeuufVgfGTYZc32EpNq hS7iQhfC4K5GW4QJQnQcLn6FzaatVNKTbQ53f2kawY/tXHnbs7e9TqELzFSK4hsFqbkn THUHsfYXWuerfJ+BulEiC5BAv6IblJ70CEDk2A83+fw1kt4n4F6eHrcHs4JKy1gKYVgR R2FVQLBgCuOB47PLhB0lZiqzcq2D2qCZkPrxjXvxMSNFZ/ToiFT9aO1UPBBvDbE8K5U3 RkiA== X-Forwarded-Encrypted: i=1; AJvYcCWRWm1Zrg/VLl2qrdXv+gC2WIvpZbef7v7c/9UQKx1TGQNvLzPlAGX8WeqP2l9xZgaU607xhiyfiVbKEq4=@vger.kernel.org X-Gm-Message-State: AOJu0YxSc5BSLKlkZciZG+I5lec/KdEzuJhq6ysWjWZB1bPvQeweJ8Vc eQdUUo6Cb09Sf3bm6ICGHiEaNLq+2wl1QuTKU6vQUMgbUJpTXQXbpG9wLmvNuteIPJo= X-Gm-Gg: ASbGncuoxR66AMMF912lVVmp+GUIh/QIY2OT6WZycsoO+6obA7wZhwsOIb21cbKCGte NYe0Kvfi21l09hnSH+TQUlHXrEcL4dEH/lgmv240e4KfbFM8vwHJTVysQB3h4G6Pm9sVF5trEQR HObUiBIWo3poCmlCMlg7W+kW2YaUheIhhCfgypa41pnTE6fLbtUGfFphYdPKUf51zymc2xZTXsW xQTIbXu9nj+SNNlL6/s6D1UIitaf3MvCIY17rGnA3YJJpBYaocZeOuOlptfJtuRktjnAd/dYuFi bRhV3QhMhwzVUtykxjqUHm8PPJTFWX2XExMAnnCTTGrWU+XavD7YbggwnJhFb13Un8P9/71j+Qd 9dl02k5rrJpUfkRi2k8CZYzbIxKet9ePmW0ZT1LN6hte5PSd00QyeE4yHVaII9qmNPmQOoXmq2S 2aOeDWLWZ3Mg8NK9Sa496ayIGSShuWaJ16BbKuSrKC4DKNjQG49keY174EO5I1dJDnmfoGue6rc UBMI62tkUj5 X-Google-Smtp-Source: AGHT+IHJ1zH4LIr+8ggxAx5kKu9ExeGv2qGaiXw+0rzT1l+A7YZxhx40PkhUi9yrE8Oc+3HdrnQIwA== X-Received: by 2002:a17:90b:38c5:b0:343:6611:f21 with SMTP id 98e67ed59e1d1-34733e2ccf1mr21792000a91.1.1764252704720; Thu, 27 Nov 2025 06:11:44 -0800 (PST) Received: from J9GPGXL7NT.bytedance.net ([61.213.176.58]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-3477b7341d2sm2030249a91.11.2025.11.27.06.11.39 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Thu, 27 Nov 2025 06:11:44 -0800 (PST) From: Xu Lu To: pjw@kernel.org, palmer@dabbelt.com, aou@eecs.berkeley.edu, alex@ghiti.fr, kees@kernel.org, mingo@redhat.com, peterz@infradead.org, juri.lelli@redhat.com, vincent.guittot@linaro.org, akpm@linux-foundation.org, david@redhat.com, apatel@ventanamicro.com, guoren@kernel.org Cc: linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Xu Lu Subject: [RFC PATCH v2 1/9] riscv: Introduce RISCV_LAZY_TLB_FLUSH config Date: Thu, 27 Nov 2025 22:11:09 +0800 Message-ID: <20251127141117.87420-2-luxu.kernel@bytedance.com> X-Mailer: git-send-email 2.50.1 In-Reply-To: <20251127141117.87420-1-luxu.kernel@bytedance.com> References: <20251127141117.87420-1-luxu.kernel@bytedance.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" This feature avoids unnecessary TLB Flush IPIs. After memory mapping modifications on certain mm_struct, instead of sending IPIs, this feature records the TLB Flush information on percpu buffer, defer the TLB Flush to the moment when target CPUs really load this mm_struct. Signed-off-by: Xu Lu --- arch/riscv/Kconfig | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig index 22cda9c452d2a..d219c7f4b129e 100644 --- a/arch/riscv/Kconfig +++ b/arch/riscv/Kconfig @@ -925,6 +925,18 @@ config RISCV_VECTOR_MISALIGNED help Enable detecting support for vector misaligned loads and stores. =20 +config RISCV_LAZY_TLB_FLUSH + bool "Defer TLB Flush to context switch to avoid IPIs" + depends on MMU && SMP + def_bool n + help + This feature avoids unnecessary TLB Flush IPIs. After memory mapping + modifications on certain mm_struct, instead of sending IPIs, this featu= re + records the TLB Flush information on percpu buffer, defer the TLB Flush + to the moment when target CPUs really load this mm_struct. + + If unsure what to do here, say N. + choice prompt "Unaligned Accesses Support" default RISCV_PROBE_UNALIGNED_ACCESS --=20 2.20.1 From nobody Mon Dec 1 22:36:56 2025 Received: from mail-pj1-f54.google.com (mail-pj1-f54.google.com [209.85.216.54]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1E25E332EB2 for ; Thu, 27 Nov 2025 14:11:50 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.54 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1764252712; cv=none; b=YXEQ14m/hCnHIaleUJ++9Zgg0DrzyWymHN9iByO1WOIFy9sPDGTH7UgQKH2hTTWEoENT9wq04baZA1VGDDdkWYLQsvV2Kcynu5v0JIBAhNX/HjPGHC23lHAlRHuXUGcQRPKsdlWjsUhcBvPWStpGjhRWVjLZmPdxCsT0oRFQIoo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1764252712; c=relaxed/simple; bh=6WgF+HUgdtvQQTctNmbv7ojAGa2BGON60tch8qSWMbg=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=Ntyjz0xzndrJ4I+oLg3xeo5ze+o2IITwz3Uv895VRW283fjwruZI1MxLU5BdxDv9U8x+kiPhADCL/kPepcfL7iC+5lkzogUqfUn1Naurcxw15/WfYUsyVWNYHaFaOk+o8XbR/3k28vea3IwU3l7ln4amv03quYaMc1OAV6yazKE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=Df/MJMnQ; arc=none smtp.client-ip=209.85.216.54 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="Df/MJMnQ" Received: by mail-pj1-f54.google.com with SMTP id 98e67ed59e1d1-3434700be69so1217415a91.1 for ; Thu, 27 Nov 2025 06:11:50 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1764252710; x=1764857510; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=OGHgnOknc2HR28gygrywMGZEYzdqi18gD0LqvxY1V9E=; b=Df/MJMnQBdMFBRz3kBeHlN4njecNvUYRp9XTAq4qul2GOSyjIajxkBcBc47cAzg/Z6 xEvrlCcff8SaI0kHns/oqVYSr0a4vMt+peIHpbgkTFEm3/41QpVxHnAEwYK4gLRMD7UK vLHNSoLxC0iVoPHD2K8tu2F7TD8zPYZLLr9CoNkZ2uueS6GgGBblPQPd/R0P+yBt4UNu 03r6ick5x2dnK2dZqGY51S8usxQte/pzbw3OaR/vgLoYsYBvxKjmnD+fUuddS0B7APMy DYYwUxStCC2IWRO+xlmV1u7Rnh529p8T0nk6X66g9xWN4PKtDyVUutXH+bG8Vxdt5Tdd 4y5Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1764252710; x=1764857510; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=OGHgnOknc2HR28gygrywMGZEYzdqi18gD0LqvxY1V9E=; b=O8f+bfC0BZdwFXuNV8frvC1D0uLVamBPr4jlhQEc+xR/jVhsHtakG7NilwbFF06tZM r/YFJ3/z79ROvvQa4kEbJLqIMj5J5tOWoi9YfB95T1GHPYYk6orcdCnS2o84eoMZ1nr4 /py5rYxP/87dvLdwAuj6pWon/R+AhWA6aHOgI/h8I6XT8YnkVGsD9KJ8RQJsY0rlbHcE cIXPW9JlW0u4l6lP6Odap2xiG3EXvBoR6M0M2o13ygB602CQt8i+c52U/FpWJArOXZ5Z 2GrQYEKZ2tPhTDhRmfbXwSeD9orVIjdHszrwK0wBi46AcT+1DE9Lu4IpfHTDytjjxj5x HFwQ== X-Forwarded-Encrypted: i=1; AJvYcCU2btPMRkYnVAel3aaXO5tSf2O5HvQ7MykEQVCSf/lA/2a6Iz+gg7hXyOJaALHfGu9jqk+NllzsAEHAKY4=@vger.kernel.org X-Gm-Message-State: AOJu0YzmhLRcbthRSwwhzcqikUyhwgdkin7UGrO4Kc82reXK4g7cSNrN sZ1Vpfm9Jw7g2LDcOJ8amxTgKWLCO2S6drkM/nNlVu0oTDNPWljzeLpPnAdcfL+GAns= X-Gm-Gg: ASbGncvMGqBwgV0JiSvG51TJ0u0lE8xI5AiEp9z12aWBHspBGasSDZ2OBARqoUX1Aqs 7KHLGv4EHu62h7HD7qMT5OYGntOwWPrUttoacDxvEmSjR1U6nRMN587HWJLBmb4Io44XPCUSbD2 GwuxVZPAufTFZaxpcEL+OyO7eKeMdGyLbvOBJUAqt/hicBPXv8NKViZf9dTXpBTNq+MTOyjYqY3 kCe/kWFd2oaSV4phctM858M/vNtoDiHk/AIwSem2nR+Nhv5sx7gWZDWi+dvqLnHxsqGgwHuh6rC j5SJFCEr4J5YEWQnfeGFIFDLQ5Z2rDV6F30EFQM9z+9LViX5+mPcrS53/XXS2iPIhj28+eDw5nP t648G22N/u7+MBKAEXc3oCjqIu3HxCRMZ5MWKVCPH2i9Y3Nk37rNK5HuWNsBMmevRnaNQjunhzh 8vhHbLiMm0v/iLaU9lFa9h0bBSVBwxjD9CmClpadGMZh17+kOzDJSXSFoUHyuPgisTIOMqZ99ke w== X-Google-Smtp-Source: AGHT+IH9shCFmiY257H7nfVJqHK2TfMQ1cqSxBQ+VJIOMrKykl/Xamof7TK0iwdrGZLERz9FgKIyJw== X-Received: by 2002:a17:90b:4c8f:b0:341:133:e13d with SMTP id 98e67ed59e1d1-3475ebd2d92mr11210465a91.5.1764252710205; Thu, 27 Nov 2025 06:11:50 -0800 (PST) Received: from J9GPGXL7NT.bytedance.net ([61.213.176.58]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-3477b7341d2sm2030249a91.11.2025.11.27.06.11.45 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Thu, 27 Nov 2025 06:11:49 -0800 (PST) From: Xu Lu To: pjw@kernel.org, palmer@dabbelt.com, aou@eecs.berkeley.edu, alex@ghiti.fr, kees@kernel.org, mingo@redhat.com, peterz@infradead.org, juri.lelli@redhat.com, vincent.guittot@linaro.org, akpm@linux-foundation.org, david@redhat.com, apatel@ventanamicro.com, guoren@kernel.org Cc: linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Xu Lu Subject: [RFC PATCH v2 2/9] riscv: mm: Apply a threshold to the number of active ASIDs on each CPU Date: Thu, 27 Nov 2025 22:11:10 +0800 Message-ID: <20251127141117.87420-3-luxu.kernel@bytedance.com> X-Mailer: git-send-email 2.50.1 In-Reply-To: <20251127141117.87420-1-luxu.kernel@bytedance.com> References: <20251127141117.87420-1-luxu.kernel@bytedance.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Since each CPU has limited TLB entries, there exist limited active ASIDs in each CPU's TLB at the same time. Thus we apply a threshold here. When a mm_struct is loaded, we mark its ASID as active. If the number of active ASIDs exceeds the threshold, we evict the mm_struct that has not been used for the longest time, flush its TLB entries, mark its ASID inactive, and clear current CPU in its mm_cpumask. Signed-off-by: Xu Lu --- arch/riscv/include/asm/tlbflush.h | 27 +++++++++++++ arch/riscv/mm/context.c | 1 + arch/riscv/mm/tlbflush.c | 66 +++++++++++++++++++++++++++++++ 3 files changed, 94 insertions(+) diff --git a/arch/riscv/include/asm/tlbflush.h b/arch/riscv/include/asm/tlb= flush.h index eed0abc405143..3f83fd5ef36db 100644 --- a/arch/riscv/include/asm/tlbflush.h +++ b/arch/riscv/include/asm/tlbflush.h @@ -66,6 +66,33 @@ void arch_tlbbatch_add_pending(struct arch_tlbflush_unma= p_batch *batch, void arch_tlbbatch_flush(struct arch_tlbflush_unmap_batch *batch); =20 extern unsigned long tlb_flush_all_threshold; + +#ifdef CONFIG_RISCV_LAZY_TLB_FLUSH + +#define MAX_LOADED_MM 6 + +struct tlb_context { + struct mm_struct *mm; + unsigned int gen; +}; + +struct tlb_info { + rwlock_t rwlock; + struct mm_struct *active_mm; + unsigned int next_gen; + struct tlb_context contexts[MAX_LOADED_MM]; +}; + +DECLARE_PER_CPU_SHARED_ALIGNED(struct tlb_info, tlbinfo); + +void local_load_tlb_mm(struct mm_struct *mm); + +#else /* CONFIG_RISCV_LAZY_TLB_FLUSH */ + +static inline void local_load_tlb_mm(struct mm_struct *mm) {} + +#endif /* CONFIG_RISCV_LAZY_TLB_FLUSH */ + #else /* CONFIG_MMU */ #define local_flush_tlb_all() do { } while (0) #endif /* CONFIG_MMU */ diff --git a/arch/riscv/mm/context.c b/arch/riscv/mm/context.c index 55c20ad1f7444..a7cf36ad34678 100644 --- a/arch/riscv/mm/context.c +++ b/arch/riscv/mm/context.c @@ -217,6 +217,7 @@ static inline void set_mm(struct mm_struct *prev, */ cpumask_set_cpu(cpu, mm_cpumask(next)); if (static_branch_unlikely(&use_asid_allocator)) { + local_load_tlb_mm(next); set_mm_asid(next, cpu); } else { cpumask_clear_cpu(cpu, mm_cpumask(prev)); diff --git a/arch/riscv/mm/tlbflush.c b/arch/riscv/mm/tlbflush.c index 8404530ec00f9..0b1c21c7aafb8 100644 --- a/arch/riscv/mm/tlbflush.c +++ b/arch/riscv/mm/tlbflush.c @@ -103,6 +103,15 @@ struct flush_tlb_range_data { unsigned long stride; }; =20 +#ifdef CONFIG_RISCV_LAZY_TLB_FLUSH +DEFINE_PER_CPU_SHARED_ALIGNED(struct tlb_info, tlbinfo) =3D { + .rwlock =3D __RW_LOCK_UNLOCKED(tlbinfo.rwlock), + .active_mm =3D NULL, + .next_gen =3D 1, + .contexts =3D { { NULL, 0, }, }, +}; +#endif /* CONFIG_RISCV_LAZY_TLB_FLUSH */ + static void __ipi_flush_tlb_range_asid(void *info) { struct flush_tlb_range_data *d =3D info; @@ -240,3 +249,60 @@ void arch_tlbbatch_flush(struct arch_tlbflush_unmap_ba= tch *batch) 0, FLUSH_TLB_MAX_SIZE, PAGE_SIZE); cpumask_clear(&batch->cpumask); } + +#ifdef CONFIG_RISCV_LAZY_TLB_FLUSH + +static inline unsigned int new_tlb_gen(struct tlb_info *info) +{ + unsigned int gen =3D info->next_gen++; + unsigned int i; + + if (unlikely(!info->next_gen)) { + for (i =3D 0; i < MAX_LOADED_MM; i++) { + if (info->contexts[i].gen) + info->contexts[i].gen =3D 1; + } + info->next_gen =3D 1; + gen =3D info->next_gen++; + } + + return gen; +} + +void local_load_tlb_mm(struct mm_struct *mm) +{ + struct tlb_info *info =3D this_cpu_ptr(&tlbinfo); + struct tlb_context *contexts =3D info->contexts; + struct mm_struct *victim =3D NULL; + unsigned int i, pos =3D 0, min =3D UINT_MAX; + + for (i =3D 0; i < MAX_LOADED_MM; i++) { + if (contexts[i].mm =3D=3D mm) { + pos =3D i; + break; + } + if (min > contexts[i].gen) { + min =3D contexts[i].gen; + pos =3D i; + } + } + + write_lock(&info->rwlock); + + info->active_mm =3D mm; + + if (contexts[pos].mm !=3D mm) { + victim =3D contexts[pos].mm; + contexts[pos].mm =3D mm; + } + contexts[pos].gen =3D new_tlb_gen(info); + + write_unlock(&info->rwlock); + + if (victim) { + cpumask_clear_cpu(raw_smp_processor_id(), mm_cpumask(victim)); + local_flush_tlb_all_asid(get_mm_asid(victim)); + } +} + +#endif /* CONFIG_RISCV_LAZY_TLB_FLUSH */ --=20 2.20.1 From nobody Mon Dec 1 22:36:56 2025 Received: from mail-pj1-f54.google.com (mail-pj1-f54.google.com [209.85.216.54]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 40744332EA5 for ; Thu, 27 Nov 2025 14:11:57 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.54 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1764252718; cv=none; b=q9Xwsdcv6G+3wlJhkzQTk+J1ywG/eYei67P/Yt0KGWFsbRctpmg4QtoyrEkLA8XORSBkqLws/RWFSnpQTg/cLtYEk0040KY9b9HdS6LuMLsEX+tU841WG5fdhb3e27j8MeKULWcidHmD+XD1aR8whgOcabMMaylpgcw2HirY3zc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1764252718; c=relaxed/simple; bh=W4Vm9ClsWQRE6Rr2tnB5g8ZJh9ErQlE4jvYs/iPj8kA=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=Xgh2ckn8+p0SI6rf0qZJPOQSg56d3IxDRrJhluv1yTk3bc4eWSPjiHSQAx/+OXt0iH3FpbEQuxNx4rILCpFDNVG1Gi/Nd2N5+5vsB1OEalUKtwbW9CJet3mWzmAggGvdxbTNe4LncdsGS4soL6BtctMFh2CLfqC+PW64M+Qi8bI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=dftJo1pa; arc=none smtp.client-ip=209.85.216.54 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="dftJo1pa" Received: by mail-pj1-f54.google.com with SMTP id 98e67ed59e1d1-34372216275so998363a91.2 for ; Thu, 27 Nov 2025 06:11:57 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1764252716; x=1764857516; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=bT1DoFK44/mxXPszG1k06ulWEZ0r0zvum0ccrRfuy/8=; b=dftJo1pa6BiywSxJA6+l3AoDntq2MAS+UbhQUQvxV7rJmbde1SWPRpy4QFWCWQApR9 TM/K+tZh9TAID6sNlETz/4w4ZoSUH+F0t28tqnDnQpxFqiZBqcafODa6QoT1T2WhDklZ HID6Tt4MOClMe67MGez/BLd3xPbVlPLS23jV6KVHV00WBsrNtA/3ZtVHjmZZ3W3ItWA4 rQmXr0OiH1hzhYDK9b0mdrd7NfnVvQi8/y6KX6GrDLQ8WKWa4Sj+1eYzJ1nZXCoEbd+k WyGSaUoaKAT5goOW+nyNvDfKGmS3qjRG4M2X1XRqwOW3tZq42H6LffJhHA6UOGYZAeb0 4WWA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1764252716; x=1764857516; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=bT1DoFK44/mxXPszG1k06ulWEZ0r0zvum0ccrRfuy/8=; b=EIxUTenUePo/PmcRijHoWlPa706nkRBlsBEYxooNv2ZsnNPxbnA6NsSc/74ixTg2qm FoyBTiscYbRxeZD7NuGVadLfofthkaJEhx8whddunPCoi6pDfv4U5NormlQkjIefeRby YR0IBesgiAOYII52GXg5nLw1X3JC0N5erZ196hxtbxaa0/dzxhOY0kvfBFbh0MNLirGG ClJZV3qZcHIRKY3GTgaNjgu2D5GQiB+br/ObBTHH2iREvNjh8u5nID8NYhg4yWlqcYRW j4zRsm6lQ3e52KYc1PVYd3fK/++9+jnvmASNZA9/u+Lo+TmqL5iWfn38oIpKlqOEGAah TsDQ== X-Forwarded-Encrypted: i=1; AJvYcCU1gSa9F8JrUnTRpUDW0qlbPGwDxVXVUoMhvVDaKjgzGag++VOsCtVuarAP2RO+MZCLXKSkSTT+vlMrTko=@vger.kernel.org X-Gm-Message-State: AOJu0Yw7BKM59rdT9XHSs9wytr07YwJCnMoE/3qF6RR7/2UfNlLLwQ4a CoiczL4fgHy4HsiKq7pRVwJQ5n8FPlseNwh0u1+4FkrflcA0uixLIk1Grwb3LvPn29o= X-Gm-Gg: ASbGnct/mZpOg2nfOMGJeLfnfwYLAePZANkk5SSD27Xj35A9i4k+AcQXw27lYS0B3iD kTwcPmsfI2gXX7/8JS9ewc3F8Bbce1MY5K4lwg6t7ZirmT3sTFf+ooayiO1Mb8fS3kyNPEd96WG i/ZVnfOxJ2JrVRn51dk98l+JAN8KrxcnGBhtkdarcgJMsgrgIJIvf8oEh0+ppn220xEyTrruDPJ NX2cds/RD9hGFma/UpKhiAamIerBdptSz/Zn/RpA7EWuQYGZdft6qvi5wIBxkCEmSC/nrRgQ3Mz 90J515/QoHrTWiWtfUwhR/DGB56eOFV7qkpqI9JaIeAacp8U2AbOiFZw7XjDWiWcEc0OJcjFwb3 Chd8mgN2fP7k4j8B2SPy41DjrrEV7FHxDC97LfejSXfnymbiWHBSX9egQpw6xFZMKfZi0jhIsRc d9WkbcEXET2CMa0aQGHTfmE+iywanjqMQ1fP5HSSG6A2KD+jP9rSDHdnKjFOprhIKth343j0qWh AWWbxKSVj41 X-Google-Smtp-Source: AGHT+IGHWS9mtIo18ykGEbjce9y2+tGQtCuaDp2x5Akdhl4bvcW7U6KnSKpb0r7Pzb23BFyTlkH5Aw== X-Received: by 2002:a17:90b:2f4b:b0:341:8bda:d0ae with SMTP id 98e67ed59e1d1-34733f2de45mr22829266a91.20.1764252716195; Thu, 27 Nov 2025 06:11:56 -0800 (PST) Received: from J9GPGXL7NT.bytedance.net ([61.213.176.58]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-3477b7341d2sm2030249a91.11.2025.11.27.06.11.50 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Thu, 27 Nov 2025 06:11:55 -0800 (PST) From: Xu Lu To: pjw@kernel.org, palmer@dabbelt.com, aou@eecs.berkeley.edu, alex@ghiti.fr, kees@kernel.org, mingo@redhat.com, peterz@infradead.org, juri.lelli@redhat.com, vincent.guittot@linaro.org, akpm@linux-foundation.org, david@redhat.com, apatel@ventanamicro.com, guoren@kernel.org Cc: linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Xu Lu Subject: [RFC PATCH v2 3/9] riscv: mm: Grab mm_count to avoid mm getting released Date: Thu, 27 Nov 2025 22:11:11 +0800 Message-ID: <20251127141117.87420-4-luxu.kernel@bytedance.com> X-Mailer: git-send-email 2.50.1 In-Reply-To: <20251127141117.87420-1-luxu.kernel@bytedance.com> References: <20251127141117.87420-1-luxu.kernel@bytedance.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" We maintain an array of mm_structs whose ASIDs are active on the current CPU. To avoid these mm_structs getting released, we grab their mm_count before loaded them into the array. And drop their mm_count via tasklet when they are evicted out of the array. Signed-off-by: Xu Lu --- arch/riscv/include/asm/mmu.h | 4 +++ arch/riscv/mm/tlbflush.c | 47 ++++++++++++++++++++++++++++++++++++ 2 files changed, 51 insertions(+) diff --git a/arch/riscv/include/asm/mmu.h b/arch/riscv/include/asm/mmu.h index cf8e6eac77d52..913fa535b3d19 100644 --- a/arch/riscv/include/asm/mmu.h +++ b/arch/riscv/include/asm/mmu.h @@ -30,6 +30,10 @@ typedef struct { #ifdef CONFIG_RISCV_ISA_SUPM u8 pmlen; #endif +#ifdef CONFIG_RISCV_LAZY_TLB_FLUSH + atomic_t lazy_tlb_cnt; + void *next; +#endif } mm_context_t; =20 /* Lock the pointer masking mode because this mm is multithreaded */ diff --git a/arch/riscv/mm/tlbflush.c b/arch/riscv/mm/tlbflush.c index 0b1c21c7aafb8..4b2ce06cbe6bd 100644 --- a/arch/riscv/mm/tlbflush.c +++ b/arch/riscv/mm/tlbflush.c @@ -104,12 +104,57 @@ struct flush_tlb_range_data { }; =20 #ifdef CONFIG_RISCV_LAZY_TLB_FLUSH + DEFINE_PER_CPU_SHARED_ALIGNED(struct tlb_info, tlbinfo) =3D { .rwlock =3D __RW_LOCK_UNLOCKED(tlbinfo.rwlock), .active_mm =3D NULL, .next_gen =3D 1, .contexts =3D { { NULL, 0, }, }, }; + +static DEFINE_PER_CPU(mm_context_t *, mmdrop_victims); + +static void mmdrop_lazy_mms(struct tasklet_struct *tasklet) +{ + mm_context_t *victim =3D xchg_relaxed(this_cpu_ptr(&mmdrop_victims), NULL= ); + struct mm_struct *mm =3D NULL; + + while (victim) { + mm =3D container_of(victim, struct mm_struct, context); + while (atomic_dec_return_relaxed(&victim->lazy_tlb_cnt) !=3D 0) + mmdrop_lazy_tlb(mm); + victim =3D victim->next; + } +} + +static DEFINE_PER_CPU(struct tasklet_struct, mmdrop_tasklets) =3D { + .count =3D ATOMIC_INIT(0), + .callback =3D mmdrop_lazy_mms, + .use_callback =3D true, +}; + +static inline void mmgrab_lazy_mm(struct mm_struct *mm) +{ + mmgrab_lazy_tlb(mm); + atomic_inc(&mm->context.lazy_tlb_cnt); +} + +static inline void mmdrop_lazy_mm(struct mm_struct *mm) +{ + mm_context_t **head, *list, *context =3D &mm->context; + + if (atomic_inc_return_relaxed(&context->lazy_tlb_cnt) =3D=3D 1) { + head =3D this_cpu_ptr(&mmdrop_victims); + + do { + list =3D *head; + context->next =3D list; + } while (cmpxchg_relaxed(head, list, context) !=3D list); + + tasklet_schedule(this_cpu_ptr(&mmdrop_tasklets)); + } +} + #endif /* CONFIG_RISCV_LAZY_TLB_FLUSH */ =20 static void __ipi_flush_tlb_range_asid(void *info) @@ -292,6 +337,7 @@ void local_load_tlb_mm(struct mm_struct *mm) info->active_mm =3D mm; =20 if (contexts[pos].mm !=3D mm) { + mmgrab_lazy_mm(mm); victim =3D contexts[pos].mm; contexts[pos].mm =3D mm; } @@ -302,6 +348,7 @@ void local_load_tlb_mm(struct mm_struct *mm) if (victim) { cpumask_clear_cpu(raw_smp_processor_id(), mm_cpumask(victim)); local_flush_tlb_all_asid(get_mm_asid(victim)); + mmdrop_lazy_mm(victim); } } =20 --=20 2.20.1 From nobody Mon Dec 1 22:36:56 2025 Received: from mail-pj1-f51.google.com (mail-pj1-f51.google.com [209.85.216.51]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5D0CA3328F2 for ; Thu, 27 Nov 2025 14:12:15 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.51 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1764252736; cv=none; b=Bn9CvO8JRDdUrgvpoXTeSiVALyxZWDriPCOy44qP6xutOUCXfFLMa1dTHMAHras8WSwNpEHl960gptR69JCY48vgST2o2YdODYbjtesb6jyhsv0VGDNfCMiaN/Hm0s7UsoPol2/VLww/47lp/7+fNqB1knR1CCmDr/CLS3gWU34= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1764252736; c=relaxed/simple; bh=CrX7CEvokCPANSuQJFMmSSRVpYnQGHR2WF5rrV0BHk8=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=doonWrltSu7QOAxuMbA7P7USnaedPiBVon79NYg3Ff0MM/IRMk5nlDq4lSBUbBvJTIH0o79bZhJD37HgWtHIAGHN6iVk2M1suvBhb9/CJr+XbDxT/ayfbhmtViBjbYRcVVVuoTVGW42dmMEVGy2lpYdQSaXKirutRRjkQnayDLs= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=L1eihxt7; arc=none smtp.client-ip=209.85.216.51 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="L1eihxt7" Received: by mail-pj1-f51.google.com with SMTP id 98e67ed59e1d1-3438d4ae152so1014024a91.1 for ; Thu, 27 Nov 2025 06:12:15 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1764252734; x=1764857534; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=f0swZ8Ypan+oXrthAo/pUrKpZWoHKJGiiGThfa6sHH8=; b=L1eihxt7aGKW18N9x251Ia77XvA2/acyMm3bvl5GLcsyW8BanHzLIXyI/rwy/KnFNP TAFCl5VKpVsm2/+0l+UmPTkr0boggu2xH900sR/0IAxhr0flmSU8/1ZXPpk1MYXIxqQC f9+r2yU0Ki2Hn3v1Ev5Rg/9oooWyEEYhUDg/chA8Tj2ZitkWfofvH6j6t0hk65bBk5hd zCD+NNIpWEqNItn/C+k4BJu9Zxim0FOwuBuuw5tEWLtR6DC0TEdDXhLDAojqFtiY7xJ9 qsrmR8m+MK7k5Kbtzdef/hHP2c2CKdEY5ehuA55mr+xL5UzOq/nx5/JwA7MDcAluScY9 X4qQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1764252734; x=1764857534; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=f0swZ8Ypan+oXrthAo/pUrKpZWoHKJGiiGThfa6sHH8=; b=Rf7PU19iYHgn/IFX2pTNPGIbtl9PM8j9qqgmzt27cNvPFOCAA6PcDs8cyQ32zI7gFF 2+dB7xuB9AR6PJd8a/7vba0v1VfZcO8mc7EXFqbzwPixzPnVXj30ramXSMsKQ4Wqgfol 5Aw4hW3QPGOmti1EXn6IG++ySex+jSksyoLUY1NcWCyQdjpldfYGM2/1nIleb+ljNmDp cU83mBw3szD1DwSekSvByzvzLz7gX3txunxjW46L3zwtJ4qmYrTnFp10rxsm4zJDdZwm ci8PeyrqA+DnXWrNo1D70TkNBXsfGMsSc8rxmCbZAHOwkqgIYZYebx4YiAZLnJ9yw777 ozyg== X-Forwarded-Encrypted: i=1; AJvYcCVy7YqK1lbjgtCSbJT0INNz26Ag3guQBuTj7TjCwE/ott1vviGDKAmlm4SaFztgtIctDoigq8jqHUhNvrk=@vger.kernel.org X-Gm-Message-State: AOJu0YyGp2GPPvRoN4DIzt7Q1hALyUqynMLKatdKcLiWjIRj0vh/gMp5 R1L7A1WKaezp2oQmAFQGR6qsYZzHJEo7r81WeTEkoBR7MF+lf8x5A6Av/ubzZ4gQY4g= X-Gm-Gg: ASbGnctKrSAVHYOcsarOwzM/W4bIubC10T0s03RtzzxQS+6oDDFFBwqRqXS9DaFvjc1 Ulrq8K3u/kVBtcl4ePjopOW0kgZnRQxOBAKxuC/0hv7GARtzhdVxHraSbdCN9BiFc1GP0Lwxs82 OW8JZcpgOOzQCaTEzo+28DdWB1Nr7RUcrMg/Gxyxnn7Ok0aYlj39R0V4WA4CWE64cqJnUfChSFZ 3xrNZhuRkCzoPgkog14PhBxDjTiEAurTYbESrsdQGg6LFqTax3/JrhZPOA2uuHhuvjM2EZW8b+k zweA/bC05vSXzLvxqQwLgyzdAcuMdUTQBiX2pSiLm/CW4CQySTOHrQrtJdc5niJvzd4YtH6beOt 9P7GKwEjHyf6IVelJuOfJ+5IXkvCZG1x/WJd1wMuxNp0pn1DkNfW2yhJZt440De3h/93V2neVRz aAnMNcKM1NJ0dkSExW0mhXSLO2AKCQAWia75Utd4z1Mv4+yhIJez42dzb0C3Xo8/FShQsxBLyGd A== X-Google-Smtp-Source: AGHT+IHkhvWqjx6FEHq30tTA44a11zGXZ1BYveiPwreNaqri3rljB5haB7+w3kfLn/irkM5FUcsJKg== X-Received: by 2002:a17:90b:35cc:b0:33b:cfac:d5c6 with SMTP id 98e67ed59e1d1-34733f487f1mr23197433a91.29.1764252734477; Thu, 27 Nov 2025 06:12:14 -0800 (PST) Received: from J9GPGXL7NT.bytedance.net ([61.213.176.58]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-3477b7341d2sm2030249a91.11.2025.11.27.06.11.57 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Thu, 27 Nov 2025 06:12:14 -0800 (PST) From: Xu Lu To: pjw@kernel.org, palmer@dabbelt.com, aou@eecs.berkeley.edu, alex@ghiti.fr, kees@kernel.org, mingo@redhat.com, peterz@infradead.org, juri.lelli@redhat.com, vincent.guittot@linaro.org, akpm@linux-foundation.org, david@redhat.com, apatel@ventanamicro.com, guoren@kernel.org Cc: linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Xu Lu Subject: [RFC PATCH v2 4/9] fork: Add arch override for do_shoot_lazy_tlb() Date: Thu, 27 Nov 2025 22:11:12 +0800 Message-ID: <20251127141117.87420-5-luxu.kernel@bytedance.com> X-Mailer: git-send-email 2.50.1 In-Reply-To: <20251127141117.87420-1-luxu.kernel@bytedance.com> References: <20251127141117.87420-1-luxu.kernel@bytedance.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" When both CONFIG_RISCV_LAZY_TLB_FLUSH and CONFIG_MMU_LSYZ_TLB_SHOOTDOWN is enabled, riscv needs an arch special method to free the mm that needs to be shot down. Thus we add arch override for do_shoot_lazy_tlb(). Signed-off-by: Xu Lu --- kernel/fork.c | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/kernel/fork.c b/kernel/fork.c index 3da0f08615a95..b6d11acd6ac10 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -612,7 +612,8 @@ static void do_check_lazy_tlb(void *arg) WARN_ON_ONCE(current->active_mm =3D=3D mm); } =20 -static void do_shoot_lazy_tlb(void *arg) +#ifndef arch_do_shoot_lazy_tlb +static void arch_do_shoot_lazy_tlb(void *arg) { struct mm_struct *mm =3D arg; =20 @@ -622,6 +623,7 @@ static void do_shoot_lazy_tlb(void *arg) switch_mm(mm, &init_mm, current); } } +#endif =20 static void cleanup_lazy_tlbs(struct mm_struct *mm) { @@ -661,7 +663,7 @@ static void cleanup_lazy_tlbs(struct mm_struct *mm) * - A delayed freeing and RCU-like quiescing sequence based on mm * switching to avoid IPIs completely. */ - on_each_cpu_mask(mm_cpumask(mm), do_shoot_lazy_tlb, (void *)mm, 1); + on_each_cpu_mask(mm_cpumask(mm), arch_do_shoot_lazy_tlb, (void *)mm, 1); if (IS_ENABLED(CONFIG_DEBUG_VM_SHOOT_LAZIES)) on_each_cpu(do_check_lazy_tlb, (void *)mm, 1); } --=20 2.20.1 From nobody Mon Dec 1 22:36:56 2025 Received: from mail-pj1-f53.google.com (mail-pj1-f53.google.com [209.85.216.53]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C7C59221D9E for ; Thu, 27 Nov 2025 14:12:26 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.53 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1764252748; cv=none; b=UXNitXANvoOxNz8UBAqTSs2TaL3FprF/pv1fCvUVx50zRbICID4oQ9V1j9F8JhWrgSvTL4p5Br/+S2j49m0t2OnRgpIgAOXVjJ750h8Bo/04ECIVtNwB7S4LFZuW8fcHThjnyZDi8aOw8jZ2NBwFlcR8mVQk5IaE9uBHuRBsJz8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1764252748; c=relaxed/simple; bh=7/P0WM8LP0Pzq4LXZ2Bwjb/hu4Jof6j4zAmCpiWAKvk=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=XzuDMTDEHE8JP5EK5XUzflGggH71MJSa6deVCakMneie0/8Gb04mTtZIvuHbQCalCblZGTTDp17tBR/zHZSusXvX6v2g6oUHLnxEKMJaI0KhYzid0IKcUNmRR9vfCilQb8jj5pYk2nLxYXzAj0Px+Lhahax70AFKla4UWA1uxsg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=BsB5hVGx; arc=none smtp.client-ip=209.85.216.53 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="BsB5hVGx" Received: by mail-pj1-f53.google.com with SMTP id 98e67ed59e1d1-3438231df5fso1142464a91.2 for ; Thu, 27 Nov 2025 06:12:26 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1764252746; x=1764857546; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=kCE7BFaxTmXT5gX1mJT6mv3jSbzO4IVr04XlUFHKTqA=; b=BsB5hVGxOFNIw63W43oZL2uGQcxWTrjoy74YVWf0Y7bSLlhRPqrSqcR/E4ggtaHxw9 hgxBeDQ6kjYv9osax1PqJIF3HVkoychdodb2NXolc2RVL1Lh807yeijls2jZvDDTX+9a S/fJnZqhADf0IliFcKrQaPmys5QL3cEkWZJFweoOjib2gOhsn6SALfdrnxOPnzkDT187 a/5/1BAfHQGDeCecae6Xuf58nmiZmzKwXi/XP1dgp2KDVsUJjEVjqYaM/MxVwfqIElQr PTnxoqZIodXTKVZaJqG3VFLQn/w1QwaEB2SZzg+IIKkqc44SuE7S9hwmWf1cucTaPZNA GRpQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1764252746; x=1764857546; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=kCE7BFaxTmXT5gX1mJT6mv3jSbzO4IVr04XlUFHKTqA=; b=UhxjiVNYDfx75CEzh7s7Fd3ZUCNYfc9h+FW6Y9F4b2KL+PkpciN3nX/XGZA2dlvxCU h5aTwSQiYLLXd3eFdllE4lsKM4UDDj2BStw/w3C6srsS9Z8eQLuIhY7FMAIzEV0ay0BO 2soBFfquwJ+7uuDw9tAteuEw8Yo/sGdzKtZOVNPc0TBfea6Q8y/nAqgKn7X09LisLYM2 DtP3wJN0PkUM1lYLjTunD/t3dEpQI/kqc3DW8Pehm1RDNxz9AqNGrYqm0RZpBlrl2yDf LOXg6U+xuJvR12K25TmdWX61zME7ZhRRF393KHbAiJZD0X3BaLVYcq6JIqEmvJuZb+wv GVqQ== X-Forwarded-Encrypted: i=1; AJvYcCUp2SP+Yf7ioCjOVKt8dUE0kw+scoUOKjsylVsJAAPZUGoo+URlvKf90LBlALJmaIqNkePl3Ia+9d2XRpY=@vger.kernel.org X-Gm-Message-State: AOJu0YzJVu66LJ9NDmVID49YV2HXEBfQaiNDk6RM47vAHNfXSixJhTyl rUQUCTbcnAezjaON/AE8ovWTKfbk8XJ96YbeYYokQM08VsZ7A6B9k9Q7jadNHblQZAk= X-Gm-Gg: ASbGncscfIDraQsGuAV4mb6wg3SQv2J6Kb3VrGRRrh4my+kjJefQZoxPAdFehsnNyy2 nkK2aUZf3aeoEoA+XGdEyjBzIuoFNm3wj1tRi1BOSy4fRXdbvLR5kRq47+JlHX1GQvIG7V+iJwg l2YZAlO57oVrTG+B/Q58nMxpKr0YMvrK1MY3XEAVUcVC6wHeMWiALHN87cVZVQ1VcQVlZmtM445 6/tJ3suLqtjesJ3/+kjRBceN7YZrSrPGTfvoVNZ7JxB2WrpSXRlQ9p+UJDYHHY/4ONkN1/k/Iu6 yKLdiF746cw/Aoidzm+M7eLT1nGoJam1HUcrRHEfI3dUyvMgRGgmfR48Cn3XqDPNGHs3XCQWJ1B GTwpykNLEmHiFcfw2N/SA1zo9k9Ousf34TwFQ4vbsFp53Z/0A3FEFItH65WKsp3S0r1/hmP8qfl IbWIu8qXo6YojDYsges+O+cWsyZOf3KTGqKfnV8rNdZKC5WJxN48MNZQoxWHs868wcJ46yW7UYX ZqcaaRdUmcj X-Google-Smtp-Source: AGHT+IHaJbhR6uoA8AsgyvN1rZkRokZKcV5EN92G/Fd+krItmdJONN0XD0mHkXfOugo46//aXXPl7A== X-Received: by 2002:a17:90b:1a84:b0:32b:65e6:ec48 with SMTP id 98e67ed59e1d1-3475ebd2f41mr10628835a91.8.1764252745954; Thu, 27 Nov 2025 06:12:25 -0800 (PST) Received: from J9GPGXL7NT.bytedance.net ([61.213.176.58]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-3477b7341d2sm2030249a91.11.2025.11.27.06.12.15 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Thu, 27 Nov 2025 06:12:25 -0800 (PST) From: Xu Lu To: pjw@kernel.org, palmer@dabbelt.com, aou@eecs.berkeley.edu, alex@ghiti.fr, kees@kernel.org, mingo@redhat.com, peterz@infradead.org, juri.lelli@redhat.com, vincent.guittot@linaro.org, akpm@linux-foundation.org, david@redhat.com, apatel@ventanamicro.com, guoren@kernel.org Cc: linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Xu Lu Subject: [RFC PATCH v2 5/9] riscv: mm: Introduce arch_do_shoot_lazy_tlb Date: Thu, 27 Nov 2025 22:11:13 +0800 Message-ID: <20251127141117.87420-6-luxu.kernel@bytedance.com> X-Mailer: git-send-email 2.50.1 In-Reply-To: <20251127141117.87420-1-luxu.kernel@bytedance.com> References: <20251127141117.87420-1-luxu.kernel@bytedance.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" When an active_mm is shot down, we switch it to the init_mm, evict it out of percpu active mm array. Signed-off-by: Xu Lu --- arch/riscv/include/asm/mmu_context.h | 5 ++++ arch/riscv/include/asm/tlbflush.h | 11 +++++++++ arch/riscv/mm/context.c | 19 ++++++++++++++++ arch/riscv/mm/tlbflush.c | 34 ++++++++++++++++++++++++---- 4 files changed, 64 insertions(+), 5 deletions(-) diff --git a/arch/riscv/include/asm/mmu_context.h b/arch/riscv/include/asm/= mmu_context.h index 8c4bc49a3a0f5..bc73cc3262ae6 100644 --- a/arch/riscv/include/asm/mmu_context.h +++ b/arch/riscv/include/asm/mmu_context.h @@ -16,6 +16,11 @@ void switch_mm(struct mm_struct *prev, struct mm_struct *next, struct task_struct *task); =20 +#ifdef CONFIG_RISCV_LAZY_TLB_FLUSH +#define arch_do_shoot_lazy_tlb arch_do_shoot_lazy_tlb +void arch_do_shoot_lazy_tlb(void *arg); +#endif + #define activate_mm activate_mm static inline void activate_mm(struct mm_struct *prev, struct mm_struct *next) diff --git a/arch/riscv/include/asm/tlbflush.h b/arch/riscv/include/asm/tlb= flush.h index 3f83fd5ef36db..e7365a53265a6 100644 --- a/arch/riscv/include/asm/tlbflush.h +++ b/arch/riscv/include/asm/tlbflush.h @@ -15,6 +15,11 @@ #define FLUSH_TLB_NO_ASID ((unsigned long)-1) =20 #ifdef CONFIG_MMU +static inline unsigned long get_mm_asid(struct mm_struct *mm) +{ + return mm ? cntx2asid(atomic_long_read(&mm->context.id)) : FLUSH_TLB_NO_A= SID; +} + static inline void local_flush_tlb_all(void) { __asm__ __volatile__ ("sfence.vma" : : : "memory"); @@ -86,11 +91,17 @@ struct tlb_info { DECLARE_PER_CPU_SHARED_ALIGNED(struct tlb_info, tlbinfo); =20 void local_load_tlb_mm(struct mm_struct *mm); +void local_flush_tlb_mm(struct mm_struct *mm); =20 #else /* CONFIG_RISCV_LAZY_TLB_FLUSH */ =20 static inline void local_load_tlb_mm(struct mm_struct *mm) {} =20 +static inline void local_flush_tlb_mm(struct mm_struct *mm) +{ + local_flush_tlb_all_asid(get_mm_asid(mm)); +} + #endif /* CONFIG_RISCV_LAZY_TLB_FLUSH */ =20 #else /* CONFIG_MMU */ diff --git a/arch/riscv/mm/context.c b/arch/riscv/mm/context.c index a7cf36ad34678..3335080e5f720 100644 --- a/arch/riscv/mm/context.c +++ b/arch/riscv/mm/context.c @@ -274,6 +274,25 @@ static int __init asids_init(void) return 0; } early_initcall(asids_init); + +#ifdef CONFIG_RISCV_LAZY_TLB_FLUSH +void arch_do_shoot_lazy_tlb(void *arg) +{ + struct mm_struct *mm =3D arg; + + if (current->active_mm =3D=3D mm) { + WARN_ON_ONCE(current->mm); + current->active_mm =3D &init_mm; + switch_mm(mm, &init_mm, current); + } + + if (!static_branch_unlikely(&use_asid_allocator) || !mm) + return; + + local_flush_tlb_mm(mm); +} +#endif /* CONFIG_RISCV_LAZY_TLB_FLUSH */ + #else static inline void set_mm(struct mm_struct *prev, struct mm_struct *next, unsigned int cpu) diff --git a/arch/riscv/mm/tlbflush.c b/arch/riscv/mm/tlbflush.c index 4b2ce06cbe6bd..a47bacf5801ab 100644 --- a/arch/riscv/mm/tlbflush.c +++ b/arch/riscv/mm/tlbflush.c @@ -164,11 +164,6 @@ static void __ipi_flush_tlb_range_asid(void *info) local_flush_tlb_range_asid(d->start, d->size, d->stride, d->asid); } =20 -static inline unsigned long get_mm_asid(struct mm_struct *mm) -{ - return mm ? cntx2asid(atomic_long_read(&mm->context.id)) : FLUSH_TLB_NO_A= SID; -} - static void __flush_tlb_range(struct mm_struct *mm, const struct cpumask *cmask, unsigned long start, unsigned long size, @@ -352,4 +347,33 @@ void local_load_tlb_mm(struct mm_struct *mm) } } =20 +void local_flush_tlb_mm(struct mm_struct *mm) +{ + struct tlb_info *info =3D this_cpu_ptr(&tlbinfo); + struct tlb_context *contexts =3D info->contexts; + unsigned long asid =3D get_mm_asid(mm); + unsigned int i; + + if (!mm || mm =3D=3D info->active_mm) { + local_flush_tlb_all_asid(asid); + return; + } + + for (i =3D 0; i < MAX_LOADED_MM; i++) { + if (contexts[i].mm !=3D mm) + continue; + + write_lock(&info->rwlock); + contexts[i].mm =3D NULL; + contexts[i].gen =3D 0; + write_unlock(&info->rwlock); + + cpumask_clear_cpu(raw_smp_processor_id(), mm_cpumask(mm)); + mmdrop_lazy_mm(mm); + break; + } + + local_flush_tlb_all_asid(asid); +} + #endif /* CONFIG_RISCV_LAZY_TLB_FLUSH */ --=20 2.20.1 From nobody Mon Dec 1 22:36:56 2025 Received: from mail-pf1-f178.google.com (mail-pf1-f178.google.com [209.85.210.178]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E576433290F for ; Thu, 27 Nov 2025 14:12:37 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.178 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1764252759; cv=none; b=sXKJK9Xns8Czjz/Q5tmYkTaZo94tlI+sbb+v8zTe9A/i5CNxzuzfqwxyXIpJVnGsT7MvXIugoxVEtjVnBUgooxOf2gcgH6ypBwqb+3XPUr55t7gVPHxPagCbJQyWJuEsvkPXIMpGReoZmyjg4D7gRJyWn+zS+9xKHM3YS4Ip6Lo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1764252759; c=relaxed/simple; bh=K/lO+vh7mnLcg0Ur+IKYTaxb8wzEl8qPzjU81LLXgJA=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=W8D4j1Ca2DLfD8y06GJjL/1hu+yejp5tGxDN/810Ve5mQGI+/gZemDT4dcOCzb66Q7oXC3irHCNdjgwgEZ2i8YNLV/k2AyQWio2tJScwh0gTFgWxgqWxfMML4jK0xCK4EBXVgxJGqeUAGe0ahF/Hg/+WWYlz2Ov23tpzhwbzYPo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=ECp05fti; arc=none smtp.client-ip=209.85.210.178 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="ECp05fti" Received: by mail-pf1-f178.google.com with SMTP id d2e1a72fcca58-794e300e20dso1065443b3a.1 for ; Thu, 27 Nov 2025 06:12:37 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1764252757; x=1764857557; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Zl77PqgzhSQnbli+DMk7tCeC4XrJE4OWFtNvT9IO89M=; b=ECp05ftinIy9U4jB0J9TnaFwgiZVqkdPatM/RxnhIIl3j9k+WDctH3iRnmjUgyuz9K zmkyNfMvNOoDp1XdxcF7RRRgDgU0oh7UMkve9eo/ns1CATNHTIdWIpW33WnRexntOXb7 0bYH13sBtoNlkCSy9lREnwlqjUPxPiIq2DYSLwSlrjESuM1xFcPzj/WW6WLBmd/m0VPs I1nR5JyS+KvODUxgNhcqv9+IvKjQm1Bf6wRpBCSuJ01ftvJir2MrP7YugEH7bA3Z38BN jAcmL82mRoeFZ8iRRdxc8LTgT4tcOj+b2IcUjrHqw7kCSfYrunhdKiFZdpAkvq7ojjmk DUfA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1764252757; x=1764857557; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=Zl77PqgzhSQnbli+DMk7tCeC4XrJE4OWFtNvT9IO89M=; b=tOL7h50uGnMulOtsaoI0dNzicKI/Oa3AjwBDY26PMH5r1v3znqegPQa6YM/15SWfVN KLPsWH4gfZthMIWknrH+J5ZvNVeUO/p5wio2dgtZhWY9TyxuCPs1pzgG3NtYpm2ecodE miLPvs8B+Y7cXTW/kLFpMrxOG4vvSUFokB9NsZhaAw8d9g95tov7ZRWETXZsSIcCKijD mVLztoakPTxFjo0SHstiA4A365jbiUVHSSG8zbY0r4HygCWVozPk9CXsdDJgA7sD0Lxd qV+Gtpj+jZysStbVenh4jUjzsJz7zFJP2nH7IGlOPP4o6BXqFl3mU9svwHNAgk4Rf0kI t8VA== X-Forwarded-Encrypted: i=1; AJvYcCU1jLsx3ojkgHZJgH+TG/0j9k1/R6328nyCqTU46yXL1I4FIkMnnBgxC4/jMbyOCB9upDq4b6toPAOtzqE=@vger.kernel.org X-Gm-Message-State: AOJu0YyS1Nc3K+sYWe/6JzI9D9illiaj3hhuzV2vb2FZhWi6bejnXSmX AR1uuiUfm21skjfsTKwAj8//+KWvRXfkyUTM3D1QgNCIBXgcVgnCvn7BXENdTECiuLA= X-Gm-Gg: ASbGncvQ4JnKdmpgweI5/U5EGexIY5dwQ2LK4OtF1JNJcAc1VxA1lTYwFhgw0Oe+FsF iE4hZMu6LUwt89itv7tmtKXQTiaVhwV6rLm3uFzmtVSo3EfiCQxf04rrtdXM8rHsDi8bg2c2Dyz 5OWSjywwUOrcq20APKc2jrBnngaPe7rEgsKTvqmw9xMRXYetlMccUnKQ/WBN8ZYjjt1C+K8deGe XMzqLea4PBEwWPA2cfy8f7+/dCalzNuLf+0iKt4hWbqMPqNwZfcgKfNMvvp7Fh7r+5Wtotg1KW7 spPG2c52PtdIeN8VmCppd8KeYpOv1+hPo83S7YVeHv6e73mwlPg2ubPUSlpiE6SUEUfL0XSb/p6 GupudpbFfJr4EVPPpLWGGfFCs5fhwQzPduX3vz3LCEa80/FXqh3fXsximkZnFejy1H794E8yYId W1ZaTuC9MFseN/aRJ/Msx+awokpo92rg+qyeTLJcRLQxYFHs9LILg5nGpZyVOejHA5tTATE5W9T g== X-Google-Smtp-Source: AGHT+IFs53VTIZhomsFYAowx9J26TGi3f3ksQWsE5AJKE20aCMEW51a+o1FMLwfvpm4UU6skwP1mVg== X-Received: by 2002:a17:90b:38ca:b0:32e:23c9:6f41 with SMTP id 98e67ed59e1d1-347331be358mr21956360a91.5.1764252756999; Thu, 27 Nov 2025 06:12:36 -0800 (PST) Received: from J9GPGXL7NT.bytedance.net ([61.213.176.58]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-3477b7341d2sm2030249a91.11.2025.11.27.06.12.26 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Thu, 27 Nov 2025 06:12:36 -0800 (PST) From: Xu Lu To: pjw@kernel.org, palmer@dabbelt.com, aou@eecs.berkeley.edu, alex@ghiti.fr, kees@kernel.org, mingo@redhat.com, peterz@infradead.org, juri.lelli@redhat.com, vincent.guittot@linaro.org, akpm@linux-foundation.org, david@redhat.com, apatel@ventanamicro.com, guoren@kernel.org Cc: linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Xu Lu Subject: [RFC PATCH v2 6/9] riscv: mm: Introduce percpu TLB Flush queue Date: Thu, 27 Nov 2025 22:11:14 +0800 Message-ID: <20251127141117.87420-7-luxu.kernel@bytedance.com> X-Mailer: git-send-email 2.50.1 In-Reply-To: <20251127141117.87420-1-luxu.kernel@bytedance.com> References: <20251127141117.87420-1-luxu.kernel@bytedance.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" When memory mapping of a mm is modified, instead of sending IPI to all CPUs recorded in its mm_cpumask, we check whether each target CPU is using this mm right now. If not, we just store the TLB Flush information in target CPU's percpu TLB Flush queue, avoiding the IPI. Signed-off-by: Xu Lu --- arch/riscv/include/asm/tlbflush.h | 19 +++++++++ arch/riscv/mm/context.c | 2 + arch/riscv/mm/tlbflush.c | 71 ++++++++++++++++++++++++++++++- 3 files changed, 90 insertions(+), 2 deletions(-) diff --git a/arch/riscv/include/asm/tlbflush.h b/arch/riscv/include/asm/tlb= flush.h index e7365a53265a6..c9630267c58cd 100644 --- a/arch/riscv/include/asm/tlbflush.h +++ b/arch/riscv/include/asm/tlbflush.h @@ -75,23 +75,40 @@ extern unsigned long tlb_flush_all_threshold; #ifdef CONFIG_RISCV_LAZY_TLB_FLUSH =20 #define MAX_LOADED_MM 6 +#define MAX_TLB_FLUSH_TASK 32 +#define FLUSH_TLB_ALL_ASID 0x1 =20 struct tlb_context { struct mm_struct *mm; unsigned int gen; + bool need_flush; }; =20 +struct tlb_flush_task { + unsigned long start; + unsigned long size; + unsigned long stride; +}; + +struct tlb_flush_queue { + atomic_t len; + unsigned int flag; + struct tlb_flush_task tasks[MAX_TLB_FLUSH_TASK]; +} ____cacheline_aligned_in_smp; + struct tlb_info { rwlock_t rwlock; struct mm_struct *active_mm; unsigned int next_gen; struct tlb_context contexts[MAX_LOADED_MM]; + struct tlb_flush_queue *flush_queues; }; =20 DECLARE_PER_CPU_SHARED_ALIGNED(struct tlb_info, tlbinfo); =20 void local_load_tlb_mm(struct mm_struct *mm); void local_flush_tlb_mm(struct mm_struct *mm); +void __init lazy_tlb_flush_init(void); =20 #else /* CONFIG_RISCV_LAZY_TLB_FLUSH */ =20 @@ -102,6 +119,8 @@ static inline void local_flush_tlb_mm(struct mm_struct = *mm) local_flush_tlb_all_asid(get_mm_asid(mm)); } =20 +static inline void lazy_tlb_flush_init(void) {} + #endif /* CONFIG_RISCV_LAZY_TLB_FLUSH */ =20 #else /* CONFIG_MMU */ diff --git a/arch/riscv/mm/context.c b/arch/riscv/mm/context.c index 3335080e5f720..c381c4ed46bfb 100644 --- a/arch/riscv/mm/context.c +++ b/arch/riscv/mm/context.c @@ -263,6 +263,8 @@ static int __init asids_init(void) =20 __set_bit(0, context_asid_map); =20 + lazy_tlb_flush_init(); + static_branch_enable(&use_asid_allocator); =20 pr_info("ASID allocator using %lu bits (%lu entries)\n", diff --git a/arch/riscv/mm/tlbflush.c b/arch/riscv/mm/tlbflush.c index a47bacf5801ab..b5a2d9874d62b 100644 --- a/arch/riscv/mm/tlbflush.c +++ b/arch/riscv/mm/tlbflush.c @@ -97,6 +97,7 @@ void flush_tlb_all(void) } =20 struct flush_tlb_range_data { + struct mm_struct *mm; unsigned long asid; unsigned long start; unsigned long size; @@ -109,7 +110,8 @@ DEFINE_PER_CPU_SHARED_ALIGNED(struct tlb_info, tlbinfo)= =3D { .rwlock =3D __RW_LOCK_UNLOCKED(tlbinfo.rwlock), .active_mm =3D NULL, .next_gen =3D 1, - .contexts =3D { { NULL, 0, }, }, + .contexts =3D { { NULL, 0, false, }, }, + .next_gen =3D 0, }; =20 static DEFINE_PER_CPU(mm_context_t *, mmdrop_victims); @@ -155,6 +157,47 @@ static inline void mmdrop_lazy_mm(struct mm_struct *mm) } } =20 +static bool should_ipi_flush(int cpu, void *data) +{ + struct tlb_info *info =3D per_cpu_ptr(&tlbinfo, cpu); + struct tlb_context *contexts =3D info->contexts; + struct tlb_flush_queue *queue =3D NULL; + struct flush_tlb_range_data *ftd =3D data; + unsigned int i, index; + unsigned long flags; + + if (info->active_mm =3D=3D ftd->mm) + return true; + + read_lock_irqsave(&info->rwlock, flags); + + if (info->active_mm =3D=3D ftd->mm) { + read_unlock_irqrestore(&info->rwlock, flags); + return true; + } + + for (i =3D 0; i < MAX_LOADED_MM; i++) { + if (contexts[i].mm !=3D ftd->mm) + continue; + + queue =3D &info->flush_queues[i]; + index =3D atomic_fetch_add_unless(&queue->len, 1, MAX_TLB_FLUSH_TASK); + if (index < MAX_TLB_FLUSH_TASK) { + queue->tasks[index].start =3D ftd->start; + queue->tasks[index].stride =3D ftd->stride; + queue->tasks[index].size =3D ftd->size; + } else { + queue->flag |=3D FLUSH_TLB_ALL_ASID; + } + contexts[i].need_flush =3D true; + break; + } + + read_unlock_irqrestore(&info->rwlock, flags); + + return false; +} + #endif /* CONFIG_RISCV_LAZY_TLB_FLUSH */ =20 static void __ipi_flush_tlb_range_asid(void *info) @@ -185,11 +228,20 @@ static void __flush_tlb_range(struct mm_struct *mm, } else { struct flush_tlb_range_data ftd; =20 + ftd.mm =3D mm; ftd.asid =3D asid; ftd.start =3D start; ftd.size =3D size; ftd.stride =3D stride; - on_each_cpu_mask(cmask, __ipi_flush_tlb_range_asid, &ftd, 1); +#ifdef CONFIG_RISCV_LAZY_TLB_FLUSH + if (static_branch_unlikely(&use_asid_allocator) && mm) + on_each_cpu_cond_mask(should_ipi_flush, + __ipi_flush_tlb_range_asid, + &ftd, 1, cmask); + else +#endif + on_each_cpu_mask(cmask, __ipi_flush_tlb_range_asid, + &ftd, 1); } =20 put_cpu(); @@ -376,4 +428,19 @@ void local_flush_tlb_mm(struct mm_struct *mm) local_flush_tlb_all_asid(asid); } =20 +void __init lazy_tlb_flush_init(void) +{ + struct tlb_flush_queue *queue; + unsigned int cpu, size; + + size =3D MAX_LOADED_MM * sizeof(struct tlb_flush_queue); + for_each_possible_cpu(cpu) { + queue =3D kzalloc_node(size, GFP_KERNEL, cpu_to_node(cpu)); + if (!queue) + panic("Failed to alloc per cpu tlb flush queue\n"); + + per_cpu(tlbinfo, cpu).flush_queues =3D queue; + } +} + #endif /* CONFIG_RISCV_LAZY_TLB_FLUSH */ --=20 2.20.1 From nobody Mon Dec 1 22:36:56 2025 Received: from mail-pj1-f45.google.com (mail-pj1-f45.google.com [209.85.216.45]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6D79A332EA7 for ; Thu, 27 Nov 2025 14:12:43 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.45 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1764252764; cv=none; b=D0wIX41+wLW0o3yR1mNFSYXlxQSuHJqieEbmDCQqJevcM9spaUoRq+DOGlz+hqHLWLMEijFqa0NxDW7y0vzDGibiN8AIt/AroZGTYm8kKhXYwhtdZvaAfe/BTzueC9abLKWKlohndqIYgSXBAxEhLsNG9WJHRuHzNG5hPdVY4Ek= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1764252764; c=relaxed/simple; bh=YBQ9qQcs6LlRk0123o2O2ePLMo1oSFfOTRC6L0Zkbk4=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=DgNahY2PSD9tk6LH7md7FIPULgAIvjYKfZbb+PLYnO8HYLwAhIBvXMLkpnNLmlutAKEJ8qFlR558/50Dg7tEdIJ99YSZ/9xCrYW/IzxHB5zH/IYoXOVBatsuZMX8I4BHxENnIJsTVg6UcFKo2tOhvs9h8GbH3oey/ZaRpUwFJiQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=RPCFH1Y3; arc=none smtp.client-ip=209.85.216.45 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="RPCFH1Y3" Received: by mail-pj1-f45.google.com with SMTP id 98e67ed59e1d1-343ff854297so1152909a91.1 for ; Thu, 27 Nov 2025 06:12:43 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1764252763; x=1764857563; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=7glhrA2/bhcJ3C4rVIYMtSTbP/A3a+HxT2eJKeURbio=; b=RPCFH1Y3u1sR+Nmn4QJNKNsJINHIn+fdzANhRz4CXDfeBLddD9PWbBAACwJrOTdMHj AS2tjBAPkvEBAI4hriV+eOXNC7St0HL9Jr1ukKLmzMgUIAfiewAvRCOT7+t5/oq8HUTB opQTkfEMZrxB1GPtXQ6zD+qZbJY//GjG1j/+XQUaiGR+g5BXKYu+ddMjKq66IzuLrtzV 7ZIJVFL90UAKXKx7NnLHGcfo4OgYCCDBi4kIFg6vtP81TDka5IbmCh4QjdW+Zj2m9oix KSnbZsYkP8WP9EmNtOpFia8H/4RFDU7I98RoYYA9VWE5DxyVTIp44UIYaYhuVB2SQTwd LOyw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1764252763; x=1764857563; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=7glhrA2/bhcJ3C4rVIYMtSTbP/A3a+HxT2eJKeURbio=; b=Msx0eu8ZoeyKoqHp/EAKGQzYubdXPOunhuxo5gumN92punZRHI1gMvbQY91TjXXID2 vJB7jfMZJB62fvsGm4HYc356MdTBRCg4sVQdVd61Hzb52ufln4LwX/UT1lMGYEMtLRjk Ytl8z81/IMHY1ANMdgCBXScyM/8h2JIWxoMT59A29dJh5qg298Jr68fs6Cdc7ZfUuX+r l1MUzlnfBCx/tkyNn0YJ3g/96e4A5jt40fnIwR32fU9wj9heL9V6KjSoMuTiLtbc+cff GPT2+BZWi0oRrGoSSaQsSY7M8fNYfl8sKRc4b2lARP8QP/1bAUM9H2ZdLwr+CGH1a96V EL4A== X-Forwarded-Encrypted: i=1; AJvYcCW/i3yuwzZsd786Lhcrps4Udb31jXmxZm7WCJCr1z7d0xf6sVJrExrv2JwzRURWRDF3pKwnElXgoJMRi40=@vger.kernel.org X-Gm-Message-State: AOJu0Yz86mbesT/8xFJx4zWEJF8Z5Be56B7p+3z/w7pf+u3NLJz/pyUh WnC3iLBF892GuzTbL7FkJCxbruYzSyUW2uXL2VDfCalrmdwatsw2rJe/GiIEBopVIH4= X-Gm-Gg: ASbGncvUBo31iHRGm3YHs10pqEmbHbp6f9dNO3aku7Cj+i2PtfAM4FmkQ8qVim4zKZ3 Ef2GWs+xCO80X8eDwz6Vxp7P5eLLMEsxBAm4sgRps9EaNsgZLGuHIEqHcPXMmgHf8xwsRisN9v9 hK9/NfQuW1rR3WpDYnCxAQtdpy4rr9SsEMXI0CVEC+5O3F8rUekEjJuBI4DW+15W8YrKcAQzHJb nGaBCYlPHUAHo81LQ67DHcAjxPRNWecgTddxkFvVRgPxkWQRHwWWEyDQ5OVkqzTeb5YKI6Vcv2g 3ZuY0Wxnl5bfHeNHYJxrJ7wndib3MzFFecXXVO4WJw+3Xiw1QwVVevjNRg/xjL4UtMTCCVIK8ub zADzdjqvAuTljSQtMGw78n7QyBogaNVccKx7ySffxFYFW4kX+JKJsHTCe59BRorIkX7U/fSA9Mv 9TbqSe7AoBtjzueyiC/2kd0O04OBseUlYIPtZWL1e0RXPtdekc+RueiTv1M0iUu9CpARhqvxPQI w== X-Google-Smtp-Source: AGHT+IGJodAk35vvTSCefFgpwaOAVIMesvU7l7N866cJmylZ3F8YVFLaZ662o0Sa9lutX8QW3ovQ6w== X-Received: by 2002:a17:90b:3502:b0:32e:a5ae:d00 with SMTP id 98e67ed59e1d1-34733e72350mr22757635a91.13.1764252762612; Thu, 27 Nov 2025 06:12:42 -0800 (PST) Received: from J9GPGXL7NT.bytedance.net ([61.213.176.58]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-3477b7341d2sm2030249a91.11.2025.11.27.06.12.37 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Thu, 27 Nov 2025 06:12:42 -0800 (PST) From: Xu Lu To: pjw@kernel.org, palmer@dabbelt.com, aou@eecs.berkeley.edu, alex@ghiti.fr, kees@kernel.org, mingo@redhat.com, peterz@infradead.org, juri.lelli@redhat.com, vincent.guittot@linaro.org, akpm@linux-foundation.org, david@redhat.com, apatel@ventanamicro.com, guoren@kernel.org Cc: linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Xu Lu Subject: [RFC PATCH v2 7/9] riscv: mm: Defer the TLB Flush to switch_mm Date: Thu, 27 Nov 2025 22:11:15 +0800 Message-ID: <20251127141117.87420-8-luxu.kernel@bytedance.com> X-Mailer: git-send-email 2.50.1 In-Reply-To: <20251127141117.87420-1-luxu.kernel@bytedance.com> References: <20251127141117.87420-1-luxu.kernel@bytedance.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Since the TLB Flush IPI is avoided when the modified mm is not running on the target CPU, next time when target CPU switches to the modified mm, it has to check percpu TLB Flush queue and perform TLB FLush for the midified mm. Signed-off-by: Xu Lu --- arch/riscv/mm/tlbflush.c | 29 ++++++++++++++++++++++++++++- 1 file changed, 28 insertions(+), 1 deletion(-) diff --git a/arch/riscv/mm/tlbflush.c b/arch/riscv/mm/tlbflush.c index b5a2d9874d62b..0083fac87c2bc 100644 --- a/arch/riscv/mm/tlbflush.c +++ b/arch/riscv/mm/tlbflush.c @@ -365,8 +365,10 @@ void local_load_tlb_mm(struct mm_struct *mm) { struct tlb_info *info =3D this_cpu_ptr(&tlbinfo); struct tlb_context *contexts =3D info->contexts; + struct tlb_flush_queue *queue =3D NULL; struct mm_struct *victim =3D NULL; - unsigned int i, pos =3D 0, min =3D UINT_MAX; + unsigned int i, len, pos =3D 0, min =3D UINT_MAX; + unsigned long asid, start, size, stride; =20 for (i =3D 0; i < MAX_LOADED_MM; i++) { if (contexts[i].mm =3D=3D mm) { @@ -387,11 +389,36 @@ void local_load_tlb_mm(struct mm_struct *mm) mmgrab_lazy_mm(mm); victim =3D contexts[pos].mm; contexts[pos].mm =3D mm; + contexts[pos].need_flush =3D false; + + queue =3D &info->flush_queues[pos]; + atomic_set(&queue->len, 0); + queue->flag =3D 0; } contexts[pos].gen =3D new_tlb_gen(info); =20 write_unlock(&info->rwlock); =20 + if (contexts[pos].need_flush) { + queue =3D &info->flush_queues[pos]; + asid =3D get_mm_asid(mm); + if (queue->flag & FLUSH_TLB_ALL_ASID) { + local_flush_tlb_all_asid(asid); + } else { + len =3D atomic_read(&queue->len); + for (i =3D 0; i < len; i++) { + start =3D queue->tasks[i].start; + size =3D queue->tasks[i].size; + stride =3D queue->tasks[i].stride; + local_flush_tlb_range_asid(start, size, + stride, asid); + } + } + contexts[pos].need_flush =3D false; + atomic_set(&queue->len, 0); + queue->flag =3D 0; + } + if (victim) { cpumask_clear_cpu(raw_smp_processor_id(), mm_cpumask(victim)); local_flush_tlb_all_asid(get_mm_asid(victim)); --=20 2.20.1 From nobody Mon Dec 1 22:36:56 2025 Received: from mail-pl1-f178.google.com (mail-pl1-f178.google.com [209.85.214.178]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3C1FD332ED1 for ; Thu, 27 Nov 2025 14:12:49 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.178 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1764252771; cv=none; b=pT/zIlRIBlcMVyj0BF+/davbv7pOby58wS3w25bPi4csUTJyOChjw62Q2eD1uHCT8bmvxI0TCv3/7ibDAL39Wzhozt980MchrIM69ao3PoqsU22u6bTu/UF2CmkfWGx836y+8zVLllggSLhjlkkJ80Q91lBWYg1KJcuOmjYRxt8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1764252771; c=relaxed/simple; bh=IvngoUZBuv8RePl4TbzYqiqK3KGpttyy7cH8vL2a6io=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=V3VXDz6Cyzi05SQfLUv1ZB7WnR0tz+MZV2HKJxPRwWw3Ar2zFHmT2leTUi92CPbQjXDtG/NtmLAuwP96QELFAQfOnDH5LZUfI4lP1b3T2DyitbQY98BuooqhZ9FAtHYVL5jhj3EP/OvPsskrxoDk3Kj6Gk04Eai1xTzqyKJHUIY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=eVb+9Mpu; arc=none smtp.client-ip=209.85.214.178 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="eVb+9Mpu" Received: by mail-pl1-f178.google.com with SMTP id d9443c01a7336-29845b06dd2so12147075ad.2 for ; Thu, 27 Nov 2025 06:12:49 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1764252768; x=1764857568; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=y6POrY+BMUj8HO1IGB0lZ0hYE1ONoquBp43+FVGHg3w=; b=eVb+9MpuuMVLGx1rajeasGZ+9Yrc8LRtEgMyHaeZPJ6iCPGigQtT5Tx85tBj9ELCcH lv49rwXzizHfaS4SPkO40l9uvIx0f8lxQE1O3qXPaqlHOEFVdNx2CmbvKumSphJj8yd1 X6z+QoFxfTZqwcNOhOsnnV1atE/9giBYyQ25eMXCe/PRhfyFQnOIaP6uLoU9iCNJHQYY loUeHb0bxooXXTygdpw2/+7TRkQJP+hwN1CiS1Lj00ZnrviX+GCkuCTnHYG3ifHcpAxm hUU/e2aBpvTxOlhi9iAtzIsmrVThcdUxCN3YY4/Y5itcDXBXLVTQQ97oWI6BAPGlvoXC iXQQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1764252768; x=1764857568; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=y6POrY+BMUj8HO1IGB0lZ0hYE1ONoquBp43+FVGHg3w=; b=o0chs3L0UH+IzEfP2L5RXky4DHA+iAPCB8lgpRLfZoPYyp7daYQyXSRSuamwleCFfF ZUbhNL1g0hl0Fx2rJ6vtKe1R7icJnutRAxDQC1+nef7LUHavJUIDRuzVAYaGm9MkvzgR rrOHesXpEaDVWqnDD4P4kbhxPa5sfK/4j0aFceHGTBgekmrmaSuAk+sUOOMskNwTb330 C2+2oQCPqS6Gbk/y6D9rPsPfacpWWSB+kqGsbAeAvMlVYy+eZPFgn5RbQgbO9yCMwzES T5YPjX7hR1Jx+xfOOwWQ8H4HSPJFOsw/xWz+1XxiYrFWcVPQRKD4q2NQKlo7f9s4Btml VA6Q== X-Forwarded-Encrypted: i=1; AJvYcCU65i8SYn0+pAnX5qp+LgSrjcb+lKLrT1ZtAox+0G0JqE6h8OOSn+vFavmAUbMGU4VOZfiQptPdSMRbxDI=@vger.kernel.org X-Gm-Message-State: AOJu0YwLdE5ySzZH6Pq3nVuDIX1zzqmWOfUUZhQemPz1KNGYHbzbqGh/ TDdAEFUfz0Qm0d6ZgChLF8MSh0GKdCGw6bONPyFlAOtz0BUIrKl0A78G0HEpyiaz8KQ= X-Gm-Gg: ASbGncsDOELyzRrB8xrK1FVMUbpqchD48da4aEIWVSixyZf5KBjJMLv+vedVtcCveSD 6EQW0Q2aqspadpUWIMisa50D9pUZk4zluGFLse/rp3LJS/GOXUwnGv24EWZsJNPTUIW3Vr2sOwO i13eDc+TZV5FRWoN7xes0nHE5NNiiq45kTl2ncHdH+VEHSCrsuILcy8nkiSRKuO/NOWDab2E3fP cQDZbBaq4FdtHksi0DMihkDvZ8jUsx0ZXg8/SLv6PkJaCNoNZtUZzrw+Vbz4jS3wvnziOsKyCE4 kyOL3bgzUR1JwHTxPD1OJHVu1Xx093Tg/CLtIa3t2GTgQlovKIjD69iAAuuRYwXUqFA8RP3ljh9 FeOuZZk0tFvcIPS1YTpfCX48mNZ/8d+nqLIDjN2aJ/U4m/VmjEQSaW1vvkWHpRWxhmOEXtM674V fnGjWtTCleymI16y656Eo+TMZYlWCNAVt5N2qlesGZcixeiaj3zM6a00CAM/Z6bdE9CbRC9DtEk A== X-Google-Smtp-Source: AGHT+IHvUpXCjjZYwqFKdgiDcxzIag75+OwjgHS8aOmS+l5BTWC/h8DmVKdTYaMfKXu5VxrbnkaKrQ== X-Received: by 2002:a17:903:1af0:b0:299:d5a5:3f7b with SMTP id d9443c01a7336-29b6c6b87bcmr256617355ad.53.1764252768286; Thu, 27 Nov 2025 06:12:48 -0800 (PST) Received: from J9GPGXL7NT.bytedance.net ([61.213.176.58]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-3477b7341d2sm2030249a91.11.2025.11.27.06.12.43 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Thu, 27 Nov 2025 06:12:47 -0800 (PST) From: Xu Lu To: pjw@kernel.org, palmer@dabbelt.com, aou@eecs.berkeley.edu, alex@ghiti.fr, kees@kernel.org, mingo@redhat.com, peterz@infradead.org, juri.lelli@redhat.com, vincent.guittot@linaro.org, akpm@linux-foundation.org, david@redhat.com, apatel@ventanamicro.com, guoren@kernel.org Cc: linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Xu Lu Subject: [RFC PATCH v2 8/9] riscv: mm: Clear mm_cpumask during local_flush_tlb_all_asid() Date: Thu, 27 Nov 2025 22:11:16 +0800 Message-ID: <20251127141117.87420-9-luxu.kernel@bytedance.com> X-Mailer: git-send-email 2.50.1 In-Reply-To: <20251127141117.87420-1-luxu.kernel@bytedance.com> References: <20251127141117.87420-1-luxu.kernel@bytedance.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" When local_flush_tlb_all_asid() is called, the target mm's TLB entries are all flushed out, then we can clear current CPU in its mm_cpumask so that next time the mm's memory mapping is modified, no IPI will be sent to current CPU. Signed-off-by: Xu Lu --- arch/riscv/mm/tlbflush.c | 32 +++++++++++++++++++------------- 1 file changed, 19 insertions(+), 13 deletions(-) diff --git a/arch/riscv/mm/tlbflush.c b/arch/riscv/mm/tlbflush.c index 0083fac87c2bc..88a1e45bcf508 100644 --- a/arch/riscv/mm/tlbflush.c +++ b/arch/riscv/mm/tlbflush.c @@ -35,7 +35,8 @@ static inline void local_sinval_vma(unsigned long vma, un= signed long asid) */ unsigned long tlb_flush_all_threshold __read_mostly =3D 64; =20 -static void local_flush_tlb_range_threshold_asid(unsigned long start, +static void local_flush_tlb_range_threshold_asid(struct mm_struct *mm, + unsigned long start, unsigned long size, unsigned long stride, unsigned long asid) @@ -44,7 +45,7 @@ static void local_flush_tlb_range_threshold_asid(unsigned= long start, int i; =20 if (nr_ptes_in_range > tlb_flush_all_threshold) { - local_flush_tlb_all_asid(asid); + local_flush_tlb_mm(mm); return; } =20 @@ -64,21 +65,26 @@ static void local_flush_tlb_range_threshold_asid(unsign= ed long start, } } =20 -static inline void local_flush_tlb_range_asid(unsigned long start, - unsigned long size, unsigned long stride, unsigned long asid) +static inline void local_flush_tlb_range_asid(struct mm_struct *mm, + unsigned long start, + unsigned long size, + unsigned long stride, + unsigned long asid) { - if (size <=3D stride) + if (size <=3D stride) { local_flush_tlb_page_asid(start, asid); - else if (size =3D=3D FLUSH_TLB_MAX_SIZE) - local_flush_tlb_all_asid(asid); - else - local_flush_tlb_range_threshold_asid(start, size, stride, asid); + } else if (size =3D=3D FLUSH_TLB_MAX_SIZE) { + local_flush_tlb_mm(mm); + } else { + local_flush_tlb_range_threshold_asid(mm, start, size, stride, + asid); + } } =20 /* Flush a range of kernel pages without broadcasting */ void local_flush_tlb_kernel_range(unsigned long start, unsigned long end) { - local_flush_tlb_range_asid(start, end - start, PAGE_SIZE, FLUSH_TLB_NO_AS= ID); + local_flush_tlb_range_asid(NULL, start, end - start, PAGE_SIZE, FLUSH_TLB= _NO_ASID); } =20 static void __ipi_flush_tlb_all(void *info) @@ -204,7 +210,7 @@ static void __ipi_flush_tlb_range_asid(void *info) { struct flush_tlb_range_data *d =3D info; =20 - local_flush_tlb_range_asid(d->start, d->size, d->stride, d->asid); + local_flush_tlb_range_asid(d->mm, d->start, d->size, d->stride, d->asid); } =20 static void __flush_tlb_range(struct mm_struct *mm, @@ -222,7 +228,7 @@ static void __flush_tlb_range(struct mm_struct *mm, =20 /* Check if the TLB flush needs to be sent to other CPUs. */ if (cpumask_any_but(cmask, cpu) >=3D nr_cpu_ids) { - local_flush_tlb_range_asid(start, size, stride, asid); + local_flush_tlb_range_asid(mm, start, size, stride, asid); } else if (riscv_use_sbi_for_rfence()) { sbi_remote_sfence_vma_asid(cmask, start, size, asid); } else { @@ -410,7 +416,7 @@ void local_load_tlb_mm(struct mm_struct *mm) start =3D queue->tasks[i].start; size =3D queue->tasks[i].size; stride =3D queue->tasks[i].stride; - local_flush_tlb_range_asid(start, size, + local_flush_tlb_range_asid(mm, start, size, stride, asid); } } --=20 2.20.1 From nobody Mon Dec 1 22:36:56 2025 Received: from mail-pl1-f173.google.com (mail-pl1-f173.google.com [209.85.214.173]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B8BB633345D for ; Thu, 27 Nov 2025 14:12:54 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.173 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1764252776; cv=none; b=n2WT6IyWJSpZexruybcJK46Xmz9WsYbUn0YUaLpoBEzO4yW5roFVBGqlmGSodeYPwz8jzXNTgs0+GiOBQ5m1o+pi+WoZpN8/3mx9hZDQtMYM0n2yJjLZ7nvtZSiQ+GfoFuhRPTAh3PSbFIT2CSXvH7PYJ/UVKpPTZs9hZy+qbpE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1764252776; c=relaxed/simple; bh=ZMUCIsJLVPQ8ZdCtIFgapy3wsFNkdaZJpiYshywHMWk=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=Nv9E7Q+AKluvMJNlYZyNwJDyjIXHrUh+mlGBjngIA57ZvFZhE6ipC0Jgmd3o14iFOu4dGk4KYL+S3rwmqJiX533sdsdWVqBc+YfgwiEDSNmuVxdohrd1Jmi6DT8YhcaljKqhdX5mn2MN/tkzD3QPMbs9MwARr3SjV4RTzlHp/Ck= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=J8qaq2kW; arc=none smtp.client-ip=209.85.214.173 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="J8qaq2kW" Received: by mail-pl1-f173.google.com with SMTP id d9443c01a7336-297f35be2ffso14192945ad.2 for ; Thu, 27 Nov 2025 06:12:54 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1764252774; x=1764857574; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=hTb9aibo/K/jflPv0dS9O1eEwweSl7PPeR+/A9D7hw4=; b=J8qaq2kWYphVIfBULyFcRrlGXtUrM64nBT88nPQTq+TrAtLueBw8Gy6J4sL2EUXIKz dVgz8xw4/dlXmYfi9caSyL/ARi+a2vIsL5bRcrqKOiYjZSBu09pfXNmmacsBoF2Uy4kU e113rL7yxZaKESS65gh/l+IKMB8E/NOpvtQ2UToNzdrCUUCQRKtZT125yOpcwGKGN9Lv 2VSMD5lE7KoyTP9jbn2R3rdUt7GuZWXlLXxvzmAnZpmXtsJfk3gPajHEilgJH3+5zIZT fQHCk2Y3BsuYL1sIVMN7/pVF30HNJtPB7dVuyxTTMhNeZJRpxSQy833L7Ot/s+Joy+FG knTA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1764252774; x=1764857574; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=hTb9aibo/K/jflPv0dS9O1eEwweSl7PPeR+/A9D7hw4=; b=cDTaeMXmfIf83MlDQppZ7Vmj0WbFZqxipj5LH4UfBpMxByk1sJmwgC/3qOc3cI1cZk j4S6hiPo7dhAkr55wxPbs0uI8gdPt6UKb/itl7Ejw7IPZigCwQ9q0Umxfhdmweed9kSG NW2WpaAYjHAkC4mfoTTy4CGJHA5Qrw3HLItBxRnaaqze+rdqK+pOdwNbeAAOeas5sc5N lfoazvdCa3uHSUe4ipGP32JGjozEhtuwsPPfpmwUuep/hPJxEofotb4VQMMkOt9X7V+G 9AWGPLOCHjCADEqwjYvSEluBoztiPOXbAzqgRqI4EKOFvNGVSh64qK6c9ff1lBVGAYlK C+4g== X-Forwarded-Encrypted: i=1; AJvYcCX4JRwJINQLsJ7B1QezJ2BiqcBA53dv+8Us+VuXt6MDL7DraDUx+YcLCGU9eejIVXNllBs8Y28FgIaMAeY=@vger.kernel.org X-Gm-Message-State: AOJu0YxprnuDtntSFJ5guGGRcah/vk82myTjZ6pzkxrbKGnowb5pIItD ziMJ6zoVqqnAegbxBznrMNEblRXh5jXr2CzgDevKyhb8IH6o/T/h5Y4UE/BtejjHn6U= X-Gm-Gg: ASbGncvtQWXVaJFBIrZ/Jb98yISMURRQhwjgsriBvHQR3pq3v00eJjZl+rqnZrQ4hqz GkftGgadu0cFAgg5Iaey72Z+RLuDcMxrDjm8KBM1MI+t12cPOqrYFj6Dn5fvPFEq1/GeYaK+qgz 5Wfl0ZUJ5aotLOspl0PCuC2nV7yWmdY5vtVYoYMSKxNutixk/C344JLXavj9mEGMQcydO7vZwRY CQIPVXErCJhGhnihk8CRGcmq1dAuI1Vt5V+1QsednJTDH0kuCEn2STI+LeXBpywaQ5IjumL2cxq FGL22G60ybZFO1gT5Oiz0p8LWOwhZw5LIHRRSZPzrbTpZ6tivZxKN2Tx2iFwnzj5akRvssdvv6R fF0qWKt0g5oyATgafPhchny5XfghlY1yudvHNAIaAp757mK961Tbtv6QN8UNmOumyCjow2w3WNZ mjXiyKskWTUwhM4QveoWijgYPnlsXqaL3YLUnXmE+Fxk6Pe4QW97m0uinmAHOSTXq00kcB/Xjxu +KjdYUX5pfr X-Google-Smtp-Source: AGHT+IGPqPAMlGbhX5jEp4J3UbDXeLl8BWDll8WVzM9luwXo+j3LxoGVrUXUo4r4lcoq2Ou8dpHZEA== X-Received: by 2002:a17:903:2c07:b0:295:f1f:65f with SMTP id d9443c01a7336-29b6c6865a6mr282588785ad.31.1764252773816; Thu, 27 Nov 2025 06:12:53 -0800 (PST) Received: from J9GPGXL7NT.bytedance.net ([61.213.176.58]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-3477b7341d2sm2030249a91.11.2025.11.27.06.12.48 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Thu, 27 Nov 2025 06:12:53 -0800 (PST) From: Xu Lu To: pjw@kernel.org, palmer@dabbelt.com, aou@eecs.berkeley.edu, alex@ghiti.fr, kees@kernel.org, mingo@redhat.com, peterz@infradead.org, juri.lelli@redhat.com, vincent.guittot@linaro.org, akpm@linux-foundation.org, david@redhat.com, apatel@ventanamicro.com, guoren@kernel.org Cc: linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Xu Lu Subject: [RFC PATCH v2 9/9] riscv: mm: Clear mm_cpumask during local_flush_tlb_all() Date: Thu, 27 Nov 2025 22:11:17 +0800 Message-ID: <20251127141117.87420-10-luxu.kernel@bytedance.com> X-Mailer: git-send-email 2.50.1 In-Reply-To: <20251127141117.87420-1-luxu.kernel@bytedance.com> References: <20251127141117.87420-1-luxu.kernel@bytedance.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Now that we maintain an array of active mms on each CPU, when local_flush_tlb_all() is called, we can clear current CPU in the mm_cpumask of all active mms on current CPU. Signed-off-by: Xu Lu --- arch/riscv/include/asm/tlbflush.h | 6 ++++++ arch/riscv/mm/context.c | 2 +- arch/riscv/mm/tlbflush.c | 31 +++++++++++++++++++++++++++++-- 3 files changed, 36 insertions(+), 3 deletions(-) diff --git a/arch/riscv/include/asm/tlbflush.h b/arch/riscv/include/asm/tlb= flush.h index c9630267c58cd..fd62b27172d4a 100644 --- a/arch/riscv/include/asm/tlbflush.h +++ b/arch/riscv/include/asm/tlbflush.h @@ -108,6 +108,7 @@ DECLARE_PER_CPU_SHARED_ALIGNED(struct tlb_info, tlbinfo= ); =20 void local_load_tlb_mm(struct mm_struct *mm); void local_flush_tlb_mm(struct mm_struct *mm); +void local_flush_tlb_all_mm(void); void __init lazy_tlb_flush_init(void); =20 #else /* CONFIG_RISCV_LAZY_TLB_FLUSH */ @@ -119,6 +120,11 @@ static inline void local_flush_tlb_mm(struct mm_struct= *mm) local_flush_tlb_all_asid(get_mm_asid(mm)); } =20 +static inline void local_flush_tlb_all_mm(void) +{ + local_flush_tlb_all(); +} + static inline void lazy_tlb_flush_init(void) {} =20 #endif /* CONFIG_RISCV_LAZY_TLB_FLUSH */ diff --git a/arch/riscv/mm/context.c b/arch/riscv/mm/context.c index c381c4ed46bfb..b6657681948f9 100644 --- a/arch/riscv/mm/context.c +++ b/arch/riscv/mm/context.c @@ -194,7 +194,7 @@ static void set_mm_asid(struct mm_struct *mm, unsigned = int cpu) satp_mode); =20 if (need_flush_tlb) - local_flush_tlb_all(); + local_flush_tlb_all_mm(); } =20 static void set_mm_noasid(struct mm_struct *mm) diff --git a/arch/riscv/mm/tlbflush.c b/arch/riscv/mm/tlbflush.c index 88a1e45bcf508..73c0a7ef61cb1 100644 --- a/arch/riscv/mm/tlbflush.c +++ b/arch/riscv/mm/tlbflush.c @@ -89,13 +89,13 @@ void local_flush_tlb_kernel_range(unsigned long start, = unsigned long end) =20 static void __ipi_flush_tlb_all(void *info) { - local_flush_tlb_all(); + local_flush_tlb_all_mm(); } =20 void flush_tlb_all(void) { if (num_online_cpus() < 2) - local_flush_tlb_all(); + local_flush_tlb_all_mm(); else if (riscv_use_sbi_for_rfence()) sbi_remote_sfence_vma_asid(NULL, 0, FLUSH_TLB_MAX_SIZE, FLUSH_TLB_NO_ASI= D); else @@ -461,6 +461,33 @@ void local_flush_tlb_mm(struct mm_struct *mm) local_flush_tlb_all_asid(asid); } =20 +void local_flush_tlb_all_mm(void) +{ + struct tlb_info *info =3D this_cpu_ptr(&tlbinfo); + struct tlb_context *contexts =3D info->contexts; + struct mm_struct *mms[MAX_LOADED_MM]; + unsigned int cpu =3D raw_smp_processor_id(); + unsigned int i, num =3D 0; + + write_lock(&info->rwlock); + for (i =3D 0; i < MAX_LOADED_MM; i++) { + if (!contexts[i].mm || contexts[i].mm =3D=3D info->active_mm) + continue; + + mms[num++] =3D contexts[i].mm; + contexts[i].mm =3D NULL; + contexts[i].gen =3D 0; + } + write_unlock(&info->rwlock); + + for (i =3D 0; i < num; i++) { + cpumask_clear_cpu(cpu, mm_cpumask(mms[i])); + mmdrop_lazy_mm(mms[i]); + } + + local_flush_tlb_all(); +} + void __init lazy_tlb_flush_init(void) { struct tlb_flush_queue *queue; --=20 2.20.1