From nobody Fri May 17 07:47:03 2024 Received: from invmail4.hynix.com (exvmail4.hynix.com [166.125.252.92]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 367E4184D for ; Wed, 17 Apr 2024 07:34:20 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=166.125.252.92 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1713339262; cv=none; b=KTZ4QRFqi+9xUdFdVk/MHe1hUYQOsz8hA92YVT1+UF1+yVFtUt7LG0SezXB/uBqiY435KRXUi8MN9Tl9dX4q/QSkJojCX2E/+8WpG8l6HMlhQG2CFCGxJwLYx0komFABGdQj3yUFjTxbADbrzgVT9t6pY7I3abh7Z5Vu3D01rEQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1713339262; c=relaxed/simple; bh=2MKSxAJ2mBZDz9V39JhFe+05XOOJBWLN/9CInRv7UqI=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References; b=c4HcZMFSjyVOw1asdJ33kf0FSJXf23U5QSe7ycTnj26UNHbPG/Gd8QxQ83MfcvX6IySyjticCk6mMgq9m4g+pX6hVL2Ic5B67KbRcAOlWKb5JANiziu2XTqtWMpjxhgiecovvE+NdUUMBFyocgvH/Bgq0SDgPGdKoBgJoFDZ5SQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=sk.com; spf=pass smtp.mailfrom=sk.com; arc=none smtp.client-ip=166.125.252.92 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=sk.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=sk.com X-AuditID: a67dfc5b-d6dff70000001748-92-661f77e2e210 From: Byungchul Park To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: kernel_team@skhynix.com, akpm@linux-foundation.org, ying.huang@intel.com, vernhao@tencent.com, mgorman@techsingularity.net, hughd@google.com, willy@infradead.org, david@redhat.com, peterz@infradead.org, luto@kernel.org, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, rjgolo@gmail.com Subject: [PATCH v9 1/8] x86/tlb: add APIs manipulating tlb batch's arch data Date: Wed, 17 Apr 2024 16:18:40 +0900 Message-Id: <20240417071847.29584-2-byungchul@sk.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20240417071847.29584-1-byungchul@sk.com> References: <20240417071847.29584-1-byungchul@sk.com> X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFnrGLMWRmVeSWpSXmKPExsXC9ZZnke6jcvk0g2eL+S3mrF/DZvF5wz82 ixcb2hktvq7/xWzx9FMfi8XlXXPYLO6t+c9qcX7XWlaLHUv3MVlcOrCAyeJ47wEmi/n3PrNZ bN40ldni+JSpjBa/fwAVn5w1mcVBwON7ax+Lx85Zd9k9Fmwq9di8Qstj8Z6XTB6bVnWyeWz6 NInd4925c+weJ2b8ZvGYdzLQ4/2+q2weW3/ZeTROvcbm8XmTXABfFJdNSmpOZllqkb5dAldG w70XLAVr+Cp6P81naWCcydPFyMkhIWAi8W72FGYY++nLfjCbTUBd4saNn2C2iICZxMHWP+wg NrPAXSaJA/1sILawgI/EqZl9YDaLgKpE7/9zYPW8AqYS8xfeZ4KYKS+xesMBsDgn0JzNzWvA 6oWAar6f+AhkcwHVvGaTWPu7hRGiQVLi4IobLBMYeRcwMqxiFMrMK8tNzMwx0cuozMus0EvO z93ECAz9ZbV/oncwfroQfIhRgINRiYfXIEouTYg1say4MvcQowQHs5IIb4uwbJoQb0piZVVq UX58UWlOavEhRmkOFiVxXqNv5SlCAumJJanZqakFqUUwWSYOTqkGRgu7/neSpzNfWF7d2Lk2 dC/DHIZZX3gaOacKqexSjTioVPPO7oNb4kpmpoPPZPmPhSwKClt2N5NzcYILa1bwnk1JvQnR ovdur2zmZ3YW+8EqyfmLceL7ULPpwtsPb77gFuQe0RvINuHedual/xecen1k2qELr+ISdhfI VCxmWxztHG+ykee+jxJLcUaioRZzUXEiANheHqx5AgAA X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFjrLLMWRmVeSWpSXmKPExsXC5WfdrPuoXD7NYNZCFos569ewWXze8I/N 4sWGdkaLr+t/MVs8/dTHYnF47klWi8u75rBZ3Fvzn9Xi/K61rBY7lu5jsrh0YAGTxfHeA0wW 8+99ZrPYvGkqs8XxKVMZLX7/ACo+OWsyi4Ogx/fWPhaPnbPusnss2FTqsXmFlsfiPS+ZPDat 6mTz2PRpErvHu3Pn2D1OzPjN4jHvZKDH+31X2TwWv/jA5LH1l51H49RrbB6fN8kF8Edx2aSk 5mSWpRbp2yVwZTTce8FSsIavovfTfJYGxpk8XYycHBICJhJPX/Yzg9hsAuoSN278BLNFBMwk Drb+YQexmQXuMkkc6GcDsYUFfCROzewDs1kEVCV6/58Dq+cVMJWYv/A+E8RMeYnVGw6AxTmB 5mxuXgNWLwRU8/3ER7YJjFwLGBlWMYpk5pXlJmbmmOoVZ2dU5mVW6CXn525iBAbysto/E3cw frnsfohRgINRiYfXIEouTYg1say4MvcQowQHs5IIb4uwbJoQb0piZVVqUX58UWlOavEhRmkO FiVxXq/w1AQhgfTEktTs1NSC1CKYLBMHp1QDo8PJ4HcKjJfDDhsqXSyp/FZhuOwrY1C3surV C1NU/jpKbZdVnZJidkB0wrQXtVX+rya5Lb7RO5H/0cov/KffX2D235J4t9zZU7Rh326/o0ur 8n85+M4Rs3q2LVbPe1Po81+dnhMeLlivkiIV/kV4u2/R/oVCWvcvtzp7tGo4nZmkYnDi+/q6 d0osxRmJhlrMRcWJALm+JTdgAgAA X-CFilter-Loop: Reflected Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" This is a preparation for migrc mechanism that needs to recognize read-only tlb entries during migration by separating tlb batch arch data into two, one is for read-only entries and the other is for writable ones, and merging those two when needed. Migrc also needs to optimize tlb shootdown by skipping CPUs that have already performed tlb flush needed for a while. To support it, added APIs manipulating arch data for x86. Signed-off-by: Byungchul Park --- arch/x86/include/asm/tlbflush.h | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) diff --git a/arch/x86/include/asm/tlbflush.h b/arch/x86/include/asm/tlbflus= h.h index 25726893c6f4..a14f77c5cdde 100644 --- a/arch/x86/include/asm/tlbflush.h +++ b/arch/x86/include/asm/tlbflush.h @@ -5,6 +5,7 @@ #include #include #include +#include =20 #include #include @@ -293,6 +294,23 @@ static inline void arch_flush_tlb_batched_pending(stru= ct mm_struct *mm) =20 extern void arch_tlbbatch_flush(struct arch_tlbflush_unmap_batch *batch); =20 +static inline void arch_tlbbatch_clear(struct arch_tlbflush_unmap_batch *b= atch) +{ + cpumask_clear(&batch->cpumask); +} + +static inline void arch_tlbbatch_fold(struct arch_tlbflush_unmap_batch *bd= st, + struct arch_tlbflush_unmap_batch *bsrc) +{ + cpumask_or(&bdst->cpumask, &bdst->cpumask, &bsrc->cpumask); +} + +static inline bool arch_tlbbatch_done(struct arch_tlbflush_unmap_batch *bd= st, + struct arch_tlbflush_unmap_batch *bsrc) +{ + return !cpumask_andnot(&bdst->cpumask, &bdst->cpumask, &bsrc->cpumask); +} + static inline bool pte_flags_need_flush(unsigned long oldflags, unsigned long newflags, bool ignore_access) --=20 2.17.1 From nobody Fri May 17 07:47:03 2024 Received: from invmail4.hynix.com (exvmail4.skhynix.com [166.125.252.92]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 2E371F516 for ; Wed, 17 Apr 2024 07:34:05 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=166.125.252.92 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1713339252; cv=none; b=pNMeIAv683y3n0c9qg0Q2MUG1QbzNrH+pAgXvWCLfVct4MfaYEUTZyKLDW3K94xja3TfGGbds+qlJExbS/0lfj9gFfgnvQ414yhuLmtC70VNDajLGfkXHaKXHFLtQgm86WLBQjEaeQsgggY2aCpEI98FppPXkt417vlYo1hPHUk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1713339252; c=relaxed/simple; bh=DwZ21YpU6DS7qxbUmMpNArY0GTu0rE1pYDwE/6PUoUk=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References; b=aFqZjnNs20soBSYIpEto3FfYwfX4OnNCzcanuaIc0kG3ZrnsbLjI4nE+RQ+TxYltJTpNVU/cPOp/xOrQsApvvGnwtXrlsfbOO3vVCpwid/bUHQ1wsKn5TM2P+ERV0b96Vuunb4ufBXWMWluLRZniGfCKjlVLCdB6Wr3R/qguEbc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=sk.com; spf=pass smtp.mailfrom=sk.com; arc=none smtp.client-ip=166.125.252.92 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=sk.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=sk.com X-AuditID: a67dfc5b-d6dff70000001748-97-661f77e263d8 From: Byungchul Park To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: kernel_team@skhynix.com, akpm@linux-foundation.org, ying.huang@intel.com, vernhao@tencent.com, mgorman@techsingularity.net, hughd@google.com, willy@infradead.org, david@redhat.com, peterz@infradead.org, luto@kernel.org, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, rjgolo@gmail.com Subject: [PATCH v9 2/8] arm64: tlbflush: add APIs manipulating tlb batch's arch data Date: Wed, 17 Apr 2024 16:18:41 +0900 Message-Id: <20240417071847.29584-3-byungchul@sk.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20240417071847.29584-1-byungchul@sk.com> References: <20240417071847.29584-1-byungchul@sk.com> X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFnrGLMWRmVeSWpSXmKPExsXC9ZZnke6jcvk0g10zJS3mrF/DZvF5wz82 ixcb2hktvq7/xWzx9FMfi8XlXXPYLO6t+c9qcX7XWlaLHUv3MVlcOrCAyeJ47wEmi/n3PrNZ bN40ldni+JSpjBa/fwAVn5w1mcVBwON7ax+Lx85Zd9k9Fmwq9di8Qstj8Z6XTB6bVnWyeWz6 NInd4925c+weJ2b8ZvGYdzLQ4/2+q2weW3/ZeTROvcbm8XmTXABfFJdNSmpOZllqkb5dAlfG wo7NrAV7uCr2/zzE2MB4k6OLkZNDQsBEYlvHPEYY+8nlA+wgNpuAusSNGz+ZQWwRATOJg61/ wOLMAneZJA70s4HYwgIhEn93tLGC2CwCqhJz/k4Dq+EVMJXYM/8GE8RMeYnVGw6AzeEEmrO5 eQ1YrxBQzfcTH4FsLqCa92wSr9s6oI6QlDi44gbLBEbeBYwMqxiFMvPKchMzc0z0MirzMiv0 kvNzNzECQ39Z7Z/oHYyfLgQfYhTgYFTi4TWIkksTYk0sK67MPcQowcGsJMLbIiybJsSbklhZ lVqUH19UmpNafIhRmoNFSZzX6Ft5ipBAemJJanZqakFqEUyWiYNTqoExu3TtzsuKX3fHfJfw /8JdUsyZU6U/nZND/NA3fx2/XP+++8kz5hmW8kpv+Myg13H11MHkv/EM33I7HWfdybYSemF9 8GtH+gVb901P/UT5qqU0fhxcu/7Mwxufst4nGHMn7P+z8VnlAiHlOwzz5kamx6YV3OzVbbPj mXbruFJrYJDgUSmNRD0lluKMREMt5qLiRAAlXqDIeQIAAA== X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFjrLLMWRmVeSWpSXmKPExsXC5WfdrPuoXD7NYNJmLos569ewWXze8I/N 4sWGdkaLr+t/MVs8/dTHYnF47klWi8u75rBZ3Fvzn9Xi/K61rBY7lu5jsrh0YAGTxfHeA0wW 8+99ZrPYvGkqs8XxKVMZLX7/ACo+OWsyi4Ogx/fWPhaPnbPusnss2FTqsXmFlsfiPS+ZPDat 6mTz2PRpErvHu3Pn2D1OzPjN4jHvZKDH+31X2TwWv/jA5LH1l51H49RrbB6fN8kF8Edx2aSk 5mSWpRbp2yVwZSzs2MxasIerYv/PQ4wNjDc5uhg5OSQETCSeXD7ADmKzCahL3LjxkxnEFhEw kzjY+gcszixwl0niQD8biC0sECLxd0cbK4jNIqAqMefvNLAaXgFTiT3zbzBBzJSXWL3hANgc TqA5m5vXgPUKAdV8P/GRbQIj1wJGhlWMIpl5ZbmJmTmmesXZGZV5mRV6yfm5mxiBgbys9s/E HYxfLrsfYhTgYFTi4TWIkksTYk0sK67MPcQowcGsJMLbIiybJsSbklhZlVqUH19UmpNafIhR moNFSZzXKzw1QUggPbEkNTs1tSC1CCbLxMEp1cCYvHTzPMOZ3TY+Oydu7l7keF925+t1HTH1 9ol3JFVP3q348H+59j/JtNr5Nnt7Wa8lt/VMKGkKECmok/eeEZa8dfZbEZmNnz7vbqicuOZT CtOEeHWGX63zTs9Vcam/b5zkuUp8G6NIgEH71fV/PruJta/iFnuwre5Wc1hsI3vPw7r1MY22 j88osRRnJBpqMRcVJwIADhOFE2ACAAA= X-CFilter-Loop: Reflected Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" This is a preparation for migrc mechanism that requires to manipulate tlb batch's arch data. Even though arm64 does nothing for it, arch with CONFIG_ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH should provide the APIs. Signed-off-by: Byungchul Park --- arch/arm64/include/asm/tlbflush.h | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) diff --git a/arch/arm64/include/asm/tlbflush.h b/arch/arm64/include/asm/tlb= flush.h index a75de2665d84..b8c7fbc1c68e 100644 --- a/arch/arm64/include/asm/tlbflush.h +++ b/arch/arm64/include/asm/tlbflush.h @@ -347,6 +347,24 @@ static inline void arch_tlbbatch_flush(struct arch_tlb= flush_unmap_batch *batch) dsb(ish); } =20 +static inline void arch_tlbbatch_clear(struct arch_tlbflush_unmap_batch *b= atch) +{ + /* nothing to do */ +} + +static inline void arch_tlbbatch_fold(struct arch_tlbflush_unmap_batch *bd= st, + struct arch_tlbflush_unmap_batch *bsrc) +{ + /* nothing to do */ +} + +static inline bool arch_tlbbatch_done(struct arch_tlbflush_unmap_batch *bd= st, + struct arch_tlbflush_unmap_batch *bsrc) +{ + /* Kernel can consider tlb batch always has been done. */ + return true; +} + /* * This is meant to avoid soft lock-ups on large TLB flushing ranges and n= ot * necessarily a performance improvement. --=20 2.17.1 From nobody Fri May 17 07:47:03 2024 Received: from invmail4.hynix.com (exvmail4.hynix.com [166.125.252.92]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 57B1B6BB56 for ; Wed, 17 Apr 2024 07:34:20 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=166.125.252.92 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1713339262; cv=none; b=al90QGI8ygIuaAXGQzhnsp9VAL3/8c/nQm1eh8oLQvqX1OJDa9/qHQDXw1D1d41hfPtJ3gzeVFkSFInFXnXHojvYcGaStsqEbtHYE/KF95dwp+DeIWiaTjCGB2f5pARnWKnrWZZ+4yt9JVZMEhgh8M9c/dCKsH6Wi+LjHA7HT5A= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1713339262; c=relaxed/simple; bh=02cOaCrEAguODxPW9vC2szkGEgfN5B2WX9+iVHZabDE=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References; b=jBNIR5es0MsfuS5Act3w88kRiqPjrDyBkEoXDJWqm7QH2/qN8JrlcO7Od6ugFo/vbIiZtbfEi8TwOjUpGIpkkU09hWoZT0YsiWBXcI3Q5jAza6QDYc1Xe5ksPBVe7SnOuxYNYqDXyTeRRwkovlmMgmmvkuGxFS4l13ZcQb2TlZA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=sk.com; spf=pass smtp.mailfrom=sk.com; arc=none smtp.client-ip=166.125.252.92 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=sk.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=sk.com X-AuditID: a67dfc5b-d6dff70000001748-9d-661f77e24455 From: Byungchul Park To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: kernel_team@skhynix.com, akpm@linux-foundation.org, ying.huang@intel.com, vernhao@tencent.com, mgorman@techsingularity.net, hughd@google.com, willy@infradead.org, david@redhat.com, peterz@infradead.org, luto@kernel.org, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, rjgolo@gmail.com Subject: [PATCH v9 3/8] mm/rmap: recognize read-only tlb entries during batched tlb flush Date: Wed, 17 Apr 2024 16:18:42 +0900 Message-Id: <20240417071847.29584-4-byungchul@sk.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20240417071847.29584-1-byungchul@sk.com> References: <20240417071847.29584-1-byungchul@sk.com> X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFnrGLMWRmVeSWpSXmKPExsXC9ZZnoe6jcvk0g+8LJC3mrF/DZvF5wz82 ixcb2hktvq7/xWzx9FMfi8XlXXPYLO6t+c9qcX7XWlaLHUv3MVlcOrCAyeJ47wEmi/n3PrNZ bN40ldni+JSpjBa/fwAVn5w1mcVBwON7ax+Lx85Zd9k9Fmwq9di8Qstj8Z6XTB6bVnWyeWz6 NInd4925c+weJ2b8ZvGYdzLQ4/2+q2weW3/ZeTROvcbm8XmTXABfFJdNSmpOZllqkb5dAlfG 3uWrmApmSldc27CHvYFxh1gXIyeHhICJxNaZX9lg7KNrusBsNgF1iRs3fjKD2CICZhIHW/+w g9jMAneZJA70g9UIC0RKPN33FizOIqAqce/RCbB6XgFTifXvOxghZspLrN5wACzOCTRnc/Ma sF4hoJrvJz4C2VxANe/ZJO5enwx1hKTEwRU3WCYw8i5gZFjFKJSZV5abmJljopdRmZdZoZec n7uJERj6y2r/RO9g/HQh+BCjAAejEg+vQZRcmhBrYllxZe4hRgkOZiUR3hZh2TQh3pTEyqrU ovz4otKc1OJDjNIcLErivEbfylOEBNITS1KzU1MLUotgskwcnFINjIkCzxZaf/OrKtnl/qVj kUek2TajHp99civ2phkp19+rXuMedmLBAwn2xYHM185e4Fk8wW7WjA+TukxrNW9PTU/9duDD vr4na1mVmIz2fprTvVVBuz1DqeSvYFutnHpI7OusJCe1pzVJy86xvLnvPPcbg5xakO2zwm2f //ItPRVTXvVQsUqLQYmlOCPRUIu5qDgRAFoa4LF5AgAA X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFjrLLMWRmVeSWpSXmKPExsXC5WfdrPuoXD7NoOMbr8Wc9WvYLD5v+Mdm 8WJDO6PF1/W/mC2efupjsTg89ySrxeVdc9gs7q35z2pxftdaVosdS/cxWVw6sIDJ4njvASaL +fc+s1ls3jSV2eL4lKmMFr9/ABWfnDWZxUHQ43trH4vHzll32T0WbCr12LxCy2PxnpdMHptW dbJ5bPo0id3j3blz7B4nZvxm8Zh3MtDj/b6rbB6LX3xg8tj6y86jceo1No/Pm+QC+KO4bFJS czLLUov07RK4MvYuX8VUMFO64tqGPewNjDvEuhg5OSQETCSOruliA7HZBNQlbtz4yQxiiwiY SRxs/cMOYjML3GWSONAPViMsECnxdN9bsDiLgKrEvUcnwOp5BUwl1r/vYISYKS+xesMBsDgn 0JzNzWvAeoWAar6f+Mg2gZFrASPDKkaRzLyy3MTMHFO94uyMyrzMCr3k/NxNjMBAXlb7Z+IO xi+X3Q8xCnAwKvHwGkTJpQmxJpYVV+YeYpTgYFYS4W0Rlk0T4k1JrKxKLcqPLyrNSS0+xCjN waIkzusVnpogJJCeWJKanZpakFoEk2Xi4JRqYDznvqdLebNf1O7N+48ZhvUndxeJ+V2u08uL 6db9vUEt9In0w5kCliUlYdmOZrcWPj/tbnbMJOd+MPv7wzOeRancuv3oOqOW2MH/IQ8Oa2zT epL/SG3/rvj9P9MdH6y58NE683qAvIuK3/ILC0suPlog3drL8YRf/XCeteTMtcle7lN/BTYc MFZiKc5INNRiLipOBAB1cBaHYAIAAA== X-CFilter-Loop: Reflected Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Functionally, no change. This is a preparation for migrc mechanism that requires to recognize read-only tlb entries and handle them in a different way. The newly introduced API, fold_ubc(), will be used by migrc mechanism. Signed-off-by: Byungchul Park --- include/linux/sched.h | 1 + mm/internal.h | 4 ++++ mm/rmap.c | 31 ++++++++++++++++++++++++++++++- 3 files changed, 35 insertions(+), 1 deletion(-) diff --git a/include/linux/sched.h b/include/linux/sched.h index 3c2abbc587b4..823d83b24364 100644 --- a/include/linux/sched.h +++ b/include/linux/sched.h @@ -1334,6 +1334,7 @@ struct task_struct { #endif =20 struct tlbflush_unmap_batch tlb_ubc; + struct tlbflush_unmap_batch tlb_ubc_ro; =20 /* Cache last used pipe for splice(): */ struct pipe_inode_info *splice_pipe; diff --git a/mm/internal.h b/mm/internal.h index 7e486f2c502c..bbaf1fd97b1c 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -1019,6 +1019,7 @@ extern struct workqueue_struct *mm_percpu_wq; void try_to_unmap_flush(void); void try_to_unmap_flush_dirty(void); void flush_tlb_batched_pending(struct mm_struct *mm); +void fold_ubc(struct tlbflush_unmap_batch *dst, struct tlbflush_unmap_batc= h *src); #else static inline void try_to_unmap_flush(void) { @@ -1029,6 +1030,9 @@ static inline void try_to_unmap_flush_dirty(void) static inline void flush_tlb_batched_pending(struct mm_struct *mm) { } +static inline void fold_ubc(struct tlbflush_unmap_batch *dst, struct tlbfl= ush_unmap_batch *src) +{ +} #endif /* CONFIG_ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH */ =20 extern const struct trace_print_flags pageflag_names[]; diff --git a/mm/rmap.c b/mm/rmap.c index 3746a5531018..2542bfe1a947 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -635,6 +635,28 @@ struct anon_vma *folio_lock_anon_vma_read(struct folio= *folio, } =20 #ifdef CONFIG_ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH + +void fold_ubc(struct tlbflush_unmap_batch *dst, + struct tlbflush_unmap_batch *src) +{ + if (!src->flush_required) + return; + + /* + * Fold src to dst. + */ + arch_tlbbatch_fold(&dst->arch, &src->arch); + dst->writable =3D dst->writable || src->writable; + dst->flush_required =3D true; + + /* + * Reset src. + */ + arch_tlbbatch_clear(&src->arch); + src->flush_required =3D false; + src->writable =3D false; +} + /* * Flush TLB entries for recently unmapped pages from remote CPUs. It is * important if a PTE was dirty when it was unmapped that it's flushed @@ -644,7 +666,9 @@ struct anon_vma *folio_lock_anon_vma_read(struct folio = *folio, void try_to_unmap_flush(void) { struct tlbflush_unmap_batch *tlb_ubc =3D ¤t->tlb_ubc; + struct tlbflush_unmap_batch *tlb_ubc_ro =3D ¤t->tlb_ubc_ro; =20 + fold_ubc(tlb_ubc, tlb_ubc_ro); if (!tlb_ubc->flush_required) return; =20 @@ -675,13 +699,18 @@ void try_to_unmap_flush_dirty(void) static void set_tlb_ubc_flush_pending(struct mm_struct *mm, pte_t pteval, unsigned long uaddr) { - struct tlbflush_unmap_batch *tlb_ubc =3D ¤t->tlb_ubc; + struct tlbflush_unmap_batch *tlb_ubc; int batch; bool writable =3D pte_dirty(pteval); =20 if (!pte_accessible(mm, pteval)) return; =20 + if (pte_write(pteval) || writable) + tlb_ubc =3D ¤t->tlb_ubc; + else + tlb_ubc =3D ¤t->tlb_ubc_ro; + arch_tlbbatch_add_pending(&tlb_ubc->arch, mm, uaddr); tlb_ubc->flush_required =3D true; =20 --=20 2.17.1 From nobody Fri May 17 07:47:03 2024 Received: from invmail4.hynix.com (exvmail4.hynix.com [166.125.252.92]) by smtp.subspace.kernel.org (Postfix) with ESMTP id E14576F060 for ; Wed, 17 Apr 2024 07:34:22 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=166.125.252.92 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1713339264; cv=none; b=PPzfIiNWILhwbOADRDREjXoi0SBzK5Hu1Wt+hGNeMjisNeNkDZiZ7WaEAqgv5HToNaDNKWCIEFes1I/HQf/nPfheK4kxRW0lUEr5Ek0I/fWJizEdYTvbsiuFQlWTfIwgQ2qexJ9StygKksX1WAItDBy1LFhOnr6zoKjhlSGszm4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1713339264; c=relaxed/simple; bh=GKKQ0ELyqnwZB6LDFYITg0PJaCDG9V1PU31qvzMDlro=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References; b=jk2dxBk4+Dimzy6vgPiXdLZQMjN4FbF1rP8gOMcxMHq4qF4MZ1yJNRz1NU/+PjFBpwctWlxfU5Ghv00mah8eOROUAXLVQ/kYokno6vqEcqKz7ZYsKjHyVTy42QT7rTlHXqDgpNGKsw8bTLZqR+c+4c5099ZW/CHiZDF3dAm5NCo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=sk.com; spf=pass smtp.mailfrom=sk.com; arc=none smtp.client-ip=166.125.252.92 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=sk.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=sk.com X-AuditID: a67dfc5b-d6dff70000001748-a2-661f77e25f54 From: Byungchul Park To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: kernel_team@skhynix.com, akpm@linux-foundation.org, ying.huang@intel.com, vernhao@tencent.com, mgorman@techsingularity.net, hughd@google.com, willy@infradead.org, david@redhat.com, peterz@infradead.org, luto@kernel.org, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, rjgolo@gmail.com Subject: [PATCH v9 4/8] x86/tlb, mm/rmap: separate arch_tlbbatch_clear() out of arch_tlbbatch_flush() Date: Wed, 17 Apr 2024 16:18:43 +0900 Message-Id: <20240417071847.29584-5-byungchul@sk.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20240417071847.29584-1-byungchul@sk.com> References: <20240417071847.29584-1-byungchul@sk.com> X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFnrGLMWRmVeSWpSXmKPExsXC9ZZnke6jcvk0g9afshZz1q9hs/i84R+b xYsN7YwWX9f/YrZ4+qmPxeLyrjlsFvfW/Ge1OL9rLavFjqX7mCwuHVjAZHG89wCTxfx7n9ks Nm+aymxxfMpURovfP4CKT86azOIg4PG9tY/FY+esu+weCzaVemxeoeWxeM9LJo9NqzrZPDZ9 msTu8e7cOXaPEzN+s3jMOxno8X7fVTaPrb/sPBqnXmPz+LxJLoAvissmJTUnsyy1SN8ugStj 9+ZVbAXHOSp+HnvM0sDYyt7FyMkhIWAi0f9lBSOM/WbGFBYQm01AXeLGjZ/MILaIgJnEwdY/ YPXMAneZJA70s4HYwgKpEgsO7AWzWQRUJSbf/wA2h1fAVGJz+zFWiJnyEqs3HACbwwk0Z3Pz GrB6IaCa7yc+AtlcQDXv2SROL/7CBNEgKXFwxQ2WCYy8CxgZVjEKZeaV5SZm5pjoZVTmZVbo JefnbmIEhv6y2j/ROxg/XQg+xCjAwajEw2sQJZcmxJpYVlyZe4hRgoNZSYS3RVg2TYg3JbGy KrUoP76oNCe1+BCjNAeLkjiv0bfyFCGB9MSS1OzU1ILUIpgsEwenVAOjlfr/49s3v/36ubL7 39fsPw9u9Cx7LPDjwGqfjy4svKrf7eN+qyS5ND+ck/f05wsvrdm/zu18N2fJsbiMvxOEznQe ebxm0sTvQtlfT6yw2Fojyv5rsqmKi2T1Z82qWWU6NzsjPu991t02P8OoO55VLeilr2/u/Nsz Tr4ym7quJijnmOz7ljez05VYijMSDbWYi4oTAeUAXwt5AgAA X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFjrLLMWRmVeSWpSXmKPExsXC5WfdrPuoXD7N4NAxYYs569ewWXze8I/N 4sWGdkaLr+t/MVs8/dTHYnF47klWi8u75rBZ3Fvzn9Xi/K61rBY7lu5jsrh0YAGTxfHeA0wW 8+99ZrPYvGkqs8XxKVMZLX7/ACo+OWsyi4Ogx/fWPhaPnbPusnss2FTqsXmFlsfiPS+ZPDat 6mTz2PRpErvHu3Pn2D1OzPjN4jHvZKDH+31X2TwWv/jA5LH1l51H49RrbB6fN8kF8Edx2aSk 5mSWpRbp2yVwZezevIqt4DhHxc9jj1kaGFvZuxg5OSQETCTezJjCAmKzCahL3LjxkxnEFhEw kzjY+geshlngLpPEgX42EFtYIFViwYG9YDaLgKrE5PsfGEFsXgFTic3tx1ghZspLrN5wAGwO J9Cczc1rwOqFgGq+n/jINoGRawEjwypGkcy8stzEzBxTveLsjMq8zAq95PzcTYzAQF5W+2fi DsYvl90PMQpwMCrx8BpEyaUJsSaWFVfmHmKU4GBWEuFtEZZNE+JNSaysSi3Kjy8qzUktPsQo zcGiJM7rFZ6aICSQnliSmp2aWpBaBJNl4uCUamCc/Fhtrcm3f1vZVc2/T4xN/ttwco908O3X s/MKlm6oWFa/8kS056Tj8ywMnaMWKXediak8u3exUapJaar24dWf32VsdZbxtOH/OkGhLfGG ycEtXxsCnk15tkLc7Xhm+vdt352+b3iy+ojohJj4n8XZ2TumvXqQdzvrd+GsP5Utlb7SXwME vCtjlViKMxINtZiLihMBeTy0bGACAAA= X-CFilter-Loop: Reflected Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" This is a preparation for migrc mechanism that requires to avoid redundant tlb flushes by manipulating tlb batch's arch data. To achieve that, it's needed to separate the part clearing the tlb batch's arch data out of arch_tlbbatch_flush(). Signed-off-by: Byungchul Park --- arch/x86/mm/tlb.c | 2 -- mm/rmap.c | 1 + 2 files changed, 1 insertion(+), 2 deletions(-) diff --git a/arch/x86/mm/tlb.c b/arch/x86/mm/tlb.c index 44ac64f3a047..24bce69222cd 100644 --- a/arch/x86/mm/tlb.c +++ b/arch/x86/mm/tlb.c @@ -1265,8 +1265,6 @@ void arch_tlbbatch_flush(struct arch_tlbflush_unmap_b= atch *batch) local_irq_enable(); } =20 - cpumask_clear(&batch->cpumask); - put_flush_tlb_info(); put_cpu(); } diff --git a/mm/rmap.c b/mm/rmap.c index 2542bfe1a947..d8671d0dc416 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -673,6 +673,7 @@ void try_to_unmap_flush(void) return; =20 arch_tlbbatch_flush(&tlb_ubc->arch); + arch_tlbbatch_clear(&tlb_ubc->arch); tlb_ubc->flush_required =3D false; tlb_ubc->writable =3D false; } --=20 2.17.1 From nobody Fri May 17 07:47:03 2024 Received: from invmail4.hynix.com (exvmail4.hynix.com [166.125.252.92]) by smtp.subspace.kernel.org (Postfix) with ESMTP id EEA9A6F062 for ; Wed, 17 Apr 2024 07:34:22 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=166.125.252.92 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1713339265; cv=none; b=RS9W4YVo7BVMm6sXXTK2n9DW9/NbrZjL8bPeyzQ9VUDM636xmp2IGRl/I0Ehh/eQFyEM/mQmbo5C4TfJU1YCkwy4xDkaqV+pzzTwsdXqmwzOe0dObnJtgg7K+16Uv1/mTcHYflYzfWgE+GzZv04OZD7W5hIObNDYS2z6Amp8omQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1713339265; c=relaxed/simple; bh=w15d3DQ2DjMA6BbDIKYVP7vubWk99uX3kubuVSEXQFg=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References; b=IhT6jk7kV+Sjt+oXKsFJyYbOL6ZahAYUbSGAph9GyAHxSH6kkLtycbB35uKMKhL5WodskjXlqZBWdXCURvY3dNmlu5E/DngsjC/FXlgHntA/N+uUhjOkEhnrhQcOHypOIv9hK0tmgRbwhwvfbuoxRAt6lI7Pmsh2rn8YXAj+Wus= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=sk.com; spf=pass smtp.mailfrom=sk.com; arc=none smtp.client-ip=166.125.252.92 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=sk.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=sk.com X-AuditID: a67dfc5b-d6dff70000001748-a7-661f77e22826 From: Byungchul Park To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: kernel_team@skhynix.com, akpm@linux-foundation.org, ying.huang@intel.com, vernhao@tencent.com, mgorman@techsingularity.net, hughd@google.com, willy@infradead.org, david@redhat.com, peterz@infradead.org, luto@kernel.org, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, rjgolo@gmail.com Subject: [PATCH v9 5/8] mm: separate move/undo parts from migrate_pages_batch() Date: Wed, 17 Apr 2024 16:18:44 +0900 Message-Id: <20240417071847.29584-6-byungchul@sk.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20240417071847.29584-1-byungchul@sk.com> References: <20240417071847.29584-1-byungchul@sk.com> X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFnrKLMWRmVeSWpSXmKPExsXC9ZZnoe6jcvk0g9fbFSzmrF/DZvF5wz82 ixcb2hktvq7/xWzx9FMfi8XlXXPYLO6t+c9qcX7XWlaLHUv3MVlcOrCAyeJ47wEmi/n3PrNZ bN40ldni+JSpjBa/fwAVn5w1mcVBwON7ax+Lx85Zd9k9Fmwq9di8Qstj8Z6XTB6bVnWyeWz6 NInd4925c+weJ2b8ZvGYdzLQ4/2+q2weW3/ZeTROvcbm8XmTXABfFJdNSmpOZllqkb5dAlfG x1tHmQu+a1c0TjvK2sA4XbmLkZNDQsBE4uDm2Uww9pr9F9lBbDYBdYkbN34yg9giAmYSB1v/ gMWZBe4ySRzoZ+ti5OAQFvCX+P5bByTMIqAq0Tn7NyuIzStgKtE0aRrUSHmJ1RsOgI3hBBqz uXkNG4gtBFTz/cRHIJsLqOY9m0TnmhY2iAZJiYMrbrBMYORdwMiwilEoM68sNzEzx0QvozIv s0IvOT93EyMw8JfV/onewfjpQvAhRgEORiUeXoMouTQh1sSy4srcQ4wSHMxKIrwtwrJpQrwp iZVVqUX58UWlOanFhxilOViUxHmNvpWnCAmkJ5akZqemFqQWwWSZODilGhh9nrSuKjAyXn3u /rKimY53Gt9Hrg1kbck2llLbsCtjetiHncd87hXkWjyOV3r58n6J7Evv2PMGUziPz67onWwv +EDm8IesT7GdAS2Tty39ldYfKC38SpiB4d+HqRpdFe2Xtiimx+fPOLepVHZD472AcrkPB7re 815/tMys4enjz/sL7FYeFtBTYinOSDTUYi4qTgQAo8zHzXgCAAA= X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFjrDLMWRmVeSWpSXmKPExsXC5WfdrPuoXD7NoHmjmMWc9WvYLD5v+Mdm 8WJDO6PF1/W/mC2efupjsTg89ySrxeVdc9gs7q35z2pxftdaVosdS/cxWVw6sIDJ4njvASaL +fc+s1ls3jSV2eL4lKmMFr9/ABWfnDWZxUHQ43trH4vHzll32T0WbCr12LxCy2PxnpdMHptW dbJ5bPo0id3j3blz7B4nZvxm8Zh3MtDj/b6rbB6LX3xg8tj6y86jceo1No/Pm+QC+KO4bFJS czLLUov07RK4Mj7eOspc8F27onHaUdYGxunKXYycHBICJhJr9l9kB7HZBNQlbtz4yQxiiwiY SRxs/QMWZxa4yyRxoJ+ti5GDQ1jAX+L7bx2QMIuAqkTn7N+sIDavgKlE06RpTBAj5SVWbzgA NoYTaMzm5jVsILYQUM33Ex/ZJjByLWBkWMUokplXlpuYmWOqV5ydUZmXWaGXnJ+7iREYxstq /0zcwfjlsvshRgEORiUeXoMouTQh1sSy4srcQ4wSHMxKIrwtwrJpQrwpiZVVqUX58UWlOanF hxilOViUxHm9wlMThATSE0tSs1NTC1KLYLJMHJxSDYwh1+w64rKdty7ffpGv4Vh9dPKcZ9KS xrvjXtn+3Trx895M86ObBRK7tkypvLtDXpXH7x/rRZXtzuWNmcdklduXVp8SCmxy/M4ddm5q 5vFIZ73y8Ff9vguOyAU4TmQ4L5R46dzezlydah0Hzm0SZYwaOiFPr/itb+nbcqyNP616Hvea G9wlIUosxRmJhlrMRcWJAKjdQyJfAgAA X-CFilter-Loop: Reflected Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Functionally, no change. This is a preparation for migrc mechanism that requires to use separated folio lists for its own handling during migration. Refactored migrate_pages_batch() and separated move/undo parts from migrate_pages_batch(). Signed-off-by: Byungchul Park --- mm/migrate.c | 134 +++++++++++++++++++++++++++++++-------------------- 1 file changed, 83 insertions(+), 51 deletions(-) diff --git a/mm/migrate.c b/mm/migrate.c index 73a052a382f1..fed3a65e9bbe 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -1609,6 +1609,81 @@ static int migrate_hugetlbs(struct list_head *from, = new_folio_t get_new_folio, return nr_failed; } =20 +static void migrate_folios_move(struct list_head *src_folios, + struct list_head *dst_folios, + free_folio_t put_new_folio, unsigned long private, + enum migrate_mode mode, int reason, + struct list_head *ret_folios, + struct migrate_pages_stats *stats, + int *retry, int *thp_retry, int *nr_failed, + int *nr_retry_pages) +{ + struct folio *folio, *folio2, *dst, *dst2; + bool is_thp; + int nr_pages; + int rc; + + dst =3D list_first_entry(dst_folios, struct folio, lru); + dst2 =3D list_next_entry(dst, lru); + list_for_each_entry_safe(folio, folio2, src_folios, lru) { + is_thp =3D folio_test_large(folio) && folio_test_pmd_mappable(folio); + nr_pages =3D folio_nr_pages(folio); + + cond_resched(); + + rc =3D migrate_folio_move(put_new_folio, private, + folio, dst, mode, + reason, ret_folios); + /* + * The rules are: + * Success: folio will be freed + * -EAGAIN: stay on the unmap_folios list + * Other errno: put on ret_folios list + */ + switch(rc) { + case -EAGAIN: + *retry +=3D 1; + *thp_retry +=3D is_thp; + *nr_retry_pages +=3D nr_pages; + break; + case MIGRATEPAGE_SUCCESS: + stats->nr_succeeded +=3D nr_pages; + stats->nr_thp_succeeded +=3D is_thp; + break; + default: + *nr_failed +=3D 1; + stats->nr_thp_failed +=3D is_thp; + stats->nr_failed_pages +=3D nr_pages; + break; + } + dst =3D dst2; + dst2 =3D list_next_entry(dst, lru); + } +} + +static void migrate_folios_undo(struct list_head *src_folios, + struct list_head *dst_folios, + free_folio_t put_new_folio, unsigned long private, + struct list_head *ret_folios) +{ + struct folio *folio, *folio2, *dst, *dst2; + + dst =3D list_first_entry(dst_folios, struct folio, lru); + dst2 =3D list_next_entry(dst, lru); + list_for_each_entry_safe(folio, folio2, src_folios, lru) { + int old_page_state =3D 0; + struct anon_vma *anon_vma =3D NULL; + + __migrate_folio_extract(dst, &old_page_state, &anon_vma); + migrate_folio_undo_src(folio, old_page_state & PAGE_WAS_MAPPED, + anon_vma, true, ret_folios); + list_del(&dst->lru); + migrate_folio_undo_dst(dst, true, put_new_folio, private); + dst =3D dst2; + dst2 =3D list_next_entry(dst, lru); + } +} + /* * migrate_pages_batch() first unmaps folios in the from list as many as * possible, then move the unmapped folios. @@ -1631,7 +1706,7 @@ static int migrate_pages_batch(struct list_head *from, int pass =3D 0; bool is_thp =3D false; bool is_large =3D false; - struct folio *folio, *folio2, *dst =3D NULL, *dst2; + struct folio *folio, *folio2, *dst =3D NULL; int rc, rc_saved =3D 0, nr_pages; LIST_HEAD(unmap_folios); LIST_HEAD(dst_folios); @@ -1767,42 +1842,11 @@ static int migrate_pages_batch(struct list_head *fr= om, thp_retry =3D 0; nr_retry_pages =3D 0; =20 - dst =3D list_first_entry(&dst_folios, struct folio, lru); - dst2 =3D list_next_entry(dst, lru); - list_for_each_entry_safe(folio, folio2, &unmap_folios, lru) { - is_thp =3D folio_test_large(folio) && folio_test_pmd_mappable(folio); - nr_pages =3D folio_nr_pages(folio); - - cond_resched(); - - rc =3D migrate_folio_move(put_new_folio, private, - folio, dst, mode, - reason, ret_folios); - /* - * The rules are: - * Success: folio will be freed - * -EAGAIN: stay on the unmap_folios list - * Other errno: put on ret_folios list - */ - switch(rc) { - case -EAGAIN: - retry++; - thp_retry +=3D is_thp; - nr_retry_pages +=3D nr_pages; - break; - case MIGRATEPAGE_SUCCESS: - stats->nr_succeeded +=3D nr_pages; - stats->nr_thp_succeeded +=3D is_thp; - break; - default: - nr_failed++; - stats->nr_thp_failed +=3D is_thp; - stats->nr_failed_pages +=3D nr_pages; - break; - } - dst =3D dst2; - dst2 =3D list_next_entry(dst, lru); - } + /* Move the unmapped folios */ + migrate_folios_move(&unmap_folios, &dst_folios, + put_new_folio, private, mode, reason, + ret_folios, stats, &retry, &thp_retry, + &nr_failed, &nr_retry_pages); } nr_failed +=3D retry; stats->nr_thp_failed +=3D thp_retry; @@ -1811,20 +1855,8 @@ static int migrate_pages_batch(struct list_head *fro= m, rc =3D rc_saved ? : nr_failed; out: /* Cleanup remaining folios */ - dst =3D list_first_entry(&dst_folios, struct folio, lru); - dst2 =3D list_next_entry(dst, lru); - list_for_each_entry_safe(folio, folio2, &unmap_folios, lru) { - int old_page_state =3D 0; - struct anon_vma *anon_vma =3D NULL; - - __migrate_folio_extract(dst, &old_page_state, &anon_vma); - migrate_folio_undo_src(folio, old_page_state & PAGE_WAS_MAPPED, - anon_vma, true, ret_folios); - list_del(&dst->lru); - migrate_folio_undo_dst(dst, true, put_new_folio, private); - dst =3D dst2; - dst2 =3D list_next_entry(dst, lru); - } + migrate_folios_undo(&unmap_folios, &dst_folios, + put_new_folio, private, ret_folios); =20 return rc; } --=20 2.17.1 From nobody Fri May 17 07:47:03 2024 Received: from invmail4.hynix.com (exvmail4.hynix.com [166.125.252.92]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 18A2E7441F for ; Wed, 17 Apr 2024 07:34:23 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=166.125.252.92 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1713339266; cv=none; b=hkRm8cxCL2tHngh7AgmbxhQNOmqtvH+5TA4iXuO460Nnv5aZMVF/CYh3kOf7zN+IjogHk7iuqQ6a36LFYhJuX/f8V/xg8MlqKBsO8Ph1orB5yL9b5dKvoatn4RC45SFVI2G4GEyCld350GqYKKSa9eSKO3J2NKKqAWVvstJOcIc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1713339266; c=relaxed/simple; bh=6UBLhiqsl7goEJIikM/TmWWuVZKMJGHJBKBpatkJWiI=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References; b=S8tGoUAhQ6CpFbZp6EOCE8ZIvSEKhglhT2mfrT+H136v/GryXHj+nNFpzZQkxIXGoQTYhabH+TEdNFa99jcdobPRqMQFb8pD/ZvbcyQADHUWPzm/8Y3uLHbK50ZSKH4a/KMSkhvezO1aKXgGXu1RSsRaQd8j2YzG5RUlg6ygvIM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=sk.com; spf=pass smtp.mailfrom=sk.com; arc=none smtp.client-ip=166.125.252.92 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=sk.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=sk.com X-AuditID: a67dfc5b-d6dff70000001748-ac-661f77e2f414 From: Byungchul Park To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: kernel_team@skhynix.com, akpm@linux-foundation.org, ying.huang@intel.com, vernhao@tencent.com, mgorman@techsingularity.net, hughd@google.com, willy@infradead.org, david@redhat.com, peterz@infradead.org, luto@kernel.org, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, rjgolo@gmail.com Subject: [PATCH v9 6/8] mm: buddy: make room for a new variable, mgen, in struct page Date: Wed, 17 Apr 2024 16:18:45 +0900 Message-Id: <20240417071847.29584-7-byungchul@sk.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20240417071847.29584-1-byungchul@sk.com> References: <20240417071847.29584-1-byungchul@sk.com> X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFnrCLMWRmVeSWpSXmKPExsXC9ZZnke6jcvk0g5dflSzmrF/DZvF5wz82 ixcb2hktvq7/xWzx9FMfi8XlXXPYLO6t+c9qcX7XWlaLHUv3MVlcOrCAyeJ47wEmi/n3PrNZ bN40ldni+JSpjBa/fwAVn5w1mcVBwON7ax+Lx85Zd9k9Fmwq9di8Qstj8Z6XTB6bVnWyeWz6 NInd4925c+weJ2b8ZvGYdzLQ4/2+q2weW3/ZeTROvcbm8XmTXABfFJdNSmpOZllqkb5dAlfG v1kX2AquK1c8fuzZwLhIpouRk0NCwESi+fFKdhj7VuMaRhCbTUBd4saNn8wgtoiAmcTB1j9g NcwCd5kkDvSzgdjCAqESsz7NZwWxWQRUJV49XgtUz8HBK2AqMaOxBmKkvMTqDQfAxnACjdnc vAasVQio5PuJj0A2F1DNZzaJV+fes0I0SEocXHGDZQIj7wJGhlWMQpl5ZbmJmTkmehmVeZkV esn5uZsYgWG/rPZP9A7GTxeCDzEKcDAq8fAaRMmlCbEmlhVX5h5ilOBgVhLhbRGWTRPiTUms rEotyo8vKs1JLT7EKM3BoiTOa/StPEVIID2xJDU7NbUgtQgmy8TBKdXAuHhWdVnnz+yK0n0y p5g0kk3sdm1K3yxft1qYbZf4rdi6Jp49kyeK9a3IlZ5yb21n4ZOGiUcS2qemPnsRsvbMva4r 87fE/LdRZ55Yzcq7z+XiVPU1x5/LRr6+Jm84wW+5gv6ts/v2FLRL/ZgaJMej5fN/6bzd/d9i g2Osv2ar7g8qPeDwx6jeQomlOCPRUIu5qDgRACm0Dwh3AgAA X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFjrNLMWRmVeSWpSXmKPExsXC5WfdrPuoXD7N4NwBSYs569ewWXze8I/N 4sWGdkaLr+t/MVs8/dTHYnF47klWi8u75rBZ3Fvzn9Xi/K61rBY7lu5jsrh0YAGTxfHeA0wW 8+99ZrPYvGkqs8XxKVMZLX7/ACo+OWsyi4Ogx/fWPhaPnbPusnss2FTqsXmFlsfiPS+ZPDat 6mTz2PRpErvHu3Pn2D1OzPjN4jHvZKDH+31X2TwWv/jA5LH1l51H49RrbB6fN8kF8Edx2aSk 5mSWpRbp2yVwZfybdYGt4LpyxePHng2Mi2S6GDk5JARMJG41rmEEsdkE1CVu3PjJDGKLCJhJ HGz9ww5iMwvcZZI40M8GYgsLhErM+jSfFcRmEVCVePV4LVA9BwevgKnEjMYaiJHyEqs3HAAb wwk0ZnPzGrBWIaCS7yc+sk1g5FrAyLCKUSQzryw3MTPHVK84O6MyL7NCLzk/dxMjMIiX1f6Z uIPxy2X3Q4wCHIxKPLwGUXJpQqyJZcWVuYcYJTiYlUR4W4Rl04R4UxIrq1KL8uOLSnNSiw8x SnOwKInzeoWnJggJpCeWpGanphakFsFkmTg4pRoYjeqZG5+ViM9xYxTSX7HtQYF43aNDl72/ 1FZe2v/XbMmD3wwtdve2i8ns/LcwpWFT4PN/4ls81lddvP7UJnBrx98oLZvcSxeEr23b9lBk +4sXMZdSf0lfT9VhZuxb5nFocd+k+yb8ev8z655f3LZ1HZ/fpaZ57zmeT52s/TXV5ardnVXt oct7lyuxFGckGmoxFxUnAgC2pfBtXgIAAA== X-CFilter-Loop: Reflected Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Functionally, no change. This is a preparation for migrc mechanism that tracks need of tlb flush for each page residing in buddy, using a generation number in struct page. Fortunately, since the private field in struct page is used only to store page order in buddy, ranging from 0 to MAX_PAGE_ORDER, that can be covered with unsigned short int. So splitted it into two smaller ones, order and mgen, so that the both can be used in buddy at the same time. Signed-off-by: Byungchul Park --- include/linux/mm_types.h | 39 ++++++++++++++++++++++++++++++++------- mm/internal.h | 4 ++-- mm/page_alloc.c | 13 ++++++++----- 3 files changed, 42 insertions(+), 14 deletions(-) diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 5240bd7bca33..4130d22977d6 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -108,13 +108,24 @@ struct page { pgoff_t index; /* Our offset within mapping. */ unsigned long share; /* share count for fsdax */ }; - /** - * @private: Mapping-private opaque data. - * Usually used for buffer_heads if PagePrivate. - * Used for swp_entry_t if PageSwapCache. - * Indicates order in the buddy system if PageBuddy. - */ - unsigned long private; + union { + /** + * @private: Mapping-private opaque data. + * Usually used for buffer_heads if PagePrivate. + * Used for swp_entry_t if PageSwapCache. + */ + unsigned long private; + struct { + /* + * Indicates order in the buddy system if PageBuddy. + */ + unsigned short int order; + /* + * Tracks need of tlb flush used by migrc + */ + unsigned short int mgen; + }; + }; }; struct { /* page_pool used by netstack */ /** @@ -520,6 +531,20 @@ static inline void set_page_private(struct page *page,= unsigned long private) page->private =3D private; } =20 +#define page_buddy_order(page) ((page)->order) + +static inline void set_page_buddy_order(struct page *page, unsigned int or= der) +{ + page->order =3D (unsigned short int)order; +} + +#define page_buddy_mgen(page) ((page)->mgen) + +static inline void set_page_buddy_mgen(struct page *page, unsigned short i= nt mgen) +{ + page->mgen =3D mgen; +} + static inline void *folio_get_private(struct folio *folio) { return folio->private; diff --git a/mm/internal.h b/mm/internal.h index bbaf1fd97b1c..e3a8b77b58ca 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -364,7 +364,7 @@ struct alloc_context { static inline unsigned int buddy_order(struct page *page) { /* PageBuddy() must be checked by the caller */ - return page_private(page); + return page_buddy_order(page); } =20 /* @@ -378,7 +378,7 @@ static inline unsigned int buddy_order(struct page *pag= e) * times, potentially observing different values in the tests and the actu= al * use of the result. */ -#define buddy_order_unsafe(page) READ_ONCE(page_private(page)) +#define buddy_order_unsafe(page) READ_ONCE(page_buddy_order(page)) =20 /* * This function checks whether a page is free && is the buddy diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 14d39f34d336..555f6d2e33ea 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -604,9 +604,12 @@ void destroy_large_folio(struct folio *folio) free_the_page(&folio->page, folio_order(folio)); } =20 -static inline void set_buddy_order(struct page *page, unsigned int order) +static inline void set_buddy_order_mgen(struct page *page, + unsigned int order, + unsigned short int mgen) { - set_page_private(page, order); + set_page_buddy_order(page, order); + set_page_buddy_mgen(page, order); __SetPageBuddy(page); } =20 @@ -824,7 +827,7 @@ static inline void __free_one_page(struct page *page, } =20 done_merging: - set_buddy_order(page, order); + set_buddy_order_mgen(page, order, 0); =20 if (fpi_flags & FPI_TO_TAIL) to_tail =3D true; @@ -1404,7 +1407,7 @@ static inline void expand(struct zone *zone, struct p= age *page, continue; =20 add_to_free_list(&page[size], zone, high, migratetype); - set_buddy_order(&page[size], high); + set_buddy_order_mgen(&page[size], high, 0); } } =20 @@ -6710,7 +6713,7 @@ static void break_down_buddy_pages(struct zone *zone,= struct page *page, continue; =20 add_to_free_list(current_buddy, zone, high, migratetype); - set_buddy_order(current_buddy, high); + set_buddy_order_mgen(current_buddy, high, 0); } } =20 --=20 2.17.1 From nobody Fri May 17 07:47:03 2024 Received: from invmail4.hynix.com (exvmail4.hynix.com [166.125.252.92]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 2CDAD7B3C1 for ; Wed, 17 Apr 2024 07:34:25 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=166.125.252.92 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1713339267; cv=none; b=q+9OtdFE8wyepEnkunIrmQyveHhO8YruGIQtYhUdq1YszwvanP/M+G8tMLHy8t5ruh1+JHt2EiM5ZIKJouF2EYG/Ao2tmy00UL1KwrdyQ9GLyN4IzRsMm4AuUefXw9VuMiy3TLg8JzP4bnHBYQaI0U1NZ0hBg6uonSYkWrnJ2Ac= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1713339267; c=relaxed/simple; bh=Kbkgd8H70pfYdspSdFN3QXQdKpQDFz84yPpR6OVboYw=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References; b=JygE4hi4HUe1C6HhNXLmoFz1uDxxCRgi2HRuisHlGvMB9OTtglsBv0W6xBoUF/CKTAZ9Ha6O7GNDBT+jbduP3I/kH+jysblQddZjTb7phpXF9WggUJ8l1DJUFwclkR7YWGiRCTDK7RUEJkwDUbpnBKplB5XIhGAagZM6OkZJWKQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=sk.com; spf=pass smtp.mailfrom=sk.com; arc=none smtp.client-ip=166.125.252.92 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=sk.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=sk.com X-AuditID: a67dfc5b-d6dff70000001748-b1-661f77e2c364 From: Byungchul Park To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: kernel_team@skhynix.com, akpm@linux-foundation.org, ying.huang@intel.com, vernhao@tencent.com, mgorman@techsingularity.net, hughd@google.com, willy@infradead.org, david@redhat.com, peterz@infradead.org, luto@kernel.org, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, rjgolo@gmail.com Subject: [PATCH v9 7/8] mm: add folio_put_mgen() to deliver migrc's generation number to pcp or buddy Date: Wed, 17 Apr 2024 16:18:46 +0900 Message-Id: <20240417071847.29584-8-byungchul@sk.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20240417071847.29584-1-byungchul@sk.com> References: <20240417071847.29584-1-byungchul@sk.com> X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFnrBLMWRmVeSWpSXmKPExsXC9ZZnoe6jcvk0gxkf1C3mrF/DZvF5wz82 ixcb2hktvq7/xWzx9FMfi8XlXXPYLO6t+c9qcX7XWlaLHUv3MVlcOrCAyeJ47wEmi/n3PrNZ bN40ldni+JSpjBa/fwAVn5w1mcVBwON7ax+Lx85Zd9k9Fmwq9di8Qstj8Z6XTB6bVnWyeWz6 NInd4925c+weJ2b8ZvGYdzLQ4/2+q2weW3/ZeTROvcbm8XmTXABfFJdNSmpOZllqkb5dAlfG mc3djAUbFjBWXP7/mK2BcUUjYxcjJ4eEgInE5Ac/WGDsu8+PMIHYbALqEjdu/GQGsUUEzCQO tv5hB7GZBe4ySRzoZwOxhQVSJQ5e3gEWZxFQlXg5fStYnFfAVKJr5gkmiJnyEqs3HACbwwk0 Z3PzGrAaIaCa7yc+AtlcQDWf2SRmr5wL1SApcXDFDZYJjLwLGBlWMQpl5pXlJmbmmOhlVOZl Vugl5+duYgSG/7LaP9E7GD9dCD7EKMDBqMTDaxAllybEmlhWXJl7iFGCg1lJhLdFWDZNiDcl sbIqtSg/vqg0J7X4EKM0B4uSOK/Rt/IUIYH0xJLU7NTUgtQimCwTB6dUA2PMyvntHbr1N2Ub UlYckI8skrr9pWdeJL/QjBnsz6Yq2N+3Fhf8raihlXjvz+7l0tdLLhcvPd6e0ntqieEUt/Cw qMIOneYn97VlvA87fp/8v8zzjyfjhPM753ayyByKPqPY0NnqfZLLQpGnSafgxiSTH1FOYh/T Q2KmTf3rtl0pZMnldZsjviixFGckGmoxFxUnAgAy+kDEewIAAA== X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFjrPLMWRmVeSWpSXmKPExsXC5WfdrPuoXD7NYN1DGYs569ewWXze8I/N 4sWGdkaLr+t/MVs8/dTHYnF47klWi8u75rBZ3Fvzn9Xi/K61rBY7lu5jsrh0YAGTxfHeA0wW 8+99ZrPYvGkqs8XxKVMZLX7/ACo+OWsyi4Ogx/fWPhaPnbPusnss2FTqsXmFlsfiPS+ZPDat 6mTz2PRpErvHu3Pn2D1OzPjN4jHvZKDH+31X2TwWv/jA5LH1l51H49RrbB6fN8kF8Edx2aSk 5mSWpRbp2yVwZZzZ3M1YsGEBY8Xl/4/ZGhhXNDJ2MXJySAiYSNx9foQJxGYTUJe4ceMnM4gt ImAmcbD1DzuIzSxwl0niQD8biC0skCpx8PIOsDiLgKrEy+lbweK8AqYSXTNPMEHMlJdYveEA 2BxOoDmbm9eA1QgB1Xw/8ZFtAiPXAkaGVYwimXlluYmZOaZ6xdkZlXmZFXrJ+bmbGIHBvKz2 z8QdjF8uux9iFOBgVOLhNYiSSxNiTSwrrsw9xCjBwawkwtsiLJsmxJuSWFmVWpQfX1Sak1p8 iFGag0VJnNcrPDVBSCA9sSQ1OzW1ILUIJsvEwSnVwNg3Q8WyRv3wgStWqjsPVP5dzmu9cm/W eWHJ79ViE9tN/mS1FhoIhWz74mXV/vVkn6Pq94RuRzbfV91hJ051fjzqtyvJIlBmbXBpoGPU Rc77cXn/Z0x9acn1II87genAmu9JlgfWPvjOE/Iu/uE+//2z36t8SGG+aZiVvHXGg9/Hvvz7 WXiwpVqJpTgj0VCLuag4EQCfLBuuYgIAAA== X-CFilter-Loop: Reflected Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Introduced a new API, folio_put_mgen(), to deliver migrc's generation number to pcp or buddy that will be used by migrc mechanism to track need of tlb flush for each page residing in pcp or buddy. migrc makes decision whether tlb flush is needed or not, based on a generation number stored in the interesting page and the global generation number, for that tlb flush required has been completed. For now, the delivery works only for the following call path but not for e.g. free_the_page(), __free_pages_ok() and free_unref_page_list() that are not for releasing source folios during migration: folio_put_mgen() __folio_put_mgen() __folio_put_small() free_unref_page() free_unref_page_commit() free_one_page() __free_one_page() The generation number should be handed over properly when pages travel between pcp and buddy, and must do necessary things on exit from pcp or buddy. It's worth noting that this patch doesn't include actual body for tlb flush on the exit, which will be filled by the main patch of migrc mechanism. Signed-off-by: Byungchul Park --- include/linux/mm.h | 22 ++++++++ include/linux/sched.h | 1 + mm/compaction.c | 10 ++++ mm/internal.h | 41 ++++++++++++++- mm/page_alloc.c | 114 +++++++++++++++++++++++++++++++++++------- mm/page_reporting.c | 10 ++++ mm/swap.c | 19 +++++-- 7 files changed, 195 insertions(+), 22 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 7b0ee64225de..f99072ca5bbc 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -1307,6 +1307,7 @@ static inline struct folio *virt_to_folio(const void = *x) } =20 void __folio_put(struct folio *folio); +void __folio_put_mgen(struct folio *folio, unsigned short int mgen); =20 void put_pages_list(struct list_head *pages); =20 @@ -1506,6 +1507,27 @@ static inline void folio_put(struct folio *folio) __folio_put(folio); } =20 +/** + * folio_put_mgen - Decrement the last reference count on a folio. + * @folio: The folio. + * @mgen: The migrc generation # of TLB flush that the folio requires. + * + * The folio's reference count should be one since the only user, folio + * migration code, calls folio_put_mgen() only when the folio has no + * reference else. The memory will be released back to the page + * allocator and may be used by another allocation immediately. Do not + * access the memory or the struct folio after calling folio_put_mgen(). + * + * Context: May be called in process or interrupt context, but not in NMI + * context. May be called while holding a spinlock. + */ +static inline void folio_put_mgen(struct folio *folio, unsigned short int = mgen) +{ + if (WARN_ON(!folio_put_testzero(folio))) + return; + __folio_put_mgen(folio, mgen); +} + /** * folio_put_refs - Reduce the reference count on a folio. * @folio: The folio. diff --git a/include/linux/sched.h b/include/linux/sched.h index 823d83b24364..74f8d106be79 100644 --- a/include/linux/sched.h +++ b/include/linux/sched.h @@ -1335,6 +1335,7 @@ struct task_struct { =20 struct tlbflush_unmap_batch tlb_ubc; struct tlbflush_unmap_batch tlb_ubc_ro; + unsigned short int mgen; =20 /* Cache last used pipe for splice(): */ struct pipe_inode_info *splice_pipe; diff --git a/mm/compaction.c b/mm/compaction.c index 807b58e6eb68..b095b159bc6b 100644 --- a/mm/compaction.c +++ b/mm/compaction.c @@ -701,6 +701,11 @@ static unsigned long isolate_freepages_block(struct co= mpact_control *cc, if (locked) spin_unlock_irqrestore(&cc->zone->lock, flags); =20 + /* + * Check and flush before using the isolated pages. + */ + check_flush_task_mgen(); + /* * Be careful to not go outside of the pageblock. */ @@ -1673,6 +1678,11 @@ static void fast_isolate_freepages(struct compact_co= ntrol *cc) =20 spin_unlock_irqrestore(&cc->zone->lock, flags); =20 + /* + * Check and flush before using the isolated pages. + */ + check_flush_task_mgen(); + /* Skip fast search if enough freepages isolated */ if (cc->nr_freepages >=3D cc->nr_migratepages) break; diff --git a/mm/internal.h b/mm/internal.h index e3a8b77b58ca..f381af27e6d1 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -544,7 +544,7 @@ extern bool free_pages_prepare(struct page *page, unsig= ned int order); =20 extern int user_min_free_kbytes; =20 -void free_unref_page(struct page *page, unsigned int order); +void free_unref_page(struct page *page, unsigned int order, unsigned short= int mgen); void free_unref_folios(struct folio_batch *fbatch); =20 extern void zone_pcp_reset(struct zone *zone); @@ -1379,4 +1379,43 @@ static inline void shrinker_debugfs_remove(struct de= ntry *debugfs_entry, void workingset_update_node(struct xa_node *node); extern struct list_lru shadow_nodes; =20 +#if defined(CONFIG_MIGRATION) && defined(CONFIG_ARCH_WANT_BATCHED_UNMAP_TL= B_FLUSH) +static inline unsigned short int mgen_latest(unsigned short int a, unsigne= d short int b) +{ + if (!a || !b) + return a + b; + + /* + * The mgen is wrapped around so let's use this trick. + */ + if ((short int)(a - b) < 0) + return b; + else + return a; +} + +static inline void update_task_mgen(unsigned short int mgen) +{ + current->mgen =3D mgen_latest(current->mgen, mgen); +} + +static inline unsigned int hand_over_task_mgen(void) +{ + return xchg(¤t->mgen, 0); +} + +static inline void check_flush_task_mgen(void) +{ + /* + * XXX: migrc mechanism will handle this. For now, do nothing + * but reset current's mgen to finalize this turn. + */ + current->mgen =3D 0; +} +#else /* CONFIG_MIGRATION && CONFIG_ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH */ +static inline unsigned short int mgen_latest(unsigned short int a, unsigne= d short int b) { return 0; } +static inline void update_task_mgen(unsigned short int mgen) {} +static inline unsigned int hand_over_task_mgen(void) { return 0; } +static inline void check_flush_task_mgen(void) {} +#endif #endif /* __MM_INTERNAL_H */ diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 555f6d2e33ea..aefa3cde62d2 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -561,7 +561,7 @@ static inline bool pcp_allowed_order(unsigned int order) static inline void free_the_page(struct page *page, unsigned int order) { if (pcp_allowed_order(order)) /* Via pcp? */ - free_unref_page(page, order); + free_unref_page(page, order, 0); else __free_pages_ok(page, order, FPI_NONE); } @@ -703,6 +703,7 @@ static inline void del_page_from_free_list(struct page = *page, struct zone *zone, if (page_reported(page)) __ClearPageReported(page); =20 + update_task_mgen(page_buddy_mgen(page)); list_del(&page->buddy_list); __ClearPageBuddy(page); set_page_private(page, 0); @@ -768,7 +769,7 @@ buddy_merge_likely(unsigned long pfn, unsigned long bud= dy_pfn, static inline void __free_one_page(struct page *page, unsigned long pfn, struct zone *zone, unsigned int order, - int migratetype, fpi_t fpi_flags) + int migratetype, fpi_t fpi_flags, unsigned short int mgen) { struct capture_control *capc =3D task_capc(zone); unsigned long buddy_pfn =3D 0; @@ -786,8 +787,17 @@ static inline void __free_one_page(struct page *page, VM_BUG_ON_PAGE(pfn & ((1 << order) - 1), page); VM_BUG_ON_PAGE(bad_range(zone, page), page); =20 + /* + * Ensure private is zero before using it inside buddy. + */ + set_page_private(page, 0); while (order < MAX_PAGE_ORDER) { if (compaction_capture(capc, page, order, migratetype)) { + /* + * Capturer will check_flush_task_mgen() through + * prep_new_page(). + */ + update_task_mgen(mgen); __mod_zone_freepage_state(zone, -(1 << order), migratetype); return; @@ -819,6 +829,11 @@ static inline void __free_one_page(struct page *page, if (page_is_guard(buddy)) clear_page_guard(zone, buddy, order, migratetype); else + /* + * del_page_from_free_list() updates current's + * mgen that pairs with hand_over_task_mgen() below + * in this funtion. + */ del_page_from_free_list(buddy, zone, order); combined_pfn =3D buddy_pfn & pfn; page =3D page + (combined_pfn - pfn); @@ -827,7 +842,8 @@ static inline void __free_one_page(struct page *page, } =20 done_merging: - set_buddy_order_mgen(page, order, 0); + mgen =3D mgen_latest(mgen, hand_over_task_mgen()); + set_buddy_order_mgen(page, order, mgen); =20 if (fpi_flags & FPI_TO_TAIL) to_tail =3D true; @@ -869,6 +885,7 @@ int split_free_page(struct page *free_page, int free_page_order; int mt; int ret =3D 0; + unsigned short int mgen; =20 if (split_pfn_offset =3D=3D 0) return ret; @@ -884,7 +901,13 @@ int split_free_page(struct page *free_page, if (likely(!is_migrate_isolate(mt))) __mod_zone_freepage_state(zone, -(1UL << order), mt); =20 + /* + * del_page_from_free_list() updates current's mgen that pairs + * with the following hand_over_task_mgen(). + */ del_page_from_free_list(free_page, zone, order); + mgen =3D hand_over_task_mgen(); + for (pfn =3D free_page_pfn; pfn < free_page_pfn + (1UL << order);) { int mt =3D get_pfnblock_migratetype(pfn_to_page(pfn), pfn); @@ -893,7 +916,7 @@ int split_free_page(struct page *free_page, pfn ? __ffs(pfn) : order, __fls(split_pfn_offset)); __free_one_page(pfn_to_page(pfn), pfn, zone, free_page_order, - mt, FPI_NONE); + mt, FPI_NONE, mgen); pfn +=3D 1UL << free_page_order; split_pfn_offset -=3D (1UL << free_page_order); /* we have done the first part, now switch to second part */ @@ -1094,6 +1117,11 @@ __always_inline bool free_pages_prepare(struct page = *page, =20 VM_BUG_ON_PAGE(PageTail(page), page); =20 + /* + * Ensure private is zero before using it inside pcp. + */ + set_page_private(page, 0); + trace_mm_page_free(page, order); kmsan_free_page(page, order); =20 @@ -1224,10 +1252,16 @@ static void free_pcppages_bulk(struct zone *zone, i= nt count, nr_pages =3D 1 << order; do { int mt; + unsigned short int mgen; =20 page =3D list_last_entry(list, struct page, pcp_list); mt =3D get_pcppage_migratetype(page); =20 + /* + * pcp uses private to store mgen. + */ + mgen =3D page_private(page); + /* must delete to avoid corrupting pcp list */ list_del(&page->pcp_list); count -=3D nr_pages; @@ -1239,7 +1273,7 @@ static void free_pcppages_bulk(struct zone *zone, int= count, if (unlikely(isolated_pageblocks)) mt =3D get_pageblock_migratetype(page); =20 - __free_one_page(page, page_to_pfn(page), zone, order, mt, FPI_NONE); + __free_one_page(page, page_to_pfn(page), zone, order, mt, FPI_NONE, mge= n); trace_mm_page_pcpu_drain(page, order, mt); } while (count > 0 && !list_empty(list)); } @@ -1250,7 +1284,8 @@ static void free_pcppages_bulk(struct zone *zone, int= count, static void free_one_page(struct zone *zone, struct page *page, unsigned long pfn, unsigned int order, - int migratetype, fpi_t fpi_flags) + int migratetype, fpi_t fpi_flags, + unsigned short int mgen) { unsigned long flags; =20 @@ -1259,7 +1294,7 @@ static void free_one_page(struct zone *zone, is_migrate_isolate(migratetype))) { migratetype =3D get_pfnblock_migratetype(page, pfn); } - __free_one_page(page, pfn, zone, order, migratetype, fpi_flags); + __free_one_page(page, pfn, zone, order, migratetype, fpi_flags, mgen); spin_unlock_irqrestore(&zone->lock, flags); } =20 @@ -1280,7 +1315,7 @@ static void __free_pages_ok(struct page *page, unsign= ed int order, */ migratetype =3D get_pfnblock_migratetype(page, pfn); =20 - free_one_page(zone, page, pfn, order, migratetype, fpi_flags); + free_one_page(zone, page, pfn, order, migratetype, fpi_flags, 0); =20 __count_vm_events(PGFREE, 1 << order); } @@ -1541,6 +1576,10 @@ inline void post_alloc_hook(struct page *page, unsig= ned int order, static void prep_new_page(struct page *page, unsigned int order, gfp_t gfp= _flags, unsigned int alloc_flags) { + /* + * Check and flush before using the pages. + */ + check_flush_task_mgen(); post_alloc_hook(page, order, gfp_flags); =20 if (order && (gfp_flags & __GFP_COMP)) @@ -1576,6 +1615,10 @@ struct page *__rmqueue_smallest(struct zone *zone, u= nsigned int order, page =3D get_page_from_free_area(area, migratetype); if (!page) continue; + /* + * del_page_from_free_list() updates current's mgen that + * pairs with check_flush_task_mgen() in prep_new_page(). + */ del_page_from_free_list(page, zone, current_order); expand(zone, page, order, current_order, migratetype); set_pcppage_migratetype(page, migratetype); @@ -2430,7 +2473,7 @@ static int nr_pcp_high(struct per_cpu_pages *pcp, str= uct zone *zone, =20 static void free_unref_page_commit(struct zone *zone, struct per_cpu_pages= *pcp, struct page *page, int migratetype, - unsigned int order) + unsigned int order, unsigned short int mgen) { int high, batch; int pindex; @@ -2444,6 +2487,11 @@ static void free_unref_page_commit(struct zone *zone= , struct per_cpu_pages *pcp, pcp->alloc_factor >>=3D 1; __count_vm_events(PGFREE, 1 << order); pindex =3D order_to_pindex(migratetype, order); + + /* + * pcp uses private to store mgen. + */ + set_page_private(page, mgen); list_add(&page->pcp_list, &pcp->lists[pindex]); pcp->count +=3D 1 << order; =20 @@ -2479,7 +2527,8 @@ static void free_unref_page_commit(struct zone *zone,= struct per_cpu_pages *pcp, /* * Free a pcp page */ -void free_unref_page(struct page *page, unsigned int order) +void free_unref_page(struct page *page, unsigned int order, + unsigned short int mgen) { unsigned long __maybe_unused UP_flags; struct per_cpu_pages *pcp; @@ -2500,7 +2549,7 @@ void free_unref_page(struct page *page, unsigned int = order) migratetype =3D pcpmigratetype =3D get_pcppage_migratetype(page); if (unlikely(migratetype >=3D MIGRATE_PCPTYPES)) { if (unlikely(is_migrate_isolate(migratetype))) { - free_one_page(page_zone(page), page, pfn, order, migratetype, FPI_NONE); + free_one_page(page_zone(page), page, pfn, order, migratetype, FPI_NONE,= mgen); return; } pcpmigratetype =3D MIGRATE_MOVABLE; @@ -2510,10 +2559,10 @@ void free_unref_page(struct page *page, unsigned in= t order) pcp_trylock_prepare(UP_flags); pcp =3D pcp_spin_trylock(zone->per_cpu_pageset); if (pcp) { - free_unref_page_commit(zone, pcp, page, pcpmigratetype, order); + free_unref_page_commit(zone, pcp, page, pcpmigratetype, order, mgen); pcp_spin_unlock(pcp); } else { - free_one_page(zone, page, pfn, order, migratetype, FPI_NONE); + free_one_page(zone, page, pfn, order, migratetype, FPI_NONE, mgen); } pcp_trylock_finish(UP_flags); } @@ -2547,7 +2596,7 @@ void free_unref_folios(struct folio_batch *folios) if (!pcp_allowed_order(order) || is_migrate_isolate(migratetype)) { free_one_page(folio_zone(folio), &folio->page, pfn, - order, migratetype, FPI_NONE); + order, migratetype, FPI_NONE, 0); continue; } folio->private =3D (void *)(unsigned long)order; @@ -2582,7 +2631,7 @@ void free_unref_folios(struct folio_batch *folios) pcp_trylock_finish(UP_flags); free_one_page(zone, &folio->page, folio_pfn(folio), order, - migratetype, FPI_NONE); + migratetype, FPI_NONE, 0); locked_zone =3D NULL; continue; } @@ -2598,7 +2647,7 @@ void free_unref_folios(struct folio_batch *folios) =20 trace_mm_page_free_batched(&folio->page); free_unref_page_commit(zone, pcp, &folio->page, migratetype, - order); + order, 0); } =20 if (pcp) { @@ -2650,6 +2699,11 @@ int __isolate_free_page(struct page *page, unsigned = int order) __mod_zone_freepage_state(zone, -(1UL << order), mt); } =20 + /* + * del_page_from_free_list() updates current's mgen. The user of + * the isolated page should check_flush_task_mgen() before using + * it. + */ del_page_from_free_list(page, zone, order); =20 /* @@ -2691,7 +2745,7 @@ void __putback_isolated_page(struct page *page, unsig= ned int order, int mt) =20 /* Return isolated page to tail of freelist. */ __free_one_page(page, page_to_pfn(page), zone, order, mt, - FPI_SKIP_REPORT_NOTIFY | FPI_TO_TAIL); + FPI_SKIP_REPORT_NOTIFY | FPI_TO_TAIL, 0); } =20 /* @@ -2836,6 +2890,11 @@ struct page *__rmqueue_pcplist(struct zone *zone, un= signed int order, } =20 page =3D list_first_entry(list, struct page, pcp_list); + + /* + * Pairs with check_flush_task_mgen() in prep_new_page(). + */ + update_task_mgen(page_private(page)); list_del(&page->pcp_list); pcp->count -=3D 1 << order; } while (check_new_pages(page, order)); @@ -6659,10 +6718,19 @@ void __offline_isolated_pages(unsigned long start_p= fn, unsigned long end_pfn) BUG_ON(page_count(page)); BUG_ON(!PageBuddy(page)); order =3D buddy_order(page); + /* + * del_page_from_free_list() updates current's mgen that + * pairs with check_flush_task_mgen() below in this function. + */ del_page_from_free_list(page, zone, order); pfn +=3D (1 << order); } spin_unlock_irqrestore(&zone->lock, flags); + + /* + * Check and flush before using it. + */ + check_flush_task_mgen(); } #endif =20 @@ -6738,6 +6806,11 @@ bool take_page_off_buddy(struct page *page) int migratetype =3D get_pfnblock_migratetype(page_head, pfn_head); =20 + /* + * del_page_from_free_list() updates current's + * mgen that pairs with check_flush_task_mgen() below + * in this function. + */ del_page_from_free_list(page_head, zone, page_order); break_down_buddy_pages(zone, page_head, page, 0, page_order, migratetype); @@ -6751,6 +6824,11 @@ bool take_page_off_buddy(struct page *page) break; } spin_unlock_irqrestore(&zone->lock, flags); + + /* + * Check and flush before using it. + */ + check_flush_task_mgen(); return ret; } =20 @@ -6768,7 +6846,7 @@ bool put_page_back_buddy(struct page *page) spin_lock_irqsave(&zone->lock, flags); if (put_page_testzero(page)) { ClearPageHWPoisonTakenOff(page); - __free_one_page(page, pfn, zone, 0, migratetype, FPI_NONE); + __free_one_page(page, pfn, zone, 0, migratetype, FPI_NONE, 0); if (TestClearPageHWPoison(page)) { ret =3D true; } diff --git a/mm/page_reporting.c b/mm/page_reporting.c index e4c428e61d8c..95b771ae4653 100644 --- a/mm/page_reporting.c +++ b/mm/page_reporting.c @@ -221,6 +221,11 @@ page_reporting_cycle(struct page_reporting_dev_info *p= rdev, struct zone *zone, /* release lock before waiting on report processing */ spin_unlock_irq(&zone->lock); =20 + /* + * Check and flush before using the isolated pages. + */ + check_flush_task_mgen(); + /* begin processing pages in local list */ err =3D prdev->report(prdev, sgl, PAGE_REPORTING_CAPACITY); =20 @@ -253,6 +258,11 @@ page_reporting_cycle(struct page_reporting_dev_info *p= rdev, struct zone *zone, =20 spin_unlock_irq(&zone->lock); =20 + /* + * Check and flush before using the isolated pages. + */ + check_flush_task_mgen(); + return err; } =20 diff --git a/mm/swap.c b/mm/swap.c index 500a09a48dfd..1c4ebe0febc6 100644 --- a/mm/swap.c +++ b/mm/swap.c @@ -112,11 +112,24 @@ static void page_cache_release(struct folio *folio) unlock_page_lruvec_irqrestore(lruvec, flags); } =20 -static void __folio_put_small(struct folio *folio) +static void __folio_put_small(struct folio *folio, unsigned short int mgen) { page_cache_release(folio); mem_cgroup_uncharge(folio); - free_unref_page(&folio->page, 0); + free_unref_page(&folio->page, 0, mgen); +} + +void __folio_put_mgen(struct folio *folio, unsigned short int mgen) +{ + if (unlikely(folio_is_zone_device(folio))) + WARN_ON(1); + else if (unlikely(folio_test_large(folio))) + WARN_ON(1); + /* + * For now, migrc supports this case only. + */ + else + __folio_put_small(folio, mgen); } =20 static void __folio_put_large(struct folio *folio) @@ -139,7 +152,7 @@ void __folio_put(struct folio *folio) else if (unlikely(folio_test_large(folio))) __folio_put_large(folio); else - __folio_put_small(folio); + __folio_put_small(folio, 0); } EXPORT_SYMBOL(__folio_put); =20 --=20 2.17.1 From nobody Fri May 17 07:47:03 2024 Received: from invmail4.hynix.com (exvmail4.hynix.com [166.125.252.92]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 7DCBA7BB14 for ; Wed, 17 Apr 2024 07:34:25 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=166.125.252.92 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1713339268; cv=none; b=Vka0Fbq24t1mVkIKNlA8NAM0onlMMFAuGWO1IJfKX/rvhzF4zynYRBcgkPoJMTYE4wF5s5CQ62ZstoHogY9rebLBehk0z7Fu5lwmHNzCGFiAPpRb8VLxfC860zvrV9/vR0Uoyd0ewxZ9yWOuHNZbBzRqX2jCyh3e4yMrFMmurP8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1713339268; c=relaxed/simple; bh=eFi5Wu0m28oGDzaL0rBLfxlUVSsbx0DZkuIuKdmE0jY=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References; b=rrRyPvynWnQ6lbji0de6bfOAG8flIXNSnrgLZf17Y0/z09ZxhPFmLNZUKtWkblCERTk8e0TplbKgnb8fg55OzLIjdHcy10Us2IDrz7icKSl0kCLEpdtMdcDR+2LGF05q4PRn+b1anxQr1QChuAD1da6SMM3azf6SFNzlWHcAmH0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=sk.com; spf=pass smtp.mailfrom=sk.com; arc=none smtp.client-ip=166.125.252.92 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=sk.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=sk.com X-AuditID: a67dfc5b-d6dff70000001748-b7-661f77e24a4a From: Byungchul Park To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: kernel_team@skhynix.com, akpm@linux-foundation.org, ying.huang@intel.com, vernhao@tencent.com, mgorman@techsingularity.net, hughd@google.com, willy@infradead.org, david@redhat.com, peterz@infradead.org, luto@kernel.org, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, rjgolo@gmail.com Subject: [PATCH v9 8/8] mm: defer tlb flush until the source folios at migration actually get used Date: Wed, 17 Apr 2024 16:18:47 +0900 Message-Id: <20240417071847.29584-9-byungchul@sk.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20240417071847.29584-1-byungchul@sk.com> References: <20240417071847.29584-1-byungchul@sk.com> X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFnrGLMWRmVeSWpSXmKPExsXC9ZZnke6jcvk0gxuNehZz1q9hs/i84R+b xYsN7YwWX9f/YrZ4+qmPxeLyrjlsFvfW/Ge1OL9rLavFjqX7mCwuHVjAZHG89wCTxfx7n9ks Nm+aymxxfMpURovfP4CKT86azOIg4PG9tY/FY+esu+weCzaVemxeoeWxeM9LJo9NqzrZPDZ9 msTu8e7cOXaPEzN+s3jMOxno8X7fVTaPrb/sPBqnXmPz+LxJLoAvissmJTUnsyy1SN8ugStj /baD7AWLtzJWnL5+iqmBsa+XsYuRk0NCwESi8+UqIJsDzL71rRwkzCagLnHjxk9mEFtEwEzi YOsfdhCbWeAuk8SBfjYQW1ggSeLhtAusIDaLgKrEkZttTCA2r4CpxJy+TUwQ4+UlVm84ADaH E2jO5uY1YL1CQDXfT3xkg6j5zibxYroIhC0pcXDFDZYJjLwLGBlWMQpl5pXlJmbmmOhlVOZl Vugl5+duYgSG/rLaP9E7GD9dCD7EKMDBqMTDaxAllybEmlhWXJl7iFGCg1lJhLdFWDZNiDcl sbIqtSg/vqg0J7X4EKM0B4uSOK/Rt/IUIYH0xJLU7NTUgtQimCwTB6dUA6Nn08adCg9FPZWm 5zI78glp13VceXrU0+hjQH50nhLHXaOKfGWePIup9dfmSVtyTknOCFteLv30sRcD9/SMouDs Gfm7rtS8q/j2a5XIu5hXZ2cf8HRt2j1vUf5x81azNsMLJeEmCj/yL+lcEHSodDY7+eTQwV6n f+fOF19OORJ93dlF76+JsRJLcUaioRZzUXEiAMlpBip5AgAA X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFjrHLMWRmVeSWpSXmKPExsXC5WfdrPuoXD7N4MgkBYs569ewWXze8I/N 4sWGdkaLr+t/MVs8/dTHYnF47klWi8u75rBZ3Fvzn9Xi/K61rBY7lu5jsrh0YAGTxfHeA0wW 8+99ZrPYvGkqs8XxKVMZLX7/ACo+OWsyi4Ogx/fWPhaPnbPusnss2FTqsXmFlsfiPS+ZPDat 6mTz2PRpErvHu3Pn2D1OzPjN4jHvZKDH+31X2TwWv/jA5LH1l51H49RrbB6fN8kF8Edx2aSk 5mSWpRbp2yVwZazfdpC9YPFWxorT108xNTD29TJ2MXJwSAiYSNz6Vt7FyMnBJqAucePGT2YQ W0TATOJg6x92EJtZ4C6TxIF+NhBbWCBJ4uG0C6wgNouAqsSRm21MIDavgKnEnL5NYLaEgLzE 6g0HwOZwAs3Z3LwGrFcIqOb7iY9sExi5FjAyrGIUycwry03MzDHVK87OqMzLrNBLzs/dxAgM 5WW1fybuYPxy2f0QowAHoxIPr0GUXJoQa2JZcWXuIUYJDmYlEd4WYdk0Id6UxMqq1KL8+KLS nNTiQ4zSHCxK4rxe4akJQgLpiSWp2ampBalFMFkmDk6pBsbrkwWZNR97HDBLl6stNNpmtlr7 /pqzXU2CN75fUJqzoqzqp3y5lXyepbalVPFP7rowg90+32smWm837LlyLTCw7//t9bPa+2yM nHbO+ff5zaqnEm9VLA50acRMeHWmMjn2ha1YaquS9vq2PWbLv/xzPrViS1Xjzo0/1vGxls0o fpf9/Pr72DwlluKMREMt5qLiRADnf6nVYQIAAA== X-CFilter-Loop: Reflected Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" This is implementation of MIGRC mechanism that stands for 'Migration Read Copy'. We always face the migration overhead at either promotion or demotion, while working with tiered memory e.g. CXL memory and found out tlb shootdown is one that is needed to get rid of if possible. Fortunately, tlb flush can be defered as long as it guarantees to be performed before the source folios at migration actually become used, of course, only if the target PTE entries have read-only permission, precisely, don't have write permission. Otherwise, no doubt the sytem might get messed up. To achieve that: 1. For the folios that map only to non-writable tlb entries, prevent tlb flush during migration but perform it just before the source folios actually become used out of buddy or pcp. 2. When any non-writable tlb entry changes to writable e.g. through fault handler, give up migrc mechanism and perform tlb flush required right away. No matter what type of workload is used for performance evaluation, the result would be positive thanks to the unconditional reduction of tlb flushes, tlb misses and interrupts. For the test, I picked up XSBench that is widely used for performance analysis on high performance computing architectures - https://github.com/ANL-CESAR/XSBench. The result would depend on memory latency and how often reclaim runs, which implies tlb miss overhead and how many times migration happens. The slower the memory is and the more reclaim runs, the better migrc works so as to obtain the better result. In my system, the result shows: 1. itlb flushes are reduced over 90%. 2. itlb misses are reduced over 30%. 3. All the other tlb numbers also get enhanced. 4. tlb shootdown interrupts are reduced over 90%. 5. The test program runtime is reduced over 5%. The test envitonment: Architecture - x86_64 QEMU - kvm enabled, host cpu Numa - 2 nodes (16 CPUs 1GB, no CPUs 99GB) Linux Kernel - v6.9-rc4, numa balancing tiering on, demotion enabled < measurement: raw data - tlb and interrupt numbers > $ perf stat -a \ -e itlb.itlb_flush \ -e tlb_flush.dtlb_thread \ -e tlb_flush.stlb_any \ -e dtlb-load-misses \ -e dtlb-store-misses \ -e itlb-load-misses \ XSBench -t 16 -p 50000000 $ grep "TLB shootdowns" /proc/interrupts BEFORE ------ 40417078 itlb.itlb_flush 234852566 tlb_flush.dtlb_thread 153192357 tlb_flush.stlb_any 119001107892 dTLB-load-misses 307921167 dTLB-store-misses 1355272118 iTLB-load-misses TLB: 1364803 1303670 1333921 1349607 1356934 1354216 1332972 1342842 1350265 1316443 1355928 1360793 1298239 1326358 1343006 1340971 TLB shootdowns AFTER ----- 3316495 itlb.itlb_flush 138912511 tlb_flush.dtlb_thread 115199341 tlb_flush.stlb_any 117610390021 dTLB-load-misses 198042233 dTLB-store-misses 840066984 iTLB-load-misses TLB: 117257 119219 117178 115737 117967 118948 117508 116079 116962 117266 117320 117215 105808 103934 115672 117610 TLB shootdowns < measurement: user experience - runtime > $ time XSBench -t 16 -p 50000000 BEFORE ------ Threads: 16 Runtime: 968.783 seconds Lookups: 1,700,000,000 Lookups/s: 1,754,778 15208.91s user 141.44s system 1564% cpu 16:20.98 total AFTER ----- Threads: 16 Runtime: 913.210 seconds Lookups: 1,700,000,000 Lookups/s: 1,861,565 14351.69s user 138.23s system 1565% cpu 15:25.47 total Signed-off-by: Byungchul Park --- include/linux/sched.h | 8 + mm/internal.h | 46 +++++- mm/memory.c | 8 + mm/migrate.c | 359 ++++++++++++++++++++++++++++++++++++++++-- mm/rmap.c | 12 +- 5 files changed, 414 insertions(+), 19 deletions(-) diff --git a/include/linux/sched.h b/include/linux/sched.h index 74f8d106be79..7fba33c1faec 100644 --- a/include/linux/sched.h +++ b/include/linux/sched.h @@ -1337,6 +1337,14 @@ struct task_struct { struct tlbflush_unmap_batch tlb_ubc_ro; unsigned short int mgen; =20 +#if defined(CONFIG_MIGRATION) && defined(CONFIG_ARCH_WANT_BATCHED_UNMAP_TL= B_FLUSH) + /* + * whether all the mappings of a folio during unmap are read-only + * so that migrc can work on the folio + */ + bool can_migrc; +#endif + /* Cache last used pipe for splice(): */ struct pipe_inode_info *splice_pipe; =20 diff --git a/mm/internal.h b/mm/internal.h index f381af27e6d1..cfdfa5908f72 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -1380,6 +1380,39 @@ void workingset_update_node(struct xa_node *node); extern struct list_lru shadow_nodes; =20 #if defined(CONFIG_MIGRATION) && defined(CONFIG_ARCH_WANT_BATCHED_UNMAP_TL= B_FLUSH) +void check_migrc_flush(unsigned short int mgen); +void migrc_flush(void); +void rmap_flush_start(void); +void rmap_flush_end(struct tlbflush_unmap_batch *batch); + +/* + * Reset the indicator indicating there are no writable mappings at the + * beginning of every rmap traverse for unmap. migrc can work only when + * all the mappings are read-only. + */ +static inline void can_migrc_init(void) +{ + current->can_migrc =3D true; +} + +/* + * Mark the folio is not applicable to migrc once it found a writble or + * dirty pte during rmap traverse for unmap. + */ +static inline void can_migrc_fail(void) +{ + current->can_migrc =3D false; +} + +/* + * Check if all the mappings are read-only and read-only mappings even + * exist. + */ +static inline bool can_migrc_test(void) +{ + return current->can_migrc && current->tlb_ubc_ro.flush_required; +} + static inline unsigned short int mgen_latest(unsigned short int a, unsigne= d short int b) { if (!a || !b) @@ -1406,13 +1439,16 @@ static inline unsigned int hand_over_task_mgen(void) =20 static inline void check_flush_task_mgen(void) { - /* - * XXX: migrc mechanism will handle this. For now, do nothing - * but reset current's mgen to finalize this turn. - */ - current->mgen =3D 0; + check_migrc_flush(xchg(¤t->mgen, 0)); } #else /* CONFIG_MIGRATION && CONFIG_ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH */ +static inline void check_migrc_flush(unsigned short int mgen) {} +static inline void migrc_flush(void) {} +static inline void rmap_flush_start(void) {} +static inline void rmap_flush_end(struct tlbflush_unmap_batch *batch) {} +static inline void can_migrc_init(void) {} +static inline void can_migrc_fail(void) {} +static inline bool can_migrc_test(void) { return false; } static inline unsigned short int mgen_latest(unsigned short int a, unsigne= d short int b) { return 0; } static inline void update_task_mgen(unsigned short int mgen) {} static inline unsigned int hand_over_task_mgen(void) { return 0; } diff --git a/mm/memory.c b/mm/memory.c index d2155ced45f8..fc4d09fdfef8 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -3611,6 +3611,14 @@ static vm_fault_t do_wp_page(struct vm_fault *vmf) if (vmf->page) folio =3D page_folio(vmf->page); =20 + /* + * The folio may or may not be one that is under migrc's control + * and about to change its permission from read-only to writable. + * Conservatively give up deferring tlb flush just in case. + */ + if (folio) + migrc_flush(); + /* * Shared mapping: we are guaranteed to have VM_WRITE and * FAULT_FLAG_WRITE set at this point. diff --git a/mm/migrate.c b/mm/migrate.c index fed3a65e9bbe..7e16cb608ac5 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -57,6 +57,279 @@ =20 #include "internal.h" =20 +#ifdef CONFIG_ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH +static struct tlbflush_unmap_batch migrc_ubc; +static DEFINE_SPINLOCK(migrc_lock); + +/* + * Don't be zero to distinguish from invalid mgen, 0. + */ +static unsigned short int mgen_next(unsigned short int a) +{ + return a + 1 ?: a + 2; +} + +static bool mgen_before(unsigned short int a, unsigned short int b) +{ + return (short int)(a - b) < 0; +} + +static void init_tlb_ubc(struct tlbflush_unmap_batch *ubc) +{ + arch_tlbbatch_clear(&ubc->arch); + ubc->flush_required =3D false; + ubc->writable =3D false; +} + +/* + * Need to synchronize between tlb flush and managing pending CPUs in + * migrc_ubc. Take a look at the following scenario, where CPU0 is in + * try_to_unmap_flush() and CPU1 is in migrate_pages_batch(): + * + * CPU0 CPU1 + * ---- ---- + * tlb flush + * unmap folios (needing tlb flush) + * add pending CPUs to migrc_ubc + * <-- not performed tlb flush needed by + * the unmap above yet but the request + * will be cleared by CPU0 shortly. bug! + * clear the CPUs from migrc_ubc + * + * The pending CPUs added in CPU1 should not be cleared from migrc_ubc + * in CPU0 because the tlb flush for migrc_ubc added in CPU1 has not + * been performed this turn. To avoid this, using 'on_flushing' + * variable, prevent adding pending CPUs to migrc_ubc and give up migrc + * mechanism if someone is in the middle of tlb flush, like: + * + * CPU0 CPU1 + * ---- ---- + * on_flushing++ + * tlb flush + * unmap folios (needing tlb flush) + * if on_flushing =3D=3D 0: + * add pending CPUs to migrc_ubc + * else: <-- hit + * give up migrc mechanism + * clear the CPUs from migrc_ubc + * on_flushing-- + * + * Only the following case would be allowed for migrc mechanism to work: + * + * CPU0 CPU1 + * ---- ---- + * unmap folios (needing tlb flush) + * if on_flushing =3D=3D 0: <-- hit + * add pending CPUs to migrc_ubc + * else: + * give up migrc mechanism + * on_flushing++ + * tlb flush + * clear the CPUs from migrc_ubc + * on_flushing-- + */ +static int on_flushing; + +/* + * When more than one thread enter check_migrc_flush() at the same + * time, each should wait for the request on progress to be done to + * avoid the following scenario, where the both CPUs are in + * check_migrc_flush(): + * + * CPU0 CPU1 + * ---- ---- + * if !migrc_ubc.flush_required: + * return + * migrc_ubc.flush_required =3D false + * if !migrc_ubc.flush_requied: <-- hit + * return <-- not performed tlb flush + * needed yet but return. bug! + * migrc_ubc.flush_required =3D false + * try_to_unmap_flush() + * finalize + * try_to_unmap_flush() <-- performs tlb flush needed + * finalize + * + * So it should be handled: + * + * CPU0 CPU1 + * ---- ---- + * atomically execute { + * if migrc_on_flushing: + * wait for the completion + * return + * if !migrc_ubc.flush_required: + * return + * migrc_ubc.flush_required =3D false + * migrc_on_flushing =3D true + * } + * atomically execute { + * if migrc_on_flushing: <-- hit + * wait for the completion + * return <-- tlb flush needed is done + * if !migrc_ubc.flush_requied: + * return + * migrc_ubc.flush_required =3D false + * migrc_on_flushing =3D true + * } + * + * try_to_unmap_flush() + * migrc_on_flushing =3D false + * finalize + * try_to_unmap_flush() <-- performs tlb flush needed + * migrc_on_flushing =3D false + * finalize + */ +static bool migrc_on_flushing; + +/* + * Generation number for the current request of deferred tlb flush. + */ +static unsigned short int migrc_gen; + +/* + * Generation number for the next request. + */ +static unsigned short int migrc_gen_next =3D 1; + +/* + * Generation number for the latest request handled. + */ +static unsigned short int migrc_gen_done; + +static unsigned short int migrc_add_pending_ubc(struct tlbflush_unmap_batc= h *ubc) +{ + struct tlbflush_unmap_batch *tlb_ubc =3D ¤t->tlb_ubc; + unsigned long flags; + unsigned short int mgen; + + spin_lock_irqsave(&migrc_lock, flags); + if (on_flushing || migrc_on_flushing) { + spin_unlock_irqrestore(&migrc_lock, flags); + + /* + * Give up migrc mechanism. Just let tlb flush needed + * handled by try_to_unmap_flush() at the caller side. + */ + fold_ubc(tlb_ubc, ubc); + return 0; + } + fold_ubc(&migrc_ubc, ubc); + mgen =3D migrc_gen =3D migrc_gen_next; + spin_unlock_irqrestore(&migrc_lock, flags); + + return mgen; +} + +void rmap_flush_start(void) +{ + unsigned long flags; + + spin_lock_irqsave(&migrc_lock, flags); + on_flushing++; + spin_unlock_irqrestore(&migrc_lock, flags); +} + +void rmap_flush_end(struct tlbflush_unmap_batch *batch) +{ + unsigned long flags; + + spin_lock_irqsave(&migrc_lock, flags); + if (arch_tlbbatch_done(&migrc_ubc.arch, &batch->arch)) { + migrc_ubc.flush_required =3D false; + migrc_ubc.writable =3D false; + } + on_flushing--; + spin_unlock_irqrestore(&migrc_lock, flags); +} + +/* + * Even if multiple contexts are requesting tlb flush at the same time, + * it must guarantee to have completed tlb flush requested on return. + */ +void check_migrc_flush(unsigned short int mgen) +{ + struct tlbflush_unmap_batch *tlb_ubc =3D ¤t->tlb_ubc; + unsigned long flags; + + /* + * Nothing has been requested. We are done. + */ + if (!mgen) + return; +retry: + /* + * We can see a larger value than or equal to migrc_gen_done, + * which means the tlb flush we need has been done. + */ + if (!mgen_before(READ_ONCE(migrc_gen_done), mgen)) + return; + + spin_lock_irqsave(&migrc_lock, flags); + + /* + * With migrc_lock held, we might read migrc_gen_done updated. + */ + if (mgen_next(migrc_gen_done) !=3D mgen) { + spin_unlock_irqrestore(&migrc_lock, flags); + return; + } + + /* + * Others are already working for us. + */ + if (migrc_on_flushing) { + spin_unlock_irqrestore(&migrc_lock, flags); + goto retry; + } + + if (!migrc_ubc.flush_required) { + spin_unlock_irqrestore(&migrc_lock, flags); + return; + } + + fold_ubc(tlb_ubc, &migrc_ubc); + migrc_gen_next =3D mgen_next(migrc_gen); + migrc_on_flushing =3D true; + spin_unlock_irqrestore(&migrc_lock, flags); + + try_to_unmap_flush(); + + spin_lock_irqsave(&migrc_lock, flags); + migrc_on_flushing =3D false; + + /* + * migrc_gen_done can be read by another with migrc_lock not + * held so use WRITE_ONCE() to prevent tearing. + */ + WRITE_ONCE(migrc_gen_done, mgen); + spin_unlock_irqrestore(&migrc_lock, flags); +} + +void migrc_flush(void) +{ + unsigned long flags; + unsigned short int mgen; + + /* + * Obtain the latest mgen number. + */ + spin_lock_irqsave(&migrc_lock, flags); + mgen =3D migrc_gen; + spin_unlock_irqrestore(&migrc_lock, flags); + + check_migrc_flush(mgen); +} +#else /* CONFIG_ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH */ +static void init_tlb_ubc(struct tlbflush_unmap_batch *ubc) +{ +} +static unsigned int migrc_add_pending_ubc(struct tlbflush_unmap_batch *ubc) +{ + return 0; +} +#endif + bool isolate_movable_page(struct page *page, isolate_mode_t mode) { struct folio *folio =3D folio_get_nontail_page(page); @@ -1090,7 +1363,8 @@ static void migrate_folio_undo_dst(struct folio *dst,= bool locked, =20 /* Cleanup src folio upon migration success */ static void migrate_folio_done(struct folio *src, - enum migrate_reason reason) + enum migrate_reason reason, + unsigned short int mgen) { /* * Compaction can migrate also non-LRU pages which are @@ -1101,8 +1375,15 @@ static void migrate_folio_done(struct folio *src, mod_node_page_state(folio_pgdat(src), NR_ISOLATED_ANON + folio_is_file_lru(src), -folio_nr_pages(src)); =20 - if (reason !=3D MR_MEMORY_FAILURE) - /* We release the page in page_handle_poison. */ + /* We release the page in page_handle_poison. */ + if (reason =3D=3D MR_MEMORY_FAILURE) { + check_migrc_flush(mgen); + return; + } + + if (mgen) + folio_put_mgen(src, mgen); + else folio_put(src); } =20 @@ -1126,7 +1407,7 @@ static int migrate_folio_unmap(new_folio_t get_new_fo= lio, folio_clear_unevictable(src); /* free_pages_prepare() will clear PG_isolated. */ list_del(&src->lru); - migrate_folio_done(src, reason); + migrate_folio_done(src, reason, 0); return MIGRATEPAGE_SUCCESS; } =20 @@ -1272,7 +1553,7 @@ static int migrate_folio_unmap(new_folio_t get_new_fo= lio, static int migrate_folio_move(free_folio_t put_new_folio, unsigned long pr= ivate, struct folio *src, struct folio *dst, enum migrate_mode mode, enum migrate_reason reason, - struct list_head *ret) + struct list_head *ret, unsigned short int mgen) { int rc; int old_page_state =3D 0; @@ -1322,11 +1603,12 @@ static int migrate_folio_move(free_folio_t put_new_= folio, unsigned long private, * and will be freed. */ list_del(&src->lru); + /* Drop an anon_vma reference if we took one */ if (anon_vma) put_anon_vma(anon_vma); folio_unlock(src); - migrate_folio_done(src, reason); + migrate_folio_done(src, reason, mgen); =20 return rc; out: @@ -1616,7 +1898,7 @@ static void migrate_folios_move(struct list_head *src= _folios, struct list_head *ret_folios, struct migrate_pages_stats *stats, int *retry, int *thp_retry, int *nr_failed, - int *nr_retry_pages) + int *nr_retry_pages, unsigned short int mgen) { struct folio *folio, *folio2, *dst, *dst2; bool is_thp; @@ -1633,7 +1915,7 @@ static void migrate_folios_move(struct list_head *src= _folios, =20 rc =3D migrate_folio_move(put_new_folio, private, folio, dst, mode, - reason, ret_folios); + reason, ret_folios, mgen); /* * The rules are: * Success: folio will be freed @@ -1706,24 +1988,36 @@ static int migrate_pages_batch(struct list_head *fr= om, int pass =3D 0; bool is_thp =3D false; bool is_large =3D false; + bool is_zone_device =3D false; struct folio *folio, *folio2, *dst =3D NULL; int rc, rc_saved =3D 0, nr_pages; LIST_HEAD(unmap_folios); LIST_HEAD(dst_folios); + LIST_HEAD(unmap_folios_migrc); + LIST_HEAD(dst_folios_migrc); bool nosplit =3D (reason =3D=3D MR_NUMA_MISPLACED); + struct tlbflush_unmap_batch pending_ubc; + struct tlbflush_unmap_batch *tlb_ubc =3D ¤t->tlb_ubc; + struct tlbflush_unmap_batch *tlb_ubc_ro =3D ¤t->tlb_ubc_ro; + unsigned short int mgen; =20 VM_WARN_ON_ONCE(mode !=3D MIGRATE_ASYNC && !list_empty(from) && !list_is_singular(from)); =20 + init_tlb_ubc(&pending_ubc); + for (pass =3D 0; pass < nr_pass && retry; pass++) { retry =3D 0; thp_retry =3D 0; nr_retry_pages =3D 0; =20 list_for_each_entry_safe(folio, folio2, from, lru) { + bool can_migrc; + is_large =3D folio_test_large(folio); is_thp =3D is_large && folio_test_pmd_mappable(folio); nr_pages =3D folio_nr_pages(folio); + is_zone_device =3D folio_is_zone_device(folio); =20 cond_resched(); =20 @@ -1750,9 +2044,25 @@ static int migrate_pages_batch(struct list_head *fro= m, continue; } =20 + can_migrc_init(); rc =3D migrate_folio_unmap(get_new_folio, put_new_folio, private, folio, &dst, mode, reason, ret_folios); + can_migrc =3D can_migrc_test(); + + /* + * XXX: No way to handle zone device folio after + * freeing. Remove the following constraint + * once migrc can handle it. + */ + can_migrc =3D can_migrc && likely(!is_zone_device); + + /* + * XXX: Remove the following constraint once + * migrc handles large folio. + */ + can_migrc =3D can_migrc && likely(!is_large); + /* * The rules are: * Success: folio will be freed @@ -1798,7 +2108,8 @@ static int migrate_pages_batch(struct list_head *from, /* nr_failed isn't updated for not used */ stats->nr_thp_failed +=3D thp_retry; rc_saved =3D rc; - if (list_empty(&unmap_folios)) + if (list_empty(&unmap_folios) && + list_empty(&unmap_folios_migrc)) goto out; else goto move; @@ -1812,8 +2123,19 @@ static int migrate_pages_batch(struct list_head *fro= m, stats->nr_thp_succeeded +=3D is_thp; break; case MIGRATEPAGE_UNMAP: - list_move_tail(&folio->lru, &unmap_folios); - list_add_tail(&dst->lru, &dst_folios); + if (can_migrc) { + list_move_tail(&folio->lru, &unmap_folios_migrc); + list_add_tail(&dst->lru, &dst_folios_migrc); + + /* + * Gather ro batch data to add + * to migrc_ubc after unmap. + */ + fold_ubc(&pending_ubc, tlb_ubc_ro); + } else { + list_move_tail(&folio->lru, &unmap_folios); + list_add_tail(&dst->lru, &dst_folios); + } break; default: /* @@ -1827,12 +2149,19 @@ static int migrate_pages_batch(struct list_head *fr= om, stats->nr_failed_pages +=3D nr_pages; break; } + /* + * Done with the current folio. Fold the ro + * batch data gathered to the normal batch. + */ + fold_ubc(tlb_ubc, tlb_ubc_ro); } } nr_failed +=3D retry; stats->nr_thp_failed +=3D thp_retry; stats->nr_failed_pages +=3D nr_retry_pages; move: + /* Should be before try_to_unmap_flush() */ + mgen =3D migrc_add_pending_ubc(&pending_ubc); /* Flush TLBs for all unmapped folios */ try_to_unmap_flush(); =20 @@ -1846,7 +2175,11 @@ static int migrate_pages_batch(struct list_head *fro= m, migrate_folios_move(&unmap_folios, &dst_folios, put_new_folio, private, mode, reason, ret_folios, stats, &retry, &thp_retry, - &nr_failed, &nr_retry_pages); + &nr_failed, &nr_retry_pages, 0); + migrate_folios_move(&unmap_folios_migrc, &dst_folios_migrc, + put_new_folio, private, mode, reason, + ret_folios, stats, &retry, &thp_retry, + &nr_failed, &nr_retry_pages, mgen); } nr_failed +=3D retry; stats->nr_thp_failed +=3D thp_retry; @@ -1857,6 +2190,8 @@ static int migrate_pages_batch(struct list_head *from, /* Cleanup remaining folios */ migrate_folios_undo(&unmap_folios, &dst_folios, put_new_folio, private, ret_folios); + migrate_folios_undo(&unmap_folios_migrc, &dst_folios_migrc, + put_new_folio, private, ret_folios); =20 return rc; } diff --git a/mm/rmap.c b/mm/rmap.c index d8671d0dc416..9c369aefd636 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -672,7 +672,9 @@ void try_to_unmap_flush(void) if (!tlb_ubc->flush_required) return; =20 + rmap_flush_start(); arch_tlbbatch_flush(&tlb_ubc->arch); + rmap_flush_end(tlb_ubc); arch_tlbbatch_clear(&tlb_ubc->arch); tlb_ubc->flush_required =3D false; tlb_ubc->writable =3D false; @@ -707,9 +709,15 @@ static void set_tlb_ubc_flush_pending(struct mm_struct= *mm, pte_t pteval, if (!pte_accessible(mm, pteval)) return; =20 - if (pte_write(pteval) || writable) + if (pte_write(pteval) || writable) { tlb_ubc =3D ¤t->tlb_ubc; - else + + /* + * migrc cannot work with the folio once it found a + * writable or dirty mapping on it. + */ + can_migrc_fail(); + } else tlb_ubc =3D ¤t->tlb_ubc_ro; =20 arch_tlbbatch_add_pending(&tlb_ubc->arch, mm, uaddr); --=20 2.17.1