From nobody Fri Sep 25 12:38:49 2026 Received: from m16.mail.163.com (m16.mail.163.com [117.135.210.4]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 47418364049 for ; Sat, 12 Sep 2026 08:26:21 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=117.135.210.4 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789201585; cv=none; b=BrQqd0QhEEoSQHsC12Mz7QqoKywE3X6f+jnuRKKqWujRmyT42Qf13yAueI9DVHXyTlAZ3mVTRbSmLxEy/0y1FkRIgT2/7W5HxsyCkJMnDCvJ04vfEJIFhPQZvcrG/V+4IOWZt3lUQSkFfA7ZrLhJldp5JUOgK7Mpyj/VhmMM4NA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789201585; c=relaxed/simple; bh=HiCLigrb3JPgZqO0yJoGKQgmzJo9QYRctcjHZfXqCrY=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=O1ez6xKcEgFaACoR6orC10DZFauJB4HXJGnBpVzQscKbeLLr6pDoTNbqGKFZrsC0yD/8ehelXSNfmBaB0PmSnpcbVJob7Jwes6ROTTzdWV/oALB83hBVImY6WnJ8SXa1syibATjYx1kDL1CLMGadVadl8afyo9Z+5M7gy/GbfgY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=163.com; spf=pass smtp.mailfrom=163.com; dkim=pass (1024-bit key) header.d=163.com header.i=@163.com header.b=p+QhlJEF; arc=none smtp.client-ip=117.135.210.4 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=163.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=163.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=163.com header.i=@163.com header.b="p+QhlJEF" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=163.com; s=s110527; h=From:To:Subject:Date:Message-ID:MIME-Version; bh=zB ogXOhOlG1kMkxMqII2ZbwGmlciHENob2EpVE6bVCc=; b=p+QhlJEFq21IlZxo+c NxJyxgDg8fE6y/NEaX/XoL6bTwDF+tDVS++xpBFt/OaZSZY0SSExhsUc+C5k1hHI kNDk49X/U5ZtJRXr/AwtjCu6H1R1oPNh8g201lbpGjrw3gnoG6yLSG0avCaek5yv QKgmBzdwo8jjm+x6m996fAOI4= Received: from VM-0-3-ubuntu.. (unknown []) by gzsmtp3 (Coremail) with SMTP id PigvCgDnr7FODKVqmfMoSQ--.44660S3; Sat, 12 Sep 2026 16:24:48 +0800 (CST) From: Longlong Xia To: xu.xin16@zte.com.cn Cc: akpm@linux-foundation.org, baohua@kernel.org, baolin.wang@linux.alibaba.com, borntraeger@linux.ibm.com, chengming.zhou@linux.dev, david@kernel.org, dev.jain@arm.com, gourry@gourry.net, harry@kernel.org, lance.yang@linux.dev, liam@infradead.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, nico.pache@linux.dev, peterz@infradead.org, rppt@kernel.org, surenb@google.com, usama.arif@linux.dev, vbabka@kernel.org, ziy@nvidia.com, ljs@kernel.org, Longlong Xia Subject: [PATCH RFC 1/3] mm/pagewalk: allow folio_walk_start() under a vma read lock Date: Sat, 12 Sep 2026 16:24:05 +0800 Message-ID: <20260912082407.75875-2-xialonglong2025@163.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20260912082407.75875-1-xialonglong2025@163.com> References: <20260911160421076_KNXun8Mpp9Xj7fxHG0i7@zte.com.cn> <20260912082407.75875-1-xialonglong2025@163.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-CM-TRANSID: PigvCgDnr7FODKVqmfMoSQ--.44660S3 X-Coremail-Antispam: 1Uf129KBjvJXoW7ur15Xr1rZr4xGr18GF1UAwb_yoW8Kw45pF Z2ka4FqFWvkr1I9w18Aa12yr45u34kKFWUC3ZFgry7ZrnIya1fKF1kt3Wjqa45Gr4xAFZ5 XrWDKrWkuF18Jr7anT9S1TB71UUUUU7qnTZGkaVYY2UrUUUUjbIjqfuFe4nvWSU5nxnvy2 9KBjDUYxBIdaVFxhVjvjDU0xZFpf9x07jSv38UUUUU= X-CM-SenderInfo: x0ldz0pqjo00rjsqjki6rwjhhfrp/xtbC+BAbFGqlDFCUsQAA3R Content-Type: text/plain; charset="utf-8" From: Longlong Xia Allow folio_walk_start() to use a vma read lock by replacing mmap_assert_locked() with vma_assert_locked(). This lets KSM look up pages without taking the mmap read lock. The vma lock stabilizes the mapping, while page table locks serialize entry updates. Callers walking another mm under the vma lock must also hold an mm_users reference to prevent exit_mmap() from freeing the page tables. Assisted-by: Zcode:GLM-5.3 Signed-off-by: Longlong Xia --- include/linux/pagewalk.h | 3 +++ mm/pagewalk.c | 11 +++++++++-- 2 files changed, 12 insertions(+), 2 deletions(-) diff --git a/include/linux/pagewalk.h b/include/linux/pagewalk.h index b41d7265c01b..cafd7b15f480 100644 --- a/include/linux/pagewalk.h +++ b/include/linux/pagewalk.h @@ -151,6 +151,9 @@ typedef int __bitwise folio_walk_flags_t; /* Walk shared zeropages (small + huge) as well. */ #define FW_ZEROPAGE ((__force folio_walk_flags_t)BIT(0)) =20 +/* The caller holds the VMA read lock instead of the mmap lock. */ +#define FW_VMA_LOCKED ((__force folio_walk_flags_t)BIT(1)) + enum folio_walk_level { FW_LEVEL_PTE, FW_LEVEL_PMD, diff --git a/mm/pagewalk.c b/mm/pagewalk.c index cc07fcf50e87..57ffaf2a85b8 100644 --- a/mm/pagewalk.c +++ b/mm/pagewalk.c @@ -894,7 +894,11 @@ int walk_page_mapping(struct address_space *mapping, p= goff_t first_index, * huge_ptep_set_*, ...). Note that the page table entry stored in @fw mig= ht * not correspond to the first physical entry of a logical hugetlb entry. * - * The mmap lock must be held in read mode. + * The mmap lock must be held in read mode. Alternatively, with + * CONFIG_PER_VMA_LOCK and @FW_VMA_LOCKED, the vma lock may be held in rea= d mode: the + * page tables of a read-locked vma cannot be torn down while the mm has + * users, so a caller that walks an mm other than its own must also hold a + * mm_users reference for the duration of the walk. * * Return: folio pointer on success, otherwise NULL. */ @@ -912,7 +916,10 @@ struct folio *folio_walk_start(struct folio_walk *fw, pgd_t *pgdp; p4d_t *p4dp; =20 - mmap_assert_locked(vma->vm_mm); + if (flags & FW_VMA_LOCKED) + vma_assert_locked(vma); + else + mmap_assert_locked(vma->vm_mm); vma_pgtable_walk_begin(vma); =20 if (WARN_ON_ONCE(addr < vma->vm_start || addr >=3D vma->vm_end)) --=20 2.43.0 From nobody Fri Sep 25 12:38:49 2026 Received: from m16.mail.163.com (m16.mail.163.com [220.197.31.2]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E57E037B407 for ; Sat, 12 Sep 2026 08:26:24 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=220.197.31.2 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789201587; cv=none; b=D24gKgtlanAgsC6WR4mHgVMdeibEllsgP0GFDqnDmr2UuAsrTKiOuhAmgOy6VJ8bhwOlWgYA88zElMqmJY1pPljWrXnmITGybWltbcCpZok/yGN0s0SR/uR9wwh++qhKg23xb2P9+tnEjmAWhd2fHTmtdg9ZqpauobuYSw3xHIA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789201587; c=relaxed/simple; bh=PUCPMn+73ZfH0uCHvI5sdQ84aDEztubs7amQmnJNj+o=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=Ty/tzG8O8ka+d4xEwqhjckCsgjnH5A04vvyfG+4zN1TdPeXQDuX9g5n8P/Z4y/3eBuj2AMzI9NK6mfzbAR49m+t2ex9AHNzuVEYUdgDFuHOxmaezKq3ifaOWfhqjjIpwUdMFfQkIDw+tN5Pgv05578qLO7s+qf9rKyLQSq+Eh+I= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=163.com; spf=pass smtp.mailfrom=163.com; dkim=pass (1024-bit key) header.d=163.com header.i=@163.com header.b=mZ8qcMQ0; arc=none smtp.client-ip=220.197.31.2 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=163.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=163.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=163.com header.i=@163.com header.b="mZ8qcMQ0" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=163.com; s=s110527; h=From:To:Subject:Date:Message-ID:MIME-Version; bh=fY YZoGWdlGWHi1GqFofODMAvVspXQpVNRnI7u63JZdA=; b=mZ8qcMQ0jNua/wD0a2 R5+bUyUTNUp77BIOXdh3ftCnuPvuOYstV0lvDqiAuvWUINtT/j36FGVtXK12uPq2 UvG0+ZU3l4GcQw4Lh1f0Df5nwZlcZ5GTlQ2UjjO7CQlIP2P3yAf6M//MA9O7SEZj GTD8sG8BhEqw6XsepLNtEcMVQ= Received: from VM-0-3-ubuntu.. (unknown []) by gzsmtp3 (Coremail) with SMTP id PigvCgDnr7FODKVqmfMoSQ--.44660S4; Sat, 12 Sep 2026 16:24:49 +0800 (CST) From: Longlong Xia To: xu.xin16@zte.com.cn Cc: akpm@linux-foundation.org, baohua@kernel.org, baolin.wang@linux.alibaba.com, borntraeger@linux.ibm.com, chengming.zhou@linux.dev, david@kernel.org, dev.jain@arm.com, gourry@gourry.net, harry@kernel.org, lance.yang@linux.dev, liam@infradead.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, nico.pache@linux.dev, peterz@infradead.org, rppt@kernel.org, surenb@google.com, usama.arif@linux.dev, vbabka@kernel.org, ziy@nvidia.com, ljs@kernel.org, Longlong Xia Subject: [PATCH RFC 2/3] mm/ksm: use the VMA lock when looking up mergeable pages Date: Sat, 12 Sep 2026 16:24:06 +0800 Message-ID: <20260912082407.75875-3-xialonglong2025@163.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20260912082407.75875-1-xialonglong2025@163.com> References: <20260911160421076_KNXun8Mpp9Xj7fxHG0i7@zte.com.cn> <20260912082407.75875-1-xialonglong2025@163.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-CM-TRANSID: PigvCgDnr7FODKVqmfMoSQ--.44660S4 X-Coremail-Antispam: 1Uf129KBjvJXoWxArW8Gr4kWr4xZFWUCw4kZwb_yoWrGw48pF WfKas8KrWakr43u34fAr4qyr4Ygwn2gayUAFy3Ja1avw1rtF43GayfJa48JFy5GrWxG3Zx Ar4jyFn3Canrt3DanT9S1TB71UUUUU7qnTZGkaVYY2UrUUUUjbIjqfuFe4nvWSU5nxnvy2 9KBjDUYxBIdaVFxhVjvjDU0xZFpf9x07j92-nUUUUU= X-CM-SenderInfo: x0ldz0pqjo00rjsqjki6rwjhhfrp/xtbC9xEbFGqlDFGSgwAA3q Content-Type: text/plain; charset="utf-8" From: Longlong Xia get_mergeable_page() currently takes mm->mmap_lock for every unstable-tree lookup, so KSM is serialized with layout changes to unrelated VMAs. Try lock_vma_under_rcu() while holding an mm_users reference and walk the page tables under that VMA's read lock. Retain the mmap read-lock path as a fallback when the VMA cannot be locked or per-VMA locking is disabled. Extend folio_walk_start() to verify a VMA read lock and keep the mm_users reference across the walk, since a VMA lock alone does not prevent exit_mmap() from freeing page tables. Drop the reference before waiting for mmap_lock on the fallback path so an exiting mm is not delayed. Assisted-by: Zcode:GLM-5.3 Signed-off-by: Longlong Xia --- mm/ksm.c | 64 +++++++++++++++++++++++++++++++++++++++++++++----------- 1 file changed, 52 insertions(+), 12 deletions(-) diff --git a/mm/ksm.c b/mm/ksm.c index 49d48d1e0998..aee1a1b49b1b 100644 --- a/mm/ksm.c +++ b/mm/ksm.c @@ -811,21 +811,20 @@ static void break_cow(struct ksm_rmap_item *rmap_item) mmap_read_unlock(mm); } =20 -static struct page *get_mergeable_page(struct ksm_rmap_item *rmap_item) +/* + * Get the page that @addr maps in @vma, with an elevated reference, or NU= LL + * when the address no longer maps an anon page. The caller must hold a l= ock + * that stabilizes @vma: either the mmap read lock, or the vma read lock + * together with an mm_users reference. + */ +static struct page *__get_mergeable_page(struct vm_area_struct *vma, + unsigned long addr, folio_walk_flags_t flags) { - struct mm_struct *mm =3D rmap_item->mm; - unsigned long addr =3D rmap_item->address; - struct vm_area_struct *vma; - struct page *page =3D NULL; struct folio_walk fw; + struct page *page =3D NULL; struct folio *folio; =20 - mmap_read_lock(mm); - vma =3D find_mergeable_vma(mm, addr); - if (!vma) - goto out; - - folio =3D folio_walk_start(&fw, vma, addr, 0); + folio =3D folio_walk_start(&fw, vma, addr, flags); if (folio) { if (!folio_is_zone_device(folio) && folio_test_anon(folio)) { @@ -834,11 +833,52 @@ static struct page *get_mergeable_page(struct ksm_rma= p_item *rmap_item) } folio_walk_end(&fw, vma); } -out: if (page) { flush_anon_page(vma, page, addr); flush_dcache_page(page); } + return page; +} + +static struct page *get_mergeable_page(struct ksm_rmap_item *rmap_item) +{ + struct mm_struct *mm =3D rmap_item->mm; + unsigned long addr =3D rmap_item->address; + struct vm_area_struct *vma; + struct page *page =3D NULL; + + /* + * Try the vma lock before the mmap lock, so that ksmd does not queue + * behind a writer that changes the address space layout anywhere in + * this mm: the vma lock only contends with modification of this very + * vma. Pin mm_users for the walk: exit_mmap() frees the page tables + * under the mmap lock alone, so a vma read lock cannot keep it away, + * but an mm_users reference can; the pin also stands in for the + * ksm_test_exit() check of find_mergeable_vma() on this path. Drop + * it again before waiting for the mmap lock below, so that an exiting + * mm is not delayed by us. + */ + if (IS_ENABLED(CONFIG_PER_VMA_LOCK) && mmget_not_zero(mm)) { + vma =3D lock_vma_under_rcu(mm, addr); + if (vma) { + if ((vma->vm_flags & VM_MERGEABLE) && vma->anon_vma) + page =3D __get_mergeable_page(vma, addr, FW_VMA_LOCKED); + vma_end_read(vma); + mmput_async(mm); + return page; + } + mmput_async(mm); + } + + /* + * The vma is being modified, or CONFIG_PER_VMA_LOCK is off: take the + * mmap read lock as before. We are prepared to wait rather than skip + * this page, so a contended mm only slows down its own merging. + */ + mmap_read_lock(mm); + vma =3D find_mergeable_vma(mm, addr); + if (vma) + page =3D __get_mergeable_page(vma, addr, 0); mmap_read_unlock(mm); return page; } --=20 2.43.0 From nobody Fri Sep 25 12:38:49 2026 Received: from m16.mail.163.com (m16.mail.163.com [117.135.210.5]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D0B8235AC1E for ; Sat, 12 Sep 2026 08:26:20 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=117.135.210.5 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789201585; cv=none; b=mdmhBnYp6/6002OYsYkUAzPeMvfJEZpi64A4dJ3lj+GBsuNhFbDJI/fjSkPbnzfSCLtxEjUqppYA54O3evk/YAOH9L0LFaaeyCEXHkNjnuqNgftdvlCn+dMaE6egQwtbXkYwWZfXVkCV1tl9Q9Lc7VgvoAXxtMp4Q/Sa0GrRFfw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789201585; c=relaxed/simple; bh=01E7MTFBXBA9FD67qbSGieQEP4bz4k4o8tXOVDhTb/4=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=lUmDgfar65HADQap/CKzLDgcUsZWJenKMzJukh/632SW+fRSug3GK45gf1jdMMpRj7Q0Fn7yGjVV+OTzTJj9X5UZ6bkd7xj/Qlmes6BkRAaj7Sg9P+Ufl/yVTW/AJG3LHq/Y0Ld0ghTIDrWiDnR7t1DQkNPcnr9flS72/gWJUDY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=163.com; spf=pass smtp.mailfrom=163.com; dkim=pass (1024-bit key) header.d=163.com header.i=@163.com header.b=ISqqJMZc; arc=none smtp.client-ip=117.135.210.5 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=163.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=163.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=163.com header.i=@163.com header.b="ISqqJMZc" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=163.com; s=s110527; h=From:To:Subject:Date:Message-ID:MIME-Version; bh=Yn cf0PxijlZoQpU9WpOljLbavmoG2PB3EBdY+B+ayfA=; b=ISqqJMZcXy0NipoBmQ DGy+LrADHOcT2G5d3Giav/XLw7t9GyjKbu8FlOV0xqA3zVQFcEWdW/cildYiwUnh YTY1sTDIiCu4d94LimVip3iwJte2sei5ZfwxWTq8G7YAXW9/itMbazDCUDSI1ULp n6m3SUEhuJr+nBhKAsZ0lzUEY= Received: from VM-0-3-ubuntu.. (unknown []) by gzsmtp3 (Coremail) with SMTP id PigvCgDnr7FODKVqmfMoSQ--.44660S5; Sat, 12 Sep 2026 16:24:50 +0800 (CST) From: Longlong Xia To: xu.xin16@zte.com.cn Cc: akpm@linux-foundation.org, baohua@kernel.org, baolin.wang@linux.alibaba.com, borntraeger@linux.ibm.com, chengming.zhou@linux.dev, david@kernel.org, dev.jain@arm.com, gourry@gourry.net, harry@kernel.org, lance.yang@linux.dev, liam@infradead.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, nico.pache@linux.dev, peterz@infradead.org, rppt@kernel.org, surenb@google.com, usama.arif@linux.dev, vbabka@kernel.org, ziy@nvidia.com, ljs@kernel.org, Longlong Xia Subject: [PATCH RFC 3/3] mm/ksm: scan VMAs with per-VMA locks Date: Sat, 12 Sep 2026 16:24:07 +0800 Message-ID: <20260912082407.75875-4-xialonglong2025@163.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20260912082407.75875-1-xialonglong2025@163.com> References: <20260911160421076_KNXun8Mpp9Xj7fxHG0i7@zte.com.cn> <20260912082407.75875-1-xialonglong2025@163.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-CM-TRANSID: PigvCgDnr7FODKVqmfMoSQ--.44660S5 X-Coremail-Antispam: 1Uf129KBjvJXoW3JrW8KFW3Xw48ZFy8GF4rXwb_yoW7Ww15pF ZrJa98Wr45Cw4fXFZ3XF48CrWagw4vga1UJr43Cwna9wnxtrsxuF4rKayUAF98Crn3Xa4a vr4jyFnxCF1avaDanT9S1TB71UUUUU7qnTZGkaVYY2UrUUUUjbIjqfuFe4nvWSU5nxnvy2 9KBjDUYxBIdaVFxhVjvjDU0xZFpf9x07j9yCXUUUUU= X-CM-SenderInfo: x0ldz0pqjo00rjsqjki6rwjhhfrp/xtbC9xMbFGqlDFOSjwAA3m Content-Type: text/plain; charset="utf-8" From: Longlong Xia KSM currently holds mm->mmap_lock while scanning the VMAs of an mm. Use lock_next_vma() and walk_page_range_vma() to acquire each VMA's read lock and release it after finding a candidate page, allowing unrelated VMA updates to proceed while ksmd scans. Pin mm_users for the duration of the lockless VMA traversal to keep page tables alive, and retain the mmap-lock scan as a fallback when a VMA lock cannot be acquired or per-VMA locking is disabled. Allocation failures preserve the existing early-stop semantics. This reduces mmap_lock contention for workloads with concurrent address-space updates while preserving existing scan semantics. When the per-VMA scan completes without finding a candidate, remember the complete traversal and revalidate the mmap-lock sequence before entering the fallback. If the address space is unchanged, skip the duplicate VMA walk; if a VMA writer raced with the traversal, restart the fallback walk. Assisted-by: Zcode:GLM-5.3 Signed-off-by: Longlong Xia --- mm/ksm.c | 107 +++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 107 insertions(+) diff --git a/mm/ksm.c b/mm/ksm.c index aee1a1b49b1b..2c0b2adb83d4 100644 --- a/mm/ksm.c +++ b/mm/ksm.c @@ -2647,6 +2647,98 @@ static struct mm_walk_ops ksm_next_page_ops =3D { .walk_lock =3D PGWALK_RDLOCK, }; =20 +#ifdef CONFIG_PER_VMA_LOCK +static const struct mm_walk_ops ksm_next_page_vma_ops =3D { + .pmd_entry =3D ksm_next_page_pmd_entry, + .walk_lock =3D PGWALK_VMA_RDLOCK_VERIFY, +}; + +/* Return true when a candidate was processed, including allocation failur= e. */ +static bool scan_get_next_rmap_item_vma(struct ksm_mm_slot *mm_slot, + struct page **page, struct ksm_rmap_item **result, + bool *complete, unsigned int *mm_wr_seq) +{ + struct mm_struct *mm =3D mm_slot->slot.mm; + unsigned long address =3D ksm_scan.address; + struct vm_area_struct *vma; + struct vma_iterator vmi; + bool done =3D false; + bool mmap_unlocked; + + if (!mmget_not_zero(mm)) + return false; + + mmap_unlocked =3D mmap_lock_speculate_try_begin(mm, mm_wr_seq); + + for (;;) { + rcu_read_lock(); + vma_iter_init(&vmi, mm, address); + vma =3D lock_next_vma(mm, &vmi, address); + rcu_read_unlock(); + if (IS_ERR_OR_NULL(vma)) + break; + + address =3D vma->vm_end; + if (!(vma->vm_flags & VM_MERGEABLE)) + goto next_vma; + if (ksm_scan.address < vma->vm_start) + ksm_scan.address =3D vma->vm_start; + if (!vma->anon_vma) + ksm_scan.address =3D vma->vm_end; + + while (ksm_scan.address < vma->vm_end) { + struct ksm_next_page_arg arg; + struct ksm_rmap_item *rmap_item; + int found; + + found =3D walk_page_range_vma(vma, ksm_scan.address, + vma->vm_end, &ksm_next_page_vma_ops, &arg); + if (found <=3D 0) { + VM_WARN_ON_ONCE(found < 0); + ksm_scan.address =3D vma->vm_end; + break; + } + + ksm_scan.address =3D arg.addr; + flush_anon_page(vma, arg.page, arg.addr); + flush_dcache_page(arg.page); + rmap_item =3D get_next_rmap_item(mm_slot, + ksm_scan.rmap_list, arg.addr); + if (rmap_item) { + ksm_scan.rmap_list =3D &rmap_item->rmap_list; + if (should_skip_rmap_item(arg.folio, rmap_item)) { + folio_put(arg.folio); + ksm_scan.address +=3D PAGE_SIZE; + cond_resched(); + continue; + } + ksm_scan.address +=3D PAGE_SIZE; + *page =3D arg.page; + } else { + folio_put(arg.folio); + } + *result =3D rmap_item; + done =3D true; + vma_end_read(vma); + goto out; + } +next_vma: + /* + * Don't advance ksm_scan.address for VMAs the mmap-lock loop + * skips with a plain continue: the cursor has to stay 0 when + * this mm holds no VM_MERGEABLE vma, so the fallback walk can + * remove the mm from the scan list at the end of the pass. + */ + vma_end_read(vma); + cond_resched(); + } + *complete =3D mmap_unlocked && !vma; +out: + mmput_async(mm); + return done; +} +#endif + static struct ksm_rmap_item *scan_get_next_rmap_item(struct page **page) { struct mm_struct *mm; @@ -2655,6 +2747,9 @@ static struct ksm_rmap_item *scan_get_next_rmap_item(= struct page **page) struct vm_area_struct *vma; struct ksm_rmap_item *rmap_item; struct vma_iterator vmi; + bool skip_vma_scan =3D false; + bool vma_scan_complete =3D false; + unsigned int mm_wr_seq; int nid; =20 if (list_empty(&ksm_mm_head.slot.mm_node)) @@ -2719,12 +2814,23 @@ static struct ksm_rmap_item *scan_get_next_rmap_ite= m(struct page **page) =20 slot =3D &mm_slot->slot; mm =3D slot->mm; +#ifdef CONFIG_PER_VMA_LOCK + rmap_item =3D NULL; + if (scan_get_next_rmap_item_vma(mm_slot, page, &rmap_item, + &vma_scan_complete, &mm_wr_seq)) + return rmap_item; +#endif + /* Recheck the end of the scan under mmap_lock before removing the mm. */ vma_iter_init(&vmi, mm, ksm_scan.address); =20 mmap_read_lock(mm); + if (vma_scan_complete && !mmap_lock_speculate_retry(mm, mm_wr_seq)) + skip_vma_scan =3D true; if (ksm_test_exit(mm)) goto no_vmas; =20 + if (skip_vma_scan) + goto scan_cleanup; for_each_vma(vmi, vma) { if (!(vma->vm_flags & VM_MERGEABLE)) continue; @@ -2785,6 +2891,7 @@ static struct ksm_rmap_item *scan_get_next_rmap_item(= struct page **page) } } =20 +scan_cleanup: if (ksm_test_exit(mm)) { no_vmas: ksm_scan.address =3D 0; --=20 2.43.0