From nobody Thu Sep 24 21:19:23 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-alma10-1.taild15c8.ts.net [100.103.45.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B241A444713; Sun, 20 Sep 2026 14:13:36 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=100.103.45.18 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789913618; cv=none; b=Z1cOiSmRhdC1LrTJwvBtpnRaAqEEobVVgMRdL0t9wSu55u5y0eOfxWd7h7OI8XTK1E/cmLXbddx002R57MIm7ZiAjmRyQu1XYsGS067fuRrcinAGLMjI6w1zG4N8xFrAlHGCFsOhe8gQ0S9rSPoKvEtyJN4FZx8vM3oyf5eTDxY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789913618; c=relaxed/simple; bh=7NKK0j5EusrBCNKL26BUuITRpi+S9GQj7txy+LzBAXE=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=LuhmVNZ2Ad791pvahtQFBYb+4LKwcTHmQKPu8vm3oRkZUzed4b5v8DlT9Z01ex+XCO6pHRku/BXA2dCQnac9PmQIkb3JBDpe6vXjzu4LFd4ZxrtqfushIKyQBPN54JYz6MGrG4tArpRhQlUTnaJX6qti6I3qGAXTfcAiLY7PwIA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=OWv++KQH; arc=none smtp.client-ip=100.103.45.18 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="OWv++KQH" Received: by smtp.kernel.org (Postfix) with ESMTPSA id A84871F00893; Sun, 20 Sep 2026 14:13:33 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1789913616; bh=wuTTB08dUEw+auFKFpO6hUoO88RZSfTJOEDWI/SO6ck=; h=From:Date:Subject:References:In-Reply-To:To:Cc; b=OWv++KQHYKsrJTLCNMAywUPNOYbKv4lS6Yb4rf2jm77yyS86ot95L3nLZJYOxx/Tb YYPhL2Q1xnx/AoOPImjKgqJb7SuOEndoICWeA+zDq0IpQsZ2RCu6ruBLe5HEO97y9l 0FaEDgaWEa0GbPGM0N+JK4W28977b3AA5NIUu2XpQGwa3luOr1FdMxbfmhG4VqEDHo PJDhqv3kGfYia8X/A6+V6/D7m4T/ooqu4csKi0lGDwKYcjbC8Obwm7Ie8pub8jSvti lpUubk43vg7ZAucfM9SmzJRTnbCXEQHL1EVkFkp1YtQgx+A1PFoIz5nLUHxYtD5SMZ Vy025jNvpbglQ== From: "Lorenzo Stoakes (ARM)" Date: Sun, 20 Sep 2026 15:13:10 +0100 Subject: [PATCH 1/2] mm/mremap: fix locked_vm leak from MREMAP_DONTUNMAP self-merge Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20260920-fix-dontunmap-partial-self-merge-v1-1-6ffb556f8f8b@kernel.org> References: <20260920-fix-dontunmap-partial-self-merge-v1-0-6ffb556f8f8b@kernel.org> In-Reply-To: <20260920-fix-dontunmap-partial-self-merge-v1-0-6ffb556f8f8b@kernel.org> To: Andrew Morton , "Liam R. Howlett" , Vlastimil Babka , Jann Horn , Pedro Falcato , Brian Geffon , Minchan Kim , Kiryl Shutsemau Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, "Lorenzo Stoakes (ARM)" , stable@vger.kernel.org X-Mailer: b4 0.14.3 X-Developer-Signature: v=1; a=openpgp-sha256; l=7383; i=ljs@kernel.org; h=from:subject:message-id; bh=7NKK0j5EusrBCNKL26BUuITRpi+S9GQj7txy+LzBAXE=; b=owGbwMvMwCV2fu7ZrsZH9SKMp9WSGLLWv+LUExRWZZg7dfWU04ciVW7JrD67S67ESGuX7cbLe rrvVa+md5SyMIhxMciKKbI8/yK+P0gkbF7nBX83mDmsTCBDGLg4BWAinz8xMmx/7SX676+F3kTH PUctGPj03dkVl+Wx8er9tko0lQliZWX4nxKpr8K0OMBeeH3wui0Pqn4qzrq1U/rKlakNK03nnjI 6yQgA X-Developer-Key: i=ljs@kernel.org; a=openpgp; fpr=E7F417BF5214569E89D04F46CF9DCD8A81E27F14 The MREMAP_DONTUNMAP feature is highly unusual in that it permits mremap() operations that keep the original VMA in place. Historically this has led to a lot of bugs where non-obvious interactions occur between existing mremap() operations and the original VMA. Fix another of these - self-merge. Self-merge occurs when a VMA is moved in front of or behind itself and the attributes of the VMA permit such a merge. Practically this can only happen for unfaulted anonymous VMAs due to the page offset equality requirement for merge: |------------| | | | v |...........||-----------||...........| | || unfaulted || | |...........||-----------||...........| ^ | | | |------------| This becomes problematic if the VMA is configured by the user to mlock-on-fault, i.e. the VMA_LOCKED_BIT, VMA_LOCKONFAULT_BIT VMA flags are set. MREMAP_DONTUNMAP clears mlock flags for the source VMA and maintains them for the destination VMA. Self-merge makes this impossible (there is only one VMA) and incorrectly clears the destination VMA's mlock flags. This causes a leak in mm->locked_vm as clearing this flag does not decrement the counter and the VMA no longer has VMA_LOCKED_BIT set so it is not decremented on unmap. Resolve this by simply disallowing a self-merge in this case - the source and destination VMAs are kept distinct and then are able to have distinct mlock() flags. Update dontunmap_complete() to make the now-redundant self-merge check a VM_WARN_ON_ONCE() instead to guard against future regressions. Also update the VMA userland tests to reflect the change. Fixes: e346b3813067 ("mm/mremap: add MREMAP_DONTUNMAP to mremap()") Cc: Signed-off-by: Lorenzo Stoakes (ARM) --- mm/mremap.c | 8 ++++++-- mm/vma.c | 17 ++++++++++++++++- mm/vma.h | 2 +- tools/testing/vma/tests/vma.c | 10 +++++----- 4 files changed, 28 insertions(+), 9 deletions(-) diff --git a/mm/mremap.c b/mm/mremap.c index 1122282a1d6a..73f52c45705c 100644 --- a/mm/mremap.c +++ b/mm/mremap.c @@ -1275,7 +1275,8 @@ static int copy_vma_and_data(struct vma_remap_struct = *vrm, PAGETABLE_MOVE(pmc, NULL, NULL, vrm->addr, vrm->new_addr, vrm->old_len); =20 new_vma =3D copy_vma(&vma, vrm->new_addr, vrm->new_len, new_pgoff, - new_anon_pgoff, &pmc.need_rmap_locks); + new_anon_pgoff, &pmc.need_rmap_locks, + vrm->flags & MREMAP_DONTUNMAP); if (!new_vma) { vrm_uncharge(vrm); *new_vma_ptr =3D NULL; @@ -1335,6 +1336,9 @@ static void dontunmap_complete(struct vma_remap_struc= t *vrm, unsigned long old_start =3D vma->vm_start; unsigned long old_end =3D vma->vm_end; =20 + /* Self-merge is disallowed. */ + VM_WARN_ON_ONCE(new_vma =3D=3D vma); + /* We always clear VMA_LOCKED[ONFAULT]_BIT on the old VMA. */ vma_clear_flags_mask(vma, VMA_LOCKED_MASK); =20 @@ -1342,7 +1346,7 @@ static void dontunmap_complete(struct vma_remap_struc= t *vrm, * anon_vma links of the old vma is no longer needed after its page * table has been moved. */ - if (new_vma !=3D vma && start =3D=3D old_start && end =3D=3D old_end) { + if (start =3D=3D old_start && end =3D=3D old_end) { const pgoff_t pgoff_unfaulted =3D vma->vm_start >> PAGE_SHIFT; =20 unlink_anon_vmas(vma); diff --git a/mm/vma.c b/mm/vma.c index 0db3f1222cad..55d4d0939129 100644 --- a/mm/vma.c +++ b/mm/vma.c @@ -1946,7 +1946,7 @@ static int vma_link(struct mm_struct *mm, struct vm_a= rea_struct *vma) */ struct vm_area_struct *copy_vma(struct vm_area_struct **vmap, unsigned long addr, unsigned long len, pgoff_t pgoff, - pgoff_t anon_pgoff, bool *need_rmap_locks) + pgoff_t anon_pgoff, bool *need_rmap_locks, bool keep_source) { struct vm_area_struct *vma =3D *vmap; unsigned long old_vma_start =3D vma->vm_start; @@ -1984,6 +1984,21 @@ struct vm_area_struct *copy_vma(struct vm_area_struc= t **vmap, vmg.pgoff =3D pgoff; vmg.anon_pgoff =3D anon_pgoff; vmg.next =3D vma_iter_next_rewind(&vmi, NULL); + + /* + * If the original VMA is kept (MREMAP_DONTUNMAP), the source and + * destination VMA must be treated distinctly. + * + * A merge violates this, so in this case disallow a self-merge. + */ + if (can_self_merge && keep_source) { + if (vmg.prev =3D=3D vma) + vmg.prev =3D NULL; + if (vmg.next =3D=3D vma) + vmg.next =3D NULL; + can_self_merge =3D false; + } + new_vma =3D vma_merge_copied_range(&vmg); =20 if (new_vma) { diff --git a/mm/vma.h b/mm/vma.h index b2c3bc832a48..03ed8afd0c1f 100644 --- a/mm/vma.h +++ b/mm/vma.h @@ -534,7 +534,7 @@ void unlink_file_vma_batch_add(struct unlink_vma_file_b= atch *vb, =20 struct vm_area_struct *copy_vma(struct vm_area_struct **vmap, unsigned long addr, unsigned long len, pgoff_t pgoff, - pgoff_t anon_pgoff, bool *need_rmap_locks); + pgoff_t anon_pgoff, bool *need_rmap_locks, bool keep_source); =20 struct anon_vma *find_mergeable_anon_vma(struct vm_area_struct *vma); =20 diff --git a/tools/testing/vma/tests/vma.c b/tools/testing/vma/tests/vma.c index c8ef7b8cd46b..e973e0a6d1a8 100644 --- a/tools/testing/vma/tests/vma.c +++ b/tools/testing/vma/tests/vma.c @@ -40,7 +40,7 @@ static bool test_copy_vma(void) vma =3D alloc_and_link_vma(&mm, 0x1000, 0x2000, 1, vma_flags); vma_set_anonymous(vma); vma_orig =3D vma; - vma_new =3D copy_vma(&vma, 0x2000, 0x1000, 1, 1, &need_locks); + vma_new =3D copy_vma(&vma, 0x2000, 0x1000, 1, 1, &need_locks, false); ASSERT_EQ(vma_new, vma_orig); ASSERT_EQ(vma, vma_orig); ASSERT_EQ(vma_new->vm_start, 0x1000); @@ -53,7 +53,7 @@ static bool test_copy_vma(void) vma =3D alloc_and_link_vma(&mm, 0x2000, 0x3000, 2, vma_flags); vma_set_anonymous(vma); vma_orig =3D vma; - vma_new =3D copy_vma(&vma, 0x1000, 0x1000, 2, 2, &need_locks); + vma_new =3D copy_vma(&vma, 0x1000, 0x1000, 2, 2, &need_locks, false); ASSERT_EQ(vma_new, vma_orig); ASSERT_EQ(vma, vma_orig); ASSERT_EQ(vma_new->vm_start, 0x1000); @@ -71,7 +71,7 @@ static bool test_copy_vma(void) vma =3D alloc_and_link_vma(&mm, 0x3000, 0x4000, 3, vma_flags); vma_set_anonymous(vma); vma_orig =3D vma; - vma_new =3D copy_vma(&vma, 0x2000, 0x1000, 3, 3, &need_locks); + vma_new =3D copy_vma(&vma, 0x2000, 0x1000, 3, 3, &need_locks, false); ASSERT_NE(vma_new, vma_orig); ASSERT_EQ(vma_new, vma); ASSERT_EQ(vma_new->vm_start, 0x1000); @@ -82,7 +82,7 @@ static bool test_copy_vma(void) /* Move backwards and do not merge. */ =20 vma =3D alloc_and_link_vma(&mm, 0x3000, 0x5000, 3, vma_flags); - vma_new =3D copy_vma(&vma, 0, 0x2000, 0, 3, &need_locks); + vma_new =3D copy_vma(&vma, 0, 0x2000, 0, 3, &need_locks, false); ASSERT_NE(vma_new, vma); ASSERT_EQ(vma_new->vm_start, 0); ASSERT_EQ(vma_new->vm_end, 0x2000); @@ -95,7 +95,7 @@ static bool test_copy_vma(void) =20 vma =3D alloc_and_link_vma(&mm, 0, 0x2000, 0, vma_flags); vma_next =3D alloc_and_link_vma(&mm, 0x6000, 0x8000, 6, vma_flags); - vma_new =3D copy_vma(&vma, 0x4000, 0x2000, 4, 4, &need_locks); + vma_new =3D copy_vma(&vma, 0x4000, 0x2000, 4, 4, &need_locks, false); vma_assert_attached(vma_new); =20 ASSERT_EQ(vma_new, vma_next); --=20 2.55.0 From nobody Thu Sep 24 21:19:23 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-alma10-1.taild15c8.ts.net [100.103.45.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9B0843C457D; Sun, 20 Sep 2026 14:13:39 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=100.103.45.18 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789913621; cv=none; b=cHDNuCSOPIW4uD1DBKSzE95wd/lONMw1uZUOKBTjb7mHdIWIFKdu6XxJqBZ5I6n8zPF5HaVTJ7o8gRqbUYb/Lrv7+RB/UxghYwnGLDUAONsRXqTrH4vrlmVBJ7j2dae7XKefTwYT8Ap7/7qSMllgc6VHd3urX5vxoUVcHJSldVs= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789913621; c=relaxed/simple; bh=yjmLeTXCkk2mbYGvT4Wqq1IUIfo5bp3zlxd9nUSq81E=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=RCyBKcwSObNWNnEyL2YChWnGPcy4l4Jj8hl7niuIXf0DmmqXe4bGkvAhQgRmzGRS8fB+AtVgu0bTVuFq0e+Pqd1ASPpiYQo8E6jq1O7xtLsKNco/O78EttzFfo0f7Yp/2ESbCKhoMHRfRFAxsoPHgTeBQwPHSVZ4j3wXuY1bNDI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=nxvZxroO; arc=none smtp.client-ip=100.103.45.18 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="nxvZxroO" Received: by smtp.kernel.org (Postfix) with ESMTPSA id D269B1F000FF; Sun, 20 Sep 2026 14:13:36 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1789913619; bh=xSrOUEQX5cEdTjwoeayswD1PWITxAREjqOm5wmTNA7M=; h=From:Date:Subject:References:In-Reply-To:To:Cc; b=nxvZxroOyGkQt3BsmWkXQQ4iPfEvmKakF2Otjj/Ox9xSBb43pEpVMox31SBQ7zzum mcKVc0L3CDb0QcAruI7IS+MsABquxtd8G6hJPEJufUUMsgu69/ebD/uBnwl8OiTz/8 1h2cFLuRySdRSGjaCdVOFdQAP4GNAAZsH57aSr7RTCRXILJJgzUUJ9WP4JZbYznSZ6 blIuihj6jAMYeN9YmjBEc4/ENED/gRqYflUJXVClt/08zDf/e2nwhIPxBwZci7RRh8 75TsGgJhfnf99HugZ77WYn2N1//b98AqHlk/0FazP7KtnzzRHdQR9Z3H6eSsYtgngR 7KLL8wWmzWBCg== From: "Lorenzo Stoakes (ARM)" Date: Sun, 20 Sep 2026 15:13:11 +0100 Subject: [PATCH 2/2] mm/mremap: fix locked_vm leak by splitting VMA for MREMAP_DONTUNMAP Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20260920-fix-dontunmap-partial-self-merge-v1-2-6ffb556f8f8b@kernel.org> References: <20260920-fix-dontunmap-partial-self-merge-v1-0-6ffb556f8f8b@kernel.org> In-Reply-To: <20260920-fix-dontunmap-partial-self-merge-v1-0-6ffb556f8f8b@kernel.org> To: Andrew Morton , "Liam R. Howlett" , Vlastimil Babka , Jann Horn , Pedro Falcato , Brian Geffon , Minchan Kim , Kiryl Shutsemau Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, "Lorenzo Stoakes (ARM)" , stable@vger.kernel.org X-Mailer: b4 0.14.3 X-Developer-Signature: v=1; a=openpgp-sha256; l=9174; i=ljs@kernel.org; h=from:subject:message-id; bh=yjmLeTXCkk2mbYGvT4Wqq1IUIfo5bp3zlxd9nUSq81E=; b=owGbwMvMwCV2fu7ZrsZH9SKMp9WSGLLWv+JS/l1vKVYZsvrnI4XsS0/faV54flr875Xv3Wv6P F53ikfs7ihlYRDjYpAVU2R5/kV8f5BI2LzOC/5uMHNYmUCGMHBxCsBEUu4w/I/Jyjz03GxiTfVu zd7ncx5wCVZFnp9d93/RN5X/pQfEE6cw/K/2mpi8wfufQ2KYSeizogbT1C9uvSXOJQ/uFd1U7PR sYwAA X-Developer-Key: i=ljs@kernel.org; a=openpgp; fpr=E7F417BF5214569E89D04F46CF9DCD8A81E27F14 The MREMAP_DONTUNMAP feature is highly unusual in that it permits mremap() operations that keep the original VMA in place. Historically this has led to a lot of bugs where non-obvious interactions occur between existing mremap() operations and the original VMA. Fix another of these - partial copies. The long-standing mremap() partial VMA logic has the baked-in assumption that the originating VMA is unmapped and thus moved. However MREMAP_DONTUNMAP defeats this by performing a partial copy instead since it keeps the source VMA around. An mremap(..., MREMAP_DONTUNMAP) operation disallows resizing of the VMA, but the operation can be performed partially: |-----------------| | | | v <------> <------> .new_sz. new_sz |--.------.--| |------| | .source. | | dest | |--.------.--| |------| <------------> old_sz The page tables in the specified range are moved, but the original VMA is kept intact. This interacts poorly with mlock()'d VMAs, as the VMA_LOCKED_BIT flag is cleared for the entire source VMA and set for the entire destination VMA. This results in an mm->locked_vm leak as the change is therefore not accounted correctly. The clear solution here is to make the portion of the source VMA which is mremap()'d distinct from the rest of it, a.k.a. split it. Therefore resolve this issue by splitting it ahead of the rest of the mremap() operation. In order to make this change re-expose split_vma() in vma.h for CONFIG_MMU (nommu doesn't compile mremap.c and uses a static helper instead). A quick search of how MREMAP_DONTUNMAP is used in the wild suggests that the partial case is either unused or rarely used, so this should not result in unreasonable VMA proliferation. Since this makes every mremap() MREMAP_DONTUNMAP operation operate across an entire, distinct, VMA, also eliminate now-redundant code checking for this in dontunmap_complete(). Finally, update the sys_map_count check to account for this case. Fixes: e346b3813067 ("mm/mremap: add MREMAP_DONTUNMAP to mremap()") Cc: Signed-off-by: Lorenzo Stoakes (ARM) --- mm/mremap.c | 81 ++++++++++++++++++++++++++++++++++++++-------------------= ---- mm/vma.c | 2 +- mm/vma.h | 5 ++++ 3 files changed, 56 insertions(+), 32 deletions(-) diff --git a/mm/mremap.c b/mm/mremap.c index 73f52c45705c..49dc25d8a34d 100644 --- a/mm/mremap.c +++ b/mm/mremap.c @@ -1037,6 +1037,7 @@ static void vrm_stat_account(struct vma_remap_struct = *vrm, } =20 static bool __check_map_count_against_split(struct mm_struct *mm, + bool is_dontunmap, bool before_unmaps) { const int sys_map_count =3D get_sysctl_max_map_count(); @@ -1088,26 +1089,38 @@ static bool __check_map_count_against_split(struct = mm_struct *mm, * Therefore we must check to ensure we have headroom of 2 additional * VMAs. */ - return map_count + 2 <=3D sys_map_count; + map_count +=3D 2; + + /* + * If MREMAP_DONTUNMAP is set and a partial operation is performed, + * the VMA is split ahead of time and the -1 observed above doesn't + * apply. + */ + if (is_dontunmap) + map_count++; + + return map_count <=3D sys_map_count; } =20 /* Do we violate the map count limit if we split VMAs when moving the VMA?= */ -static bool check_map_count_against_split(void) +static bool check_map_count_against_split(struct vma_remap_struct *vrm) { return __check_map_count_against_split(current->mm, + vrm->flags & MREMAP_DONTUNMAP, /*before_unmaps=3D*/false); } =20 /* Do we violate the map count limit if we split VMAs prior to early unmap= s? */ -static bool check_map_count_against_split_early(void) +static bool check_map_count_against_split_early(struct vma_remap_struct *v= rm) { return __check_map_count_against_split(current->mm, + vrm->flags & MREMAP_DONTUNMAP, /*before_unmaps=3D*/true); } =20 /* - * Perform checks before attempting to write a VMA prior to it being - * moved. + * Perform checks and preparation before attempting to write a VMA prior t= o it + * being moved. */ static unsigned long prep_move_vma(struct vma_remap_struct *vrm) { @@ -1116,19 +1129,17 @@ static unsigned long prep_move_vma(struct vma_remap= _struct *vrm) unsigned long old_addr =3D vrm->addr; unsigned long old_len =3D vrm->old_len; vm_flags_t dummy =3D vma->vm_flags; + const bool split_before =3D vma->vm_start !=3D old_addr; + const bool split_after =3D vma->vm_end !=3D old_addr + old_len; =20 - /* - * We'd prefer to avoid failure later on in do_munmap: we copy a VMA, - * which may not merge, then (if MREMAP_DONTUNMAP is not set) unmap the - * source, which may split, causing a net increase of 2 mappings. - */ - if (!check_map_count_against_split()) + /* Avoid failure later on. */ + if (!check_map_count_against_split(vrm)) return -ENOMEM; =20 if (vma->vm_ops && vma->vm_ops->may_split) { - if (vma->vm_start !=3D old_addr) + if (split_before) err =3D vma->vm_ops->may_split(vma, old_addr); - if (!err && vma->vm_end !=3D old_addr + old_len) + if (!err && split_after) err =3D vma->vm_ops->may_split(vma, old_addr + old_len); if (err) return err; @@ -1146,6 +1157,21 @@ static unsigned long prep_move_vma(struct vma_remap_= struct *vrm) if (err) return err; =20 + /* + * To account mlock()'d pages correctly in the MREMAP_DONTUNMAP + * case perform any split ahead of time. + */ + if (vrm->flags & MREMAP_DONTUNMAP) { + VMA_ITERATOR(vmi, vma->vm_mm, old_addr); + + if (split_before) + err =3D split_vma(&vmi, vma, old_addr, 1); + if (!err && split_after) + err =3D split_vma(&vmi, vma, old_addr + old_len, 0); + vrm->vmi_needs_invalidate =3D true; + return err; + } + return 0; } =20 @@ -1330,11 +1356,8 @@ static int copy_vma_and_data(struct vma_remap_struct= *vrm, static void dontunmap_complete(struct vma_remap_struct *vrm, struct vm_area_struct *new_vma) { - unsigned long start =3D vrm->addr; - unsigned long end =3D vrm->addr + vrm->old_len; struct vm_area_struct *vma =3D vrm->vma; - unsigned long old_start =3D vma->vm_start; - unsigned long old_end =3D vma->vm_end; + const pgoff_t pgoff_unfaulted =3D vma->vm_start >> PAGE_SHIFT; =20 /* Self-merge is disallowed. */ VM_WARN_ON_ONCE(new_vma =3D=3D vma); @@ -1346,19 +1369,15 @@ static void dontunmap_complete(struct vma_remap_str= uct *vrm, * anon_vma links of the old vma is no longer needed after its page * table has been moved. */ - if (start =3D=3D old_start && end =3D=3D old_end) { - const pgoff_t pgoff_unfaulted =3D vma->vm_start >> PAGE_SHIFT; - - unlink_anon_vmas(vma); - /* - * The VMA is now unfaulted and it is an invariant that - * unfaulted anonymous VMAs have page offset equal to - * vma->vm_start >> PAGE_SHIFT. - */ - vma_set_anon_pgoff(vma, pgoff_unfaulted); - if (vma_is_anonymous(vma) && !vma->vm_file) - vma_set_pgoff(vma, pgoff_unfaulted); - } + unlink_anon_vmas(vma); + /* + * The VMA is now unfaulted and it is an invariant that + * unfaulted anonymous VMAs have page offset equal to + * vma->vm_start >> PAGE_SHIFT. + */ + vma_set_anon_pgoff(vma, pgoff_unfaulted); + if (vma_is_anonymous(vma) && !vma->vm_file) + vma_set_pgoff(vma, pgoff_unfaulted); } =20 static unsigned long move_vma(struct vma_remap_struct *vrm) @@ -2005,7 +2024,7 @@ static unsigned long do_mremap(struct vma_remap_struc= t *vrm) return -EINTR; vrm->mmap_locked =3D true; =20 - if (!check_map_count_against_split_early()) { + if (!check_map_count_against_split_early(vrm)) { mmap_write_unlock(mm); return -ENOMEM; } diff --git a/mm/vma.c b/mm/vma.c index 55d4d0939129..92a2fafd66ae 100644 --- a/mm/vma.c +++ b/mm/vma.c @@ -634,7 +634,7 @@ __split_vma(struct vma_iterator *vmi, struct vm_area_st= ruct *vma, * Split a vma into two pieces at address 'addr', a new vma is allocated * either for the first part or the tail. */ -static int split_vma(struct vma_iterator *vmi, struct vm_area_struct *vma, +int split_vma(struct vma_iterator *vmi, struct vm_area_struct *vma, unsigned long addr, int new_below) { if (vma->vm_mm->map_count >=3D get_sysctl_max_map_count()) diff --git a/mm/vma.h b/mm/vma.h index 03ed8afd0c1f..b9b99fa02a86 100644 --- a/mm/vma.h +++ b/mm/vma.h @@ -555,6 +555,11 @@ int do_brk_flags(struct vma_iterator *vmi, struct vm_a= rea_struct *brkvma, unsigned long unmapped_area(struct vm_unmapped_area_info *info); unsigned long unmapped_area_topdown(struct vm_unmapped_area_info *info); =20 +#ifdef CONFIG_MMU +int split_vma(struct vma_iterator *vmi, struct vm_area_struct *vma, + unsigned long addr, int new_below); +#endif + static inline bool vma_wants_manual_pte_write_upgrade(struct vm_area_struc= t *vma) { /* --=20 2.55.0