From nobody Tue Jun 23 03:00:31 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id F20C1C433F5 for ; Fri, 11 Mar 2022 17:46:15 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S241148AbiCKRrR (ORCPT ); Fri, 11 Mar 2022 12:47:17 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:57166 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237361AbiCKRrH (ORCPT ); Fri, 11 Mar 2022 12:47:07 -0500 Received: from mail-ej1-x634.google.com (mail-ej1-x634.google.com [IPv6:2a00:1450:4864:20::634]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 01FB1133955 for ; Fri, 11 Mar 2022 09:46:03 -0800 (PST) Received: by mail-ej1-x634.google.com with SMTP id qt6so20619848ejb.11 for ; Fri, 11 Mar 2022 09:46:03 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=09Pa4q2b4vX67JHZNNkgoh3+KCy7fCLjcgKDWwRT+M0=; b=DLAOcfyY4RSMcyqKyRAYZJw4xE61oYMJpgI+6BgAA5I8tDn6gOw/xkp0RylEQ+BNG+ X39US6T7op8ybAYMVIohzhKnlFqbUyGI1nmeDVMJu4r0EFookP8UKAVCn0b9I0S4r9EG zRr2tJSNzseIfWg5zifBxJn5RpDYyC6hTMj1hhMmGpTKHVIS5wJ6w834mHZiFJgkcJED zLZiHIeQa0DH1RPSwXLHUOxDA+GJzV6U8W0o4sDbKCiRwu/qAd9UKUu8cIULaI1Hjfp9 o9zV7HGVWbHNVSdUBl4Wu+d+et3xkhKh96P8usb+0UMW6piuQlF2y+YZWK1gs2rGV1Dw CVBw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=09Pa4q2b4vX67JHZNNkgoh3+KCy7fCLjcgKDWwRT+M0=; b=ZhiyCqQax/lKK1qdCYjb83iJvve79GA9BCUgyAxXFX89qM43Rfw6CJHMgLLaHjW6IL cpjJ856mqibazi7lSVz7EmiJE/J8nvywtwveFctnh9ams0OQtXwb42s6NwtXNbOW467Q c3t1vg26HolKFgTZ6+CQJkbBzxsRHCl+xVuZoLVMYwPGeUKn6ST9P402unCpLGsBaZ5P ZqF3zSjtPDwTZ//hcwWEFAlNzJWE8cMHBUnFPnzudipHpkGaHyq3Se6luSmqfJ3bHsOR BWqSQ56EQ9ypzFWJGmb1KJAo595AMr/focNDfwXBc5E6RLs1IKtllw8u84fhAYkJv8PJ KFPw== X-Gm-Message-State: AOAM531alcf+AlEagu1Z1umnJKp3x0mA4zfSBPs7Ilg434rLJUscrOwP iot0be3VMypTDaCS+Mu2wkw= X-Google-Smtp-Source: ABdhPJyHfDu2o5XZ2wRtC0e82YeqMCLO+OmeSyH71sQYMclcdq3RHn7j70Faw1aLHMPlviZ/+d4w2w== X-Received: by 2002:a17:906:9b95:b0:6db:38fc:5ff4 with SMTP id dd21-20020a1709069b9500b006db38fc5ff4mr9143643ejc.114.1647020762408; Fri, 11 Mar 2022 09:46:02 -0800 (PST) Received: from orion.localdomain ([93.99.228.15]) by smtp.gmail.com with ESMTPSA id i21-20020a1709061cd500b006da62ab503csm3159886ejh.157.2022.03.11.09.46.00 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 11 Mar 2022 09:46:01 -0800 (PST) Received: by orion.localdomain (Postfix, from userid 1003) id 814DCA00A8; Fri, 11 Mar 2022 18:46:03 +0100 (CET) From: =?UTF-8?q?Jakub=20Mat=C4=9Bna?= To: linux-mm@kvack.org Cc: patches@lists.linux.dev, linux-kernel@vger.kernel.org, vbabka@suse.cz, mhocko@kernel.org, mgorman@techsingularity.net, willy@infradead.org, liam.howlett@oracle.com, hughd@google.com, kirill@shutemov.name, riel@surriel.com, rostedt@goodmis.org, peterz@infradead.org, =?UTF-8?q?Jakub=20Mat=C4=9Bna?= Subject: [RFC PATCH v2 1/4] [PATCH 1/4] mm: refactor of vma_merge() Date: Fri, 11 Mar 2022 18:45:59 +0100 Message-Id: <20220311174602.288010-2-matenajakub@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20220311174602.288010-1-matenajakub@gmail.com> References: <20220311174602.288010-1-matenajakub@gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Refactor vma_merge() to make it shorter, more understandable and suitable for tracing of successful merges that are made possible by following patches in the series. Main change is the elimination of code duplicity in the case of merge next check. This is done by first doing checks and caching the results before executing the merge itself. Exit paths are also unified. Signed-off-by: Jakub Mat=C4=9Bna Reviewed-by: Vlastimil Babka --- mm/mmap.c | 81 +++++++++++++++++++++++++------------------------------ 1 file changed, 36 insertions(+), 45 deletions(-) diff --git a/mm/mmap.c b/mm/mmap.c index 1e8fdb0b51ed..8d817b11c656 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -1171,7 +1171,9 @@ struct vm_area_struct *vma_merge(struct mm_struct *mm, { pgoff_t pglen =3D (end - addr) >> PAGE_SHIFT; struct vm_area_struct *area, *next; - int err; + int err =3D -1; + bool merge_prev =3D false; + bool merge_next =3D false; =20 /* * We later require that vma->vm_flags =3D=3D vm_flags, @@ -1190,66 +1192,55 @@ struct vm_area_struct *vma_merge(struct mm_struct *= mm, VM_WARN_ON(area && end > area->vm_end); VM_WARN_ON(addr >=3D end); =20 - /* - * Can it merge with the predecessor? - */ + /* Can we merge the predecessor? */ if (prev && prev->vm_end =3D=3D addr && mpol_equal(vma_policy(prev), policy) && can_vma_merge_after(prev, vm_flags, anon_vma, file, pgoff, vm_userfaultfd_ctx, anon_name)) { - /* - * OK, it can. Can we now merge in the successor as well? - */ - if (next && end =3D=3D next->vm_start && - mpol_equal(policy, vma_policy(next)) && - can_vma_merge_before(next, vm_flags, - anon_vma, file, - pgoff+pglen, - vm_userfaultfd_ctx, anon_name) && - is_mergeable_anon_vma(prev->anon_vma, - next->anon_vma, NULL)) { - /* cases 1, 6 */ - err =3D __vma_adjust(prev, prev->vm_start, - next->vm_end, prev->vm_pgoff, NULL, - prev); - } else /* cases 2, 5, 7 */ - err =3D __vma_adjust(prev, prev->vm_start, - end, prev->vm_pgoff, NULL, prev); - if (err) - return NULL; - khugepaged_enter_vma_merge(prev, vm_flags); - return prev; + merge_prev =3D true; + area =3D prev; } - - /* - * Can this new request be merged in front of next? - */ + /* Can we merge the successor? */ if (next && end =3D=3D next->vm_start && mpol_equal(policy, vma_policy(next)) && can_vma_merge_before(next, vm_flags, anon_vma, file, pgoff+pglen, vm_userfaultfd_ctx, anon_name)) { + merge_next =3D true; + } + /* Can we merge both the predecessor and the successor? */ + if (merge_prev && merge_next && + is_mergeable_anon_vma(prev->anon_vma, + next->anon_vma, NULL)) { /* cases 1, 6 */ + err =3D __vma_adjust(prev, prev->vm_start, + next->vm_end, prev->vm_pgoff, NULL, + prev); + } else if (merge_prev) { /* cases 2, 5, 7 */ + err =3D __vma_adjust(prev, prev->vm_start, + end, prev->vm_pgoff, NULL, prev); + } else if (merge_next) { if (prev && addr < prev->vm_end) /* case 4 */ err =3D __vma_adjust(prev, prev->vm_start, - addr, prev->vm_pgoff, NULL, next); - else { /* cases 3, 8 */ + addr, prev->vm_pgoff, NULL, next); + else /* cases 3, 8 */ err =3D __vma_adjust(area, addr, next->vm_end, - next->vm_pgoff - pglen, NULL, next); - /* - * In case 3 area is already equal to next and - * this is a noop, but in case 8 "area" has - * been removed and next was expanded over it. - */ - area =3D next; - } - if (err) - return NULL; - khugepaged_enter_vma_merge(area, vm_flags); - return area; + next->vm_pgoff - pglen, NULL, next); + /* + * In case 3 and 4 area is already equal to next and + * this is a noop, but in case 8 "area" has + * been removed and next was expanded over it. + */ + area =3D next; } =20 - return NULL; + /* + * Cannot merge with predecessor or successor or error in __vma_adjust? + */ + if (err) + return NULL; + khugepaged_enter_vma_merge(area, vm_flags); + return area; } =20 /* --=20 2.34.1 From nobody Tue Jun 23 03:00:31 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 87176C433EF for ; Fri, 11 Mar 2022 17:46:26 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S241941AbiCKRr2 (ORCPT ); Fri, 11 Mar 2022 12:47:28 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:57192 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S242962AbiCKRrJ (ORCPT ); Fri, 11 Mar 2022 12:47:09 -0500 Received: from mail-ej1-x635.google.com (mail-ej1-x635.google.com [IPv6:2a00:1450:4864:20::635]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6CAC813C27B for ; Fri, 11 Mar 2022 09:46:05 -0800 (PST) Received: by mail-ej1-x635.google.com with SMTP id p15so20669850ejc.7 for ; Fri, 11 Mar 2022 09:46:05 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=0lq4ouGfnimqyWNmVOB40NOttTua5Z3ILiH3wkeBRpo=; b=QnqdrUzB4vfjCnIzoOfEpuPmFEt4g9jkQcTZFN44Pc20qzILMoaIngY4z3nAZQHLdP bHULc4fwCM+FmxZ15Ytz64Apd/uLaFUR2LiYth7iKww5zbl1cSw5i6KhiwBrkRPrde5K J0vQS3q7PwLlLjANJyDnm298DU/3V8MC5oMo/3aDpYMjOpCUqZkdC92JQlE/m5OgszZa OFmG5glJIoAEMOW/FpHLOIYs4Si+Fn5/henMZHrXHH365zzYeecfkdtbHPv52tEP/E7J Y1IBP2oQQtljVYN26XK4pMGMnA7vnCVwekXvd+/Xs4cGBrGJkYOb8tCUV8kLKbSX+l9b +MGw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=0lq4ouGfnimqyWNmVOB40NOttTua5Z3ILiH3wkeBRpo=; b=q2buUmy/6sIzrW2hzjIxEKeInX0LuPF6gty2fc78dBrrjbNMCmEdNYKaQ0dXv9qTpp 4+IrpCnw1a2KDEWe/Ms7RHRnKYfUTCq6cw5W1oezQQ+PFBsmY1PtbjTlcYKIB9/6BTfa kXjwqg1cRA/iR00ZGptnXYJyvJuDvatfpcZE5ItZ1jz/5U966BkRbjI8MNabe3WPMsAA lFTl5y1ubId0CtfAFvxPdoQBYzLMp/+NUCW+D7iU5LIMBY2xeCPPIJdaWjEhJECjBdTs nIktuwkNkVGDdMNdFPUg1VoeNyyQQRdSRpmPO1k+OJFujNiWkN3h0SjdqY8puaowsw73 Tl/Q== X-Gm-Message-State: AOAM532/wFfPSjhwgTRA9RoUzQV7VUw/adNeCR9eyhuGXtQmaCJ/HnlJ DmjfXFm63I2Xw3FJU4pwdpA= X-Google-Smtp-Source: ABdhPJyM8VtM3eyZK31TyYDcdc2wS0xsQUrceRwXb8S6txY7mKuo4AJdTWKm7HoKnRMb7p62AGiJeQ== X-Received: by 2002:a17:906:52c7:b0:6ce:a880:50a3 with SMTP id w7-20020a17090652c700b006cea88050a3mr9345194ejn.437.1647020763935; Fri, 11 Mar 2022 09:46:03 -0800 (PST) Received: from orion.localdomain ([93.99.228.15]) by smtp.gmail.com with ESMTPSA id cf17-20020a170906b2d100b006daa59af421sm3232771ejb.149.2022.03.11.09.46.01 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 11 Mar 2022 09:46:01 -0800 (PST) Received: by orion.localdomain (Postfix, from userid 1003) id 855A9A00C0; Fri, 11 Mar 2022 18:46:03 +0100 (CET) From: =?UTF-8?q?Jakub=20Mat=C4=9Bna?= To: linux-mm@kvack.org Cc: patches@lists.linux.dev, linux-kernel@vger.kernel.org, vbabka@suse.cz, mhocko@kernel.org, mgorman@techsingularity.net, willy@infradead.org, liam.howlett@oracle.com, hughd@google.com, kirill@shutemov.name, riel@surriel.com, rostedt@goodmis.org, peterz@infradead.org, =?UTF-8?q?Jakub=20Mat=C4=9Bna?= Subject: [RFC PATCH v2 2/4] [PATCH 2/4] mm: adjust page offset in mremap Date: Fri, 11 Mar 2022 18:46:00 +0100 Message-Id: <20220311174602.288010-3-matenajakub@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20220311174602.288010-1-matenajakub@gmail.com> References: <20220311174602.288010-1-matenajakub@gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Adjust page offset of a VMA when it's moved to a new location by mremap. This is made possible for all VMAs that do not share their anonymous pages with other processes. Previously this was possible only for not yet faulted VMAs. When the page offset does not correspond to the virtual address of the anonymous VMA any merge attempt with another VMA will fail. Signed-off-by: Jakub Mat=C4=9Bna --- mm/mmap.c | 69 ++++++++++++++++++++++++++++++++++++++++++++++++++----- mm/rmap.c | 37 +++++++++++++++++++++++++++++ 2 files changed, 100 insertions(+), 6 deletions(-) diff --git a/mm/mmap.c b/mm/mmap.c index 8d817b11c656..4f9c6ca7ff4e 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -3218,6 +3218,59 @@ int insert_vm_struct(struct mm_struct *mm, struct vm= _area_struct *vma) return 0; } =20 +/** + * update_faulted_pgoff() - Update faulted pages of a vma + * @vma: VMA being moved + * @addr: new virtual address + * @pgoff: pointer to pgoff which is updated + * If the vma and its pages are not shared with another process, update + * the new pgoff and also update index parameter (copy of the pgoff) in + * all faulted pages. + */ +bool update_faulted_pgoff(struct vm_area_struct *vma, unsigned long addr, = pgoff_t *pgoff) +{ + unsigned long pg_iter =3D 0; + unsigned long pg_iters =3D (vma->vm_end - vma->vm_start) >> PAGE_SHIFT; + + /* Check vma is not shared with other processes */ + if (vma->anon_vma->root !=3D vma->anon_vma || !rbt_no_children(vma->anon_= vma)) + return false; + + /* Check all pages are not shared */ + for (; pg_iter < pg_iters; ++pg_iter) { + bool pages_not_shared =3D true; + unsigned long shift =3D pg_iter << PAGE_SHIFT; + struct page *phys_page =3D follow_page(vma, vma->vm_start + shift, FOLL_= GET); + + if (phys_page =3D=3D NULL) + continue; + + /* Check page is not shared with other processes */ + if (page_mapcount(phys_page) > 1) + pages_not_shared =3D false; + put_page(phys_page); + if (!pages_not_shared) + return false; + } + + /* Update index in all pages to this new pgoff */ + pg_iter =3D 0; + *pgoff =3D addr >> PAGE_SHIFT; + + for (; pg_iter < pg_iters; ++pg_iter) { + unsigned long shift =3D pg_iter << PAGE_SHIFT; + struct page *phys_page =3D follow_page(vma, vma->vm_start + shift, FOLL_= GET); + + if (phys_page =3D=3D NULL) + continue; + lock_page(phys_page); + phys_page->index =3D *pgoff + pg_iter; + unlock_page(phys_page); + put_page(phys_page); + } + return true; +} + /* * Copy the vma structure to a new location in the same mm, * prior to moving page table entries, to effect an mremap move. @@ -3231,15 +3284,19 @@ struct vm_area_struct *copy_vma(struct vm_area_stru= ct **vmap, struct mm_struct *mm =3D vma->vm_mm; struct vm_area_struct *new_vma, *prev; struct rb_node **rb_link, *rb_parent; - bool faulted_in_anon_vma =3D true; + bool anon_pgoff_updated =3D false; =20 /* - * If anonymous vma has not yet been faulted, update new pgoff + * Try to update new pgoff for anonymous vma * to match new location, to increase its chance of merging. */ - if (unlikely(vma_is_anonymous(vma) && !vma->anon_vma)) { - pgoff =3D addr >> PAGE_SHIFT; - faulted_in_anon_vma =3D false; + if (unlikely(vma_is_anonymous(vma))) { + if (!vma->anon_vma) { + pgoff =3D addr >> PAGE_SHIFT; + anon_pgoff_updated =3D true; + } else { + anon_pgoff_updated =3D update_faulted_pgoff(vma, addr, &pgoff); + } } =20 if (find_vma_links(mm, addr, addr + len, &prev, &rb_link, &rb_parent)) @@ -3265,7 +3322,7 @@ struct vm_area_struct *copy_vma(struct vm_area_struct= **vmap, * safe. It is only safe to keep the vm_pgoff * linear if there are no pages mapped yet. */ - VM_BUG_ON_VMA(faulted_in_anon_vma, new_vma); + VM_BUG_ON_VMA(!anon_pgoff_updated, new_vma); *vmap =3D vma =3D new_vma; } *need_rmap_locks =3D (new_vma->vm_pgoff <=3D vma->vm_pgoff); diff --git a/mm/rmap.c b/mm/rmap.c index 6a1e8c7f6213..96273d6a9796 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -387,6 +387,43 @@ int anon_vma_fork(struct vm_area_struct *vma, struct v= m_area_struct *pvma) return -ENOMEM; } =20 +/* + * Used by rbt_no_children to check node subtree. + * Check if none of the VMAs connected to the node subtree via + * anon_vma_chain are in child relationship to the given anon_vma. + */ +bool rbst_no_children(struct anon_vma *av, struct rb_node *node) +{ + struct anon_vma_chain *model; + struct anon_vma_chain *avc; + + if (node =3D=3D NULL) /* leaf node */ + return true; + avc =3D container_of(node, typeof(*(model)), rb); + if (avc->vma->anon_vma !=3D av) + /* + * Inequality implies avc belongs + * to a VMA of a child process + */ + return false; + return (rbst_no_children(av, node->rb_left) && + rbst_no_children(av, node->rb_right)); +} + +/* + * Check if none of the VMAs connected to the given + * anon_vma via anon_vma_chain are in child relationship + */ +bool rbt_no_children(struct anon_vma *av) +{ + struct rb_node *root_node; + + if (av =3D=3D NULL || av->degree <=3D 1) /* Higher degree might not neces= sarily imply children */ + return true; + root_node =3D av->rb_root.rb_root.rb_node; + return rbst_no_children(av, root_node); +} + void unlink_anon_vmas(struct vm_area_struct *vma) { struct anon_vma_chain *avc, *next; --=20 2.34.1 From nobody Tue Jun 23 03:00:31 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id C01E2C433F5 for ; Fri, 11 Mar 2022 17:46:11 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S239095AbiCKRrN (ORCPT ); Fri, 11 Mar 2022 12:47:13 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:57158 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1346629AbiCKRrH (ORCPT ); Fri, 11 Mar 2022 12:47:07 -0500 Received: from mail-ej1-x632.google.com (mail-ej1-x632.google.com [IPv6:2a00:1450:4864:20::632]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B4E8814A6DF for ; Fri, 11 Mar 2022 09:46:03 -0800 (PST) Received: by mail-ej1-x632.google.com with SMTP id qt6so20619822ejb.11 for ; Fri, 11 Mar 2022 09:46:03 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=46hR8qn+FvV21nPx7MnpWlG2AiI7do5YUIgYG1Cv7cg=; b=ccCwwfBqjf2cVwLVdbFKTirEaQR5eqblzaW5tzpXkhLDCeMz14RXQg3cgWpYmopJuC UayK8wqQCxTeafeMNClzwFdgnNC6gFoZnpmxSIpsX8NOTF42N9Kwa/ccyN9Q8KieESwo 0DSRUaZkiZor8fAoqjJ09A1UuoRIbbUHaoOcwPEnOWblwGh4dApEJfT2y8PxAEv5KU0m CHsrvO3pm8mtxWqeKXbgIECQwf+2C8cQ/ORW/Omn0etA8qsmKu7cU3Tsuj4C5ZqscAwb OgdxwRzYJWKmoi40IMCcmUzO0NdX6D2WfkvKvVf8vTkjqBDIwhMsfo6NEsJbli2mAF9f j7kg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=46hR8qn+FvV21nPx7MnpWlG2AiI7do5YUIgYG1Cv7cg=; b=hl6oXEZUvpj8MnhZkYEEg5vyK755Fxkz0RYvPRqvcCsfZ5FJozwHCYN6+xqm4KNIBe r25ICz6KfOHJGKL/NMOy4FdLpP//ly0/sIP3eUfyWTQeetpny8Kw6ydS0/hNs1BObkya n1/LubN1E7Yhqvgzvgbo6wxjOOYnBVDEw3Er+A+mLothwquZz/RmzRDaIgI6MYkpo0By HQzg73HZjQ4ckDN1QTRw+cw+9grV7CcqyrBVDLXUJa45Aa+mnsRa7ScwwBVX30xjdOUX tQVkCLeAMAnCLv8NNUX+Ah0pNV3b/E/nmoYI0z/MiciF4m7V1OYCyPZnoxma6JFa0c1P 3sQg== X-Gm-Message-State: AOAM531CPmyQ+h2sdC1Fyee5FndEaresgAmEzuSItKSnalWlCjDx907Z fztnskUz5IqEQ3xAe8/dbxc= X-Google-Smtp-Source: ABdhPJzr5vVq5kXp5nER8koKW+0CLiQKDEP5mHuCuUineelZnECEWO40tLksEo7SeQ+Tfn2AjOxVrw== X-Received: by 2002:a17:906:3ad1:b0:6ce:a880:7745 with SMTP id z17-20020a1709063ad100b006cea8807745mr9552418ejd.46.1647020762124; Fri, 11 Mar 2022 09:46:02 -0800 (PST) Received: from orion.localdomain ([93.99.228.15]) by smtp.gmail.com with ESMTPSA id fs6-20020a170907600600b006da8ec6e4a6sm3185068ejc.26.2022.03.11.09.46.00 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 11 Mar 2022 09:46:01 -0800 (PST) Received: by orion.localdomain (Postfix, from userid 1003) id 88CDAA00C3; Fri, 11 Mar 2022 18:46:03 +0100 (CET) From: =?UTF-8?q?Jakub=20Mat=C4=9Bna?= To: linux-mm@kvack.org Cc: patches@lists.linux.dev, linux-kernel@vger.kernel.org, vbabka@suse.cz, mhocko@kernel.org, mgorman@techsingularity.net, willy@infradead.org, liam.howlett@oracle.com, hughd@google.com, kirill@shutemov.name, riel@surriel.com, rostedt@goodmis.org, peterz@infradead.org, =?UTF-8?q?Jakub=20Mat=C4=9Bna?= Subject: [RFC PATCH v2 3/4] [PATCH 3/4] mm: enable merging of VMAs with different anon_vmas Date: Fri, 11 Mar 2022 18:46:01 +0100 Message-Id: <20220311174602.288010-4-matenajakub@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20220311174602.288010-1-matenajakub@gmail.com> References: <20220311174602.288010-1-matenajakub@gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Enable merging of a VMA even when it is linked to different anon_vma than the one it is being merged to, but only if the VMA in question does not share any page with a parent or child process. Every anonymous page stores a pointer to its anon_vma in the parameter mapping, which is now updated as part of the merge process. Signed-off-by: Jakub Mat=C4=9Bna --- include/linux/rmap.h | 17 ++++++++++++++++- mm/mmap.c | 15 ++++++++++++++- mm/rmap.c | 40 ++++++++++++++++++++++++++++++++++++++++ 3 files changed, 70 insertions(+), 2 deletions(-) diff --git a/include/linux/rmap.h b/include/linux/rmap.h index e704b1a4c06c..c8508a4ebc46 100644 --- a/include/linux/rmap.h +++ b/include/linux/rmap.h @@ -137,10 +137,13 @@ static inline void anon_vma_unlock_read(struct anon_v= ma *anon_vma) */ void anon_vma_init(void); /* create anon_vma_cachep */ int __anon_vma_prepare(struct vm_area_struct *); +void reconnect_pages(struct vm_area_struct *vma, struct vm_area_struct *ne= xt); void unlink_anon_vmas(struct vm_area_struct *); int anon_vma_clone(struct vm_area_struct *, struct vm_area_struct *); int anon_vma_fork(struct vm_area_struct *, struct vm_area_struct *); =20 +bool rbt_no_children(struct anon_vma *av); + static inline int anon_vma_prepare(struct vm_area_struct *vma) { if (likely(vma->anon_vma)) @@ -149,10 +152,22 @@ static inline int anon_vma_prepare(struct vm_area_str= uct *vma) return __anon_vma_prepare(vma); } =20 +/** + * anon_vma_merge() - Merge anon_vmas of the given VMAs + * @vma: VMA being merged to + * @next: VMA being merged + */ static inline void anon_vma_merge(struct vm_area_struct *vma, struct vm_area_struct *next) { - VM_BUG_ON_VMA(vma->anon_vma !=3D next->anon_vma, vma); + struct anon_vma *anon_vma1 =3D vma->anon_vma; + struct anon_vma *anon_vma2 =3D next->anon_vma; + + VM_BUG_ON_VMA(anon_vma1 && anon_vma2 && anon_vma1 !=3D anon_vma2 && + ((anon_vma2 !=3D anon_vma2->root) + || !rbt_no_children(anon_vma2)), vma); + + reconnect_pages(vma, next); unlink_anon_vmas(next); } =20 diff --git a/mm/mmap.c b/mm/mmap.c index 4f9c6ca7ff4e..ccb24862e670 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -1065,7 +1065,20 @@ static inline int is_mergeable_anon_vma(struct anon_= vma *anon_vma1, if ((!anon_vma1 || !anon_vma2) && (!vma || list_is_singular(&vma->anon_vma_chain))) return 1; - return anon_vma1 =3D=3D anon_vma2; + if (anon_vma1 =3D=3D anon_vma2) + return 1; + /* + * Different anon_vma but not shared by several processes + */ + else if ((anon_vma1 && anon_vma2) && + (anon_vma1 =3D=3D anon_vma1->root) + && (rbt_no_children(anon_vma1))) + return 1; + /* + * Different anon_vma and shared -> unmergeable + */ + else + return 0; } =20 /* diff --git a/mm/rmap.c b/mm/rmap.c index 96273d6a9796..b296e1e1aec3 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -387,6 +387,46 @@ int anon_vma_fork(struct vm_area_struct *vma, struct v= m_area_struct *pvma) return -ENOMEM; } =20 +/** + * reconnect_pages() - Reconnect physical pages from old to vma + * @vma: VMA to newly contain all physical pages of old + * @old: old VMA being merged to vma + */ +void reconnect_pages(struct vm_area_struct *vma, struct vm_area_struct *ol= d) +{ + struct anon_vma *anon_vma1 =3D vma->anon_vma; + struct anon_vma *anon_vma2 =3D old->anon_vma; + unsigned long pg_iter; + int pg_iters; + + if (anon_vma1 =3D=3D anon_vma2 || anon_vma1 =3D=3D NULL || anon_vma2 =3D= =3D NULL) + return; /* Nothing to do */ + + /* Modify page->mapping for all pages in old */ + pg_iter =3D 0; + pg_iters =3D (old->vm_end - old->vm_start) >> PAGE_SHIFT; + + for (; pg_iter < pg_iters; ++pg_iter) { + /* Get the physical page */ + unsigned long shift =3D pg_iter << PAGE_SHIFT; + struct page *phys_page =3D follow_page(old, old->vm_start + shift, FOLL_= GET); + struct anon_vma *page_anon_vma; + + /* Do some checks and lock the page */ + if (phys_page =3D=3D NULL) + continue; /* Virtual memory page is not mapped */ + lock_page(phys_page); + page_anon_vma =3D page_get_anon_vma(phys_page); + if (page_anon_vma !=3D NULL) { /* NULL in case of ZERO_PAGE */ + VM_BUG_ON_VMA(page_anon_vma !=3D old->anon_vma, old); + /* Update physical page's mapping */ + page_move_anon_rmap(phys_page, vma); + } + unlock_page(phys_page); + put_page(phys_page); + } +} + /* * Used by rbt_no_children to check node subtree. * Check if none of the VMAs connected to the node subtree via --=20 2.34.1 From nobody Tue Jun 23 03:00:31 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7E019C433EF for ; Fri, 11 Mar 2022 17:46:08 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237423AbiCKRrK (ORCPT ); Fri, 11 Mar 2022 12:47:10 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:57156 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1348689AbiCKRrH (ORCPT ); Fri, 11 Mar 2022 12:47:07 -0500 Received: from mail-ej1-x635.google.com (mail-ej1-x635.google.com [IPv6:2a00:1450:4864:20::635]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8A2E513A1CC for ; Fri, 11 Mar 2022 09:46:03 -0800 (PST) Received: by mail-ej1-x635.google.com with SMTP id d10so20635243eje.10 for ; Fri, 11 Mar 2022 09:46:03 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=16FTiziTuHY7luYyqhUgZZQe9XBDNooPz6CdRNgABNY=; b=oxr+bbXplN7JLlsYtHsTMq87tB9qjjwJovy4Qmphj7sIBHnv5iE72yNg7suTzlkG7R oINry5tchEi9fK7FZ7LUmaPJ+jCwkyU4D/uYP1hDIDuAHb5l5rVXjHcij7yoXtKuJ2Zp enoa7/O0Z/su2Af9pCcQsKXgeplEeEeWn/P99XC2ffJEz02lS85YB1M8YGyi/hpMUmf1 ChQ9zzI46OgPcsPId44t45Xpz1NWniqZIQo00/15hexDGgVRQkUiwrC0I30gCFe2bue3 CtMBKIRZHu7FIrQ/cg6ZOxrbEKmIPRnJXzW/DvCQWv2G24+B32J7iO8vAxujZM6w1EVv xiyg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=16FTiziTuHY7luYyqhUgZZQe9XBDNooPz6CdRNgABNY=; b=t54SVLhMP6VldT7Dul8nPdCjC9sRKbeeDxI9C6ugqoT91eod3R2A9EaYgIm9oPzTUz SnIX6mMrmsC0n8Xr21PXzfWIRRPZD//PP3Qhvv5qmUITKpxCBFlEGC5qVNG7LZxlIz2P XcTDiR+pNPbTPqhWLR7iU9fxB9Q3LdwP+pJg3h94FXd6Lgwcz+bvIdQi85fqIlkI/Mqt IYrScLxv2Mj5qfP6iGQKaartVkOGIe5ddMxtY7NmvtduYQE70y+OPaepcGeU7KIAxfPV yHmsJe2JczgFMzRyrwgw/pDifB2qim+iBCgQRDnUYhvjDvKUIdwtIJ1w6LQuYHDU7HPw 48fw== X-Gm-Message-State: AOAM530TJa+2o3eTAs0U6JxRV3SS+qVHcUWQdF68ZDveErFS2V2+BSx2 8Li7MxxfSW5aPL0jwnBs+mY= X-Google-Smtp-Source: ABdhPJwrEvlJ1jGKDiP915MpwaiTas/VtSW176/53uKYGRh7P+2bKjGc5/e1w+Yl1fYaK6RmZ7HAyQ== X-Received: by 2002:a17:907:8a0a:b0:6d8:85a6:4d42 with SMTP id sc10-20020a1709078a0a00b006d885a64d42mr9993383ejc.138.1647020761822; Fri, 11 Mar 2022 09:46:01 -0800 (PST) Received: from orion.localdomain ([93.99.228.15]) by smtp.gmail.com with ESMTPSA id k23-20020a1709062a5700b006ccd8fdc300sm3166909eje.180.2022.03.11.09.46.00 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 11 Mar 2022 09:46:01 -0800 (PST) Received: by orion.localdomain (Postfix, from userid 1003) id 8C736A0482; Fri, 11 Mar 2022 18:46:03 +0100 (CET) From: =?UTF-8?q?Jakub=20Mat=C4=9Bna?= To: linux-mm@kvack.org Cc: patches@lists.linux.dev, linux-kernel@vger.kernel.org, vbabka@suse.cz, mhocko@kernel.org, mgorman@techsingularity.net, willy@infradead.org, liam.howlett@oracle.com, hughd@google.com, kirill@shutemov.name, riel@surriel.com, rostedt@goodmis.org, peterz@infradead.org, =?UTF-8?q?Jakub=20Mat=C4=9Bna?= Subject: [RFC PATCH v2 4/4] [PATCH 4/4] mm: add tracing for VMA merges Date: Fri, 11 Mar 2022 18:46:02 +0100 Message-Id: <20220311174602.288010-5-matenajakub@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20220311174602.288010-1-matenajakub@gmail.com> References: <20220311174602.288010-1-matenajakub@gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Adds trace support for vma_merge to measure successful and unsuccessful merges of two VMAs with distinct anon_vmas and also trace support for merges made possible by update of page offset made possible by a previous patch in this series. Signed-off-by: Jakub Mat=C4=9Bna --- include/trace/events/mmap.h | 83 +++++++++++++++++++++++++++++++++++++ mm/internal.h | 12 ++++++ mm/mmap.c | 69 ++++++++++++++++-------------- 3 files changed, 133 insertions(+), 31 deletions(-) diff --git a/include/trace/events/mmap.h b/include/trace/events/mmap.h index 4661f7ba07c0..bad7abe4899c 100644 --- a/include/trace/events/mmap.h +++ b/include/trace/events/mmap.h @@ -6,6 +6,27 @@ #define _TRACE_MMAP_H =20 #include +#include <../mm/internal.h> + +#define AV_MERGE_TYPES \ + EM(MERGE_FAILED) \ + EM(AV_MERGE_FAILED) \ + EM(AV_MERGE_NULL) \ + EM(AV_MERGE_SAME) \ + EMe(AV_MERGE_DIFFERENT) + +#undef EM +#undef EMe +#define EM(a) TRACE_DEFINE_ENUM(a); +#define EMe(a) TRACE_DEFINE_ENUM(a); + +AV_MERGE_TYPES + +#undef EM +#undef EMe + +#define EM(a) { a, #a }, +#define EMe(a) { a, #a } =20 TRACE_EVENT(vm_unmapped_area, =20 @@ -42,6 +63,68 @@ TRACE_EVENT(vm_unmapped_area, __entry->low_limit, __entry->high_limit, __entry->align_mask, __entry->align_offset) ); + +TRACE_EVENT(vm_av_merge, + + TP_PROTO(int merged, enum vma_merge_res merge_prev, + enum vma_merge_res merge_next, enum vma_merge_res merge_both), + + TP_ARGS(merged, merge_prev, merge_next, merge_both), + + TP_STRUCT__entry( + __field(int, merged) + __field(enum vma_merge_res, predecessor_different_av) + __field(enum vma_merge_res, successor_different_av) + __field(enum vma_merge_res, predecessor_with_successor_different_av) + __field(int, same_count) + __field(int, diff_count) + __field(int, failed_count) + ), + + TP_fast_assign( + __entry->merged =3D merged =3D=3D 0; + __entry->predecessor_different_av =3D merge_prev; + __entry->successor_different_av =3D merge_next; + __entry->predecessor_with_successor_different_av =3D merge_both; + __entry->same_count =3D (merge_prev =3D=3D AV_MERGE_SAME) + + (merge_next =3D=3D AV_MERGE_SAME) + + (merge_both =3D=3D AV_MERGE_SAME); + __entry->diff_count =3D (merge_prev =3D=3D AV_MERGE_DIFFERENT) + + (merge_next =3D=3D AV_MERGE_DIFFERENT) + + (merge_both =3D=3D AV_MERGE_DIFFERENT); + __entry->failed_count =3D (merge_prev =3D=3D AV_MERGE_FAILED) + + (merge_next =3D=3D AV_MERGE_FAILED) + + (merge_both =3D=3D AV_MERGE_FAILED); + ), + + TP_printk("merged=3D%d predecessor=3D%s successor=3D%s predecessor_with_s= uccessor=3D%s same_count=3D%d diff_count=3D%d failed_count=3D%d", + __entry->merged, + __print_symbolic(__entry->predecessor_different_av, AV_MERGE_TYPES), + __print_symbolic(__entry->successor_different_av, AV_MERGE_TYPES), + __print_symbolic(__entry->predecessor_with_successor_different_av, AV_ME= RGE_TYPES), + __entry->same_count, __entry->diff_count, __entry->failed_count) + +); + +TRACE_EVENT(vm_pgoff_merge, + + TP_PROTO(struct vm_area_struct *vma, bool anon_pgoff_updated), + + TP_ARGS(vma, anon_pgoff_updated), + + TP_STRUCT__entry( + __field(bool, faulted) + __field(bool, updated) + ), + + TP_fast_assign( + __entry->faulted =3D vma->anon_vma; + __entry->updated =3D anon_pgoff_updated; + ), + + TP_printk("faulted=3D%d updated=3D%d\n", + __entry->faulted, __entry->updated) +); #endif =20 /* This part must be outside protection */ diff --git a/mm/internal.h b/mm/internal.h index d80300392a19..860169612192 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -34,6 +34,18 @@ struct folio_batch; /* Do not use these with a slab allocator */ #define GFP_SLAB_BUG_MASK (__GFP_DMA32|__GFP_HIGHMEM|~__GFP_BITS_MASK) =20 +/* + * Following values indicate reason for merge success or failure. + */ +enum vma_merge_res { + MERGE_FAILED, + AV_MERGE_FAILED, + AV_MERGE_NULL, + MERGE_OK =3D AV_MERGE_NULL, + AV_MERGE_SAME, + AV_MERGE_DIFFERENT, +}; + void page_writeback_init(void); =20 static inline void *folio_raw_mapping(struct folio *folio) diff --git a/mm/mmap.c b/mm/mmap.c index ccb24862e670..663f8ec46f2c 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -1064,21 +1064,21 @@ static inline int is_mergeable_anon_vma(struct anon= _vma *anon_vma1, */ if ((!anon_vma1 || !anon_vma2) && (!vma || list_is_singular(&vma->anon_vma_chain))) - return 1; + return AV_MERGE_NULL; if (anon_vma1 =3D=3D anon_vma2) - return 1; + return AV_MERGE_SAME; /* * Different anon_vma but not shared by several processes */ else if ((anon_vma1 && anon_vma2) && (anon_vma1 =3D=3D anon_vma1->root) && (rbt_no_children(anon_vma1))) - return 1; + return AV_MERGE_DIFFERENT; /* * Different anon_vma and shared -> unmergeable */ else - return 0; + return AV_MERGE_FAILED; } =20 /* @@ -1099,12 +1099,10 @@ can_vma_merge_before(struct vm_area_struct *vma, un= signed long vm_flags, struct vm_userfaultfd_ctx vm_userfaultfd_ctx, const char *anon_name) { - if (is_mergeable_vma(vma, file, vm_flags, vm_userfaultfd_ctx, anon_name) = && - is_mergeable_anon_vma(anon_vma, vma->anon_vma, vma)) { + if (is_mergeable_vma(vma, file, vm_flags, vm_userfaultfd_ctx, anon_name)) if (vma->vm_pgoff =3D=3D vm_pgoff) - return 1; - } - return 0; + return is_mergeable_anon_vma(anon_vma, vma->anon_vma, vma); + return MERGE_FAILED; } =20 /* @@ -1121,14 +1119,13 @@ can_vma_merge_after(struct vm_area_struct *vma, uns= igned long vm_flags, struct vm_userfaultfd_ctx vm_userfaultfd_ctx, const char *anon_name) { - if (is_mergeable_vma(vma, file, vm_flags, vm_userfaultfd_ctx, anon_name) = && - is_mergeable_anon_vma(anon_vma, vma->anon_vma, vma)) { + if (is_mergeable_vma(vma, file, vm_flags, vm_userfaultfd_ctx, anon_name))= { pgoff_t vm_pglen; vm_pglen =3D vma_pages(vma); if (vma->vm_pgoff + vm_pglen =3D=3D vm_pgoff) - return 1; + return is_mergeable_anon_vma(anon_vma, vma->anon_vma, vma); } - return 0; + return MERGE_FAILED; } =20 /* @@ -1185,8 +1182,14 @@ struct vm_area_struct *vma_merge(struct mm_struct *m= m, pgoff_t pglen =3D (end - addr) >> PAGE_SHIFT; struct vm_area_struct *area, *next; int err =3D -1; - bool merge_prev =3D false; - bool merge_next =3D false; + /* + * Following three variables are used to store values + * indicating wheather this VMA and its anon_vma can + * be merged and also the type of failure or success. + */ + enum vma_merge_res merge_prev =3D MERGE_FAILED; + enum vma_merge_res merge_both =3D MERGE_FAILED; + enum vma_merge_res merge_next =3D MERGE_FAILED; =20 /* * We later require that vma->vm_flags =3D=3D vm_flags, @@ -1207,32 +1210,34 @@ struct vm_area_struct *vma_merge(struct mm_struct *= mm, =20 /* Can we merge the predecessor? */ if (prev && prev->vm_end =3D=3D addr && - mpol_equal(vma_policy(prev), policy) && - can_vma_merge_after(prev, vm_flags, + mpol_equal(vma_policy(prev), policy)) { + merge_prev =3D can_vma_merge_after(prev, vm_flags, anon_vma, file, pgoff, - vm_userfaultfd_ctx, anon_name)) { - merge_prev =3D true; - area =3D prev; + vm_userfaultfd_ctx, anon_name); } + /* Can we merge the successor? */ if (next && end =3D=3D next->vm_start && - mpol_equal(policy, vma_policy(next)) && - can_vma_merge_before(next, vm_flags, - anon_vma, file, pgoff+pglen, - vm_userfaultfd_ctx, anon_name)) { - merge_next =3D true; + mpol_equal(policy, vma_policy(next))) { + merge_next =3D can_vma_merge_before(next, vm_flags, + anon_vma, file, pgoff+pglen, + vm_userfaultfd_ctx, anon_name); } + /* Can we merge both the predecessor and the successor? */ - if (merge_prev && merge_next && - is_mergeable_anon_vma(prev->anon_vma, - next->anon_vma, NULL)) { /* cases 1, 6 */ + if (merge_prev >=3D MERGE_OK && merge_next >=3D MERGE_OK) + merge_both =3D is_mergeable_anon_vma(prev->anon_vma, next->anon_vma, NUL= L); + + if (merge_both >=3D MERGE_OK) { /* cases 1, 6 */ err =3D __vma_adjust(prev, prev->vm_start, next->vm_end, prev->vm_pgoff, NULL, prev); - } else if (merge_prev) { /* cases 2, 5, 7 */ + area =3D prev; + } else if (merge_prev >=3D MERGE_OK) { /* cases 2, 5, 7 */ err =3D __vma_adjust(prev, prev->vm_start, end, prev->vm_pgoff, NULL, prev); - } else if (merge_next) { + area =3D prev; + } else if (merge_next >=3D MERGE_OK) { if (prev && addr < prev->vm_end) /* case 4 */ err =3D __vma_adjust(prev, prev->vm_start, addr, prev->vm_pgoff, NULL, next); @@ -1246,7 +1251,7 @@ struct vm_area_struct *vma_merge(struct mm_struct *mm, */ area =3D next; } - + trace_vm_av_merge(err, merge_prev, merge_next, merge_both); /* * Cannot merge with predecessor or successor or error in __vma_adjust? */ @@ -3321,6 +3326,8 @@ struct vm_area_struct *copy_vma(struct vm_area_struct= **vmap, /* * Source vma may have been merged into new_vma */ + trace_vm_pgoff_merge(vma, anon_pgoff_updated); + if (unlikely(vma_start >=3D new_vma->vm_start && vma_start < new_vma->vm_end)) { /* --=20 2.34.1