From nobody Tue Apr 28 09:06:12 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7E9B9C433EF for ; Fri, 3 Jun 2022 14:57:09 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S245258AbiFCO5I (ORCPT ); Fri, 3 Jun 2022 10:57:08 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46202 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S245252AbiFCO4x (ORCPT ); Fri, 3 Jun 2022 10:56:53 -0400 Received: from mail-ej1-x631.google.com (mail-ej1-x631.google.com [IPv6:2a00:1450:4864:20::631]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C276AAE78 for ; Fri, 3 Jun 2022 07:56:50 -0700 (PDT) Received: by mail-ej1-x631.google.com with SMTP id s6so4864125eja.0 for ; Fri, 03 Jun 2022 07:56:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=xDGUdvx1pZTgyR7PJdBJDYNZdw+7CJ97LAoW275dhVs=; b=RSfJURm0QkP/9Vq/fBoprT4aTS2dTIKLUzM/4tcnKiBXK0q++Pvc6+HW05p0AqIUYG BY0IZq8t7He8sycbNSRuJgs0AE4a0HsC552jBOp6reIGUO0fkfJ56KuGWF+D1Pme6BOP OKhRndSEHmQaxthqzxydfKYJv9CQXEjn7sm70oJNM8qD/J/8nYgw4Qy6y/DpkCnqkloS 4cxKR8gPKRa/0Km47eUPxu3aWehYH6JGgXxX9MHVwla+IGMRIzzUzW2qftJRSjOY9uY5 qKs6sLNbJCWBbmWAxINU/Qy2nP8+oVKt2GIBxkdUv44hb/FEtTGkohnG2sYsEm2Y16Y+ sWhw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=xDGUdvx1pZTgyR7PJdBJDYNZdw+7CJ97LAoW275dhVs=; b=PWGz34P15I9J1KQ9ykFuDqz2Hj/4WkrWZ/enVURc8Q18R0uWPqH/0WdvMqxRW/XWSV xR7SfmG5ZGpcCfit8vNtMSyYVy0A2NPWMSTReVlFkRuUBsR00dNnV+eCNsem/dWy17h3 LbBKVDU4ZXfNYYb36G2OWL6GsCuF7N5Aq3JPnw5keWWTmCpowkF2wOSOZDTBmm+oS/cZ IFSgWxzw/LB7+atqKZVkVCabjOIvbWNKwh9/fjFEz1t673qojony6jYOa2YvQCdZoJ+f Ef94ZjeQEW4VFCF2GC9+GoADy7xqktTfy2XzlBWxIiKbxfuhydcMI6CMQvtwGR6KDNl8 n07Q== X-Gm-Message-State: AOAM531tCgaVfHGL7ufDpPwXA0S8Rw473LWKFCM3BSX4Om6/LOpK2Q2v FuEjU9yo3Wv7eyDwlVRP/3Y= X-Google-Smtp-Source: ABdhPJx0+/KUkuEWOOtQ4+NUTALnnZ3ui7c8UbzzYK8djtmi4+WlxYaLTp0wVIUGCmcnTLAXkWvg0Q== X-Received: by 2002:a17:907:3e18:b0:6fe:cf44:8692 with SMTP id hp24-20020a1709073e1800b006fecf448692mr9050521ejc.601.1654268209258; Fri, 03 Jun 2022 07:56:49 -0700 (PDT) Received: from orion.localdomain ([93.99.228.15]) by smtp.gmail.com with ESMTPSA id k9-20020a1709061c0900b006fec27575f1sm2890627ejg.123.2022.06.03.07.56.45 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 03 Jun 2022 07:56:46 -0700 (PDT) Received: by orion.localdomain (Postfix, from userid 1003) id ABBE8A0013; Fri, 3 Jun 2022 16:57:20 +0200 (CEST) From: =?UTF-8?q?Jakub=20Mat=C4=9Bna?= To: akpm@linux-foundation.org Cc: linux-mm@kvack.org, patches@lists.linux.dev, linux-kernel@vger.kernel.org, vbabka@suse.cz, mhocko@kernel.org, mgorman@techsingularity.net, willy@infradead.org, liam.howlett@oracle.com, hughd@google.com, kirill@shutemov.name, riel@surriel.com, rostedt@goodmis.org, peterz@infradead.org, =?UTF-8?q?Jakub=20Mat=C4=9Bna?= Subject: [PATCH v4 1/2] [PATCH 1/2] mm: refactor of vma_merge() Date: Fri, 3 Jun 2022 16:57:18 +0200 Message-Id: <20220603145719.1012094-2-matenajakub@gmail.com> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20220603145719.1012094-1-matenajakub@gmail.com> References: <20220603145719.1012094-1-matenajakub@gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Refactor vma_merge() to make it shorter and more understandable. Main change is the elimination of code duplicity in the case of merge next check. This is done by first doing checks and caching the results before executing the merge itself. The variable 'area' is divided into 'mid' and 'res' as previously it was used for two purposes, as the middle VMA between prev and next and also as the result of the merge itself. Exit paths are also unified. Signed-off-by: Jakub Mat=C4=9Bna Reviewed-by: Vlastimil Babka --- mm/mmap.c | 87 +++++++++++++++++++++++-------------------------------- 1 file changed, 37 insertions(+), 50 deletions(-) diff --git a/mm/mmap.c b/mm/mmap.c index 313b57d55a63..91100fdc400a 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -1170,8 +1170,10 @@ struct vm_area_struct *vma_merge(struct mm_struct *m= m, struct anon_vma_name *anon_name) { pgoff_t pglen =3D (end - addr) >> PAGE_SHIFT; - struct vm_area_struct *area, *next; - int err; + struct vm_area_struct *mid, *next, *res; + int err =3D -1; + bool merge_prev =3D false; + bool merge_next =3D false; =20 /* * We later require that vma->vm_flags =3D=3D vm_flags, @@ -1181,75 +1183,60 @@ struct vm_area_struct *vma_merge(struct mm_struct *= mm, return NULL; =20 next =3D find_vma(mm, prev ? prev->vm_end : 0); - area =3D next; - if (area && area->vm_end =3D=3D end) /* cases 6, 7, 8 */ + mid =3D next; + if (next && next->vm_end =3D=3D end) /* cases 6, 7, 8 */ next =3D find_vma(mm, next->vm_end); =20 /* verify some invariant that must be enforced by the caller */ VM_WARN_ON(prev && addr <=3D prev->vm_start); - VM_WARN_ON(area && end > area->vm_end); + VM_WARN_ON(mid && end > mid->vm_end); VM_WARN_ON(addr >=3D end); =20 - /* - * Can it merge with the predecessor? - */ + /* Can we merge the predecessor? */ if (prev && prev->vm_end =3D=3D addr && mpol_equal(vma_policy(prev), policy) && can_vma_merge_after(prev, vm_flags, anon_vma, file, pgoff, vm_userfaultfd_ctx, anon_name)) { - /* - * OK, it can. Can we now merge in the successor as well? - */ - if (next && end =3D=3D next->vm_start && - mpol_equal(policy, vma_policy(next)) && - can_vma_merge_before(next, vm_flags, - anon_vma, file, - pgoff+pglen, - vm_userfaultfd_ctx, anon_name) && - is_mergeable_anon_vma(prev->anon_vma, - next->anon_vma, NULL)) { - /* cases 1, 6 */ - err =3D __vma_adjust(prev, prev->vm_start, - next->vm_end, prev->vm_pgoff, NULL, - prev); - } else /* cases 2, 5, 7 */ - err =3D __vma_adjust(prev, prev->vm_start, - end, prev->vm_pgoff, NULL, prev); - if (err) - return NULL; - khugepaged_enter_vma(prev, vm_flags); - return prev; + merge_prev =3D true; } - - /* - * Can this new request be merged in front of next? - */ + /* Can we merge the successor? */ if (next && end =3D=3D next->vm_start && mpol_equal(policy, vma_policy(next)) && can_vma_merge_before(next, vm_flags, anon_vma, file, pgoff+pglen, vm_userfaultfd_ctx, anon_name)) { + merge_next =3D true; + } + /* Can we merge both the predecessor and the successor? */ + if (merge_prev && merge_next && + is_mergeable_anon_vma(prev->anon_vma, + next->anon_vma, NULL)) { /* cases 1, 6 */ + err =3D __vma_adjust(prev, prev->vm_start, + next->vm_end, prev->vm_pgoff, NULL, + prev); + res =3D prev; + } else if (merge_prev) { /* cases 2, 5, 7 */ + err =3D __vma_adjust(prev, prev->vm_start, + end, prev->vm_pgoff, NULL, prev); + res =3D prev; + } else if (merge_next) { if (prev && addr < prev->vm_end) /* case 4 */ err =3D __vma_adjust(prev, prev->vm_start, - addr, prev->vm_pgoff, NULL, next); - else { /* cases 3, 8 */ - err =3D __vma_adjust(area, addr, next->vm_end, - next->vm_pgoff - pglen, NULL, next); - /* - * In case 3 area is already equal to next and - * this is a noop, but in case 8 "area" has - * been removed and next was expanded over it. - */ - area =3D next; - } - if (err) - return NULL; - khugepaged_enter_vma(area, vm_flags); - return area; + addr, prev->vm_pgoff, NULL, next); + else /* cases 3, 8 */ + err =3D __vma_adjust(mid, addr, next->vm_end, + next->vm_pgoff - pglen, NULL, next); + res =3D next; } =20 - return NULL; + /* + * Cannot merge with predecessor or successor or error in __vma_adjust? + */ + if (err) + return NULL; + khugepaged_enter_vma(res, vm_flags); + return res; } =20 /* --=20 2.35.1 From nobody Tue Apr 28 09:06:12 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 41779C43334 for ; Fri, 3 Jun 2022 14:57:15 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S245279AbiFCO5L (ORCPT ); Fri, 3 Jun 2022 10:57:11 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46358 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S245260AbiFCO44 (ORCPT ); Fri, 3 Jun 2022 10:56:56 -0400 Received: from mail-ed1-x534.google.com (mail-ed1-x534.google.com [IPv6:2a00:1450:4864:20::534]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4E361140DD for ; Fri, 3 Jun 2022 07:56:52 -0700 (PDT) Received: by mail-ed1-x534.google.com with SMTP id c2so10490307edf.5 for ; Fri, 03 Jun 2022 07:56:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=plkRHxZRk9CMIn7YOznv+vuUGRy66YRIHBjRF967Ovg=; b=OFHroXywrVa28TjmeuM2CMND/lkHNeB16Lt8Jq4E+glKJ+M7g6kanfCClZ/vFLbsov r8u80hXBHLksypvSYoPqwzFpAvd//6QXleUWoJK13uqoAIkBMosyf14qgs97WAoRRFMS 2KC6Aydv3OfnpRsyHYsxy5s2+idac3Jr5ftWPg1lOOMhX2cY9q1vxiYomkU9wyA37TPs y092tj0B+WAidkzySllTdAu4D4oFoTQV5zFbZgVofWXuBfipvdQrWKcYFYeiqnrtaMUH 4A/YjJ4MwVGW9fEPcYOu8uvC7WZOVe5qBOe1jEhd3lz2/ZtjlZn/oDr2bvMun5l0roiS 2EfA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=plkRHxZRk9CMIn7YOznv+vuUGRy66YRIHBjRF967Ovg=; b=MCDJSDFkVXxaMvmyMMb90wrTOGCYkeK47Wq0wVrwfD2aj2MVi3AFkDoECVbk5BH/lr 0umAK+VdW791MIPdPaRd1tGeL7CDcK80PoFiqFy0mDd2o2J69uxORxBXq9kyaV7U2mOW DJa+EP4r9K+ZS8qaqMvr5DwyMyd5TQB/6RSOCmL6XWH7AfsL0NhGN1wn4ubDXD7PzkmL 2pDz6hx0cRkXzPwsQP/cnMPBnPFmsXYBZmOkj1oMwQiMONQrn4QfpE8xaw8xv00L5pFu DyaktMX1AyYfgspz8iBdMCWi5v7J6XTf1YHjjwRqWp+mAkBHjuheKK3Q8boVPKWROAb5 tcvw== X-Gm-Message-State: AOAM5328Ri5rrARgz249O2DA8ya7UfFcKBA0Sy86bJoj+/Lyw8k4OpYX mFOrxls+f2fA7Rxl5e1anvM= X-Google-Smtp-Source: ABdhPJx1tqTWq+bg3K48QBR4njKSxXrtNNk36G7Ebems4gSc3jniGgD8tYlpAFClGJES+20LMJVByg== X-Received: by 2002:a05:6402:b45:b0:42d:bf43:8db1 with SMTP id bx5-20020a0564020b4500b0042dbf438db1mr11221191edb.100.1654268210815; Fri, 03 Jun 2022 07:56:50 -0700 (PDT) Received: from orion.localdomain ([93.99.228.15]) by smtp.gmail.com with ESMTPSA id e2-20020a1709062d4200b006fec2097d53sm2917509eji.118.2022.06.03.07.56.45 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 03 Jun 2022 07:56:46 -0700 (PDT) Received: by orion.localdomain (Postfix, from userid 1003) id AEEECA0366; Fri, 3 Jun 2022 16:57:20 +0200 (CEST) From: =?UTF-8?q?Jakub=20Mat=C4=9Bna?= To: akpm@linux-foundation.org Cc: linux-mm@kvack.org, patches@lists.linux.dev, linux-kernel@vger.kernel.org, vbabka@suse.cz, mhocko@kernel.org, mgorman@techsingularity.net, willy@infradead.org, liam.howlett@oracle.com, hughd@google.com, kirill@shutemov.name, riel@surriel.com, rostedt@goodmis.org, peterz@infradead.org, =?UTF-8?q?Jakub=20Mat=C4=9Bna?= Subject: [PATCH v4 2/2] [PATCH 2/2] mm: add merging after mremap resize Date: Fri, 3 Jun 2022 16:57:19 +0200 Message-Id: <20220603145719.1012094-3-matenajakub@gmail.com> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20220603145719.1012094-1-matenajakub@gmail.com> References: <20220603145719.1012094-1-matenajakub@gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org When mremap call results in expansion, it might be possible to merge the VMA with the next VMA which might become adjacent. This patch adds vma_merge call after the expansion is done to try and merge. Signed-off-by: Jakub Mat=C4=9Bna Reviewed-by: Vlastimil Babka --- mm/mremap.c | 19 +++++++++- tools/testing/selftests/vm/mremap_test.c | 47 +++++++++++++++++++++++- 2 files changed, 63 insertions(+), 3 deletions(-) diff --git a/mm/mremap.c b/mm/mremap.c index 0b93fac76851..66970dcd636a 100644 --- a/mm/mremap.c +++ b/mm/mremap.c @@ -9,6 +9,7 @@ */ =20 #include +#include #include #include #include @@ -23,6 +24,7 @@ #include #include #include +#include =20 #include #include @@ -1014,6 +1016,9 @@ SYSCALL_DEFINE5(mremap, unsigned long, addr, unsigned= long, old_len, /* can we just expand the current mapping? */ if (vma_expandable(vma, new_len - old_len)) { long pages =3D (new_len - old_len) >> PAGE_SHIFT; + unsigned long extension_start =3D addr + old_len; + unsigned long extension_end =3D addr + new_len; + pgoff_t extension_pgoff =3D vma->vm_pgoff + (old_len >> PAGE_SHIFT); =20 if (vma->vm_flags & VM_ACCOUNT) { if (security_vm_enough_memory_mm(mm, pages)) { @@ -1022,8 +1027,18 @@ SYSCALL_DEFINE5(mremap, unsigned long, addr, unsigne= d long, old_len, } } =20 - if (vma_adjust(vma, vma->vm_start, addr + new_len, - vma->vm_pgoff, NULL)) { + /* + * Function vma_merge() is called on the extension we are adding to + * the already existing vma, vma_merge() will merge this extension with + * the already existing vma (expand operation itself) and possibly also + * with the next vma if it becomes adjacent to the expanded vma and + * otherwise compatible. + */ + vma =3D vma_merge(mm, vma, extension_start, extension_end, + vma->vm_flags, vma->anon_vma, vma->vm_file, + extension_pgoff, vma_policy(vma), + vma->vm_userfaultfd_ctx, anon_vma_name(vma)); + if (!vma) { vm_unacct_memory(pages); ret =3D -ENOMEM; goto out; diff --git a/tools/testing/selftests/vm/mremap_test.c b/tools/testing/selft= ests/vm/mremap_test.c index db0270127aeb..0865a6cb5bdb 100644 --- a/tools/testing/selftests/vm/mremap_test.c +++ b/tools/testing/selftests/vm/mremap_test.c @@ -118,6 +118,48 @@ static unsigned long long get_mmap_min_addr(void) return addr; } =20 +/* + * This test validates that merge is called when expanding a mapping. + * Mapping containing three pages is created, middle page is unmapped + * and then the mapping containing the first page is expanded so that + * it fills the created hole. The two parts should merge creating + * single mapping with three pages. + */ +static void mremap_expand_merge(unsigned long page_size) +{ + char *test_name =3D "mremap expand merge"; + FILE *fp; + char *line =3D NULL; + size_t len =3D 0; + bool success =3D false; + + char *start =3D mmap(NULL, 3 * page_size, PROT_READ | PROT_WRITE, MAP_PRI= VATE | MAP_ANONYMOUS, -1, 0); + munmap(start + page_size, page_size); + mremap(start, page_size, 2 * page_size, 0); + + fp =3D fopen("/proc/self/maps", "r"); + if (fp =3D=3D NULL) { + ksft_test_result_fail("%s\n", test_name); + return; + } + + while(getline(&line, &len, fp) !=3D -1) { + char *first =3D strtok(line,"- "); + void *first_val =3D (void *) strtol(first, NULL, 16); + char *second =3D strtok(NULL,"- "); + void *second_val =3D (void *) strtol(second, NULL, 16); + if (first_val =3D=3D start && second_val =3D=3D start + 3 * page_size) { + success =3D true; + break; + } + } + if (success) + ksft_test_result_pass("%s\n", test_name); + else + ksft_test_result_fail("%s\n", test_name); + fclose(fp); +} + /* * Returns the start address of the mapping on success, else returns * NULL on failure. @@ -336,6 +378,7 @@ int main(int argc, char **argv) int i, run_perf_tests; unsigned int threshold_mb =3D VALIDATION_DEFAULT_THRESHOLD; unsigned int pattern_seed; + int num_expand_tests =3D 1; struct test test_cases[MAX_TEST]; struct test perf_test_cases[MAX_PERF_TEST]; int page_size; @@ -407,12 +450,14 @@ int main(int argc, char **argv) (threshold_mb * _1MB >=3D _1GB); =20 ksft_set_plan(ARRAY_SIZE(test_cases) + (run_perf_tests ? - ARRAY_SIZE(perf_test_cases) : 0)); + ARRAY_SIZE(perf_test_cases) : 0) + num_expand_tests); =20 for (i =3D 0; i < ARRAY_SIZE(test_cases); i++) run_mremap_test_case(test_cases[i], &failures, threshold_mb, pattern_seed); =20 + mremap_expand_merge(page_size); + if (run_perf_tests) { ksft_print_msg("\n%s\n", "mremap HAVE_MOVE_PMD/PUD optimization time comparison for 1GB region:"= ); --=20 2.35.1