From nobody Tue Apr 7 09:06:15 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5BBD9ECAAD2 for ; Mon, 29 Aug 2022 21:25:48 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229756AbiH2VZp (ORCPT ); Mon, 29 Aug 2022 17:25:45 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59916 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229589AbiH2VZj (ORCPT ); Mon, 29 Aug 2022 17:25:39 -0400 Received: from mail-pl1-x649.google.com (mail-pl1-x649.google.com [IPv6:2607:f8b0:4864:20::649]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 341E9543FC for ; Mon, 29 Aug 2022 14:25:38 -0700 (PDT) Received: by mail-pl1-x649.google.com with SMTP id l16-20020a170902f69000b00175138bcd25so362062plg.2 for ; Mon, 29 Aug 2022 14:25:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc; bh=g4qOnyYThXS8aPbz4q0EiUiPOz/BrsimeRGT1yiKfMQ=; b=mWzgq+g1hUcWORjEKpHNvgndKK5fYzq5Kwo71ogdoOHT+8RKeEQm1dWhj+jH+fzB93 eAtnxNDhkrvgMQV0NbZW6cX2N1R1+vx9KkVCTcRNm2sUroxHvRgBw6l3WRg5Zt3VpzAp Gtf7jyf0iomGZE2cusptU53XV2fFMuQkz5kSqUL16OlTWKpS5jWEYB7QcGrOHtGU2amu R0EpT4v+6Ubf6b7FtWlOEkEYwXOj5Qkd+op5DjD/MSrHyHjvI8tucin/+y2TJJ3NqFBm d0kZdCDqO99cL/X5NSt2crz3dZBhVMVoGOKJ9zBhEnqaTkK0+up1kisxNJ5uiuFsJR5L UYkA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc; bh=g4qOnyYThXS8aPbz4q0EiUiPOz/BrsimeRGT1yiKfMQ=; b=uwFGmBl2ELAOIhHyVDWacmEGuDMzznR8FFLgLWCh7Hr7mGP57wlksBgWG2pOWjXeQu YjF5ocozMTpmD2UrQCwuWhQBrwK1Qu22sLM66lRt10LWLIHwL0HgXuIs/8ibCW1X3eC/ RaCaybqTqgL4k6nZ/cdtGsUyxrbYybVV5LKrBHxuznnbN0Tw8eG3/snYbq7PL8uNaZSa htPZnmMuItOFI4jcQvRCMZEaMc7Rbn1Ruob5ET9SuTFbMpzQnPBcBg1B/b7+5QZeLEaz 68iRdVjWPlxyOZAs/C2S8aP/qOz+5f2VaUCd56CiaU+cQ4Flzyi0IsJMSR40KL5kr/et CGag== X-Gm-Message-State: ACgBeo08Sr49CYXMs28KR6n/c1NMlOCJJ5F37MmA9ShjhmC4sMkWXtWr umgCXxwlR1CGdreSUdBqE1oPsG5Atw8= X-Google-Smtp-Source: AA6agR70MCvwbcIgBEgSXgn9wLeLPhFpHFOZXigYtzK5DEim0KxhWXMFn5l7ORAicxSiaBREEUpeMlkp2eE= X-Received: from surenb-spec.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:3e30]) (user=surenb job=sendgmr) by 2002:a62:5ac6:0:b0:537:f0fa:4ae1 with SMTP id o189-20020a625ac6000000b00537f0fa4ae1mr12574833pfb.70.1661808337643; Mon, 29 Aug 2022 14:25:37 -0700 (PDT) Date: Mon, 29 Aug 2022 21:25:04 +0000 In-Reply-To: <20220829212531.3184856-1-surenb@google.com> Mime-Version: 1.0 References: <20220829212531.3184856-1-surenb@google.com> X-Mailer: git-send-email 2.37.2.672.g94769d06f0-goog Message-ID: <20220829212531.3184856-2-surenb@google.com> Subject: [RFC PATCH 01/28] mm: introduce CONFIG_PER_VMA_LOCK From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, riel@surriel.com, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, surenb@google.com, kernel-team@android.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" This configuration variable will be used to build the support for VMA locking during page fault handling. This is enabled by default on supported architectures with SMP and MMU set. The architecture support is needed since the page fault handler is called from the architecture's page faulting code which needs modifications to handle faults under VMA lock. Signed-off-by: Suren Baghdasaryan --- mm/Kconfig | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/mm/Kconfig b/mm/Kconfig index 0331f1461f81..58c20fad9cf9 100644 --- a/mm/Kconfig +++ b/mm/Kconfig @@ -1124,6 +1124,19 @@ config PTE_MARKER_UFFD_WP purposes. It is required to enable userfaultfd write protection on file-backed memory types like shmem and hugetlbfs. =20 +config ARCH_SUPPORTS_PER_VMA_LOCK + def_bool n + +config PER_VMA_LOCK + bool "Per-vma locking support" + default y + depends on ARCH_SUPPORTS_PER_VMA_LOCK && MMU && SMP + help + Allow per-vma locking during page fault handling. + + This feature allows locking each virtual memory area separately when + handling page faults instead of taking mmap_lock. + source "mm/damon/Kconfig" =20 endmenu --=20 2.37.2.672.g94769d06f0-goog From nobody Tue Apr 7 09:06:15 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3DE85ECAAD4 for ; Mon, 29 Aug 2022 21:25:52 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229475AbiH2VZu (ORCPT ); Mon, 29 Aug 2022 17:25:50 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59994 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229682AbiH2VZl (ORCPT ); Mon, 29 Aug 2022 17:25:41 -0400 Received: from mail-pj1-x1049.google.com (mail-pj1-x1049.google.com [IPv6:2607:f8b0:4864:20::1049]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 28E5ED53 for ; Mon, 29 Aug 2022 14:25:39 -0700 (PDT) Received: by mail-pj1-x1049.google.com with SMTP id n30-20020a17090a5aa100b001fb0c492d5eso3876529pji.3 for ; Mon, 29 Aug 2022 14:25:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc; bh=cPhW2Jsm5VYfGL+tNkfF3HbuaaNlLZ4rflaa6thsEZ0=; b=Ef83aGWTdm10niN8twSSpG3ds5UuJPZXIlmBxHFg2pzrVEdcovVnDh3P/RYOinFzyL 56g3s4OBiiWq4sYS4zyOn+I3sJdcQzzrlMjpsAwGpe/6DNb56LumMLV46gfcGkdOoroQ Xn2Ez1+n3LVumJ9ul/e3Fo4g/eC812QXSiR/GOAlZ/X37IBJZ8GhOEEF+zxhXhrBOhrA wyQ3RxsgNeUblE3TddZPmpCUfJPQ1JoHkMsqbRXnrreQjskaAOAntkxj5MpJLWS1ppBQ s3y364HBN4NsSi7BBvtTGWNjoBJVZ6VaguwCOOzVv7z+u5EVx5HkEViGicNoS6cS9car 5ukQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc; bh=cPhW2Jsm5VYfGL+tNkfF3HbuaaNlLZ4rflaa6thsEZ0=; b=vUCD+/JSTxgS+AoyvoIDXIUt01raJEQlBSaZpvI6jplZP6B1Q4OEAeMAqcBp0SgtUi EcTqYFYNv2ca1RZJr9KWHoOT2saek8Cz+gD9DGQ7iEQF2zt3GJC7f9X29yoHMq4piVf6 xDIfepskCEvaXpuWcGC3aQ/xvnUDvjTkp3dEGcF0l8Gw/HP8dp/W8RIG21Gt/BMaAktz pvVQvM644X2iMRH6Raupdq2qQU5w1fTeJzSG3ha6LVqlIdjqmZ15QV1nz7+tMfLnxSC7 5K+VFYsZ3bzHmtPVfrv3YWsFLhiR7OLiHyX3rTKHWcY1bLRLqfPjrzTzloprg8Hmq2kA dvGA== X-Gm-Message-State: ACgBeo2uOf5/c/tGHAbA0727BoEKXzPv3a29VlUGs2f1p2zARP7B25vq RXI6kv9TPRZHV2gd/dZya8nvIHk+HatCUt/Wf9Iad+eqv1mdua3IPeCUlME6q3suIiGque3/lbx rkpqoz7Jt2E97FMPrjDBc X-Google-Smtp-Source: AA6agR4h8d6XU8wf7Rba89JuzvO6/M5QQD9xRgO2erAMFyngZx2nEIGpax2LUa1YM3lFKVFmr9HgbHhBg9U= X-Received: from surenb-spec.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:3e30]) (user=surenb job=sendgmr) by 2002:a17:902:694a:b0:16e:e270:1f84 with SMTP id k10-20020a170902694a00b0016ee2701f84mr17988146plt.89.1661808339339; Mon, 29 Aug 2022 14:25:39 -0700 (PDT) Date: Mon, 29 Aug 2022 21:25:05 +0000 In-Reply-To: <20220829212531.3184856-1-surenb@google.com> Mime-Version: 1.0 References: <20220829212531.3184856-1-surenb@google.com> X-Mailer: git-send-email 2.37.2.672.g94769d06f0-goog Message-ID: <20220829212531.3184856-3-surenb@google.com> Subject: [RFC PATCH 02/28] mm: rcu safe VMA freeing From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, riel@surriel.com, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, surenb@google.com, kernel-team@android.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org X-ccpol: medium Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Michel Lespinasse This prepares for page faults handling under VMA lock, looking up VMAs under protection of an rcu read lock, instead of the usual mmap read lock. Signed-off-by: Michel Lespinasse Signed-off-by: Suren Baghdasaryan --- include/linux/mm_types.h | 16 +++++++++++----- kernel/fork.c | 13 +++++++++++++ 2 files changed, 24 insertions(+), 5 deletions(-) diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index cf97f3884fda..bed25ef7c994 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -403,12 +403,18 @@ struct anon_vma_name { struct vm_area_struct { /* The first cache line has the info for VMA tree walking. */ =20 - unsigned long vm_start; /* Our start address within vm_mm. */ - unsigned long vm_end; /* The first byte after our end address - within vm_mm. */ + union { + struct { + /* VMA covers [vm_start; vm_end) addresses within mm */ + unsigned long vm_start, vm_end; =20 - /* linked list of VM areas per task, sorted by address */ - struct vm_area_struct *vm_next, *vm_prev; + /* linked list of VMAs per task, sorted by address */ + struct vm_area_struct *vm_next, *vm_prev; + }; +#ifdef CONFIG_PER_VMA_LOCK + struct rcu_head vm_rcu; /* Used for deferred freeing. */ +#endif + }; =20 struct rb_node vm_rb; =20 diff --git a/kernel/fork.c b/kernel/fork.c index 90c85b17bf69..614872438393 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -481,10 +481,23 @@ struct vm_area_struct *vm_area_dup(struct vm_area_str= uct *orig) return new; } =20 +#ifdef CONFIG_PER_VMA_LOCK +static void __vm_area_free(struct rcu_head *head) +{ + struct vm_area_struct *vma =3D container_of(head, struct vm_area_struct, + vm_rcu); + kmem_cache_free(vm_area_cachep, vma); +} +#endif + void vm_area_free(struct vm_area_struct *vma) { free_anon_vma_name(vma); +#ifdef CONFIG_PER_VMA_LOCK + call_rcu(&vma->vm_rcu, __vm_area_free); +#else kmem_cache_free(vm_area_cachep, vma); +#endif } =20 static void account_kernel_stack(struct task_struct *tsk, int account) --=20 2.37.2.672.g94769d06f0-goog From nobody Tue Apr 7 09:06:15 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 35137ECAAD8 for ; Mon, 29 Aug 2022 21:25:57 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229770AbiH2VZy (ORCPT ); Mon, 29 Aug 2022 17:25:54 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60112 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229575AbiH2VZn (ORCPT ); Mon, 29 Aug 2022 17:25:43 -0400 Received: from mail-yw1-x1149.google.com (mail-yw1-x1149.google.com [IPv6:2607:f8b0:4864:20::1149]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2778F57200 for ; Mon, 29 Aug 2022 14:25:42 -0700 (PDT) Received: by mail-yw1-x1149.google.com with SMTP id 00721157ae682-33daeaa6b8eso146119777b3.7 for ; Mon, 29 Aug 2022 14:25:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc; bh=r66X5QU6Hd1zKKrQM4VdH4G/hUbDUrS2PjBFocCJ7kE=; b=f0ZITQpEExaLaSKDQb+h3/jYVFf5BjWfu0A0p9NMJ80xDBwkhscojqDzgDQ6W2gLIU JqyvlNc6N0TYQxhXpPiBzGuyzj1xr5cqDIbNzDR0v00qVutFwkDUQtyKnSj2Dn+NYRmF XbYPUhr7ZlEh9zlxoVr+7UyC5/R2ipvFX7ptRmhGZgjobP0J3ITaBrEOKND2CAY9iU2J fnwalrszPulqwRWfSp/VwWSeSLynEBR581yqV4QALN8e0aPzId5HIRtAjN2fmV4usJpw ghPowcSLA4CLZg9Bm0wFQDYbe3eb9NsWWsLPsurv1jkHcSRK8KYhRIFhxtRmb7OcOQkd NeDw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc; bh=r66X5QU6Hd1zKKrQM4VdH4G/hUbDUrS2PjBFocCJ7kE=; b=UiwEdxtkM35KNZtuQzLnuh0nrNVjlMUE52Z9AXby6hwIZJDjDYJnjADX7d0nZ8ZEyM Rd1f1OCrDEhVig6XlcOMbVai6ZaW/ce8wWZuYM0u80H+tsKsXeW155ghhzG9LFtj9+t+ ZmVudga/Yf492Am9bsll8uWNtdpl6PDoA6jDYohQ/cmrIrvWvrxPUePRRM4rD6294CZM JuwSkR12Cb744EttyQc9zPGrjDPzJNGy0P+cswJQuXVytovXzo2+y1y4zFkzy/XsL56S pHdtNOwsrG0vjzZWLGumnx9eTuv/a4e4txP36rFLkpMN819tzOUPK6vESFTdtk9MvUH5 eYFQ== X-Gm-Message-State: ACgBeo0dcufAtSkoipyKWMKr11nSprS7MiM8ARNRoedFSGwxe8Rz3xlW 5juPM5PxuUZPeHZ37xuruGVBWD2k3Y0= X-Google-Smtp-Source: AA6agR7RORyYEvgHp7XQu1QQYhj/biZijMsx0xRWZsAvcqgghjoomkWy7B4TiExDCzWJq5q+aaEXQCYL4WE= X-Received: from surenb-spec.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:3e30]) (user=surenb job=sendgmr) by 2002:a25:254f:0:b0:695:7c37:9202 with SMTP id l76-20020a25254f000000b006957c379202mr10232936ybl.79.1661808341087; Mon, 29 Aug 2022 14:25:41 -0700 (PDT) Date: Mon, 29 Aug 2022 21:25:06 +0000 In-Reply-To: <20220829212531.3184856-1-surenb@google.com> Mime-Version: 1.0 References: <20220829212531.3184856-1-surenb@google.com> X-Mailer: git-send-email 2.37.2.672.g94769d06f0-goog Message-ID: <20220829212531.3184856-4-surenb@google.com> Subject: [RFC PATCH 03/28] mm: introduce __find_vma to be used without mmap_lock protection From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, riel@surriel.com, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, surenb@google.com, kernel-team@android.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Add __find_vma function to be used for VMA lookup under rcu protection. Signed-off-by: Suren Baghdasaryan --- drivers/gpu/drm/i915/i915_gpu_error.c | 4 ++-- include/linux/mm.h | 9 ++++++++- mm/mmap.c | 6 ++---- 3 files changed, 12 insertions(+), 7 deletions(-) diff --git a/drivers/gpu/drm/i915/i915_gpu_error.c b/drivers/gpu/drm/i915/i= 915_gpu_error.c index 32e92651ef7c..fc94985c95c8 100644 --- a/drivers/gpu/drm/i915/i915_gpu_error.c +++ b/drivers/gpu/drm/i915/i915_gpu_error.c @@ -507,7 +507,7 @@ static void error_print_context(struct drm_i915_error_s= tate_buf *m, } =20 static struct i915_vma_coredump * -__find_vma(struct i915_vma_coredump *vma, const char *name) +__i915_find_vma(struct i915_vma_coredump *vma, const char *name) { while (vma) { if (strcmp(vma->name, name) =3D=3D 0) @@ -521,7 +521,7 @@ __find_vma(struct i915_vma_coredump *vma, const char *n= ame) struct i915_vma_coredump * intel_gpu_error_find_batch(const struct intel_engine_coredump *ee) { - return __find_vma(ee->vma, "batch"); + return __i915_find_vma(ee->vma, "batch"); } =20 static void error_print_engine(struct drm_i915_error_state_buf *m, diff --git a/include/linux/mm.h b/include/linux/mm.h index 21f8b27bd9fd..7d322a979455 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -2712,7 +2712,14 @@ extern int expand_upwards(struct vm_area_struct *vma= , unsigned long address); #endif =20 /* Look up the first VMA which satisfies addr < vm_end, NULL if none. */ -extern struct vm_area_struct * find_vma(struct mm_struct * mm, unsigned lo= ng addr); +extern struct vm_area_struct *__find_vma(struct mm_struct *mm, unsigned lo= ng addr); +static inline +struct vm_area_struct *find_vma(struct mm_struct *mm, unsigned long addr) +{ + mmap_assert_locked(mm); + return __find_vma(mm, addr); +} + extern struct vm_area_struct * find_vma_prev(struct mm_struct * mm, unsign= ed long addr, struct vm_area_struct **pprev); =20 diff --git a/mm/mmap.c b/mm/mmap.c index 9d780f415be3..693e6776be39 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -2250,12 +2250,11 @@ get_unmapped_area(struct file *file, unsigned long = addr, unsigned long len, EXPORT_SYMBOL(get_unmapped_area); =20 /* Look up the first VMA which satisfies addr < vm_end, NULL if none. */ -struct vm_area_struct *find_vma(struct mm_struct *mm, unsigned long addr) +struct vm_area_struct *__find_vma(struct mm_struct *mm, unsigned long addr) { struct rb_node *rb_node; struct vm_area_struct *vma; =20 - mmap_assert_locked(mm); /* Check the cache first. */ vma =3D vmacache_find(mm, addr); if (likely(vma)) @@ -2281,8 +2280,7 @@ struct vm_area_struct *find_vma(struct mm_struct *mm,= unsigned long addr) vmacache_update(addr, vma); return vma; } - -EXPORT_SYMBOL(find_vma); +EXPORT_SYMBOL(__find_vma); =20 /* * Same as find_vma, but also return a pointer to the previous VMA in *ppr= ev. --=20 2.37.2.672.g94769d06f0-goog From nobody Tue Apr 7 09:06:15 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 15147ECAAD4 for ; Mon, 29 Aug 2022 21:26:01 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229777AbiH2VZ6 (ORCPT ); Mon, 29 Aug 2022 17:25:58 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60202 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229751AbiH2VZp (ORCPT ); Mon, 29 Aug 2022 17:25:45 -0400 Received: from mail-yw1-x114a.google.com (mail-yw1-x114a.google.com [IPv6:2607:f8b0:4864:20::114a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C6F945A168 for ; Mon, 29 Aug 2022 14:25:43 -0700 (PDT) Received: by mail-yw1-x114a.google.com with SMTP id 00721157ae682-33f8988daecso138732817b3.12 for ; Mon, 29 Aug 2022 14:25:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc; bh=VHFOj4kfUa3xGA3HxCxfqSH2fyPnkF//iB8ZL7PxHTQ=; b=XClgxH8bXaTLY5kgySHmNtvDCCVAB8mjOROUY69yCIkvhLsUcaeiTHEG390LZTJ0UJ f+bTq7v3mEXk1WZIYhpvHYfgX0MddKEDnFBtUXG7eI4mzGho1uyo+BRigftRuUZpe5ed yTJv9FwpqBMqwvSi+XVWkQi2sCAbfOGrPpu1lkrGFWjYvxB7buAqdmGdNV7OSfwNKssU SvorUO1mGFzFR5b/hO2rDfP9WSX1xgC98rCcdyQIaEqfPxTvpPYobfXt4iE+GIx2WNMy khAj4t26F+rdvp0rGETv3PRP0ykKrNZPVAZGmSPMWGLG1X6s1pkbIB5IUW2TBK/ZgFFb KwQg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc; bh=VHFOj4kfUa3xGA3HxCxfqSH2fyPnkF//iB8ZL7PxHTQ=; b=CKBxuEUKK1wm6WDvv6DRsswUfsnUVsg2BJHaYquEFKBSpGybGZB4WFgPEMKa2ILwj5 wALUf6z/NgXSHlW3A61pkSC74qcH6nOLGjQ3SuOQU9un1To4Wt1fVQX290d9B2J1GiQv L6VNQ+pXuqJ63z7ResMfoOBn1MpvUAB/Qsny/ozbBHwf7bPoMMQR+ANnjFWBoiIULfg5 JASVgcjs9YTRJRQtebc7BxSknVfvTtrbw901gTaAov88xdQanp8ChjHUBO2yVzpis/j8 ARZm1p9DwQ6avEFzg83cBrsMsWvD+Bgkuc8G8OHUWLkKCjEvvP5svrKfQB03fPhv5Oxf oLNw== X-Gm-Message-State: ACgBeo0vbAOGdzJKPRQyHr5GB1VLe8SS3rhXsxUebF8Wgf9YPeIwudzq 6YLPnbsdf77l1rhdKgdQ2rJYGhzALiU= X-Google-Smtp-Source: AA6agR6wyKOqZ82JQBWMvaTOKlqh8vry2Onq3V9mMTZsNJ+QDBtzKC1cVl90fnxR++4fsXM2QhKkAh66mSQ= X-Received: from surenb-spec.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:3e30]) (user=surenb job=sendgmr) by 2002:a0d:eac9:0:b0:33d:c025:8a74 with SMTP id t192-20020a0deac9000000b0033dc0258a74mr11722581ywe.430.1661808342911; Mon, 29 Aug 2022 14:25:42 -0700 (PDT) Date: Mon, 29 Aug 2022 21:25:07 +0000 In-Reply-To: <20220829212531.3184856-1-surenb@google.com> Mime-Version: 1.0 References: <20220829212531.3184856-1-surenb@google.com> X-Mailer: git-send-email 2.37.2.672.g94769d06f0-goog Message-ID: <20220829212531.3184856-5-surenb@google.com> Subject: [RFC PATCH 04/28] mm: move mmap_lock assert function definitions From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, riel@surriel.com, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, surenb@google.com, kernel-team@android.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Move mmap_lock assert function definitions up so that they can be used by other mmap_lock routines. Signed-off-by: Suren Baghdasaryan --- include/linux/mmap_lock.h | 24 ++++++++++++------------ 1 file changed, 12 insertions(+), 12 deletions(-) diff --git a/include/linux/mmap_lock.h b/include/linux/mmap_lock.h index 96e113e23d04..e49ba91bb1f0 100644 --- a/include/linux/mmap_lock.h +++ b/include/linux/mmap_lock.h @@ -60,6 +60,18 @@ static inline void __mmap_lock_trace_released(struct mm_= struct *mm, bool write) =20 #endif /* CONFIG_TRACING */ =20 +static inline void mmap_assert_locked(struct mm_struct *mm) +{ + lockdep_assert_held(&mm->mmap_lock); + VM_BUG_ON_MM(!rwsem_is_locked(&mm->mmap_lock), mm); +} + +static inline void mmap_assert_write_locked(struct mm_struct *mm) +{ + lockdep_assert_held_write(&mm->mmap_lock); + VM_BUG_ON_MM(!rwsem_is_locked(&mm->mmap_lock), mm); +} + static inline void mmap_init_lock(struct mm_struct *mm) { init_rwsem(&mm->mmap_lock); @@ -150,18 +162,6 @@ static inline void mmap_read_unlock_non_owner(struct m= m_struct *mm) up_read_non_owner(&mm->mmap_lock); } =20 -static inline void mmap_assert_locked(struct mm_struct *mm) -{ - lockdep_assert_held(&mm->mmap_lock); - VM_BUG_ON_MM(!rwsem_is_locked(&mm->mmap_lock), mm); -} - -static inline void mmap_assert_write_locked(struct mm_struct *mm) -{ - lockdep_assert_held_write(&mm->mmap_lock); - VM_BUG_ON_MM(!rwsem_is_locked(&mm->mmap_lock), mm); -} - static inline int mmap_lock_is_contended(struct mm_struct *mm) { return rwsem_is_contended(&mm->mmap_lock); --=20 2.37.2.672.g94769d06f0-goog From nobody Tue Apr 7 09:06:15 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id DEAB0ECAAD4 for ; Mon, 29 Aug 2022 21:26:17 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229817AbiH2V0O (ORCPT ); Mon, 29 Aug 2022 17:26:14 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60840 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229781AbiH2V0E (ORCPT ); Mon, 29 Aug 2022 17:26:04 -0400 Received: from mail-pl1-x64a.google.com (mail-pl1-x64a.google.com [IPv6:2607:f8b0:4864:20::64a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id AD77857884 for ; Mon, 29 Aug 2022 14:25:45 -0700 (PDT) Received: by mail-pl1-x64a.google.com with SMTP id p8-20020a170902e74800b0017307429ca3so6839759plf.17 for ; Mon, 29 Aug 2022 14:25:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc; bh=N9bIms4Zgtt1w5mfZvOohRmDSgEYydYD4bEwHw/V5ow=; b=CDYqj2724k+j3UZuEqBII6d0mYlErNKF+DONYkTH+E04Ngvdyx6CFcIzGI3JpE+Ocv IfFz52ELd/oHC5JbA4KJiqR41o93UU1oNs91Ejb3C2WUYMd8rJbvn4wGpH2TVPFpMmJT SgaAVJGTguR7jHY2klbLNlMZG17tv+P143dcZzCLTWGQf5RYnpB7E01bhfNNtgZSKJd4 8iRdR4KuZii8bgu+gGfhXIXfMfDbZfPU8zkTguml14FGrqbs//qWRwWzX5xJRRQfxW3M XOIZ22KxRY8zlSKN1mF3msZnGKAdWpIuRmlb/EMY5nhFhrrOHs7u2uNNMJtGec19XB/4 ScVA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc; bh=N9bIms4Zgtt1w5mfZvOohRmDSgEYydYD4bEwHw/V5ow=; b=yco3RAXEBf0k8WkDJ3iuJyBlDhkokQJoBW8rheLvyqDRcgVSNOrrG3A69kvPsHlpf2 rWoNfaeIrw7H0m24xWuo19vtNKUsb9hALeylcXGG9OnJ7jtOLLiTE5iuTo6+zEu4no2b J4Dt7yQsbRd8gneTnybvLHUGypxdCRybiNGhJX+bKsyMVWTTvbd9W+Wr3g84Z9FBt1Ty N/1lkz7JINOKu4E64EC7oQSRmjF+EB0axQnCUg2GDWXO0ndhrDwt+t3FGi7DkiVeYEEU YIvFibP6zb79mv3UAd7gFM3EIVvcSicXL7z7D9UvNPwd+JA5lD9yZK0KQiw5W3cLcZSn 9EUw== X-Gm-Message-State: ACgBeo3KMhRyBVPc32WWPZhbTSIkmLAa8z5EY51lK+ZFxb3na+BKe7tj tNv7yofd+zdjsFliwxfdyVduBQHHob2Mer74DWQRG1CWysKPaYyUr5lQ4oX+09VagwT0gYvgUiy 5l77Q6TWbUMceNp4Nuf/L X-Google-Smtp-Source: AA6agR7o7Ix1gbIIukXoy+30trWr4dJQ0Q9jpP73SljSWj/tMusjSLFwSN2Bk21+x1fzs5P+BkoH3vuJaG0= X-Received: from surenb-spec.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:3e30]) (user=surenb job=sendgmr) by 2002:aa7:8a0b:0:b0:537:f5f1:fd91 with SMTP id m11-20020aa78a0b000000b00537f5f1fd91mr12449109pfa.41.1661808344609; Mon, 29 Aug 2022 14:25:44 -0700 (PDT) Date: Mon, 29 Aug 2022 21:25:08 +0000 In-Reply-To: <20220829212531.3184856-1-surenb@google.com> Mime-Version: 1.0 References: <20220829212531.3184856-1-surenb@google.com> X-Mailer: git-send-email 2.37.2.672.g94769d06f0-goog Message-ID: <20220829212531.3184856-6-surenb@google.com> Subject: [RFC PATCH 05/28] mm: add per-VMA lock and helper functions to control it From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, riel@surriel.com, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, surenb@google.com, kernel-team@android.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org X-ccpol: medium Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Introduce a per-VMA rw_semaphore to be used during page fault handling instead of mmap_lock. Because there are cases when multiple VMAs need to be exclusively locked during VMA tree modifications, instead of the usual lock/unlock patter we mark a VMA as locked by taking per-VMA lock exclusively and setting vma->lock_seq to the current mm->lock_seq. When mmap_write_lock holder is done with all modifications and drops mmap_lock, it will increment mm->lock_seq, effectively unlocking all VMAs marked as locked. Signed-off-by: Suren Baghdasaryan --- include/linux/mm.h | 78 +++++++++++++++++++++++++++++++++++++++ include/linux/mm_types.h | 7 ++++ include/linux/mmap_lock.h | 13 +++++++ kernel/fork.c | 4 ++ mm/init-mm.c | 3 ++ 5 files changed, 105 insertions(+) diff --git a/include/linux/mm.h b/include/linux/mm.h index 7d322a979455..476bf936c5f0 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -611,6 +611,83 @@ struct vm_operations_struct { unsigned long addr); }; =20 +#ifdef CONFIG_PER_VMA_LOCK +static inline void vma_init_lock(struct vm_area_struct *vma) +{ + init_rwsem(&vma->lock); + vma->vm_lock_seq =3D -1; +} + +static inline void vma_mark_locked(struct vm_area_struct *vma) +{ + int mm_lock_seq; + + mmap_assert_write_locked(vma->vm_mm); + + /* + * current task is holding mmap_write_lock, both vma->vm_lock_seq and + * mm->mm_lock_seq can't be concurrently modified. + */ + mm_lock_seq =3D READ_ONCE(vma->vm_mm->mm_lock_seq); + if (vma->vm_lock_seq =3D=3D mm_lock_seq) + return; + + down_write(&vma->lock); + vma->vm_lock_seq =3D mm_lock_seq; + up_write(&vma->lock); +} + +static inline bool vma_read_trylock(struct vm_area_struct *vma) +{ + if (unlikely(down_read_trylock(&vma->lock) =3D=3D 0)) + return false; + + /* + * Overflow might produce false locked result but it's not critical. + * False unlocked result is critical but is impossible because we + * modify and check vma->vm_lock_seq under vma->lock protection and + * mm->mm_lock_seq modification invalidates all existing locks. + */ + if (vma->vm_lock_seq =3D=3D READ_ONCE(vma->vm_mm->mm_lock_seq)) { + up_read(&vma->lock); + return false; + } + return true; +} + +static inline void vma_read_unlock(struct vm_area_struct *vma) +{ + up_read(&vma->lock); +} + +static inline void vma_assert_locked(struct vm_area_struct *vma) +{ + lockdep_assert_held(&vma->lock); + VM_BUG_ON_VMA(!rwsem_is_locked(&vma->lock), vma); +} + +static inline void vma_assert_write_locked(struct vm_area_struct *vma, int= pos) +{ + mmap_assert_write_locked(vma->vm_mm); + /* + * current task is holding mmap_write_lock, both vma->vm_lock_seq and + * mm->mm_lock_seq can't be concurrently modified. + */ + VM_BUG_ON_VMA(vma->vm_lock_seq !=3D READ_ONCE(vma->vm_mm->mm_lock_seq), v= ma); +} + +#else /* CONFIG_PER_VMA_LOCK */ + +static inline void vma_init_lock(struct vm_area_struct *vma) {} +static inline void vma_mark_locked(struct vm_area_struct *vma) {} +static inline bool vma_read_trylock(struct vm_area_struct *vma) + { return false; } +static inline void vma_read_unlock(struct vm_area_struct *vma) {} +static inline void vma_assert_locked(struct vm_area_struct *vma) {} +static inline void vma_assert_write_locked(struct vm_area_struct *vma, int= pos) {} + +#endif /* CONFIG_PER_VMA_LOCK */ + static inline void vma_init(struct vm_area_struct *vma, struct mm_struct *= mm) { static const struct vm_operations_struct dummy_vm_ops =3D {}; @@ -619,6 +696,7 @@ static inline void vma_init(struct vm_area_struct *vma,= struct mm_struct *mm) vma->vm_mm =3D mm; vma->vm_ops =3D &dummy_vm_ops; INIT_LIST_HEAD(&vma->anon_vma_chain); + vma_init_lock(vma); } =20 static inline void vma_set_anonymous(struct vm_area_struct *vma) diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index bed25ef7c994..6a03f59c1e78 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -486,6 +486,10 @@ struct vm_area_struct { struct mempolicy *vm_policy; /* NUMA policy for the VMA */ #endif struct vm_userfaultfd_ctx vm_userfaultfd_ctx; +#ifdef CONFIG_PER_VMA_LOCK + struct rw_semaphore lock; + int vm_lock_seq; +#endif } __randomize_layout; =20 struct kioctx_table; @@ -567,6 +571,9 @@ struct mm_struct { * init_mm.mmlist, and are protected * by mmlist_lock */ +#ifdef CONFIG_PER_VMA_LOCK + int mm_lock_seq; +#endif =20 =20 unsigned long hiwater_rss; /* High-watermark of RSS usage */ diff --git a/include/linux/mmap_lock.h b/include/linux/mmap_lock.h index e49ba91bb1f0..a391ae226564 100644 --- a/include/linux/mmap_lock.h +++ b/include/linux/mmap_lock.h @@ -72,6 +72,17 @@ static inline void mmap_assert_write_locked(struct mm_st= ruct *mm) VM_BUG_ON_MM(!rwsem_is_locked(&mm->mmap_lock), mm); } =20 +#ifdef CONFIG_PER_VMA_LOCK +static inline void vma_mark_unlocked_all(struct mm_struct *mm) +{ + mmap_assert_write_locked(mm); + /* No races during update due to exclusive mmap_lock being held */ + WRITE_ONCE(mm->mm_lock_seq, mm->mm_lock_seq + 1); +} +#else +static inline void vma_mark_unlocked_all(struct mm_struct *mm) {} +#endif + static inline void mmap_init_lock(struct mm_struct *mm) { init_rwsem(&mm->mmap_lock); @@ -114,12 +125,14 @@ static inline bool mmap_write_trylock(struct mm_struc= t *mm) static inline void mmap_write_unlock(struct mm_struct *mm) { __mmap_lock_trace_released(mm, true); + vma_mark_unlocked_all(mm); up_write(&mm->mmap_lock); } =20 static inline void mmap_write_downgrade(struct mm_struct *mm) { __mmap_lock_trace_acquire_returned(mm, false, true); + vma_mark_unlocked_all(mm); downgrade_write(&mm->mmap_lock); } =20 diff --git a/kernel/fork.c b/kernel/fork.c index 614872438393..bfab31ecd11e 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -475,6 +475,7 @@ struct vm_area_struct *vm_area_dup(struct vm_area_struc= t *orig) */ *new =3D data_race(*orig); INIT_LIST_HEAD(&new->anon_vma_chain); + vma_init_lock(new); new->vm_next =3D new->vm_prev =3D NULL; dup_anon_vma_name(orig, new); } @@ -1130,6 +1131,9 @@ static struct mm_struct *mm_init(struct mm_struct *mm= , struct task_struct *p, seqcount_init(&mm->write_protect_seq); mmap_init_lock(mm); INIT_LIST_HEAD(&mm->mmlist); +#ifdef CONFIG_PER_VMA_LOCK + WRITE_ONCE(mm->mm_lock_seq, 0); +#endif mm_pgtables_bytes_init(mm); mm->map_count =3D 0; mm->locked_vm =3D 0; diff --git a/mm/init-mm.c b/mm/init-mm.c index fbe7844d0912..8399f90d631c 100644 --- a/mm/init-mm.c +++ b/mm/init-mm.c @@ -37,6 +37,9 @@ struct mm_struct init_mm =3D { .page_table_lock =3D __SPIN_LOCK_UNLOCKED(init_mm.page_table_lock), .arg_lock =3D __SPIN_LOCK_UNLOCKED(init_mm.arg_lock), .mmlist =3D LIST_HEAD_INIT(init_mm.mmlist), +#ifdef CONFIG_PER_VMA_LOCK + .mm_lock_seq =3D 0, +#endif .user_ns =3D &init_user_ns, .cpu_bitmap =3D CPU_BITS_NONE, #ifdef CONFIG_IOMMU_SVA --=20 2.37.2.672.g94769d06f0-goog From nobody Tue Apr 7 09:06:15 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 925ECECAAD8 for ; Mon, 29 Aug 2022 21:26:24 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229875AbiH2V0U (ORCPT ); Mon, 29 Aug 2022 17:26:20 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:32838 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229788AbiH2V0I (ORCPT ); Mon, 29 Aug 2022 17:26:08 -0400 Received: from mail-pl1-x649.google.com (mail-pl1-x649.google.com [IPv6:2607:f8b0:4864:20::649]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E7F2C6E2C2 for ; Mon, 29 Aug 2022 14:25:47 -0700 (PDT) Received: by mail-pl1-x649.google.com with SMTP id s8-20020a170902ea0800b00172e456031eso6759721plg.3 for ; Mon, 29 Aug 2022 14:25:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc; bh=Ymgoo1txtx46eok3IPOZQ+7oQJBGfBlI8J01hhP7hOA=; b=V7GGgx774wbENy2sL3aG99hkTB/qGZUI2IMkW5Z7Ox/AY4k1IvJaJqWcS2+hJrQTXz qJiaho12WNhOPkKsDNI4kFEEE0MJklisQUNbnL2MLe3SXi7RR3OSC49CxUHm5rWNBWOA wHnBj0rB97rbw4NvbtXfeTLWG+Vyzon45DGcHKdD67SN4QP16ppc4Mfgkavx0/TZazKT jRNtTwjBlBNTzgM29WASwCLg6OtowkNwGUZwoCyX8zuUMvza2TeNlSBp6kFeN5K2wkE2 aGwm4Y8asgOKWmz3tiq46nHGojR+kREyp0h1B2grUgz1rUK/O35dZpO1+I8qz72cYmP5 /MNQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc; bh=Ymgoo1txtx46eok3IPOZQ+7oQJBGfBlI8J01hhP7hOA=; b=1/UjouWp4gqJwzjTTXsbnRLV3k5pNGC1rv+gf99KSMNOSzLwFhKiMzPtpIGEsZBmih wPMB2xLLsi+D+bMSW71Iwr4kkfhRqE/sMnUY5g4t8k6nMYwj/kOK7Vk8mZfL39oqbRUz 0JRD/SS72t/FGc1FgiJPyfIbypxyu16O1ZR7Dl996+M0ThOfYNIDwuasqb7Nttqsw9a1 LjWCfS7BhT6ciuJUVeaF6gIqXqoo+rKqDzyHicoRVY7C9lGe4LCbVb6QimIDz30uTAIz QO+YiPwIaWH/hPpJJHfdrnC5x0cNeP9e/zXPrpjOws4Hc/9EviMcqa6twPQFaZDkJfra mTMw== X-Gm-Message-State: ACgBeo3HG3Ug74rNEJsAH7yqTIWW7JwQKVSoTqEpf3Ww3qGyvB2YTvjU aIED7Y4Pi+31tM/hzjwnwM+VSXkpGqY= X-Google-Smtp-Source: AA6agR5xVu1GUsjULFDqf0en50jiRHsHcqxte0InGO36KTqnyV9YWBjqbQ+B0gx1566Kx/T+OnpxLJSWOq4= X-Received: from surenb-spec.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:3e30]) (user=surenb job=sendgmr) by 2002:a17:902:d48a:b0:16f:c31:7005 with SMTP id c10-20020a170902d48a00b0016f0c317005mr18369268plg.173.1661808346364; Mon, 29 Aug 2022 14:25:46 -0700 (PDT) Date: Mon, 29 Aug 2022 21:25:09 +0000 In-Reply-To: <20220829212531.3184856-1-surenb@google.com> Mime-Version: 1.0 References: <20220829212531.3184856-1-surenb@google.com> X-Mailer: git-send-email 2.37.2.672.g94769d06f0-goog Message-ID: <20220829212531.3184856-7-surenb@google.com> Subject: [RFC PATCH 06/28] mm: mark VMA as locked whenever vma->vm_flags are modified From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, riel@surriel.com, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, surenb@google.com, kernel-team@android.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" VMA flag modifications should be done under VMA lock to prevent concurrent page fault handling in that area. Signed-off-by: Suren Baghdasaryan --- fs/proc/task_mmu.c | 1 + fs/userfaultfd.c | 6 ++++++ mm/madvise.c | 1 + mm/mlock.c | 2 ++ mm/mmap.c | 1 + mm/mprotect.c | 1 + 6 files changed, 12 insertions(+) diff --git a/fs/proc/task_mmu.c b/fs/proc/task_mmu.c index 4e0023643f8b..ceffa5c2c650 100644 --- a/fs/proc/task_mmu.c +++ b/fs/proc/task_mmu.c @@ -1285,6 +1285,7 @@ static ssize_t clear_refs_write(struct file *file, co= nst char __user *buf, for (vma =3D mm->mmap; vma; vma =3D vma->vm_next) { if (!(vma->vm_flags & VM_SOFTDIRTY)) continue; + vma_mark_locked(vma); vma->vm_flags &=3D ~VM_SOFTDIRTY; vma_set_page_prot(vma); } diff --git a/fs/userfaultfd.c b/fs/userfaultfd.c index 175de70e3adf..fe557b3d1c07 100644 --- a/fs/userfaultfd.c +++ b/fs/userfaultfd.c @@ -620,6 +620,7 @@ static void userfaultfd_event_wait_completion(struct us= erfaultfd_ctx *ctx, mmap_write_lock(mm); for (vma =3D mm->mmap; vma; vma =3D vma->vm_next) if (vma->vm_userfaultfd_ctx.ctx =3D=3D release_new_ctx) { + vma_mark_locked(vma); vma->vm_userfaultfd_ctx =3D NULL_VM_UFFD_CTX; vma->vm_flags &=3D ~__VM_UFFD_FLAGS; } @@ -653,6 +654,7 @@ int dup_userfaultfd(struct vm_area_struct *vma, struct = list_head *fcs) =20 octx =3D vma->vm_userfaultfd_ctx.ctx; if (!octx || !(octx->features & UFFD_FEATURE_EVENT_FORK)) { + vma_mark_locked(vma); vma->vm_userfaultfd_ctx =3D NULL_VM_UFFD_CTX; vma->vm_flags &=3D ~__VM_UFFD_FLAGS; return 0; @@ -734,6 +736,7 @@ void mremap_userfaultfd_prep(struct vm_area_struct *vma, atomic_inc(&ctx->mmap_changing); } else { /* Drop uffd context if remap feature not enabled */ + vma_mark_locked(vma); vma->vm_userfaultfd_ctx =3D NULL_VM_UFFD_CTX; vma->vm_flags &=3D ~__VM_UFFD_FLAGS; } @@ -891,6 +894,7 @@ static int userfaultfd_release(struct inode *inode, str= uct file *file) vma =3D prev; else prev =3D vma; + vma_mark_locked(vma); vma->vm_flags =3D new_flags; vma->vm_userfaultfd_ctx =3D NULL_VM_UFFD_CTX; } @@ -1449,6 +1453,7 @@ static int userfaultfd_register(struct userfaultfd_ct= x *ctx, * the next vma was merged into the current one and * the current one has not been updated yet. */ + vma_mark_locked(vma); vma->vm_flags =3D new_flags; vma->vm_userfaultfd_ctx.ctx =3D ctx; =20 @@ -1630,6 +1635,7 @@ static int userfaultfd_unregister(struct userfaultfd_= ctx *ctx, * the next vma was merged into the current one and * the current one has not been updated yet. */ + vma_mark_locked(vma); vma->vm_flags =3D new_flags; vma->vm_userfaultfd_ctx =3D NULL_VM_UFFD_CTX; =20 diff --git a/mm/madvise.c b/mm/madvise.c index 5f0f0948a50e..a173f0025abd 100644 --- a/mm/madvise.c +++ b/mm/madvise.c @@ -181,6 +181,7 @@ static int madvise_update_vma(struct vm_area_struct *vm= a, /* * vm_flags is protected by the mmap_lock held in write mode. */ + vma_mark_locked(vma); vma->vm_flags =3D new_flags; if (!vma->vm_file) { error =3D replace_anon_vma_name(vma, anon_name); diff --git a/mm/mlock.c b/mm/mlock.c index b14e929084cc..f62e1a4d05f2 100644 --- a/mm/mlock.c +++ b/mm/mlock.c @@ -380,6 +380,7 @@ static void mlock_vma_pages_range(struct vm_area_struct= *vma, */ if (newflags & VM_LOCKED) newflags |=3D VM_IO; + vma_mark_locked(vma); WRITE_ONCE(vma->vm_flags, newflags); =20 lru_add_drain(); @@ -456,6 +457,7 @@ static int mlock_fixup(struct vm_area_struct *vma, stru= ct vm_area_struct **prev, =20 if ((newflags & VM_LOCKED) && (oldflags & VM_LOCKED)) { /* No work to do, and mlocking twice would be wrong */ + vma_mark_locked(vma); vma->vm_flags =3D newflags; } else { mlock_vma_pages_range(vma, start, end, newflags); diff --git a/mm/mmap.c b/mm/mmap.c index 693e6776be39..f89c9b058105 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -1818,6 +1818,7 @@ unsigned long mmap_region(struct file *file, unsigned= long addr, out: perf_event_mmap(vma); =20 + vma_mark_locked(vma); vm_stat_account(mm, vm_flags, len >> PAGE_SHIFT); if (vm_flags & VM_LOCKED) { if ((vm_flags & VM_SPECIAL) || vma_is_dax(vma) || diff --git a/mm/mprotect.c b/mm/mprotect.c index bc6bddd156ca..df47fc21b0e4 100644 --- a/mm/mprotect.c +++ b/mm/mprotect.c @@ -621,6 +621,7 @@ mprotect_fixup(struct mmu_gather *tlb, struct vm_area_s= truct *vma, * vm_flags and vm_page_prot are protected by the mmap_lock * held in write mode. */ + vma_mark_locked(vma); vma->vm_flags =3D newflags; /* * We want to check manually if we can change individual PTEs writable --=20 2.37.2.672.g94769d06f0-goog From nobody Tue Apr 7 09:06:15 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2A21EECAAD2 for ; Mon, 29 Aug 2022 21:26:29 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229730AbiH2V00 (ORCPT ); Mon, 29 Aug 2022 17:26:26 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:32922 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229800AbiH2V0K (ORCPT ); Mon, 29 Aug 2022 17:26:10 -0400 Received: from mail-pj1-x104a.google.com (mail-pj1-x104a.google.com [IPv6:2607:f8b0:4864:20::104a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4DF4870E60 for ; Mon, 29 Aug 2022 14:25:49 -0700 (PDT) Received: by mail-pj1-x104a.google.com with SMTP id x8-20020a17090a1f8800b001faa9857ef2so3894749pja.0 for ; Mon, 29 Aug 2022 14:25:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc; bh=knlkft8CSs4ctk+Wf5EGoFcT9IapAOL4+B6Ai/+T1Ew=; b=P37RAGhPa0fj1eu5oWQ75AvAaLtugohQsYRWKzKsYPy82nTGbYS2huH/UiZNVvoE/s 88nyZ5hMCHsleX96EzwxSRzBq/7CUKc/bHbJ31Ts1GLCvmDOU9CmcQD2WQXDTk1c+h7j 3oIUBNbln4Bxomo0AKLbatKVQNVrzgPz+A4aQoGdDS5LbHfkkFmAcE6pV6uk/ymeBjbJ 1nc4oAeG1urj8/jUSameTDeH8ziLxBTZVHP9LQuagPlJAxjluMekWoC6NIdC1YJBWEbt nn2+QyaYYhDKNwmBnD8p146GgfwAnKteHIAkGMz5qwPGMxUQt1Cd0qMSAU4DAqNB0mVB aVgw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc; bh=knlkft8CSs4ctk+Wf5EGoFcT9IapAOL4+B6Ai/+T1Ew=; b=hY5S5bMd/JZYq9RE32ihPedbkWPG7PCCRMym9mdGI4Uci7RTcqJtbERTxylkmtSKF8 WAR3wklTTrgWrQ4fzg6N30Zqe72kLTpEUtQff3Yne/0pl6wCVhmW8yMaAlBlHrLxuBf/ PYGkNypsAN8cgshtSA7Cscn4U3VIu6sYrYABkUclXKn0gYgtCmh4c98tYkYMdhwaMriT e/EbWSm1mSftx9rzPhlJMf0r2XqBd1+lK5S6AZ8UpX80d7+g1K9lqIysRefdxPa3ftv4 +/2e+RzS1rh8BJo11Cd2F8uUYiU0fikYuBT/tllcWYTh3Z/Oo1BhrpCt29M11VohPhLp 0kXA== X-Gm-Message-State: ACgBeo2g4G0mfzNvDld+iHgc23k7Nv3Rb3cmh84MNGym5QCFX6CJOu7x 8MFfXlT4sY5lhnaz6HhkDeWEye179Tc= X-Google-Smtp-Source: AA6agR7ucJtBhSAnuB+PqV9qzppZFKeEkxBUzS6um9UDd2r/XYBs5DAVo1iLh4VL9E0CKz9mlSvmLd+h5RQ= X-Received: from surenb-spec.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:3e30]) (user=surenb job=sendgmr) by 2002:aa7:8d08:0:b0:538:105a:eb6a with SMTP id j8-20020aa78d08000000b00538105aeb6amr10112054pfe.42.1661808348114; Mon, 29 Aug 2022 14:25:48 -0700 (PDT) Date: Mon, 29 Aug 2022 21:25:10 +0000 In-Reply-To: <20220829212531.3184856-1-surenb@google.com> Mime-Version: 1.0 References: <20220829212531.3184856-1-surenb@google.com> X-Mailer: git-send-email 2.37.2.672.g94769d06f0-goog Message-ID: <20220829212531.3184856-8-surenb@google.com> Subject: [RFC PATCH 07/28] kernel/fork: mark VMAs as locked before copying pages during fork From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, riel@surriel.com, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, surenb@google.com, kernel-team@android.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Protect VMAs from concurrent page fault handler while performing copy_page_range for VMAs having VM_WIPEONFORK flag set. Signed-off-by: Suren Baghdasaryan --- kernel/fork.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/kernel/fork.c b/kernel/fork.c index bfab31ecd11e..1872ad549fed 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -709,8 +709,10 @@ static __latent_entropy int dup_mmap(struct mm_struct = *mm, rb_parent =3D &tmp->vm_rb; =20 mm->map_count++; - if (!(tmp->vm_flags & VM_WIPEONFORK)) + if (!(tmp->vm_flags & VM_WIPEONFORK)) { + vma_mark_locked(mpnt); retval =3D copy_page_range(tmp, mpnt); + } =20 if (tmp->vm_ops && tmp->vm_ops->open) tmp->vm_ops->open(tmp); --=20 2.37.2.672.g94769d06f0-goog From nobody Tue Apr 7 09:06:15 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id C5E7EECAAD4 for ; Mon, 29 Aug 2022 21:26:33 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229829AbiH2V0b (ORCPT ); Mon, 29 Aug 2022 17:26:31 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:32838 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229449AbiH2V0N (ORCPT ); Mon, 29 Aug 2022 17:26:13 -0400 Received: from mail-pf1-x449.google.com (mail-pf1-x449.google.com [IPv6:2607:f8b0:4864:20::449]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id BCE5874B96 for ; Mon, 29 Aug 2022 14:25:51 -0700 (PDT) Received: by mail-pf1-x449.google.com with SMTP id 125-20020a621483000000b0053814ac4b8bso1875870pfu.16 for ; Mon, 29 Aug 2022 14:25:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc; bh=ooNZAin4dExMhxnzVFEqRWXW+87JR2sLE0fevcXKZcM=; b=I4AROHhc91xv1qcH3nCi9ZpRLfIZ0YYOknUuCx0UCVegrJDEOhp2xXx6sXe9aCLn1C FzO21jqONtvqpljlp8Pb4Y+CdSx9zRBCbsYcOU1CMA6tT/zAnvmRorbY0bmK2T02uMkI Atv+IFZLAVnvs+tVzbF09RqlClFEsd0HUKtTSkQ+YRbpf3d4jB4ZfuXvxVp5TLyaLLyx AxdLnxV0cUJcHwzUd+Lxky4P92oXj4Xhl4or9fJrlsG5GoIvzarsCtdrQ2b29IJKR80y NIBWQdZUPTZNVXZAtYrsTzYsNpAhtyZg3hqM9pNuOBs9nY5UdbnMfXBZJ7yLdA5TAp1d MxBA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc; bh=ooNZAin4dExMhxnzVFEqRWXW+87JR2sLE0fevcXKZcM=; b=nRixvNdDUbgvxI/59c0ksaZZh8aXERwYbz4uRgcusa62rZxma/LuKlReAZGUE1XjvJ GGdvVFvt+eMmmDqDyEoVJSzpq4Bni8A9DcwwTZCS2Vm31hb4jH2YaiTQJECwFR2DW00U h3VY5i0yP2znJQJ+95nqM8MuQw25KBzGj2Rmfn6wV9rJfShWHV7OKYNEjX7Vw9NUI9yH kaIk8CHU04RBBFBhUHd8NmPbhd2ewrj9g0eCISL1ugNn6UZFsx0Dsa/AxzQ8iKKoVT2h J7nS9oDRte4Cf5W//WAZuGNrFUMhutPsTYuMjwFWKKAWGvdGRb/T5ihDVoeN7jeshJb1 XAzg== X-Gm-Message-State: ACgBeo3TzOsfZIwUjsbnSt9vnMOJecOyYdr1cZAT9VrP7IvDNKlQzAsJ iZjoocg6wsDydyc92BRPsRnnbHE30lM= X-Google-Smtp-Source: AA6agR6Ch+8H2P4ZiI/nyVX9BprKXIg4wSUP+mENwsodRVHrGORNiLr4bLOGgyZYxzbEnnQphTeE0ezRPcY= X-Received: from surenb-spec.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:3e30]) (user=surenb job=sendgmr) by 2002:a17:90a:4607:b0:1fa:f794:2c0d with SMTP id w7-20020a17090a460700b001faf7942c0dmr20939608pjg.24.1661808349897; Mon, 29 Aug 2022 14:25:49 -0700 (PDT) Date: Mon, 29 Aug 2022 21:25:11 +0000 In-Reply-To: <20220829212531.3184856-1-surenb@google.com> Mime-Version: 1.0 References: <20220829212531.3184856-1-surenb@google.com> X-Mailer: git-send-email 2.37.2.672.g94769d06f0-goog Message-ID: <20220829212531.3184856-9-surenb@google.com> Subject: [RFC PATCH 08/28] mm/khugepaged: mark VMA as locked while collapsing a hugepage From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, riel@surriel.com, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, surenb@google.com, kernel-team@android.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Protect VMA from concurrent page fault handler while modifying it in collapse_huge_page. Signed-off-by: Suren Baghdasaryan --- mm/khugepaged.c | 1 + 1 file changed, 1 insertion(+) diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 01f71786d530..030680633989 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -1072,6 +1072,7 @@ static void collapse_huge_page(struct mm_struct *mm, if (mm_find_pmd(mm, address) !=3D pmd) goto out_up_write; =20 + vma_mark_locked(vma); anon_vma_lock_write(vma->anon_vma); =20 mmu_notifier_range_init(&range, MMU_NOTIFY_CLEAR, 0, NULL, mm, --=20 2.37.2.672.g94769d06f0-goog From nobody Tue Apr 7 09:06:15 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id D6E2EECAAD2 for ; Mon, 29 Aug 2022 21:26:38 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229917AbiH2V0g (ORCPT ); Mon, 29 Aug 2022 17:26:36 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:32860 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229826AbiH2V0P (ORCPT ); Mon, 29 Aug 2022 17:26:15 -0400 Received: from mail-pj1-x1049.google.com (mail-pj1-x1049.google.com [IPv6:2607:f8b0:4864:20::1049]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 11F3E6EF11 for ; Mon, 29 Aug 2022 14:25:52 -0700 (PDT) Received: by mail-pj1-x1049.google.com with SMTP id 92-20020a17090a09e500b001d917022847so3728626pjo.1 for ; Mon, 29 Aug 2022 14:25:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc; bh=tMRmbcwOZDIFaTPxLY27tBFqvxpOSXiDeb2XTcL9iTc=; b=c5Ge838GLG1Fn9WCiifxSW0yY2J286Z3nb7CHVsgQeKl819oavnaneDjs0pHM1pBgc 6/VXxA27srsSSYsH3q/5JQwqbG7gvhz/dxlqpfYYnChp/oSAOK1144eGZRxL5H4xfU4+ gXjPirwbsBHls6abdkbd/Ru5neoY+5POLJLE67tmQyRoTq3+UExyJwB4xZoCc8lFIPHT 1JLKkVoSr3hMYFs+x98zjjRcyw2xrS1rGfAXHEq+RyZfpGyhDpyGAbYsGbpBS3qHA7OY gUezPbUinXA9UrR8RhJyAbOsAhf2VqCoI/O3T8n1+HeqEuYCNoxrQ8z3jvR4TT/rnWyA atPg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc; bh=tMRmbcwOZDIFaTPxLY27tBFqvxpOSXiDeb2XTcL9iTc=; b=WIUSLTwVEHYT0+Jviji8lyFug0u5mvHgjPSDqRHNXPXEfN+0/7ZqPLNXWJ5CAQX6NP 2vGXH3lecPomVFQR+ObmUZFy9+vKvykol2eAVIa5r0o0A69vTkO3MHlWA+hMbC+qMmMw hXNX79bRrt7cG4njvWGkq8pf0jkO+A0+DVKQrwFE4lOWFYAJdBa7LXdn12Wc6+FymxGi JcYxUuuPSSKif8eDDY+g6S8XT+yqL3G8jPWWvHiWCvz0nhn5HbP+qTHd9BFQYz4BRu4o McahDhNKQen4b89UoVfco1mcW/fYtVNYBUygn6pKbRCEEsTmeApTvYQq8chqqgAkZ8Ke WmWg== X-Gm-Message-State: ACgBeo0ZTkR5yWD+ZTR9YY+jKYDStAlOGEFerZGJwmHOvE/CK2xLJhj2 YcVRAwioMFlCcbCFsbx2O9rXAdhD5iE= X-Google-Smtp-Source: AA6agR6WFXWG6lBERn8fKgbng4pdY7+dFZ275Tti+4gL6MGpsob4z29fC0owAPSMM8vQ0mXhvHK4dQ+aJ1U= X-Received: from surenb-spec.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:3e30]) (user=surenb job=sendgmr) by 2002:a17:902:ab08:b0:173:320e:3316 with SMTP id ik8-20020a170902ab0800b00173320e3316mr17541232plb.38.1661808351535; Mon, 29 Aug 2022 14:25:51 -0700 (PDT) Date: Mon, 29 Aug 2022 21:25:12 +0000 In-Reply-To: <20220829212531.3184856-1-surenb@google.com> Mime-Version: 1.0 References: <20220829212531.3184856-1-surenb@google.com> X-Mailer: git-send-email 2.37.2.672.g94769d06f0-goog Message-ID: <20220829212531.3184856-10-surenb@google.com> Subject: [RFC PATCH 09/28] mm/mempolicy: mark VMA as locked when changing protection policy From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, riel@surriel.com, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, surenb@google.com, kernel-team@android.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Protect VMA from concurrent page fault handler while performing VMA protection policy changes. Signed-off-by: Suren Baghdasaryan --- mm/mempolicy.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/mm/mempolicy.c b/mm/mempolicy.c index b73d3248d976..6be1e5c75556 100644 --- a/mm/mempolicy.c +++ b/mm/mempolicy.c @@ -383,8 +383,10 @@ void mpol_rebind_mm(struct mm_struct *mm, nodemask_t *= new) struct vm_area_struct *vma; =20 mmap_write_lock(mm); - for (vma =3D mm->mmap; vma; vma =3D vma->vm_next) + for (vma =3D mm->mmap; vma; vma =3D vma->vm_next) { + vma_mark_locked(vma); mpol_rebind_policy(vma->vm_policy, new); + } mmap_write_unlock(mm); } =20 @@ -632,6 +634,7 @@ unsigned long change_prot_numa(struct vm_area_struct *v= ma, struct mmu_gather tlb; int nr_updated; =20 + vma_mark_locked(vma); tlb_gather_mmu(&tlb, vma->vm_mm); =20 nr_updated =3D change_protection(&tlb, vma, addr, end, PAGE_NONE, @@ -765,6 +768,7 @@ static int vma_replace_policy(struct vm_area_struct *vm= a, if (IS_ERR(new)) return PTR_ERR(new); =20 + vma_mark_locked(vma); if (vma->vm_ops && vma->vm_ops->set_policy) { err =3D vma->vm_ops->set_policy(vma, new); if (err) --=20 2.37.2.672.g94769d06f0-goog From nobody Tue Apr 7 09:06:15 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9C7DBECAAD2 for ; Mon, 29 Aug 2022 21:26:43 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229923AbiH2V0m (ORCPT ); Mon, 29 Aug 2022 17:26:42 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33370 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229843AbiH2V0R (ORCPT ); Mon, 29 Aug 2022 17:26:17 -0400 Received: from mail-pl1-x649.google.com (mail-pl1-x649.google.com [IPv6:2607:f8b0:4864:20::649]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9F8385AC7E for ; Mon, 29 Aug 2022 14:25:56 -0700 (PDT) Received: by mail-pl1-x649.google.com with SMTP id y9-20020a17090322c900b00174c881abaeso2531249plg.6 for ; Mon, 29 Aug 2022 14:25:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc; bh=nWFFKiz2JrcFJbZNC6h3WaeWKSGRlWYuGKA4mg+JUNY=; b=PCjrqm9NhycPwB3kYWPhGmirug2EAINieO21R1tkNllFgsambVeodQV2SWE+SPKnLk B35opb3MQlODsxNphVo3gt2yyCr1QpWXQKLcx1/vKzcN/4STqp6cRFJdHuKeBk+NsfTT tDOTylgaSdZ5nEY1ymyrW4L9evpwJBVsw8WgcZFUheAteEM8kYWXWY1xc4VUjrsXgS7n TvpEAgLiNbXgMlGNPVj+MFOM++dtX+BoFzGnROjLhFZB70sHOrsgtuWSCJIJNO8R1S+2 S3QUIvJ5IyCicdMqNDVbeJ892T5I6yDVEmvFVjdkZe2DnjLJIY2H8E2Qt303ElEJo4Tc HSMQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc; bh=nWFFKiz2JrcFJbZNC6h3WaeWKSGRlWYuGKA4mg+JUNY=; b=c14P+UMmU6epo/yay91QTvQ/VSrvIdMQsuGha5IlMmVsY1Q4GVwe21/mqXW9XrqoA0 cDnvBKKNwdpZArBZJmXgVQ6fK4SQTS1IpFU5kEfKNxfqt4d8qLpG/l+h87olxxktTYh1 xmm454LVNbonaz6qZCrJMXSH+dO/F8KbJ65frBtEo3+w96AEh0Vf7hidoDkpdlTC2asA atJoe5ixaFJVmknKWRRHUzpAtp5Qarct7GsfN/gg1BIrBznsXlJGKIZaC6b4RcJ3kBVL hqs10K0oeRbxqQ9ViYMj2tcc88rw1uZvU8u7QhNB8J2EexQhaX6KfY/qVceMKPg+YE6q dZ6g== X-Gm-Message-State: ACgBeo2MROoQ6Bui0mEpsjBAYI3rqw8C61/775K41bpFp9iyvVn/vxi7 uqBPTR47EXujXh2EL+/3qdA7beyoOeI= X-Google-Smtp-Source: AA6agR4dglPLGzQSnoqHwxEZwo0Lk0EwUGiOTYsdKQuDkqIVlV54iwLYmhw7nxi8GDICS/TsbTVUTOyC/V4= X-Received: from surenb-spec.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:3e30]) (user=surenb job=sendgmr) by 2002:a17:90a:249:b0:1e0:a8a3:3c6c with SMTP id t9-20020a17090a024900b001e0a8a33c6cmr21107pje.0.1661808353487; Mon, 29 Aug 2022 14:25:53 -0700 (PDT) Date: Mon, 29 Aug 2022 21:25:13 +0000 In-Reply-To: <20220829212531.3184856-1-surenb@google.com> Mime-Version: 1.0 References: <20220829212531.3184856-1-surenb@google.com> X-Mailer: git-send-email 2.37.2.672.g94769d06f0-goog Message-ID: <20220829212531.3184856-11-surenb@google.com> Subject: [RFC PATCH 10/28] mm/mmap: mark VMAs as locked in vma_adjust From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, riel@surriel.com, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, surenb@google.com, kernel-team@android.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" vma_adjust modifies a VMA and possibly its neighbors. Mark them as locked before making the modifications. Signed-off-by: Suren Baghdasaryan --- mm/mmap.c | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/mm/mmap.c b/mm/mmap.c index f89c9b058105..ed58cf0689b2 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -710,6 +710,10 @@ int __vma_adjust(struct vm_area_struct *vma, unsigned = long start, long adjust_next =3D 0; int remove_next =3D 0; =20 + vma_mark_locked(vma); + if (next) + vma_mark_locked(next); + if (next && !insert) { struct vm_area_struct *exporter =3D NULL, *importer =3D NULL; =20 @@ -754,8 +758,11 @@ int __vma_adjust(struct vm_area_struct *vma, unsigned = long start, * If next doesn't have anon_vma, import from vma after * next, if the vma overlaps with it. */ - if (remove_next =3D=3D 2 && !next->anon_vma) + if (remove_next =3D=3D 2 && !next->anon_vma) { exporter =3D next->vm_next; + if (exporter) + vma_mark_locked(exporter); + } =20 } else if (end > next->vm_start) { /* @@ -931,6 +938,8 @@ int __vma_adjust(struct vm_area_struct *vma, unsigned l= ong start, * "vma->vm_next" gap must be updated. */ next =3D vma->vm_next; + if (next) + vma_mark_locked(next); } else { /* * For the scope of the comment "next" and --=20 2.37.2.672.g94769d06f0-goog From nobody Tue Apr 7 09:06:15 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 41150ECAAD2 for ; Mon, 29 Aug 2022 21:26:54 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229959AbiH2V0v (ORCPT ); Mon, 29 Aug 2022 17:26:51 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:32794 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229881AbiH2V03 (ORCPT ); Mon, 29 Aug 2022 17:26:29 -0400 Received: from mail-pj1-x104a.google.com (mail-pj1-x104a.google.com [IPv6:2607:f8b0:4864:20::104a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7BCD8870A0 for ; Mon, 29 Aug 2022 14:25:58 -0700 (PDT) Received: by mail-pj1-x104a.google.com with SMTP id e11-20020a17090a630b00b001f8b2deb88dso10114750pjj.1 for ; Mon, 29 Aug 2022 14:25:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc; bh=iuFA+HN3YQAhGF4lKe/pszwBtaIFpTgsToRqXaNjG7k=; b=OQFmY5p6Pe2rtWh8HoOtibHP8P5GG7l2q0OSc7HNXnd2wSmQX9hqgXeo/WRhgSbde1 3GOMVRznNWUQpWpM6oRymYxlQgfSmhokZzxHMaOud6hAHMI7hRSvRQgnpTP2D9VCRijV tvYOdRbm6MoDu4UgOmxt7sejA6CQywedGjOTRBl30ijbrDgQETYn5oZOqnGCtaAYcPZu MjsJRXAG04ApiH5Z6mU8Nc2C+hXCaIEB6zH7hUa5Up7nSwAgqcd3I/44d6H2f140ISuN zqINoaSTwBSBF/Nj+3hF7XpzbCbRiSR9QvteChaGaDG9beNmWGZas127F5+VFT8rM/v5 R54A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc; bh=iuFA+HN3YQAhGF4lKe/pszwBtaIFpTgsToRqXaNjG7k=; b=u/uylYtf8O8l9e2tYsxOIICBD831cj0Mr+VuPREElkRd5WgHSt4F8SYUk43taoUBqm 0vSTrpSZhC9Bc4PxN78WIAz8MH3Y9qzpfpfV2c+oJ/2sdpmkgIEELxqzO4n2xCkOG4Kt f9pX/2h1/Olm45DVhet9v+r6PqxYNM26I/iByF+giiGixYkANadeaM5ZC7k/SvLFAS1s 4NNTlg9kg854xFrfc0Bx/OInt7QxNjl+SEzmkWETKCaWgVpJrdlxSl25VRLYYCw7d5P3 gubfo2DPo0bfcJH9lI70wFdTZcKMaM39+GRZuxlFcUeJZOvOP/HZ5ufd1TV4Jmsr272h N70Q== X-Gm-Message-State: ACgBeo0NILJutH+bdlTPj5U2pY1QBtiq0Cjd19jWeyhQrheZ1mCA6xyu hDKOwckSaMozu31SdJ1T0q6NgsQAx4g= X-Google-Smtp-Source: AA6agR67llFT6eQ0+3Cyyc4Z2yCAV5KwAp4HusV+mlw9qyUeJ6m9H2aBfcEzix7gu6JpzsX42cg/TQ27QPw= X-Received: from surenb-spec.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:3e30]) (user=surenb job=sendgmr) by 2002:a17:90a:e558:b0:1fb:c4b7:1a24 with SMTP id ei24-20020a17090ae55800b001fbc4b71a24mr21023pjb.1.1661808355887; Mon, 29 Aug 2022 14:25:55 -0700 (PDT) Date: Mon, 29 Aug 2022 21:25:14 +0000 In-Reply-To: <20220829212531.3184856-1-surenb@google.com> Mime-Version: 1.0 References: <20220829212531.3184856-1-surenb@google.com> X-Mailer: git-send-email 2.37.2.672.g94769d06f0-goog Message-ID: <20220829212531.3184856-12-surenb@google.com> Subject: [RFC PATCH 11/28] mm/mmap: mark VMAs as locked before merging or splitting them From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, riel@surriel.com, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, surenb@google.com, kernel-team@android.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Decisions about whether VMAs can be merged or split must be made while VMAs are protected from the changes which can affect that decision. For example, merge_vma uses vma->anon_vma in its decision whether the VMA can be merged. Meanwhile, page fault handler changes vma->anon_vma during COW operation. Mark all VMAs which might be affected by a merge or split operation as locked before making decision how such operations should be performed. Signed-off-by: Suren Baghdasaryan --- mm/mmap.c | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/mm/mmap.c b/mm/mmap.c index ed58cf0689b2..ade3909c89b4 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -1147,10 +1147,17 @@ struct vm_area_struct *vma_merge(struct mm_struct *= mm, if (vm_flags & VM_SPECIAL) return NULL; =20 + if (prev) + vma_mark_locked(prev); next =3D vma_next(mm, prev); area =3D next; - if (area && area->vm_end =3D=3D end) /* cases 6, 7, 8 */ + if (area) + vma_mark_locked(area); + if (area && area->vm_end =3D=3D end) { /* cases 6, 7, 8 */ next =3D next->vm_next; + if (next) + vma_mark_locked(next); + } =20 /* verify some invariant that must be enforced by the caller */ VM_WARN_ON(prev && addr <=3D prev->vm_start); @@ -2687,6 +2694,7 @@ int __split_vma(struct mm_struct *mm, struct vm_area_= struct *vma, struct vm_area_struct *new; int err; =20 + vma_mark_locked(vma); if (vma->vm_ops && vma->vm_ops->may_split) { err =3D vma->vm_ops->may_split(vma, addr); if (err) --=20 2.37.2.672.g94769d06f0-goog From nobody Tue Apr 7 09:06:15 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id B2172ECAAD2 for ; Mon, 29 Aug 2022 21:26:50 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229949AbiH2V0s (ORCPT ); Mon, 29 Aug 2022 17:26:48 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:32840 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229873AbiH2V0U (ORCPT ); Mon, 29 Aug 2022 17:26:20 -0400 Received: from mail-pl1-x649.google.com (mail-pl1-x649.google.com [IPv6:2607:f8b0:4864:20::649]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 19B5E8C470 for ; Mon, 29 Aug 2022 14:25:59 -0700 (PDT) Received: by mail-pl1-x649.google.com with SMTP id m5-20020a170902f64500b0016d313f3ce7so6864671plg.23 for ; Mon, 29 Aug 2022 14:25:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc; bh=TWnM2nl+IWV2Xsj5xnZFMNMq0JvYeydCw4P9njL7O34=; b=QxyTYU3FERVCw2VziTt0AktgHLl//+1LB6ej8pZ2s1TWfQT3cvsAWfnuU7caTHeT9p cUZ3ty459UM3cwcXwNVqpzu7fZxDdvEiQ7oppDRkd/R2+Ilj86/8iy+0q3ZJXoZ4hnye zLhHEgIO/fbjIHEiSfoXxsUyX6O+Z8dic5XVruk2KEpor5fjgGmf4uNln1+WOsbYbTO8 1++i5QRJ3Dh2yxyVyOmY5Dm8Y1A3ze3Xt4uhd4NBeKcnHbK+ENDvs3AO4SaOBTp9Wi+f EGkkrDdrwisqruuTTVNnHMbKcZefZFlW3jnAC7mX9edrYOzqBxLpV3pgyWig/2c0mbAr sZlg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc; bh=TWnM2nl+IWV2Xsj5xnZFMNMq0JvYeydCw4P9njL7O34=; b=qfOf+APAbzS0pxkYmRjhl2XSWCmGK0n7PHkNW5n/V1FRvkRcHvSxVe/pp0s3XfjCbO UAV+XV36Kpfnxi9ZLwqpJ9XGWBRzqtfyRbFRvswHmS4tyZCuaEvvNHpB/GaqxSDbhEE+ P7FSt+qQA0YgqICg+Rx1lvvpFux6EgrYbuGmOK7Besc6WAUVBSZ+juFBgAmGWw5qCUv1 YQKUWcLvnLfZHAqbHKEkqxUIdLtq1Jg7uyuWDMYPngVQDYqs68euL3ZHPtQnUddeAxqM oFS73/zAcCJe+ZL1mVKUjBx+Vifkgr496qnItJItD1TaCYQDfgS9EGSqRyUYLMreuK11 O1dQ== X-Gm-Message-State: ACgBeo2Ka2o95WGiO3F/6X/U9ZaCTmq9ya/ZlszOUctY+CzEojWxP7xp NUgnfN+CftnjswHku7/52TcBEMmRp3A= X-Google-Smtp-Source: AA6agR76cA5Bw6pPdfILXZxKwAftwtVJj++YaWYUCk6VRMlDSa2HJ+OTsxxi3Jretb4HksKEr9YwTk9TbU4= X-Received: from surenb-spec.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:3e30]) (user=surenb job=sendgmr) by 2002:a17:903:18b:b0:16e:f09c:919b with SMTP id z11-20020a170903018b00b0016ef09c919bmr18604991plg.135.1661808358173; Mon, 29 Aug 2022 14:25:58 -0700 (PDT) Date: Mon, 29 Aug 2022 21:25:15 +0000 In-Reply-To: <20220829212531.3184856-1-surenb@google.com> Mime-Version: 1.0 References: <20220829212531.3184856-1-surenb@google.com> X-Mailer: git-send-email 2.37.2.672.g94769d06f0-goog Message-ID: <20220829212531.3184856-13-surenb@google.com> Subject: [RFC PATCH 12/28] mm/mremap: mark VMA as locked while remapping it to a new address range From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, riel@surriel.com, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, surenb@google.com, kernel-team@android.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Mark VMA as locked before copying it and when copy_vma produces a new VMA. Signed-off-by: Suren Baghdasaryan --- mm/mmap.c | 1 + mm/mremap.c | 1 + 2 files changed, 2 insertions(+) diff --git a/mm/mmap.c b/mm/mmap.c index ade3909c89b4..121544fd90de 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -3248,6 +3248,7 @@ struct vm_area_struct *copy_vma(struct vm_area_struct= **vmap, get_file(new_vma->vm_file); if (new_vma->vm_ops && new_vma->vm_ops->open) new_vma->vm_ops->open(new_vma); + vma_mark_locked(new_vma); vma_link(mm, new_vma, prev, rb_link, rb_parent); *need_rmap_locks =3D false; } diff --git a/mm/mremap.c b/mm/mremap.c index b522cd0259a0..bdbf96254e43 100644 --- a/mm/mremap.c +++ b/mm/mremap.c @@ -620,6 +620,7 @@ static unsigned long move_vma(struct vm_area_struct *vm= a, return -ENOMEM; } =20 + vma_mark_locked(vma); new_pgoff =3D vma->vm_pgoff + ((old_addr - vma->vm_start) >> PAGE_SHIFT); new_vma =3D copy_vma(&vma, new_addr, new_len, new_pgoff, &need_rmap_locks); --=20 2.37.2.672.g94769d06f0-goog From nobody Tue Apr 7 09:06:15 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8A7E6ECAAD4 for ; Mon, 29 Aug 2022 21:26:57 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229962AbiH2V0z (ORCPT ); Mon, 29 Aug 2022 17:26:55 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:32866 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229886AbiH2V0a (ORCPT ); Mon, 29 Aug 2022 17:26:30 -0400 Received: from mail-pf1-x449.google.com (mail-pf1-x449.google.com [IPv6:2607:f8b0:4864:20::449]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 5831591D01 for ; Mon, 29 Aug 2022 14:26:00 -0700 (PDT) Received: by mail-pf1-x449.google.com with SMTP id dc10-20020a056a0035ca00b0053870674be9so756906pfb.12 for ; Mon, 29 Aug 2022 14:26:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc; bh=i5dTvRyWQzPS3ycEVnnedNfB+iHCFx9RUaZKzBMQ1fM=; b=OIPy78dmVL2e9cDtVRNWLABLgvKsSbZL8TpHXpGFOXD1iGGPPAnyGMcNCguJqgzcNF vE+9cNpICzTMp48ey6JmMRyIlEctkJFDoL9MrqQuHHLDpAc2aXZC9NHJQw5sThX00eJ4 cjI0k3jAy5gFEX5VEHv4cVfUWBvRtfOC3iZWstqSQy3ibplhtgaSEpWxPcs9EVcm88qg LMXuGuQRDlQwRuQ7YNFSCH+8XULwWBK9BYSC5nbMXIDpOIovOb1rbCvinLUjcMpntl6K B3Pgew8zey/TtmyxMcXnTuT0BjzA6VIn4rP9dbnKnx4PgoHjYUplxPVEMwf3o1trOT0a 21qg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc; bh=i5dTvRyWQzPS3ycEVnnedNfB+iHCFx9RUaZKzBMQ1fM=; b=fCwnfPlM9TCFEmHVd0/zlh9jJu98NSi/YVwhwZy6nbNVG15kX0t0wut+YAQggMu6TM AwO0JSKrKpCi4wDMT/q0kiqgjBXJvGSHG/KP821EiFPCBMW7KtvI9RQ5nx3w2ib1hX5+ I5RNiu5jAo34YUHCjb41Sq7oqL4OzljgUFJwwFa0QhpOMp/wcrB0OSn/H46rVi+G9Nz0 YSobgnPaL5Q1MbmxZpgKg0mTHbPbmLIt3AUD2/sUFPt7K77wIN4RT0YYOOnkd2Nx74aS AIjJOZNKfSOayZu0/H8Cy/o5vYdU1XundLe6QxYJPDt96+7ji6tIDOfjj4ULPZxjhk+r P8Gg== X-Gm-Message-State: ACgBeo1360aT0vOSB8FT98JV9TX0z8s/1U6Jur6/sDUCP80WovQUlS5I 7lv8Voqhvi0Lu6pfyVLqPA0xMcWWgZU= X-Google-Smtp-Source: AA6agR4q+ezvBgIOmovwk/KulEVJwqDq2XJXfw+za6wVj0A/SjlqYDTJGsWkQl5JJSuDpwxpjjn7kfBBqpA= X-Received: from surenb-spec.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:3e30]) (user=surenb job=sendgmr) by 2002:a17:902:76c5:b0:174:e586:f6e2 with SMTP id j5-20020a17090276c500b00174e586f6e2mr4373889plt.37.1661808359839; Mon, 29 Aug 2022 14:25:59 -0700 (PDT) Date: Mon, 29 Aug 2022 21:25:16 +0000 In-Reply-To: <20220829212531.3184856-1-surenb@google.com> Mime-Version: 1.0 References: <20220829212531.3184856-1-surenb@google.com> X-Mailer: git-send-email 2.37.2.672.g94769d06f0-goog Message-ID: <20220829212531.3184856-14-surenb@google.com> Subject: [RFC PATCH 13/28] mm: conditionally mark VMA as locked in free_pgtables and unmap_page_range From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, riel@surriel.com, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, surenb@google.com, kernel-team@android.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" free_pgtables and unmap_page_range functions can be called with mmap_lock held for write (e.g. in mmap_region), held for read (e.g in madvise_pageout) or not held at all (e.g in madvise_remove might drop mmap_lock before calling vfs_fallocate, which ends up calling unmap_page_range). Provide free_pgtables and unmap_page_range with additional argument indicating whether to mark the VMA as locked or not based on the usage. The parameter is set based on whether mmap_lock is held in write mode during the call. This ensures no change in behavior between mmap_lock and per-vma locks. Signed-off-by: Suren Baghdasaryan --- include/linux/mm.h | 2 +- mm/internal.h | 4 ++-- mm/memory.c | 32 +++++++++++++++++++++----------- mm/mmap.c | 17 +++++++++-------- mm/oom_kill.c | 3 ++- 5 files changed, 35 insertions(+), 23 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 476bf936c5f0..dc72be923e5b 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -1874,7 +1874,7 @@ void zap_vma_ptes(struct vm_area_struct *vma, unsigne= d long address, void zap_page_range(struct vm_area_struct *vma, unsigned long address, unsigned long size); void unmap_vmas(struct mmu_gather *tlb, struct vm_area_struct *start_vma, - unsigned long start, unsigned long end); + unsigned long start, unsigned long end, bool lock_vma); =20 struct mmu_notifier_range; =20 diff --git a/mm/internal.h b/mm/internal.h index 785409805ed7..e6c0f999e0cb 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -85,14 +85,14 @@ bool __folio_end_writeback(struct folio *folio); void deactivate_file_folio(struct folio *folio); =20 void free_pgtables(struct mmu_gather *tlb, struct vm_area_struct *start_vm= a, - unsigned long floor, unsigned long ceiling); + unsigned long floor, unsigned long ceiling, bool lock_vma); void pmd_install(struct mm_struct *mm, pmd_t *pmd, pgtable_t *pte); =20 struct zap_details; void unmap_page_range(struct mmu_gather *tlb, struct vm_area_struct *vma, unsigned long addr, unsigned long end, - struct zap_details *details); + struct zap_details *details, bool lock_vma); =20 void page_cache_ra_order(struct readahead_control *, struct file_ra_state = *, unsigned int order); diff --git a/mm/memory.c b/mm/memory.c index 4ba73f5aa8bb..9ac9944e8c62 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -403,7 +403,7 @@ void free_pgd_range(struct mmu_gather *tlb, } =20 void free_pgtables(struct mmu_gather *tlb, struct vm_area_struct *vma, - unsigned long floor, unsigned long ceiling) + unsigned long floor, unsigned long ceiling, bool lock_vma) { while (vma) { struct vm_area_struct *next =3D vma->vm_next; @@ -413,6 +413,8 @@ void free_pgtables(struct mmu_gather *tlb, struct vm_ar= ea_struct *vma, * Hide vma from rmap and truncate_pagecache before freeing * pgtables */ + if (lock_vma) + vma_mark_locked(vma); unlink_anon_vmas(vma); unlink_file_vma(vma); =20 @@ -427,6 +429,8 @@ void free_pgtables(struct mmu_gather *tlb, struct vm_ar= ea_struct *vma, && !is_vm_hugetlb_page(next)) { vma =3D next; next =3D vma->vm_next; + if (lock_vma) + vma_mark_locked(vma); unlink_anon_vmas(vma); unlink_file_vma(vma); } @@ -1631,12 +1635,16 @@ static inline unsigned long zap_p4d_range(struct mm= u_gather *tlb, void unmap_page_range(struct mmu_gather *tlb, struct vm_area_struct *vma, unsigned long addr, unsigned long end, - struct zap_details *details) + struct zap_details *details, + bool lock_vma) { pgd_t *pgd; unsigned long next; =20 BUG_ON(addr >=3D end); + if (lock_vma) + vma_mark_locked(vma); + tlb_start_vma(tlb, vma); pgd =3D pgd_offset(vma->vm_mm, addr); do { @@ -1652,7 +1660,7 @@ void unmap_page_range(struct mmu_gather *tlb, static void unmap_single_vma(struct mmu_gather *tlb, struct vm_area_struct *vma, unsigned long start_addr, unsigned long end_addr, - struct zap_details *details) + struct zap_details *details, bool lock_vma) { unsigned long start =3D max(vma->vm_start, start_addr); unsigned long end; @@ -1691,7 +1699,7 @@ static void unmap_single_vma(struct mmu_gather *tlb, i_mmap_unlock_write(vma->vm_file->f_mapping); } } else - unmap_page_range(tlb, vma, start, end, details); + unmap_page_range(tlb, vma, start, end, details, lock_vma); } } =20 @@ -1715,7 +1723,7 @@ static void unmap_single_vma(struct mmu_gather *tlb, */ void unmap_vmas(struct mmu_gather *tlb, struct vm_area_struct *vma, unsigned long start_addr, - unsigned long end_addr) + unsigned long end_addr, bool lock_vma) { struct mmu_notifier_range range; struct zap_details details =3D { @@ -1728,7 +1736,8 @@ void unmap_vmas(struct mmu_gather *tlb, start_addr, end_addr); mmu_notifier_invalidate_range_start(&range); for ( ; vma && vma->vm_start < end_addr; vma =3D vma->vm_next) - unmap_single_vma(tlb, vma, start_addr, end_addr, &details); + unmap_single_vma(tlb, vma, start_addr, end_addr, &details, + lock_vma); mmu_notifier_invalidate_range_end(&range); } =20 @@ -1753,7 +1762,7 @@ void zap_page_range(struct vm_area_struct *vma, unsig= ned long start, update_hiwater_rss(vma->vm_mm); mmu_notifier_invalidate_range_start(&range); for ( ; vma && vma->vm_start < range.end; vma =3D vma->vm_next) - unmap_single_vma(&tlb, vma, start, range.end, NULL); + unmap_single_vma(&tlb, vma, start, range.end, NULL, false); mmu_notifier_invalidate_range_end(&range); tlb_finish_mmu(&tlb); } @@ -1768,7 +1777,7 @@ void zap_page_range(struct vm_area_struct *vma, unsig= ned long start, * The range must fit into one VMA. */ static void zap_page_range_single(struct vm_area_struct *vma, unsigned lon= g address, - unsigned long size, struct zap_details *details) + unsigned long size, struct zap_details *details, bool lock_vma) { struct mmu_notifier_range range; struct mmu_gather tlb; @@ -1779,7 +1788,7 @@ static void zap_page_range_single(struct vm_area_stru= ct *vma, unsigned long addr tlb_gather_mmu(&tlb, vma->vm_mm); update_hiwater_rss(vma->vm_mm); mmu_notifier_invalidate_range_start(&range); - unmap_single_vma(&tlb, vma, address, range.end, details); + unmap_single_vma(&tlb, vma, address, range.end, details, lock_vma); mmu_notifier_invalidate_range_end(&range); tlb_finish_mmu(&tlb); } @@ -1802,7 +1811,7 @@ void zap_vma_ptes(struct vm_area_struct *vma, unsigne= d long address, !(vma->vm_flags & VM_PFNMAP)) return; =20 - zap_page_range_single(vma, address, size, NULL); + zap_page_range_single(vma, address, size, NULL, true); } EXPORT_SYMBOL_GPL(zap_vma_ptes); =20 @@ -3483,7 +3492,8 @@ static void unmap_mapping_range_vma(struct vm_area_st= ruct *vma, unsigned long start_addr, unsigned long end_addr, struct zap_details *details) { - zap_page_range_single(vma, start_addr, end_addr - start_addr, details); + zap_page_range_single(vma, start_addr, end_addr - start_addr, details, + false); } =20 static inline void unmap_mapping_range_tree(struct rb_root_cached *root, diff --git a/mm/mmap.c b/mm/mmap.c index 121544fd90de..094678b4434b 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -79,7 +79,7 @@ core_param(ignore_rlimit_data, ignore_rlimit_data, bool, = 0644); =20 static void unmap_region(struct mm_struct *mm, struct vm_area_struct *vma, struct vm_area_struct *prev, - unsigned long start, unsigned long end); + unsigned long start, unsigned long end, bool lock_vma); =20 static pgprot_t vm_pgprot_modify(pgprot_t oldprot, unsigned long vm_flags) { @@ -1866,7 +1866,7 @@ unsigned long mmap_region(struct file *file, unsigned= long addr, vma->vm_file =3D NULL; =20 /* Undo any partial mapping done by a device driver. */ - unmap_region(mm, vma, prev, vma->vm_start, vma->vm_end); + unmap_region(mm, vma, prev, vma->vm_start, vma->vm_end, true); if (vm_flags & VM_SHARED) mapping_unmap_writable(file->f_mapping); free_vma: @@ -2626,7 +2626,7 @@ static void remove_vma_list(struct mm_struct *mm, str= uct vm_area_struct *vma) */ static void unmap_region(struct mm_struct *mm, struct vm_area_struct *vma, struct vm_area_struct *prev, - unsigned long start, unsigned long end) + unsigned long start, unsigned long end, bool lock_vma) { struct vm_area_struct *next =3D vma_next(mm, prev); struct mmu_gather tlb; @@ -2634,9 +2634,10 @@ static void unmap_region(struct mm_struct *mm, lru_add_drain(); tlb_gather_mmu(&tlb, mm); update_hiwater_rss(mm); - unmap_vmas(&tlb, vma, start, end); + unmap_vmas(&tlb, vma, start, end, lock_vma); free_pgtables(&tlb, vma, prev ? prev->vm_end : FIRST_USER_ADDRESS, - next ? next->vm_start : USER_PGTABLES_CEILING); + next ? next->vm_start : USER_PGTABLES_CEILING, + lock_vma); tlb_finish_mmu(&tlb); } =20 @@ -2849,7 +2850,7 @@ int __do_munmap(struct mm_struct *mm, unsigned long s= tart, size_t len, if (downgrade) mmap_write_downgrade(mm); =20 - unmap_region(mm, vma, prev, start, end); + unmap_region(mm, vma, prev, start, end, !downgrade); =20 /* Fix up all other VM information */ remove_vma_list(mm, vma); @@ -3129,8 +3130,8 @@ void exit_mmap(struct mm_struct *mm) tlb_gather_mmu_fullmm(&tlb, mm); /* update_hiwater_rss(mm) here? but nobody should be looking */ /* Use -1 here to ensure all VMAs in the mm are unmapped */ - unmap_vmas(&tlb, vma, 0, -1); - free_pgtables(&tlb, vma, FIRST_USER_ADDRESS, USER_PGTABLES_CEILING); + unmap_vmas(&tlb, vma, 0, -1, true); + free_pgtables(&tlb, vma, FIRST_USER_ADDRESS, USER_PGTABLES_CEILING, true); tlb_finish_mmu(&tlb); =20 /* Walk the list again, actually closing and freeing it. */ diff --git a/mm/oom_kill.c b/mm/oom_kill.c index 3c6cf9e3cd66..6ffa7c511aa3 100644 --- a/mm/oom_kill.c +++ b/mm/oom_kill.c @@ -549,7 +549,8 @@ bool __oom_reap_task_mm(struct mm_struct *mm) ret =3D false; continue; } - unmap_page_range(&tlb, vma, range.start, range.end, NULL); + unmap_page_range(&tlb, vma, range.start, range.end, + NULL, false); mmu_notifier_invalidate_range_end(&range); tlb_finish_mmu(&tlb); } --=20 2.37.2.672.g94769d06f0-goog From nobody Tue Apr 7 09:06:15 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 49177ECAAD4 for ; Mon, 29 Aug 2022 21:27:12 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229942AbiH2V1K (ORCPT ); Mon, 29 Aug 2022 17:27:10 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33004 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229927AbiH2V0m (ORCPT ); Mon, 29 Aug 2022 17:26:42 -0400 Received: from mail-pj1-x104a.google.com (mail-pj1-x104a.google.com [IPv6:2607:f8b0:4864:20::104a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A886798588 for ; Mon, 29 Aug 2022 14:26:02 -0700 (PDT) Received: by mail-pj1-x104a.google.com with SMTP id k1-20020a17090a658100b001fb35f86ccdso10084925pjj.9 for ; Mon, 29 Aug 2022 14:26:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc; bh=J1u/YkZ7r4U6qP3j3HXey3MGQlv66sCtBUAUQwGtkmo=; b=bssUgHF5w47sbr7l7KTrCUB9BNCgN6M5weE9wkVgXyWLQuTUzyGpg0OJvcyz0sW5UD jBcbuP4XXvP/J4pS6AqtYRXplk3ioZfRWOizdHfYubwqdaFV7iqu+wCAfN30wOXuSJR5 rOeSjLAnMTV/UMrEb+PillTsGibQBZHi92r8Zd7PRmmwTLk5upCdMt9/ApQU+Kr/zUNy g3HC4+2oFiondWWpmL/CQDHhjiEfoJFEyDpo5hDjW6J9yE1A4zjv6LFdAo5+gh9e4xsA 3B8ihRdAo9AsX2EZ1zty6Setu8LqB98WdsMEL3dnRqIhFnd/9nqnY8Jcva4fprwPuNko 5pyg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc; bh=J1u/YkZ7r4U6qP3j3HXey3MGQlv66sCtBUAUQwGtkmo=; b=chesZP8fpu+lqS1YYS84XLjXNgxOzZ14Jb0YwZ8iRxjSwPVX0adPUfGp86rplQgHDo vvXW7HO6NLGOaB53JTK9EEegCegFEnjdsWMjnrh1vF+XNaged6+cUMN/Es36O7/HAtik oTPV1xlRo7WZPdamDPBEMZggDpSH7eea+RKyrS5pFi08pSikpZtvZIXCT2n3lq5WMUrU PwzMhpsVEGtFEefDzY2CSt0tY+ipzhsEeKGA544B1p2ecRYFTUZE6aGgfGMupp29Fk6q /VPWRIMUWqs+Zei8kQUKgIh90RnJPWp+QwkU9K2E2XUQDL/tDp3k9Jv4pbfnGpu6YuBc cq0w== X-Gm-Message-State: ACgBeo2tip4lytpdthssvLFycjYafVVaC9G+hB2P3B2CAzdwHB70GyDx SoZtb1uqFkViVw2fo3RPgAIOdfkuwuA= X-Google-Smtp-Source: AA6agR7FrxHuf3ebSGip6Dsrc2j7pCNXLJVmg+GJ9+F6P4LGjbGDMIiC+XxX5XFr/bvFDyLyzfk2y9GLnRc= X-Received: from surenb-spec.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:3e30]) (user=surenb job=sendgmr) by 2002:a17:903:248:b0:172:7520:db07 with SMTP id j8-20020a170903024800b001727520db07mr18052271plh.76.1661808361566; Mon, 29 Aug 2022 14:26:01 -0700 (PDT) Date: Mon, 29 Aug 2022 21:25:17 +0000 In-Reply-To: <20220829212531.3184856-1-surenb@google.com> Mime-Version: 1.0 References: <20220829212531.3184856-1-surenb@google.com> X-Mailer: git-send-email 2.37.2.672.g94769d06f0-goog Message-ID: <20220829212531.3184856-15-surenb@google.com> Subject: [RFC PATCH 14/28] mm: mark VMAs as locked before isolating them From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, riel@surriel.com, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, surenb@google.com, kernel-team@android.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Mark VMAs as locked before isolating them and clear their tree node so that isolated VMAs are easily identifiable. In the later patches page fault handlers will try locking the found VMA and will check whether the VMA was isolated. Locking VMAs before isolating them ensures that page fault handlers don't operate on isolated VMAs. Signed-off-by: Suren Baghdasaryan --- mm/mmap.c | 2 ++ mm/nommu.c | 2 ++ 2 files changed, 4 insertions(+) diff --git a/mm/mmap.c b/mm/mmap.c index 094678b4434b..b0d78bdc0de0 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -421,12 +421,14 @@ static inline void vma_rb_insert(struct vm_area_struc= t *vma, =20 static void __vma_rb_erase(struct vm_area_struct *vma, struct rb_root *roo= t) { + vma_mark_locked(vma); /* * Note rb_erase_augmented is a fairly large inline function, * so make sure we instantiate it only once with our desired * augmented rbtree callbacks. */ rb_erase_augmented(&vma->vm_rb, root, &vma_gap_callbacks); + RB_CLEAR_NODE(&vma->vm_rb); } =20 static __always_inline void vma_rb_erase_ignore(struct vm_area_struct *vma, diff --git a/mm/nommu.c b/mm/nommu.c index e819cbc21b39..ff9933e57501 100644 --- a/mm/nommu.c +++ b/mm/nommu.c @@ -622,6 +622,7 @@ static void delete_vma_from_mm(struct vm_area_struct *v= ma) struct mm_struct *mm =3D vma->vm_mm; struct task_struct *curr =3D current; =20 + vma_mark_locked(vma); mm->map_count--; for (i =3D 0; i < VMACACHE_SIZE; i++) { /* if the vma is cached, invalidate the entire cache */ @@ -644,6 +645,7 @@ static void delete_vma_from_mm(struct vm_area_struct *v= ma) =20 /* remove from the MM's tree and list */ rb_erase(&vma->vm_rb, &mm->mm_rb); + RB_CLEAR_NODE(&vma->vm_rb); =20 __vma_unlink_list(mm, vma); } --=20 2.37.2.672.g94769d06f0-goog From nobody Tue Apr 7 09:06:15 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id CF3AAECAAD4 for ; Mon, 29 Aug 2022 21:27:28 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230002AbiH2V11 (ORCPT ); Mon, 29 Aug 2022 17:27:27 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33298 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229800AbiH2V0s (ORCPT ); Mon, 29 Aug 2022 17:26:48 -0400 Received: from mail-pg1-x549.google.com (mail-pg1-x549.google.com [IPv6:2607:f8b0:4864:20::549]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A5BC49E2D7 for ; Mon, 29 Aug 2022 14:26:12 -0700 (PDT) Received: by mail-pg1-x549.google.com with SMTP id q14-20020a6557ce000000b0041da9c3c244so4589013pgr.22 for ; Mon, 29 Aug 2022 14:26:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc; bh=KQbM9RENeAKpMamE0B+V4MUsqhzbhGXDlXXjmQQUA6A=; b=B9JYMkX5xydBJOgcGzuYw+brQarCWmRWS+0/Aqinb/OY26MbR1xT5axtOfpYrKlmA6 qGVi7LP6LaJMDcQJmEECw3J6IqaSMiez5M7VUbq2lZEZKkVSQHjZtuqy9g7s10YpHljB L9LJY/vXV43u8aIw7/j+LcG51+Bdx5jgIt7f2hLczg1JgYzlwTyYBe/lmm+nmwMuVZWc thodE+2BI084oTXFC0MD73T0ggmHuAL+ik6FuitgxrIkPY+yM+/axTrbJYGoWGabXb1/ atpqGKZe5bOiVGcNY/fR8gQx2/9qiwRjzQjgpRwLojxFdTcGDVT+d94c/I/T/XPhcGEF rEbQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc; bh=KQbM9RENeAKpMamE0B+V4MUsqhzbhGXDlXXjmQQUA6A=; b=RfwsNpyC3LqPhSXyAshMhZDi4oprDbnC90X+B8ciWreFBIwzFr+0vElvtUBl0la/9o BjiFSdQc5ti8mt1NbmMa9XyctMt3zlh+6iUOnJfkNcGDkrnizi6xhtiVCErRk6pqEcao wX0JcYTSHTi4Ihk+sQPfK9L59UPshNUA2BIcXfpU7kZpkA2hiNfccJn4OboZHHfEzmhQ eh8h4GcokCImGl53P8gl0ESl0rVcnTHfv3Yn3l/yqIstz63M3yTIdN9pNiE9mN8PBP+W yhbb8yg9Z8sN/bctryQ6uNxvKnRxFVPkvhGuD5TbILnuRcEhtLSecPy77+v12txyPwB3 QE+Q== X-Gm-Message-State: ACgBeo28Gq0uxKLao158logo/5iAqadwRQy2l4rb9dCepQkq9qwvAu3k jc0dfOELacaKs9+7cVTwJDgpJdIS1NY= X-Google-Smtp-Source: AA6agR7f4s+E2TUUdzlInwBIWNCTTjaS2dHpmImL6O3kfzXJ6rCAm5H+xn8nFf0AEemGeSJLYPdhv1d/+n8= X-Received: from surenb-spec.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:3e30]) (user=surenb job=sendgmr) by 2002:a05:6a00:b8a:b0:537:f81:203a with SMTP id g10-20020a056a000b8a00b005370f81203amr18341111pfj.80.1661808363176; Mon, 29 Aug 2022 14:26:03 -0700 (PDT) Date: Mon, 29 Aug 2022 21:25:18 +0000 In-Reply-To: <20220829212531.3184856-1-surenb@google.com> Mime-Version: 1.0 References: <20220829212531.3184856-1-surenb@google.com> X-Mailer: git-send-email 2.37.2.672.g94769d06f0-goog Message-ID: <20220829212531.3184856-16-surenb@google.com> Subject: [RFC PATCH 15/28] mm/mmap: mark adjacent VMAs as locked if they can grow into unmapped area From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, riel@surriel.com, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, surenb@google.com, kernel-team@android.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" While unmapping VMAs, adjacent VMAs might be able to grow into the area being unmapped. In such cases mark adjacent VMAs as locked to prevent this growth. Signed-off-by: Suren Baghdasaryan --- mm/mmap.c | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/mm/mmap.c b/mm/mmap.c index b0d78bdc0de0..b31cc97c2803 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -2680,10 +2680,14 @@ detach_vmas_to_be_unmapped(struct mm_struct *mm, st= ruct vm_area_struct *vma, * VM_GROWSUP VMA. Such VMAs can change their size under * down_read(mmap_lock) and collide with the VMA we are about to unmap. */ - if (vma && (vma->vm_flags & VM_GROWSDOWN)) + if (vma && (vma->vm_flags & VM_GROWSDOWN)) { + vma_mark_locked(vma); return false; - if (prev && (prev->vm_flags & VM_GROWSUP)) + } + if (prev && (prev->vm_flags & VM_GROWSUP)) { + vma_mark_locked(prev); return false; + } return true; } =20 --=20 2.37.2.672.g94769d06f0-goog From nobody Tue Apr 7 09:06:15 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 065FBECAAD4 for ; Mon, 29 Aug 2022 21:27:47 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230039AbiH2V1p (ORCPT ); Mon, 29 Aug 2022 17:27:45 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:34508 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229915AbiH2V1H (ORCPT ); Mon, 29 Aug 2022 17:27:07 -0400 Received: from mail-yw1-x1149.google.com (mail-yw1-x1149.google.com [IPv6:2607:f8b0:4864:20::1149]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D42A99E12A for ; Mon, 29 Aug 2022 14:26:11 -0700 (PDT) Received: by mail-yw1-x1149.google.com with SMTP id 00721157ae682-340a4dcb403so126518207b3.22 for ; Mon, 29 Aug 2022 14:26:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc; bh=NYe8oOJNfY11QXfRhGHbzq3PtP5qYF5OBWlNRgI/ICw=; b=gO3BAu2tPnt5ajc2wonv12EyC/cky1iEagPrWs8TIsmrfOPEdnZICXVlt39nBHt3WW tTV55lawhiZGMRBmA7gdT3YGawg8JjnJFEe5+7PrxS+L7PuBHZcRqu/eGZS5+7HkD+/v WBe3l/+DgKFH8adTgdOoThkCyU3oUkPdhTm9bGePyXaQUn5Kf04YxKMvEOmZ7WCBWmLN BXoUJ7D1rnsd47mFF47xSWH/cS8u1UFuDHVuHAEZYgUQ7/tjgR1JcWJgnN0k47FfJT+A /f4moE4fkIQHN9Nb1qOXqxym1C/DCmqaA6CVw+YBif80n7h7hRdtT5oL02KCkdJe7ned iBKA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc; bh=NYe8oOJNfY11QXfRhGHbzq3PtP5qYF5OBWlNRgI/ICw=; b=cnfgOFBEp44v6e9buEgtVfvaaXKmdShB/IIcT+Bp5/Zy+lp2GKtDZW6UpsMUZcOjqo Ew9FVrZ13MMVqJuulrjotw3Zli0ySBEg1cuDtZy0SJOjsfL4mA/0aIabimI09ctfIDBN aNvyaCVuINvMn6DLXhVPCmnYBek56T30IaPUDKZ2wzXuAyMiiELNdnSr5ln2CLZDPzl5 QZC2OD/mDbgOrIkI7WuszVyqF/9AMlGf8vx+nDVxVSLLTe31/2sCsWHufRK44erwaXFd +ai9MiC8rJZXA34ZLU5S1Apq0fw2nNxyhCA6spm6+cHGk4atGk9wgsKJ+UGxRhHE7bLc PBSA== X-Gm-Message-State: ACgBeo2VXBr/PkgDQ9LqivGKKOjX55ARFSlFFXK70a7COJKh8U/7bG1E Bm8I3BAS4Dn71C1v8GkrLwOTDGQXdXs= X-Google-Smtp-Source: AA6agR7nmkBRROpS79vTx3reOak4mrTZffxlmMbexsEub45N/wO5CnqJNjif+IJpyxwlCbKERnK2QUttv8k= X-Received: from surenb-spec.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:3e30]) (user=surenb job=sendgmr) by 2002:a25:d944:0:b0:695:6056:7dda with SMTP id q65-20020a25d944000000b0069560567ddamr9840482ybg.277.1661808364958; Mon, 29 Aug 2022 14:26:04 -0700 (PDT) Date: Mon, 29 Aug 2022 21:25:19 +0000 In-Reply-To: <20220829212531.3184856-1-surenb@google.com> Mime-Version: 1.0 References: <20220829212531.3184856-1-surenb@google.com> X-Mailer: git-send-email 2.37.2.672.g94769d06f0-goog Message-ID: <20220829212531.3184856-17-surenb@google.com> Subject: [RFC PATCH 16/28] kernel/fork: assert no VMA readers during its destruction From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, riel@surriel.com, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, surenb@google.com, kernel-team@android.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Assert there are no holders of VMA lock for reading when it is about to be destroyed. Signed-off-by: Suren Baghdasaryan --- include/linux/mm.h | 8 ++++++++ kernel/fork.c | 2 ++ 2 files changed, 10 insertions(+) diff --git a/include/linux/mm.h b/include/linux/mm.h index dc72be923e5b..0d9c1563c354 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -676,6 +676,13 @@ static inline void vma_assert_write_locked(struct vm_a= rea_struct *vma, int pos) VM_BUG_ON_VMA(vma->vm_lock_seq !=3D READ_ONCE(vma->vm_mm->mm_lock_seq), v= ma); } =20 +static inline void vma_assert_no_reader(struct vm_area_struct *vma) +{ + VM_BUG_ON_VMA(rwsem_is_locked(&vma->lock) && + vma->vm_lock_seq !=3D READ_ONCE(vma->vm_mm->mm_lock_seq), + vma); +} + #else /* CONFIG_PER_VMA_LOCK */ =20 static inline void vma_init_lock(struct vm_area_struct *vma) {} @@ -685,6 +692,7 @@ static inline bool vma_read_trylock(struct vm_area_stru= ct *vma) static inline void vma_read_unlock(struct vm_area_struct *vma) {} static inline void vma_assert_locked(struct vm_area_struct *vma) {} static inline void vma_assert_write_locked(struct vm_area_struct *vma, int= pos) {} +static inline void vma_assert_no_reader(struct vm_area_struct *vma) {} =20 #endif /* CONFIG_PER_VMA_LOCK */ =20 diff --git a/kernel/fork.c b/kernel/fork.c index 1872ad549fed..b443ba3a247a 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -487,6 +487,8 @@ static void __vm_area_free(struct rcu_head *head) { struct vm_area_struct *vma =3D container_of(head, struct vm_area_struct, vm_rcu); + /* The vma should either have no lock holders or be write-locked. */ + vma_assert_no_reader(vma); kmem_cache_free(vm_area_cachep, vma); } #endif --=20 2.37.2.672.g94769d06f0-goog From nobody Tue Apr 7 09:06:15 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id D280EECAAD4 for ; Mon, 29 Aug 2022 21:27:15 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229863AbiH2V1O (ORCPT ); Mon, 29 Aug 2022 17:27:14 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:32838 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229940AbiH2V0n (ORCPT ); Mon, 29 Aug 2022 17:26:43 -0400 Received: from mail-pj1-x1049.google.com (mail-pj1-x1049.google.com [IPv6:2607:f8b0:4864:20::1049]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3A19F9E891 for ; Mon, 29 Aug 2022 14:26:15 -0700 (PDT) Received: by mail-pj1-x1049.google.com with SMTP id q9-20020a17090aa00900b001fd93514a68so4327603pjp.3 for ; Mon, 29 Aug 2022 14:26:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc; bh=+ZeFgC+onLi5H2Gk/r+wfB2lUIAEpTgiMaISz2j8Yh8=; b=sfdQ+2NyYwxA7WE4W+U26tjnRsGpTwH+W4yCMsULIhDPYOMjHM3COZzPg/S6KTqrLL mCgPnd6W8rrgwvL37XiGeYHG8ArgrFQG0UGf2gLsolv/ME8hJmTyRg7Fg1LBPlsPaNI3 te4whtYEHF8c2GeRvQHa/rDrJjk62KSTUlVEPYqGvjKgQGKn8f4Dp8C4KiASRefP1Qh1 bQmVBjd8KI32N8+vGT1M5VRcf57cOjajmNaLyJUUpin1ccvlQLSGjkUAJOTaDxWOJrJ5 Pk3cN7LOZgsgUwKnowGjucplCuekHbu//K343iXFhb3F2AjszKhuURdvTUr7Fsy7vLjJ fKug== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc; bh=+ZeFgC+onLi5H2Gk/r+wfB2lUIAEpTgiMaISz2j8Yh8=; b=0WhHNp+Psn1uZcM7tnssYkTXQuKK03hKUOCjWoI3BEFXnigAtIVartwiKm+ZzS8BjU OjUl/sN13/ZIO7hL/OXNN3+peDvajdpysxrfaB8juq0pTADEHWFNEtxjs51KWkMFvhl4 YjgCMMyuTG4PodFxqcqBdEhjzHZJztpsp3ulp7PJkfQU9OIsgoMqegwIVy91V59fVfO9 r/tlpOe/CrvuanRmqML8+qT/YEvgDihwJs0GO3ZZRshToxT5aMtQpEi60hFb9Ho9HXzm AxF5D8JUpNORXF19ChKDyXf4Svo6RQFZNQnpki2WKEkC+IsQLJfXXYaUoxt5Xh744B4N ivfQ== X-Gm-Message-State: ACgBeo3xUpGG5Do3ibkTCxHCYw70BGUOuACDkVIbsmIxlA4A05uFT5ip 2GBFo17U4aoFElorfMw5A3CnCfvFz6o= X-Google-Smtp-Source: AA6agR60gqv1XJF9S7yhKpnItHkmCdUjQh3CpTVXyWE3xiDHKhNiTBjmbY9F8YGX1B9vpUO9txTmTcIisVc= X-Received: from surenb-spec.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:3e30]) (user=surenb job=sendgmr) by 2002:a17:902:6b88:b0:173:c1:691b with SMTP id p8-20020a1709026b8800b0017300c1691bmr18372367plk.18.1661808366708; Mon, 29 Aug 2022 14:26:06 -0700 (PDT) Date: Mon, 29 Aug 2022 21:25:20 +0000 In-Reply-To: <20220829212531.3184856-1-surenb@google.com> Mime-Version: 1.0 References: <20220829212531.3184856-1-surenb@google.com> X-Mailer: git-send-email 2.37.2.672.g94769d06f0-goog Message-ID: <20220829212531.3184856-18-surenb@google.com> Subject: [RFC PATCH 17/28] mm/mmap: prevent pagefault handler from racing with mmu_notifier registration From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, riel@surriel.com, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, surenb@google.com, kernel-team@android.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Pagefault handlers might need to fire MMU notifications while a new notifier is being registered. Modify mm_take_all_locks to mark all VMAs as locked and prevent this race with fault handlers that would hold VMA locks. Signed-off-by: Suren Baghdasaryan --- mm/mmap.c | 3 +++ 1 file changed, 3 insertions(+) diff --git a/mm/mmap.c b/mm/mmap.c index b31cc97c2803..1edfcd384f5e 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -3538,6 +3538,7 @@ static void vm_lock_mapping(struct mm_struct *mm, str= uct address_space *mapping) * hugetlb mapping); * - all i_mmap_rwsem locks; * - all anon_vma->rwseml + * - all vmas marked locked * * We can take all locks within these types randomly because the VM code * doesn't nest them and we protected from parallel mm_take_all_locks() by @@ -3579,6 +3580,7 @@ int mm_take_all_locks(struct mm_struct *mm) if (vma->anon_vma) list_for_each_entry(avc, &vma->anon_vma_chain, same_vma) vm_lock_anon_vma(mm, avc->anon_vma); + vma_mark_locked(vma); } =20 return 0; @@ -3636,6 +3638,7 @@ void mm_drop_all_locks(struct mm_struct *mm) mmap_assert_write_locked(mm); BUG_ON(!mutex_is_locked(&mm_all_locks_mutex)); =20 + vma_mark_unlocked_all(mm); for (vma =3D mm->mmap; vma; vma =3D vma->vm_next) { if (vma->anon_vma) list_for_each_entry(avc, &vma->anon_vma_chain, same_vma) --=20 2.37.2.672.g94769d06f0-goog From nobody Tue Apr 7 09:06:15 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5FAD6ECAAD4 for ; Mon, 29 Aug 2022 21:27:19 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229951AbiH2V1R (ORCPT ); Mon, 29 Aug 2022 17:27:17 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33018 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229941AbiH2V0n (ORCPT ); Mon, 29 Aug 2022 17:26:43 -0400 Received: from mail-pf1-x449.google.com (mail-pf1-x449.google.com [IPv6:2607:f8b0:4864:20::449]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 161459E881 for ; Mon, 29 Aug 2022 14:26:14 -0700 (PDT) Received: by mail-pf1-x449.google.com with SMTP id by13-20020a056a00400d00b0052ec5a1cd4dso3623103pfb.21 for ; Mon, 29 Aug 2022 14:26:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc; bh=MrRcKlTR7zCSrJMQg2Z93mfggQWMfDwWeAi691n34gQ=; b=cVg/tOxX8sxsAXVi/E7ByEF0x14KVPhZxGLvuX25A7I6HfsUpP9Yw5fhBx8ozeM60j nuDrEi09QA9+LTKC8GajK7CQiyi7gN9srLPx+SGJoGK9+OesuSLiR+9oEBujQJz23p/X P3t9ZdRyZDRuaF1wtpM+xaufjpbccFTUx81s4naxMNNB/ygFAADiWbtKwBK4coZj6Khw l+lkYHWl+iss5qkPjBxcVIZuDHxj8lggM4wnCJgOdoiaVpSSSGOBOp9823YRpnW7UBz5 1Wlgpr7ZzU0IbMK3CpPADXeW2X4v/4Fep7gHMya9NlwRNw5kCRaApBpvKmaw4X2L8jbG 4C6A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc; bh=MrRcKlTR7zCSrJMQg2Z93mfggQWMfDwWeAi691n34gQ=; b=ICsr0sx9qYmw2ydEzrTVu9NK8yud1dQVIMO3+EY5E4JGncbrctJt+d6ZJ6LS4S7qSV EfE1804kHStW+qRVq/hQvhPGZOnCuyaAVTkZ0EcGmhaokFY6+U3mbg7b4paAwR+AcW6b Py0UjO2pUvjAVdoM9Ys2Zu84xyuF4aGSgwxwEQspDra4wv6WxZLJikutl+5ciCZxMEJb xgWAfECr6nrWhksF6bGe3hPJOF/C7ui6C2Pwb6EjLCN+3kXFygB6ugbtrFrNHjjbxWE3 hOSCfbrMvhxvpcV4fu6SoXAGfp9eCR+KOGmTYsHyoDGLLlQAhoOV8Nmv+iO0ESvHPUds iUAA== X-Gm-Message-State: ACgBeo1MOD9E5o9ANDH9O/mW4KSsNMar21IdORL23XidO/QNUfYrluVx RkSQTbAOY1yFFAHoJkYJ8PwusqXofkU= X-Google-Smtp-Source: AA6agR6U/5CcVqFo3qhw78HCsOiNZ4QUVwqc0LtUdoOd4ifUOM6ozQmoAsAgpNZOBFg7UV9Y6fDIpWBGaXE= X-Received: from surenb-spec.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:3e30]) (user=surenb job=sendgmr) by 2002:aa7:8742:0:b0:537:ee75:601a with SMTP id g2-20020aa78742000000b00537ee75601amr12578569pfo.37.1661808368234; Mon, 29 Aug 2022 14:26:08 -0700 (PDT) Date: Mon, 29 Aug 2022 21:25:21 +0000 In-Reply-To: <20220829212531.3184856-1-surenb@google.com> Mime-Version: 1.0 References: <20220829212531.3184856-1-surenb@google.com> X-Mailer: git-send-email 2.37.2.672.g94769d06f0-goog Message-ID: <20220829212531.3184856-19-surenb@google.com> Subject: [RFC PATCH 18/28] mm: add FAULT_FLAG_VMA_LOCK flag From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, riel@surriel.com, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, surenb@google.com, kernel-team@android.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Add a new flag to distinguish page faults handled under protection of per-vma lock. Signed-off-by: Suren Baghdasaryan --- include/linux/mm.h | 3 ++- include/linux/mm_types.h | 1 + 2 files changed, 3 insertions(+), 1 deletion(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 0d9c1563c354..7c3190eaabd7 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -466,7 +466,8 @@ static inline bool fault_flag_allow_retry_first(enum fa= ult_flag flags) { FAULT_FLAG_USER, "USER" }, \ { FAULT_FLAG_REMOTE, "REMOTE" }, \ { FAULT_FLAG_INSTRUCTION, "INSTRUCTION" }, \ - { FAULT_FLAG_INTERRUPTIBLE, "INTERRUPTIBLE" } + { FAULT_FLAG_INTERRUPTIBLE, "INTERRUPTIBLE" }, \ + { FAULT_FLAG_VMA_LOCK, "VMA_LOCK" } =20 /* * vm_fault is filled by the pagefault handler and passed to the vma's diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 6a03f59c1e78..36562e702baf 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -886,6 +886,7 @@ enum fault_flag { FAULT_FLAG_INTERRUPTIBLE =3D 1 << 9, FAULT_FLAG_UNSHARE =3D 1 << 10, FAULT_FLAG_ORIG_PTE_VALID =3D 1 << 11, + FAULT_FLAG_VMA_LOCK =3D 1 << 12, }; =20 typedef unsigned int __bitwise zap_flags_t; --=20 2.37.2.672.g94769d06f0-goog From nobody Tue Apr 7 09:06:15 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 90E72ECAAD4 for ; Mon, 29 Aug 2022 21:27:37 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229967AbiH2V1g (ORCPT ); Mon, 29 Aug 2022 17:27:36 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33312 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229956AbiH2V0u (ORCPT ); Mon, 29 Aug 2022 17:26:50 -0400 Received: from mail-pg1-x549.google.com (mail-pg1-x549.google.com [IPv6:2607:f8b0:4864:20::549]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 39CAE9E888 for ; Mon, 29 Aug 2022 14:26:15 -0700 (PDT) Received: by mail-pg1-x549.google.com with SMTP id i25-20020a635859000000b0042bbb74be8bso2624710pgm.5 for ; Mon, 29 Aug 2022 14:26:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc; bh=K93TFWAb7JwQdWCvuFbBF/NwLgqzppA/TxAstG/QPvU=; b=f8ZHy6CiTcDXROb3pFMLfetBSV+kFVTeZws7OjDiiohYptMrJjGynWZ4jN+J/j3SJj dCEDPAmPkmnX5B3X46hqeY2mb8ZvMpLB7j64gw3uDHSYMAUM2B6ucp8KKPug6MxLnEu7 BmebG6DC6czjsVXnb7ENc9u7p9QGYJElvc7qRNsIMqXvgjewnR2ulhT9EyPiZ2zz1B9D 56/cpf1MfCjBxFK5vFsjvAIH2zjjfVoYxcmJWgmvbc3eCKVdp1qhJM+tuigQjNUJJMgZ jrHBPRDF7lp+rxYzH8zskIPXprMRBcWxeh9qo4POOeyeUnThUPILDVIwum49CxVeKLEs T9zA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc; bh=K93TFWAb7JwQdWCvuFbBF/NwLgqzppA/TxAstG/QPvU=; b=yguYos31KDmZgoOyO8nJfP16xaXx0z1lgP88ijLpXJHVPgTHRit6rUC0gMN0a1GM9/ Pk63QZUh48bjR1BMHvO5DliukMhHp+CHpr9N2hlxv9RVqDoPeTUilVRcp70yG+26CBf9 7i4atw6V8nGbhNOrcQwnFy02upV3gv/Gu/ArIdacVcp8pP09TKoedJuUew1au7AHY2ZA C6r/YHlM6Nh0IEMLRCROJZUM8D9+NdEYprL5f6WFoxyRvC2j0ATpZsVxOT7s3IMLmWcI 56ixFtTHqhm0kX2MunIhKf2SbzqdxhIpkpyxBZpJlBRei7y6PQ190fhadaXwgxHleXhK Hhsg== X-Gm-Message-State: ACgBeo0okVUtIzeeQKkMM2cxSjbQ+s3Y5jzsZV/Xc3cF4GZeYocaP+uv 6QMc8/SvUZz737/Bq1M3OqQVhnwEdN4= X-Google-Smtp-Source: AA6agR7nf9+ItGZZ0+7hqrcNY7xNhTop6ny3nOph/OxIOFh5WmAm4Kp1l2Jp+ca/L6EEvcmkimt0snMwAs0= X-Received: from surenb-spec.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:3e30]) (user=surenb job=sendgmr) by 2002:a17:902:b094:b0:173:12d1:4ece with SMTP id p20-20020a170902b09400b0017312d14ecemr18383730plr.117.1661808370030; Mon, 29 Aug 2022 14:26:10 -0700 (PDT) Date: Mon, 29 Aug 2022 21:25:22 +0000 In-Reply-To: <20220829212531.3184856-1-surenb@google.com> Mime-Version: 1.0 References: <20220829212531.3184856-1-surenb@google.com> X-Mailer: git-send-email 2.37.2.672.g94769d06f0-goog Message-ID: <20220829212531.3184856-20-surenb@google.com> Subject: [RFC PATCH 19/28] mm: disallow do_swap_page to handle page faults under VMA lock From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, riel@surriel.com, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, surenb@google.com, kernel-team@android.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Due to the possibility of do_swap_page dropping mmap_lock, abort fault handling under VMA lock and retry holding mmap_lock. This can be handled more gracefully in the future. Signed-off-by: Suren Baghdasaryan --- mm/memory.c | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/mm/memory.c b/mm/memory.c index 9ac9944e8c62..29d2f49f922a 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -3738,6 +3738,11 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) vm_fault_t ret =3D 0; void *shadow =3D NULL; =20 + if (vmf->flags & FAULT_FLAG_VMA_LOCK) { + ret =3D VM_FAULT_RETRY; + goto out; + } + if (!pte_unmap_same(vmf)) goto out; =20 --=20 2.37.2.672.g94769d06f0-goog From nobody Tue Apr 7 09:06:15 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 14C0BECAAD4 for ; Mon, 29 Aug 2022 21:27:25 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229995AbiH2V1W (ORCPT ); Mon, 29 Aug 2022 17:27:22 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33278 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229862AbiH2V0q (ORCPT ); Mon, 29 Aug 2022 17:26:46 -0400 Received: from mail-pg1-x54a.google.com (mail-pg1-x54a.google.com [IPv6:2607:f8b0:4864:20::54a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E634D985A4 for ; Mon, 29 Aug 2022 14:26:16 -0700 (PDT) Received: by mail-pg1-x54a.google.com with SMTP id m34-20020a634c62000000b0042aff6dff12so4547542pgl.14 for ; Mon, 29 Aug 2022 14:26:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc; bh=bUksEMWzQ2sQtj6CAseSDP7Wnd3Y9YWuXlHuoAiHS/w=; b=R4f3AmhYQvfrJmCukv/e3Nij13PuVUa8k8O646df0PoObwBnO/xDcemk6+SqKXvVGr hJkm2649shhdYcclI4xYPbiKHXEe7XEvjq6ebwkp6YZFsZtQX3MH5xTV757BPS7Zeyib IALyiqYSG4f0kCJDHQ3+fMXa5F0OzkQKlh2EgahzIDAp6HMSPtfd8qN17COdyNthUkyK vH/w1n6XgnYHxRHIzAjyfQH31twkRddq+jpJViGP9dMWFd2ba2cwGPVTVH8mdAvdjCY4 84OD9xZ4r8HlzIfM65gveSjGxaILhsxn9vJXDz/VODVoOMzJt9RI4cwlcbKnFfX3PjSN BY7A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc; bh=bUksEMWzQ2sQtj6CAseSDP7Wnd3Y9YWuXlHuoAiHS/w=; b=l/Y0aMF0B+iVY2xu5dVlCbcxy2dr5pCTArjwe9X0x3iaUN47Mi05RvDrVKwoN+9TcE pbJ0QoWJgbtvg13goGjeunHxUIXGY5Ai04qsYX8CCBh2l/Jx5FrFdpin4pkkNkfNBAcX ctKOiDhLjKWm22m+zkv9gIwS4WwFlXk9CjNDk5lxv3PbBnrZPEYhQjFqjX+IKW0wkaOw 4iVj16gulpQ1ygTGZpg0r+LsSFDIrJ3r68jzf6XLJMXcpz4AByGWcOxYulT3JV9kp9tZ Qibs6W6LiXClY2LFNscbCVsSo6T2WgB57UtR09eKFlWTWqWjh8MYaA2ltwKKenad2QX6 cuTA== X-Gm-Message-State: ACgBeo32bGo42XvCuDgrDzWFKlpuiuEtVQ1NuXP2YxmYVuu+ij1VMZ4E uDEItCgs1RNsru8zRn74UWN09dTwY58= X-Google-Smtp-Source: AA6agR4CSgZfYughTw1oP2TGtJ/Gx4qpWPKVumSGbjZh4UG9V5skHPeijnTVviS5CargVPsvloUs2cN/SRk= X-Received: from surenb-spec.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:3e30]) (user=surenb job=sendgmr) by 2002:a17:90b:2c11:b0:1fd:e56c:79de with SMTP id rv17-20020a17090b2c1100b001fde56c79demr4481996pjb.201.1661808371494; Mon, 29 Aug 2022 14:26:11 -0700 (PDT) Date: Mon, 29 Aug 2022 21:25:23 +0000 In-Reply-To: <20220829212531.3184856-1-surenb@google.com> Mime-Version: 1.0 References: <20220829212531.3184856-1-surenb@google.com> X-Mailer: git-send-email 2.37.2.672.g94769d06f0-goog Message-ID: <20220829212531.3184856-21-surenb@google.com> Subject: [RFC PATCH 20/28] mm: introduce per-VMA lock statistics From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, riel@surriel.com, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, surenb@google.com, kernel-team@android.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Add a new CONFIG_PER_VMA_LOCK_STATS config option to dump extra statistics about handling page fault under VMA lock. Signed-off-by: Suren Baghdasaryan --- include/linux/vm_event_item.h | 6 ++++++ include/linux/vmstat.h | 6 ++++++ mm/Kconfig.debug | 8 ++++++++ mm/vmstat.c | 6 ++++++ 4 files changed, 26 insertions(+) diff --git a/include/linux/vm_event_item.h b/include/linux/vm_event_item.h index f3fc36cd2276..a325783ed05d 100644 --- a/include/linux/vm_event_item.h +++ b/include/linux/vm_event_item.h @@ -150,6 +150,12 @@ enum vm_event_item { PGPGIN, PGPGOUT, PSWPIN, PSWPOUT, #ifdef CONFIG_X86 DIRECT_MAP_LEVEL2_SPLIT, DIRECT_MAP_LEVEL3_SPLIT, +#endif +#ifdef CONFIG_PER_VMA_LOCK_STATS + VMA_LOCK_SUCCESS, + VMA_LOCK_ABORT, + VMA_LOCK_RETRY, + VMA_LOCK_MISS, #endif NR_VM_EVENT_ITEMS }; diff --git a/include/linux/vmstat.h b/include/linux/vmstat.h index bfe38869498d..0c2611899cfc 100644 --- a/include/linux/vmstat.h +++ b/include/linux/vmstat.h @@ -131,6 +131,12 @@ static inline void vm_events_fold_cpu(int cpu) #define count_vm_vmacache_event(x) do {} while (0) #endif =20 +#ifdef CONFIG_PER_VMA_LOCK_STATS +#define count_vm_vma_lock_event(x) count_vm_event(x) +#else +#define count_vm_vma_lock_event(x) do {} while (0) +#endif + #define __count_zid_vm_events(item, zid, delta) \ __count_vm_events(item##_NORMAL - ZONE_NORMAL + zid, delta) =20 diff --git a/mm/Kconfig.debug b/mm/Kconfig.debug index ce8dded36de9..075642763a03 100644 --- a/mm/Kconfig.debug +++ b/mm/Kconfig.debug @@ -207,3 +207,11 @@ config PTDUMP_DEBUGFS kernel. =20 If in doubt, say N. + + +config PER_VMA_LOCK_STATS + bool "Statistics for per-vma locks" + depends on PER_VMA_LOCK + help + Statistics for per-vma locks. + If in doubt, say N. diff --git a/mm/vmstat.c b/mm/vmstat.c index 90af9a8572f5..3f3804c846a6 100644 --- a/mm/vmstat.c +++ b/mm/vmstat.c @@ -1411,6 +1411,12 @@ const char * const vmstat_text[] =3D { "direct_map_level2_splits", "direct_map_level3_splits", #endif +#ifdef CONFIG_PER_VMA_LOCK_STATS + "vma_lock_success", + "vma_lock_abort", + "vma_lock_retry", + "vma_lock_miss", +#endif #endif /* CONFIG_VM_EVENT_COUNTERS || CONFIG_MEMCG */ }; #endif /* CONFIG_PROC_FS || CONFIG_SYSFS || CONFIG_NUMA || CONFIG_MEMCG */ --=20 2.37.2.672.g94769d06f0-goog From nobody Tue Apr 7 09:06:15 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id DA187ECAAD2 for ; Mon, 29 Aug 2022 21:28:25 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230101AbiH2V2X (ORCPT ); Mon, 29 Aug 2022 17:28:23 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:34012 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230000AbiH2V1Z (ORCPT ); Mon, 29 Aug 2022 17:27:25 -0400 Received: from mail-pf1-x44a.google.com (mail-pf1-x44a.google.com [IPv6:2607:f8b0:4864:20::44a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 37DDE7268E for ; Mon, 29 Aug 2022 14:26:18 -0700 (PDT) Received: by mail-pf1-x44a.google.com with SMTP id 200-20020a6217d1000000b00538090d37f3so2068608pfx.3 for ; Mon, 29 Aug 2022 14:26:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc; bh=TAH5MnJT3oUowhISL8bra7V0NlPN+Gm7GlObU5EM5DA=; b=WI+iAkKYdp/dw+q+sE6vVFwD5mLSWBWMhRHboF9Ttbardy/unLndkxHVrzFVqulr1A yPcJLa3Ur0z+PJer1B1IamGoCGgxEpMLeAxgWf9CgasGK7cypr9FW+usqow4Tv90c3k6 hSoUgkPn+GgzurgQcPzRD2rwA/PJHxIAlhRbX6toXLNmtMd1EOWFLhXluJKlLDk1UBbT O0I6rKmCm2uwsJ4JnyJZiGAk3wWQ6Qh6RctA9jkOKMvzLM+0NSmJ3ovBUaPQyVoGm6YI JXf6mvqPKdqWiUAL/+XE2dov1V39lipJcNcZcI/t4fYcyAX1GpBk//UErawM2Q1iWoKr uzug== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc; bh=TAH5MnJT3oUowhISL8bra7V0NlPN+Gm7GlObU5EM5DA=; b=FnnfvDCIBw0zjAyK4HDilGoC17RsYobiWHfAB7/kGxrNb+dprDONboH34M13fL2Mhp +uAkd40/wbODIgpllCUcSFg2IBArmdL+dhNNA/aP8QXFN8sA6mir0Syr1AFN/f91jUcu yFL//kic3MAWDXIhiAjc+B/UUwJTQK7bEXIZoI7f8GYohFF3FrDqXRxqEJGiCluQIEMF 1zKJ3PDT1ayA/6i0eGC8ZyZtKD3/MbgL5qGrfRa35M5Bh2TmUTyYvOwRAUUxXHqX+4QN lznlANGwjkbsh4XnDDWu3Q6q5HidDGCZG7qx5yN3JCJXRTdIzdAlqMaJYJp+O2IrySLQ 1s7Q== X-Gm-Message-State: ACgBeo1DjhYQBSIGsbwCw0i7qQ0aesyfeoDDbb4PPPlwKtP6HxL5UxS9 m3vNVACiYGKbWS62RK2y8NdixM/fXzw= X-Google-Smtp-Source: AA6agR5aic644PZ0OuL1YK8O5lnv8GmzPsmAclgLgLZvZKPuRUB75AtHpFEjRVbtgx264YmpiRiioO9wBJc= X-Received: from surenb-spec.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:3e30]) (user=surenb job=sendgmr) by 2002:a17:90a:249:b0:1e0:a8a3:3c6c with SMTP id t9-20020a17090a024900b001e0a8a33c6cmr21187pje.0.1661808373054; Mon, 29 Aug 2022 14:26:13 -0700 (PDT) Date: Mon, 29 Aug 2022 21:25:24 +0000 In-Reply-To: <20220829212531.3184856-1-surenb@google.com> Mime-Version: 1.0 References: <20220829212531.3184856-1-surenb@google.com> X-Mailer: git-send-email 2.37.2.672.g94769d06f0-goog Message-ID: <20220829212531.3184856-22-surenb@google.com> Subject: [RFC PATCH 21/28] mm: introduce find_and_lock_anon_vma to be used from arch-specific code From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, riel@surriel.com, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, surenb@google.com, kernel-team@android.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Introduce find_and_lock_anon_vma function to lookup and lock an anonymous VMA during page fault handling. When VMA is not found, can't be locked or changes after being locked, the function returns NULL. The lookup is performed under RCU protection to prevent the found VMA from being destroyed before the VMA lock is acquired. VMA lock statistics are updated according to the results. Signed-off-by: Suren Baghdasaryan --- include/linux/mm.h | 3 +++ mm/memory.c | 45 +++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 48 insertions(+) diff --git a/include/linux/mm.h b/include/linux/mm.h index 7c3190eaabd7..a3cbaa7b9119 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -684,6 +684,9 @@ static inline void vma_assert_no_reader(struct vm_area_= struct *vma) vma); } =20 +struct vm_area_struct *find_and_lock_anon_vma(struct mm_struct *mm, + unsigned long address); + #else /* CONFIG_PER_VMA_LOCK */ =20 static inline void vma_init_lock(struct vm_area_struct *vma) {} diff --git a/mm/memory.c b/mm/memory.c index 29d2f49f922a..bf557f7056de 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -5183,6 +5183,51 @@ vm_fault_t handle_mm_fault(struct vm_area_struct *vm= a, unsigned long address, } EXPORT_SYMBOL_GPL(handle_mm_fault); =20 +#ifdef CONFIG_PER_VMA_LOCK +static inline struct vm_area_struct *find_vma_under_rcu(struct mm_struct *= mm, + unsigned long address) +{ + struct vm_area_struct *vma =3D __find_vma(mm, address); + + if (!vma || vma->vm_start > address) + return NULL; + + if (!vma_is_anonymous(vma)) + return NULL; + + if (!vma_read_trylock(vma)) { + count_vm_vma_lock_event(VMA_LOCK_ABORT); + return NULL; + } + + /* Check if the VMA got isolated after we found it */ + if (RB_EMPTY_NODE(&vma->vm_rb)) { + vma_read_unlock(vma); + count_vm_vma_lock_event(VMA_LOCK_MISS); + return NULL; + } + + return vma; +} + +/* + * Lookup and lock and anonymous VMA. Returned VMA is guaranteed to be sta= ble + * and not isolated. If the VMA is not found of is being modified the func= tion + * returns NULL. + */ +struct vm_area_struct *find_and_lock_anon_vma(struct mm_struct *mm, + unsigned long address) +{ + struct vm_area_struct *vma; + + rcu_read_lock(); + vma =3D find_vma_under_rcu(mm, address); + rcu_read_unlock(); + + return vma; +} +#endif /* CONFIG_PER_VMA_LOCK */ + #ifndef __PAGETABLE_P4D_FOLDED /* * Allocate p4d page table. --=20 2.37.2.672.g94769d06f0-goog From nobody Tue Apr 7 09:06:15 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id ED844ECAAD2 for ; Mon, 29 Aug 2022 21:28:03 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230011AbiH2V15 (ORCPT ); Mon, 29 Aug 2022 17:27:57 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33442 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229600AbiH2V1I (ORCPT ); Mon, 29 Aug 2022 17:27:08 -0400 Received: from mail-pl1-x64a.google.com (mail-pl1-x64a.google.com [IPv6:2607:f8b0:4864:20::64a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 143D8979F0 for ; Mon, 29 Aug 2022 14:26:18 -0700 (PDT) Received: by mail-pl1-x64a.google.com with SMTP id t13-20020a170902e84d00b00174b03be629so3066595plg.16 for ; Mon, 29 Aug 2022 14:26:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc; bh=plnLK4otWaGxR+W8sd3fBxFm06LW87+WFj6YX7JmKJI=; b=O7Fq7HWZ7z8n2j/HcxIFYSMFkVCbPksq++2+VIS66vXRQwbI7dU8JTklRI8MPiby2e edzsUySG6IImo6mrh8ENCQ2+wxA6BfMFPfkVn5AwXOsZupGcnzhGGSKXttVqFe7CqMCi 1h3acQOuI10/ig3PP8xkKsm4NtzNbHGpaQpLwoGjKN2BSoc/HFLvkEbHUIkKXG5J3g9z 9q6Jpb0Ph5QlVq/4hF27NyIuM3/V2/KAkuHoouSJodEv2dWsfVfE58VeJPPY/+QOUQfz r+QwUf2vpfx2TV1s2fbWWB8eIKopRLMSEdWWEDiBiVoTpxt/dnwqSCw7tnlmHUsJj3fT 5A2Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc; bh=plnLK4otWaGxR+W8sd3fBxFm06LW87+WFj6YX7JmKJI=; b=An68qMb3BQkaDFoaTCxxaamXw5oAoZjEZ7URrhoAlVLhpPPeIod/NjRIeZ/XAJELtP SdGjfhkagpI+4ovYHYfoh0wHCiouDbkhANyMYodmk0cGBhsLdPWXs/5aPwHYd3afAgaY 0L8TekAY3mwVPvfb8uyAIpm9+E5T2Cn8658i28tbHqjQyPyzGNqlkYvay1fVTBV/gxWT TEUEycUn29XF8BshCGVVxbGfqJmNLiDxs/Zl6VLMe2hYtMQlco6srBdoi8B5umRZfSks /FIKwo7LY2PSn61X5Smh3RNo7PsvmoKSDpL+idWeY7NCj6PgOiPkZTG7/MbEtUmqW0lw UwmQ== X-Gm-Message-State: ACgBeo2EsFHSUIqvf1DUgRM5kcCBycJEV/nx+1wFHuzeZIFIEHnQjKAU O99mch1GH+nLfRxtKzykcdvKqt1VyDM= X-Google-Smtp-Source: AA6agR6koDuvaTUSC7OU/WlAja75jS99jSayBh5syt6ErVM0PHA+biyxUvs5Ar838YXdhWnvkXm6gonES4M= X-Received: from surenb-spec.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:3e30]) (user=surenb job=sendgmr) by 2002:a05:6a00:14c7:b0:52e:efb7:bd05 with SMTP id w7-20020a056a0014c700b0052eefb7bd05mr17908102pfu.24.1661808374985; Mon, 29 Aug 2022 14:26:14 -0700 (PDT) Date: Mon, 29 Aug 2022 21:25:25 +0000 In-Reply-To: <20220829212531.3184856-1-surenb@google.com> Mime-Version: 1.0 References: <20220829212531.3184856-1-surenb@google.com> X-Mailer: git-send-email 2.37.2.672.g94769d06f0-goog Message-ID: <20220829212531.3184856-23-surenb@google.com> Subject: [RFC PATCH 22/28] x86/mm: try VMA lock-based page fault handling first From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, riel@surriel.com, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, surenb@google.com, kernel-team@android.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Attempt VMA lock-based page fault handling first, and fall back to the existing mmap_lock-based handling if that fails. Signed-off-by: Suren Baghdasaryan --- arch/x86/mm/fault.c | 36 ++++++++++++++++++++++++++++++++++++ 1 file changed, 36 insertions(+) diff --git a/arch/x86/mm/fault.c b/arch/x86/mm/fault.c index fa71a5d12e87..35e74e3dc2c1 100644 --- a/arch/x86/mm/fault.c +++ b/arch/x86/mm/fault.c @@ -19,6 +19,7 @@ #include /* faulthandler_disabled() */ #include /* efi_crash_gracefully_on_page_fault()*/ #include +#include /* find_and_lock_vma() */ =20 #include /* boot_cpu_has, ... */ #include /* dotraplinkage, ... */ @@ -1323,6 +1324,38 @@ void do_user_addr_fault(struct pt_regs *regs, } #endif =20 +#ifdef CONFIG_PER_VMA_LOCK + if (!(flags & FAULT_FLAG_USER) || atomic_read(&mm->mm_users) =3D=3D 1) + goto lock_mmap; + + vma =3D find_and_lock_anon_vma(mm, address); + if (!vma) + goto lock_mmap; + + if (unlikely(access_error(error_code, vma))) { + vma_read_unlock(vma); + goto lock_mmap; + } + fault =3D handle_mm_fault(vma, address, flags | FAULT_FLAG_VMA_LOCK, regs= ); + vma_read_unlock(vma); + + if (!(fault & VM_FAULT_RETRY)) { + count_vm_vma_lock_event(VMA_LOCK_SUCCESS); + goto done; + } + count_vm_vma_lock_event(VMA_LOCK_RETRY); + + /* Quick path to respond to signals */ + if (fault_signal_pending(fault, regs)) { + if (!user_mode(regs)) + kernelmode_fixup_or_oops(regs, error_code, address, + SIGBUS, BUS_ADRERR, + ARCH_DEFAULT_PKEY); + return; + } +lock_mmap: +#endif /* CONFIG_PER_VMA_LOCK */ + /* * Kernel-mode access to the user address space should only occur * on well-defined single instructions listed in the exception @@ -1423,6 +1456,9 @@ void do_user_addr_fault(struct pt_regs *regs, } =20 mmap_read_unlock(mm); +#ifdef CONFIG_PER_VMA_LOCK +done: +#endif if (likely(!(fault & VM_FAULT_ERROR))) return; =20 --=20 2.37.2.672.g94769d06f0-goog From nobody Tue Apr 7 09:06:15 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8171BECAAD2 for ; Mon, 29 Aug 2022 21:27:34 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229958AbiH2V1c (ORCPT ); Mon, 29 Aug 2022 17:27:32 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:32794 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229950AbiH2V0t (ORCPT ); Mon, 29 Aug 2022 17:26:49 -0400 Received: from mail-pj1-x104a.google.com (mail-pj1-x104a.google.com [IPv6:2607:f8b0:4864:20::104a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E04D474DCB for ; Mon, 29 Aug 2022 14:26:18 -0700 (PDT) Received: by mail-pj1-x104a.google.com with SMTP id 36-20020a17090a0fa700b001fd64c962afso3800076pjz.5 for ; Mon, 29 Aug 2022 14:26:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc; bh=2fd/wGll+ATU6BJRTn7wtpRUT69Ps19K4L9NuJFwTCo=; b=CptXAEbTEIrjcI1hpAhqMGzwstwkcEJJxmfb7we2Tkw33BFQo0d25Vl71IxE1hgwCE +/zdjUxBz/kfwlykJ2mpRLD2ASHh0Jx/8NPF+GvmOj+OIPDuJ9h8tJ0qgl8jIPnA0gyY HL/K18g+JnPGw8ocd38eG5KHL97qZgAE2vHMSq8EmM7kYFzwJqKpJAdVhwHK8rQ0CEUk zlwtiJI41bKlNxlGW7NWWuuvJOTrejKhLXTonY1HBOreH+6SF0a8TMKsBr5d7H138Lnw MYC4Krs42a4eoKPM77oYkjvTzwtNbqixe2Jty6RY0Tg1xXxoDrCaGfCoHKiNRbMG4ac9 6NkA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc; bh=2fd/wGll+ATU6BJRTn7wtpRUT69Ps19K4L9NuJFwTCo=; b=HSqoH+9LDmkeOjE5+0qscOIrkwQQdDtmrOd/0Ibs+OSb6voR6T3yUT9R4md7TDZgpc Brn8EP8jIYdKS3b7xZrak0H7vh2vPFNFTtUXyebg4t6uOVesUTeITl9Zf3bJsstqg3/2 UnzcDX//LsbKdkGD/79llnmFHn94PVMnL7KOxwbZHtckSuFFPBvzZCY67CJWjxLG3N9l MO0TpDFNLLWKX5mPQSzLVe1ige5h5oCzn7ei7D/4X69JwQDN/7zRkh1eONABiIJxDbDa d/5KsS+xQx+wWXfWCuV/U0/q/VYjKiAj0t18EiFuNk4TPVawzh7wdCR+rDUfjgyRY7oh VNqQ== X-Gm-Message-State: ACgBeo3aootgj6cTHOBsoHv74Q6EiVC9mJmAKtj8ztKyrpYmPOMJQ2+P peTNQDQYR/+CbgpAFljQPK7vlrckh+I= X-Google-Smtp-Source: AA6agR5QjqhJOQoDiATaLtKSN9vwads3L2SWpCSYlmwHfKycrJZLy9F13OZO0MNqEBq8/z8yofCbHCaZE68= X-Received: from surenb-spec.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:3e30]) (user=surenb job=sendgmr) by 2002:a17:902:d54b:b0:174:af35:4ba0 with SMTP id z11-20020a170902d54b00b00174af354ba0mr7860599plf.74.1661808376452; Mon, 29 Aug 2022 14:26:16 -0700 (PDT) Date: Mon, 29 Aug 2022 21:25:26 +0000 In-Reply-To: <20220829212531.3184856-1-surenb@google.com> Mime-Version: 1.0 References: <20220829212531.3184856-1-surenb@google.com> X-Mailer: git-send-email 2.37.2.672.g94769d06f0-goog Message-ID: <20220829212531.3184856-24-surenb@google.com> Subject: [RFC PATCH 23/28] x86/mm: define ARCH_SUPPORTS_PER_VMA_LOCK From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, riel@surriel.com, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, surenb@google.com, kernel-team@android.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Set ARCH_SUPPORTS_PER_VMA_LOCK so that the per-VMA lock support can be compiled on this architecture. Signed-off-by: Suren Baghdasaryan --- arch/x86/Kconfig | 1 + 1 file changed, 1 insertion(+) diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig index f9920f1341c8..ee19de020b27 100644 --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -27,6 +27,7 @@ config X86_64 # Options that are inherently 64-bit kernel only: select ARCH_HAS_GIGANTIC_PAGE select ARCH_SUPPORTS_INT128 if CC_HAS_INT128 + select ARCH_SUPPORTS_PER_VMA_LOCK select ARCH_USE_CMPXCHG_LOCKREF select HAVE_ARCH_SOFT_DIRTY select MODULES_USE_ELF_RELA --=20 2.37.2.672.g94769d06f0-goog From nobody Tue Apr 7 09:06:15 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3DB77ECAAD8 for ; Mon, 29 Aug 2022 21:27:52 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229941AbiH2V1t (ORCPT ); Mon, 29 Aug 2022 17:27:49 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33002 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229918AbiH2V1H (ORCPT ); Mon, 29 Aug 2022 17:27:07 -0400 Received: from mail-pf1-x449.google.com (mail-pf1-x449.google.com [IPv6:2607:f8b0:4864:20::449]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id EFFB49F0DA for ; Mon, 29 Aug 2022 14:26:19 -0700 (PDT) Received: by mail-pf1-x449.google.com with SMTP id x25-20020aa79199000000b005358eeebf49so3609848pfa.17 for ; Mon, 29 Aug 2022 14:26:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc; bh=3FeDd/uESQnEyJqRWNnPK+iT++ObIkcBX2+m7Yxyetk=; b=WuXmy7spYby6FKlYZxVkneAX6CYLeedEW25AaCdm5U/xwDtXPaTsukbnW+BREr3wmK Yp+DdB6epKs/SB7JvycD5yPQfOWs3SBJlayjVM1bzBPF4xDdscUUlmQ491HoasCmixWz EDQApyvlKJ0m9j6IZ5Bl5eAvYAMUvQ5PzTt+MmJPi2OMUuj7wlttfCVr1jtd4buEfno/ mHPpjk9OptKM0QuTB8OPa5JPYvPxCHU8q8PlG/PC9yFoVmLZ3eeWZBp1XntU6iG84/rA BgcQG1UC8Z471LrYLx/b3SNnNzdfHV9f4k2uE0RZvNOL2F/Wod5mOLtISQc7G5F1GR1Y 7dZg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc; bh=3FeDd/uESQnEyJqRWNnPK+iT++ObIkcBX2+m7Yxyetk=; b=dY/pvieqME8M6CCzMo8hHMq7UMxLZQz+RzdFj3I9WPsEKkTL2pBTpuNCHOtykGYKoF 9he7Q7AfIWt82nXkScTsr75fPHfpK2fCfyGSWnSrv+Laoudi1gs8/YVsCiIqABOXguH9 kGab+DKH0iMqPmgE/qMwIZ8TPiAKkkCwhNjV9+b41E4F2QkfaTxdxASjDAO9XPxbZuAc PhzATRSSFiJoIiADifggCgmhI7dvCXWgg4p+8DIPFjF17WdnaQAlwOWuqBn6dxZrgBLw 0bER5sC8eH38bp8WDDNtj13hvcSmdYv0V1f1kuQcQ0StgCCcrZxOL7eR09fl27Uns8+8 5VRg== X-Gm-Message-State: ACgBeo3VhfVFg8TL4yf7UsxbKF8+2ViNxobEb6lUPLxuZ0KGcsbj5Q5T FpHq09vUWa2HlnXbvmW0uc7DkmZxwSc= X-Google-Smtp-Source: AA6agR65SMIHezE5eUhR7FLJy4sKjKwDRErJ8crUMropZEJrhJg/maBEVhsE6eunhKdLtoV+/pEsLbtUb5g= X-Received: from surenb-spec.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:3e30]) (user=surenb job=sendgmr) by 2002:a62:b519:0:b0:537:9723:5cf2 with SMTP id y25-20020a62b519000000b0053797235cf2mr18436343pfe.15.1661808377917; Mon, 29 Aug 2022 14:26:17 -0700 (PDT) Date: Mon, 29 Aug 2022 21:25:27 +0000 In-Reply-To: <20220829212531.3184856-1-surenb@google.com> Mime-Version: 1.0 References: <20220829212531.3184856-1-surenb@google.com> X-Mailer: git-send-email 2.37.2.672.g94769d06f0-goog Message-ID: <20220829212531.3184856-25-surenb@google.com> Subject: [RFC PATCH 24/28] arm64/mm: try VMA lock-based page fault handling first From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, riel@surriel.com, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, surenb@google.com, kernel-team@android.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Attempt VMA lock-based page fault handling first, and fall back to the existing mmap_lock-based handling if that fails. Signed-off-by: Suren Baghdasaryan --- arch/arm64/mm/fault.c | 36 ++++++++++++++++++++++++++++++++++++ 1 file changed, 36 insertions(+) diff --git a/arch/arm64/mm/fault.c b/arch/arm64/mm/fault.c index c33f1fad2745..f05ce40ff32b 100644 --- a/arch/arm64/mm/fault.c +++ b/arch/arm64/mm/fault.c @@ -525,6 +525,9 @@ static int __kprobes do_page_fault(unsigned long far, u= nsigned long esr, unsigned long vm_flags; unsigned int mm_flags =3D FAULT_FLAG_DEFAULT; unsigned long addr =3D untagged_addr(far); +#ifdef CONFIG_PER_VMA_LOCK + struct vm_area_struct *vma; +#endif =20 if (kprobe_page_fault(regs, esr)) return 0; @@ -575,6 +578,36 @@ static int __kprobes do_page_fault(unsigned long far, = unsigned long esr, =20 perf_sw_event(PERF_COUNT_SW_PAGE_FAULTS, 1, regs, addr); =20 +#ifdef CONFIG_PER_VMA_LOCK + if (!(mm_flags & FAULT_FLAG_USER) || atomic_read(&mm->mm_users) =3D=3D 1) + goto lock_mmap; + + vma =3D find_and_lock_anon_vma(mm, addr); + if (!vma) + goto lock_mmap; + + if (!(vma->vm_flags & vm_flags)) { + vma_read_unlock(vma); + goto lock_mmap; + } + fault =3D handle_mm_fault(vma, addr & PAGE_MASK, + mm_flags | FAULT_FLAG_VMA_LOCK, regs); + vma_read_unlock(vma); + + if (!(fault & VM_FAULT_RETRY)) { + count_vm_vma_lock_event(VMA_LOCK_SUCCESS); + goto done; + } + count_vm_vma_lock_event(VMA_LOCK_RETRY); + + /* Quick path to respond to signals */ + if (fault_signal_pending(fault, regs)) { + if (!user_mode(regs)) + goto no_context; + return 0; + } +lock_mmap: +#endif /* CONFIG_PER_VMA_LOCK */ /* * As per x86, we may deadlock here. However, since the kernel only * validly references user space from well defined areas of the code, @@ -618,6 +651,9 @@ static int __kprobes do_page_fault(unsigned long far, u= nsigned long esr, } mmap_read_unlock(mm); =20 +#ifdef CONFIG_PER_VMA_LOCK +done: +#endif /* * Handle the "normal" (no error) case first. */ --=20 2.37.2.672.g94769d06f0-goog From nobody Tue Apr 7 09:06:15 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 533C7ECAAD4 for ; Mon, 29 Aug 2022 21:28:15 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229840AbiH2V2M (ORCPT ); Mon, 29 Aug 2022 17:28:12 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60840 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229944AbiH2V1K (ORCPT ); Mon, 29 Aug 2022 17:27:10 -0400 Received: from mail-pj1-x104a.google.com (mail-pj1-x104a.google.com [IPv6:2607:f8b0:4864:20::104a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B75B29F0EB for ; Mon, 29 Aug 2022 14:26:21 -0700 (PDT) Received: by mail-pj1-x104a.google.com with SMTP id y16-20020a17090aa41000b001fdf0a76a4eso848687pjp.3 for ; Mon, 29 Aug 2022 14:26:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc; bh=M/5Y14x1SSOe03/jfupiRT6TG6dq1ngDrWV9hskObuI=; b=keg1i0nk+izk1klWl7kacG2dqLVFCBBFr7DTBR9Vkh347LyX6fgOptbA6FX9RaUao+ vJTGW+XEi+UTAE6ZSui/ACoLYvw+CzBRdshi7UOcb6GKyXUkGWluK1e9ViX0u2MOFUby n+VIqiL1KgbaHmBILZRqDKCT7yq9OcaG3eWuWfyFbeDAc3tNJSfOayUDVB763p8OZugP N6WyMveSXRyt1x9RwdvL650Tufs+g4oBOJJ5iU1MJBZM9QMXfDComRZRSTq/8D/ye7ef ZSEe/ZXHQtclALcTUMn5apyFMWCE/g3tgTBxLG+Y3u9vQ4Ps1pm7jxw7nOrCfEth2Ax0 A9EQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc; bh=M/5Y14x1SSOe03/jfupiRT6TG6dq1ngDrWV9hskObuI=; b=8L+vhLiaG5I09SNP640CmMj8yj3iWksJ8Y2EL5WpsyMe3XjdZKywC8sNoJqywIvx8/ 3GBtZn723DbZ/X7e8iS2RoAmVKMYWI+xhHos5iIvjoQJ1jfVel1kAVjRXq7YZ7Oi/I2X /hmA41E/b6z599rdCu7giHzWX3YVjiy7cka75Qldi9cfXP3cT1uOmVnP4WPnoB6wld9V NTg5iT5h/k31o1pdohiVY1+8x2mhqZn1zGsMyRxxfSa3T5REG8/F9+n+kcBmk1KkTGVg r17ImoxXKAqZP3QyofbtzTw8/WziEmqFdWDLitkcYjeFGfYR67T+mhGS7I6XbfgJQS3I geZw== X-Gm-Message-State: ACgBeo3qb091RusDWYEJFZ7N/+DgnXm9kT5l4IzsrSkmsAp2xlyLM2xO x1b2v19WctA+telW24ozkr1p74EmWco= X-Google-Smtp-Source: AA6agR4Lx1uzJZ7T/fCqu0c+3ou3bFPxSP7qIc7HtsIZjbU9yCjHgv7Z+4FP1+5TPgAu6qrR46XQccXnjfE= X-Received: from surenb-spec.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:3e30]) (user=surenb job=sendgmr) by 2002:a17:90a:e558:b0:1fb:c4b7:1a24 with SMTP id ei24-20020a17090ae55800b001fbc4b71a24mr21113pjb.1.1661808379380; Mon, 29 Aug 2022 14:26:19 -0700 (PDT) Date: Mon, 29 Aug 2022 21:25:28 +0000 In-Reply-To: <20220829212531.3184856-1-surenb@google.com> Mime-Version: 1.0 References: <20220829212531.3184856-1-surenb@google.com> X-Mailer: git-send-email 2.37.2.672.g94769d06f0-goog Message-ID: <20220829212531.3184856-26-surenb@google.com> Subject: [RFC PATCH 25/28] arm64/mm: define ARCH_SUPPORTS_PER_VMA_LOCK From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, riel@surriel.com, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, surenb@google.com, kernel-team@android.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Set ARCH_SUPPORTS_PER_VMA_LOCK so that the per-VMA lock support can be compiled on this architecture. Signed-off-by: Suren Baghdasaryan --- arch/arm64/Kconfig | 1 + 1 file changed, 1 insertion(+) diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index 9fb9fff08c94..0747ae1f3b39 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -93,6 +93,7 @@ config ARM64 select ARCH_SUPPORTS_INT128 if CC_HAS_INT128 select ARCH_SUPPORTS_NUMA_BALANCING select ARCH_SUPPORTS_PAGE_TABLE_CHECK + select ARCH_SUPPORTS_PER_VMA_LOCK select ARCH_WANT_COMPAT_IPC_PARSE_VERSION if COMPAT select ARCH_WANT_DEFAULT_BPF_JIT select ARCH_WANT_DEFAULT_TOPDOWN_MMAP_LAYOUT --=20 2.37.2.672.g94769d06f0-goog From nobody Tue Apr 7 09:06:15 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5CB17ECAAD4 for ; Mon, 29 Aug 2022 21:28:09 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230073AbiH2V2G (ORCPT ); Mon, 29 Aug 2022 17:28:06 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:34572 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229977AbiH2V1K (ORCPT ); Mon, 29 Aug 2022 17:27:10 -0400 Received: from mail-pg1-x549.google.com (mail-pg1-x549.google.com [IPv6:2607:f8b0:4864:20::549]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 695F59F1B5 for ; Mon, 29 Aug 2022 14:26:23 -0700 (PDT) Received: by mail-pg1-x549.google.com with SMTP id k62-20020a638441000000b0042b66a99b6aso4565641pgd.18 for ; Mon, 29 Aug 2022 14:26:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc; bh=o1aoXsAZRKWKp3E60vO3aTfa7LhhJu+wloCiW42SL3o=; b=hVadOQBIf7hIcCCPF4k/V8ishbMycFmV23IYK2/L+oJFUn4snhEXCB8nhiJ0XvBmGq IPVx7m/qpumur19aTHX425elTvPhBwnAWLq2eu14tCxMYLfyVDPtnXMD7eRtBYdE8ib6 g6FY2QFmIprsXVeiDN0/TJA6z8yPl6Ga6H+AU+5BAz5wvK9q2eG8K6M5/G5IT26iqvDR sF5KQOom085C4/LO0v/IjmuRTEiouanpvkUJ/WwC7Eqe/f3fgCYq7VPZ9r9nrS+kR+CY ydqosByt6O340qUe8wv/4Yn0djwvN5x2tceHdnG95lN+D/X9zA9TkMcItH0K4FkMtDqG cLTQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc; bh=o1aoXsAZRKWKp3E60vO3aTfa7LhhJu+wloCiW42SL3o=; b=v6A9chmvQp+ELHi2dDyqGF4FtldW0N0uyu1UnHDbGszi390jX54OufIbIfhbff+/bT 2irXbMdwo8OrusGu5hFXsWkgHhdRa0TW272BNHsBdmD6uarIOfOLZqloCotWaZ0Iowjh H8Y/C5AiUZqZkF//lkR052QjnStLuGj51usJiGlp22c6V7aIPLsdWFPqyX+GMjtc2r1z sq6c1wMn/FoAkKrPj3ZIxRAo2qejpC5hzVuKVWq4/NRaqxREkG/MP8gwp8AxqQRC/hGz e2n8SlJ9//QqQ7M8/BRpTEg8qdrcEGet81tJzrA7vUdCw+H+Vpy/kDPfdbNyAtaxG8OC 5HhA== X-Gm-Message-State: ACgBeo2FLzFSSQOOzZU998cKoRkBBWkHpwfjssHVFm6PIPsXFLUll5a1 1EmmkxT65bdfMOFAaz9YQkfKZTr+x0o= X-Google-Smtp-Source: AA6agR5CaUs6YJBbyF56NgpgIgB6cgapfFaHPRJdugast+FKzlkUBYXObPd4n8zhAA8UFAnqt1i7b7mpRsg= X-Received: from surenb-spec.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:3e30]) (user=surenb job=sendgmr) by 2002:a17:903:41c6:b0:174:3acf:8294 with SMTP id u6-20020a17090341c600b001743acf8294mr17683670ple.118.1661808381376; Mon, 29 Aug 2022 14:26:21 -0700 (PDT) Date: Mon, 29 Aug 2022 21:25:29 +0000 In-Reply-To: <20220829212531.3184856-1-surenb@google.com> Mime-Version: 1.0 References: <20220829212531.3184856-1-surenb@google.com> X-Mailer: git-send-email 2.37.2.672.g94769d06f0-goog Message-ID: <20220829212531.3184856-27-surenb@google.com> Subject: [RFC PATCH 26/28] powerc/mm: try VMA lock-based page fault handling first From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, riel@surriel.com, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, surenb@google.com, kernel-team@android.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Laurent Dufour Attempt VMA lock-based page fault handling first, and fall back to the existing mmap_lock-based handling if that fails. Copied from "x86/mm: try VMA lock-based page fault handling first" Signed-off-by: Laurent Dufour --- arch/powerpc/mm/fault.c | 41 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 41 insertions(+) diff --git a/arch/powerpc/mm/fault.c b/arch/powerpc/mm/fault.c index 014005428687..c92bdfcd1796 100644 --- a/arch/powerpc/mm/fault.c +++ b/arch/powerpc/mm/fault.c @@ -450,6 +450,44 @@ static int ___do_page_fault(struct pt_regs *regs, unsi= gned long address, if (is_exec) flags |=3D FAULT_FLAG_INSTRUCTION; =20 +#ifdef CONFIG_PER_VMA_LOCK + if (!(flags & FAULT_FLAG_USER) || atomic_read(&mm->mm_users) =3D=3D 1) + goto lock_mmap; + + vma =3D find_and_lock_anon_vma(mm, address); + if (!vma) + goto lock_mmap; + + if (unlikely(access_pkey_error(is_write, is_exec, + (error_code & DSISR_KEYFAULT), vma))) { + int rc =3D bad_access_pkey(regs, address, vma); + + vma_read_unlock(vma); + return rc; + } + + if (unlikely(access_error(is_write, is_exec, vma))) { + int rc =3D bad_access(regs, address); + + vma_read_unlock(vma); + return rc; + } + + fault =3D handle_mm_fault(vma, address, flags | FAULT_FLAG_VMA_LOCK, regs= ); + vma_read_unlock(vma); + + if (!(fault & VM_FAULT_RETRY)) { + count_vm_vma_lock_event(VMA_LOCK_SUCCESS); + goto done; + } + count_vm_vma_lock_event(VMA_LOCK_RETRY); + + if (fault_signal_pending(fault, regs)) + return user_mode(regs) ? 0 : SIGBUS; + +lock_mmap: +#endif /* CONFIG_PER_VMA_LOCK */ + /* When running in the kernel we expect faults to occur only to * addresses in user space. All other faults represent errors in the * kernel and should generate an OOPS. Unfortunately, in the case of an @@ -526,6 +564,9 @@ static int ___do_page_fault(struct pt_regs *regs, unsig= ned long address, =20 mmap_read_unlock(current->mm); =20 +#ifdef CONFIG_PER_VMA_LOCK +done: +#endif if (unlikely(fault & VM_FAULT_ERROR)) return mm_fault_error(regs, address, fault); =20 --=20 2.37.2.672.g94769d06f0-goog From nobody Tue Apr 7 09:06:15 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 194ACECAAD2 for ; Mon, 29 Aug 2022 21:28:17 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229970AbiH2V2P (ORCPT ); Mon, 29 Aug 2022 17:28:15 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:32840 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229802AbiH2V1M (ORCPT ); Mon, 29 Aug 2022 17:27:12 -0400 Received: from mail-pl1-x649.google.com (mail-pl1-x649.google.com [IPv6:2607:f8b0:4864:20::649]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id AD5E49F750 for ; Mon, 29 Aug 2022 14:26:24 -0700 (PDT) Received: by mail-pl1-x649.google.com with SMTP id e16-20020a17090301d000b00172fbf52e7dso6844085plh.11 for ; Mon, 29 Aug 2022 14:26:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc; bh=WpdYqLbUAWRRR7NpWeIwifObGgR3/WEcKlSas1duxWs=; b=A8iXlM7kWh/+WLxmr7Wxu633pJ33WbnreWCJkhO2mN51OTMuCRIiZbXSJTMwU7nnLp 8HrWKhSa16qOE9IHNHTIDFaumIghV/X3vZxzBwxJyopuUJchdMXN0oUY2EJvc97qw23/ 002SGSF6e/GfvGXcSJZJ9OH52bwQ+17bBv1qwUEkh+/hxiA3XGJZS3QMw2bsNQo6K8n5 ASVJIXUmfSCyDSBXlWu7+9C2NTyQxO6WdEpOrWiiN0G0XOZ0xYY++9szDB3PW4z/iVNr kXUF0efo+Jl4A1NDPbFsv2i2F8cuEbcHbyZjyx1JQrC6JX3bDooGkyCEfiAtjL3LX2kX L4Og== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc; bh=WpdYqLbUAWRRR7NpWeIwifObGgR3/WEcKlSas1duxWs=; b=KtpWuzJ8jnDu+y2GbboetjTSaMA9QWs+Xr+EMf7i+nowQIjNQ+GoQ8inWXk+4Wy9uw +j8aj9mtiy7oTtKpIy8mHaYyA6acQCiWWx9LsT3EjdF6nEhzZl1TWF2EWDLaD82w+C/7 P2ePyuavhm1rnLtZ0l6B0MX+ykbL6wsLT4ZFpriy5eGvlYmea2wIVQDMFY+Jc3w+mMAQ mWy17nNNUwzJj66G/DMXCZFDj7ULecfjMAnIf3L5zpHZdlksnpg2EVoA03cjAFIevnrF //3quUC1StQZL7QBSNszY9PSPmBuwEeRld+1dZqeb/MTXIo+jyWOps5opUUguzNuyBIH m8Gg== X-Gm-Message-State: ACgBeo1N1Do4CayCTXdY+o3ScTNpeZA5NhrdDDEEbr2rtvBg+GMaJe8i iKW+WAbF7ClV3ZGYG0iTdVNLva+D0Ew= X-Google-Smtp-Source: AA6agR7BIF7zDBFRMtaQFIX+dYsK3ioA4U218I6Y8n5CHeAbnavsVuqAEK5kMo+KsvAOcZae2RDMSpS8XIE= X-Received: from surenb-spec.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:3e30]) (user=surenb job=sendgmr) by 2002:a17:903:2d1:b0:171:3773:b95 with SMTP id s17-20020a17090302d100b0017137730b95mr17499271plk.173.1661808383315; Mon, 29 Aug 2022 14:26:23 -0700 (PDT) Date: Mon, 29 Aug 2022 21:25:30 +0000 In-Reply-To: <20220829212531.3184856-1-surenb@google.com> Mime-Version: 1.0 References: <20220829212531.3184856-1-surenb@google.com> X-Mailer: git-send-email 2.37.2.672.g94769d06f0-goog Message-ID: <20220829212531.3184856-28-surenb@google.com> Subject: [RFC PATCH 27/28] powerpc/mm: define ARCH_SUPPORTS_PER_VMA_LOCK From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, riel@surriel.com, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, surenb@google.com, kernel-team@android.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Laurent Dufour Set ARCH_SUPPORTS_PER_VMA_LOCK so that the per-VMA lock support can be compiled on powernv and pseries. It may be use on the other platforms but I can't test that seriously. Signed-off-by: Laurent Dufour --- arch/powerpc/platforms/powernv/Kconfig | 1 + arch/powerpc/platforms/pseries/Kconfig | 1 + 2 files changed, 2 insertions(+) diff --git a/arch/powerpc/platforms/powernv/Kconfig b/arch/powerpc/platform= s/powernv/Kconfig index ae248a161b43..70a46acc70d6 100644 --- a/arch/powerpc/platforms/powernv/Kconfig +++ b/arch/powerpc/platforms/powernv/Kconfig @@ -16,6 +16,7 @@ config PPC_POWERNV select PPC_DOORBELL select MMU_NOTIFIER select FORCE_SMP + select ARCH_SUPPORTS_PER_VMA_LOCK default y =20 config OPAL_PRD diff --git a/arch/powerpc/platforms/pseries/Kconfig b/arch/powerpc/platform= s/pseries/Kconfig index fb6499977f99..7d13a2de3475 100644 --- a/arch/powerpc/platforms/pseries/Kconfig +++ b/arch/powerpc/platforms/pseries/Kconfig @@ -21,6 +21,7 @@ config PPC_PSERIES select HOTPLUG_CPU select FORCE_SMP select SWIOTLB + select ARCH_SUPPORTS_PER_VMA_LOCK default y =20 config PARAVIRT_SPINLOCKS --=20 2.37.2.672.g94769d06f0-goog From nobody Tue Apr 7 09:06:15 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id ED1C6ECAAD2 for ; Mon, 29 Aug 2022 21:28:21 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230094AbiH2V2S (ORCPT ); Mon, 29 Aug 2022 17:28:18 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:32860 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229985AbiH2V1P (ORCPT ); Mon, 29 Aug 2022 17:27:15 -0400 Received: from mail-pf1-x449.google.com (mail-pf1-x449.google.com [IPv6:2607:f8b0:4864:20::449]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7EA079C50A for ; Mon, 29 Aug 2022 14:26:26 -0700 (PDT) Received: by mail-pf1-x449.google.com with SMTP id c142-20020a621c94000000b005324991c5b8so3612270pfc.15 for ; Mon, 29 Aug 2022 14:26:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc; bh=T8fB8hXi5puWeYGLUKktDcvprMtt+Dm3Upe4kr4atwA=; b=YPMK3AJpw0lWwzYZfDLzh3K7/2W07auDNNnlJ3qDkZJloq+AWVG8rnBi3EjSTi8Sdc 7WJv0veKl0lZAyzfelPpHbd7cV4CAy5dCoMcdYFQ7CqbidAfsv5F6022/J5YgvQf67DO L1YTRMppSCyvCaIew0iiAysCdKjsWOFUDjcZZGzRH+sdcxnSvug7vs9F56KVsee2c0Zo YLbMsX94AWYL+h4w9bE7JCdOjOkowcodvSgMnxQ2NHaXBQhv7Fitxu5qAmsw46U72GSa SpD2ueQh2UQYjoAvP3IZCnzfNryUU6OHaypVe7QLBtTEl+UaqGMhHtnVjouLDn/cSCG8 82Iw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc; bh=T8fB8hXi5puWeYGLUKktDcvprMtt+Dm3Upe4kr4atwA=; b=StG2vmGlfbVf2Q/ZJf8V5mQ7Z75uDI9ME1QdTReuiFIxCeKGdKCbPmhJzu+HBVOG4k /DMgbT9Hs0SlHKqJAJy9Hoo59PWihcxFFVwgovlDX6MvWMFTgh9cd76YsFRCT7k34EV5 GcqAxKmwqAw4Ear9qob5CsgfqXG/CTG3oZoaSvyMTB7hp9tdcejW2Xets+SyYt1fjFjO Fhj4tPfWzqO0w/3JGpS26IQlLqK6r2Xxwr1XEm+towvv32UwnK7c7JImMTCZguFETAYz qUpyWTvFAGAOdG0w6110/0N8sSB8ienZpoywk6R108c+mDO3H+wv/hDjbhA9C97ICCaD /wKg== X-Gm-Message-State: ACgBeo0fxwgBnCAS87aW3R8SV/TqBsR6pfuyhyy/Wt4ZKojjAyW5P7rY aosh3o16xa5ilmyoTb6pD/NfCNcuYbI= X-Google-Smtp-Source: AA6agR5dzmEiiCM1ZfmS4IP8jubPYXR49V7KmmeKiHJ1f+/Bhg9nkxqem44jNkystynSPzi1e7aYTBBx0e8= X-Received: from surenb-spec.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:3e30]) (user=surenb job=sendgmr) by 2002:aa7:8b44:0:b0:537:a35d:3c11 with SMTP id i4-20020aa78b44000000b00537a35d3c11mr18188023pfd.76.1661808384779; Mon, 29 Aug 2022 14:26:24 -0700 (PDT) Date: Mon, 29 Aug 2022 21:25:31 +0000 In-Reply-To: <20220829212531.3184856-1-surenb@google.com> Mime-Version: 1.0 References: <20220829212531.3184856-1-surenb@google.com> X-Mailer: git-send-email 2.37.2.672.g94769d06f0-goog Message-ID: <20220829212531.3184856-29-surenb@google.com> Subject: [RFC PATCH 28/28] kernel/fork: throttle call_rcu() calls in vm_area_free From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, riel@surriel.com, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, surenb@google.com, kernel-team@android.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" call_rcu() can take a long time when callback offloading is enabled. Its use in the vm_area_free can cause regressions in the exit path when multiple VMAs are being freed. To minimize that impact, place VMAs into a list and free them in groups using one call_rcu() call per group. Signed-off-by: Suren Baghdasaryan --- include/linux/mm.h | 1 + include/linux/mm_types.h | 11 ++++++- kernel/fork.c | 68 +++++++++++++++++++++++++++++++++++----- mm/init-mm.c | 3 ++ mm/mmap.c | 1 + 5 files changed, 75 insertions(+), 9 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index a3cbaa7b9119..81dff694ac14 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -249,6 +249,7 @@ void setup_initial_init_mm(void *start_code, void *end_= code, struct vm_area_struct *vm_area_alloc(struct mm_struct *); struct vm_area_struct *vm_area_dup(struct vm_area_struct *); void vm_area_free(struct vm_area_struct *); +void drain_free_vmas(struct mm_struct *mm); =20 #ifndef CONFIG_MMU extern struct rb_root nommu_region_tree; diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 36562e702baf..6f3effc493b1 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -412,7 +412,11 @@ struct vm_area_struct { struct vm_area_struct *vm_next, *vm_prev; }; #ifdef CONFIG_PER_VMA_LOCK - struct rcu_head vm_rcu; /* Used for deferred freeing. */ + struct { + struct list_head vm_free_list; + /* Used for deferred freeing. */ + struct rcu_head vm_rcu; + }; #endif }; =20 @@ -573,6 +577,11 @@ struct mm_struct { */ #ifdef CONFIG_PER_VMA_LOCK int mm_lock_seq; + struct { + struct list_head head; + spinlock_t lock; + int size; + } vma_free_list; #endif =20 =20 diff --git a/kernel/fork.c b/kernel/fork.c index b443ba3a247a..7c88710aed72 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -483,26 +483,75 @@ struct vm_area_struct *vm_area_dup(struct vm_area_str= uct *orig) } =20 #ifdef CONFIG_PER_VMA_LOCK -static void __vm_area_free(struct rcu_head *head) +static inline void __vm_area_free(struct vm_area_struct *vma) { - struct vm_area_struct *vma =3D container_of(head, struct vm_area_struct, - vm_rcu); /* The vma should either have no lock holders or be write-locked. */ vma_assert_no_reader(vma); kmem_cache_free(vm_area_cachep, vma); } -#endif + +static void vma_free_rcu_callback(struct rcu_head *head) +{ + struct vm_area_struct *first_vma; + struct vm_area_struct *vma, *vma2; + + first_vma =3D container_of(head, struct vm_area_struct, vm_rcu); + list_for_each_entry_safe(vma, vma2, &first_vma->vm_free_list, vm_free_lis= t) + __vm_area_free(vma); + __vm_area_free(first_vma); +} + +void drain_free_vmas(struct mm_struct *mm) +{ + struct vm_area_struct *first_vma; + LIST_HEAD(to_destroy); + + spin_lock(&mm->vma_free_list.lock); + list_splice_init(&mm->vma_free_list.head, &to_destroy); + mm->vma_free_list.size =3D 0; + spin_unlock(&mm->vma_free_list.lock); + + if (list_empty(&to_destroy)) + return; + + first_vma =3D list_first_entry(&to_destroy, struct vm_area_struct, vm_fre= e_list); + /* Remove the head which is allocated on the stack */ + list_del(&to_destroy); + + call_rcu(&first_vma->vm_rcu, vma_free_rcu_callback); +} + +#define VM_AREA_FREE_LIST_MAX 32 + +void vm_area_free(struct vm_area_struct *vma) +{ + struct mm_struct *mm =3D vma->vm_mm; + bool drain; + + free_anon_vma_name(vma); + + spin_lock(&mm->vma_free_list.lock); + list_add(&vma->vm_free_list, &mm->vma_free_list.head); + mm->vma_free_list.size++; + drain =3D mm->vma_free_list.size > VM_AREA_FREE_LIST_MAX; + spin_unlock(&mm->vma_free_list.lock); + + if (drain) + drain_free_vmas(mm); +} + +#else /* CONFIG_PER_VMA_LOCK */ + +void drain_free_vmas(struct mm_struct *mm) {} =20 void vm_area_free(struct vm_area_struct *vma) { free_anon_vma_name(vma); -#ifdef CONFIG_PER_VMA_LOCK - call_rcu(&vma->vm_rcu, __vm_area_free); -#else kmem_cache_free(vm_area_cachep, vma); -#endif } =20 +#endif /* CONFIG_PER_VMA_LOCK */ + static void account_kernel_stack(struct task_struct *tsk, int account) { if (IS_ENABLED(CONFIG_VMAP_STACK)) { @@ -1137,6 +1186,9 @@ static struct mm_struct *mm_init(struct mm_struct *mm= , struct task_struct *p, INIT_LIST_HEAD(&mm->mmlist); #ifdef CONFIG_PER_VMA_LOCK WRITE_ONCE(mm->mm_lock_seq, 0); + INIT_LIST_HEAD(&mm->vma_free_list.head); + spin_lock_init(&mm->vma_free_list.lock); + mm->vma_free_list.size =3D 0; #endif mm_pgtables_bytes_init(mm); mm->map_count =3D 0; diff --git a/mm/init-mm.c b/mm/init-mm.c index 8399f90d631c..7b6d2460545f 100644 --- a/mm/init-mm.c +++ b/mm/init-mm.c @@ -39,6 +39,9 @@ struct mm_struct init_mm =3D { .mmlist =3D LIST_HEAD_INIT(init_mm.mmlist), #ifdef CONFIG_PER_VMA_LOCK .mm_lock_seq =3D 0, + .vma_free_list.head =3D LIST_HEAD_INIT(init_mm.vma_free_list.head), + .vma_free_list.lock =3D __SPIN_LOCK_UNLOCKED(init_mm.vma_free_list.lock), + .vma_free_list.size =3D 0, #endif .user_ns =3D &init_user_ns, .cpu_bitmap =3D CPU_BITS_NONE, diff --git a/mm/mmap.c b/mm/mmap.c index 1edfcd384f5e..d61b7ef84ba6 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -3149,6 +3149,7 @@ void exit_mmap(struct mm_struct *mm) } mm->mmap =3D NULL; mmap_write_unlock(mm); + drain_free_vmas(mm); vm_unacct_memory(nr_accounted); } =20 --=20 2.37.2.672.g94769d06f0-goog