From nobody Sat Sep 26 01:04:33 2026 Received: from mail-pg1-f198.google.com (mail-pg1-f198.google.com [209.85.215.198]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 21E053FFFA9 for ; Mon, 7 Sep 2026 06:39:24 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.215.198 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788763166; cv=none; b=FpjfDlrnmujhs6O2Rvmx16Yz1iZ4p31HpPvSeYXl0yX5gmhlZvLW63GWBfjgOO7uKOacwYUgVLZvNvkfKX5shKxJ78xvmUiu1LC3f+ga4iKoAjY6/xXxzc8FqpFegbL0qeLFgs7WHOl9LqBBlfzeAigZbEU4qH4x+xGqewhmS3k= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788763166; c=relaxed/simple; bh=3OjcKa0V3SVb9pxz6PWF4t6frVrFTR1l1Lz6h9HZkQI=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=K0BZUmMc6Pc35gZdljI/JOdfsnxcZRC95CXMuvDmGlW4l4cdAOPHqisUaHZ+JA4AdpnmSKkkYH/5hWgIj3WHpBv5W5r8FYgsLi9/nvMSkVa6SmwE0wazhtYmadJIhknxbbTbgAgnuKpJvtpe9X1RzkJFoPMpw6OGCTNL391r2qw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--surenb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=jN0S3UDt; arc=none smtp.client-ip=209.85.215.198 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--surenb.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="jN0S3UDt" Received: by mail-pg1-f198.google.com with SMTP id 41be03b00d2f7-cb6cf425e86so3900394a12.1 for ; Sun, 06 Sep 2026 23:39:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1788763164; x=1789367964; darn=vger.kernel.org; h=content-type:cc:to:from:subject:message-id:references:mime-version :in-reply-to:date:from:to:cc:subject:date:message-id:reply-to :content-type; bh=w4B0N/rPIgypiVNXAn0XPbbBc+Skfopg6i5mY1KeQYA=; b=jN0S3UDtKmdNH2JWpgrWQpmNmEUJwMBhzTL7f3yFfcLFW6yuoZAHMrVaDTF/IV92uj 2hdwKGCVuPULttBlXvObtIcFgdzqkebJC9i3LN45bfD37P0H4GadgVyDNNNr2zIRB+pu GfHfjxZanpeNNMPm5yTjpYCpeo6ZapHBkVaT7MEYMxGvN3S/JSyT74DXLvjQteqgrC7/ WCyP3BT+o27WnhY7qkpuEd8A/dVq0q787UvRmKU15TdCGO265F6u4TzCRumldjizKpwt DlV5zVcaOtwhL0nJ+KOb70IPcTkQkM7VX0MgIsU5BeNME1HzMFmAjrwF07j0lATMm5UC db0Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1788763164; x=1789367964; h=content-type:cc:to:from:subject:message-id:references:mime-version :in-reply-to:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to:content-type; bh=w4B0N/rPIgypiVNXAn0XPbbBc+Skfopg6i5mY1KeQYA=; b=T7+/hnlJplaXbvhlQEyqeYfPiLarjh7DOx8DQinW4ymieMSl4cG2vZx76B27YkeKJc iigQrXiZehfQ8dAkNwte8zZpO2WaoG667cr8e6zUWYD2oFjDDQp1K1+EpRPIjzZuKhka V5XAAl79CgqVM0JxSp+M1qP7CQBFnbuYFdXcU/eLDsCfBYD+WSnB3pribfJ5uTa4JN5c q2I2Hc0JowyJT1SHaXF5BYIfGb87gl4qhy6vWBakdeJ/J8pIjbYptR7DBd0W5CkOkX8C XagaUTuCMzNfF4UWbtG0uCHEJYLcOh+YcdJ6ZZlamb7H+571zU7/TCmS4qSw+eF+Cb8e ujCA== X-Forwarded-Encrypted: i=1; AKwUvBzEAYblIugLOIUSLVECJWpzeJB+c36X2IvEk7MYBZUH3NRIBZfQgJcgZ4+eg742bbDy/hf3HAqvQx8zDhw=@vger.kernel.org X-Gm-Message-State: AFuF++lMC0h1lpO+f0+nLKq4hIwaE6dTdVf6CvVkiyE29H2dzdZBXZ9s P7dk4taQAAVyzRjdqeoKa6XkB6OXvdAW0axsu50KtMotCbrZbbQy1cOAS2dZ4PgG9aMiP75y5YA mp4jUHA== X-Received: from dlbro6.prod.google.com ([2002:a05:7022:1586:b0:141:ac8:b5ae]) (user=surenb job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6a20:3ca8:b0:3c4:46ca:334b with SMTP id adf61e73a8af0-3da39cf4744mr31781029637.9.1788763164089; Sun, 06 Sep 2026 23:39:24 -0700 (PDT) Date: Sun, 6 Sep 2026 23:39:14 -0700 In-Reply-To: <20260907063918.3432401-1-surenb@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20260907063918.3432401-1-surenb@google.com> X-Mailer: git-send-email 2.55.0.979.g7e5102b832-goog Message-ID: <20260907063918.3432401-2-surenb@google.com> Subject: [PATCH v2 1/5] proc/task_mmu: remove unnecessary helpers From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: liam@infradead.org, ljs@kernel.org, vbabka@kernel.org, david@redhat.com, willy@infradead.org, jannh@google.com, paulmck@kernel.org, pfalcato@suse.de, linux-mm@kvack.org, linux-kernel@vger.kernel.org, linux-fsdevel@vger.kernel.org, surenb@google.com Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" When per-vma locks were behind a config option, a number of helper functions were needed to simplify the locking code. Now that these locks are universally available, we can do a little cleanup. Remove lock_vma_range(), unlock_vma_range(), query_vma_setup(), query_vma_teardown() helpers. No functional change intended. Signed-off-by: Suren Baghdasaryan Acked-by: David Hildenbrand (Arm) Acked-by: Usama Arif Reviewed-by: Liam R. Howlett (Oracle) Reviewed-by: Lorenzo Stoakes (ARM) Tested-by: Xueyuan Chen --- fs/proc/task_mmu.c | 67 ++++++++++++---------------------------------- 1 file changed, 17 insertions(+), 50 deletions(-) diff --git a/fs/proc/task_mmu.c b/fs/proc/task_mmu.c index e671b4fd8ded..2f500d639db5 100644 --- a/fs/proc/task_mmu.c +++ b/fs/proc/task_mmu.c @@ -160,25 +160,6 @@ static void unlock_ctx_vma(struct proc_maps_locking_ct= x *lock_ctx) } } =20 -static inline bool lock_vma_range(struct seq_file *m, - struct proc_maps_locking_ctx *lock_ctx) -{ - rcu_read_lock(); - reset_lock_ctx(lock_ctx); - - return true; -} - -static inline void unlock_vma_range(struct proc_maps_locking_ctx *lock_ctx) -{ - if (lock_ctx->mmap_locked) { - unlock_ctx_mm(lock_ctx); - } else { - unlock_ctx_vma(lock_ctx); - rcu_read_unlock(); - } -} - static struct vm_area_struct *get_next_vma(struct proc_maps_private *priv, loff_t last_pos) { @@ -286,13 +267,8 @@ static void *m_start(struct seq_file *m, loff_t *ppos) return NULL; } =20 - if (!lock_vma_range(m, lock_ctx)) { - mmput(mm); - put_task_struct(priv->task); - priv->task =3D NULL; - return ERR_PTR(-EINTR); - } - + rcu_read_lock(); + reset_lock_ctx(lock_ctx); /* * Reset current position if last_addr was set before * and it's not a sentinel. @@ -325,7 +301,12 @@ static void m_stop(struct seq_file *m, void *v) return; =20 release_task_mempolicy(priv); - unlock_vma_range(&priv->lock_ctx); + if (priv->lock_ctx.mmap_locked) { + unlock_ctx_mm(&priv->lock_ctx); + } else { + unlock_ctx_vma(&priv->lock_ctx); + rcu_read_unlock(); + } mmput(mm); put_task_struct(priv->task); priv->task =3D NULL; @@ -518,21 +499,6 @@ static int pid_maps_open(struct inode *inode, struct f= ile *file) PROCMAP_QUERY_VMA_FLAGS \ ) =20 -static int query_vma_setup(struct proc_maps_locking_ctx *lock_ctx) -{ - reset_lock_ctx(lock_ctx); - - return 0; -} - -static void query_vma_teardown(struct proc_maps_locking_ctx *lock_ctx) -{ - if (lock_ctx->mmap_locked) - unlock_ctx_mm(lock_ctx); - else - unlock_ctx_vma(lock_ctx); -} - static struct vm_area_struct *query_vma_find_by_addr(struct proc_maps_lock= ing_ctx *lock_ctx, unsigned long addr) { @@ -653,12 +619,7 @@ static int do_procmap_query(struct mm_struct *mm, void= __user *uarg) if (!mm || !mmget_not_zero(mm)) return -ESRCH; =20 - err =3D query_vma_setup(&lock_ctx); - if (err) { - mmput(mm); - return err; - } - + reset_lock_ctx(&lock_ctx); vma =3D query_matching_vma(&lock_ctx, karg.query_addr, karg.query_flags); if (IS_ERR(vma)) { err =3D PTR_ERR(vma); @@ -732,7 +693,10 @@ static int do_procmap_query(struct mm_struct *mm, void= __user *uarg) vm_file =3D get_file(vma->vm_file); =20 /* unlock vma or mmap_lock, and put mm_struct before copying data to user= */ - query_vma_teardown(&lock_ctx); + if (lock_ctx.mmap_locked) + unlock_ctx_mm(&lock_ctx); + else + unlock_ctx_vma(&lock_ctx); mmput(mm); =20 if (karg.build_id_size) { @@ -773,7 +737,10 @@ static int do_procmap_query(struct mm_struct *mm, void= __user *uarg) return 0; =20 out: - query_vma_teardown(&lock_ctx); + if (lock_ctx.mmap_locked) + unlock_ctx_mm(&lock_ctx); + else + unlock_ctx_vma(&lock_ctx); mmput(mm); out_file: if (vm_file) --=20 2.55.0.979.g7e5102b832-goog From nobody Sat Sep 26 01:04:33 2026 Received: from mail-pj1-f72.google.com (mail-pj1-f72.google.com [209.85.216.72]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3D4084028E2 for ; Mon, 7 Sep 2026 06:39:26 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.72 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788763168; cv=none; b=Mbuu4sG3kl8AmLcXcv/sMbDA0PUAvQr3wJRRqpLD7JBhB6HUu4IegGexqCH9p3DlBgGfYRdH7VQwYQ0Wg0Sum+J/FCB/t2uazf5sdsPGcV810TOriqie6hkv9xenn7spDoslh0y+Hr2U1odjRk4FgXXhj0a+f9zc1hW88U9A5hg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788763168; c=relaxed/simple; bh=XvZKlwyqByri/054hcrh9tw8Y9rV224J0uA2+z0txnI=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=RZS5PZw9BtHlVAkX2nJfCmgo9gxOysFzZJQcQIDq2EZgGSNL73PNUvVWLBYCSMSYuIIR2x8sbbjrdo6jBXTf5rSEA6hbP1m0ZPpcVd5mhM8IHZFexU9SbMQowg/ziZXQDt252dQCgpP7m4gKn9d9/FX4JgT/Al4KxFaZ4HhWdmM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--surenb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=YYdN4OFK; arc=none smtp.client-ip=209.85.216.72 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--surenb.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="YYdN4OFK" Received: by mail-pj1-f72.google.com with SMTP id 98e67ed59e1d1-38e7ff7b375so4052404a91.1 for ; Sun, 06 Sep 2026 23:39:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1788763166; x=1789367966; darn=vger.kernel.org; h=content-type:cc:to:from:subject:message-id:references:mime-version :in-reply-to:date:from:to:cc:subject:date:message-id:reply-to :content-type; bh=WAyGHFgPAjMeDjtoxxSVMLam0i0jWSVnA20CIJf1N7E=; b=YYdN4OFKG57HIQBHYMhnuulPCJKBgeYSfI0ajPPMZ8wEd0wEGvKNgW7dezPfBPIurt JfntccQGotBJBdXhC8dm/djiNPc3Ls5ZgWTS6lVYgkXgOkCv/67Z8eieQf9DsOfh2Uz+ DHtNUlLUQnvGGRylSUxcOEC10W93TartqpTcy21tAfeG4FP++FmR+1ZC/FgMZBeEW3TA GNp7ipGvr4uW2mtAOfb/QefXJt4LGvLVF1sNF7xXPCcGU2PVKwXjiIEMIowYGPkjsiAI TYJH2FJYwI+qmcAZUWYngByJQfrDdgCCitx714FUPyMsSG3ZcIPx/e6KBLD4cDWjBN5x B97w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1788763166; x=1789367966; h=content-type:cc:to:from:subject:message-id:references:mime-version :in-reply-to:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to:content-type; bh=WAyGHFgPAjMeDjtoxxSVMLam0i0jWSVnA20CIJf1N7E=; b=A4sYDs4ThovHy6f/IUubwQOhLDUQ2zHgcx0hulJV/DW2+DHXpQJ92l5V9DDLfIeoIH SWu++yRCai+wdbPZnR+atTIyR/If368mLhRWkW1uU6x+30axaiDA/2tpZzMyZqqnUjBN va36ABoji88TZfbqTT8V5nuPF7haAOmrOoRvdzzPLjVJfD6Nxa57y5cG98eOeIotffOD 4Wcvr0Xlpl/GGGseiyn2g51hUiOrpPdbNCU1SbmdhFvyr6eiK7Y9JBswmAOiBqjMQ0zp fATFPbsfBhr5HiTaEZCi6ptXe2BcQdFLCXo9T6oB8/GRC3l3iM+iD8Glwa7tTad4Vk15 XeKA== X-Forwarded-Encrypted: i=1; AKwUvBz+qaokpaP2erQZlZO5xfBcjakFb11X8EkBcTkFlyE9tIX3xweWLcqpRDa59aFdBZdyPZ62uD4LggQshZk=@vger.kernel.org X-Gm-Message-State: AFuF++nUrPz0MsCpNJAxF3LY4olN6KyAek+omjTiQstXNelHcPeLMo1k BYMkFqtYaSnrGg1OnME5AMT8ReIlgeOd9MbHlZZ40PfQWInzcaYrTgU5cfKCLTtT9N8EGjy99dp g7O6K3g== X-Received: from dlea18-n2.prod.google.com ([2002:a05:701b:4212:20b0:141:4de6:26b2]) (user=surenb job=prod-delivery.src-stubby-dispatcher) by 2002:a17:90b:5246:b0:39a:e983:d4bd with SMTP id 98e67ed59e1d1-39b262a3cf7mr34335443a91.24.1788763166262; Sun, 06 Sep 2026 23:39:26 -0700 (PDT) Date: Sun, 6 Sep 2026 23:39:15 -0700 In-Reply-To: <20260907063918.3432401-1-surenb@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20260907063918.3432401-1-surenb@google.com> X-Mailer: git-send-email 2.55.0.979.g7e5102b832-goog Message-ID: <20260907063918.3432401-3-surenb@google.com> Subject: [PATCH v2 2/5] proc/task_mmu: remove unnecessary inlines in function definitions From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: liam@infradead.org, ljs@kernel.org, vbabka@kernel.org, david@redhat.com, willy@infradead.org, jannh@google.com, paulmck@kernel.org, pfalcato@suse.de, linux-mm@kvack.org, linux-kernel@vger.kernel.org, linux-fsdevel@vger.kernel.org, surenb@google.com Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" It was pointed out in the previous reviews of this code that many functions are specified as inline, which is unnecessary as the compile can make that decision by itself. Cleanup these definitions. No functional change intended. Signed-off-by: Suren Baghdasaryan Acked-by: Usama Arif Reviewed-by: Liam R. Howlett (Oracle) Reviewed-by: Lorenzo Stoakes (ARM) Tested-by: Xueyuan Chen --- fs/proc/task_mmu.c | 32 ++++++++++++++++---------------- 1 file changed, 16 insertions(+), 16 deletions(-) diff --git a/fs/proc/task_mmu.c b/fs/proc/task_mmu.c index 2f500d639db5..9908ba32f180 100644 --- a/fs/proc/task_mmu.c +++ b/fs/proc/task_mmu.c @@ -130,7 +130,7 @@ static void release_task_mempolicy(struct proc_maps_pri= vate *priv) } #endif =20 -static inline int lock_ctx_mm(struct proc_maps_locking_ctx *lock_ctx) +static int lock_ctx_mm(struct proc_maps_locking_ctx *lock_ctx) { int ret =3D mmap_read_lock_killable(lock_ctx->mm); =20 @@ -140,7 +140,7 @@ static inline int lock_ctx_mm(struct proc_maps_locking_= ctx *lock_ctx) return ret; } =20 -static inline void unlock_ctx_mm(struct proc_maps_locking_ctx *lock_ctx) +static void unlock_ctx_mm(struct proc_maps_locking_ctx *lock_ctx) { mmap_read_unlock(lock_ctx->mm); lock_ctx->mmap_locked =3D false; @@ -177,7 +177,7 @@ static struct vm_area_struct *get_next_vma(struct proc_= maps_private *priv, return vma; } =20 -static inline bool fallback_to_mmap_lock(struct proc_maps_private *priv, +static bool fallback_to_mmap_lock(struct proc_maps_private *priv, loff_t pos) { struct proc_maps_locking_ctx *lock_ctx =3D &priv->lock_ctx; @@ -194,7 +194,7 @@ static inline bool fallback_to_mmap_lock(struct proc_ma= ps_private *priv, return true; } =20 -static inline void drop_rcu(struct proc_maps_private *priv) +static void drop_rcu(struct proc_maps_private *priv) { if (priv->lock_ctx.mmap_locked) return; @@ -202,7 +202,7 @@ static inline void drop_rcu(struct proc_maps_private *p= riv) rcu_read_unlock(); } =20 -static inline void reacquire_rcu(struct proc_maps_private *priv) +static void reacquire_rcu(struct proc_maps_private *priv) { if (priv->lock_ctx.mmap_locked) return; @@ -1230,7 +1230,7 @@ static const struct mm_walk_ops smaps_shmem_walk_vma_= lock_ops =3D { .walk_lock =3D PGWALK_VMA_RDLOCK_VERIFY, }; =20 -static inline const struct mm_walk_ops * +static const struct mm_walk_ops * get_smaps_walk_ops(struct proc_maps_private *priv) { if (priv->lock_ctx.mmap_locked) @@ -1238,7 +1238,7 @@ get_smaps_walk_ops(struct proc_maps_private *priv) return &smaps_walk_vma_lock_ops; } =20 -static inline const struct mm_walk_ops * +static const struct mm_walk_ops * get_smaps_shmem_walk_ops(struct proc_maps_private *priv) { if (priv->lock_ctx.mmap_locked) @@ -1572,7 +1572,7 @@ struct clear_refs_private { enum clear_refs_types type; }; =20 -static inline bool pte_is_pinned(struct vm_area_struct *vma, unsigned long= addr, pte_t pte) +static bool pte_is_pinned(struct vm_area_struct *vma, unsigned long addr, = pte_t pte) { struct folio *folio; =20 @@ -1588,8 +1588,8 @@ static inline bool pte_is_pinned(struct vm_area_struc= t *vma, unsigned long addr, return folio_maybe_dma_pinned(folio); } =20 -static inline void clear_soft_dirty(struct vm_area_struct *vma, - unsigned long addr, pte_t *pte) +static void clear_soft_dirty(struct vm_area_struct *vma, unsigned long add= r, + pte_t *pte) { if (!pgtable_supports_soft_dirty()) return; @@ -1620,8 +1620,8 @@ static inline void clear_soft_dirty(struct vm_area_st= ruct *vma, } =20 #if defined(CONFIG_TRANSPARENT_HUGEPAGE) -static inline void clear_soft_dirty_pmd(struct vm_area_struct *vma, - unsigned long addr, pmd_t *pmdp) +static void clear_soft_dirty_pmd(struct vm_area_struct *vma, + unsigned long addr, pmd_t *pmdp) { pmd_t old, pmd =3D *pmdp; =20 @@ -1646,8 +1646,8 @@ static inline void clear_soft_dirty_pmd(struct vm_are= a_struct *vma, } } #else -static inline void clear_soft_dirty_pmd(struct vm_area_struct *vma, - unsigned long addr, pmd_t *pmdp) +static void clear_soft_dirty_pmd(struct vm_area_struct *vma, + unsigned long addr, pmd_t *pmdp) { } #endif @@ -1846,7 +1846,7 @@ struct pagemapread { =20 #define PM_END_OF_BUFFER 1 =20 -static inline pagemap_entry_t make_pme(u64 frame, u64 flags) +static pagemap_entry_t make_pme(u64 frame, u64 flags) { return (pagemap_entry_t) { .pme =3D (frame & PM_PFRAME_MASK) | flags }; } @@ -3388,7 +3388,7 @@ static const struct mm_walk_ops show_numa_vma_lock_op= s =3D { .walk_lock =3D PGWALK_VMA_RDLOCK_VERIFY, }; =20 -static inline const struct mm_walk_ops * +static const struct mm_walk_ops * get_show_numa_ops(struct proc_maps_private *priv) { if (priv->lock_ctx.mmap_locked) --=20 2.55.0.979.g7e5102b832-goog From nobody Sat Sep 26 01:04:33 2026 Received: from mail-pj1-f71.google.com (mail-pj1-f71.google.com [209.85.216.71]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 57CFE403EB5 for ; Mon, 7 Sep 2026 06:39:29 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.71 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788763170; cv=none; b=joDxi273BFqQhzKzRc45RfUW+zeldA06FikzgWWTu2tcOllOqDI50XnfoEtdpiDvvyM5UhblxAdtJK0cVQYio8zKM3ObmjXU9BYnq6sZQAF/ZCUNzV+FB1j9C0Vtxcr20T0k00K2WIU1CIjETloE4WWEubh4IWSLRNL5Gog+eTA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788763170; c=relaxed/simple; bh=vys6AT7bnyxQQxULzvDgCzmeOLcybKvobkQ/3Vu4NVA=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=e5fBz0wxezKcUP2eDPx48bwWbqEhYVxpzo/jLJhSHQ0fWkz5PCCs3BrQuep4UL0jk7w5xsqdoYwweCHJ69U8upJrZO/chx8QiesELKRZIgd1eGjsdyIlnb5qoYp4tSX3xZwqvw52U5NtFeWrAYxjUzm0rTrnHXxflWce71oBT6Y= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--surenb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=CNMWYCpT; arc=none smtp.client-ip=209.85.216.71 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--surenb.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="CNMWYCpT" Received: by mail-pj1-f71.google.com with SMTP id 98e67ed59e1d1-39b6416441eso1585654a91.1 for ; Sun, 06 Sep 2026 23:39:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1788763168; x=1789367968; darn=vger.kernel.org; h=content-type:cc:to:from:subject:message-id:references:mime-version :in-reply-to:date:from:to:cc:subject:date:message-id:reply-to :content-type; bh=l82YwPFJoqBct/wpgBVxgb9B5/Qm0auI8Z1CtPsM/dw=; b=CNMWYCpTbZVDpPjAv2vErU87jRba4RASpQbaGg6ib7zNPrmnP5VEsp3jThCRjU9bmI 4tXq4jk1ZmoIZUp8VV7y3e33fty0deBq2TYbnO4/iqC/Ox9chcsLthZMYk2yoSONc38X BXPFsIsesa9IUhzqP8EpQAXKwwKS1rrBg0Wu92ivpslEjKMFCtLhgeHJ0xxxcYFa2E6O 2/x9h1VksO6flvT23NJohvSnIJD1NSa8oxqvwGOE+az0Tzy9EMf658cZa2tkt/QbLwtZ uC9dW/H5AF7/iiK3h+s9mhBsplnm+hq3VFjKnkr6VPPIdq7feft++TA1WySWl4EVXA2u w9Ng== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1788763168; x=1789367968; h=content-type:cc:to:from:subject:message-id:references:mime-version :in-reply-to:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to:content-type; bh=l82YwPFJoqBct/wpgBVxgb9B5/Qm0auI8Z1CtPsM/dw=; b=sRRADMTud02zprHhUpQ94mKHR4BbchAK49eM3b45GeAIV29f9f6QJjPgRP3hChDykW +XS5HGVt31x8r0iVVvDUo9pfxmjnTCBHQCchJFweOQZopg20fLdYGjgaHiGVC5ziuDyz y9DQwdlCnr+UCWpExL6DWvy7DLI82hYP24IhpydWH+T2vyecsqgu4aL1EqO+ooSU21yl Sdp/21+N206uUQqHfNKtFqBNeCWLxJrusu4TeiMO/w1n0SMlCJCv7kUrt+kS9ulbCF+f JI4+7jHXGQy5y6BkhdCrjK/d/tBrGbsJmGU8mwu+QvnTQHcsfNyrBzbLpLnz0CLK+sTs 08gw== X-Forwarded-Encrypted: i=1; AKwUvBzjjaZAecGZ0AZ486/pH2RRnEyrqOqA7p8ME9JApD8TZTdH0u+AFiEIXvGtePrV6jyRiB2ndMou5rDH6eo=@vger.kernel.org X-Gm-Message-State: AFuF++k1pWZiJ9uPExMmEeTa7a+DNKyNCmDaq0QTpvUvD6H9bNoBV1RS P+mQgKvDYZsZaDFvXJFuGGAYURweRZIcfUH8QKKISVTY/mMu7jDylRyfRC6EFWVsIlGfSfQwdI5 tkm/SIw== X-Received: from dyjo27.prod.google.com ([2002:a05:7300:339b:b0:327:c2c1:f06f]) (user=surenb job=prod-delivery.src-stubby-dispatcher) by 2002:a17:90b:5866:b0:398:e73e:5a13 with SMTP id 98e67ed59e1d1-39b2613272amr31362696a91.9.1788763168352; Sun, 06 Sep 2026 23:39:28 -0700 (PDT) Date: Sun, 6 Sep 2026 23:39:16 -0700 In-Reply-To: <20260907063918.3432401-1-surenb@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20260907063918.3432401-1-surenb@google.com> X-Mailer: git-send-email 2.55.0.979.g7e5102b832-goog Message-ID: <20260907063918.3432401-4-surenb@google.com> Subject: [PATCH v2 3/5] proc/task_mmu: remove special-casing of smap_gather_stats() start parameter From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: liam@infradead.org, ljs@kernel.org, vbabka@kernel.org, david@redhat.com, willy@infradead.org, jannh@google.com, paulmck@kernel.org, pfalcato@suse.de, linux-mm@kvack.org, linux-kernel@vger.kernel.org, linux-fsdevel@vger.kernel.org, surenb@google.com Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" smap_gather_stats() interprets its start parameter to mean vma->vm_start when it's set to 0. Eliminate this special interpretation and pass vma->vm_start explicitly when needed. Since smap_gather_stats() operates within a single VMA, we can replace walk_page_vma()/walk_page_range() calls with walk_page_range_vma() which is simpler and also can be called while holding per-VMA lock. No functional change intended. Suggested by: Lorenzo Stoakes Signed-off-by: Suren Baghdasaryan Reviewed-by: Liam R. Howlett (Oracle) Tested-by: Xueyuan Chen --- fs/proc/task_mmu.c | 40 ++++++++++++++++++++++------------------ 1 file changed, 22 insertions(+), 18 deletions(-) diff --git a/fs/proc/task_mmu.c b/fs/proc/task_mmu.c index 9908ba32f180..3351decd1172 100644 --- a/fs/proc/task_mmu.c +++ b/fs/proc/task_mmu.c @@ -1246,20 +1246,27 @@ get_smaps_shmem_walk_ops(struct proc_maps_private *= priv) return &smaps_shmem_walk_vma_lock_ops; } =20 -/* - * Gather mem stats from @vma with the indicated beginning - * address @start, and keep them in @mss. +/** + * smap_gather_stats() - Gather mem stats from @vma. + * @priv: proc maps private state. + * @vma: The VMA to gather stats for. + * @mss: The accumulated stats. + * @start: The address from which to start. * - * Use vm_start of @vma as the beginning address if @start is 0. + * This gathers stats for the whole of the VMA unless the lock was dropped + * and VMA grew or got merged and we found it again, in which case we only + * gather stats for the remainder of the VMA range. */ static void smap_gather_stats(struct proc_maps_private *priv, struct vm_area_struct *vma, - struct mem_size_stats *mss, unsigned long start) + struct mem_size_stats *mss, + unsigned long start) { const struct mm_walk_ops *ops =3D get_smaps_walk_ops(priv); + const bool is_partial =3D start > vma->vm_start; =20 /* Invalid start */ - if (start >=3D vma->vm_end) + if (start < vma->vm_start || start >=3D vma->vm_end) return; =20 if (vma =3D=3D get_gate_vma(priv->lock_ctx.mm)) @@ -1279,20 +1286,17 @@ static void smap_gather_stats(struct proc_maps_priv= ate *priv, * Unless we know that the shmem object (or the part mapped by * our VMA) has no swapped out pages at all. */ - unsigned long shmem_swapped =3D shmem_swap_usage(vma); + const unsigned long shmem_swapped =3D shmem_swap_usage(vma); + const bool shared_or_ro =3D vma_test(vma, VMA_SHARED_BIT) || + !vma_test(vma, VMA_WRITE_BIT); =20 - if (!start && (!shmem_swapped || (vma->vm_flags & VM_SHARED) || - !(vma->vm_flags & VM_WRITE))) { + if (!is_partial && (!shmem_swapped || shared_or_ro)) mss->swap +=3D shmem_swapped; - } else { + else ops =3D get_smaps_shmem_walk_ops(priv); - } } =20 - if (!start) - walk_page_vma(vma, ops, mss); - else - walk_page_range(vma->vm_mm, start, vma->vm_end, ops, mss); + walk_page_range_vma(vma, start, vma->vm_end, ops, mss); =20 reacquire_rcu(priv); } @@ -1347,7 +1351,7 @@ static int show_smap(struct seq_file *m, void *v) struct vm_area_struct *vma =3D v; struct mem_size_stats mss =3D {}; =20 - smap_gather_stats(priv, vma, &mss, 0); + smap_gather_stats(priv, vma, &mss, vma->vm_start); =20 show_map_vma(m, vma); =20 @@ -1400,7 +1404,7 @@ static int show_smaps_rollup(struct seq_file *m, void= *v) =20 vma_start =3D vma->vm_start; do { - smap_gather_stats(priv, vma, &mss, 0); + smap_gather_stats(priv, vma, &mss, vma->vm_start); last_vma_end =3D vma->vm_end; =20 /* @@ -1459,7 +1463,7 @@ static int show_smaps_rollup(struct seq_file *m, void= *v) =20 /* Case 1 and 2 above */ if (vma->vm_start >=3D last_vma_end) { - smap_gather_stats(priv, vma, &mss, 0); + smap_gather_stats(priv, vma, &mss, vma->vm_start); last_vma_end =3D vma->vm_end; continue; } --=20 2.55.0.979.g7e5102b832-goog From nobody Sat Sep 26 01:04:33 2026 Received: from mail-pj1-f71.google.com (mail-pj1-f71.google.com [209.85.216.71]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 77AD0211A09 for ; Mon, 7 Sep 2026 06:39:31 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.71 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788763173; cv=none; b=ppwTNVZecxReYAPbRdxooxxMQgX8F5RZ8NO6bvqKxNUstqv7l3mk4epW8oS/Fzp3KxYnF3Qozux1GUycelAEVTh9iEC4jn5pod1HCslkMx65ZxWjBFos1hkTLGjea7ofnNPSKr4YCkbVKw8wJ7nOp5nYyA8Q07XKn2SZa2OF520= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788763173; c=relaxed/simple; bh=MT5QQxPfK/3q54Cew+r4LE8jvjX183O0yXfDE6L9Les=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=lbSIDdsBPFwsPpu1bpL1Sbxdh6bXfp40GutTTMJSEeqYWVlv04OlSkOtm9RnSlf4bIuM+ruNAOf+Wqp+89EvgddYzRAW6CYIU0UmhRIGPyS/abASvpZaMgjg+cNK9o7EjwKYkHrJCdHoKPHcytA/3Fdtsbe9yAuMXvg9g+Vva/k= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--surenb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=WZ4b6sFR; arc=none smtp.client-ip=209.85.216.71 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--surenb.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="WZ4b6sFR" Received: by mail-pj1-f71.google.com with SMTP id 98e67ed59e1d1-38f283baf1fso3873038a91.3 for ; Sun, 06 Sep 2026 23:39:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1788763171; x=1789367971; darn=vger.kernel.org; h=content-type:cc:to:from:subject:message-id:references:mime-version :in-reply-to:date:from:to:cc:subject:date:message-id:reply-to :content-type; bh=hRgAJEesmnVK4x+kJuwts5FXMtr9BAdCkMEInHLpvrU=; b=WZ4b6sFRBMuI2h2fHacsFggjm250NomujOv9980SWJqdTjZEr5UdwuC9rZs+Tb9hGB orL2B4iEf7HEPvLR+evqZc/4NBlMvFEbRlfCqk/ztwwkRjHGIAqBBa5Clq7wBplmDall CgacXpMxuuktXq2zrbHn4TLg9AK28LGcvta5V3uuCjIdNINMdoRZPPr4s3X8eDMG0OSb oiWnuaaoXK92uSWvKf6r629V9r8dBfgf+DyG0vUqQf9iDXl14+wtlPGU62uHi8PlZwqH BsfrZcsNhrih2TngqjSGHpMbnasRIrLCvg+4x36OTXap+Zx4DOiPlAVYLVwdaZ73N+XN uoWg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1788763171; x=1789367971; h=content-type:cc:to:from:subject:message-id:references:mime-version :in-reply-to:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to:content-type; bh=hRgAJEesmnVK4x+kJuwts5FXMtr9BAdCkMEInHLpvrU=; b=IKnCtaI4fzhsw9kPAeHCpThmfbL0jo43YaYHMKrSP/lQp8Gh5sdcGeq4c1wNNOIb+W 8oQpmBULxExfeC1/OoxLHZFO9x7a9nWyRbp336q15NjY+obUxIs0q6xGIwZGz5UfZ4S+ slfyz3FZaVBQ6TnReKK1J6OjY9Aubg6qToZFOR23H4LF6QQQ318yzhnTW4PapFRGp73+ aP6vfmHVOOqItsc9HzrJ9S2QImQGEXF+nECRF61pz8ostGD1csLmaIUgirQz6jct/BLu bhstczEzuO7P0Ky9PeW8wMci8jaq0e2/Y4F9Wqloii3FWdIVzSTuBeL9F3VDBRrX0+Fh AnxA== X-Forwarded-Encrypted: i=1; AKwUvBzlsS2/iwLdpcGDbYjEBiLPbHdcyspdpAimeDFDIumYZNfeE8Dh4CCyxuyHePSpyZwsr9tPMx1tIpt7+pA=@vger.kernel.org X-Gm-Message-State: AFuF++mneQ3kmHg6hJnwRHbVpEH97VGB1IRFYcHR+yRN57HDa2hj7RR3 EfpH+Lwrh5qfeA/nyySDGpv/6Mf8Z+mHgh+ierdeTau4bl64nBJB7PYdD9U+l62EsuOwpBiQUN+ GG1Zq2Q== X-Received: from dlec19-n2.prod.google.com ([2002:a05:701b:4293:20b0:13d:2ef5:67a6]) (user=surenb job=prod-delivery.src-stubby-dispatcher) by 2002:a17:90b:4c06:b0:396:b98b:a3c2 with SMTP id 98e67ed59e1d1-39b26116d9fmr27528420a91.8.1788763170216; Sun, 06 Sep 2026 23:39:30 -0700 (PDT) Date: Sun, 6 Sep 2026 23:39:17 -0700 In-Reply-To: <20260907063918.3432401-1-surenb@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20260907063918.3432401-1-surenb@google.com> X-Mailer: git-send-email 2.55.0.979.g7e5102b832-goog Message-ID: <20260907063918.3432401-5-surenb@google.com> Subject: [PATCH v2 4/5] proc/task_mmu: read proc/pid/smaps_rollup under per-vma lock From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: liam@infradead.org, ljs@kernel.org, vbabka@kernel.org, david@redhat.com, willy@infradead.org, jannh@google.com, paulmck@kernel.org, pfalcato@suse.de, linux-mm@kvack.org, linux-kernel@vger.kernel.org, linux-fsdevel@vger.kernel.org, surenb@google.com Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" proc/pid/smaps_rollup can be read using the combination of RCU and VMA read locks, similar to proc/pid/{maps|smaps|numa_maps}. RCU is required to safely traverse the VMA tree and VMA lock stabilizes the VMA being processed and the pagetable walk. Note that we have to keep the logic to drop mmap_lock on contention because even when using per-VMA locks we might have to fall back to holding the mmap_lock. Running Paul's contention benchmark [1] shows considerable improvement both in median and in the worst case latencies: Execution command: run-proc-vs-map.sh --nsamples 20 --rawdata -- \ --busyduration 2 --procfile smaps_rollup Baseline: Median Minimum Maximum 0.174 0.161 2.553 0.174 0.164 2.663 0.174 0.165 2.664 0.174 0.166 2.679 0.174 0.167 2.691 0.174 0.168 2.704 0.174 0.169 2.729 0.174 0.172 2.741 0.174 0.174 2.745 0.174 0.174 2.755 0.174 0.175 2.790 0.174 0.177 2.809 0.174 0.179 3.096 0.174 0.183 3.144 0.174 0.184 3.158 0.174 0.185 3.175 0.174 0.185 4.568 0.174 0.198 4.821 0.174 0.214 5.143 0.174 0.251 5.220 Patched: Median Minimum Maximum 0.007 0.007 1.952 0.007 0.007 1.955 0.007 0.007 1.955 0.007 0.007 1.955 0.007 0.007 1.957 0.007 0.007 1.969 0.007 0.007 2.065 0.007 0.007 2.075 0.007 0.007 2.146 0.007 0.007 2.195 0.007 0.007 2.223 0.007 0.007 2.259 0.007 0.007 2.488 0.007 0.007 2.562 0.007 0.007 2.599 0.007 0.007 2.697 0.007 0.007 3.030 0.007 0.007 3.075 0.007 0.007 3.145 0.007 0.007 3.225 Remove now unused lock_ctx_mm() and move unlock_ctx_vma() next to unlock_ctx_mm() as they are logically related. Remove a long comment about 4 cases that we handle when dropping the mmap lock in the middle of VMA walk due to contention. The first 3 cases explained there are handled naturally and only case 4 needs to be handled in a special way, which is done in smap_gather_stats() by gathering stats from the portion of the VMA that has not yet been processed. For posterity, moving this comment here: After dropping the lock, there are four cases to consider. See the following example for explanation. +------+------+-----------+ | VMA1 | VMA2 | VMA3 | +------+------+-----------+ | | | | 4k 8k 16k 400k Suppose we drop the lock after reading VMA2 due to contention, then we get: last_vma_end =3D 16k 1) VMA2 is freed, but VMA3 exists: vma_next(vmi) will return VMA3. In this case, just continue from VMA3. 2) VMA2 still exists: vma_next(vmi) will return VMA3. In this case, just continue from VMA3. 3) No more VMAs can be found: vma_next(vmi) will return NULL. No more things to do, just break. 4) (last_vma_end - 1) is the middle of a vma (VMA'): vma_next(vmi) will return VMA' whose range contains last_vma_end. Iterate VMA' from last_vma_end. [1] https://github.com/paulmckrcu/proc-mmap_sem-test Signed-off-by: Suren Baghdasaryan Reviewed-by: Liam R. Howlett (Oracle) Tested-by: Xueyuan Chen --- fs/proc/task_mmu.c | 153 ++++++++++++++++++--------------------------- 1 file changed, 60 insertions(+), 93 deletions(-) diff --git a/fs/proc/task_mmu.c b/fs/proc/task_mmu.c index 3351decd1172..641a155b0c61 100644 --- a/fs/proc/task_mmu.c +++ b/fs/proc/task_mmu.c @@ -130,28 +130,12 @@ static void release_task_mempolicy(struct proc_maps_p= rivate *priv) } #endif =20 -static int lock_ctx_mm(struct proc_maps_locking_ctx *lock_ctx) -{ - int ret =3D mmap_read_lock_killable(lock_ctx->mm); - - if (!ret) - lock_ctx->mmap_locked =3D true; - - return ret; -} - static void unlock_ctx_mm(struct proc_maps_locking_ctx *lock_ctx) { mmap_read_unlock(lock_ctx->mm); lock_ctx->mmap_locked =3D false; } =20 -static void reset_lock_ctx(struct proc_maps_locking_ctx *lock_ctx) -{ - lock_ctx->locked_vma =3D NULL; - lock_ctx->mmap_locked =3D false; -} - static void unlock_ctx_vma(struct proc_maps_locking_ctx *lock_ctx) { if (lock_ctx->locked_vma) { @@ -160,6 +144,12 @@ static void unlock_ctx_vma(struct proc_maps_locking_ct= x *lock_ctx) } } =20 +static void reset_lock_ctx(struct proc_maps_locking_ctx *lock_ctx) +{ + lock_ctx->locked_vma =3D NULL; + lock_ctx->mmap_locked =3D false; +} + static struct vm_area_struct *get_next_vma(struct proc_maps_private *priv, loff_t last_pos) { @@ -1376,12 +1366,14 @@ static int show_smap(struct seq_file *m, void *v) static int show_smaps_rollup(struct seq_file *m, void *v) { struct proc_maps_private *priv =3D m->private; + struct proc_maps_locking_ctx *lock_ctx =3D &priv->lock_ctx; + struct mm_struct *mm =3D lock_ctx->mm; struct mem_size_stats mss =3D {}; - struct mm_struct *mm =3D priv->lock_ctx.mm; + unsigned long last_vma_end =3D 0; + unsigned long vma_start =3D 0; struct vm_area_struct *vma; - unsigned long vma_start =3D 0, last_vma_end =3D 0; + loff_t pos =3D 0; int ret =3D 0; - VMA_ITERATOR(vmi, mm, 0); =20 priv->task =3D get_proc_task(priv->inode); if (!priv->task) @@ -1392,89 +1384,60 @@ static int show_smaps_rollup(struct seq_file *m, vo= id *v) goto out_put_task; } =20 - ret =3D lock_ctx_mm(&priv->lock_ctx); - if (ret) - goto out_put_mm; - hold_task_mempolicy(priv); - vma =3D vma_next(&vmi); + rcu_read_lock(); + reset_lock_ctx(lock_ctx); =20 + vma_iter_init(&priv->iter, mm, 0); + vma =3D proc_get_vma(m, &pos); if (unlikely(!vma)) goto empty_set; =20 - vma_start =3D vma->vm_start; - do { - smap_gather_stats(priv, vma, &mss, vma->vm_start); - last_vma_end =3D vma->vm_end; + if (!IS_ERR(vma) && vma !=3D get_gate_vma(lock_ctx->mm)) + vma_start =3D vma->vm_start; + + while (vma) { + if (IS_ERR(vma)) { + ret =3D PTR_ERR(vma); + goto out_unlock; + } + + if (vma =3D=3D get_gate_vma(lock_ctx->mm)) + break; =20 /* - * Release mmap_lock temporarily if someone wants to - * access it for write request. + * If after retaking the lock, already reported VMA grew or + * merged with the next one, smap_gather_stats() will gather + * stats for the remaining portion by starting at last_vma_end. */ - if (mmap_lock_is_contended(mm)) { - vma_iter_invalidate(&vmi); - unlock_ctx_mm(&priv->lock_ctx); - ret =3D lock_ctx_mm(&priv->lock_ctx); - if (ret) { - release_task_mempolicy(priv); - goto out_put_mm; - } + smap_gather_stats(priv, vma, &mss, last_vma_end); + last_vma_end =3D vma->vm_end; =20 + /* + * If the VMA lock is not taken, we hold the often contended + * mmap lock. This can happen if we had to fall back to the + * mmap lock. + * + * To relieve pressure, check if it is indeed contended, then + * temporarily release it. + */ + if (lock_ctx->mmap_locked && + mmap_lock_is_contended(lock_ctx->mm)) { + unlock_ctx_mm(lock_ctx); /* - * After dropping the lock, there are four cases to - * consider. See the following example for explanation. - * - * +------+------+-----------+ - * | VMA1 | VMA2 | VMA3 | - * +------+------+-----------+ - * | | | | - * 4k 8k 16k 400k - * - * Suppose we drop the lock after reading VMA2 due to - * contention, then we get: - * - * last_vma_end =3D 16k - * - * 1) VMA2 is freed, but VMA3 exists: - * - * vma_next(vmi) will return VMA3. - * In this case, just continue from VMA3. - * - * 2) VMA2 still exists: - * - * vma_next(vmi) will return VMA3. - * In this case, just continue from VMA3. - * - * 3) No more VMAs can be found: - * - * vma_next(vmi) will return NULL. - * No more things to do, just break. - * - * 4) (last_vma_end - 1) is the middle of a vma (VMA'): - * - * vma_next(vmi) will return VMA' whose range - * contains last_vma_end. - * Iterate VMA' from last_vma_end. + * Even though we previously fell back to mmap lock, + * we try taking VMA lock for the next VMA, since it + * might not be under modification. In the worst case + * we will fall back to mmap lock again. */ - vma =3D vma_next(&vmi); - /* Case 3 above */ - if (!vma) - break; - - /* Case 1 and 2 above */ - if (vma->vm_start >=3D last_vma_end) { - smap_gather_stats(priv, vma, &mss, vma->vm_start); - last_vma_end =3D vma->vm_end; - continue; - } - - /* Case 4 above */ - if (vma->vm_end > last_vma_end) { - smap_gather_stats(priv, vma, &mss, last_vma_end); - last_vma_end =3D vma->vm_end; - } + rcu_read_lock(); + reset_lock_ctx(lock_ctx); + /* Resume from the last position. */ + pos =3D last_vma_end; + vma_iter_init(&priv->iter, mm, pos); } - } for_each_vma(vmi, vma); + vma =3D proc_get_vma(m, &pos); + } =20 empty_set: show_vma_header_prefix(m, vma_start, last_vma_end, 0, 0, 0, 0); @@ -1483,10 +1446,14 @@ static int show_smaps_rollup(struct seq_file *m, vo= id *v) =20 __show_smap(m, &mss, true); =20 +out_unlock: + if (lock_ctx->mmap_locked) { + unlock_ctx_mm(lock_ctx); + } else { + unlock_ctx_vma(lock_ctx); + rcu_read_unlock(); + } release_task_mempolicy(priv); - unlock_ctx_mm(&priv->lock_ctx); - -out_put_mm: mmput(mm); out_put_task: put_task_struct(priv->task); --=20 2.55.0.979.g7e5102b832-goog From nobody Sat Sep 26 01:04:33 2026 Received: from mail-pg1-f197.google.com (mail-pg1-f197.google.com [209.85.215.197]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4A86F3FF1D0 for ; Mon, 7 Sep 2026 06:39:33 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.215.197 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788763175; cv=none; b=dlti4fE57EcaxoOZ8byHTk07nMxHEkvU0qguTefCOqMCoevJF+AyCqk2DfbbYtt5PnROSRtefmx+iABhgGazyknhOXAUPlBLK5fb+hs3FH8prctmtuaIvA5YBEnN9NaJzegnRTaVUqHI3ieZ3REOUAenUsC8JMw9A+i2jVvxlS0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788763175; c=relaxed/simple; bh=KtiDqItAIqcOo5yX7TKSo5IrBmn9uko0OiM/lrd3JH8=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=B48OKk7GKHwxRsS9psTmgCB89yKKwb7UqbA0q8A2X7DrixkPFWE6kJhff3HtgqFgOaKf2jAEsfOFgqQPHB+IC5z9uVn7ILeKn0vAysKkrtFCMamgRtgQgbsbBTKGC5j9gtOc3kRpfIM4TPMgoT6i3j4OwYr5ZQyY/aoA6gH0r5g= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--surenb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=He8mhBPT; arc=none smtp.client-ip=209.85.215.197 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--surenb.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="He8mhBPT" Received: by mail-pg1-f197.google.com with SMTP id 41be03b00d2f7-cc18ced1a5aso5823411a12.1 for ; Sun, 06 Sep 2026 23:39:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1788763173; x=1789367973; darn=vger.kernel.org; h=content-type:cc:to:from:subject:message-id:references:mime-version :in-reply-to:date:from:to:cc:subject:date:message-id:reply-to :content-type; bh=T7MPdcOKwaU52QY/pYkhfSWrRgRTHQ7GCQeg5bNkckA=; b=He8mhBPT8AYaXLpJVm6fIzABnjlPCchRHX1MZohICs8TN56jeuSnKM7oJIpzqYIqKA foLMPS/8pe5t8XZVDQOZcRHyc6o+/mggwBKyjJdw6FYy9sO/IAcZTo8hVaJzU8NP76Yh qGf/ZaRzqqmvACD4tGrekDzrUooKGvCyp1uI/6mkv6SqJhM+j2LitXjsHDI/OqQM+liP 6VjQFDaFuntvrBiKyE0nEA2JrvTPcBfa3GFM/fu0joRTrOcb+NM+4QWsr7+pNRGEtJ6N t6lK2/kzcyNCsKrP8t87jJ3TuLIWVsUPcH3uiO1+Ie9+vm8xx8n6/rtIpY6t2zOGocmi 5h/g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1788763173; x=1789367973; h=content-type:cc:to:from:subject:message-id:references:mime-version :in-reply-to:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to:content-type; bh=T7MPdcOKwaU52QY/pYkhfSWrRgRTHQ7GCQeg5bNkckA=; b=AxJBpQir1mzPXKa7JkpkE43hblHVRIhxqNiO2HobcRB8nr1itA8A3Cl0vL4ceYa1Ll CpBNfMf787PkOIj6EMa7WUfnyitYYI2uKgu6b+dgxt5oDpFCwTKjNGfG1jNK0H/IbIYd Q6EYuDUI3+AYUbC2jqaEG+QkRiLzi+QMLv7FtAxnCSMnIK6zAXcRKccd0lJ2/dx1mLlM NRYWng84sUCdA0VLVWBQCXwB3Yu9E9vUns4ZdRDZsVkDVYJjuBGWHOU2ULFxzwn43wLt YULXV1m1ecogUHZEi4yc2HjQVxEdc97aikf7U+VPO52ABgSymgajC/9wYgIn+mJRvfbU FOCg== X-Forwarded-Encrypted: i=1; AKwUvBz2TEhVMq6VZECoPX0M7qu4dMbzA5LLeWgW+SAegk3e4O7q3ijpfQ9I0Eh7Woey70Rfh+4zV88LPwB4STQ=@vger.kernel.org X-Gm-Message-State: AFuF++lq1uGYwjQZY4oDpDdZxyPSOQ5ZQTgabF7xDBWo1rGAiYjU64Lc XjQoJ3ndJZSM+ExNaof8F/4kofu7V6RE5VX7pcNub+SChFN07D9uYpdcybrkCNohmkqvoy8cH/7 gp5sTvg== X-Received: from dybdx7.prod.google.com ([2002:a05:7300:b187:b0:32c:7ca7:8f62]) (user=surenb job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6a21:62c2:b0:3d0:d0ed:f141 with SMTP id adf61e73a8af0-3da39d18dfcmr32776365637.2.1788763172413; Sun, 06 Sep 2026 23:39:32 -0700 (PDT) Date: Sun, 6 Sep 2026 23:39:18 -0700 In-Reply-To: <20260907063918.3432401-1-surenb@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20260907063918.3432401-1-surenb@google.com> X-Mailer: git-send-email 2.55.0.979.g7e5102b832-goog Message-ID: <20260907063918.3432401-6-surenb@google.com> Subject: [PATCH v2 5/5] selftests/proc: add /proc/pid/smaps_rollup tearing tests From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: liam@infradead.org, ljs@kernel.org, vbabka@kernel.org, david@redhat.com, willy@infradead.org, jannh@google.com, paulmck@kernel.org, pfalcato@suse.de, linux-mm@kvack.org, linux-kernel@vger.kernel.org, linux-fsdevel@vger.kernel.org, surenb@google.com Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" During tearing tests, smaps_rollup Pss* metrics should stay constant. Extend /proc/pid/smaps tearing tests to also check for smaps_rollup consistency. Signed-off-by: Suren Baghdasaryan Acked-by: Liam R. Howlett (Oracle) Tested-by: Xueyuan Chen --- tools/testing/selftests/proc/proc-maps-race.c | 187 +++++++++++++++++- 1 file changed, 182 insertions(+), 5 deletions(-) diff --git a/tools/testing/selftests/proc/proc-maps-race.c b/tools/testing/= selftests/proc/proc-maps-race.c index 415eccb70468..8d00d7db1c65 100644 --- a/tools/testing/selftests/proc/proc-maps-race.c +++ b/tools/testing/selftests/proc/proc-maps-race.c @@ -80,6 +80,61 @@ enum maps_file { =20 struct vma_modifier_info; =20 +enum smaps_rollup_stat { + Rss, + Pss, + Pss_Dirty, + Pss_Anon, + Pss_File, + Pss_Shmem, + Shared_Clean, + Shared_Dirty, + Private_Clean, + Private_Dirty, + Referenced, + Anonymous, + KSM, + LazyFree, + AnonHugePages, + ShmemPmdMapped, + FilePmdMapped, + Shared_Hugetlb, + Private_Hugetlb, + Swap, + SwapPss, + Locked, + RollupFieldCount +}; + +static const char *smaps_rollup_stat_names[RollupFieldCount] =3D { + "Rss", + "Pss", + "Pss_Dirty", + "Pss_Anon", + "Pss_File", + "Pss_Shmem", + "Shared_Clean", + "Shared_Dirty", + "Private_Clean", + "Private_Dirty", + "Referenced", + "Anonymous", + "KSM", + "LazyFree", + "AnonHugePages", + "ShmemPmdMapped", + "FilePmdMapped", + "Shared_Hugetlb", + "Private_Hugetlb", + "Swap", + "SwapPss", + "Locked", +}; + +struct smaps_rollup_stats { + unsigned long values[RollupFieldCount]; +}; + FIXTURE(proc_maps_race) { struct vma_modifier_info *mod_info; @@ -91,6 +146,7 @@ FIXTURE(proc_maps_race) enum maps_file maps_file; int shared_mem_size; int skip_pages; + int rollup_fd; int page_size; int vma_count; bool verbose; @@ -132,12 +188,12 @@ struct vma_modifier_info { void *child_mapped_addr[]; }; =20 -static bool read_page(FIXTURE_DATA(proc_maps_race) *self, +static bool read_page(FIXTURE_DATA(proc_maps_race) *self, int fd, struct page_content *page) { ssize_t bytes_read; =20 - bytes_read =3D read(self->maps_fd, page->data, self->page_size); + bytes_read =3D read(fd, page->data, self->page_size); if (bytes_read <=3D 0) return false; =20 @@ -175,7 +231,7 @@ static int locate_containing_page(FIXTURE_DATA(proc_map= s_race) *self, char *curr_pos; char *end_pos; =20 - if (!read_page(self, &self->page1)) + if (!read_page(self, self->maps_fd, &self->page1)) return -1; =20 curr_pos =3D self->page1.data; @@ -205,10 +261,11 @@ static bool read_two_pages(FIXTURE_DATA(proc_maps_rac= e) *self) return false; =20 for (int i =3D 0; i < self->skip_pages; i++) - if (!read_page(self, &self->page1)) + if (!read_page(self, self->maps_fd, &self->page1)) return false; =20 - return read_page(self, &self->page1) && read_page(self, &self->page2); + return read_page(self, self->maps_fd, &self->page1) && + read_page(self, self->maps_fd, &self->page2); } =20 static void copy_line(const char *line_start, const char *line_end, @@ -317,6 +374,61 @@ static bool read_boundary_lines(FIXTURE_DATA(proc_maps= _race) *self, &first_line->end_addr) =3D=3D 2; } =20 +static bool parse_smaps_rollup(FIXTURE_DATA(proc_maps_race) *self, + struct smaps_rollup_stats *stats) +{ + unsigned int dev_maj, dev_min, inode; + unsigned long start, end, offs; + unsigned long value; + char name[32], perm[5]; + char *curr_pos; + char *end_pos; + char *line_end; + + if (lseek(self->rollup_fd, 0, SEEK_SET) < 0) + return false; + + if (!read_page(self, self->rollup_fd, &self->page1)) + return false; + + curr_pos =3D self->page1.data; + end_pos =3D self->page1.data + self->page1.size; + + line_end =3D strchr(curr_pos, '\n'); + if (!line_end) + return false; + + if (sscanf(curr_pos, "%lx-%lx %4s %lx %u:%u %u %31s", + &start, &end, perm, &offs, &dev_maj, &dev_min, &inode, name) !=3D 8) + return false; + + if (strcmp(name, "[rollup]")) + return false; + + for (int stat =3D 0; stat < ARRAY_SIZE(smaps_rollup_stat_names); stat++) { + int len; + + curr_pos =3D line_end + 1; + if (curr_pos >=3D end_pos) + return false; + + line_end =3D strchr(curr_pos, '\n'); + if (!line_end) + return false; + + if (sscanf(curr_pos, "%31s %lu kB", name, &value) !=3D 2) + return false; + + len =3D strlen(name); + if (name[len - 1] !=3D ':' || strncmp(name, smaps_rollup_stat_names[stat= ], len - 1)) + return false; + + stats->values[stat] =3D value; + } + + return true; +} + /* Thread synchronization routines */ static void wait_for_state(struct vma_modifier_info *mod_info, enum test_s= tate state) { @@ -397,6 +509,41 @@ static bool print_boundaries_on(bool condition, const = char *title, return condition; } =20 +static void print_smaps_rollup_stats(const char *title, FIXTURE_DATA(proc_= maps_race) *self, + struct smaps_rollup_stats *stats) +{ + printf("%s", title); + for (int stat =3D 0; stat < ARRAY_SIZE(smaps_rollup_stat_names); stat++) + printf("%64s %lu kB\n", smaps_rollup_stat_names[stat], stats->values[sta= t]); +} + +static bool cmp_smaps_rollup_stat(struct smaps_rollup_stats *s1, + struct smaps_rollup_stats *s2, + enum smaps_rollup_stat stat) +{ + return s1->values[stat] =3D=3D s2->values[stat]; +} + +static bool compare_smaps_rollup(FIXTURE_DATA(proc_maps_race) *self, + struct smaps_rollup_stats *expected, + struct smaps_rollup_stats *actual) +{ + /* + * Clean/dirty metrics might change but Pss-related ones + * should stay constant. + */ + if (cmp_smaps_rollup_stat(expected, actual, Pss) && + cmp_smaps_rollup_stat(expected, actual, Pss_Anon) && + cmp_smaps_rollup_stat(expected, actual, Pss_File) && + cmp_smaps_rollup_stat(expected, actual, Pss_Shmem)) + return true; + + print_smaps_rollup_stats("Expected stats:", self, expected); + print_smaps_rollup_stats("Actual stats:", self, actual); + + return false; +} + static void report_test_start(const char *name, bool verbose) { if (verbose) @@ -572,6 +719,7 @@ FIXTURE_SETUP(proc_maps_race) unsigned long first_map_addr; unsigned long last_map_addr; unsigned long duration_sec; + char rollup_fname[32]; char fname[32]; =20 self->page_size =3D (unsigned long)sysconf(_SC_PAGESIZE); @@ -649,6 +797,9 @@ FIXTURE_SETUP(proc_maps_race) break; case SMAPS: sprintf(fname, "/proc/%d/smaps", self->pid); + sprintf(rollup_fname, "/proc/%d/smaps_rollup", self->pid); + self->rollup_fd =3D open(rollup_fname, O_RDONLY); + ASSERT_NE(self->rollup_fd, -1); break; default: ksft_exit_fail(); @@ -711,6 +862,8 @@ FIXTURE_TEARDOWN(proc_maps_race) for (int i =3D 0; i < self->vma_count; i++) munmap(self->mod_info->child_mapped_addr[i], self->page_size); close(self->maps_fd); + if (self->maps_file =3D=3D SMAPS) + close(self->rollup_fd); waitpid(self->pid, &status, 0); munmap(self->mod_info, self->shared_mem_size); } @@ -723,6 +876,7 @@ TEST_F(proc_maps_race, test_maps_tearing_from_split) struct line_content split_first_line; struct line_content restored_last_line; struct line_content restored_first_line; + struct smaps_rollup_stats orig_stats; =20 wait_for_state(mod_info, SETUP_READY); =20 @@ -736,6 +890,8 @@ TEST_F(proc_maps_race, test_maps_tearing_from_split) report_test_start("Tearing from split", self->verbose); ASSERT_TRUE(capture_mod_pattern(self, &split_last_line, &split_first_line, &restored_last_line, &restored_first_line)); + if (self->maps_file =3D=3D SMAPS) + ASSERT_TRUE(parse_smaps_rollup(self, &orig_stats)); =20 /* Now start concurrent modifications for self->duration_sec */ signal_state(mod_info, TEST_READY); @@ -799,6 +955,11 @@ TEST_F(proc_maps_race, test_maps_tearing_from_split) vma_end =3D=3D self->last_line.end_addr) || (vma_start =3D=3D split_first_line.start_addr && vma_end =3D=3D split_first_line.end_addr)); + } else { + struct smaps_rollup_stats stats; + + ASSERT_TRUE(parse_smaps_rollup(self, &stats)); + ASSERT_TRUE(compare_smaps_rollup(self, &orig_stats, &stats)); } clock_gettime(CLOCK_MONOTONIC_COARSE, &end_ts); end_test_iteration(&end_ts, self->verbose); @@ -817,6 +978,7 @@ TEST_F(proc_maps_race, test_maps_tearing_from_resize) struct line_content shrunk_first_line; struct line_content restored_last_line; struct line_content restored_first_line; + struct smaps_rollup_stats orig_stats; =20 wait_for_state(mod_info, SETUP_READY); =20 @@ -830,6 +992,8 @@ TEST_F(proc_maps_race, test_maps_tearing_from_resize) report_test_start("Tearing from resize", self->verbose); ASSERT_TRUE(capture_mod_pattern(self, &shrunk_last_line, &shrunk_first_li= ne, &restored_last_line, &restored_first_line)); + if (self->maps_file =3D=3D SMAPS) + ASSERT_TRUE(parse_smaps_rollup(self, &orig_stats)); =20 /* Now start concurrent modifications for self->duration_sec */ signal_state(mod_info, TEST_READY); @@ -880,6 +1044,11 @@ TEST_F(proc_maps_race, test_maps_tearing_from_resize) ASSERT_TRUE(vma_start =3D=3D self->last_line.start_addr && (vma_end - vma_start =3D=3D self->page_size * 3 || vma_end - vma_start =3D=3D self->page_size)); + } else { + struct smaps_rollup_stats stats; + + ASSERT_TRUE(parse_smaps_rollup(self, &stats)); + ASSERT_TRUE(compare_smaps_rollup(self, &orig_stats, &stats)); } clock_gettime(CLOCK_MONOTONIC_COARSE, &end_ts); end_test_iteration(&end_ts, self->verbose); @@ -898,6 +1067,7 @@ TEST_F(proc_maps_race, test_maps_tearing_from_remap) struct line_content remapped_first_line; struct line_content restored_last_line; struct line_content restored_first_line; + struct smaps_rollup_stats orig_stats; =20 wait_for_state(mod_info, SETUP_READY); =20 @@ -911,6 +1081,8 @@ TEST_F(proc_maps_race, test_maps_tearing_from_remap) report_test_start("Tearing from remap", self->verbose); ASSERT_TRUE(capture_mod_pattern(self, &remapped_last_line, &remapped_firs= t_line, &restored_last_line, &restored_first_line)); + if (self->maps_file =3D=3D SMAPS) + ASSERT_TRUE(parse_smaps_rollup(self, &orig_stats)); =20 /* Now start concurrent modifications for self->duration_sec */ signal_state(mod_info, TEST_READY); @@ -963,6 +1135,11 @@ TEST_F(proc_maps_race, test_maps_tearing_from_remap) vma_end - vma_start =3D=3D self->page_size * 3) || (vma_start =3D=3D self->last_line.start_addr + self->page_size && vma_end - vma_start =3D=3D self->page_size)); + } else { + struct smaps_rollup_stats stats; + + ASSERT_TRUE(parse_smaps_rollup(self, &stats)); + ASSERT_TRUE(compare_smaps_rollup(self, &orig_stats, &stats)); } clock_gettime(CLOCK_MONOTONIC_COARSE, &end_ts); end_test_iteration(&end_ts, self->verbose); --=20 2.55.0.979.g7e5102b832-goog