From nobody Mon Sep 28 06:35:13 2026 Received: from mx1.zhaoxin.com (MX1.ZHAOXIN.COM [210.0.225.12]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 55E153F4854 for ; Tue, 25 Aug 2026 10:56:01 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=210.0.225.12 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787655368; cv=none; b=OBGz02+arTMmuweNGIpLglCW4x3btnItx/8wEaFcVVj7pM4r1c6fySLDJcBb+5f2oBkh37jFXZprjRtRQYx46YvfwjFdxmcJ1vgnq1mdX2IzkB+BQK8SV9bYRT6MRWwxhkUI51uAPtO4vzB1tigMF98Jl7i1tdYYnXiSKkxzmqU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787655368; c=relaxed/simple; bh=FUcg1afzptYcOr+2HOEcgqzBbupimwWvPel8jzo8qjg=; h=From:To:Cc:Subject:Date:Message-ID:MIME-Version; b=saG/NT4tVi4KMU4qOnCpxELhu+IL+OtriVQcb6VmZ0Stq7Sl3A12T/rwyJMqAPj9rUqmic4+ZG9Lp9y25Ml5AAPrgx890J8rgxjSL+38gfLFMyypjzyKU0aydosBe07lSDKzjFBb9OPg1bbPwWf9Ma1ib6ji5ommLOdszcjV+rk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=zhaoxin.com; spf=pass smtp.mailfrom=zhaoxin.com; arc=none smtp.client-ip=210.0.225.12 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=zhaoxin.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=zhaoxin.com Received: from zhaoxin.com (unknown [127.0.0.1]) by mx1.zhaoxin.com (MTA) with ESMTP id 4hTkxX3mRrzb0tjv; Tue, 25 Aug 2026 18:47:12 +0800 (CST) Received: from zhaoxin.com (unknown [10.28.208.166]) by mx1.zhaoxin.com (MTA) with ESMTP id 4hTkxT6xn3zb0tjv; Tue, 25 Aug 2026 18:47:09 +0800 (CST) Received: from zjh-os.zhaoxin.com (zjh-os.zhaoxin.com [10.28.24.13]) by zhaoxin.com (8.30) with ESMTP0dcdd9a0ab20575ea6a786a0762250d2 Tue, 25 Aug 2026 18:47:08 +0800 X-Eyou-Smtpauth: jonaszhou-oc@zhaoxin.com X-Eyou-EnvelopeSender: jonaszhou-oc@zhaoxin.com X-Eyou-From: JonasZhou From: "=?UTF-8?B?Sm9uYXNaaG91LW9j?=" To: Andrew Morton , Uladzislau Rezki Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, jianhuizzzzz@gmail.com Subject: [PATCH] mm/vmalloc: avoid false sharing with drain_vmap_work Date: Tue, 25 Aug 2026 18:46:59 +0800 Message-ID: <20260825104659.100134-1-jonaszhou-oc@zhaoxin.com> X-Mailer: git-send-email 2.43.0 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Eyou-Sender: Content-Type: text/plain; charset="utf-8" free_vmap_area_noflush() queues drain_vmap_work after the number of lazily freed pages exceeds lazy_max_pages(). Until the worker purges those pages, concurrent frees keep calling schedule_work(). Even if the work is already pending, queue_work_on() performs a locked test_and_set_bit() on the pending bit in the work item. On the tested x86-64 build, drain_vmap_work and vmap_nodes occupy the same 64-byte cache line. The work item starts at offset 0 and the vmap_nodes pointer at offset 32. The latter is read by vmap allocation and free paths, so updates to the work item invalidate a cache line read by all CPUs. Put drain_vmap_work in the cacheline-aligned data section. Tests were run on Linux 7.2. On a two-socket Intel Xeon Silver 4208 system using 16 workers, the runtimes of vmalloc.fix_align, vmalloc.fix_size, and vmalloc.no_block_alloc decreased by 12.61%, 5.78%, and 6.87%, respectively. HITM samples for the affected cache line and total HITM samples decreased by 96.55% and 13.36%, respectively. Signed-off-by: JonasZhou Reviewed-by: Uladzislau Rezki (Sony) --- mm/vmalloc.c | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/mm/vmalloc.c b/mm/vmalloc.c index f4fa227a8d7f..6b4b287e91f6 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -1088,7 +1088,12 @@ RB_DECLARE_CALLBACKS_MAX(static, free_vmap_area_rb_a= ugment_cb, static void reclaim_and_purge_vmap_areas(void); static BLOCKING_NOTIFIER_HEAD(vmap_notify_list); static void drain_vmap_area_work(struct work_struct *work); -static DECLARE_WORK(drain_vmap_work, drain_vmap_area_work); +/* + * Keep the work item, whose pending bit is updated by freeing CPUs, + * away from vmap metadata read by allocation and free paths. + */ +static __cacheline_aligned_in_smp +DECLARE_WORK(drain_vmap_work, drain_vmap_area_work); =20 static __cacheline_aligned_in_smp atomic_long_t vmap_lazy_nr; =20 --=20 2.43.0