From nobody Tue Sep 29 05:34:18 2026 Received: from m16.mail.126.com (m16.mail.126.com [117.135.210.6]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id CDCDA29A9E9 for ; Wed, 12 Aug 2026 07:05:54 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=117.135.210.6 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1786518357; cv=none; b=PLlU6uncyMEMEjL5J580RmV1tI7aeOb/sBoOv+bQddmEcfqeWfReQyxjI+7nLtSz5y0sgdP5+WRu1+E05x/xIl852FjQbOTKFXqA8pdNSUWc+QvRUDwBxCIGBjZiIGC+XwheipJo0pHE+RjevOoX/5SPlapWCLn3cA2yOnRXhrQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1786518357; c=relaxed/simple; bh=1uxIjFkRmICnZj5EXTBRiqYe0WNXXHEFBzcOBHsjp30=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=r67DXlQyZe3qLJQGvWkRMFpnitQV1/m1ona7Wnu82Nq1uaUTPCd8Sq8TdsoVFMUNviBxNXORuW9pAWc+A+XvZsakg9/gR+YdAroexB5G1g7EG7KdW2Tin2+vmLT9CZeHDlb72imSIuAYULzb9hqLsjc2y7L5iS1bwMiBKzBh1Z8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=126.com; spf=pass smtp.mailfrom=126.com; dkim=pass (1024-bit key) header.d=126.com header.i=@126.com header.b=FU3I/tQ0; arc=none smtp.client-ip=117.135.210.6 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=126.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=126.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=126.com header.i=@126.com header.b="FU3I/tQ0" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=126.com; s=s110527; h=From:To:Subject:Date:Message-ID:MIME-Version; bh=hb A1FO4npc2gsMgg8g0Ly6fhRFpwGtK0z+vodPecqwc=; b=FU3I/tQ0Ewkk+b5G1L NzSY8oJmk+YHxWB88szTogj6RKRnoag1chEfQRLWPbOWosqXCqSfQjmkZhF39Qvy NOO6291FUFHW2K3q6S4NR6etiDndS0xjVhKdoLmx0gSePHtcASLigFg825yfDx0U 2W6wmyOCl4Q8BzIZNvE77xpVA= Received: from frank-Vostro-3671.bbrouter (unknown []) by gzga-smtp-mtada-g0-3 (Coremail) with SMTP id _____wD3t7AeG3xqOIphFg--.17455S2; Wed, 12 Aug 2026 15:05:02 +0800 (CST) From: Frank Chen To: mhklinux@outlook.com, hch@lst.de, m.szyprowski@samsung.com, robin.murphy@arm.com Cc: linux-kernel@vger.kernel.org, iommu@lists.linux.dev, fan.du@intel.com, jun.miao@intel.com, chenhgs@chinatelecom.cn Subject: [PATCH v4] dma/swiotlb: decouple high watermark tracking from CONFIG_DEBUG_FS Date: Wed, 12 Aug 2026 15:04:59 +0800 Message-ID: <20260812070459.637077-1-frankchen158@126.com> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260622122114.2563254-1-jun.miao@intel.com> References: <20260622122114.2563254-1-jun.miao@intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-CM-TRANSID: _____wD3t7AeG3xqOIphFg--.17455S2 X-Coremail-Antispam: 1Uf129KBjvJXoWfGr4fGrWfAryktFyUJFyUGFg_yoWDXFW5pF y8Jw1UtF4DXr1jkFWUC348GFnIyw4fC39xWrW3Kr9ag3ZxtrnavFnakrWYvFW5JrW8ZF43 Xa43WrWrCrsrCrJanT9S1TB71UUUUU7qnTZGkaVYY2UrUUUUjbIjqfuFe4nvWSU5nxnvy2 9KBjDUYxBIdaVFxhVjvjDU0xZFpf9x07UpuWJUUUUU= X-CM-SenderInfo: xiud0yxfkh0iivy6ij2wof0z/xtbBoh+HGmp8Gx9lMAAA3n Content-Type: text/plain; charset="utf-8" From: chenhuguanshen Under heavy concurrent DMA traffic on CoCo VMs, inc_used_and_hiwater() performs an atomic_long_add_return() plus a CAS loop on the global used_hiwater, and dec_used() performs an atomic_long_sub() on total_used. All CPUs contend on the same cacheline, causing measurable throughput degradation at scale. Historically these counters were only compiled in under CONFIG_DEBUG_FS, which means production kernels with debugfs paid the atomic overhead unconditionally. Make the tracking boot-time opt-in instead so that it is disabled by default with near-zero overhead via static_call, and can be enabled via "swiotlb=3Dtrack_hiwater" parameter on demand for debugging. Note that when CONFIG_DEBUG_FS is enabled but hiwater tracking is disabled, the "io_tlb_used" metric reports an approximate value rather than an instantaneously exact one. Suggested-by: Fan Du Signed-off-by: Jun Miao Co-developed-by: Fan Du Signed-off-by: Fan Du Tested-by: chenhuguanshen Signed-off-by: chenhuguanshen --- v1 -> v2: - Change the patch title. - Doing the exact hiwater calculation is dynamic and defaults to "off", dynamic config would replace being under #ifdef CONFIG_DEBUG_FS - The mechanism used for dynamic config needs to be one that is selectable on the kernel boot line so that the exact hiwater mark during boot is easily available. v2 -> v3: - When track_hiwater is enabled, we keep the original precise hiwater calculation backed by the global total_used atomic counter. If disabled, we switch to the approximate approach that sums per-area counters to derive the total used slot count. v3 -> v4: - Change commit message and documentation. --- .../admin-guide/kernel-parameters.txt | 4 +- include/linux/swiotlb.h | 8 +- kernel/dma/swiotlb.c | 153 +++++++++++------- 3 files changed, 101 insertions(+), 64 deletions(-) diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentatio= n/admin-guide/kernel-parameters.txt index b5493a7f8f22..eb3e8db2c178 100644 --- a/Documentation/admin-guide/kernel-parameters.txt +++ b/Documentation/admin-guide/kernel-parameters.txt @@ -7477,7 +7477,7 @@ Kernel parameters Execution Facility on pSeries. =20 swiotlb=3D [ARM,PPC,MIPS,X86,S390,EARLY] - Format: { [,] | force | noforce } + Format: { [,] | force | noforce | track_hiwater} -- Number of I/O TLB slabs -- Second integer after comma. Number of swiotlb areas with their own lock. Will be rounded up @@ -7485,6 +7485,8 @@ Kernel parameters force -- force using of bounce buffers even if they wouldn't be automatically used by the kernel noforce -- Never use bounce buffers (for debugging) + track_hiwater -- Track high watermark of swiotlb buffers. + Only available when CONFIG_DEBUG_FS is set. =20 switches=3D [HW,M68k,EARLY] =20 diff --git a/include/linux/swiotlb.h b/include/linux/swiotlb.h index 3dae0f592063..008f169e4006 100644 --- a/include/linux/swiotlb.h +++ b/include/linux/swiotlb.h @@ -98,10 +98,10 @@ struct io_tlb_pool { * @pools: List of IO TLB memory pool descriptors (if dynamic). * @dyn_alloc: Dynamic IO TLB pool allocation work. * @total_used: The total number of slots in the pool that are currently u= sed - * across all areas. Used only for calculating used_hiwater in - * debugfs. - * @used_hiwater: The high water mark for total_used. Used only for repor= ting - * in debugfs. + * across all areas. Used only for calculating used_hiwater via boot + * parameter swiotlb=3Dtrack_hiwater and exposed via debugfs. + * @used_hiwater: The high water mark for total_used. Can be enabled at b= oot + * time via swiotlb=3Dtrack_hiwater and exposed via debugfs. * @transient_nslabs: The total number of slots in all transient pools that * are currently used across all areas. */ diff --git a/kernel/dma/swiotlb.c b/kernel/dma/swiotlb.c index 1abd3e6146f4..e0bc7ca4a7ad 100644 --- a/kernel/dma/swiotlb.c +++ b/kernel/dma/swiotlb.c @@ -180,6 +180,74 @@ static unsigned int limit_nareas(unsigned int nareas, = unsigned long nslots) return nareas; } =20 +#ifdef CONFIG_DEBUG_FS +/* + * Track the total used slots with a global atomic value in order to have + * correct information to determine the high water mark. + */ +static void inc_used_and_hiwater_real(struct io_tlb_mem *mem, + unsigned int nslots) +{ + unsigned long old_hiwater, new_used; + + new_used =3D atomic_long_add_return(nslots, &mem->total_used); + old_hiwater =3D atomic_long_read(&mem->used_hiwater); + do { + if (new_used <=3D old_hiwater) + break; + } while (!atomic_long_try_cmpxchg(&mem->used_hiwater, + &old_hiwater, new_used)); +} + +static void dec_used_real(struct io_tlb_mem *mem, unsigned int nslots) +{ + atomic_long_sub(nslots, &mem->total_used); +} + +static void inc_used_and_hiwater_nop(struct io_tlb_mem *mem, + unsigned int nslots) +{ +} +static void dec_used_nop(struct io_tlb_mem *mem, unsigned int nslots) +{ +} + +DEFINE_STATIC_CALL(swiotlb_inc_used, inc_used_and_hiwater_nop); +DEFINE_STATIC_CALL(swiotlb_dec_used, dec_used_nop); + +static __always_inline void inc_used_and_hiwater(struct io_tlb_mem *mem, + unsigned int nslots) +{ + static_call(swiotlb_inc_used)(mem, nslots); +} + +static __always_inline void dec_used(struct io_tlb_mem *mem, + unsigned int nslots) +{ + static_call(swiotlb_dec_used)(mem, nslots); +} + +static bool track_hiwater_enabled __read_mostly; + +#else + +static __always_inline void inc_used_and_hiwater(struct io_tlb_mem *mem, + unsigned int nslots) +{ +} + +static __always_inline void dec_used(struct io_tlb_mem *mem, + unsigned int nslots) +{ +} +#endif + +/* + * The tracking of used slots high watermark can be enabled + * by appending "track_hiwater" to the swiotlb=3D boot parameter. + * When disabled the tracking functions are no-ops with near-zero + * overhead via static_call. + */ static int __init setup_io_tlb_npages(char *str) { @@ -194,10 +262,24 @@ setup_io_tlb_npages(char *str) swiotlb_adjust_nareas(simple_strtoul(str, &str, 0)); if (*str =3D=3D ',') ++str; - if (!strcmp(str, "force")) + if (!strncmp(str, "force", 5)) { swiotlb_force_bounce =3D true; - else if (!strcmp(str, "noforce")) + str +=3D 5; + } else if (!strncmp(str, "noforce", 7)) { swiotlb_force_disable =3D true; + str +=3D 7; + } + +#ifdef CONFIG_DEBUG_FS + if (*str =3D=3D ',') + ++str; + if (!strncmp(str, "track_hiwater", 13)) { + track_hiwater_enabled =3D true; + static_call_update(swiotlb_inc_used, + inc_used_and_hiwater_real); + static_call_update(swiotlb_dec_used, dec_used_real); + } +#endif =20 return 0; } @@ -959,40 +1041,6 @@ static unsigned int wrap_area_index(struct io_tlb_poo= l *mem, unsigned int index) return index; } =20 -/* - * Track the total used slots with a global atomic value in order to have - * correct information to determine the high water mark. The mem_used() - * function gives imprecise results because there's no locking across - * multiple areas. - */ -#ifdef CONFIG_DEBUG_FS -static void inc_used_and_hiwater(struct io_tlb_mem *mem, unsigned int nslo= ts) -{ - unsigned long old_hiwater, new_used; - - new_used =3D atomic_long_add_return(nslots, &mem->total_used); - old_hiwater =3D atomic_long_read(&mem->used_hiwater); - do { - if (new_used <=3D old_hiwater) - break; - } while (!atomic_long_try_cmpxchg(&mem->used_hiwater, - &old_hiwater, new_used)); -} - -static void dec_used(struct io_tlb_mem *mem, unsigned int nslots) -{ - atomic_long_sub(nslots, &mem->total_used); -} - -#else /* !CONFIG_DEBUG_FS */ -static void inc_used_and_hiwater(struct io_tlb_mem *mem, unsigned int nslo= ts) -{ -} -static void dec_used(struct io_tlb_mem *mem, unsigned int nslots) -{ -} -#endif /* CONFIG_DEBUG_FS */ - #ifdef CONFIG_SWIOTLB_DYNAMIC #ifdef CONFIG_DEBUG_FS static void inc_transient_used(struct io_tlb_mem *mem, unsigned int nslots) @@ -1295,24 +1343,6 @@ static int swiotlb_find_slots(struct device *dev, ph= ys_addr_t orig_addr, =20 #endif /* CONFIG_SWIOTLB_DYNAMIC */ =20 -#ifdef CONFIG_DEBUG_FS - -/** - * mem_used() - get number of used slots in an allocator - * @mem: Software IO TLB allocator. - * - * The result is accurate in this version of the function, because an atom= ic - * counter is available if CONFIG_DEBUG_FS is set. - * - * Return: Number of used slots. - */ -static unsigned long mem_used(struct io_tlb_mem *mem) -{ - return atomic_long_read(&mem->total_used); -} - -#else /* !CONFIG_DEBUG_FS */ - /** * mem_pool_used() - get number of used slots in a memory pool * @pool: Software IO TLB memory pool. @@ -1335,13 +1365,20 @@ static unsigned long mem_pool_used(struct io_tlb_po= ol *pool) * mem_used() - get number of used slots in an allocator * @mem: Software IO TLB allocator. * - * The result is not accurate, because there is no locking of individual - * areas. + * When trace_hiwater and CONFIG_DEBUG_FS is enabled, the result is accura= te + * because the total number of used slots is tracked in mem->total_used. + * Otherwise, the result is an approximation, because there is no locking = of + * individual areas. * - * Return: Approximate number of used slots. + * Return: Number of used slots. */ static unsigned long mem_used(struct io_tlb_mem *mem) { +#ifdef CONFIG_DEBUG_FS + if (track_hiwater_enabled) + return atomic_long_read(&mem->total_used); +#endif + #ifdef CONFIG_SWIOTLB_DYNAMIC struct io_tlb_pool *pool; unsigned long used =3D 0; @@ -1357,8 +1394,6 @@ static unsigned long mem_used(struct io_tlb_mem *mem) #endif } =20 -#endif /* CONFIG_DEBUG_FS */ - /** * swiotlb_tbl_map_single() - bounce buffer map a single contiguous physic= al area * @dev: Device which maps the buffer. --=20 2.53.0