From nobody Sat Jul 25 20:06:06 2026 Received: from m16.mail.126.com (m16.mail.126.com [220.197.31.7]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id BB76A18D636 for ; Tue, 14 Jul 2026 03:26:11 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=220.197.31.7 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1783999575; cv=none; b=ekcR2tHzQpgHgPDiVcMh1Egt+wlZ49xO/hM4V8WKUGnZdOR/Tz4R5gsCfi/u35Q3s5tiLDrrUKNaPNVmjfKVJzNnOIXCLJ3+qd8c2NRJJpMAqFBChj89o0MYRckMFeuZ5kj6EzwWS6wj3eu7w9ICl0EUXznWLZpaojaLH/OgUJ4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1783999575; c=relaxed/simple; bh=jOdn7UTBgY2phBbMHnlhwqr1ZT77wXny2YADsIndDFU=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=HlhkBiTjKN4XkR4NH01YZFd2nZXMHcZ4zA4QevAa8jWB+iKlv0lRrGHQf/Lt5dmJ5KzftPNYvD2DXTsaecuDPim5GhD3ULJ1+fD4hjUGLJwQmttpdPshXKrr1jjmT3Na0dOUXCrCZGoZGsrCw1GJ641Dcu2hQVmkdk7XzV8PHsI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=126.com; spf=pass smtp.mailfrom=126.com; dkim=pass (1024-bit key) header.d=126.com header.i=@126.com header.b=bU/+N/TP; arc=none smtp.client-ip=220.197.31.7 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=126.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=126.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=126.com header.i=@126.com header.b="bU/+N/TP" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=126.com; s=s110527; h=From:To:Subject:Date:Message-Id:MIME-Version; bh=+n Eb+WyoimJppanfUCbOf7pNZFAH1VXUny4LwRn7Elg=; b=bU/+N/TP+ILdl+iPy3 vmfExR0yn1r1P4CJeMevYKsloXzxNozgNvwXsJKUIkgpJcroZlak5/bG5EIBnpOD i9dWZyWKyHYlcLpBzyQEpav5ZAZeMGeEQfSFx8k0BmCQ0RWTW5gMMDX1LaGiQuNB DTzPLzyNq5V4jFKMA9pk4lWcE= Received: from frank-pc.bbrouter (unknown []) by gzga-smtp-mtada-g1-3 (Coremail) with SMTP id _____wD3B7YlrFVqtlSFDQ--.32903S2; Tue, 14 Jul 2026 11:25:26 +0800 (CST) From: Frank Chen To: mhklinux@outlook.com, hch@lst.de, m.szyprowski@samsung.com, robin.murphy@arm.com Cc: linux-kernel@vger.kernel.org, iommu@lists.linux.dev, fan.du@intel.com, jun.miao@intel.com, chenhgs@chinatelecom.cn Subject: [PATCH] dma/swiotlb: make high watermark tracking boot-time opt-in Date: Tue, 14 Jul 2026 11:25:20 +0800 Message-Id: <20260714032520.1058536-1-frankchen158@126.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20260622122114.2563254-1-jun.miao@intel.com> References: <20260622122114.2563254-1-jun.miao@intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-CM-TRANSID: _____wD3B7YlrFVqtlSFDQ--.32903S2 X-Coremail-Antispam: 1Uf129KBjvJXoWfGr4DAFWDuw48GFyDuw1Utrb_yoWDWw1kpF y5Jw1DtF4DXF10kFWDC348GF1ayw4S939xWrW5Kr9aq3Zxtrn3XFn2yrWYvFW5JrW8ZF4a qFya9FyrCrsrCrJanT9S1TB71UUUUU7qnTZGkaVYY2UrUUUUjbIjqfuFe4nvWSU5nxnvy2 9KBjDUYxBIdaVFxhVjvjDU0xZFpf9x07UAHUkUUUUU= X-CM-SenderInfo: xiud0yxfkh0iivy6ij2wof0z/xtbBogYktmpVrCbgRgAA3l Content-Type: text/plain; charset="utf-8" From: chenhuguanshen Under heavy concurrent DMA traffic on CoCo VMs, inc_used_and_hiwater() performs an atomic_long_add_return() plus a CAS loop on the global used_hiwater, and dec_used() performs an atomic_long_sub() on total_used. All CPUs contend on the same cacheline, causing measurable throughput degradation at scale. Historically these counters were only compiled in under CONFIG_DEBUG_FS, which means production kernels with debugfs paid the atomic overhead unconditionally once the symbols were present. Make the tracking boot-time opt-in instead so that it is disabled by default with near-zero overhead via static_call, and can be enabled via "swiotlb=3Dtrack_hiwater" parameter on demand for debugging. Changes: - The previous inc_used_and_hiwater() maintained a separate atomic total_used counter for accurate hiwater updates. This is replaced with mem_used() which sums across areas. While less precise (no cross-area locking), the result is adequate for high watermark statistics and avoids the overhead of maintaining total_used on every alloc/free. - Refactor high watermark tracking to use static_call, allowing it to be enabled at boot via the swiotlb=3Dtrack_hiwater parameter when CONFIG_DEBUG_FS is set. When disabled (the default), the static_call resolves to a no-op function, ensuring near-zero overhead. - Remove the total_used atomic counter from io_tlb_mem since it is no longer needed. The debugfs "io_tlb_used" file already uses mem_used() to report current usage. Suggested-by: Fan Du Signed-off-by: Jun Miao Co-developed-by: Fan Du Signed-off-by: Fan Du Tested-by: chenhuguanshen Signed-off-by: chenhuguanshen --- v1 -> v2: = =20 - Change the patch title. = =20 - Doing the exact hiwater calculation is dynamic and defaults to "off", = =20 dynamic config would replace being under #ifdef CONFIG_DEBUG_FS = =20 - The mechanism used for dynamic config needs to be one that is selectable= =20 on the kernel boot line so that the exact hiwater mark during boot is ea= sily available. --- .../admin-guide/kernel-parameters.txt | 3 +- include/linux/swiotlb.h | 10 +- kernel/dma/swiotlb.c | 125 +++++++++--------- 3 files changed, 68 insertions(+), 70 deletions(-) diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentatio= n/admin-guide/kernel-parameters.txt index b2d7d3540ded..bb60c55383a0 100644 --- a/Documentation/admin-guide/kernel-parameters.txt +++ b/Documentation/admin-guide/kernel-parameters.txt @@ -7485,7 +7485,7 @@ Kernel parameters Execution Facility on pSeries. =20 swiotlb=3D [ARM,PPC,MIPS,X86,S390,EARLY] - Format: { [,] | force | noforce } + Format: { [,] | force | noforce | track_hiwater } -- Number of I/O TLB slabs -- Second integer after comma. Number of swiotlb areas with their own lock. Will be rounded up @@ -7493,6 +7493,7 @@ Kernel parameters force -- force using of bounce buffers even if they wouldn't be automatically used by the kernel noforce -- Never use bounce buffers (for debugging) + track_hiwater -- Track high watermark of swiotlb buffers =20 switches=3D [HW,M68k,EARLY] =20 diff --git a/include/linux/swiotlb.h b/include/linux/swiotlb.h index 3dae0f592063..e308792243cc 100644 --- a/include/linux/swiotlb.h +++ b/include/linux/swiotlb.h @@ -97,11 +97,8 @@ struct io_tlb_pool { * @lock: Lock to synchronize changes to the list. * @pools: List of IO TLB memory pool descriptors (if dynamic). * @dyn_alloc: Dynamic IO TLB pool allocation work. - * @total_used: The total number of slots in the pool that are currently u= sed - * across all areas. Used only for calculating used_hiwater in - * debugfs. - * @used_hiwater: The high water mark for total_used. Used only for repor= ting - * in debugfs. + * @used_hiwater: The high watermark of used slots. Can be enabled at + * boot time via swiotlb=3Dtrack_hiwater. * @transient_nslabs: The total number of slots in all transient pools that * are currently used across all areas. */ @@ -119,9 +116,8 @@ struct io_tlb_mem { struct work_struct dyn_alloc; #endif #ifdef CONFIG_DEBUG_FS - atomic_long_t total_used; - atomic_long_t used_hiwater; atomic_long_t transient_nslabs; + atomic_long_t used_hiwater; #endif }; =20 diff --git a/kernel/dma/swiotlb.c b/kernel/dma/swiotlb.c index 1abd3e6146f4..5828163c15f7 100644 --- a/kernel/dma/swiotlb.c +++ b/kernel/dma/swiotlb.c @@ -180,8 +180,55 @@ static unsigned int limit_nareas(unsigned int nareas, = unsigned long nslots) return nareas; } =20 -static int __init -setup_io_tlb_npages(char *str) +#ifdef CONFIG_DEBUG_FS +static unsigned long mem_used(struct io_tlb_mem *mem); + +/* + * Track the total used slots to determine the high watermark. + * Unlike the previous total_used atomic counter which provided + * exact values, mem_used() iterates areas without cross-area + * locking and may return imprecise results under concurrent + * alloc/free. This is acceptable for high watermark statistics + * and avoids the overhead of maintaining a dedicated atomic + * counter on every allocation and free. + */ +static void account_hiwater_real(struct io_tlb_mem *mem) +{ + unsigned long old_hiwater, new_used; + + new_used =3D mem_used(mem); + old_hiwater =3D atomic_long_read(&mem->used_hiwater); + do { + if (new_used <=3D old_hiwater) + break; + } while (!atomic_long_try_cmpxchg(&mem->used_hiwater, + &old_hiwater, new_used)); +} + +static void account_hiwater_nop(struct io_tlb_mem *mem) +{ +} + +DEFINE_STATIC_CALL(swiotlb_account_hiwater, account_hiwater_nop); + +static __always_inline void account_hiwater(struct io_tlb_mem *mem) +{ + static_call(swiotlb_account_hiwater)(mem); +} + +#else +static __always_inline void account_hiwater(struct io_tlb_mem *mem) +{ +} +#endif + +/* + * The tracking of used slots high watermark can be enabled + * by appending "track_hiwater" to the swiotlb=3D boot parameter. + * When disabled the tracking functions are no-ops with near-zero + * overhead via static_call. + */ +static int __init setup_io_tlb_npages(char *str) { if (isdigit(*str)) { /* avoid tail segment of size < IO_TLB_SEGSIZE */ @@ -194,10 +241,21 @@ setup_io_tlb_npages(char *str) swiotlb_adjust_nareas(simple_strtoul(str, &str, 0)); if (*str =3D=3D ',') ++str; - if (!strcmp(str, "force")) + if (!strncmp(str, "force", 5)) { swiotlb_force_bounce =3D true; - else if (!strcmp(str, "noforce")) + str +=3D 5; + } else if (!strncmp(str, "noforce", 7)) { swiotlb_force_disable =3D true; + str +=3D 7; + } +#ifdef CONFIG_DEBUG_FS + if (*str =3D=3D ',') + ++str; + if (!strncmp(str, "track_hiwater", 13)) { + static_call_update(swiotlb_account_hiwater, + account_hiwater_real); + } +#endif =20 return 0; } @@ -959,40 +1017,6 @@ static unsigned int wrap_area_index(struct io_tlb_poo= l *mem, unsigned int index) return index; } =20 -/* - * Track the total used slots with a global atomic value in order to have - * correct information to determine the high water mark. The mem_used() - * function gives imprecise results because there's no locking across - * multiple areas. - */ -#ifdef CONFIG_DEBUG_FS -static void inc_used_and_hiwater(struct io_tlb_mem *mem, unsigned int nslo= ts) -{ - unsigned long old_hiwater, new_used; - - new_used =3D atomic_long_add_return(nslots, &mem->total_used); - old_hiwater =3D atomic_long_read(&mem->used_hiwater); - do { - if (new_used <=3D old_hiwater) - break; - } while (!atomic_long_try_cmpxchg(&mem->used_hiwater, - &old_hiwater, new_used)); -} - -static void dec_used(struct io_tlb_mem *mem, unsigned int nslots) -{ - atomic_long_sub(nslots, &mem->total_used); -} - -#else /* !CONFIG_DEBUG_FS */ -static void inc_used_and_hiwater(struct io_tlb_mem *mem, unsigned int nslo= ts) -{ -} -static void dec_used(struct io_tlb_mem *mem, unsigned int nslots) -{ -} -#endif /* CONFIG_DEBUG_FS */ - #ifdef CONFIG_SWIOTLB_DYNAMIC #ifdef CONFIG_DEBUG_FS static void inc_transient_used(struct io_tlb_mem *mem, unsigned int nslots) @@ -1134,7 +1158,7 @@ static int swiotlb_search_pool_area(struct device *de= v, struct io_tlb_pool *pool area->used +=3D nslots; spin_unlock_irqrestore(&area->lock, flags); =20 - inc_used_and_hiwater(dev->dma_io_tlb_mem, nslots); + account_hiwater(dev->dma_io_tlb_mem); return slot_index; } =20 @@ -1295,24 +1319,6 @@ static int swiotlb_find_slots(struct device *dev, ph= ys_addr_t orig_addr, =20 #endif /* CONFIG_SWIOTLB_DYNAMIC */ =20 -#ifdef CONFIG_DEBUG_FS - -/** - * mem_used() - get number of used slots in an allocator - * @mem: Software IO TLB allocator. - * - * The result is accurate in this version of the function, because an atom= ic - * counter is available if CONFIG_DEBUG_FS is set. - * - * Return: Number of used slots. - */ -static unsigned long mem_used(struct io_tlb_mem *mem) -{ - return atomic_long_read(&mem->total_used); -} - -#else /* !CONFIG_DEBUG_FS */ - /** * mem_pool_used() - get number of used slots in a memory pool * @pool: Software IO TLB memory pool. @@ -1357,8 +1363,6 @@ static unsigned long mem_used(struct io_tlb_mem *mem) #endif } =20 -#endif /* CONFIG_DEBUG_FS */ - /** * swiotlb_tbl_map_single() - bounce buffer map a single contiguous physic= al area * @dev: Device which maps the buffer. @@ -1508,8 +1512,6 @@ static void swiotlb_release_slots(struct device *dev,= phys_addr_t tlb_addr, mem->slots[i].list =3D ++count; area->used -=3D nslots; spin_unlock_irqrestore(&area->lock, flags); - - dec_used(dev->dma_io_tlb_mem, nslots); } =20 #ifdef CONFIG_SWIOTLB_DYNAMIC @@ -1531,7 +1533,6 @@ static bool swiotlb_del_transient(struct device *dev,= phys_addr_t tlb_addr, if (!pool->transient) return false; =20 - dec_used(dev->dma_io_tlb_mem, pool->nslabs); swiotlb_del_pool(dev, pool); dec_transient_used(dev->dma_io_tlb_mem, pool->nslabs); return true; --=20 2.34.1