From nobody Thu Sep 18 15:40:48 2025 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id B50F4C4321E for ; Mon, 5 Dec 2022 15:00:29 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231770AbiLEPA2 (ORCPT ); Mon, 5 Dec 2022 10:00:28 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:32924 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231599AbiLEPAZ (ORCPT ); Mon, 5 Dec 2022 10:00:25 -0500 Received: from mx0a-00082601.pphosted.com (mx0a-00082601.pphosted.com [67.231.145.42]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id AB206BC85 for ; Mon, 5 Dec 2022 07:00:24 -0800 (PST) Received: from pps.filterd (m0044010.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 2B5EZsYo010692 for ; Mon, 5 Dec 2022 07:00:24 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=meta.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=s2048-2021-q4; bh=jW6N9lNs9aHxnq0Gv9Edz1amHHS3PbY2z2d5+0qT2t4=; b=ZDve+bapQAu7yDP8M4eZ++3X7Atj7Oi/MPdhOX8GOyN0o96Th+aa0aFtIrmp5gNK3eZS HzF3xxhFebuIGv3tqNRl5XfJNX9K6+4lD6lgY1dexHfkteTL8RPOFw97DAomCtSxxIsr 8acc7jar+0AVMiwkV8Vx32lnj+RfkqLMkgaenN7XvgNfN1U+/1N4HHpWpGQ3o1urOyRK 3Ti0b6tZ4n5xjP0Q2j7bZESHDlrEai4sXBg3oCxGH4HeWr2AeTRCouymw4bcvB/XmFO4 MvgXj2mCpC2mM6M7tvXpT6At2VZAC++PdbSJSZkV5CO26DjSDOIEXr/LZODVH26ZmerN qQ== Received: from mail.thefacebook.com ([163.114.132.120]) by mx0a-00082601.pphosted.com (PPS) with ESMTPS id 3m82ntux7x-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Mon, 05 Dec 2022 07:00:23 -0800 Received: from twshared10308.07.ash9.facebook.com (2620:10d:c085:208::11) by mail.thefacebook.com (2620:10d:c085:21d::4) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.31; Mon, 5 Dec 2022 07:00:22 -0800 Received: by devbig007.nao1.facebook.com (Postfix, from userid 544533) id A52D2C6156DD; Mon, 5 Dec 2022 07:00:09 -0800 (PST) From: Keith Busch To: , , Matthew Wilcox CC: Tony Battersby , Keith Busch Subject: [PATCH 01/11] dmapool: add alloc/free performance test Date: Mon, 5 Dec 2022 06:59:27 -0800 Message-ID: <20221205145937.54367-2-kbusch@meta.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20221205145937.54367-1-kbusch@meta.com> References: <20221205145937.54367-1-kbusch@meta.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-FB-Internal: Safe X-Proofpoint-ORIG-GUID: 0P1HgfABaooqcOIgzK216V-MDpPTcDvG X-Proofpoint-GUID: 0P1HgfABaooqcOIgzK216V-MDpPTcDvG X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.205,Aquarius:18.0.923,Hydra:6.0.545,FMLib:17.11.122.1 definitions=2022-12-05_01,2022-12-05_01,2022-06-22_01 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Keith Busch Provide a module that allocates and frees many blocks of various sizes and report how long it takes. This is intended to provide a consistent way to measure how changes to the dma_pool_alloc/free routines affect timing. Signed-off-by: Keith Busch --- mm/Kconfig | 9 +++ mm/Makefile | 1 + mm/dmapool_test.c | 138 ++++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 148 insertions(+) create mode 100644 mm/dmapool_test.c diff --git a/mm/Kconfig b/mm/Kconfig index 57e1d8c5b5052..12a78d64eeb72 100644 --- a/mm/Kconfig +++ b/mm/Kconfig @@ -1044,6 +1044,15 @@ config GUP_TEST comment "GUP_TEST needs to have DEBUG_FS enabled" depends on !GUP_TEST && !DEBUG_FS =20 +config DMAPOOL_TEST + tristate "Enable a module to run time tests on dma_pool" + depends on HAS_DMA + help + Provides a module that will allocate and free many blocks of various + sizes and report how long it takes. This is intended to provide a + consistent way to measure how changes to the dma_pool_alloc/free + routines affect performance. + config GUP_GET_PTE_LOW_HIGH bool =20 diff --git a/mm/Makefile b/mm/Makefile index 8e105e5b3e293..3a08f5d7b1782 100644 --- a/mm/Makefile +++ b/mm/Makefile @@ -103,6 +103,7 @@ obj-$(CONFIG_MEMCG) +=3D swap_cgroup.o endif obj-$(CONFIG_CGROUP_HUGETLB) +=3D hugetlb_cgroup.o obj-$(CONFIG_GUP_TEST) +=3D gup_test.o +obj-$(CONFIG_DMAPOOL_TEST) +=3D dmapool_test.o obj-$(CONFIG_MEMORY_FAILURE) +=3D memory-failure.o obj-$(CONFIG_HWPOISON_INJECT) +=3D hwpoison-inject.o obj-$(CONFIG_DEBUG_KMEMLEAK) +=3D kmemleak.o diff --git a/mm/dmapool_test.c b/mm/dmapool_test.c new file mode 100644 index 0000000000000..2873d91f509b4 --- /dev/null +++ b/mm/dmapool_test.c @@ -0,0 +1,138 @@ +#include +#include +#include +#include +#include +#include + +#define NR_TESTS (100) + +struct dma_pool_pair { + dma_addr_t dma; + void *v; +}; + +static const int block_sizes[] =3D { + 16, + 64, + 256, + 1024, + 4096 +}; + +static struct dma_pool *pool; +struct device test_dev; +u64 dma_mask; + +static inline int nr_blocks(int size) +{ + return clamp_t(int, (PAGE_SIZE / size) * 512, 1024, 8192); +} + +static int dmapool_test_alloc(struct dma_pool_pair *p, int blocks) +{ + int i; + + for (i =3D 0; i < blocks; i++) { + p[i].v =3D dma_pool_alloc(pool, GFP_KERNEL, + &p[i].dma); + if (!p[i].v) + goto pool_fail; + } + + for (i =3D 0; i < blocks; i++) + dma_pool_free(pool, p[i].v, p[i].dma); + + return 0; + +pool_fail: + for (--i; i >=3D 0; i--) + dma_pool_free(pool, p[i].v, p[i].dma); + return -ENOMEM; +} + +static int dmapool_test_block(int block_size) +{ + int blocks =3D nr_blocks(block_size); + ktime_t start_time, end_time; + struct dma_pool_pair *p; + int i, ret; + + p =3D kcalloc(blocks, sizeof(*p), GFP_KERNEL); + if (!p) + return -ENOMEM; + + pool =3D dma_pool_create("test pool", &test_dev, block_size, + block_size, 0); + if (!pool) { + ret =3D -ENOMEM; + goto free_pairs; + } + + start_time =3D ktime_get(); + for (i =3D 0; i < NR_TESTS; i++) { + ret =3D dmapool_test_alloc(p, blocks); + if (ret) + goto free_pool; + if (need_resched()) + cond_resched(); + } + end_time =3D ktime_get(); + + printk("dmapool test: size:%-4d blocks:%-6d time:%llu\n", block_size, + blocks, ktime_us_delta(end_time, start_time)); + +free_pool: + dma_pool_destroy(pool); +free_pairs: + kfree(p); + return ret; +} + +static void dmapool_test_release(struct device *dev) +{ +} + +static int dmapool_checks(void) +{ + int i, ret; + + ret =3D dev_set_name(&test_dev, "dmapool-test"); + if (ret) + return ret; + + ret =3D device_register(&test_dev); + if (ret) { + printk("%s: register failed:%d\n", __func__, ret); + goto put_device; + } + + test_dev.release =3D dmapool_test_release; + test_dev.dma_ops =3D NULL; + test_dev.dma_mask =3D &dma_mask; + ret =3D dma_set_mask_and_coherent(&test_dev, DMA_BIT_MASK(64)); + if (ret) { + printk("%s: mask failed:%d\n", __func__, ret); + goto del_device; + } + + for (i =3D 0; i < ARRAY_SIZE(block_sizes); i++) { + ret =3D dmapool_test_block(block_sizes[i]); + if (ret) + break; + } + +del_device: + device_del(&test_dev); +put_device: + put_device(&test_dev); + return ret; +} + +static void dmapool_exit(void) +{ +} + +module_init(dmapool_checks); +module_exit(dmapool_exit); +MODULE_LICENSE("GPL"); --=20 2.30.2 From nobody Thu Sep 18 15:40:48 2025 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 87020C47088 for ; Mon, 5 Dec 2022 15:00:34 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231909AbiLEPAb (ORCPT ); Mon, 5 Dec 2022 10:00:31 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:32976 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231923AbiLEPA2 (ORCPT ); Mon, 5 Dec 2022 10:00:28 -0500 Received: from mx0a-00082601.pphosted.com (mx0b-00082601.pphosted.com [67.231.153.30]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id BA071DFCE for ; Mon, 5 Dec 2022 07:00:25 -0800 (PST) Received: from pps.filterd (m0001303.ppops.net [127.0.0.1]) by m0001303.ppops.net (8.17.1.19/8.17.1.19) with ESMTP id 2B5C0SIX000740 for ; Mon, 5 Dec 2022 07:00:25 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=meta.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=s2048-2021-q4; bh=YU1RvihRnvDJ0EI7aPZRW/ndJTmtKABPXXQKd2XE/QM=; b=h72aB84uF535vhqcSF/upzVkRKlcy1O4YHQ33llTxkXXqGNEv07PJm/9UT0GamvqFm69 letw054sf96K2yDECmaBk5enLnO0+h+gBqXZpn3F9n3gVsFu4VqjzsvjyQUHNSWvgrsH ZYetGQPpLXxAeBwNCLvHfg+F6A153f/jl5EwJ17EI7tbQ5br8Zz0ZbAHA16VCXqzRSKp yMbMqL1mylZiV7qWL1Zmk1MRN/59JJuWSaNU8zWZiPNEAKZ11tvspWwFc9X4MX/uo989 +XykDO8PcS9Hz6d+i8hEd+kGzILJyBRYQoaU9DJOf749WEbwODmDaNso2QkV4OFWjpcB vA== Received: from mail.thefacebook.com ([163.114.132.120]) by m0001303.ppops.net (PPS) with ESMTPS id 3m9g8c1kks-3 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Mon, 05 Dec 2022 07:00:24 -0800 Received: from twshared10308.07.ash9.facebook.com (2620:10d:c085:208::11) by mail.thefacebook.com (2620:10d:c085:21d::6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.31; Mon, 5 Dec 2022 07:00:22 -0800 Received: by devbig007.nao1.facebook.com (Postfix, from userid 544533) id C5CE2C6156E0; Mon, 5 Dec 2022 07:00:09 -0800 (PST) From: Keith Busch To: , , Matthew Wilcox CC: Tony Battersby , Keith Busch Subject: [PATCH 02/11] dmapool: remove checks for dev == NULL Date: Mon, 5 Dec 2022 06:59:28 -0800 Message-ID: <20221205145937.54367-3-kbusch@meta.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20221205145937.54367-1-kbusch@meta.com> References: <20221205145937.54367-1-kbusch@meta.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-FB-Internal: Safe X-Proofpoint-GUID: SIAsFnmQprUPdP9asPoy9-ng49KiGAL9 X-Proofpoint-ORIG-GUID: SIAsFnmQprUPdP9asPoy9-ng49KiGAL9 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.205,Aquarius:18.0.923,Hydra:6.0.545,FMLib:17.11.122.1 definitions=2022-12-05_01,2022-12-05_01,2022-06-22_01 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Tony Battersby dmapool originally tried to support pools without a device because dma_alloc_coherent() supports allocations without a device. But nobody ended up using dma pools without a device, and trying to do so will result in an oops. So remove the checks for pool->dev =3D=3D NULL since th= ey are unneeded bloat. Signed-off-by: Tony Battersby [added check for null dev on create] Signed-off-by: Keith Busch --- mm/dmapool.c | 45 ++++++++++++++------------------------------- 1 file changed, 14 insertions(+), 31 deletions(-) diff --git a/mm/dmapool.c b/mm/dmapool.c index a7eb5d0eb2da7..559207e1c3339 100644 --- a/mm/dmapool.c +++ b/mm/dmapool.c @@ -134,6 +134,9 @@ struct dma_pool *dma_pool_create(const char *name, stru= ct device *dev, size_t allocation; bool empty =3D false; =20 + if (!dev) + return NULL; + if (align =3D=3D 0) align =3D 1; else if (align & (align - 1)) @@ -275,7 +278,7 @@ void dma_pool_destroy(struct dma_pool *pool) mutex_lock(&pools_reg_lock); mutex_lock(&pools_lock); list_del(&pool->pools); - if (pool->dev && list_empty(&pool->dev->dma_pools)) + if (list_empty(&pool->dev->dma_pools)) empty =3D true; mutex_unlock(&pools_lock); if (empty) @@ -284,12 +287,8 @@ void dma_pool_destroy(struct dma_pool *pool) =20 list_for_each_entry_safe(page, tmp, &pool->page_list, page_list) { if (is_page_busy(page)) { - if (pool->dev) - dev_err(pool->dev, "%s %s, %p busy\n", __func__, - pool->name, page->vaddr); - else - pr_err("%s %s, %p busy\n", __func__, - pool->name, page->vaddr); + dev_err(pool->dev, "%s %s, %p busy\n", __func__, + pool->name, page->vaddr); /* leak the still-in-use consistent memory */ list_del(&page->page_list); kfree(page); @@ -351,12 +350,8 @@ void *dma_pool_alloc(struct dma_pool *pool, gfp_t mem_= flags, for (i =3D sizeof(page->offset); i < pool->size; i++) { if (data[i] =3D=3D POOL_POISON_FREED) continue; - if (pool->dev) - dev_err(pool->dev, "%s %s, %p (corrupted)\n", - __func__, pool->name, retval); - else - pr_err("%s %s, %p (corrupted)\n", - __func__, pool->name, retval); + dev_err(pool->dev, "%s %s, %p (corrupted)\n", + __func__, pool->name, retval); =20 /* * Dump the first 4 bytes even if they are not @@ -411,12 +406,8 @@ void dma_pool_free(struct dma_pool *pool, void *vaddr,= dma_addr_t dma) page =3D pool_find_page(pool, dma); if (!page) { spin_unlock_irqrestore(&pool->lock, flags); - if (pool->dev) - dev_err(pool->dev, "%s %s, %p/%pad (bad dma)\n", - __func__, pool->name, vaddr, &dma); - else - pr_err("%s %s, %p/%pad (bad dma)\n", - __func__, pool->name, vaddr, &dma); + dev_err(pool->dev, "%s %s, %p/%pad (bad dma)\n", + __func__, pool->name, vaddr, &dma); return; } =20 @@ -426,12 +417,8 @@ void dma_pool_free(struct dma_pool *pool, void *vaddr,= dma_addr_t dma) #ifdef DMAPOOL_DEBUG if ((dma - page->dma) !=3D offset) { spin_unlock_irqrestore(&pool->lock, flags); - if (pool->dev) - dev_err(pool->dev, "%s %s, %p (bad vaddr)/%pad\n", - __func__, pool->name, vaddr, &dma); - else - pr_err("%s %s, %p (bad vaddr)/%pad\n", - __func__, pool->name, vaddr, &dma); + dev_err(pool->dev, "%s %s, %p (bad vaddr)/%pad\n", + __func__, pool->name, vaddr, &dma); return; } { @@ -442,12 +429,8 @@ void dma_pool_free(struct dma_pool *pool, void *vaddr,= dma_addr_t dma) continue; } spin_unlock_irqrestore(&pool->lock, flags); - if (pool->dev) - dev_err(pool->dev, "%s %s, dma %pad already free\n", - __func__, pool->name, &dma); - else - pr_err("%s %s, dma %pad already free\n", - __func__, pool->name, &dma); + dev_err(pool->dev, "%s %s, dma %pad already free\n", + __func__, pool->name, &dma); return; } } --=20 2.30.2 From nobody Thu Sep 18 15:40:48 2025 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 099CBC4321E for ; Mon, 5 Dec 2022 15:14:21 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232295AbiLEPOT (ORCPT ); Mon, 5 Dec 2022 10:14:19 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45512 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230522AbiLEPNe (ORCPT ); Mon, 5 Dec 2022 10:13:34 -0500 Received: from mx0a-00082601.pphosted.com (mx0a-00082601.pphosted.com [67.231.145.42]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8340D5F84 for ; Mon, 5 Dec 2022 07:13:27 -0800 (PST) Received: from pps.filterd (m0044010.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 2B51d4Sq028659 for ; Mon, 5 Dec 2022 07:13:26 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=meta.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=s2048-2021-q4; bh=MLWHDUS20C0jmHJ39hBO1H02jIUsWm5bZEXT2SOrcPk=; b=ejYF75fU7LKM8wl5jpImHC6ob7OmFqntqgSVGc7q7fBRtNTo2YbmogV9/hYldchrM5wM G0+tjYboDIyd5JbXawla0H4uvIOtKIYTIurU5zdZBP0kxApMAZGRIOvXKAig2QNU3KMf eIiafVB8VOpgv6FqOoo28oMSYTlMz1YLZA6rOqxa7tMTjf+eHGdJuwl4dbcwngVrLne1 l16/SnAwj5orN2eNpyh6zDEi0moXxB76qmVzGa0EU66PzkXkQAohAew/QcGDljqEA6kz WusGmDZ0R0+69WEmS1e817ytYI09qRBuM4V9W9N5PPvhKdasT7p0qd7KndjZCpcdkU7Z yw== Received: from maileast.thefacebook.com ([163.114.130.16]) by mx0a-00082601.pphosted.com (PPS) with ESMTPS id 3m82ntv1f1-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Mon, 05 Dec 2022 07:13:26 -0800 Received: from twshared0551.06.ash8.facebook.com (2620:10d:c0a8:1b::d) by mail.thefacebook.com (2620:10d:c0a8:83::5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.31; Mon, 5 Dec 2022 07:13:24 -0800 Received: by devbig007.nao1.facebook.com (Postfix, from userid 544533) id 6027CC6156E3; Mon, 5 Dec 2022 07:00:09 -0800 (PST) From: Keith Busch To: , , Matthew Wilcox CC: Tony Battersby , Keith Busch Subject: [PATCH 03/11] dmapool: use sysfs_emit() instead of scnprintf() Date: Mon, 5 Dec 2022 06:59:29 -0800 Message-ID: <20221205145937.54367-4-kbusch@meta.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20221205145937.54367-1-kbusch@meta.com> References: <20221205145937.54367-1-kbusch@meta.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-FB-Internal: Safe X-Proofpoint-ORIG-GUID: gIKqeh2PxnGikkroHP_deWhRD57Zjpuo X-Proofpoint-GUID: gIKqeh2PxnGikkroHP_deWhRD57Zjpuo X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.205,Aquarius:18.0.923,Hydra:6.0.545,FMLib:17.11.122.1 definitions=2022-12-05_01,2022-12-05_01,2022-06-22_01 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Tony Battersby Use sysfs_emit instead of scnprintf, snprintf or sprintf. Signed-off-by: Tony Battersby Signed-off-by: Keith Busch --- mm/dmapool.c | 23 +++++++---------------- 1 file changed, 7 insertions(+), 16 deletions(-) diff --git a/mm/dmapool.c b/mm/dmapool.c index 559207e1c3339..20616b760bb9c 100644 --- a/mm/dmapool.c +++ b/mm/dmapool.c @@ -64,18 +64,11 @@ static DEFINE_MUTEX(pools_reg_lock); =20 static ssize_t pools_show(struct device *dev, struct device_attribute *att= r, char *buf) { - unsigned temp; - unsigned size; - char *next; + int size; struct dma_page *page; struct dma_pool *pool; =20 - next =3D buf; - size =3D PAGE_SIZE; - - temp =3D scnprintf(next, size, "poolinfo - 0.1\n"); - size -=3D temp; - next +=3D temp; + size =3D sysfs_emit(buf, "poolinfo - 0.1\n"); =20 mutex_lock(&pools_lock); list_for_each_entry(pool, &dev->dma_pools, pools) { @@ -90,16 +83,14 @@ static ssize_t pools_show(struct device *dev, struct de= vice_attribute *attr, cha spin_unlock_irq(&pool->lock); =20 /* per-pool info, no real statistics yet */ - temp =3D scnprintf(next, size, "%-16s %4u %4zu %4zu %2u\n", - pool->name, blocks, - pages * (pool->allocation / pool->size), - pool->size, pages); - size -=3D temp; - next +=3D temp; + size +=3D sysfs_emit_at(buf, size, "%-16s %4u %4zu %4zu %2u\n", + pool->name, blocks, + pages * (pool->allocation / pool->size), + pool->size, pages); } mutex_unlock(&pools_lock); =20 - return PAGE_SIZE - size; + return size; } =20 static DEVICE_ATTR_RO(pools); --=20 2.30.2 From nobody Thu Sep 18 15:40:48 2025 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 20C6DC47088 for ; Mon, 5 Dec 2022 15:03:29 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231586AbiLEPD2 (ORCPT ); Mon, 5 Dec 2022 10:03:28 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35936 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232287AbiLEPDX (ORCPT ); Mon, 5 Dec 2022 10:03:23 -0500 Received: from mx0b-00082601.pphosted.com (mx0b-00082601.pphosted.com [67.231.153.30]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C4C411CFD0 for ; Mon, 5 Dec 2022 07:03:22 -0800 (PST) Received: from pps.filterd (m0109331.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 2B5EI7CO031279 for ; Mon, 5 Dec 2022 07:03:22 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=meta.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=s2048-2021-q4; bh=TsEKLROJI6bk+d/af34BtH3nHi+XT+AwCEfmZNUH3/Y=; b=n2s7vjs8KN+BGlwCWmAM3FI6+I2qNWWEDl7YsCA2k46CZ/7DMyFdU9VzlF7oyomPrQjJ CfuFPPABnbZaJqnY6LBHCNxBnnudgd+ny2xYQa5twOoeOH8PiiuNDG/PIhj35Y/pRqRl lCw1f4Qjwi3OvNjhgPhE0s9C8tClj/2iR4oNOYZSI8yk25zwioYHnbSdkkDEY1aNveT8 CppP79V42qM4gc+WFQPxq+0/6I1dVHulvlndcRSiUFUUvpvO7f0FA4964+xLgjz0vrJr Fq1NxJLX8Tjy4iuD1KQgJFp47CgphSlIs3EzpDKgmkhuI5qtOzIA9UNG8Z5D7xLrFqwV zA== Received: from mail.thefacebook.com ([163.114.132.120]) by mx0a-00082601.pphosted.com (PPS) with ESMTPS id 3m8561asg3-11 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Mon, 05 Dec 2022 07:03:21 -0800 Received: from twshared26225.38.frc1.facebook.com (2620:10d:c085:108::8) by mail.thefacebook.com (2620:10d:c085:11d::6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.31; Mon, 5 Dec 2022 07:03:20 -0800 Received: by devbig007.nao1.facebook.com (Postfix, from userid 544533) id D6291C6156EA; Mon, 5 Dec 2022 07:00:10 -0800 (PST) From: Keith Busch To: , , Matthew Wilcox CC: Tony Battersby , Keith Busch Subject: [PATCH 04/11] dmapool: cleanup integer types Date: Mon, 5 Dec 2022 06:59:30 -0800 Message-ID: <20221205145937.54367-5-kbusch@meta.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20221205145937.54367-1-kbusch@meta.com> References: <20221205145937.54367-1-kbusch@meta.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-FB-Internal: Safe X-Proofpoint-GUID: CJwgNXB_5qrWdRXOstgq8w2xv39hTJQF X-Proofpoint-ORIG-GUID: CJwgNXB_5qrWdRXOstgq8w2xv39hTJQF X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.205,Aquarius:18.0.923,Hydra:6.0.545,FMLib:17.11.122.1 definitions=2022-12-05_01,2022-12-05_01,2022-06-22_01 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Tony Battersby To represent the size of a single allocation, dmapool currently uses 'unsigned int' in some places and 'size_t' in other places. Standardize on 'unsigned int' to reduce overhead, but use 'size_t' when counting all the blocks in the entire pool. Signed-off-by: Tony Battersby Signed-off-by: Keith Busch --- mm/dmapool.c | 19 +++++++++++-------- 1 file changed, 11 insertions(+), 8 deletions(-) diff --git a/mm/dmapool.c b/mm/dmapool.c index 20616b760bb9c..ee993bb59fc27 100644 --- a/mm/dmapool.c +++ b/mm/dmapool.c @@ -43,10 +43,10 @@ struct dma_pool { /* the pool */ struct list_head page_list; spinlock_t lock; - size_t size; struct device *dev; - size_t allocation; - size_t boundary; + unsigned int size; + unsigned int allocation; + unsigned int boundary; char name[32]; struct list_head pools; }; @@ -73,7 +73,7 @@ static ssize_t pools_show(struct device *dev, struct devi= ce_attribute *attr, cha mutex_lock(&pools_lock); list_for_each_entry(pool, &dev->dma_pools, pools) { unsigned pages =3D 0; - unsigned blocks =3D 0; + size_t blocks =3D 0; =20 spin_lock_irq(&pool->lock); list_for_each_entry(page, &pool->page_list, page_list) { @@ -83,9 +83,10 @@ static ssize_t pools_show(struct device *dev, struct dev= ice_attribute *attr, cha spin_unlock_irq(&pool->lock); =20 /* per-pool info, no real statistics yet */ - size +=3D sysfs_emit_at(buf, size, "%-16s %4u %4zu %4zu %2u\n", + size +=3D sysfs_emit_at(buf, size, "%-16s %4zu %4zu %4u %2u\n", pool->name, blocks, - pages * (pool->allocation / pool->size), + (size_t) pages * + (pool->allocation / pool->size), pool->size, pages); } mutex_unlock(&pools_lock); @@ -133,7 +134,7 @@ struct dma_pool *dma_pool_create(const char *name, stru= ct device *dev, else if (align & (align - 1)) return NULL; =20 - if (size =3D=3D 0) + if (size =3D=3D 0 || size > INT_MAX) return NULL; else if (size < 4) size =3D 4; @@ -146,6 +147,8 @@ struct dma_pool *dma_pool_create(const char *name, stru= ct device *dev, else if ((boundary < size) || (boundary & (boundary - 1))) return NULL; =20 + boundary =3D min(boundary, allocation); + retval =3D kmalloc(sizeof(*retval), GFP_KERNEL); if (!retval) return retval; @@ -306,7 +309,7 @@ void *dma_pool_alloc(struct dma_pool *pool, gfp_t mem_f= lags, { unsigned long flags; struct dma_page *page; - size_t offset; + unsigned int offset; void *retval; =20 might_alloc(mem_flags); --=20 2.30.2 From nobody Thu Sep 18 15:40:48 2025 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id F2F8EC4321E for ; Mon, 5 Dec 2022 15:20:55 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232712AbiLEPUx (ORCPT ); Mon, 5 Dec 2022 10:20:53 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:52096 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232530AbiLEPTc (ORCPT ); Mon, 5 Dec 2022 10:19:32 -0500 Received: from mx0a-00082601.pphosted.com (mx0a-00082601.pphosted.com [67.231.145.42]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4F0871A229 for ; Mon, 5 Dec 2022 07:19:30 -0800 (PST) Received: from pps.filterd (m0109333.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 2B50UkFr019563 for ; Mon, 5 Dec 2022 07:19:30 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=meta.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=s2048-2021-q4; bh=P/KRpfmdlQUKxAAPrLlGtIIYCnA6qEIJS0aGmE3w/OM=; b=jtG+mgTh7Z80eRamWInP2j8Uv8btmTRrxRZYCGOm49bRo9tMfiR3ycsoe+/u5rgu88g1 6kQimJqXC5ofllDbbygKkFozEB4Q7dQGFA+kDt13DPL7oEo7qneol1qQwWdO5bFa6zDS adSeduIBU4TheBfJhIuKOujGMpOAJsP6Uz9gX1fYHTn0yUTywwBCzXvXVfnDxJvqTAiB /Eix5ptf30dLZBp5uO/U18JJB0Nipzn/VCIwePnENXY5CiVs92DfRKgAcF3V2fD3CXdN cnLIyCxpwRm6BYUk9V2c0WKQda8vOCZwnXdz0aDdjCpVsuudjMBn2wFjkhbpinTTVvYg KQ== Received: from mail.thefacebook.com ([163.114.132.120]) by mx0a-00082601.pphosted.com (PPS) with ESMTPS id 3m83cskxvd-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Mon, 05 Dec 2022 07:19:30 -0800 Received: from twshared16963.27.frc3.facebook.com (2620:10d:c085:108::8) by mail.thefacebook.com (2620:10d:c085:21d::5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.31; Mon, 5 Dec 2022 07:19:29 -0800 Received: by devbig007.nao1.facebook.com (Postfix, from userid 544533) id 7A815C6156F4; Mon, 5 Dec 2022 07:00:10 -0800 (PST) From: Keith Busch To: , , Matthew Wilcox CC: Tony Battersby , Keith Busch Subject: [PATCH 05/11] dmapool: speedup DMAPOOL_DEBUG with init_on_alloc Date: Mon, 5 Dec 2022 06:59:31 -0800 Message-ID: <20221205145937.54367-6-kbusch@meta.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20221205145937.54367-1-kbusch@meta.com> References: <20221205145937.54367-1-kbusch@meta.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-FB-Internal: Safe X-Proofpoint-GUID: o7lCOkJUmQNJNock67g4mqJzwFPvpAl5 X-Proofpoint-ORIG-GUID: o7lCOkJUmQNJNock67g4mqJzwFPvpAl5 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.205,Aquarius:18.0.923,Hydra:6.0.545,FMLib:17.11.122.1 definitions=2022-12-05_01,2022-12-05_01,2022-06-22_01 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Tony Battersby Avoid double-memset of the same allocated memory in dma_pool_alloc() when both DMAPOOL_DEBUG is enabled and init_on_alloc=3D1. Signed-off-by: Tony Battersby Signed-off-by: Keith Busch --- mm/dmapool.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mm/dmapool.c b/mm/dmapool.c index ee993bb59fc27..eaed3ffb42aa8 100644 --- a/mm/dmapool.c +++ b/mm/dmapool.c @@ -356,7 +356,7 @@ void *dma_pool_alloc(struct dma_pool *pool, gfp_t mem_f= lags, break; } } - if (!(mem_flags & __GFP_ZERO)) + if (!want_init_on_alloc(mem_flags)) memset(retval, POOL_POISON_ALLOCATED, pool->size); #endif spin_unlock_irqrestore(&pool->lock, flags); --=20 2.30.2 From nobody Thu Sep 18 15:40:48 2025 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id C9546C47089 for ; Mon, 5 Dec 2022 15:03:32 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232292AbiLEPDb (ORCPT ); Mon, 5 Dec 2022 10:03:31 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36036 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232239AbiLEPD2 (ORCPT ); Mon, 5 Dec 2022 10:03:28 -0500 Received: from mx0b-00082601.pphosted.com (mx0b-00082601.pphosted.com [67.231.153.30]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3DF9C13D04 for ; Mon, 5 Dec 2022 07:03:27 -0800 (PST) Received: from pps.filterd (m0148460.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 2B4NuwBT029063 for ; Mon, 5 Dec 2022 07:03:26 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=meta.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=s2048-2021-q4; bh=TZNkpjr8/3ibxb7IcC5QH+0+yx1Gj6xL4H95QeGVkb8=; b=WRuVaZkAiQ0pCRZ7ZVydL+UdVVfjJf4lRIdFT/06c2sfdAzJUZ5JuBl4nHPnVQYVhF5z lEpFBpyQtbaOSKEIyWmfg/AHDFJyZOwAadrl++UGZHMkz7sj2GnmEpB/WAGdLAz0Pn1S C1+/kJcxNbH0tiqjl9O9FfIIHhwyDhrxkrMQFlY5+GTu9GFU7o/VcIQug2eVSvoQdNxx dgxLO3JjXsw5mFBb1F7SB/ZC4q6uaw3HHNAIfFAWDfmJtVGUTIgcFQQUCie9GzSKDmTt itseNgU6bmMtNzmOB474PNHt8DaAKRGy+1v7AmhXFzS3ZTkqzZ9UG6/QMkxTkeE0FVK/ mw== Received: from mail.thefacebook.com ([163.114.132.120]) by mx0a-00082601.pphosted.com (PPS) with ESMTPS id 3m84v5jwf3-16 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Mon, 05 Dec 2022 07:03:26 -0800 Received: from twshared0705.02.ash8.facebook.com (2620:10d:c085:208::11) by mail.thefacebook.com (2620:10d:c085:11d::4) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.31; Mon, 5 Dec 2022 07:03:21 -0800 Received: by devbig007.nao1.facebook.com (Postfix, from userid 544533) id 29613C6156F6; Mon, 5 Dec 2022 07:00:11 -0800 (PST) From: Keith Busch To: , , Matthew Wilcox CC: Tony Battersby , Keith Busch Subject: [PATCH 06/11] dmapool: move debug code to own functions Date: Mon, 5 Dec 2022 06:59:32 -0800 Message-ID: <20221205145937.54367-7-kbusch@meta.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20221205145937.54367-1-kbusch@meta.com> References: <20221205145937.54367-1-kbusch@meta.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-FB-Internal: Safe X-Proofpoint-ORIG-GUID: e5kUmKCFh95jWDr0nKtOqt6HqEWBPCci X-Proofpoint-GUID: e5kUmKCFh95jWDr0nKtOqt6HqEWBPCci X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.205,Aquarius:18.0.923,Hydra:6.0.545,FMLib:17.11.122.1 definitions=2022-12-05_01,2022-12-05_01,2022-06-22_01 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Keith Busch Clean up the normal path by moving the debug code outside it. Signed-off-by: Keith Busch --- mm/dmapool.c | 96 +++++++++++++++++++++++++++++----------------------- 1 file changed, 54 insertions(+), 42 deletions(-) diff --git a/mm/dmapool.c b/mm/dmapool.c index eaed3ffb42aa8..8a7aa19e650a1 100644 --- a/mm/dmapool.c +++ b/mm/dmapool.c @@ -294,6 +294,32 @@ void dma_pool_destroy(struct dma_pool *pool) } EXPORT_SYMBOL(dma_pool_destroy); =20 +static inline void pool_check_block(struct dma_pool *pool, void *retval, + unsigned int offset, gfp_t mem_flags) +{ +#ifdef DMAPOOL_DEBUG + int i; + u8 *data =3D retval; + /* page->offset is stored in first 4 bytes */ + for (i =3D sizeof(offset); i < pool->size; i++) { + if (data[i] =3D=3D POOL_POISON_FREED) + continue; + dev_err(pool->dev, "%s %s, %p (corrupted)\n", + __func__, pool->name, retval); + + /* + * Dump the first 4 bytes even if they are not + * POOL_POISON_FREED + */ + print_hex_dump(KERN_ERR, "", DUMP_PREFIX_OFFSET, 16, 1, + data, pool->size, 1); + break; + } + if (!want_init_on_alloc(mem_flags)) + memset(retval, POOL_POISON_ALLOCATED, pool->size); +#endif +} + /** * dma_pool_alloc - get a block of consistent memory * @pool: dma pool that will produce the block @@ -336,29 +362,7 @@ void *dma_pool_alloc(struct dma_pool *pool, gfp_t mem_= flags, page->offset =3D *(int *)(page->vaddr + offset); retval =3D offset + page->vaddr; *handle =3D offset + page->dma; -#ifdef DMAPOOL_DEBUG - { - int i; - u8 *data =3D retval; - /* page->offset is stored in first 4 bytes */ - for (i =3D sizeof(page->offset); i < pool->size; i++) { - if (data[i] =3D=3D POOL_POISON_FREED) - continue; - dev_err(pool->dev, "%s %s, %p (corrupted)\n", - __func__, pool->name, retval); - - /* - * Dump the first 4 bytes even if they are not - * POOL_POISON_FREED - */ - print_hex_dump(KERN_ERR, "", DUMP_PREFIX_OFFSET, 16, 1, - data, pool->size, 1); - break; - } - } - if (!want_init_on_alloc(mem_flags)) - memset(retval, POOL_POISON_ALLOCATED, pool->size); -#endif + pool_check_block(pool, retval, offset, mem_flags); spin_unlock_irqrestore(&pool->lock, flags); =20 if (want_init_on_alloc(mem_flags)) @@ -381,6 +385,32 @@ static struct dma_page *pool_find_page(struct dma_pool= *pool, dma_addr_t dma) return NULL; } =20 +static inline bool pool_page_err(struct dma_pool *pool, struct dma_page *p= age, + void *vaddr) +{ +#ifdef DMAPOOL_DEBUG + unsigned int chain =3D page->offset; + + if ((dma - page->dma) !=3D offset) { + dev_err(pool->dev, "%s %s, %p (bad vaddr)/%pad\n", + __func__, pool->name, vaddr, &dma); + return true; + } + + while (chain < pool->allocation) { + if (chain !=3D offset) { + chain =3D *(int *)(page->vaddr + chain); + continue; + } + dev_err(pool->dev, "%s %s, dma %pad already free\n", + __func__, pool->name, &dma); + return true; + } + memset(vaddr, POOL_POISON_FREED, pool->size); +#endif + return false; +} + /** * dma_pool_free - put block back into dma pool * @pool: the dma pool holding the block @@ -408,28 +438,10 @@ void dma_pool_free(struct dma_pool *pool, void *vaddr= , dma_addr_t dma) offset =3D vaddr - page->vaddr; if (want_init_on_free()) memset(vaddr, 0, pool->size); -#ifdef DMAPOOL_DEBUG - if ((dma - page->dma) !=3D offset) { + if (pool_page_err(pool, page, vaddr)) { spin_unlock_irqrestore(&pool->lock, flags); - dev_err(pool->dev, "%s %s, %p (bad vaddr)/%pad\n", - __func__, pool->name, vaddr, &dma); return; } - { - unsigned int chain =3D page->offset; - while (chain < pool->allocation) { - if (chain !=3D offset) { - chain =3D *(int *)(page->vaddr + chain); - continue; - } - spin_unlock_irqrestore(&pool->lock, flags); - dev_err(pool->dev, "%s %s, dma %pad already free\n", - __func__, pool->name, &dma); - return; - } - } - memset(vaddr, POOL_POISON_FREED, pool->size); -#endif =20 page->in_use--; *(int *)vaddr =3D page->offset; --=20 2.30.2 From nobody Thu Sep 18 15:40:48 2025 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 06A6FC47089 for ; Mon, 5 Dec 2022 15:21:10 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232743AbiLEPVH (ORCPT ); Mon, 5 Dec 2022 10:21:07 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51120 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232621AbiLEPTx (ORCPT ); Mon, 5 Dec 2022 10:19:53 -0500 Received: from mx0b-00082601.pphosted.com (mx0b-00082601.pphosted.com [67.231.153.30]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 5E0691DF2B for ; Mon, 5 Dec 2022 07:19:34 -0800 (PST) Received: from pps.filterd (m0109332.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 2B5CVba5012959 for ; Mon, 5 Dec 2022 07:19:33 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=meta.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=s2048-2021-q4; bh=ADvvFQUdwkTabb+mgOJxdznef25hLYACuBWcCNuRK/M=; b=ELaivGWdKmR+7P/NGl0TQf7pvFbhWJmnbb62RC3z369pUeY6UlSZuwWKDKnOMSokdVJs Arw3dzuIgZEFFiZKh7MI7upKp3GquRl9YtAp/b9OeNH9VpE3fc6BuIEppTY6lc31y+FK LaHIinB+i7JmHxo6VpvquK9mcBDYlPHu/VKMFVu0hGibcaFJHo9ZodKMSPrw5DipBopS uGmULh1f9rX4DcZCFpIYeeDEGjI873/8R0AuQD5HMA8ykYMx3wM2CDemmSaePK0SWjpn j4fDvG28OUbNt9QR1pEaZAc+xeP4vszHywKECkkB0CfBOhvspsI4pCJ29Qp/DKwpnGas FA== Received: from mail.thefacebook.com ([163.114.132.120]) by mx0a-00082601.pphosted.com (PPS) with ESMTPS id 3m9gqj9gct-6 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Mon, 05 Dec 2022 07:19:33 -0800 Received: from snc-exhub201.TheFacebook.com (2620:10d:c085:21d::7) by snc-exhub103.TheFacebook.com (2620:10d:c085:11d::7) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.31; Mon, 5 Dec 2022 07:19:30 -0800 Received: from twshared16963.27.frc3.facebook.com (2620:10d:c085:108::8) by mail.thefacebook.com (2620:10d:c085:21d::7) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.31; Mon, 5 Dec 2022 07:19:29 -0800 Received: by devbig007.nao1.facebook.com (Postfix, from userid 544533) id 6E86FC615703; Mon, 5 Dec 2022 07:00:11 -0800 (PST) From: Keith Busch To: , , Matthew Wilcox CC: Tony Battersby , Keith Busch Subject: [PATCH 07/11] dmapool: rearrange page alloc failure handling Date: Mon, 5 Dec 2022 06:59:33 -0800 Message-ID: <20221205145937.54367-8-kbusch@meta.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20221205145937.54367-1-kbusch@meta.com> References: <20221205145937.54367-1-kbusch@meta.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-FB-Internal: Safe X-Proofpoint-GUID: M-RugQYgPLFsi3Q5ETmSydZwIfhtxjAM X-Proofpoint-ORIG-GUID: M-RugQYgPLFsi3Q5ETmSydZwIfhtxjAM X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.205,Aquarius:18.0.923,Hydra:6.0.545,FMLib:17.11.122.1 definitions=2022-12-05_01,2022-12-05_01,2022-06-22_01 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Keith Busch Handle the error in a condition so the good path can be in the normal flow. Signed-off-by: Keith Busch --- mm/dmapool.c | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/mm/dmapool.c b/mm/dmapool.c index 8a7aa19e650a1..f77e2af4941de 100644 --- a/mm/dmapool.c +++ b/mm/dmapool.c @@ -222,17 +222,17 @@ static struct dma_page *pool_alloc_page(struct dma_po= ol *pool, gfp_t mem_flags) return NULL; page->vaddr =3D dma_alloc_coherent(pool->dev, pool->allocation, &page->dma, mem_flags); - if (page->vaddr) { -#ifdef DMAPOOL_DEBUG - memset(page->vaddr, POOL_POISON_FREED, pool->allocation); -#endif - pool_initialise_page(pool, page); - page->in_use =3D 0; - page->offset =3D 0; - } else { + if (!page->vaddr) { kfree(page); - page =3D NULL; + return NULL; } +#ifdef DMAPOOL_DEBUG + memset(page->vaddr, POOL_POISON_FREED, pool->allocation); +#endif + pool_initialise_page(pool, page); + page->in_use =3D 0; + page->offset =3D 0; + return page; } =20 --=20 2.30.2 From nobody Thu Sep 18 15:40:48 2025 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id B3AE6C47089 for ; Mon, 5 Dec 2022 15:07:29 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232289AbiLEPH2 (ORCPT ); Mon, 5 Dec 2022 10:07:28 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38324 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232239AbiLEPHX (ORCPT ); Mon, 5 Dec 2022 10:07:23 -0500 Received: from mx0a-00082601.pphosted.com (mx0a-00082601.pphosted.com [67.231.145.42]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0F0A41D333 for ; Mon, 5 Dec 2022 07:07:22 -0800 (PST) Received: from pps.filterd (m0044010.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 2B5Er24M016570 for ; Mon, 5 Dec 2022 07:07:21 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=meta.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=s2048-2021-q4; bh=5sHDLhRqo9/RufklnGKFVoQ7JnNipMp+QhHHPmz6D+s=; b=ZZKdYNlsm54rW2y2lXptkIcf6E1liXqw1DNspW49oxKbTmP2vwjiCs4DRRXeems2hYR3 6qapFMudkYne5j5GHbcpsX1UVn2043+GI8tuykSaN824e9yky3kjr5XSyQhbXBShv3uo NQHPOFuGNTr8Ol3n3UahOu1iLedUpdBbIHN0UnVBPhBsHj2PfSxdwWA5qdtDph2S3d3f 3QKeKpPxulwP8jxrB5HSC1fW1vli3JuotxSJ5ZVo+77ZD0hoHMG8DSKDdlSIhOFB1cfS GB6t71Ixo6lHkqw4IkY1RGIVUR9WNGVVcuaR5fcvMul8+T9d7d5pgnQfyKg+Nz+8PnBN /g== Received: from maileast.thefacebook.com ([163.114.130.16]) by mx0a-00082601.pphosted.com (PPS) with ESMTPS id 3m82ntv08u-2 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Mon, 05 Dec 2022 07:07:21 -0800 Received: from twshared13940.35.frc1.facebook.com (2620:10d:c0a8:1b::d) by mail.thefacebook.com (2620:10d:c0a8:83::7) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.31; Mon, 5 Dec 2022 07:07:20 -0800 Received: by devbig007.nao1.facebook.com (Postfix, from userid 544533) id 8B08CC615706; Mon, 5 Dec 2022 07:00:12 -0800 (PST) From: Keith Busch To: , , Matthew Wilcox CC: Tony Battersby , Keith Busch Subject: [PATCH 08/11] dmapool: consolidate page initialization Date: Mon, 5 Dec 2022 06:59:34 -0800 Message-ID: <20221205145937.54367-9-kbusch@meta.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20221205145937.54367-1-kbusch@meta.com> References: <20221205145937.54367-1-kbusch@meta.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-FB-Internal: Safe X-Proofpoint-ORIG-GUID: 7YLAVJ-68DODBvOn3Nk_Ms-XVOl_65d_ X-Proofpoint-GUID: 7YLAVJ-68DODBvOn3Nk_Ms-XVOl_65d_ X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.205,Aquarius:18.0.923,Hydra:6.0.545,FMLib:17.11.122.1 definitions=2022-12-05_01,2022-12-05_01,2022-06-22_01 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Keith Busch Various fields of the dma pool are set in different places. Move it all to one function. Signed-off-by: Keith Busch --- mm/dmapool.c | 11 +++++------ 1 file changed, 5 insertions(+), 6 deletions(-) diff --git a/mm/dmapool.c b/mm/dmapool.c index f77e2af4941de..33d20ceff18c5 100644 --- a/mm/dmapool.c +++ b/mm/dmapool.c @@ -202,6 +202,8 @@ static void pool_initialise_page(struct dma_pool *pool,= struct dma_page *page) unsigned int offset =3D 0; unsigned int next_boundary =3D pool->boundary; =20 + page->in_use =3D 0; + page->offset =3D 0; do { unsigned int next =3D offset + pool->size; if (unlikely((next + pool->size) >=3D next_boundary)) { @@ -211,6 +213,8 @@ static void pool_initialise_page(struct dma_pool *pool,= struct dma_page *page) *(int *)(page->vaddr + offset) =3D next; offset =3D next; } while (offset < pool->allocation); + + list_add(&page->page_list, &pool->page_list); } =20 static struct dma_page *pool_alloc_page(struct dma_pool *pool, gfp_t mem_f= lags) @@ -229,10 +233,6 @@ static struct dma_page *pool_alloc_page(struct dma_poo= l *pool, gfp_t mem_flags) #ifdef DMAPOOL_DEBUG memset(page->vaddr, POOL_POISON_FREED, pool->allocation); #endif - pool_initialise_page(pool, page); - page->in_use =3D 0; - page->offset =3D 0; - return page; } =20 @@ -354,8 +354,7 @@ void *dma_pool_alloc(struct dma_pool *pool, gfp_t mem_f= lags, return NULL; =20 spin_lock_irqsave(&pool->lock, flags); - - list_add(&page->page_list, &pool->page_list); + pool_initialise_page(pool, page); ready: page->in_use++; offset =3D page->offset; --=20 2.30.2 From nobody Thu Sep 18 15:40:48 2025 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id E3694C4321E for ; Mon, 5 Dec 2022 15:21:00 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232716AbiLEPU7 (ORCPT ); Mon, 5 Dec 2022 10:20:59 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:52166 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232553AbiLEPTe (ORCPT ); Mon, 5 Dec 2022 10:19:34 -0500 Received: from mx0b-00082601.pphosted.com (mx0b-00082601.pphosted.com [67.231.153.30]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1EEE71A3AD for ; Mon, 5 Dec 2022 07:19:31 -0800 (PST) Received: from pps.filterd (m0109331.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 2B4LV1RP002801 for ; Mon, 5 Dec 2022 07:19:31 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=meta.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=s2048-2021-q4; bh=vVY03djUbGuw9RyCWd6Eexf1lAzlqKJXlcJv1cor+cc=; b=QLN4GBhnbA83aq6qOliJgBUpxscTILYzIYn4FBmCQUXsY0guqxzDTGzMbv1wiLpjeJd1 1kXhbx5PtNDp8z7ZTQ8fX09HUzubN2G3RIFZwNrb6q6/an9SdF20Ts13Qs2pjagC42Sm Juvu/nK1ng81PI8ZbYGbcNHF5ep+mp6hU1cA4DUFQyi3EOL0TJamGK9WLtGOlf098kqF YxeRX/gIKjuwc6oYotHgsf9+jhI3xzlBhUTYWLp0F5JSGlJkNiaalY/WlhzJwKKyrFy3 06/uLfeJU75Ff9urSFA88fxTXl+9jGCXavrFP6auX4NXkV9+GpQKEcIw8u7pTAeM7mef cQ== Received: from mail.thefacebook.com ([163.114.132.120]) by mx0a-00082601.pphosted.com (PPS) with ESMTPS id 3m8561avw9-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Mon, 05 Dec 2022 07:19:31 -0800 Received: from twshared16963.27.frc3.facebook.com (2620:10d:c085:108::8) by mail.thefacebook.com (2620:10d:c085:21d::4) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.31; Mon, 5 Dec 2022 07:19:29 -0800 Received: by devbig007.nao1.facebook.com (Postfix, from userid 544533) id BDE46C615708; Mon, 5 Dec 2022 07:00:12 -0800 (PST) From: Keith Busch To: , , Matthew Wilcox CC: Tony Battersby , Keith Busch Subject: [PATCH 09/11] dmapool: simplify freeing Date: Mon, 5 Dec 2022 06:59:35 -0800 Message-ID: <20221205145937.54367-10-kbusch@meta.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20221205145937.54367-1-kbusch@meta.com> References: <20221205145937.54367-1-kbusch@meta.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-FB-Internal: Safe X-Proofpoint-GUID: PymH5d3YwBk1q4cny15SOprF69vKwoqV X-Proofpoint-ORIG-GUID: PymH5d3YwBk1q4cny15SOprF69vKwoqV X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.205,Aquarius:18.0.923,Hydra:6.0.545,FMLib:17.11.122.1 definitions=2022-12-05_01,2022-12-05_01,2022-06-22_01 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Keith Busch The actions for busy and not busy are mostly the same, so combine these and remove the unnecessary function. Also, the pool is about to be freed so there's no need to poison the page data since we only check for poison on alloc, which can't be done on a freed pool. Signed-off-by: Keith Busch --- mm/dmapool.c | 29 ++++++----------------------- 1 file changed, 6 insertions(+), 23 deletions(-) diff --git a/mm/dmapool.c b/mm/dmapool.c index 33d20ceff18c5..44622f2bf4641 100644 --- a/mm/dmapool.c +++ b/mm/dmapool.c @@ -241,18 +241,6 @@ static inline bool is_page_busy(struct dma_page *page) return page->in_use !=3D 0; } =20 -static void pool_free_page(struct dma_pool *pool, struct dma_page *page) -{ - dma_addr_t dma =3D page->dma; - -#ifdef DMAPOOL_DEBUG - memset(page->vaddr, POOL_POISON_FREED, pool->allocation); -#endif - dma_free_coherent(pool->dev, pool->allocation, page->vaddr, dma); - list_del(&page->page_list); - kfree(page); -} - /** * dma_pool_destroy - destroys a pool of dma memory blocks. * @pool: dma pool that will be destroyed @@ -280,14 +268,14 @@ void dma_pool_destroy(struct dma_pool *pool) mutex_unlock(&pools_reg_lock); =20 list_for_each_entry_safe(page, tmp, &pool->page_list, page_list) { - if (is_page_busy(page)) { + if (!is_page_busy(page)) + dma_free_coherent(pool->dev, pool->allocation, + page->vaddr, page->dma); + else dev_err(pool->dev, "%s %s, %p busy\n", __func__, pool->name, page->vaddr); - /* leak the still-in-use consistent memory */ - list_del(&page->page_list); - kfree(page); - } else - pool_free_page(pool, page); + list_del(&page->page_list); + kfree(page); } =20 kfree(pool); @@ -445,11 +433,6 @@ void dma_pool_free(struct dma_pool *pool, void *vaddr,= dma_addr_t dma) page->in_use--; *(int *)vaddr =3D page->offset; page->offset =3D offset; - /* - * Resist a temptation to do - * if (!is_page_busy(page)) pool_free_page(pool, page); - * Better have a few empty pages hang around. - */ spin_unlock_irqrestore(&pool->lock, flags); } EXPORT_SYMBOL(dma_pool_free); --=20 2.30.2 From nobody Thu Sep 18 15:40:48 2025 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9C207C4708D for ; Mon, 5 Dec 2022 15:07:26 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232235AbiLEPHZ (ORCPT ); Mon, 5 Dec 2022 10:07:25 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38314 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232070AbiLEPHW (ORCPT ); Mon, 5 Dec 2022 10:07:22 -0500 Received: from mx0a-00082601.pphosted.com (mx0a-00082601.pphosted.com [67.231.145.42]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 88D7C1D32E for ; Mon, 5 Dec 2022 07:07:21 -0800 (PST) Received: from pps.filterd (m0044010.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 2B5Er24L016570 for ; Mon, 5 Dec 2022 07:07:21 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=meta.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=s2048-2021-q4; bh=F5ndiXKSIYwdaNgVa7CtE+9m7dnA1GNZSuFoZd2plto=; b=mchHDRAi5Y0NSIA/d5ZQByFsbGCqJGxyw/MVRIdaNFXNvPVeaq988bd12h7IoioZs6EI SD3zJCyGVReY0pzNdSxOeJf0mpdvBksWs7eulu083Kaw7Nt4wVr0j2r9fPfgxbhGelIq TFbLZfXKIiP/0F9VtKtg0fo7M35YRx/T5YOxFFL93ljc4clxOAq4ILIpu71JEHGgKLHO u97Z05whH18sMXo18co/ua/e3GQksMUQjhtUyV8cbzMmJj1JJ3Rev4+6w9x9tjl5H/5s JahH1T9bKot/R3RA3a2HtRbL19m1hjz87pPdVDQAHQ84L6zHNbRHf+XwlaQzr3VkyNRL Xw== Received: from maileast.thefacebook.com ([163.114.130.16]) by mx0a-00082601.pphosted.com (PPS) with ESMTPS id 3m82ntv08u-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Mon, 05 Dec 2022 07:07:21 -0800 Received: from twshared16963.27.frc3.facebook.com (2620:10d:c0a8:1b::d) by mail.thefacebook.com (2620:10d:c0a8:83::7) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.31; Mon, 5 Dec 2022 07:07:19 -0800 Received: by devbig007.nao1.facebook.com (Postfix, from userid 544533) id 49143C615720; Mon, 5 Dec 2022 07:00:12 -0800 (PST) From: Keith Busch To: , , Matthew Wilcox CC: Tony Battersby , Keith Busch Subject: [PATCH 10/11] dmapool: don't memset on free twice Date: Mon, 5 Dec 2022 06:59:36 -0800 Message-ID: <20221205145937.54367-11-kbusch@meta.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20221205145937.54367-1-kbusch@meta.com> References: <20221205145937.54367-1-kbusch@meta.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-FB-Internal: Safe X-Proofpoint-ORIG-GUID: 7zteIjRPGoiqQ6V4rJv-8VpjGRg8hYF0 X-Proofpoint-GUID: 7zteIjRPGoiqQ6V4rJv-8VpjGRg8hYF0 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.205,Aquarius:18.0.923,Hydra:6.0.545,FMLib:17.11.122.1 definitions=2022-12-05_01,2022-12-05_01,2022-06-22_01 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Keith Busch If debug is enabled, dmapool will poison the range, so no need to clear it to 0 immediately before writing over it. Signed-off-by: Keith Busch --- mm/dmapool.c | 14 ++++++++++---- 1 file changed, 10 insertions(+), 4 deletions(-) diff --git a/mm/dmapool.c b/mm/dmapool.c index 44622f2bf4641..f5b79c3268856 100644 --- a/mm/dmapool.c +++ b/mm/dmapool.c @@ -372,10 +372,10 @@ static struct dma_page *pool_find_page(struct dma_poo= l *pool, dma_addr_t dma) return NULL; } =20 +#ifdef DMAPOOL_DEBUG static inline bool pool_page_err(struct dma_pool *pool, struct dma_page *p= age, void *vaddr) { -#ifdef DMAPOOL_DEBUG unsigned int chain =3D page->offset; =20 if ((dma - page->dma) !=3D offset) { @@ -394,9 +394,17 @@ static inline bool pool_page_err(struct dma_pool *pool= , struct dma_page *page, return true; } memset(vaddr, POOL_POISON_FREED, pool->size); -#endif return false; } +#else +static inline bool pool_page_err(struct dma_pool *pool, struct dma_page *p= age, + void *vaddr) +{ + if (want_init_on_free()) + memset(vaddr, 0, pool->size); + return false; +} +#endif =20 /** * dma_pool_free - put block back into dma pool @@ -423,8 +431,6 @@ void dma_pool_free(struct dma_pool *pool, void *vaddr, = dma_addr_t dma) } =20 offset =3D vaddr - page->vaddr; - if (want_init_on_free()) - memset(vaddr, 0, pool->size); if (pool_page_err(pool, page, vaddr)) { spin_unlock_irqrestore(&pool->lock, flags); return; --=20 2.30.2 From nobody Thu Sep 18 15:40:48 2025 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3F914C4321E for ; Mon, 5 Dec 2022 15:14:24 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231230AbiLEPOX (ORCPT ); Mon, 5 Dec 2022 10:14:23 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45546 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231838AbiLEPNf (ORCPT ); Mon, 5 Dec 2022 10:13:35 -0500 Received: from mx0b-00082601.pphosted.com (mx0b-00082601.pphosted.com [67.231.153.30]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 61B571010 for ; Mon, 5 Dec 2022 07:13:33 -0800 (PST) Received: from pps.filterd (m0109331.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 2B5B8jGS025632 for ; Mon, 5 Dec 2022 07:13:33 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=meta.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=s2048-2021-q4; bh=1ZsQ24c7xBxg+bQGPfjzx0xptWB/LenrbaJZ6LhGCHo=; b=U69QM2oJlw0o5rHLUMeD4J7s5YpqYUxubHHE6brB0x+xvIWzZ1cCMB52Hc4FMFjYCbG0 FjRk3vG1H9SQ+Is+D1H6+E0h409TS1Ff1pZo1IuHIqvaeI36x5GIqZ2wozARJc1Bbkrp stMq0YLh45JigeZonkayo/DtZZgCktoO1ldw1RExXp29COnNW7q0p8kaGh4WmHVwJpmN dUyT1vIGkBAynotsuKsNeBEGsFLmYTtEQyN1tlE8b2OH9gXfeQduxTc4o8agNHV45r8o 1h/wUNZDm8vEy+lWxsSW8N1ttrrKJ/rlxBK0pJK5pAnXLVlQRKat0FcnevujYxzJfSJY Ig== Received: from mail.thefacebook.com ([163.114.132.120]) by mx0a-00082601.pphosted.com (PPS) with ESMTPS id 3m8561auny-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Mon, 05 Dec 2022 07:13:33 -0800 Received: from twshared2003.08.ash9.facebook.com (2620:10d:c085:108::4) by mail.thefacebook.com (2620:10d:c085:11d::5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.31; Mon, 5 Dec 2022 07:13:31 -0800 Received: by devbig007.nao1.facebook.com (Postfix, from userid 544533) id 6A86CC61572C; Mon, 5 Dec 2022 07:00:13 -0800 (PST) From: Keith Busch To: , , Matthew Wilcox CC: Tony Battersby , Keith Busch Subject: [PATCH 11/11] dmapool: link blocks across pages Date: Mon, 5 Dec 2022 06:59:37 -0800 Message-ID: <20221205145937.54367-12-kbusch@meta.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20221205145937.54367-1-kbusch@meta.com> References: <20221205145937.54367-1-kbusch@meta.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-FB-Internal: Safe X-Proofpoint-GUID: q02K3_yzZF6hoTErihxaKG4rddpwEent X-Proofpoint-ORIG-GUID: q02K3_yzZF6hoTErihxaKG4rddpwEent X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.205,Aquarius:18.0.923,Hydra:6.0.545,FMLib:17.11.122.1 definitions=2022-12-05_01,2022-12-05_01,2022-06-22_01 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Keith Busch The allocated dmapool pages are never freed for the lifetime of the pool. There is no need for the two level list+stack lookup for finding a free block since nothing is ever removed from the list. Just use a simple stack, reducing time complexity to constant. The implementation inserts the stack linking elements and the dma handle of the block within itself when freed. This means the smallest possible dmapool block is increased to at most 16 bytes to accomodate these fields, but there are no exisiting users requesting a dma pool smaller than that anyway. Removing the list has a significant change in performance. Using the kernel's micro-benchmarking self test: Before: # modprobe dmapool_test dmapool test: size:16 blocks:8192 time:57282 dmapool test: size:64 blocks:8192 time:172562 dmapool test: size:256 blocks:8192 time:789247 dmapool test: size:1024 blocks:2048 time:371823 dmapool test: size:4096 blocks:1024 time:362237 After: # modprobe dmapool_test dmapool test: size:16 blocks:8192 time:24997 dmapool test: size:64 blocks:8192 time:26584 dmapool test: size:256 blocks:8192 time:33542 dmapool test: size:1024 blocks:2048 time:9022 dmapool test: size:4096 blocks:1024 time:6045 The module test allocates quite a few blocks that may not accurately represent how these pools are used in real life. For a more marco level benchmark, running fio high-depth + high-batched on nvme, this patch shows submission and completion latency reduced by ~100usec each, 1% IOPs improvement, and perf record's time spent in dma_pool_alloc/free were reduced by half. Signed-off-by: Keith Busch --- mm/dmapool.c | 213 ++++++++++++++++++++++++++------------------------- 1 file changed, 108 insertions(+), 105 deletions(-) diff --git a/mm/dmapool.c b/mm/dmapool.c index f5b79c3268856..ca6cc5d3d9e53 100644 --- a/mm/dmapool.c +++ b/mm/dmapool.c @@ -40,13 +40,22 @@ #define DMAPOOL_DEBUG 1 #endif =20 +struct dma_block { + struct dma_block *next_block; + dma_addr_t dma; +}; + struct dma_pool { /* the pool */ struct list_head page_list; spinlock_t lock; struct device *dev; + struct dma_block *next_block; unsigned int size; unsigned int allocation; unsigned int boundary; + unsigned int nr_blocks; + unsigned int nr_active; + unsigned int nr_pages; char name[32]; struct list_head pools; }; @@ -55,8 +64,6 @@ struct dma_page { /* cacheable header for 'allocation' b= ytes */ struct list_head page_list; void *vaddr; dma_addr_t dma; - unsigned int in_use; - unsigned int offset; }; =20 static DEFINE_MUTEX(pools_lock); @@ -64,30 +71,18 @@ static DEFINE_MUTEX(pools_reg_lock); =20 static ssize_t pools_show(struct device *dev, struct device_attribute *att= r, char *buf) { - int size; - struct dma_page *page; struct dma_pool *pool; + unsigned size; =20 size =3D sysfs_emit(buf, "poolinfo - 0.1\n"); =20 mutex_lock(&pools_lock); list_for_each_entry(pool, &dev->dma_pools, pools) { - unsigned pages =3D 0; - size_t blocks =3D 0; - - spin_lock_irq(&pool->lock); - list_for_each_entry(page, &pool->page_list, page_list) { - pages++; - blocks +=3D page->in_use; - } - spin_unlock_irq(&pool->lock); - /* per-pool info, no real statistics yet */ - size +=3D sysfs_emit_at(buf, size, "%-16s %4zu %4zu %4u %2u\n", - pool->name, blocks, - (size_t) pages * - (pool->allocation / pool->size), - pool->size, pages); + size +=3D sysfs_emit_at(buf, size, "%-16s %4u %4u %4u %2u\n", + pool->name, pool->nr_active, + pool->nr_blocks, pool->size, + pool->nr_pages); } mutex_unlock(&pools_lock); =20 @@ -96,6 +91,25 @@ static ssize_t pools_show(struct device *dev, struct dev= ice_attribute *attr, cha =20 static DEVICE_ATTR_RO(pools); =20 +static inline struct dma_block *pool_block_pop(struct dma_pool *pool) +{ + struct dma_block *block =3D pool->next_block; + + if (block) { + pool->next_block =3D block->next_block; + pool->nr_active++; + } + return block; +} + +static inline void pool_block_push(struct dma_pool *pool, struct dma_block= *block, + dma_addr_t dma) +{ + block->dma =3D dma; + block->next_block =3D pool->next_block; + pool->next_block =3D block; +} + /** * dma_pool_create - Creates a pool of consistent memory blocks, for dma. * @name: name of pool, for diagnostics @@ -136,8 +150,8 @@ struct dma_pool *dma_pool_create(const char *name, stru= ct device *dev, =20 if (size =3D=3D 0 || size > INT_MAX) return NULL; - else if (size < 4) - size =3D 4; + if (size < sizeof(struct dma_block)) + size =3D sizeof(struct dma_block); =20 size =3D ALIGN(size, align); allocation =3D max_t(size_t, size, PAGE_SIZE); @@ -162,6 +176,10 @@ struct dma_pool *dma_pool_create(const char *name, str= uct device *dev, retval->size =3D size; retval->boundary =3D boundary; retval->allocation =3D allocation; + retval->nr_blocks =3D 0; + retval->nr_active =3D 0; + retval->nr_pages =3D 0; + retval->next_block =3D NULL; =20 INIT_LIST_HEAD(&retval->pools); =20 @@ -199,22 +217,24 @@ EXPORT_SYMBOL(dma_pool_create); =20 static void pool_initialise_page(struct dma_pool *pool, struct dma_page *p= age) { - unsigned int offset =3D 0; - unsigned int next_boundary =3D pool->boundary; - - page->in_use =3D 0; - page->offset =3D 0; - do { - unsigned int next =3D offset + pool->size; - if (unlikely((next + pool->size) >=3D next_boundary)) { - next =3D next_boundary; + unsigned int next_boundary =3D pool->boundary, offset =3D 0; + struct dma_block *block; + + while (offset < pool->allocation) { + if (offset > next_boundary) { + offset =3D next_boundary; next_boundary +=3D pool->boundary; + continue; } - *(int *)(page->vaddr + offset) =3D next; - offset =3D next; - } while (offset < pool->allocation); + + block =3D page->vaddr + offset; + pool_block_push(pool, block, page->dma + offset); + offset +=3D pool->size; + pool->nr_blocks++; + } =20 list_add(&page->page_list, &pool->page_list); + pool->nr_pages++; } =20 static struct dma_page *pool_alloc_page(struct dma_pool *pool, gfp_t mem_f= lags) @@ -236,11 +256,6 @@ static struct dma_page *pool_alloc_page(struct dma_poo= l *pool, gfp_t mem_flags) return page; } =20 -static inline bool is_page_busy(struct dma_page *page) -{ - return page->in_use !=3D 0; -} - /** * dma_pool_destroy - destroys a pool of dma memory blocks. * @pool: dma pool that will be destroyed @@ -252,7 +267,7 @@ static inline bool is_page_busy(struct dma_page *page) void dma_pool_destroy(struct dma_pool *pool) { struct dma_page *page, *tmp; - bool empty =3D false; + bool empty =3D false, busy =3D false; =20 if (unlikely(!pool)) return; @@ -267,13 +282,15 @@ void dma_pool_destroy(struct dma_pool *pool) device_remove_file(pool->dev, &dev_attr_pools); mutex_unlock(&pools_reg_lock); =20 + if (pool->nr_active) { + dev_err(pool->dev, "%s %s busy\n", __func__, pool->name); + busy =3D true; + } + list_for_each_entry_safe(page, tmp, &pool->page_list, page_list) { - if (!is_page_busy(page)) + if (!busy) dma_free_coherent(pool->dev, pool->allocation, page->vaddr, page->dma); - else - dev_err(pool->dev, "%s %s, %p busy\n", __func__, - pool->name, page->vaddr); list_del(&page->page_list); kfree(page); } @@ -282,18 +299,18 @@ void dma_pool_destroy(struct dma_pool *pool) } EXPORT_SYMBOL(dma_pool_destroy); =20 -static inline void pool_check_block(struct dma_pool *pool, void *retval, - unsigned int offset, gfp_t mem_flags) +static inline void pool_check_block(struct dma_pool *pool, struct dma_bloc= k *block, + gfp_t mem_flags) { -#ifdef DMAPOOL_DEBUG +#ifdef DMAPOOL_DEBUG + u8 *data =3D block; int i; - u8 *data =3D retval; - /* page->offset is stored in first 4 bytes */ - for (i =3D sizeof(offset); i < pool->size; i++) { + + for (i =3D sizeof(struct dma_block); i < pool->size; i++) { if (data[i] =3D=3D POOL_POISON_FREED) continue; - dev_err(pool->dev, "%s %s, %p (corrupted)\n", - __func__, pool->name, retval); + dev_err(pool->dev, "%s %s, %p (corrupted)\n", __func__, + pool->name, block); =20 /* * Dump the first 4 bytes even if they are not @@ -303,8 +320,9 @@ static inline void pool_check_block(struct dma_pool *po= ol, void *retval, data, pool->size, 1); break; } + if (!want_init_on_alloc(mem_flags)) - memset(retval, POOL_POISON_ALLOCATED, pool->size); + memset(block, POOL_POISON_ALLOCATED, pool->size); #endif } =20 @@ -321,44 +339,41 @@ static inline void pool_check_block(struct dma_pool *= pool, void *retval, void *dma_pool_alloc(struct dma_pool *pool, gfp_t mem_flags, dma_addr_t *handle) { - unsigned long flags; + struct dma_block *block; struct dma_page *page; - unsigned int offset; - void *retval; + unsigned long flags; =20 might_alloc(mem_flags); =20 spin_lock_irqsave(&pool->lock, flags); - list_for_each_entry(page, &pool->page_list, page_list) { - if (page->offset < pool->allocation) - goto ready; - } - - /* pool_alloc_page() might sleep, so temporarily drop &pool->lock */ - spin_unlock_irqrestore(&pool->lock, flags); + block =3D pool_block_pop(pool); + if (!block) { + /* + * pool_alloc_page() might sleep, so temporarily drop + * &pool->lock + */ + spin_unlock_irqrestore(&pool->lock, flags); =20 - page =3D pool_alloc_page(pool, mem_flags & (~__GFP_ZERO)); - if (!page) - return NULL; + page =3D pool_alloc_page(pool, mem_flags & (~__GFP_ZERO)); + if (!page) + return NULL; =20 - spin_lock_irqsave(&pool->lock, flags); - pool_initialise_page(pool, page); - ready: - page->in_use++; - offset =3D page->offset; - page->offset =3D *(int *)(page->vaddr + offset); - retval =3D offset + page->vaddr; - *handle =3D offset + page->dma; - pool_check_block(pool, retval, offset, mem_flags); + spin_lock_irqsave(&pool->lock, flags); + pool_initialise_page(pool, page); + block =3D pool_block_pop(pool); + } spin_unlock_irqrestore(&pool->lock, flags); =20 + *handle =3D block->dma; + pool_check_block(pool, block, mem_flags); if (want_init_on_alloc(mem_flags)) - memset(retval, 0, pool->size); + memset(block, 0, pool->size); =20 - return retval; + return block; } EXPORT_SYMBOL(dma_pool_alloc); =20 +#ifdef DMAPOOL_DEBUG static struct dma_page *pool_find_page(struct dma_pool *pool, dma_addr_t d= ma) { struct dma_page *page; @@ -372,33 +387,35 @@ static struct dma_page *pool_find_page(struct dma_poo= l *pool, dma_addr_t dma) return NULL; } =20 -#ifdef DMAPOOL_DEBUG -static inline bool pool_page_err(struct dma_pool *pool, struct dma_page *p= age, - void *vaddr) +static inline bool pool_block_err(struct dma_pool *pool, void *vaddr, + dma_addr_t dma) { - unsigned int chain =3D page->offset; + struct dma_block *block =3D pool->next_block; + struct dma_page *page; =20 - if ((dma - page->dma) !=3D offset) { - dev_err(pool->dev, "%s %s, %p (bad vaddr)/%pad\n", + page =3D pool_find_page(pool, dma); + if (!page) { + dev_err(pool->dev, "%s %s, %p/%pad (bad dma)\n", __func__, pool->name, vaddr, &dma); return true; } =20 - while (chain < pool->allocation) { - if (chain !=3D offset) { - chain =3D *(int *)(page->vaddr + chain); + while (block) { + if (block !=3D vaddr) { + block =3D block->next_block; continue; } dev_err(pool->dev, "%s %s, dma %pad already free\n", __func__, pool->name, &dma); return true; } + memset(vaddr, POOL_POISON_FREED, pool->size); return false; } #else -static inline bool pool_page_err(struct dma_pool *pool, struct dma_page *p= age, - void *vaddr) +static inline bool pool_block_err(struct dma_pool *pool, void *vaddr, + dma_addr_t dma) { if (want_init_on_free()) memset(vaddr, 0, pool->size); @@ -417,28 +434,14 @@ static inline bool pool_page_err(struct dma_pool *poo= l, struct dma_page *page, */ void dma_pool_free(struct dma_pool *pool, void *vaddr, dma_addr_t dma) { - struct dma_page *page; + struct dma_block *block =3D vaddr; unsigned long flags; - unsigned int offset; =20 spin_lock_irqsave(&pool->lock, flags); - page =3D pool_find_page(pool, dma); - if (!page) { - spin_unlock_irqrestore(&pool->lock, flags); - dev_err(pool->dev, "%s %s, %p/%pad (bad dma)\n", - __func__, pool->name, vaddr, &dma); - return; + if (!pool_block_err(pool, vaddr, dma)) { + pool_block_push(pool, block, dma); + pool->nr_active--; } - - offset =3D vaddr - page->vaddr; - if (pool_page_err(pool, page, vaddr)) { - spin_unlock_irqrestore(&pool->lock, flags); - return; - } - - page->in_use--; - *(int *)vaddr =3D page->offset; - page->offset =3D offset; spin_unlock_irqrestore(&pool->lock, flags); } EXPORT_SYMBOL(dma_pool_free); --=20 2.30.2