From nobody Tue Feb 10 05:10:08 2026 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) client-ip=192.237.175.120; envelope-from=xen-devel-bounces@lists.xenproject.org; helo=lists.xenproject.org; Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=pass(p=none dis=none) header.from=chromium.org ARC-Seal: i=1; a=rsa-sha256; t=1609911813; cv=none; d=zohomail.com; s=zohoarc; b=O5gqm41pOrb/dQgdLClc72YVJCbcU8uxIBmKr7uKJ98dFk9AEUkZPFMFyUAZi209jQHmuj2iFo8aW6iHYfraAqFmVS08iBJVj5JXqBVDGJGIXA9J9gqeWU5MX3oznLtGcwQbQfaCR4ovpta7PKVsHqBBN0gpqPM7djsZMVq0F6k= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1609911813; h=Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=iaVOsKMvFDJYn55B6p38dczfl91LGu9UJUDGkSKIIuA=; b=GnsbYIJCEwFatX08jWWcM7nG9kOjsOlmsyiLM+lfiKV5T+B+WZ2+3/ZdqyineKPOyMM19TqyLVA6lAFnZmUs+rwdrTjcrO2klijzt8KGV76cNtS8QnR+qnegaTrY0EN6kPImo1q8SJmUygGChbiecvIYT8kdPBxCHi16lvA2mjs= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=pass header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) by mx.zohomail.com with SMTPS id 160991181299749.84328131971597; Tue, 5 Jan 2021 21:43:32 -0800 (PST) Received: from list by lists.xenproject.org with outflank-mailman.62298.110303 (Exim 4.92) (envelope-from ) id 1kx1bC-00018D-Sv; Wed, 06 Jan 2021 05:43:10 +0000 Received: by outflank-mailman (output) from mailman id 62298.110303; Wed, 06 Jan 2021 05:43:10 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1kx1bC-000183-Mv; Wed, 06 Jan 2021 05:43:10 +0000 Received: by outflank-mailman (input) for mailman id 62298; Wed, 06 Jan 2021 03:41:42 +0000 Received: from us1-rack-iad1.inumbo.com ([172.99.69.81]) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1kwzhe-0005e4-7x for xen-devel@lists.xenproject.org; Wed, 06 Jan 2021 03:41:42 +0000 Received: from mail-pl1-x62d.google.com (unknown [2607:f8b0:4864:20::62d]) by us1-rack-iad1.inumbo.com (Halon) with ESMTPS id cd4f283c-986b-4984-b9ab-f4a3892b60de; Wed, 06 Jan 2021 03:41:40 +0000 (UTC) Received: by mail-pl1-x62d.google.com with SMTP id q4so876723plr.7 for ; Tue, 05 Jan 2021 19:41:40 -0800 (PST) Received: from localhost ([2401:fa00:1:10:3e52:82ff:fe5e:cc9d]) by smtp.gmail.com with ESMTPSA id v15sm690974pfn.217.2021.01.05.19.41.33 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Tue, 05 Jan 2021 19:41:38 -0800 (PST) X-Outflank-Mailman: Message body and most headers restored to incoming version X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: cd4f283c-986b-4984-b9ab-f4a3892b60de DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=iaVOsKMvFDJYn55B6p38dczfl91LGu9UJUDGkSKIIuA=; b=Wd25yMi/LH9CrpDNCdod8ZuwAhKMpkRR0yK0iyOtUJX5+JrAlM8LRrWgdVy0/ZmO7b 1jsT6wpnhxzgyKVtcN0/irI63tatCC+aG0Ke+zE9Hqqq+IC5FUQ/gEsRtxo1Iz0zjdp4 mrFQo9eapIqiFBskqxNfIZwWHXaV3iOAHVbX8= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=iaVOsKMvFDJYn55B6p38dczfl91LGu9UJUDGkSKIIuA=; b=p3KDIqSi4Q6raR3j5zBuv6eFpcDJqCGgguSbuVJpt8yD2FmGlIV8HPplZsVcXdTdoc 6r86QrW/EDEmtwRKhdVX1AbN2NbxHwXB44/0Jt4FiPrxWvrYUeJKrSVYuty5hnTqQRcO w3/AELR/m+YB03075gWVNeOOo8MrbDjgCGtcFvSzDRcSbTAmlwHgKPzfnJ0jrg4H0KaK PcGSpgd32OjI9VyWRC7MRzWJpOBfMX/EjQIfXzHNmTe23VSgZuvU/qg/hD6hCZC3jiPt L5izKr6qY3uqBNJcWrZzUYD2vFUO9tPxhF6XjOjSCNxC9ItYgRRPT9gk7Qd3m2HH2R5o aUXg== X-Gm-Message-State: AOAM532WdOKLWs/tLpnhHoKBT1On/9nNQ7l/m7gisvG85I91jNN4ETVz NPfXLXeI9nbD0VTHFJltyVsRmw== X-Google-Smtp-Source: ABdhPJyuJpkeVt0ODIilijkP6Uq2ZjzjhDDZESbhcJhHxYV7NpHV8otDe9U+HR7alXQpTjFLXj3ueQ== X-Received: by 2002:a17:90b:1882:: with SMTP id mn2mr2249148pjb.236.1609904499318; Tue, 05 Jan 2021 19:41:39 -0800 (PST) From: Claire Chang To: robh+dt@kernel.org, mpe@ellerman.id.au, benh@kernel.crashing.org, paulus@samba.org, joro@8bytes.org, will@kernel.org, frowand.list@gmail.com, konrad.wilk@oracle.com, boris.ostrovsky@oracle.com, jgross@suse.com, sstabellini@kernel.org, hch@lst.de, m.szyprowski@samsung.com, robin.murphy@arm.com Cc: grant.likely@arm.com, xypron.glpk@gmx.de, treding@nvidia.com, mingo@kernel.org, bauerman@linux.ibm.com, peterz@infradead.org, gregkh@linuxfoundation.org, saravanak@google.com, rafael.j.wysocki@intel.com, heikki.krogerus@linux.intel.com, andriy.shevchenko@linux.intel.com, rdunlap@infradead.org, dan.j.williams@intel.com, bgolaszewski@baylibre.com, devicetree@vger.kernel.org, linux-kernel@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, iommu@lists.linux-foundation.org, xen-devel@lists.xenproject.org, tfiga@chromium.org, drinkcat@chromium.org, Claire Chang Subject: [RFC PATCH v3 1/6] swiotlb: Add io_tlb_mem struct Date: Wed, 6 Jan 2021 11:41:19 +0800 Message-Id: <20210106034124.30560-2-tientzu@chromium.org> X-Mailer: git-send-email 2.29.2.729.g45daf8777d-goog In-Reply-To: <20210106034124.30560-1-tientzu@chromium.org> References: <20210106034124.30560-1-tientzu@chromium.org> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ZohoMail-DKIM: pass (identity @chromium.org) Content-Type: text/plain; charset="utf-8" Added a new struct, io_tlb_mem, as the IO TLB memory pool descriptor and moved relevant global variables into that struct. This will be useful later to allow for restricted DMA pool. Signed-off-by: Claire Chang --- arch/powerpc/platforms/pseries/svm.c | 4 +- drivers/xen/swiotlb-xen.c | 4 +- include/linux/swiotlb.h | 39 +++- kernel/dma/swiotlb.c | 292 +++++++++++++-------------- 4 files changed, 178 insertions(+), 161 deletions(-) diff --git a/arch/powerpc/platforms/pseries/svm.c b/arch/powerpc/platforms/= pseries/svm.c index 7b739cc7a8a9..2b767f1ca5fd 100644 --- a/arch/powerpc/platforms/pseries/svm.c +++ b/arch/powerpc/platforms/pseries/svm.c @@ -55,8 +55,8 @@ void __init svm_swiotlb_init(void) if (vstart && !swiotlb_init_with_tbl(vstart, io_tlb_nslabs, false)) return; =20 - if (io_tlb_start) - memblock_free_early(io_tlb_start, + if (io_tlb_default_mem.start) + memblock_free_early(io_tlb_default_mem.start, PAGE_ALIGN(io_tlb_nslabs << IO_TLB_SHIFT)); panic("SVM: Cannot allocate SWIOTLB buffer"); } diff --git a/drivers/xen/swiotlb-xen.c b/drivers/xen/swiotlb-xen.c index 2b385c1b4a99..4d17dff7ffd2 100644 --- a/drivers/xen/swiotlb-xen.c +++ b/drivers/xen/swiotlb-xen.c @@ -192,8 +192,8 @@ int __ref xen_swiotlb_init(int verbose, bool early) /* * IO TLB memory already allocated. Just use it. */ - if (io_tlb_start !=3D 0) { - xen_io_tlb_start =3D phys_to_virt(io_tlb_start); + if (io_tlb_default_mem.start !=3D 0) { + xen_io_tlb_start =3D phys_to_virt(io_tlb_default_mem.start); goto end; } =20 diff --git a/include/linux/swiotlb.h b/include/linux/swiotlb.h index d9c9fc9ca5d2..dd8eb57cbb8f 100644 --- a/include/linux/swiotlb.h +++ b/include/linux/swiotlb.h @@ -70,11 +70,46 @@ dma_addr_t swiotlb_map(struct device *dev, phys_addr_t = phys, =20 #ifdef CONFIG_SWIOTLB extern enum swiotlb_force swiotlb_force; -extern phys_addr_t io_tlb_start, io_tlb_end; + +/** + * struct io_tlb_mem - IO TLB Memory Pool Descriptor + * + * @start: The start address of the swiotlb memory pool. Used to do a quick + * range check to see if the memory was in fact allocated by this + * API. + * @end: The end address of the swiotlb memory pool. Used to do a quick + * range check to see if the memory was in fact allocated by this + * API. + * @nslabs: The number of IO TLB blocks (in groups of 64) between @start a= nd + * @end. This is command line adjustable via setup_io_tlb_npages. + * @used: The number of used IO TLB block. + * @list: The free list describing the number of free entries available + * from each index. + * @index: The index to start searching in the next round. + * @orig_addr: The original address corresponding to a mapped entry for the + * sync operations. + * @lock: The lock to protect the above data structures in the map and + * unmap calls. + * @debugfs: The dentry to debugfs. + */ +struct io_tlb_mem { + phys_addr_t start; + phys_addr_t end; + unsigned long nslabs; + unsigned long used; + unsigned int *list; + unsigned int index; + phys_addr_t *orig_addr; + spinlock_t lock; + struct dentry *debugfs; +}; +extern struct io_tlb_mem io_tlb_default_mem; =20 static inline bool is_swiotlb_buffer(phys_addr_t paddr) { - return paddr >=3D io_tlb_start && paddr < io_tlb_end; + struct io_tlb_mem *mem =3D &io_tlb_default_mem; + + return paddr >=3D mem->start && paddr < mem->end; } =20 void __init swiotlb_exit(void); diff --git a/kernel/dma/swiotlb.c b/kernel/dma/swiotlb.c index 7c42df6e6100..e4368159f88a 100644 --- a/kernel/dma/swiotlb.c +++ b/kernel/dma/swiotlb.c @@ -61,33 +61,11 @@ * allocate a contiguous 1MB, we're probably in trouble anyway. */ #define IO_TLB_MIN_SLABS ((1<<20) >> IO_TLB_SHIFT) +#define INVALID_PHYS_ADDR (~(phys_addr_t)0) =20 enum swiotlb_force swiotlb_force; =20 -/* - * Used to do a quick range check in swiotlb_tbl_unmap_single and - * swiotlb_tbl_sync_single_*, to see if the memory was in fact allocated b= y this - * API. - */ -phys_addr_t io_tlb_start, io_tlb_end; - -/* - * The number of IO TLB blocks (in groups of 64) between io_tlb_start and - * io_tlb_end. This is command line adjustable via setup_io_tlb_npages. - */ -static unsigned long io_tlb_nslabs; - -/* - * The number of used IO TLB block - */ -static unsigned long io_tlb_used; - -/* - * This is a free list describing the number of free entries available from - * each index - */ -static unsigned int *io_tlb_list; -static unsigned int io_tlb_index; +struct io_tlb_mem io_tlb_default_mem; =20 /* * Max segment that we can provide which (if pages are contingous) will @@ -95,27 +73,17 @@ static unsigned int io_tlb_index; */ static unsigned int max_segment; =20 -/* - * We need to save away the original address corresponding to a mapped ent= ry - * for the sync operations. - */ -#define INVALID_PHYS_ADDR (~(phys_addr_t)0) -static phys_addr_t *io_tlb_orig_addr; - -/* - * Protect the above data structures in the map and unmap calls - */ -static DEFINE_SPINLOCK(io_tlb_lock); - static int late_alloc; =20 static int __init setup_io_tlb_npages(char *str) { + struct io_tlb_mem *mem =3D &io_tlb_default_mem; + if (isdigit(*str)) { - io_tlb_nslabs =3D simple_strtoul(str, &str, 0); + mem->nslabs =3D simple_strtoul(str, &str, 0); /* avoid tail segment of size < IO_TLB_SEGSIZE */ - io_tlb_nslabs =3D ALIGN(io_tlb_nslabs, IO_TLB_SEGSIZE); + mem->nslabs =3D ALIGN(mem->nslabs, IO_TLB_SEGSIZE); } if (*str =3D=3D ',') ++str; @@ -123,7 +91,7 @@ setup_io_tlb_npages(char *str) swiotlb_force =3D SWIOTLB_FORCE; } else if (!strcmp(str, "noforce")) { swiotlb_force =3D SWIOTLB_NO_FORCE; - io_tlb_nslabs =3D 1; + mem->nslabs =3D 1; } =20 return 0; @@ -134,7 +102,7 @@ static bool no_iotlb_memory; =20 unsigned long swiotlb_nr_tbl(void) { - return unlikely(no_iotlb_memory) ? 0 : io_tlb_nslabs; + return unlikely(no_iotlb_memory) ? 0 : io_tlb_default_mem.nslabs; } EXPORT_SYMBOL_GPL(swiotlb_nr_tbl); =20 @@ -156,13 +124,14 @@ unsigned long swiotlb_size_or_default(void) { unsigned long size; =20 - size =3D io_tlb_nslabs << IO_TLB_SHIFT; + size =3D io_tlb_default_mem.nslabs << IO_TLB_SHIFT; =20 return size ? size : (IO_TLB_DEFAULT_SIZE); } =20 void __init swiotlb_adjust_size(unsigned long new_size) { + struct io_tlb_mem *mem =3D &io_tlb_default_mem; unsigned long size; =20 /* @@ -170,10 +139,10 @@ void __init swiotlb_adjust_size(unsigned long new_siz= e) * architectures such as those supporting memory encryption to * adjust/expand SWIOTLB size for their use. */ - if (!io_tlb_nslabs) { + if (!mem->nslabs) { size =3D ALIGN(new_size, 1 << IO_TLB_SHIFT); - io_tlb_nslabs =3D size >> IO_TLB_SHIFT; - io_tlb_nslabs =3D ALIGN(io_tlb_nslabs, IO_TLB_SEGSIZE); + mem->nslabs =3D size >> IO_TLB_SHIFT; + mem->nslabs =3D ALIGN(mem->nslabs, IO_TLB_SEGSIZE); =20 pr_info("SWIOTLB bounce buffer size adjusted to %luMB", size >> 20); } @@ -181,14 +150,15 @@ void __init swiotlb_adjust_size(unsigned long new_siz= e) =20 void swiotlb_print_info(void) { - unsigned long bytes =3D io_tlb_nslabs << IO_TLB_SHIFT; + struct io_tlb_mem *mem =3D &io_tlb_default_mem; + unsigned long bytes =3D mem->nslabs << IO_TLB_SHIFT; =20 if (no_iotlb_memory) { pr_warn("No low mem\n"); return; } =20 - pr_info("mapped [mem %pa-%pa] (%luMB)\n", &io_tlb_start, &io_tlb_end, + pr_info("mapped [mem %pa-%pa] (%luMB)\n", &mem->start, &mem->end, bytes >> 20); } =20 @@ -200,57 +170,59 @@ void swiotlb_print_info(void) */ void __init swiotlb_update_mem_attributes(void) { + struct io_tlb_mem *mem =3D &io_tlb_default_mem; void *vaddr; unsigned long bytes; =20 if (no_iotlb_memory || late_alloc) return; =20 - vaddr =3D phys_to_virt(io_tlb_start); - bytes =3D PAGE_ALIGN(io_tlb_nslabs << IO_TLB_SHIFT); + vaddr =3D phys_to_virt(mem->start); + bytes =3D PAGE_ALIGN(mem->nslabs << IO_TLB_SHIFT); set_memory_decrypted((unsigned long)vaddr, bytes >> PAGE_SHIFT); memset(vaddr, 0, bytes); } =20 int __init swiotlb_init_with_tbl(char *tlb, unsigned long nslabs, int verb= ose) { + struct io_tlb_mem *mem =3D &io_tlb_default_mem; unsigned long i, bytes; size_t alloc_size; =20 bytes =3D nslabs << IO_TLB_SHIFT; =20 - io_tlb_nslabs =3D nslabs; - io_tlb_start =3D __pa(tlb); - io_tlb_end =3D io_tlb_start + bytes; + mem->nslabs =3D nslabs; + mem->start =3D __pa(tlb); + mem->end =3D mem->start + bytes; =20 /* * Allocate and initialize the free list array. This array is used * to find contiguous free memory regions of size up to IO_TLB_SEGSIZE - * between io_tlb_start and io_tlb_end. + * between mem->start and mem->end. */ - alloc_size =3D PAGE_ALIGN(io_tlb_nslabs * sizeof(int)); - io_tlb_list =3D memblock_alloc(alloc_size, PAGE_SIZE); - if (!io_tlb_list) + alloc_size =3D PAGE_ALIGN(mem->nslabs * sizeof(int)); + mem->list =3D memblock_alloc(alloc_size, PAGE_SIZE); + if (!mem->list) panic("%s: Failed to allocate %zu bytes align=3D0x%lx\n", __func__, alloc_size, PAGE_SIZE); =20 - alloc_size =3D PAGE_ALIGN(io_tlb_nslabs * sizeof(phys_addr_t)); - io_tlb_orig_addr =3D memblock_alloc(alloc_size, PAGE_SIZE); - if (!io_tlb_orig_addr) + alloc_size =3D PAGE_ALIGN(mem->nslabs * sizeof(phys_addr_t)); + mem->orig_addr =3D memblock_alloc(alloc_size, PAGE_SIZE); + if (!mem->orig_addr) panic("%s: Failed to allocate %zu bytes align=3D0x%lx\n", __func__, alloc_size, PAGE_SIZE); =20 - for (i =3D 0; i < io_tlb_nslabs; i++) { - io_tlb_list[i] =3D IO_TLB_SEGSIZE - OFFSET(i, IO_TLB_SEGSIZE); - io_tlb_orig_addr[i] =3D INVALID_PHYS_ADDR; + for (i =3D 0; i < mem->nslabs; i++) { + mem->list[i] =3D IO_TLB_SEGSIZE - OFFSET(i, IO_TLB_SEGSIZE); + mem->orig_addr[i] =3D INVALID_PHYS_ADDR; } - io_tlb_index =3D 0; + mem->index =3D 0; no_iotlb_memory =3D false; =20 if (verbose) swiotlb_print_info(); =20 - swiotlb_set_max_segment(io_tlb_nslabs << IO_TLB_SHIFT); + swiotlb_set_max_segment(mem->nslabs << IO_TLB_SHIFT); return 0; } =20 @@ -261,26 +233,27 @@ int __init swiotlb_init_with_tbl(char *tlb, unsigned = long nslabs, int verbose) void __init swiotlb_init(int verbose) { + struct io_tlb_mem *mem =3D &io_tlb_default_mem; size_t default_size =3D IO_TLB_DEFAULT_SIZE; unsigned char *vstart; unsigned long bytes; =20 - if (!io_tlb_nslabs) { - io_tlb_nslabs =3D (default_size >> IO_TLB_SHIFT); - io_tlb_nslabs =3D ALIGN(io_tlb_nslabs, IO_TLB_SEGSIZE); + if (!mem->nslabs) { + mem->nslabs =3D (default_size >> IO_TLB_SHIFT); + mem->nslabs =3D ALIGN(mem->nslabs, IO_TLB_SEGSIZE); } =20 - bytes =3D io_tlb_nslabs << IO_TLB_SHIFT; + bytes =3D mem->nslabs << IO_TLB_SHIFT; =20 /* Get IO TLB memory from the low pages */ vstart =3D memblock_alloc_low(PAGE_ALIGN(bytes), PAGE_SIZE); - if (vstart && !swiotlb_init_with_tbl(vstart, io_tlb_nslabs, verbose)) + if (vstart && !swiotlb_init_with_tbl(vstart, mem->nslabs, verbose)) return; =20 - if (io_tlb_start) { - memblock_free_early(io_tlb_start, - PAGE_ALIGN(io_tlb_nslabs << IO_TLB_SHIFT)); - io_tlb_start =3D 0; + if (mem->start) { + memblock_free_early(mem->start, + PAGE_ALIGN(mem->nslabs << IO_TLB_SHIFT)); + mem->start =3D 0; } pr_warn("Cannot allocate buffer"); no_iotlb_memory =3D true; @@ -294,22 +267,23 @@ swiotlb_init(int verbose) int swiotlb_late_init_with_default_size(size_t default_size) { - unsigned long bytes, req_nslabs =3D io_tlb_nslabs; + struct io_tlb_mem *mem =3D &io_tlb_default_mem; + unsigned long bytes, req_nslabs =3D mem->nslabs; unsigned char *vstart =3D NULL; unsigned int order; int rc =3D 0; =20 - if (!io_tlb_nslabs) { - io_tlb_nslabs =3D (default_size >> IO_TLB_SHIFT); - io_tlb_nslabs =3D ALIGN(io_tlb_nslabs, IO_TLB_SEGSIZE); + if (!mem->nslabs) { + mem->nslabs =3D (default_size >> IO_TLB_SHIFT); + mem->nslabs =3D ALIGN(mem->nslabs, IO_TLB_SEGSIZE); } =20 /* * Get IO TLB memory from the low pages */ - order =3D get_order(io_tlb_nslabs << IO_TLB_SHIFT); - io_tlb_nslabs =3D SLABS_PER_PAGE << order; - bytes =3D io_tlb_nslabs << IO_TLB_SHIFT; + order =3D get_order(mem->nslabs << IO_TLB_SHIFT); + mem->nslabs =3D SLABS_PER_PAGE << order; + bytes =3D mem->nslabs << IO_TLB_SHIFT; =20 while ((SLABS_PER_PAGE << order) > IO_TLB_MIN_SLABS) { vstart =3D (void *)__get_free_pages(GFP_DMA | __GFP_NOWARN, @@ -320,15 +294,15 @@ swiotlb_late_init_with_default_size(size_t default_si= ze) } =20 if (!vstart) { - io_tlb_nslabs =3D req_nslabs; + mem->nslabs =3D req_nslabs; return -ENOMEM; } if (order !=3D get_order(bytes)) { pr_warn("only able to allocate %ld MB\n", (PAGE_SIZE << order) >> 20); - io_tlb_nslabs =3D SLABS_PER_PAGE << order; + mem->nslabs =3D SLABS_PER_PAGE << order; } - rc =3D swiotlb_late_init_with_tbl(vstart, io_tlb_nslabs); + rc =3D swiotlb_late_init_with_tbl(vstart, mem->nslabs); if (rc) free_pages((unsigned long)vstart, order); =20 @@ -337,22 +311,25 @@ swiotlb_late_init_with_default_size(size_t default_si= ze) =20 static void swiotlb_cleanup(void) { - io_tlb_end =3D 0; - io_tlb_start =3D 0; - io_tlb_nslabs =3D 0; + struct io_tlb_mem *mem =3D &io_tlb_default_mem; + + mem->end =3D 0; + mem->start =3D 0; + mem->nslabs =3D 0; max_segment =3D 0; } =20 int swiotlb_late_init_with_tbl(char *tlb, unsigned long nslabs) { + struct io_tlb_mem *mem =3D &io_tlb_default_mem; unsigned long i, bytes; =20 bytes =3D nslabs << IO_TLB_SHIFT; =20 - io_tlb_nslabs =3D nslabs; - io_tlb_start =3D virt_to_phys(tlb); - io_tlb_end =3D io_tlb_start + bytes; + mem->nslabs =3D nslabs; + mem->start =3D virt_to_phys(tlb); + mem->end =3D mem->start + bytes; =20 set_memory_decrypted((unsigned long)tlb, bytes >> PAGE_SHIFT); memset(tlb, 0, bytes); @@ -360,39 +337,39 @@ swiotlb_late_init_with_tbl(char *tlb, unsigned long n= slabs) /* * Allocate and initialize the free list array. This array is used * to find contiguous free memory regions of size up to IO_TLB_SEGSIZE - * between io_tlb_start and io_tlb_end. + * between mem->start and mem->end. */ - io_tlb_list =3D (unsigned int *)__get_free_pages(GFP_KERNEL, - get_order(io_tlb_nslabs * sizeof(int))); - if (!io_tlb_list) + mem->list =3D (unsigned int *)__get_free_pages(GFP_KERNEL, + get_order(mem->nslabs * sizeof(int))); + if (!mem->list) goto cleanup3; =20 - io_tlb_orig_addr =3D (phys_addr_t *) + mem->orig_addr =3D (phys_addr_t *) __get_free_pages(GFP_KERNEL, - get_order(io_tlb_nslabs * + get_order(mem->nslabs * sizeof(phys_addr_t))); - if (!io_tlb_orig_addr) + if (!mem->orig_addr) goto cleanup4; =20 - for (i =3D 0; i < io_tlb_nslabs; i++) { - io_tlb_list[i] =3D IO_TLB_SEGSIZE - OFFSET(i, IO_TLB_SEGSIZE); - io_tlb_orig_addr[i] =3D INVALID_PHYS_ADDR; + for (i =3D 0; i < mem->nslabs; i++) { + mem->list[i] =3D IO_TLB_SEGSIZE - OFFSET(i, IO_TLB_SEGSIZE); + mem->orig_addr[i] =3D INVALID_PHYS_ADDR; } - io_tlb_index =3D 0; + mem->index =3D 0; no_iotlb_memory =3D false; =20 swiotlb_print_info(); =20 late_alloc =3D 1; =20 - swiotlb_set_max_segment(io_tlb_nslabs << IO_TLB_SHIFT); + swiotlb_set_max_segment(mem->nslabs << IO_TLB_SHIFT); =20 return 0; =20 cleanup4: - free_pages((unsigned long)io_tlb_list, get_order(io_tlb_nslabs * - sizeof(int))); - io_tlb_list =3D NULL; + free_pages((unsigned long)mem->list, + get_order(mem->nslabs * sizeof(int))); + mem->list =3D NULL; cleanup3: swiotlb_cleanup(); return -ENOMEM; @@ -400,23 +377,25 @@ swiotlb_late_init_with_tbl(char *tlb, unsigned long n= slabs) =20 void __init swiotlb_exit(void) { - if (!io_tlb_orig_addr) + struct io_tlb_mem *mem =3D &io_tlb_default_mem; + + if (!mem->orig_addr) return; =20 if (late_alloc) { - free_pages((unsigned long)io_tlb_orig_addr, - get_order(io_tlb_nslabs * sizeof(phys_addr_t))); - free_pages((unsigned long)io_tlb_list, get_order(io_tlb_nslabs * - sizeof(int))); - free_pages((unsigned long)phys_to_virt(io_tlb_start), - get_order(io_tlb_nslabs << IO_TLB_SHIFT)); + free_pages((unsigned long)mem->orig_addr, + get_order(mem->nslabs * sizeof(phys_addr_t))); + free_pages((unsigned long)mem->list, + get_order(mem->nslabs * sizeof(int))); + free_pages((unsigned long)phys_to_virt(mem->start), + get_order(mem->nslabs << IO_TLB_SHIFT)); } else { - memblock_free_late(__pa(io_tlb_orig_addr), - PAGE_ALIGN(io_tlb_nslabs * sizeof(phys_addr_t))); - memblock_free_late(__pa(io_tlb_list), - PAGE_ALIGN(io_tlb_nslabs * sizeof(int))); - memblock_free_late(io_tlb_start, - PAGE_ALIGN(io_tlb_nslabs << IO_TLB_SHIFT)); + memblock_free_late(__pa(mem->orig_addr), + PAGE_ALIGN(mem->nslabs * sizeof(phys_addr_t))); + memblock_free_late(__pa(mem->list), + PAGE_ALIGN(mem->nslabs * sizeof(int))); + memblock_free_late(mem->start, + PAGE_ALIGN(mem->nslabs << IO_TLB_SHIFT)); } swiotlb_cleanup(); } @@ -465,7 +444,8 @@ phys_addr_t swiotlb_tbl_map_single(struct device *hwdev= , phys_addr_t orig_addr, size_t mapping_size, size_t alloc_size, enum dma_data_direction dir, unsigned long attrs) { - dma_addr_t tbl_dma_addr =3D phys_to_dma_unencrypted(hwdev, io_tlb_start); + struct io_tlb_mem *mem =3D &io_tlb_default_mem; + dma_addr_t tbl_dma_addr =3D phys_to_dma_unencrypted(hwdev, mem->start); unsigned long flags; phys_addr_t tlb_addr; unsigned int nslots, stride, index, wrap; @@ -516,13 +496,13 @@ phys_addr_t swiotlb_tbl_map_single(struct device *hwd= ev, phys_addr_t orig_addr, * Find suitable number of IO TLB entries size that will fit this * request and allocate a buffer from that IO TLB pool. */ - spin_lock_irqsave(&io_tlb_lock, flags); + spin_lock_irqsave(&mem->lock, flags); =20 - if (unlikely(nslots > io_tlb_nslabs - io_tlb_used)) + if (unlikely(nslots > mem->nslabs - mem->used)) goto not_found; =20 - index =3D ALIGN(io_tlb_index, stride); - if (index >=3D io_tlb_nslabs) + index =3D ALIGN(mem->index, stride); + if (index >=3D mem->nslabs) index =3D 0; wrap =3D index; =20 @@ -530,7 +510,7 @@ phys_addr_t swiotlb_tbl_map_single(struct device *hwdev= , phys_addr_t orig_addr, while (iommu_is_span_boundary(index, nslots, offset_slots, max_slots)) { index +=3D stride; - if (index >=3D io_tlb_nslabs) + if (index >=3D mem->nslabs) index =3D 0; if (index =3D=3D wrap) goto not_found; @@ -541,40 +521,40 @@ phys_addr_t swiotlb_tbl_map_single(struct device *hwd= ev, phys_addr_t orig_addr, * contiguous buffers, we allocate the buffers from that slot * and mark the entries as '0' indicating unavailable. */ - if (io_tlb_list[index] >=3D nslots) { + if (mem->list[index] >=3D nslots) { int count =3D 0; =20 for (i =3D index; i < (int) (index + nslots); i++) - io_tlb_list[i] =3D 0; - for (i =3D index - 1; (OFFSET(i, IO_TLB_SEGSIZE) !=3D IO_TLB_SEGSIZE - = 1) && io_tlb_list[i]; i--) - io_tlb_list[i] =3D ++count; - tlb_addr =3D io_tlb_start + (index << IO_TLB_SHIFT); + mem->list[i] =3D 0; + for (i =3D index - 1; (OFFSET(i, IO_TLB_SEGSIZE) !=3D IO_TLB_SEGSIZE - = 1) && mem->list[i]; i--) + mem->list[i] =3D ++count; + tlb_addr =3D mem->start + (index << IO_TLB_SHIFT); =20 /* * Update the indices to avoid searching in the next * round. */ - io_tlb_index =3D ((index + nslots) < io_tlb_nslabs - ? (index + nslots) : 0); + mem->index =3D ((index + nslots) < mem->nslabs + ? (index + nslots) : 0); =20 goto found; } index +=3D stride; - if (index >=3D io_tlb_nslabs) + if (index >=3D mem->nslabs) index =3D 0; } while (index !=3D wrap); =20 not_found: - tmp_io_tlb_used =3D io_tlb_used; + tmp_io_tlb_used =3D mem->used; =20 - spin_unlock_irqrestore(&io_tlb_lock, flags); + spin_unlock_irqrestore(&mem->lock, flags); if (!(attrs & DMA_ATTR_NO_WARN) && printk_ratelimit()) dev_warn(hwdev, "swiotlb buffer is full (sz: %zd bytes), total %lu (slot= s), used %lu (slots)\n", - alloc_size, io_tlb_nslabs, tmp_io_tlb_used); + alloc_size, mem->nslabs, tmp_io_tlb_used); return (phys_addr_t)DMA_MAPPING_ERROR; found: - io_tlb_used +=3D nslots; - spin_unlock_irqrestore(&io_tlb_lock, flags); + mem->used +=3D nslots; + spin_unlock_irqrestore(&mem->lock, flags); =20 /* * Save away the mapping from the original address to the DMA address. @@ -582,7 +562,7 @@ phys_addr_t swiotlb_tbl_map_single(struct device *hwdev= , phys_addr_t orig_addr, * needed. */ for (i =3D 0; i < nslots; i++) - io_tlb_orig_addr[index+i] =3D orig_addr + (i << IO_TLB_SHIFT); + mem->orig_addr[index+i] =3D orig_addr + (i << IO_TLB_SHIFT); if (!(attrs & DMA_ATTR_SKIP_CPU_SYNC) && (dir =3D=3D DMA_TO_DEVICE || dir =3D=3D DMA_BIDIRECTIONAL)) swiotlb_bounce(orig_addr, tlb_addr, mapping_size, DMA_TO_DEVICE); @@ -597,10 +577,11 @@ void swiotlb_tbl_unmap_single(struct device *hwdev, p= hys_addr_t tlb_addr, size_t mapping_size, size_t alloc_size, enum dma_data_direction dir, unsigned long attrs) { + struct io_tlb_mem *mem =3D &io_tlb_default_mem; unsigned long flags; int i, count, nslots =3D ALIGN(alloc_size, 1 << IO_TLB_SHIFT) >> IO_TLB_S= HIFT; - int index =3D (tlb_addr - io_tlb_start) >> IO_TLB_SHIFT; - phys_addr_t orig_addr =3D io_tlb_orig_addr[index]; + int index =3D (tlb_addr - mem->start) >> IO_TLB_SHIFT; + phys_addr_t orig_addr =3D mem->orig_addr[index]; =20 /* * First, sync the memory before unmapping the entry @@ -616,36 +597,37 @@ void swiotlb_tbl_unmap_single(struct device *hwdev, p= hys_addr_t tlb_addr, * While returning the entries to the free list, we merge the entries * with slots below and above the pool being returned. */ - spin_lock_irqsave(&io_tlb_lock, flags); + spin_lock_irqsave(&mem->lock, flags); { count =3D ((index + nslots) < ALIGN(index + 1, IO_TLB_SEGSIZE) ? - io_tlb_list[index + nslots] : 0); + mem->list[index + nslots] : 0); /* * Step 1: return the slots to the free list, merging the * slots with superceeding slots */ for (i =3D index + nslots - 1; i >=3D index; i--) { - io_tlb_list[i] =3D ++count; - io_tlb_orig_addr[i] =3D INVALID_PHYS_ADDR; + mem->list[i] =3D ++count; + mem->orig_addr[i] =3D INVALID_PHYS_ADDR; } /* * Step 2: merge the returned slots with the preceding slots, * if available (non zero) */ - for (i =3D index - 1; (OFFSET(i, IO_TLB_SEGSIZE) !=3D IO_TLB_SEGSIZE -1)= && io_tlb_list[i]; i--) - io_tlb_list[i] =3D ++count; + for (i =3D index - 1; (OFFSET(i, IO_TLB_SEGSIZE) !=3D IO_TLB_SEGSIZE -1)= && mem->list[i]; i--) + mem->list[i] =3D ++count; =20 - io_tlb_used -=3D nslots; + mem->used -=3D nslots; } - spin_unlock_irqrestore(&io_tlb_lock, flags); + spin_unlock_irqrestore(&mem->lock, flags); } =20 void swiotlb_tbl_sync_single(struct device *hwdev, phys_addr_t tlb_addr, size_t size, enum dma_data_direction dir, enum dma_sync_target target) { - int index =3D (tlb_addr - io_tlb_start) >> IO_TLB_SHIFT; - phys_addr_t orig_addr =3D io_tlb_orig_addr[index]; + struct io_tlb_mem *mem =3D &io_tlb_default_mem; + int index =3D (tlb_addr - mem->start) >> IO_TLB_SHIFT; + phys_addr_t orig_addr =3D mem->orig_addr[index]; =20 if (orig_addr =3D=3D INVALID_PHYS_ADDR) return; @@ -713,21 +695,21 @@ size_t swiotlb_max_mapping_size(struct device *dev) bool is_swiotlb_active(void) { /* - * When SWIOTLB is initialized, even if io_tlb_start points to physical - * address zero, io_tlb_end surely doesn't. + * When SWIOTLB is initialized, even if mem->start points to physical + * address zero, mem->end surely doesn't. */ - return io_tlb_end !=3D 0; + return io_tlb_default_mem.end !=3D 0; } =20 #ifdef CONFIG_DEBUG_FS =20 static int __init swiotlb_create_debugfs(void) { - struct dentry *root; + struct io_tlb_mem *mem =3D &io_tlb_default_mem; =20 - root =3D debugfs_create_dir("swiotlb", NULL); - debugfs_create_ulong("io_tlb_nslabs", 0400, root, &io_tlb_nslabs); - debugfs_create_ulong("io_tlb_used", 0400, root, &io_tlb_used); + mem->debugfs =3D debugfs_create_dir("swiotlb", NULL); + debugfs_create_ulong("io_tlb_nslabs", 0400, mem->debugfs, &mem->nslabs); + debugfs_create_ulong("io_tlb_used", 0400, mem->debugfs, &mem->used); return 0; } =20 --=20 2.29.2.729.g45daf8777d-goog To make this change as mechanical as possible, I didn't fix any checkpatch.pl ERROR/WARNING.