From nobody Sun Feb 8 20:59:44 2026 Received: from frasgout12.his.huawei.com (frasgout12.his.huawei.com [14.137.139.154]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 06358153569 for ; Mon, 25 Mar 2024 08:31:42 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=14.137.139.154 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711355505; cv=none; b=RtOhTGhRUjcg4d8bq3duuxza8cnzN4jSuzmOAm5SGpZC1a8Ulr2G/jcQPSX9O8CkXvs/k/HHvIR2yBJOWpHJkejfeZObumpCvV7bPNNnoitlwYjUnegvxhnGpbu0dk+3zHQb9Ip6yq05K85LffTMol8ki4hW5kXWXe9FcwF0aC0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711355505; c=relaxed/simple; bh=EkQxsNvCXCeshoxUED/Toq5N6LaWQfiIkoXL3HFn6ok=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=J6oQDNGQYUDn6iesa5jH4/JZ30h2+rFipzHtTJwS1M5yzjUnarSIEum14IExJ5A/WAl3ssVrmP90UcyWfDwbVO64xBeh0MwfkXyd8LXY13LXCjGT3RGN0q9G/HLmjlwWPH6gBKzkVNaBB8yEX4/lRsUGMmof/onxiSpDmuoDKnc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=huaweicloud.com; spf=pass smtp.mailfrom=huaweicloud.com; arc=none smtp.client-ip=14.137.139.154 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=huaweicloud.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=huaweicloud.com Received: from mail.maildlp.com (unknown [172.18.186.51]) by frasgout12.his.huawei.com (SkyGuard) with ESMTP id 4V35H93l05z9xHMY for ; Mon, 25 Mar 2024 16:11:17 +0800 (CST) Received: from mail02.huawei.com (unknown [7.182.16.27]) by mail.maildlp.com (Postfix) with ESMTP id 02A0D14059F for ; Mon, 25 Mar 2024 16:31:28 +0800 (CST) Received: from huaweicloud.com (unknown [10.81.223.214]) by APP2 (Coremail) with SMTP id GxC2BwB3sCRPNgFmUPztBA--.12038S3; Mon, 25 Mar 2024 09:31:27 +0100 (CET) From: Petr Tesarik To: Christoph Hellwig , Marek Szyprowski , Robin Murphy , Petr Tesarik , Michael Kelley , Will Deacon , linux-kernel@vger.kernel.org (open list), iommu@lists.linux.dev (open list:DMA MAPPING HELPERS) Cc: Roberto Sassu , Petr Tesarik Subject: [PATCH v4 1/2] swiotlb: extend buffer pre-padding to alloc_align_mask if necessary Date: Mon, 25 Mar 2024 09:31:04 +0100 Message-Id: <20240325083105.658-2-petrtesarik@huaweicloud.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240325083105.658-1-petrtesarik@huaweicloud.com> References: <20240325083105.658-1-petrtesarik@huaweicloud.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-CM-TRANSID: GxC2BwB3sCRPNgFmUPztBA--.12038S3 X-Coremail-Antispam: 1UD129KBjvJXoW3Xw4rZr4xXw1furyDCrW5Awb_yoW3JrW5pF 4fJa1rtFWIqF1xCwsF9a95GF1F9w1kCry7Gr4SgryY9ryDXF98XF98A3yYga4FqrWkuF47 Zas5ur48CF47Jr7anT9S1TB71UUUUU7qnTZGkaVYY2UrUUUUjbIjqfuFe4nvWSU5nxnvy2 9KBjDU0xBIdaVrnRJUUUQCb4IE77IF4wAFF20E14v26ryj6rWUM7CY07I20VC2zVCF04k2 6cxKx2IYs7xG6rWj6s0DM7CIcVAFz4kK6r1j6r18M28IrcIa0xkI8VA2jI8067AKxVWUGw A2048vs2IY020Ec7CjxVAFwI0_Gr0_Xr1l8cAvFVAK0II2c7xJM28CjxkF64kEwVA0rcxS w2x7M28EF7xvwVC0I7IYx2IY67AKxVWUJVWUCwA2z4x0Y4vE2Ix0cI8IcVCY1x0267AKxV WxJVW8Jr1l84ACjcxK6I8E87Iv67AKxVW8JVWxJwA2z4x0Y4vEx4A2jsIEc7CjxVAFwI0_ Gr0_Gr1UM2AIxVAIcxkEcVAq07x20xvEncxIr21l5I8CrVACY4xI64kE6c02F40Ex7xfMc Ij6xIIjxv20xvE14v26r106r15McIj6I8E87Iv67AKxVWUJVW8JwAm72CE4IkC6x0Yz7v_ Jr0_Gr1lF7xvr2IYc2Ij64vIr41lF7I21c0EjII2zVCS5cI20VAGYxC7M4IIrI8v6xkF7I 0E8cxan2IY04v7MxkF7I0En4kS14v26r4a6rW5MxkF7I0Ew4C26cxK6c8Ij28IcwCF04k2 0xvY0x0EwIxGrwCFx2IqxVCFs4IE7xkEbVWUJVW8JwC20s026c02F40E14v26r1j6r18MI 8I3I0E7480Y4vE14v26r106r1rMI8E67AF67kF1VAFwI0_Jw0_GFylIxkGc2Ij64vIr41l IxAIcVC0I7IYx2IY67AKxVWUJVWUCwCI42IY6xIIjxv20xvEc7CjxVAFwI0_Cr0_Gr1UMI IF0xvE42xK8VAvwI8IcIk0rVWUJVWUCwCI42IY6I8E87Iv67AKxVWUJVW8JwCI42IY6I8E 87Iv6xkF7I0E14v26r4j6r4UJbIYCTnIWIevJa73UjIFyTuYvjTRsL0oUUUUU X-CM-SenderInfo: hshw23xhvd2x3n6k3tpzhluzxrxghudrp/ Content-Type: text/plain; charset="utf-8" From: Petr Tesarik Allow a buffer pre-padding of up to alloc_align_mask, even if it requires allocating additional IO TLB slots. If the allocation alignment is bigger than IO_TLB_SIZE and min_align_mask covers any non-zero bits in the original address between IO_TLB_SIZE and alloc_align_mask, these bits are not preserved in the swiotlb buffer address. To fix this case, increase the allocation size and use a larger offset within the allocated buffer. As a result, extra padding slots may be allocated before the mapping start address. Leave orig_addr in these padding slots initialized to INVALID_PHYS_ADDR. These slots do not correspond to any CPU buffer, so attempts to sync the data should be ignored. The padding slots should be automatically released when the buffer is unmapped. However, swiotlb_tbl_unmap_single() takes only the address of the DMA buffer slot, not the first padding slot. Save the number of padding slots in struct io_tlb_slot and use it to adjust the slot index in swiotlb_release_slots(), so all allocated slots are properly freed. Fixes: 2fd4fa5d3fb5 ("swiotlb: Fix alignment checks when both allocation an= d DMA masks are present") Link: https://lore.kernel.org/linux-iommu/20240311210507.217daf8b@meshulam.= tesarici.cz/ Signed-off-by: Petr Tesarik --- kernel/dma/swiotlb.c | 59 ++++++++++++++++++++++++++++++++++---------- 1 file changed, 46 insertions(+), 13 deletions(-) diff --git a/kernel/dma/swiotlb.c b/kernel/dma/swiotlb.c index 86fe172b5958..d7a8cb93ef2d 100644 --- a/kernel/dma/swiotlb.c +++ b/kernel/dma/swiotlb.c @@ -69,11 +69,14 @@ * @alloc_size: Size of the allocated buffer. * @list: The free list describing the number of free entries available * from each index. + * @pad_slots: Number of preceding padding slots. Valid only in the first + * allocated non-padding slot. */ struct io_tlb_slot { phys_addr_t orig_addr; size_t alloc_size; - unsigned int list; + unsigned short list; + unsigned short pad_slots; }; =20 static bool swiotlb_force_bounce; @@ -287,6 +290,7 @@ static void swiotlb_init_io_tlb_pool(struct io_tlb_pool= *mem, phys_addr_t start, mem->nslabs - i); mem->slots[i].orig_addr =3D INVALID_PHYS_ADDR; mem->slots[i].alloc_size =3D 0; + mem->slots[i].pad_slots =3D 0; } =20 memset(vaddr, 0, bytes); @@ -821,12 +825,30 @@ void swiotlb_dev_init(struct device *dev) #endif } =20 -/* - * Return the offset into a iotlb slot required to keep the device happy. +/** + * swiotlb_align_offset() - Get required offset into an IO TLB allocation. + * @dev: Owning device. + * @align_mask: Allocation alignment mask. + * @addr: DMA address. + * + * Return the minimum offset from the start of an IO TLB allocation which = is + * required for a given buffer address and allocation alignment to keep the + * device happy. + * + * First, the address bits covered by min_align_mask must be identical in = the + * original address and the bounce buffer address. High bits are preserved= by + * choosing a suitable IO TLB slot, but bits below IO_TLB_SHIFT require ex= tra + * padding bytes before the bounce buffer. + * + * Second, @align_mask specifies which bits of the first allocated slot mu= st + * be zero. This may require allocating additional padding slots, and then= the + * offset (in bytes) from the first such padding slot is returned. */ -static unsigned int swiotlb_align_offset(struct device *dev, u64 addr) +static unsigned int swiotlb_align_offset(struct device *dev, + unsigned int align_mask, u64 addr) { - return addr & dma_get_min_align_mask(dev) & (IO_TLB_SIZE - 1); + return addr & dma_get_min_align_mask(dev) & + (align_mask | (IO_TLB_SIZE - 1)); } =20 /* @@ -847,7 +869,7 @@ static void swiotlb_bounce(struct device *dev, phys_add= r_t tlb_addr, size_t size return; =20 tlb_offset =3D tlb_addr & (IO_TLB_SIZE - 1); - orig_addr_offset =3D swiotlb_align_offset(dev, orig_addr); + orig_addr_offset =3D swiotlb_align_offset(dev, 0, orig_addr); if (tlb_offset < orig_addr_offset) { dev_WARN_ONCE(dev, 1, "Access before mapping start detected. orig offset %u, requested offset= %u.\n", @@ -1005,7 +1027,7 @@ static int swiotlb_search_pool_area(struct device *de= v, struct io_tlb_pool *pool unsigned long max_slots =3D get_max_slots(boundary_mask); unsigned int iotlb_align_mask =3D dma_get_min_align_mask(dev); unsigned int nslots =3D nr_slots(alloc_size), stride; - unsigned int offset =3D swiotlb_align_offset(dev, orig_addr); + unsigned int offset =3D swiotlb_align_offset(dev, 0, orig_addr); unsigned int index, slots_checked, count =3D 0, i; unsigned long flags; unsigned int slot_base; @@ -1328,11 +1350,12 @@ phys_addr_t swiotlb_tbl_map_single(struct device *d= ev, phys_addr_t orig_addr, unsigned long attrs) { struct io_tlb_mem *mem =3D dev->dma_io_tlb_mem; - unsigned int offset =3D swiotlb_align_offset(dev, orig_addr); + unsigned int offset; struct io_tlb_pool *pool; unsigned int i; int index; phys_addr_t tlb_addr; + unsigned short pad_slots; =20 if (!mem || !mem->nslabs) { dev_warn_ratelimited(dev, @@ -1349,6 +1372,7 @@ phys_addr_t swiotlb_tbl_map_single(struct device *dev= , phys_addr_t orig_addr, return (phys_addr_t)DMA_MAPPING_ERROR; } =20 + offset =3D swiotlb_align_offset(dev, alloc_align_mask, orig_addr); index =3D swiotlb_find_slots(dev, orig_addr, alloc_size + offset, alloc_align_mask, &pool); if (index =3D=3D -1) { @@ -1364,6 +1388,10 @@ phys_addr_t swiotlb_tbl_map_single(struct device *de= v, phys_addr_t orig_addr, * This is needed when we sync the memory. Then we sync the buffer if * needed. */ + pad_slots =3D offset >> IO_TLB_SHIFT; + offset &=3D (IO_TLB_SIZE - 1); + index +=3D pad_slots; + pool->slots[index].pad_slots =3D pad_slots; for (i =3D 0; i < nr_slots(alloc_size + offset); i++) pool->slots[index + i].orig_addr =3D slot_addr(orig_addr, i); tlb_addr =3D slot_addr(pool->start, index) + offset; @@ -1384,13 +1412,17 @@ static void swiotlb_release_slots(struct device *de= v, phys_addr_t tlb_addr) { struct io_tlb_pool *mem =3D swiotlb_find_pool(dev, tlb_addr); unsigned long flags; - unsigned int offset =3D swiotlb_align_offset(dev, tlb_addr); - int index =3D (tlb_addr - offset - mem->start) >> IO_TLB_SHIFT; - int nslots =3D nr_slots(mem->slots[index].alloc_size + offset); - int aindex =3D index / mem->area_nslabs; - struct io_tlb_area *area =3D &mem->areas[aindex]; + unsigned int offset =3D swiotlb_align_offset(dev, 0, tlb_addr); + int index, nslots, aindex; + struct io_tlb_area *area; int count, i; =20 + index =3D (tlb_addr - offset - mem->start) >> IO_TLB_SHIFT; + index -=3D mem->slots[index].pad_slots; + nslots =3D nr_slots(mem->slots[index].alloc_size + offset); + aindex =3D index / mem->area_nslabs; + area =3D &mem->areas[aindex]; + /* * Return the buffer to the free list by setting the corresponding * entries to indicate the number of contiguous entries available. @@ -1413,6 +1445,7 @@ static void swiotlb_release_slots(struct device *dev,= phys_addr_t tlb_addr) mem->slots[i].list =3D ++count; mem->slots[i].orig_addr =3D INVALID_PHYS_ADDR; mem->slots[i].alloc_size =3D 0; + mem->slots[i].pad_slots =3D 0; } =20 /* --=20 2.34.1 From nobody Sun Feb 8 20:59:44 2026 Received: from frasgout13.his.huawei.com (frasgout13.his.huawei.com [14.137.139.46]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 07D4B15356D for ; Mon, 25 Mar 2024 08:31:47 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=14.137.139.46 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711355509; cv=none; b=pE8QDK/x4bzzA2XM23oVUSAe+hwwim9Rxxi+MwgnD1NYFQSoonRWJT6BChf9zm7LaZDDKVO5cDJ2dA1iuiht9Q21l1+q3Imm3M58pue1z8riKrOYwBZIU3ycLGMOMUsKI0mPuwGZhz9HNTUfGwjEyBEmFJnSZLUoqQwT4KvgteY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711355509; c=relaxed/simple; bh=QlaPSqdowMBdrujZOZu5SFX0h6wJ//n1ObBSKHR/fro=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=uxNm/nb4WmVASHjCnak7sKRZ82If4ZisyQRzXM9baomuTDXeGUf1bW0rZa4Wq33Ia+9YmuBXoK11xHRsOhjmnCIT6fnNOmN5x1MIv24lsy4GBGqzaZdQb0J/0iSiwUloLTu1BL8/6OZlsEpbCLZyhMGPs8L+3DOzJuoYkgtrois= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=huaweicloud.com; spf=pass smtp.mailfrom=huaweicloud.com; arc=none smtp.client-ip=14.137.139.46 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=huaweicloud.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=huaweicloud.com Received: from mail.maildlp.com (unknown [172.18.186.29]) by frasgout13.his.huawei.com (SkyGuard) with ESMTP id 4V35NH2yr6z9v7HR for ; Mon, 25 Mar 2024 16:15:43 +0800 (CST) Received: from mail02.huawei.com (unknown [7.182.16.27]) by mail.maildlp.com (Postfix) with ESMTP id 146A7140636 for ; Mon, 25 Mar 2024 16:31:35 +0800 (CST) Received: from huaweicloud.com (unknown [10.81.223.214]) by APP2 (Coremail) with SMTP id GxC2BwB3sCRPNgFmUPztBA--.12038S4; Mon, 25 Mar 2024 09:31:34 +0100 (CET) From: Petr Tesarik To: Christoph Hellwig , Marek Szyprowski , Robin Murphy , Petr Tesarik , Michael Kelley , Will Deacon , linux-kernel@vger.kernel.org (open list), iommu@lists.linux.dev (open list:DMA MAPPING HELPERS) Cc: Roberto Sassu , Petr Tesarik Subject: [PATCH v4 2/2] bug: introduce ASSERT_VAR_CAN_HOLD() Date: Mon, 25 Mar 2024 09:31:05 +0100 Message-Id: <20240325083105.658-3-petrtesarik@huaweicloud.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240325083105.658-1-petrtesarik@huaweicloud.com> References: <20240325083105.658-1-petrtesarik@huaweicloud.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-CM-TRANSID: GxC2BwB3sCRPNgFmUPztBA--.12038S4 X-Coremail-Antispam: 1UD129KBjvJXoW7Kw1DWF1xKw4ktr45GrW8tFb_yoW8WFy7pF 9xArn5KF4jqFyxZF12k3srCF1fK34vk347Cas0gryYvr12qF9aqayDKrW3WFyqqr4vgF4a kw1FgrWYyw1UArDanT9S1TB71UUUUU7qnTZGkaVYY2UrUUUUjbIjqfuFe4nvWSU5nxnvy2 9KBjDU0xBIdaVrnRJUUUQI14x267AKxVWrJVCq3wAFc2x0x2IEx4CE42xK8VAvwI8IcIk0 rVWrJVCq3wAFIxvE14AKwVWUJVWUGwA2048vs2IY020E87I2jVAFwI0_Jryl82xGYIkIc2 x26xkF7I0E14v26ryj6s0DM28lY4IEw2IIxxk0rwA2F7IY1VAKz4vEj48ve4kI8wA2z4x0 Y4vE2Ix0cI8IcVAFwI0_JFI_Gr1l84ACjcxK6xIIjxv20xvEc7CjxVAFwI0_Cr0_Gr1UM2 8EF7xvwVC2z280aVAFwI0_Gr0_Cr1l84ACjcxK6I8E87Iv6xkF7I0E14v26r4j6r4UJwAS 0I0E0xvYzxvE52x082IY62kv0487Mc02F40EFcxC0VAKzVAqx4xG6I80ewAv7VC0I7IYx2 IY67AKxVWUGVWUXwAv7VC2z280aVAFwI0_Jr0_Gr1lOx8S6xCaFVCjc4AY6r1j6r4UM4x0 Y48IcxkI7VAKI48JM4x0x7Aq67IIx4CEVc8vx2IErcIFxwACI402YVCY1x02628vn2kIc2 xKxwCY1x0262kKe7AKxVW8ZVWrXwCY1x0264kExVAvwVAq07x20xyl42xK82IYc2Ij64vI r41l4I8I3I0E4IkC6x0Yz7v_Jr0_Gr1lx2IqxVAqx4xG67AKxVWUJVWUGwC20s026x8Gjc xK67AKxVWUGVWUWwC2zVAF1VAY17CE14v26r1q6r43MIIYrxkI7VAKI48JMIIF0xvE2Ix0 cI8IcVAFwI0_JFI_Gr1lIxAIcVC0I7IYx2IY6xkF7I0E14v26F4j6r4UJwCI42IY6xAIw2 0EY4v20xvaj40_Jr0_JF4lIxAIcVC2z280aVAFwI0_Jr0_Gr1lIxAIcVC2z280aVCY1x02 67AKxVW8JVW8JrUvcSsGvfC2KfnxnUUI43ZEXa7sR_yv3UUUUUU== X-CM-SenderInfo: hshw23xhvd2x3n6k3tpzhluzxrxghudrp/ Content-Type: text/plain; charset="utf-8" From: Petr Tesarik Introduce an ASSERT_VAR_CAN_HOLD() macro to check at build time that a variable can hold the given value. Use this macro in swiotlb to make sure that the list and pad_slots fields of struct io_tlb_slot are big enough to hold the maximum possible value of IO_TLB_SEGSIZE. Signed-off-by: Petr Tesarik --- include/linux/build_bug.h | 10 ++++++++++ kernel/dma/swiotlb.c | 2 ++ 2 files changed, 12 insertions(+) diff --git a/include/linux/build_bug.h b/include/linux/build_bug.h index 3aa3640f8c18..6e2486508af0 100644 --- a/include/linux/build_bug.h +++ b/include/linux/build_bug.h @@ -86,4 +86,14 @@ "Offset of " #field " in " #type " has changed.") =20 =20 +/* + * Compile time check that a variable can hold the given value + */ +#define ASSERT_VAR_CAN_HOLD(var, value) ({ \ + typeof(value) __val =3D (value); \ + typeof(var) __tmp =3D __val; \ + BUILD_BUG_ON_MSG(__tmp !=3D __val, \ + #var " cannot hold " #value "."); \ +}) + #endif /* _LINUX_BUILD_BUG_H */ diff --git a/kernel/dma/swiotlb.c b/kernel/dma/swiotlb.c index d7a8cb93ef2d..a34ea2c35446 100644 --- a/kernel/dma/swiotlb.c +++ b/kernel/dma/swiotlb.c @@ -285,6 +285,8 @@ static void swiotlb_init_io_tlb_pool(struct io_tlb_pool= *mem, phys_addr_t start, mem->areas[i].used =3D 0; } =20 + ASSERT_VAR_CAN_HOLD(mem->slots[0].list, IO_TLB_SEGSIZE); + ASSERT_VAR_CAN_HOLD(mem->slots[0].pad_slots, IO_TLB_SEGSIZE); for (i =3D 0; i < mem->nslabs; i++) { mem->slots[i].list =3D min(IO_TLB_SEGSIZE - io_tlb_offset(i), mem->nslabs - i); --=20 2.34.1