From nobody Sat Dec 27 20:58:10 2025 Received: from aposti.net (aposti.net [89.234.176.197]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 96DFF2D791; Fri, 15 Dec 2023 13:14:03 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=crapouillou.net Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=crapouillou.net Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=crapouillou.net header.i=@crapouillou.net header.b="Srd2SsiP" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=crapouillou.net; s=mail; t=1702646029; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=21xrIJTonl2JSPX53jr10oHQ64LwYtc2R0VBvZ9/BIs=; b=Srd2SsiPVhfJfsyWju8h+Dm7x5+BPCtFQl+HxHQbQDmI+PE4bBDNFI8XEZ/F00eK+WVL0E 1WRaVjqdmk26n/7H3E5q1RbKKjB/g34lRMRLvxcKJPC2sqozzM8vmaANDeLiVdaMqTPdyM /RQRKKg/IfZopXUG219fOc6KrlKNbXY= From: Paul Cercueil To: Vinod Koul Cc: Lars-Peter Clausen , =?UTF-8?q?Nuno=20S=C3=A1?= , Michael Hennerich , dmaengine@vger.kernel.org, linux-kernel@vger.kernel.org, Paul Cercueil Subject: [PATCH v2 1/5] dmaengine: axi-dmac: Small code cleanup Date: Fri, 15 Dec 2023 14:13:09 +0100 Message-ID: <20231215131313.23840-2-paul@crapouillou.net> In-Reply-To: <20231215131313.23840-1-paul@crapouillou.net> References: <20231215131313.23840-1-paul@crapouillou.net> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Spam: Yes Content-Type: text/plain; charset="utf-8" Use a for() loop instead of a while() loop in axi_dmac_fill_linear_sg(). This makes the code leaner and cleaner overall, and does not introduce any functional change. Signed-off-by: Paul Cercueil --- v2: Improve commit message --- drivers/dma/dma-axi-dmac.c | 5 +---- 1 file changed, 1 insertion(+), 4 deletions(-) diff --git a/drivers/dma/dma-axi-dmac.c b/drivers/dma/dma-axi-dmac.c index 2457a420c13d..760940b21eab 100644 --- a/drivers/dma/dma-axi-dmac.c +++ b/drivers/dma/dma-axi-dmac.c @@ -508,16 +508,13 @@ static struct axi_dmac_sg *axi_dmac_fill_linear_sg(st= ruct axi_dmac_chan *chan, segment_size =3D ((segment_size - 1) | chan->length_align_mask) + 1; =20 for (i =3D 0; i < num_periods; i++) { - len =3D period_len; - - while (len > segment_size) { + for (len =3D period_len; len > segment_size; sg++) { if (direction =3D=3D DMA_DEV_TO_MEM) sg->dest_addr =3D addr; else sg->src_addr =3D addr; sg->x_len =3D segment_size; sg->y_len =3D 1; - sg++; addr +=3D segment_size; len -=3D segment_size; } --=20 2.42.0 From nobody Sat Dec 27 20:58:10 2025 Received: from aposti.net (aposti.net [89.234.176.197]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A43082DB78; Fri, 15 Dec 2023 13:14:11 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=crapouillou.net Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=crapouillou.net Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=crapouillou.net header.i=@crapouillou.net header.b="n73GF01n" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=crapouillou.net; s=mail; t=1702646030; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=dX1OOI1l+mZGv2ZWGYC1ua/+SRhxUb4vYorhXQG2m0s=; b=n73GF01nlvHq339iRA74SaYN1veT8I1ymfb0jyl4RXfsslvWnxW8Sigt6UGYAq0KKmDwt/ l0mmWMMHKRX4jyKHL6DdB9ueF2czOEgP0AbgpUzqhX4I8+vBvKmer9DU0oUwCM/txc2hdU TmL1k3pWMFkaKPOw9u2qHVQN4x3utqk= From: Paul Cercueil To: Vinod Koul Cc: Lars-Peter Clausen , =?UTF-8?q?Nuno=20S=C3=A1?= , Michael Hennerich , dmaengine@vger.kernel.org, linux-kernel@vger.kernel.org, Paul Cercueil Subject: [PATCH v2 2/5] dmaengine: axi-dmac: Allocate hardware descriptors Date: Fri, 15 Dec 2023 14:13:10 +0100 Message-ID: <20231215131313.23840-3-paul@crapouillou.net> In-Reply-To: <20231215131313.23840-1-paul@crapouillou.net> References: <20231215131313.23840-1-paul@crapouillou.net> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Spam: Yes Content-Type: text/plain; charset="utf-8" Change where and how the DMA transfers meta-data is stored, to prepare for the upcoming introduction of scatter-gather support. Allocate hardware descriptors in the format that the HDL core will be expecting them when the scatter-gather feature is enabled, and use these fields to store the data that was previously stored in the axi_dmac_sg structure. Note that the 'x_len' and 'y_len' fields now contain the transfer length minus one, since that's what the hardware will expect in these fields. Signed-off-by: Paul Cercueil --- drivers/dma/dma-axi-dmac.c | 134 ++++++++++++++++++++++++------------- 1 file changed, 88 insertions(+), 46 deletions(-) diff --git a/drivers/dma/dma-axi-dmac.c b/drivers/dma/dma-axi-dmac.c index 760940b21eab..185230a769b9 100644 --- a/drivers/dma/dma-axi-dmac.c +++ b/drivers/dma/dma-axi-dmac.c @@ -97,20 +97,31 @@ /* The maximum ID allocated by the hardware is 31 */ #define AXI_DMAC_SG_UNUSED 32U =20 +struct axi_dmac_hw_desc { + u32 flags; + u32 id; + u64 dest_addr; + u64 src_addr; + u64 __unused; + u32 y_len; + u32 x_len; + u32 src_stride; + u32 dst_stride; + u64 __pad[2]; +}; + struct axi_dmac_sg { - dma_addr_t src_addr; - dma_addr_t dest_addr; - unsigned int x_len; - unsigned int y_len; - unsigned int dest_stride; - unsigned int src_stride; - unsigned int id; unsigned int partial_len; bool schedule_when_free; + + struct axi_dmac_hw_desc *hw; + dma_addr_t hw_phys; }; =20 struct axi_dmac_desc { struct virt_dma_desc vdesc; + struct axi_dmac_chan *chan; + bool cyclic; bool have_partial_xfer; =20 @@ -229,7 +240,7 @@ static void axi_dmac_start_transfer(struct axi_dmac_cha= n *chan) sg =3D &desc->sg[desc->num_submitted]; =20 /* Already queued in cyclic mode. Wait for it to finish */ - if (sg->id !=3D AXI_DMAC_SG_UNUSED) { + if (sg->hw->id !=3D AXI_DMAC_SG_UNUSED) { sg->schedule_when_free =3D true; return; } @@ -246,16 +257,16 @@ static void axi_dmac_start_transfer(struct axi_dmac_c= han *chan) chan->next_desc =3D desc; } =20 - sg->id =3D axi_dmac_read(dmac, AXI_DMAC_REG_TRANSFER_ID); + sg->hw->id =3D axi_dmac_read(dmac, AXI_DMAC_REG_TRANSFER_ID); =20 if (axi_dmac_dest_is_mem(chan)) { - axi_dmac_write(dmac, AXI_DMAC_REG_DEST_ADDRESS, sg->dest_addr); - axi_dmac_write(dmac, AXI_DMAC_REG_DEST_STRIDE, sg->dest_stride); + axi_dmac_write(dmac, AXI_DMAC_REG_DEST_ADDRESS, sg->hw->dest_addr); + axi_dmac_write(dmac, AXI_DMAC_REG_DEST_STRIDE, sg->hw->dst_stride); } =20 if (axi_dmac_src_is_mem(chan)) { - axi_dmac_write(dmac, AXI_DMAC_REG_SRC_ADDRESS, sg->src_addr); - axi_dmac_write(dmac, AXI_DMAC_REG_SRC_STRIDE, sg->src_stride); + axi_dmac_write(dmac, AXI_DMAC_REG_SRC_ADDRESS, sg->hw->src_addr); + axi_dmac_write(dmac, AXI_DMAC_REG_SRC_STRIDE, sg->hw->src_stride); } =20 /* @@ -270,8 +281,8 @@ static void axi_dmac_start_transfer(struct axi_dmac_cha= n *chan) if (chan->hw_partial_xfer) flags |=3D AXI_DMAC_FLAG_PARTIAL_REPORT; =20 - axi_dmac_write(dmac, AXI_DMAC_REG_X_LENGTH, sg->x_len - 1); - axi_dmac_write(dmac, AXI_DMAC_REG_Y_LENGTH, sg->y_len - 1); + axi_dmac_write(dmac, AXI_DMAC_REG_X_LENGTH, sg->hw->x_len); + axi_dmac_write(dmac, AXI_DMAC_REG_Y_LENGTH, sg->hw->y_len); axi_dmac_write(dmac, AXI_DMAC_REG_FLAGS, flags); axi_dmac_write(dmac, AXI_DMAC_REG_START_TRANSFER, 1); } @@ -286,9 +297,9 @@ static inline unsigned int axi_dmac_total_sg_bytes(stru= ct axi_dmac_chan *chan, struct axi_dmac_sg *sg) { if (chan->hw_2d) - return sg->x_len * sg->y_len; + return (sg->hw->x_len + 1) * (sg->hw->y_len + 1); else - return sg->x_len; + return (sg->hw->x_len + 1); } =20 static void axi_dmac_dequeue_partial_xfers(struct axi_dmac_chan *chan) @@ -307,9 +318,9 @@ static void axi_dmac_dequeue_partial_xfers(struct axi_d= mac_chan *chan) list_for_each_entry(desc, &chan->active_descs, vdesc.node) { for (i =3D 0; i < desc->num_sgs; i++) { sg =3D &desc->sg[i]; - if (sg->id =3D=3D AXI_DMAC_SG_UNUSED) + if (sg->hw->id =3D=3D AXI_DMAC_SG_UNUSED) continue; - if (sg->id =3D=3D id) { + if (sg->hw->id =3D=3D id) { desc->have_partial_xfer =3D true; sg->partial_len =3D len; found_sg =3D true; @@ -376,12 +387,12 @@ static bool axi_dmac_transfer_done(struct axi_dmac_ch= an *chan, =20 do { sg =3D &active->sg[active->num_completed]; - if (sg->id =3D=3D AXI_DMAC_SG_UNUSED) /* Not yet submitted */ + if (sg->hw->id =3D=3D AXI_DMAC_SG_UNUSED) /* Not yet submitted */ break; - if (!(BIT(sg->id) & completed_transfers)) + if (!(BIT(sg->hw->id) & completed_transfers)) break; active->num_completed++; - sg->id =3D AXI_DMAC_SG_UNUSED; + sg->hw->id =3D AXI_DMAC_SG_UNUSED; if (sg->schedule_when_free) { sg->schedule_when_free =3D false; start_next =3D true; @@ -476,22 +487,52 @@ static void axi_dmac_issue_pending(struct dma_chan *c) spin_unlock_irqrestore(&chan->vchan.lock, flags); } =20 -static struct axi_dmac_desc *axi_dmac_alloc_desc(unsigned int num_sgs) +static struct axi_dmac_desc * +axi_dmac_alloc_desc(struct axi_dmac_chan *chan, unsigned int num_sgs) { + struct axi_dmac *dmac =3D chan_to_axi_dmac(chan); + struct device *dev =3D dmac->dma_dev.dev; + struct axi_dmac_hw_desc *hws; struct axi_dmac_desc *desc; + dma_addr_t hw_phys; unsigned int i; =20 desc =3D kzalloc(struct_size(desc, sg, num_sgs), GFP_NOWAIT); if (!desc) return NULL; desc->num_sgs =3D num_sgs; + desc->chan =3D chan; =20 - for (i =3D 0; i < num_sgs; i++) - desc->sg[i].id =3D AXI_DMAC_SG_UNUSED; + hws =3D dma_alloc_coherent(dev, PAGE_ALIGN(num_sgs * sizeof(*hws)), + &hw_phys, GFP_ATOMIC); + if (!hws) { + kfree(desc); + return NULL; + } + + for (i =3D 0; i < num_sgs; i++) { + desc->sg[i].hw =3D &hws[i]; + desc->sg[i].hw_phys =3D hw_phys + i * sizeof(*hws); + + hws[i].id =3D AXI_DMAC_SG_UNUSED; + hws[i].flags =3D 0; + } =20 return desc; } =20 +static void axi_dmac_free_desc(struct axi_dmac_desc *desc) +{ + struct axi_dmac *dmac =3D chan_to_axi_dmac(desc->chan); + struct device *dev =3D dmac->dma_dev.dev; + struct axi_dmac_hw_desc *hw =3D desc->sg[0].hw; + dma_addr_t hw_phys =3D desc->sg[0].hw_phys; + + dma_free_coherent(dev, PAGE_ALIGN(desc->num_sgs * sizeof(*hw)), + hw, hw_phys); + kfree(desc); +} + static struct axi_dmac_sg *axi_dmac_fill_linear_sg(struct axi_dmac_chan *c= han, enum dma_transfer_direction direction, dma_addr_t addr, unsigned int num_periods, unsigned int period_len, @@ -510,21 +551,22 @@ static struct axi_dmac_sg *axi_dmac_fill_linear_sg(st= ruct axi_dmac_chan *chan, for (i =3D 0; i < num_periods; i++) { for (len =3D period_len; len > segment_size; sg++) { if (direction =3D=3D DMA_DEV_TO_MEM) - sg->dest_addr =3D addr; + sg->hw->dest_addr =3D addr; else - sg->src_addr =3D addr; - sg->x_len =3D segment_size; - sg->y_len =3D 1; + sg->hw->src_addr =3D addr; + sg->hw->x_len =3D segment_size - 1; + sg->hw->y_len =3D 0; + sg->hw->flags =3D 0; addr +=3D segment_size; len -=3D segment_size; } =20 if (direction =3D=3D DMA_DEV_TO_MEM) - sg->dest_addr =3D addr; + sg->hw->dest_addr =3D addr; else - sg->src_addr =3D addr; - sg->x_len =3D len; - sg->y_len =3D 1; + sg->hw->src_addr =3D addr; + sg->hw->x_len =3D len - 1; + sg->hw->y_len =3D 0; sg++; addr +=3D len; } @@ -551,7 +593,7 @@ static struct dma_async_tx_descriptor *axi_dmac_prep_sl= ave_sg( for_each_sg(sgl, sg, sg_len, i) num_sgs +=3D DIV_ROUND_UP(sg_dma_len(sg), chan->max_length); =20 - desc =3D axi_dmac_alloc_desc(num_sgs); + desc =3D axi_dmac_alloc_desc(chan, num_sgs); if (!desc) return NULL; =20 @@ -560,7 +602,7 @@ static struct dma_async_tx_descriptor *axi_dmac_prep_sl= ave_sg( for_each_sg(sgl, sg, sg_len, i) { if (!axi_dmac_check_addr(chan, sg_dma_address(sg)) || !axi_dmac_check_len(chan, sg_dma_len(sg))) { - kfree(desc); + axi_dmac_free_desc(desc); return NULL; } =20 @@ -595,7 +637,7 @@ static struct dma_async_tx_descriptor *axi_dmac_prep_dm= a_cyclic( num_periods =3D buf_len / period_len; num_segments =3D DIV_ROUND_UP(period_len, chan->max_length); =20 - desc =3D axi_dmac_alloc_desc(num_periods * num_segments); + desc =3D axi_dmac_alloc_desc(chan, num_periods * num_segments); if (!desc) return NULL; =20 @@ -650,26 +692,26 @@ static struct dma_async_tx_descriptor *axi_dmac_prep_= interleaved( return NULL; } =20 - desc =3D axi_dmac_alloc_desc(1); + desc =3D axi_dmac_alloc_desc(chan, 1); if (!desc) return NULL; =20 if (axi_dmac_src_is_mem(chan)) { - desc->sg[0].src_addr =3D xt->src_start; - desc->sg[0].src_stride =3D xt->sgl[0].size + src_icg; + desc->sg[0].hw->src_addr =3D xt->src_start; + desc->sg[0].hw->src_stride =3D xt->sgl[0].size + src_icg; } =20 if (axi_dmac_dest_is_mem(chan)) { - desc->sg[0].dest_addr =3D xt->dst_start; - desc->sg[0].dest_stride =3D xt->sgl[0].size + dst_icg; + desc->sg[0].hw->dest_addr =3D xt->dst_start; + desc->sg[0].hw->dst_stride =3D xt->sgl[0].size + dst_icg; } =20 if (chan->hw_2d) { - desc->sg[0].x_len =3D xt->sgl[0].size; - desc->sg[0].y_len =3D xt->numf; + desc->sg[0].hw->x_len =3D xt->sgl[0].size - 1; + desc->sg[0].hw->y_len =3D xt->numf - 1; } else { - desc->sg[0].x_len =3D xt->sgl[0].size * xt->numf; - desc->sg[0].y_len =3D 1; + desc->sg[0].hw->x_len =3D xt->sgl[0].size * xt->numf - 1; + desc->sg[0].hw->y_len =3D 0; } =20 if (flags & DMA_CYCLIC) @@ -685,7 +727,7 @@ static void axi_dmac_free_chan_resources(struct dma_cha= n *c) =20 static void axi_dmac_desc_free(struct virt_dma_desc *vdesc) { - kfree(container_of(vdesc, struct axi_dmac_desc, vdesc)); + axi_dmac_free_desc(to_axi_dmac_desc(vdesc)); } =20 static bool axi_dmac_regmap_rdwr(struct device *dev, unsigned int reg) --=20 2.42.0 From nobody Sat Dec 27 20:58:10 2025 Received: from aposti.net (aposti.net [89.234.176.197]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 896C62DF8F; Fri, 15 Dec 2023 13:14:25 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=crapouillou.net Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=crapouillou.net Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=crapouillou.net header.i=@crapouillou.net header.b="UIcXbG7C" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=crapouillou.net; s=mail; t=1702646030; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=DfTyNXzfHCBEsP75+j9tuzi+q9BjmgIsVWTMiU/h6PU=; b=UIcXbG7CBMh88X7t8sZevtwrmaHlWv/g0RP3uQ6xH5EHBISmC30BrAsOhMogM5QvY/RQb9 sDnWahVnoDWEl0gC976GnESmk1OOQRn/iBQleCIREPLL1vGp1FCpwEzb4yq+K0It3rjtmn 7c4/ox+xPUj1aoI9v8eALHbOI5fi1bo= From: Paul Cercueil To: Vinod Koul Cc: Lars-Peter Clausen , =?UTF-8?q?Nuno=20S=C3=A1?= , Michael Hennerich , dmaengine@vger.kernel.org, linux-kernel@vger.kernel.org, Paul Cercueil Subject: [PATCH v2 3/5] dmaengine: axi-dmac: Add support for scatter-gather transfers Date: Fri, 15 Dec 2023 14:13:11 +0100 Message-ID: <20231215131313.23840-4-paul@crapouillou.net> In-Reply-To: <20231215131313.23840-1-paul@crapouillou.net> References: <20231215131313.23840-1-paul@crapouillou.net> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Spam: Yes Content-Type: text/plain; charset="utf-8" Implement support for scatter-gather transfers. Build a chain of hardware descriptors, each one corresponding to a segment of the transfer, and linked to the next one. The hardware will transfer the chain and only fire interrupts when the whole chain has been transferred. Support for scatter-gather is automatically enabled when the driver detects that the hardware supports it, by writing then reading the AXI_DMAC_REG_SG_ADDRESS register. If not available, the driver will fall back to standard DMA transfers. Signed-off-by: Paul Cercueil --- drivers/dma/dma-axi-dmac.c | 135 +++++++++++++++++++++++++------------ 1 file changed, 93 insertions(+), 42 deletions(-) diff --git a/drivers/dma/dma-axi-dmac.c b/drivers/dma/dma-axi-dmac.c index 185230a769b9..5109530b66de 100644 --- a/drivers/dma/dma-axi-dmac.c +++ b/drivers/dma/dma-axi-dmac.c @@ -81,9 +81,13 @@ #define AXI_DMAC_REG_CURRENT_DEST_ADDR 0x438 #define AXI_DMAC_REG_PARTIAL_XFER_LEN 0x44c #define AXI_DMAC_REG_PARTIAL_XFER_ID 0x450 +#define AXI_DMAC_REG_CURRENT_SG_ID 0x454 +#define AXI_DMAC_REG_SG_ADDRESS 0x47c +#define AXI_DMAC_REG_SG_ADDRESS_HIGH 0x4bc =20 #define AXI_DMAC_CTRL_ENABLE BIT(0) #define AXI_DMAC_CTRL_PAUSE BIT(1) +#define AXI_DMAC_CTRL_ENABLE_SG BIT(2) =20 #define AXI_DMAC_IRQ_SOT BIT(0) #define AXI_DMAC_IRQ_EOT BIT(1) @@ -97,12 +101,16 @@ /* The maximum ID allocated by the hardware is 31 */ #define AXI_DMAC_SG_UNUSED 32U =20 +/* Flags for axi_dmac_hw_desc.flags */ +#define AXI_DMAC_HW_FLAG_LAST BIT(0) +#define AXI_DMAC_HW_FLAG_IRQ BIT(1) + struct axi_dmac_hw_desc { u32 flags; u32 id; u64 dest_addr; u64 src_addr; - u64 __unused; + u64 next_sg_addr; u32 y_len; u32 x_len; u32 src_stride; @@ -150,6 +158,7 @@ struct axi_dmac_chan { bool hw_partial_xfer; bool hw_cyclic; bool hw_2d; + bool hw_sg; }; =20 struct axi_dmac { @@ -224,9 +233,11 @@ static void axi_dmac_start_transfer(struct axi_dmac_ch= an *chan) unsigned int flags =3D 0; unsigned int val; =20 - val =3D axi_dmac_read(dmac, AXI_DMAC_REG_START_TRANSFER); - if (val) /* Queue is full, wait for the next SOT IRQ */ - return; + if (!chan->hw_sg) { + val =3D axi_dmac_read(dmac, AXI_DMAC_REG_START_TRANSFER); + if (val) /* Queue is full, wait for the next SOT IRQ */ + return; + } =20 desc =3D chan->next_desc; =20 @@ -245,9 +256,10 @@ static void axi_dmac_start_transfer(struct axi_dmac_ch= an *chan) return; } =20 - desc->num_submitted++; - if (desc->num_submitted =3D=3D desc->num_sgs || - desc->have_partial_xfer) { + if (chan->hw_sg) { + chan->next_desc =3D NULL; + } else if (++desc->num_submitted =3D=3D desc->num_sgs || + desc->have_partial_xfer) { if (desc->cyclic) desc->num_submitted =3D 0; /* Start again */ else @@ -259,14 +271,16 @@ static void axi_dmac_start_transfer(struct axi_dmac_c= han *chan) =20 sg->hw->id =3D axi_dmac_read(dmac, AXI_DMAC_REG_TRANSFER_ID); =20 - if (axi_dmac_dest_is_mem(chan)) { - axi_dmac_write(dmac, AXI_DMAC_REG_DEST_ADDRESS, sg->hw->dest_addr); - axi_dmac_write(dmac, AXI_DMAC_REG_DEST_STRIDE, sg->hw->dst_stride); - } + if (!chan->hw_sg) { + if (axi_dmac_dest_is_mem(chan)) { + axi_dmac_write(dmac, AXI_DMAC_REG_DEST_ADDRESS, sg->hw->dest_addr); + axi_dmac_write(dmac, AXI_DMAC_REG_DEST_STRIDE, sg->hw->dst_stride); + } =20 - if (axi_dmac_src_is_mem(chan)) { - axi_dmac_write(dmac, AXI_DMAC_REG_SRC_ADDRESS, sg->hw->src_addr); - axi_dmac_write(dmac, AXI_DMAC_REG_SRC_STRIDE, sg->hw->src_stride); + if (axi_dmac_src_is_mem(chan)) { + axi_dmac_write(dmac, AXI_DMAC_REG_SRC_ADDRESS, sg->hw->src_addr); + axi_dmac_write(dmac, AXI_DMAC_REG_SRC_STRIDE, sg->hw->src_stride); + } } =20 /* @@ -281,8 +295,14 @@ static void axi_dmac_start_transfer(struct axi_dmac_ch= an *chan) if (chan->hw_partial_xfer) flags |=3D AXI_DMAC_FLAG_PARTIAL_REPORT; =20 - axi_dmac_write(dmac, AXI_DMAC_REG_X_LENGTH, sg->hw->x_len); - axi_dmac_write(dmac, AXI_DMAC_REG_Y_LENGTH, sg->hw->y_len); + if (chan->hw_sg) { + axi_dmac_write(dmac, AXI_DMAC_REG_SG_ADDRESS, (u32)sg->hw_phys); + axi_dmac_write(dmac, AXI_DMAC_REG_SG_ADDRESS_HIGH, + (u64)sg->hw_phys >> 32); + } else { + axi_dmac_write(dmac, AXI_DMAC_REG_X_LENGTH, sg->hw->x_len); + axi_dmac_write(dmac, AXI_DMAC_REG_Y_LENGTH, sg->hw->y_len); + } axi_dmac_write(dmac, AXI_DMAC_REG_FLAGS, flags); axi_dmac_write(dmac, AXI_DMAC_REG_START_TRANSFER, 1); } @@ -359,6 +379,9 @@ static void axi_dmac_compute_residue(struct axi_dmac_ch= an *chan, rslt->result =3D DMA_TRANS_NOERROR; rslt->residue =3D 0; =20 + if (chan->hw_sg) + return; + /* * We get here if the last completed segment is partial, which * means we can compute the residue from that segment onwards @@ -385,36 +408,46 @@ static bool axi_dmac_transfer_done(struct axi_dmac_ch= an *chan, (completed_transfers & AXI_DMAC_FLAG_PARTIAL_XFER_DONE)) axi_dmac_dequeue_partial_xfers(chan); =20 - do { - sg =3D &active->sg[active->num_completed]; - if (sg->hw->id =3D=3D AXI_DMAC_SG_UNUSED) /* Not yet submitted */ - break; - if (!(BIT(sg->hw->id) & completed_transfers)) - break; - active->num_completed++; - sg->hw->id =3D AXI_DMAC_SG_UNUSED; - if (sg->schedule_when_free) { - sg->schedule_when_free =3D false; - start_next =3D true; + if (chan->hw_sg) { + if (active->cyclic) { + vchan_cyclic_callback(&active->vdesc); + } else { + list_del(&active->vdesc.node); + vchan_cookie_complete(&active->vdesc); + active =3D axi_dmac_active_desc(chan); } + } else { + do { + sg =3D &active->sg[active->num_completed]; + if (sg->hw->id =3D=3D AXI_DMAC_SG_UNUSED) /* Not yet submitted */ + break; + if (!(BIT(sg->hw->id) & completed_transfers)) + break; + active->num_completed++; + sg->hw->id =3D AXI_DMAC_SG_UNUSED; + if (sg->schedule_when_free) { + sg->schedule_when_free =3D false; + start_next =3D true; + } =20 - if (sg->partial_len) - axi_dmac_compute_residue(chan, active); + if (sg->partial_len) + axi_dmac_compute_residue(chan, active); =20 - if (active->cyclic) - vchan_cyclic_callback(&active->vdesc); + if (active->cyclic) + vchan_cyclic_callback(&active->vdesc); =20 - if (active->num_completed =3D=3D active->num_sgs || - sg->partial_len) { - if (active->cyclic) { - active->num_completed =3D 0; /* wrap around */ - } else { - list_del(&active->vdesc.node); - vchan_cookie_complete(&active->vdesc); - active =3D axi_dmac_active_desc(chan); + if (active->num_completed =3D=3D active->num_sgs || + sg->partial_len) { + if (active->cyclic) { + active->num_completed =3D 0; /* wrap around */ + } else { + list_del(&active->vdesc.node); + vchan_cookie_complete(&active->vdesc); + active =3D axi_dmac_active_desc(chan); + } } - } - } while (active); + } while (active); + } =20 return start_next; } @@ -478,8 +511,12 @@ static void axi_dmac_issue_pending(struct dma_chan *c) struct axi_dmac_chan *chan =3D to_axi_dmac_chan(c); struct axi_dmac *dmac =3D chan_to_axi_dmac(chan); unsigned long flags; + u32 ctrl =3D AXI_DMAC_CTRL_ENABLE; + + if (chan->hw_sg) + ctrl |=3D AXI_DMAC_CTRL_ENABLE_SG; =20 - axi_dmac_write(dmac, AXI_DMAC_REG_CTRL, AXI_DMAC_CTRL_ENABLE); + axi_dmac_write(dmac, AXI_DMAC_REG_CTRL, ctrl); =20 spin_lock_irqsave(&chan->vchan.lock, flags); if (vchan_issue_pending(&chan->vchan)) @@ -516,8 +553,14 @@ axi_dmac_alloc_desc(struct axi_dmac_chan *chan, unsign= ed int num_sgs) =20 hws[i].id =3D AXI_DMAC_SG_UNUSED; hws[i].flags =3D 0; + + /* Link hardware descriptors */ + hws[i].next_sg_addr =3D hw_phys + (i + 1) * sizeof(*hws); } =20 + /* The last hardware descriptor will trigger an interrupt */ + desc->sg[num_sgs - 1].hw->flags =3D AXI_DMAC_HW_FLAG_LAST | AXI_DMAC_HW_F= LAG_IRQ; + return desc; } =20 @@ -753,6 +796,9 @@ static bool axi_dmac_regmap_rdwr(struct device *dev, un= signed int reg) case AXI_DMAC_REG_CURRENT_DEST_ADDR: case AXI_DMAC_REG_PARTIAL_XFER_LEN: case AXI_DMAC_REG_PARTIAL_XFER_ID: + case AXI_DMAC_REG_CURRENT_SG_ID: + case AXI_DMAC_REG_SG_ADDRESS: + case AXI_DMAC_REG_SG_ADDRESS_HIGH: return true; default: return false; @@ -905,6 +951,10 @@ static int axi_dmac_detect_caps(struct axi_dmac *dmac,= unsigned int version) if (axi_dmac_read(dmac, AXI_DMAC_REG_FLAGS) =3D=3D AXI_DMAC_FLAG_CYCLIC) chan->hw_cyclic =3D true; =20 + axi_dmac_write(dmac, AXI_DMAC_REG_SG_ADDRESS, 0xffffffff); + if (axi_dmac_read(dmac, AXI_DMAC_REG_SG_ADDRESS)) + chan->hw_sg =3D true; + axi_dmac_write(dmac, AXI_DMAC_REG_Y_LENGTH, 1); if (axi_dmac_read(dmac, AXI_DMAC_REG_Y_LENGTH) =3D=3D 1) chan->hw_2d =3D true; @@ -1005,6 +1055,7 @@ static int axi_dmac_probe(struct platform_device *pde= v) dma_dev->dst_addr_widths =3D BIT(dmac->chan.dest_width); dma_dev->directions =3D BIT(dmac->chan.direction); dma_dev->residue_granularity =3D DMA_RESIDUE_GRANULARITY_DESCRIPTOR; + dma_dev->max_sg_burst =3D 31; /* 31 SGs maximum in one burst */ INIT_LIST_HEAD(&dma_dev->channels); =20 dmac->chan.vchan.desc_free =3D axi_dmac_desc_free; --=20 2.42.0 From nobody Sat Dec 27 20:58:10 2025 Received: from aposti.net (aposti.net [89.234.176.197]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 06ABE30333; Fri, 15 Dec 2023 13:14:34 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=crapouillou.net Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=crapouillou.net Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=crapouillou.net header.i=@crapouillou.net header.b="3r0VeLvc" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=crapouillou.net; s=mail; t=1702646031; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=LZDpp4w6x3Qe5YwK/dl5k4LHlomX4XHjxnuuTbRgfVA=; b=3r0VeLvc+diNuJCQ549YKzeVipFNByVP6djLsD20mm7iuqlmv/dAO415SlHWgLomrGZejG Wu9jBwmb8Fxxu2gYB1V7gcWIoRPEYObosJdkYoLyZGKdMTX6057PnYXEs8zeEDx7zFR2ik M7KqINO1qBecKD/eGbLBfbc0X+OVMM8= From: Paul Cercueil To: Vinod Koul Cc: Lars-Peter Clausen , =?UTF-8?q?Nuno=20S=C3=A1?= , Michael Hennerich , dmaengine@vger.kernel.org, linux-kernel@vger.kernel.org, Paul Cercueil Subject: [PATCH v2 4/5] dmaengine: axi-dmac: Use only EOT interrupts when doing scatter-gather Date: Fri, 15 Dec 2023 14:13:12 +0100 Message-ID: <20231215131313.23840-5-paul@crapouillou.net> In-Reply-To: <20231215131313.23840-1-paul@crapouillou.net> References: <20231215131313.23840-1-paul@crapouillou.net> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Spam: Yes Content-Type: text/plain; charset="utf-8" Instead of notifying userspace in the end-of-transfer (EOT) interrupt and program the hardware in the start-of-transfer (SOT) interrupt, we can do both things in the EOT, allowing us to mask the SOT, and halve the number of interrupts sent by the HDL core. Signed-off-by: Paul Cercueil --- v2: Make sure cyclic buffers are restarted as well --- drivers/dma/dma-axi-dmac.c | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/drivers/dma/dma-axi-dmac.c b/drivers/dma/dma-axi-dmac.c index 5109530b66de..f63acae511fb 100644 --- a/drivers/dma/dma-axi-dmac.c +++ b/drivers/dma/dma-axi-dmac.c @@ -411,10 +411,12 @@ static bool axi_dmac_transfer_done(struct axi_dmac_ch= an *chan, if (chan->hw_sg) { if (active->cyclic) { vchan_cyclic_callback(&active->vdesc); + start_next =3D true; } else { list_del(&active->vdesc.node); vchan_cookie_complete(&active->vdesc); active =3D axi_dmac_active_desc(chan); + start_next =3D !!active; } } else { do { @@ -1000,6 +1002,7 @@ static int axi_dmac_probe(struct platform_device *pde= v) struct axi_dmac *dmac; struct regmap *regmap; unsigned int version; + u32 irq_mask =3D 0; int ret; =20 dmac =3D devm_kzalloc(&pdev->dev, sizeof(*dmac), GFP_KERNEL); @@ -1067,7 +1070,10 @@ static int axi_dmac_probe(struct platform_device *pd= ev) =20 dma_dev->copy_align =3D (dmac->chan.address_align_mask + 1); =20 - axi_dmac_write(dmac, AXI_DMAC_REG_IRQ_MASK, 0x00); + if (dmac->chan.hw_sg) + irq_mask |=3D AXI_DMAC_IRQ_SOT; + + axi_dmac_write(dmac, AXI_DMAC_REG_IRQ_MASK, irq_mask); =20 if (of_dma_is_coherent(pdev->dev.of_node)) { ret =3D axi_dmac_read(dmac, AXI_DMAC_REG_COHERENCY_DESC); --=20 2.42.0 From nobody Sat Dec 27 20:58:10 2025 Received: from aposti.net (aposti.net [89.234.176.197]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C44693065B; Fri, 15 Dec 2023 13:14:41 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=crapouillou.net Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=crapouillou.net Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=crapouillou.net header.i=@crapouillou.net header.b="DIiMSFMx" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=crapouillou.net; s=mail; t=1702646031; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=kAeMuIy1XyR2uI2Lp4H2je64QseMIBGkzrSyKbKGd4Y=; b=DIiMSFMxTT1aXlDXsy/PFjuLsM+XpoHYg0O5x93fbK0H4ObtJC8KMmoG5cWrdP7CLuV480 S8lKlFIBKgMSWMsyyCDK0whoT0VT03xXOTmqvGWPh3Z1LLgqLy5DfaF0oe8XdNfIl+nw6X aD/W6o2E4+ikDbHVRK0jH8lN3mUCMko= From: Paul Cercueil To: Vinod Koul Cc: Lars-Peter Clausen , =?UTF-8?q?Nuno=20S=C3=A1?= , Michael Hennerich , dmaengine@vger.kernel.org, linux-kernel@vger.kernel.org, Paul Cercueil Subject: [PATCH v2 5/5] dmaengine: axi-dmac: Improve cyclic DMA transfers in SG mode Date: Fri, 15 Dec 2023 14:13:13 +0100 Message-ID: <20231215131313.23840-6-paul@crapouillou.net> In-Reply-To: <20231215131313.23840-1-paul@crapouillou.net> References: <20231215131313.23840-1-paul@crapouillou.net> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Spam: Yes Content-Type: text/plain; charset="utf-8" For cyclic transfers, chain the last descriptor to the first one, and disable IRQ generation if there is no callback registered with the cyclic transfer. Signed-off-by: Paul Cercueil --- v2: New patch --- drivers/dma/dma-axi-dmac.c | 22 ++++++++++++++-------- 1 file changed, 14 insertions(+), 8 deletions(-) diff --git a/drivers/dma/dma-axi-dmac.c b/drivers/dma/dma-axi-dmac.c index f63acae511fb..4e339c04fc1e 100644 --- a/drivers/dma/dma-axi-dmac.c +++ b/drivers/dma/dma-axi-dmac.c @@ -285,12 +285,14 @@ static void axi_dmac_start_transfer(struct axi_dmac_c= han *chan) =20 /* * If the hardware supports cyclic transfers and there is no callback to - * call and only a single segment, enable hw cyclic mode to avoid - * unnecessary interrupts. + * call, enable hw cyclic mode to avoid unnecessary interrupts. */ - if (chan->hw_cyclic && desc->cyclic && !desc->vdesc.tx.callback && - desc->num_sgs =3D=3D 1) - flags |=3D AXI_DMAC_FLAG_CYCLIC; + if (chan->hw_cyclic && desc->cyclic && !desc->vdesc.tx.callback) { + if (chan->hw_sg) + desc->sg[desc->num_sgs - 1].hw->flags &=3D ~AXI_DMAC_HW_FLAG_IRQ; + else if (desc->num_sgs =3D=3D 1) + flags |=3D AXI_DMAC_FLAG_CYCLIC; + } =20 if (chan->hw_partial_xfer) flags |=3D AXI_DMAC_FLAG_PARTIAL_REPORT; @@ -411,7 +413,6 @@ static bool axi_dmac_transfer_done(struct axi_dmac_chan= *chan, if (chan->hw_sg) { if (active->cyclic) { vchan_cyclic_callback(&active->vdesc); - start_next =3D true; } else { list_del(&active->vdesc.node); vchan_cookie_complete(&active->vdesc); @@ -667,7 +668,7 @@ static struct dma_async_tx_descriptor *axi_dmac_prep_dm= a_cyclic( { struct axi_dmac_chan *chan =3D to_axi_dmac_chan(c); struct axi_dmac_desc *desc; - unsigned int num_periods, num_segments; + unsigned int num_periods, num_segments, num_sgs; =20 if (direction !=3D chan->direction) return NULL; @@ -681,11 +682,16 @@ static struct dma_async_tx_descriptor *axi_dmac_prep_= dma_cyclic( =20 num_periods =3D buf_len / period_len; num_segments =3D DIV_ROUND_UP(period_len, chan->max_length); + num_sgs =3D num_periods * num_segments; =20 - desc =3D axi_dmac_alloc_desc(chan, num_periods * num_segments); + desc =3D axi_dmac_alloc_desc(chan, num_sgs); if (!desc) return NULL; =20 + /* Chain the last descriptor to the first, and remove its "last" flag */ + desc->sg[num_sgs - 1].hw->next_sg_addr =3D desc->sg[0].hw_phys; + desc->sg[num_sgs - 1].hw->flags &=3D ~AXI_DMAC_HW_FLAG_LAST; + axi_dmac_fill_linear_sg(chan, direction, buf_addr, num_periods, period_len, desc->sg); =20 --=20 2.42.0