From nobody Sat Jul 25 20:08:11 2026 Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.11]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 401E533260D; Tue, 14 Jul 2026 04:11:10 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=198.175.65.11 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784002272; cv=none; b=B2Ysij2V36CXoFH2YSk/SVwH55IPznoAESF2U/Xe2OK6xqtBnUkxcfnNaW9vJ5eAa9Oz6z/E8pRveZOLw01npChrs1AflOBy6+AEXaGtwfTRSjm2AXtdWFHb+xh6z5WbP24hkaBkkcKHp1rxumlyeVd2kXG38RqBvJsMIQxQQ8c= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784002272; c=relaxed/simple; bh=L3Y/ZhlIUAdgtnzQ8lUOWFMcEcKiOgfI+ea8lVssSmA=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=h//ydWTU/hZhPT3EFMfFkiXLT6467kbcpYJ2Jffs8di4aZCN7jQ6+kmoUUOiLOsD1JprNWkCEsYgaIvCWTWBEu8ziR13s8FgqwibkTYWa6pLtYoxWKGW+fmj7rAXObpXW/qAtS2cnYdxhHJSf9yQy/J73FrN5O9lAPrbp9aVo3k= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com; spf=pass smtp.mailfrom=intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=AaehM/n+; arc=none smtp.client-ip=198.175.65.11 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="AaehM/n+" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1784002271; x=1815538271; h=from:date:subject:mime-version:content-transfer-encoding: message-id:references:in-reply-to:to:cc; bh=L3Y/ZhlIUAdgtnzQ8lUOWFMcEcKiOgfI+ea8lVssSmA=; b=AaehM/n+B2n8hEnathO+HSjTdP/rI0tDVuhR4xnNjzlV8PVgNxAEgE2v X7DoCmcJLoLrxNOkPeZ8SlijZ9jZpjgFKnif0mNwbMF+13mKNqZ4I9Guz gIBRRwQPBfNVIviO2eR2g+TCFhkp2AgewhKUze9ud/+RxSA7CfLMcSmte M4jwOm7MDLwjn5c9ZBVnj+f9NPHq7MmYTDe02H2ukaeQ1BZ3hLpUjfFnV xwyHFnMmAN7g1BW68ULvtT8vj0Qk65KOFBypVopjyNGSS9xSR7ZlkhgAD BwLMjSIpW8VUr6WlEdK60u54aP0BHgfeiXJ2ZlgAW10b7kWVhT4w1A8+Z Q==; X-CSE-ConnectionGUID: miEbIP2xR9epszjUC4zuvQ== X-CSE-MsgGUID: EvGUMusRSJqyfUcSYAHWcg== X-IronPort-AV: E=McAfee;i="6800,10657,11841"; a="94970475" X-IronPort-AV: E=Sophos;i="6.25,154,1779174000"; d="scan'208";a="94970475" Received: from fmviesa009.fm.intel.com ([10.60.135.149]) by orvoesa103.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 13 Jul 2026 21:11:09 -0700 X-CSE-ConnectionGUID: odysYgFMRKOoBMNAGYmqaw== X-CSE-MsgGUID: P+KZyl3DTPCvt9IndM10Sg== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.25,154,1779174000"; d="scan'208";a="249383514" Received: from vcostago-desk1.jf.intel.com (HELO [10.88.27.144]) ([10.88.27.144]) by fmviesa009-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 13 Jul 2026 21:11:08 -0700 From: Vinicius Costa Gomes Date: Mon, 13 Jul 2026 21:10:53 -0700 Subject: [PATCH 1/4] dmaengine: idxd: assign all engines to group 0 in IAA defaults Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20260713-iaa-crypto-fixes-zswap-v1-1-65cac23c684d@intel.com> References: <20260713-iaa-crypto-fixes-zswap-v1-0-65cac23c684d@intel.com> In-Reply-To: <20260713-iaa-crypto-fixes-zswap-v1-0-65cac23c684d@intel.com> To: Dave Jiang , Vinod Koul , Frank Li , Kristen Accardi , Herbert Xu , "David S. Miller" , Andrew Morton , Yosry Ahmed , Nhat Pham Cc: dmaengine@vger.kernel.org, linux-kernel@vger.kernel.org, linux-crypto@vger.kernel.org, Vinicius Costa Gomes , Giovanni Cabiddu X-Mailer: b4 0.16-dev-4217c X-Developer-Signature: v=1; a=ed25519-sha256; t=1784002267; l=1499; i=vinicius.gomes@intel.com; s=20230921; h=from:subject:message-id; bh=xEF5rE7vTRk6IDCo/9ghupdOtafn6T+gs8mY4/r6VAw=; b=N6Mw9Q1NKOEhwmlGvfA96K5AEcH6ElUOu9RTH1fesoeFaQrwVNyXc9ioi1SsD2gRhppk7AHRO SCUVYSb0rH3BZZi8A101C/N3/ZNEhrAOooUX5iujkeHsSVTGY8kqBPA X-Developer-Key: i=vinicius.gomes@intel.com; a=ed25519; pk=aJkrtgqgT6TZ8iIHSG8/rTPsmlYnjMrUjCsMYvCzntk= From: Giovanni Cabiddu The IAA device defaults only assigned engine 0 to group 0, leaving engines 1 through max_engines-1 unassigned (group_id =3D -1). This means that by default only a single engine processed descriptors, limiting throughput to one engine's capacity. Assign all available engines to group 0 so that the full hardware parallelism is used out of the box without requiring manual accel-config setup. Signed-off-by: Giovanni Cabiddu Reviewed-by: Dave Jiang --- drivers/dma/idxd/defaults.c | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-) diff --git a/drivers/dma/idxd/defaults.c b/drivers/dma/idxd/defaults.c index 2bbbcd02a0da..26ebfa2ca144 100644 --- a/drivers/dma/idxd/defaults.c +++ b/drivers/dma/idxd/defaults.c @@ -8,6 +8,7 @@ int idxd_load_iaa_device_defaults(struct idxd_device *idxd) struct idxd_engine *engine; struct idxd_group *group; struct idxd_wq *wq; + int i; =20 if (!test_bit(IDXD_FLAG_CONFIGURABLE, &idxd->flags)) return 0; @@ -41,11 +42,12 @@ int idxd_load_iaa_device_defaults(struct idxd_device *i= dxd) /* set driver_name to "crypto" */ strscpy_pad(wq->driver_name, "crypto"); =20 - engine =3D idxd->engines[0]; - - /* set engine group to 0 */ - engine->group =3D idxd->groups[0]; - engine->group->num_engines++; + /* assign all engines to group 0 */ + for (i =3D 0; i < idxd->max_engines; i++) { + engine =3D idxd->engines[i]; + engine->group =3D group; + group->num_engines++; + } =20 return 0; } --=20 2.55.0 From nobody Sat Jul 25 20:08:11 2026 Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.11]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7627B343D7F; Tue, 14 Jul 2026 04:11:11 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=198.175.65.11 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784002273; cv=none; b=RwSs6nKtFKbpd9pg50dCnIkWoZuxgoe8FPnpz/MO975o2oCXiajV1xH9o/5nf8Br7dpZyzE1MFOsaGhZCbAePLsqe5T7eKNjVbTAQrVPHJ0/Izg3Vawt02eTvheFOkn7R5eTeEIHTk3xsbr3Gj6YTwKQbr036uhRW88404FFOnk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784002273; c=relaxed/simple; bh=09BE9selRtpP3oz5pmJ7CJLIrOIiQa1BymMDmz+BJIQ=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=hmSQ4KDr4iCovwm5zcLtXeQn0wVdXs3Ou/Tv2GTowRqYQCgIkbREPYxkO3sRbI6FusLNUYE/hgCix91TJjEBXchxEcnv2ObSnEA0OefPgWYcN8WAub6/KTg6gJEyAI7QcehR3uZPs54IAvBp2mQw57YZIAr6u1AQfuro3e5VDGM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com; spf=pass smtp.mailfrom=intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=bYhwwRAN; arc=none smtp.client-ip=198.175.65.11 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="bYhwwRAN" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1784002272; x=1815538272; h=from:date:subject:mime-version:content-transfer-encoding: message-id:references:in-reply-to:to:cc; bh=09BE9selRtpP3oz5pmJ7CJLIrOIiQa1BymMDmz+BJIQ=; b=bYhwwRANa9Sn247Yl6OY03HON5+IbxlGfDY/2TvUAPzdTQqwsxfOuoZ5 ZHRSxGgQCo+GuhhnFjNUBuDNdOg+XfVxv3hQ7wkUSdsCta0s5SspAyJNF IprBEaq9nNmswf6kkw7aFHezqOCNF+nUCeL1Kz3LqJcB546kLTMGhvHD2 qoNbfnMBRyCkeCC24pdEgbdeIT7sTM/lj0i599f0oKARBm7AI4iQgPe+P Xx0w1ga3MJU3geug0ulJA2TTXoL8EiYeifeNjn67IYTcSgRwv1zuZHefi yh9b8qDiqXOjsJ1SsU8cZSb4W5/42zX7LUvO49FZEHPqMRhKVMpjjivNj Q==; X-CSE-ConnectionGUID: sv6EqI2EQW651R/ZVtHgRw== X-CSE-MsgGUID: Y2tLdt0yT7OaVgfi7h3RNg== X-IronPort-AV: E=McAfee;i="6800,10657,11841"; a="94970481" X-IronPort-AV: E=Sophos;i="6.25,154,1779174000"; d="scan'208";a="94970481" Received: from fmviesa009.fm.intel.com ([10.60.135.149]) by orvoesa103.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 13 Jul 2026 21:11:09 -0700 X-CSE-ConnectionGUID: gs5wa2SdQD6pHLWQ6uMI5w== X-CSE-MsgGUID: 1NE5wgYGRSyxamZM9wGjUg== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.25,154,1779174000"; d="scan'208";a="249383517" Received: from vcostago-desk1.jf.intel.com (HELO [10.88.27.144]) ([10.88.27.144]) by fmviesa009-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 13 Jul 2026 21:11:08 -0700 From: Vinicius Costa Gomes Date: Mon, 13 Jul 2026 21:10:54 -0700 Subject: [PATCH 2/4] crypto: iaa - fall back to software for multi-entry scatterlists Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20260713-iaa-crypto-fixes-zswap-v1-2-65cac23c684d@intel.com> References: <20260713-iaa-crypto-fixes-zswap-v1-0-65cac23c684d@intel.com> In-Reply-To: <20260713-iaa-crypto-fixes-zswap-v1-0-65cac23c684d@intel.com> To: Dave Jiang , Vinod Koul , Frank Li , Kristen Accardi , Herbert Xu , "David S. Miller" , Andrew Morton , Yosry Ahmed , Nhat Pham Cc: dmaengine@vger.kernel.org, linux-kernel@vger.kernel.org, linux-crypto@vger.kernel.org, Vinicius Costa Gomes , Giovanni Cabiddu X-Mailer: b4 0.16-dev-4217c X-Developer-Signature: v=1; a=ed25519-sha256; t=1784002267; l=11954; i=vinicius.gomes@intel.com; s=20230921; h=from:subject:message-id; bh=JxUaw3ygUdJNqJtaG4NLTuT4daGheeTOoI2pAXEeb2E=; b=LcsvZU+AwCRXY2K2gEuIX/HGF0HO+lu+E1KVT35t+pIJjv7q+HYme8a5yHnKPeTOz82nW46q4 uoG5HW+qQRIA+VMTEf50tW5r5a5X6qlJyynYS+YD6/zHMdua8uKxCse X-Developer-Key: i=vinicius.gomes@intel.com; a=ed25519; pk=aJkrtgqgT6TZ8iIHSG8/rTPsmlYnjMrUjCsMYvCzntk= From: Giovanni Cabiddu IAA cannot process source or destination scatterlists with more than one entry directly. Instead of failing these requests, route them through a separate deflate acomp transform and keep the request alive in software. Since commit e2c3b6b21c77 ("mm: zswap: use SG list decompression APIs from zsmalloc"), zswap passes the raw zsmalloc SG list directly to crypto drivers, so objects spanning multiple pages now reach IAA as multi-entry sources and would otherwise fail decompression. Fallback to the generic DEFLATE implementation for scatterlists with more than one entry. After the multi-entry cases fall back early, simplify the DMA mapping path to a single scatterlist entry and fall back on mapping failure as well. Add counters to track the number of bytes processed by the software implementation on the compression direction. Fixes: e2c3b6b21c77 ("mm: zswap: use SG list decompression APIs from zsmall= oc") Signed-off-by: Giovanni Cabiddu --- drivers/crypto/intel/iaa/iaa_crypto_main.c | 122 ++++++++++++++++--------= ---- drivers/crypto/intel/iaa/iaa_crypto_stats.c | 11 ++- drivers/crypto/intel/iaa/iaa_crypto_stats.h | 2 + 3 files changed, 84 insertions(+), 51 deletions(-) diff --git a/drivers/crypto/intel/iaa/iaa_crypto_main.c b/drivers/crypto/in= tel/iaa/iaa_crypto_main.c index f62b994e18e5..fb154959c2aa 100644 --- a/drivers/crypto/intel/iaa/iaa_crypto_main.c +++ b/drivers/crypto/intel/iaa/iaa_crypto_main.c @@ -2,6 +2,7 @@ /* Copyright(c) 2021 Intel Corporation. All rights rsvd. */ =20 #include +#include #include #include #include @@ -983,17 +984,43 @@ static inline int check_completion(struct device *dev, return ret; } =20 -static int deflate_generic_decompress(struct acomp_req *req) +static int deflate_fallback(struct acomp_req *req, bool compress) { ACOMP_FBREQ_ON_STACK(fbreq, req); int ret; =20 - ret =3D crypto_acomp_decompress(fbreq); + ret =3D compress ? + crypto_acomp_compress(fbreq) : + crypto_acomp_decompress(fbreq); req->dlen =3D fbreq->dlen; =20 + return ret; +} + +static int deflate_generic_decompress(struct acomp_req *req) +{ + int ret; + + ret =3D deflate_fallback(req, false); + if (ret) + return ret; + update_total_sw_decomp_calls(); =20 - return ret; + return 0; +} + +static int deflate_generic_compress(struct acomp_req *req) +{ + int ret; + + ret =3D deflate_fallback(req, true); + if (ret) + return ret; + + update_total_sw_comp_calls(); + + return 0; } =20 static int iaa_remap_for_verify(struct device *dev, struct iaa_wq *iaa_wq, @@ -1472,7 +1499,7 @@ static int iaa_comp_acompress(struct acomp_req *req) struct iaa_compression_ctx *compression_ctx; struct crypto_tfm *tfm =3D req->base.tfm; dma_addr_t src_addr, dst_addr; - int nr_sgs, cpu, ret =3D 0; + int cpu, ret =3D 0; struct iaa_wq *iaa_wq; struct idxd_wq *wq; struct device *dev; @@ -1489,6 +1516,10 @@ static int iaa_comp_acompress(struct acomp_req *req) return -EINVAL; } =20 + /* Fall back to software if src or dst has multiple sg entries */ + if (sg_nents(req->src) > 1 || sg_nents(req->dst) > 1) + return deflate_generic_compress(req); + cpu =3D get_cpu(); wq =3D wq_table_next_wq(cpu); put_cpu(); @@ -1507,30 +1538,25 @@ static int iaa_comp_acompress(struct acomp_req *req) =20 dev =3D &wq->idxd->pdev->dev; =20 - nr_sgs =3D dma_map_sg(dev, req->src, sg_nents(req->src), DMA_TO_DEVICE); - if (nr_sgs <=3D 0 || nr_sgs > 1) { - dev_dbg(dev, "couldn't map src sg for iaa device %d," - " wq %d: ret=3D%d\n", iaa_wq->iaa_device->idxd->id, - iaa_wq->wq->id, ret); - ret =3D -EIO; - goto out; + if (!dma_map_sg(dev, req->src, 1, DMA_TO_DEVICE)) { + dev_dbg(dev, "couldn't map src sg for iaa device %d, wq %d\n", + iaa_wq->iaa_device->idxd->id, iaa_wq->wq->id); + iaa_wq_put(wq); + return deflate_generic_compress(req); } src_addr =3D sg_dma_address(req->src); - dev_dbg(dev, "dma_map_sg, src_addr %llx, nr_sgs %d, req->src %p," - " req->slen %d, sg_dma_len(sg) %d\n", src_addr, nr_sgs, + dev_dbg(dev, "map src %llx req->src %p slen %d sg_len %d\n", src_addr, req->src, req->slen, sg_dma_len(req->src)); =20 - nr_sgs =3D dma_map_sg(dev, req->dst, sg_nents(req->dst), DMA_FROM_DEVICE); - if (nr_sgs <=3D 0 || nr_sgs > 1) { - dev_dbg(dev, "couldn't map dst sg for iaa device %d," - " wq %d: ret=3D%d\n", iaa_wq->iaa_device->idxd->id, - iaa_wq->wq->id, ret); - ret =3D -EIO; - goto err_map_dst; + if (!dma_map_sg(dev, req->dst, 1, DMA_FROM_DEVICE)) { + dev_dbg(dev, "couldn't map dst sg for iaa device %d, wq %d\n", + iaa_wq->iaa_device->idxd->id, iaa_wq->wq->id); + dma_unmap_sg(dev, req->src, 1, DMA_TO_DEVICE); + iaa_wq_put(wq); + return deflate_generic_compress(req); } dst_addr =3D sg_dma_address(req->dst); - dev_dbg(dev, "dma_map_sg, dst_addr %llx, nr_sgs %d, req->dst %p," - " req->dlen %d, sg_dma_len(sg) %d\n", dst_addr, nr_sgs, + dev_dbg(dev, "map dst %llx req->dst %p dlen %d sg_len %d\n", dst_addr, req->dst, req->dlen, sg_dma_len(req->dst)); =20 ret =3D iaa_compress(tfm, req, wq, src_addr, req->slen, dst_addr, @@ -1550,8 +1576,8 @@ static int iaa_comp_acompress(struct acomp_req *req) if (ret) dev_dbg(dev, "asynchronous compress verification failed ret=3D%d\n", re= t); =20 - dma_unmap_sg(dev, req->dst, sg_nents(req->dst), DMA_TO_DEVICE); - dma_unmap_sg(dev, req->src, sg_nents(req->src), DMA_FROM_DEVICE); + dma_unmap_sg(dev, req->dst, 1, DMA_TO_DEVICE); + dma_unmap_sg(dev, req->src, 1, DMA_FROM_DEVICE); =20 goto out; } @@ -1559,9 +1585,8 @@ static int iaa_comp_acompress(struct acomp_req *req) if (ret) dev_dbg(dev, "asynchronous compress failed ret=3D%d\n", ret); =20 - dma_unmap_sg(dev, req->dst, sg_nents(req->dst), DMA_FROM_DEVICE); -err_map_dst: - dma_unmap_sg(dev, req->src, sg_nents(req->src), DMA_TO_DEVICE); + dma_unmap_sg(dev, req->dst, 1, DMA_FROM_DEVICE); + dma_unmap_sg(dev, req->src, 1, DMA_TO_DEVICE); out: iaa_wq_put(wq); =20 @@ -1572,7 +1597,7 @@ static int iaa_comp_adecompress(struct acomp_req *req) { struct crypto_tfm *tfm =3D req->base.tfm; dma_addr_t src_addr, dst_addr; - int nr_sgs, cpu, ret =3D 0; + int cpu, ret =3D 0; struct iaa_wq *iaa_wq; struct device *dev; struct idxd_wq *wq; @@ -1587,6 +1612,10 @@ static int iaa_comp_adecompress(struct acomp_req *re= q) return -EINVAL; } =20 + /* Fall back to software if src or dst has multiple sg entries */ + if (sg_nents(req->src) > 1 || sg_nents(req->dst) > 1) + return deflate_generic_decompress(req); + cpu =3D get_cpu(); wq =3D wq_table_next_wq(cpu); put_cpu(); @@ -1605,30 +1634,25 @@ static int iaa_comp_adecompress(struct acomp_req *r= eq) =20 dev =3D &wq->idxd->pdev->dev; =20 - nr_sgs =3D dma_map_sg(dev, req->src, sg_nents(req->src), DMA_TO_DEVICE); - if (nr_sgs <=3D 0 || nr_sgs > 1) { - dev_dbg(dev, "couldn't map src sg for iaa device %d," - " wq %d: ret=3D%d\n", iaa_wq->iaa_device->idxd->id, - iaa_wq->wq->id, ret); - ret =3D -EIO; - goto out; + if (!dma_map_sg(dev, req->src, 1, DMA_TO_DEVICE)) { + dev_dbg(dev, "couldn't map src sg for iaa device %d, wq %d\n", + iaa_wq->iaa_device->idxd->id, iaa_wq->wq->id); + iaa_wq_put(wq); + return deflate_generic_decompress(req); } src_addr =3D sg_dma_address(req->src); - dev_dbg(dev, "dma_map_sg, src_addr %llx, nr_sgs %d, req->src %p," - " req->slen %d, sg_dma_len(sg) %d\n", src_addr, nr_sgs, + dev_dbg(dev, "map src %llx req->src %p slen %d sg_len %d\n", src_addr, req->src, req->slen, sg_dma_len(req->src)); =20 - nr_sgs =3D dma_map_sg(dev, req->dst, sg_nents(req->dst), DMA_FROM_DEVICE); - if (nr_sgs <=3D 0 || nr_sgs > 1) { - dev_dbg(dev, "couldn't map dst sg for iaa device %d," - " wq %d: ret=3D%d\n", iaa_wq->iaa_device->idxd->id, - iaa_wq->wq->id, ret); - ret =3D -EIO; - goto err_map_dst; + if (!dma_map_sg(dev, req->dst, 1, DMA_FROM_DEVICE)) { + dev_dbg(dev, "couldn't map dst sg for iaa device %d, wq %d\n", + iaa_wq->iaa_device->idxd->id, iaa_wq->wq->id); + dma_unmap_sg(dev, req->src, 1, DMA_TO_DEVICE); + iaa_wq_put(wq); + return deflate_generic_decompress(req); } dst_addr =3D sg_dma_address(req->dst); - dev_dbg(dev, "dma_map_sg, dst_addr %llx, nr_sgs %d, req->dst %p," - " req->dlen %d, sg_dma_len(sg) %d\n", dst_addr, nr_sgs, + dev_dbg(dev, "map dst %llx req->dst %p dlen %d sg_len %d\n", dst_addr, req->dst, req->dlen, sg_dma_len(req->dst)); =20 ret =3D iaa_decompress(tfm, req, wq, src_addr, req->slen, @@ -1639,10 +1663,8 @@ static int iaa_comp_adecompress(struct acomp_req *re= q) if (ret !=3D 0) dev_dbg(dev, "asynchronous decompress failed ret=3D%d\n", ret); =20 - dma_unmap_sg(dev, req->dst, sg_nents(req->dst), DMA_FROM_DEVICE); -err_map_dst: - dma_unmap_sg(dev, req->src, sg_nents(req->src), DMA_TO_DEVICE); -out: + dma_unmap_sg(dev, req->dst, 1, DMA_FROM_DEVICE); + dma_unmap_sg(dev, req->src, 1, DMA_TO_DEVICE); iaa_wq_put(wq); =20 return ret; diff --git a/drivers/crypto/intel/iaa/iaa_crypto_stats.c b/drivers/crypto/i= ntel/iaa/iaa_crypto_stats.c index f5cc3d29ca19..55d0d9c6f61a 100644 --- a/drivers/crypto/intel/iaa/iaa_crypto_stats.c +++ b/drivers/crypto/intel/iaa/iaa_crypto_stats.c @@ -19,6 +19,7 @@ =20 static atomic64_t total_comp_calls; static atomic64_t total_decomp_calls; +static atomic64_t total_sw_comp_calls; static atomic64_t total_sw_decomp_calls; static atomic64_t total_comp_bytes_out; static atomic64_t total_decomp_bytes_in; @@ -43,6 +44,11 @@ void update_total_decomp_calls(void) atomic64_inc(&total_decomp_calls); } =20 +void update_total_sw_comp_calls(void) +{ + atomic64_inc(&total_sw_comp_calls); +} + void update_total_sw_decomp_calls(void) { atomic64_inc(&total_sw_decomp_calls); @@ -104,6 +110,7 @@ static void reset_iaa_crypto_stats(void) { atomic64_set(&total_comp_calls, 0); atomic64_set(&total_decomp_calls, 0); + atomic64_set(&total_sw_comp_calls, 0); atomic64_set(&total_sw_decomp_calls, 0); atomic64_set(&total_comp_bytes_out, 0); atomic64_set(&total_decomp_bytes_in, 0); @@ -174,6 +181,8 @@ static int global_stats_show(struct seq_file *m, void *= v) atomic64_read(&total_comp_calls)); seq_printf(m, " total_decomp_calls: %llu\n", atomic64_read(&total_decomp_calls)); + seq_printf(m, " total_sw_comp_calls: %llu\n", + atomic64_read(&total_sw_comp_calls)); seq_printf(m, " total_sw_decomp_calls: %llu\n", atomic64_read(&total_sw_decomp_calls)); seq_printf(m, " total_comp_bytes_out: %llu\n", @@ -263,7 +272,7 @@ int __init iaa_crypto_debugfs_init(void) return 0; } =20 -void __exit iaa_crypto_debugfs_cleanup(void) +void iaa_crypto_debugfs_cleanup(void) { debugfs_remove_recursive(iaa_crypto_debugfs_root); } diff --git a/drivers/crypto/intel/iaa/iaa_crypto_stats.h b/drivers/crypto/i= ntel/iaa/iaa_crypto_stats.h index 3787a5f507eb..6e0c6f9939bf 100644 --- a/drivers/crypto/intel/iaa/iaa_crypto_stats.h +++ b/drivers/crypto/intel/iaa/iaa_crypto_stats.h @@ -11,6 +11,7 @@ void iaa_crypto_debugfs_cleanup(void); void update_total_comp_calls(void); void update_total_comp_bytes_out(int n); void update_total_decomp_calls(void); +void update_total_sw_comp_calls(void); void update_total_sw_decomp_calls(void); void update_total_decomp_bytes_in(int n); void update_completion_einval_errs(void); @@ -29,6 +30,7 @@ static inline void iaa_crypto_debugfs_cleanup(void) {} static inline void update_total_comp_calls(void) {} static inline void update_total_comp_bytes_out(int n) {} static inline void update_total_decomp_calls(void) {} +static inline void update_total_sw_comp_calls(void) {} static inline void update_total_sw_decomp_calls(void) {} static inline void update_total_decomp_bytes_in(int n) {} static inline void update_completion_einval_errs(void) {} --=20 2.55.0 From nobody Sat Jul 25 20:08:11 2026 Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.11]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B9A4E36AB77; Tue, 14 Jul 2026 04:11:11 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=198.175.65.11 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784002273; cv=none; b=Q5VeEBNrkhDLZzdEHwQohrlTKBER4mF0o03d2z3nq1l7t//TMv5I9RapklxvD2CCkA1RjvzHyXPuYBwRkKue4THoEKSwcmCuDuNgFEztoe9Ot+Gj2b9rLoz1jwzHHpe926Mj3UXaSTHAPthGi9LM6cssC2oLadt6wWHXI5fblsk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784002273; c=relaxed/simple; bh=DimQniesOwePrAQW8cknWA9ZqhNGU3Pe01x+pNQhS2w=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=r/mxyEsyUg7Zl5dCrzH0HIXZc2h59v+1BZFbyj9GTYQFuTmBux5v04+VVeXecVyA4lJXAGNQhoia3CAHm6UJzBZWMPZecd4sGpOj6jcWu3GscnQdC/maesz+0tea3xBFA4onduciOuFPdxfyjYxsLQw6A9yUFhMW0GDaa82PU00= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com; spf=pass smtp.mailfrom=intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=CSmEQYGz; arc=none smtp.client-ip=198.175.65.11 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="CSmEQYGz" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1784002272; x=1815538272; h=from:date:subject:mime-version:content-transfer-encoding: message-id:references:in-reply-to:to:cc; bh=DimQniesOwePrAQW8cknWA9ZqhNGU3Pe01x+pNQhS2w=; b=CSmEQYGzIq8IHCPxhqzhl6IutPivQMinfoTw2H9DeN/HdZN6aZVXq0xR ixjP3lognvdl5MyvUCdtT/Sf0CwiOUARqHtNJhVR0DZVb8Sy771mQcSTV sz2Dk02MBhhdEidlLDgrZJEJbI917NKeEv60D9yy0yA7VRipOuT7c1aqS YmBKnMJPWnE9arEUw80LOhx1eZhMEu/AJNQAf1F1ateV2m1EOsIdC800t gsFcxTjrwupvBOHmbDZ4icTjE/ck7ED7+vE107lV5mulK0HkLeRXvq4Km heFWYaisVBeymDVHnymW7HKNe4aplf9uuaXzcG1Mf7CXs3gKpy6UpSE4A A==; X-CSE-ConnectionGUID: VICxpeFRSM2UOKjnxTni7w== X-CSE-MsgGUID: dthsovrtSoeIqIJpmwWd0w== X-IronPort-AV: E=McAfee;i="6800,10657,11841"; a="94970487" X-IronPort-AV: E=Sophos;i="6.25,154,1779174000"; d="scan'208";a="94970487" Received: from fmviesa009.fm.intel.com ([10.60.135.149]) by orvoesa103.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 13 Jul 2026 21:11:10 -0700 X-CSE-ConnectionGUID: JYn3O0UXR0KxeqAW3xbqaA== X-CSE-MsgGUID: df/vGAT3S42mrFH0iBy5TA== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.25,154,1779174000"; d="scan'208";a="249383520" Received: from vcostago-desk1.jf.intel.com (HELO [10.88.27.144]) ([10.88.27.144]) by fmviesa009-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 13 Jul 2026 21:11:09 -0700 From: Vinicius Costa Gomes Date: Mon, 13 Jul 2026 21:10:55 -0700 Subject: [PATCH 3/4] crypto: iaa - avoid counting fallback decompression bytes Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20260713-iaa-crypto-fixes-zswap-v1-3-65cac23c684d@intel.com> References: <20260713-iaa-crypto-fixes-zswap-v1-0-65cac23c684d@intel.com> In-Reply-To: <20260713-iaa-crypto-fixes-zswap-v1-0-65cac23c684d@intel.com> To: Dave Jiang , Vinod Koul , Frank Li , Kristen Accardi , Herbert Xu , "David S. Miller" , Andrew Morton , Yosry Ahmed , Nhat Pham Cc: dmaengine@vger.kernel.org, linux-kernel@vger.kernel.org, linux-crypto@vger.kernel.org, Vinicius Costa Gomes , Giovanni Cabiddu X-Mailer: b4 0.16-dev-4217c X-Developer-Signature: v=1; a=ed25519-sha256; t=1784002267; l=1906; i=vinicius.gomes@intel.com; s=20230921; h=from:subject:message-id; bh=Bzzsxw3x1+Awdq1YVut9OLn/yRKJ1WIjPIJmOnYqEXs=; b=jcqR1EHjhR2o3aNkqtGmNPbjADHFM/OXYYkPRUBbe8ylwoaFOvv+3NpmqKMgiTsJ+2RWgFNSp yi3C0wYFidkCEWTrr+IEC+9EyxeqBdCYuyt1hXv9EpwvWYijnmRXq5/ X-Developer-Key: i=vinicius.gomes@intel.com; a=ed25519; pk=aJkrtgqgT6TZ8iIHSG8/rTPsmlYnjMrUjCsMYvCzntk= From: Giovanni Cabiddu When decompression falls back to deflate-generic after an analytics error, the request no longer completes through IAA. Move decompression byte accounting into the successful IAA completion path in both the synchronous and asynchronous flows so decomp_bytes only reflects bytes actually processed by IAA. Signed-off-by: Giovanni Cabiddu Reviewed-by: Dave Jiang --- drivers/crypto/intel/iaa/iaa_crypto_main.c | 16 +++++++++------- 1 file changed, 9 insertions(+), 7 deletions(-) diff --git a/drivers/crypto/intel/iaa/iaa_crypto_main.c b/drivers/crypto/in= tel/iaa/iaa_crypto_main.c index fb154959c2aa..8f68b1478476 100644 --- a/drivers/crypto/intel/iaa/iaa_crypto_main.c +++ b/drivers/crypto/intel/iaa/iaa_crypto_main.c @@ -1084,15 +1084,17 @@ static void iaa_desc_complete(struct idxd_desc *idx= d_desc, } } else { ctx->req->dlen =3D idxd_desc->iax_completion->output_size; + + if (!ctx->compress) { + update_total_decomp_bytes_in(ctx->req->slen); + update_wq_decomp_bytes(iaa_wq->wq, ctx->req->slen); + } } =20 /* Update stats */ if (ctx->compress) { update_total_comp_bytes_out(ctx->req->dlen); update_wq_comp_bytes(iaa_wq->wq, ctx->req->dlen); - } else { - update_total_decomp_bytes_in(ctx->req->slen); - update_wq_decomp_bytes(iaa_wq->wq, ctx->req->slen); } =20 if (ctx->compress && compression_ctx->verify_compress) { @@ -1475,16 +1477,16 @@ static int iaa_decompress(struct crypto_tfm *tfm, s= truct acomp_req *req, } } else { req->dlen =3D idxd_desc->iax_completion->output_size; + + /* Update stats */ + update_total_decomp_bytes_in(slen); + update_wq_decomp_bytes(wq, slen); } =20 *dlen =3D req->dlen; =20 if (!ctx->async_mode) idxd_free_desc(wq, idxd_desc); - - /* Update stats */ - update_total_decomp_bytes_in(slen); - update_wq_decomp_bytes(wq, slen); out: return ret; err: --=20 2.55.0 From nobody Sat Jul 25 20:08:11 2026 Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.11]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4DCD03B0AEF; Tue, 14 Jul 2026 04:11:13 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=198.175.65.11 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784002284; cv=none; b=Xn1/md4VXWicWMtKqn+nWKuKbJYcgJ5op+GiF8ZF9VxDDeHNy05UpmeO93U+x2Er35j1vQLfRWEPh22odHZ2WWZdmjsf0Q27RTsfiPCEqL9oyXuIUYGmL/mAhq2qhIV1rxvsrCKSMPtBpvXFM5nQ51vhtHxiEUSC7G2N77JsxYM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784002284; c=relaxed/simple; bh=dDkV8rRh4v6dOr4vEdgVLR9a6liTDDRDpJ0Z/lOagTA=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=SwYaF0/DaTzFupaUTCLhl+Xk1kMFd1DwJc2kWgCGIhnpawDV/cE9NmtUB4FYO4GBCJVTGIB/K/ztaj4zs3j2ecEXH3X3hWddf0hd7ZpfspzNic3oe3uisA3Vl1DYohyL3mF7Z4yrkEkIsKfRPZJTMaL5gswu/wbdnXpXll7VJ3k= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com; spf=pass smtp.mailfrom=intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=CSHuPKr+; arc=none smtp.client-ip=198.175.65.11 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="CSHuPKr+" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1784002274; x=1815538274; h=from:date:subject:mime-version:content-transfer-encoding: message-id:references:in-reply-to:to:cc; bh=dDkV8rRh4v6dOr4vEdgVLR9a6liTDDRDpJ0Z/lOagTA=; b=CSHuPKr+ImYRNzce3h5gC0Eu0H/YHbPJ1YprOm3WR5Lf/GMryBGj65TV +scftWhr2Q5VrTnmHnW+WLkAnTcacMJ4kgCnM3ILBX5i4zu+hnnuDI67i 4mORDAYSfVHRFIE+HHm8YmiM/T33k6wNTTZf/IDm8Y2+LVCyg4nxbJad7 RbYYJEyI4wPCoHTrSbL6bh8K39/o9OTe3VH8z2TLWAUVSYpUlMQnZ8Rih S81sY80yIEotgaubYZTjfNYcd+eOsRfRf3EpPwhMARITFip/MYspandA7 9h4DLIxnuMYhmwgMxrljXHruTa+2yQHiF5VTH4TGFFtWeRiP93FxlpeTU A==; X-CSE-ConnectionGUID: ix6heqfAT/e+K7ovqoVu7w== X-CSE-MsgGUID: X+fbh165TyGe2GNtv2+eTQ== X-IronPort-AV: E=McAfee;i="6800,10657,11841"; a="94970496" X-IronPort-AV: E=Sophos;i="6.25,154,1779174000"; d="scan'208";a="94970496" Received: from fmviesa009.fm.intel.com ([10.60.135.149]) by orvoesa103.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 13 Jul 2026 21:11:10 -0700 X-CSE-ConnectionGUID: fYLFWutNRISUtZmR/uFicw== X-CSE-MsgGUID: ODymvmhySKyOv3vMci0N2A== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.25,154,1779174000"; d="scan'208";a="249383524" Received: from vcostago-desk1.jf.intel.com (HELO [10.88.27.144]) ([10.88.27.144]) by fmviesa009-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 13 Jul 2026 21:11:09 -0700 From: Vinicius Costa Gomes Date: Mon, 13 Jul 2026 21:10:56 -0700 Subject: [PATCH 4/4] crypto: iaa - use bounce buffer for multi-sg decompress input Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20260713-iaa-crypto-fixes-zswap-v1-4-65cac23c684d@intel.com> References: <20260713-iaa-crypto-fixes-zswap-v1-0-65cac23c684d@intel.com> In-Reply-To: <20260713-iaa-crypto-fixes-zswap-v1-0-65cac23c684d@intel.com> To: Dave Jiang , Vinod Koul , Frank Li , Kristen Accardi , Herbert Xu , "David S. Miller" , Andrew Morton , Yosry Ahmed , Nhat Pham Cc: dmaengine@vger.kernel.org, linux-kernel@vger.kernel.org, linux-crypto@vger.kernel.org, Vinicius Costa Gomes , Giovanni Cabiddu X-Mailer: b4 0.16-dev-4217c X-Developer-Signature: v=1; a=ed25519-sha256; t=1784002267; l=10830; i=vinicius.gomes@intel.com; s=20230921; h=from:subject:message-id; bh=p1gGEqzD5ux4EjagTqMYvoTxA8Qq3vs/vbF7VLcJ3v4=; b=JWQEMEY+i1NfiJymTtZdvm9RYje9J++66fiOppgbrDc/BS81M30YUNJYIs4IbbAbCFsfMe3WW dpdYgYST3BWCnsoH3fONfwI9+OfyQ26IRYvcjxXoAQuw1G/e7YGQ0/X X-Developer-Key: i=vinicius.gomes@intel.com; a=ed25519; pk=aJkrtgqgT6TZ8iIHSG8/rTPsmlYnjMrUjCsMYvCzntk= From: Giovanni Cabiddu Since commit e2c3b6b21c77 ("mm: zswap: use SG list decompression APIs from zsmalloc"), zswap passes the raw zsmalloc SG list directly to crypto drivers, so a compressed object spanning multiple pages reaches IAA as a multi-entry source. Such requests currently fall back to software decompression. As IAA hardware requires a single DMA source buffer, linearize small multi-entry sources into a pre-allocated bounce page and submit that to the hardware instead of falling back to software. Keep the software fallback only for multi-entry destinations. This recovers most of the performance lost by using the software fallback. Store the bounce-page state in the acomp request context alongside the existing compression CRC, free it through a shared source-unmap helper, and back the pages with a small module-wide mempool so the path remains available in reclaim-driven callers. Signed-off-by: Giovanni Cabiddu Reviewed-by: Dave Jiang --- drivers/crypto/intel/iaa/iaa_crypto_main.c | 110 ++++++++++++++++++++++++-= ---- 1 file changed, 92 insertions(+), 18 deletions(-) diff --git a/drivers/crypto/intel/iaa/iaa_crypto_main.c b/drivers/crypto/in= tel/iaa/iaa_crypto_main.c index 8f68b1478476..54bde11c454c 100644 --- a/drivers/crypto/intel/iaa/iaa_crypto_main.c +++ b/drivers/crypto/intel/iaa/iaa_crypto_main.c @@ -9,6 +9,7 @@ #include #include #include +#include #include #include #include @@ -157,6 +158,15 @@ static bool async_mode; /* Use interrupts */ static bool use_irq; =20 +struct iaa_req_ctx { + u32 compression_crc; + struct page *bounce_src; + struct scatterlist bounce_sg; +}; + +static mempool_t *iaa_bounce_pool; +#define IAA_BOUNCE_POOL_SIZE 128 + /** * set_iaa_sync_mode - Set IAA sync mode * @name: The name of the sync mode @@ -984,6 +994,19 @@ static inline int check_completion(struct device *dev, return ret; } =20 +static void iaa_unmap_src(struct device *dev, struct acomp_req *req) +{ + struct iaa_req_ctx *req_ctx =3D acomp_request_ctx(req); + struct scatterlist *src =3D req_ctx->bounce_src ? &req_ctx->bounce_sg : r= eq->src; + + dma_unmap_sg(dev, src, 1, DMA_TO_DEVICE); + + if (req_ctx->bounce_src) { + mempool_free(req_ctx->bounce_src, iaa_bounce_pool); + req_ctx->bounce_src =3D NULL; + } +} + static int deflate_fallback(struct acomp_req *req, bool compress) { ACOMP_FBREQ_ON_STACK(fbreq, req); @@ -1040,6 +1063,7 @@ static void iaa_desc_complete(struct idxd_desc *idxd_= desc, struct iaa_device_compression_mode *active_compression_mode; struct iaa_compression_ctx *compression_ctx; struct crypto_ctx *ctx =3D __ctx; + struct iaa_req_ctx *req_ctx =3D acomp_request_ctx(ctx->req); struct iaa_device *iaa_device; struct idxd_device *idxd; struct iaa_wq *iaa_wq; @@ -1098,10 +1122,9 @@ static void iaa_desc_complete(struct idxd_desc *idxd= _desc, } =20 if (ctx->compress && compression_ctx->verify_compress) { - u32 *compression_crc =3D acomp_request_ctx(ctx->req); dma_addr_t src_addr, dst_addr; =20 - *compression_crc =3D idxd_desc->iax_completion->crc; + req_ctx->compression_crc =3D idxd_desc->iax_completion->crc; =20 ret =3D iaa_remap_for_verify(dev, iaa_wq, ctx->req, &src_addr, &dst_addr= ); if (ret) { @@ -1124,7 +1147,7 @@ static void iaa_desc_complete(struct idxd_desc *idxd_= desc, } err: dma_unmap_sg(dev, ctx->req->dst, sg_nents(ctx->req->dst), DMA_FROM_DEVICE= ); - dma_unmap_sg(dev, ctx->req->src, sg_nents(ctx->req->src), DMA_TO_DEVICE); + iaa_unmap_src(dev, ctx->req); out: if (ret !=3D 0) dev_dbg(dev, "asynchronous compress failed ret=3D%d\n", ret); @@ -1144,7 +1167,7 @@ static int iaa_compress(struct crypto_tfm *tfm, struc= t acomp_req *req, { struct iaa_device_compression_mode *active_compression_mode; struct iaa_compression_ctx *ctx =3D crypto_tfm_ctx(tfm); - u32 *compression_crc =3D acomp_request_ctx(req); + struct iaa_req_ctx *req_ctx =3D acomp_request_ctx(req); struct iaa_device *iaa_device; struct idxd_desc *idxd_desc; struct iax_hw_desc *desc; @@ -1235,7 +1258,7 @@ static int iaa_compress(struct crypto_tfm *tfm, struc= t acomp_req *req, update_total_comp_bytes_out(*dlen); update_wq_comp_bytes(wq, *dlen); =20 - *compression_crc =3D idxd_desc->iax_completion->crc; + req_ctx->compression_crc =3D idxd_desc->iax_completion->crc; =20 if (!ctx->async_mode) idxd_free_desc(wq, idxd_desc); @@ -1295,7 +1318,7 @@ static int iaa_compress_verify(struct crypto_tfm *tfm= , struct acomp_req *req, { struct iaa_device_compression_mode *active_compression_mode; struct iaa_compression_ctx *ctx =3D crypto_tfm_ctx(tfm); - u32 *compression_crc =3D acomp_request_ctx(req); + struct iaa_req_ctx *req_ctx =3D acomp_request_ctx(req); struct iaa_device *iaa_device; struct idxd_desc *idxd_desc; struct iax_hw_desc *desc; @@ -1355,10 +1378,10 @@ static int iaa_compress_verify(struct crypto_tfm *t= fm, struct acomp_req *req, goto err; } =20 - if (*compression_crc !=3D idxd_desc->iax_completion->crc) { + if (req_ctx->compression_crc !=3D idxd_desc->iax_completion->crc) { ret =3D -EINVAL; - dev_dbg(dev, "(verify) iaa comp/decomp crc mismatch:" - " comp=3D0x%x, decomp=3D0x%x\n", *compression_crc, + dev_dbg(dev, "(verify) iaa comp/decomp crc mismatch: comp=3D0x%x, decomp= =3D0x%x\n", + req_ctx->compression_crc, idxd_desc->iax_completion->crc); print_hex_dump(KERN_INFO, "cmp-rec: ", DUMP_PREFIX_OFFSET, 8, 1, idxd_desc->iax_completion, 64, 0); @@ -1498,6 +1521,7 @@ static int iaa_decompress(struct crypto_tfm *tfm, str= uct acomp_req *req, =20 static int iaa_comp_acompress(struct acomp_req *req) { + struct iaa_req_ctx *req_ctx =3D acomp_request_ctx(req); struct iaa_compression_ctx *compression_ctx; struct crypto_tfm *tfm =3D req->base.tfm; dma_addr_t src_addr, dst_addr; @@ -1506,6 +1530,8 @@ static int iaa_comp_acompress(struct acomp_req *req) struct idxd_wq *wq; struct device *dev; =20 + req_ctx->bounce_src =3D NULL; + compression_ctx =3D crypto_tfm_ctx(tfm); =20 if (!iaa_crypto_enabled) { @@ -1597,12 +1623,18 @@ static int iaa_comp_acompress(struct acomp_req *req) =20 static int iaa_comp_adecompress(struct acomp_req *req) { + struct iaa_req_ctx *req_ctx =3D acomp_request_ctx(req); struct crypto_tfm *tfm =3D req->base.tfm; + struct scatterlist *src =3D req->src; dma_addr_t src_addr, dst_addr; + bool use_bounce_src =3D false; int cpu, ret =3D 0; struct iaa_wq *iaa_wq; struct device *dev; struct idxd_wq *wq; + struct page *page; + + req_ctx->bounce_src =3D NULL; =20 if (!iaa_crypto_enabled) { pr_debug("iaa_crypto disabled, not decompressing\n"); @@ -1614,10 +1646,16 @@ static int iaa_comp_adecompress(struct acomp_req *r= eq) return -EINVAL; } =20 - /* Fall back to software if src or dst has multiple sg entries */ - if (sg_nents(req->src) > 1 || sg_nents(req->dst) > 1) + /* Fall back to software if dst has multiple sg entries */ + if (sg_nents(req->dst) > 1) return deflate_generic_decompress(req); =20 + if (sg_nents(req->src) > 1) { + if (req->slen > PAGE_SIZE) + return deflate_generic_decompress(req); + use_bounce_src =3D true; + } + cpu =3D get_cpu(); wq =3D wq_table_next_wq(cpu); put_cpu(); @@ -1636,20 +1674,45 @@ static int iaa_comp_adecompress(struct acomp_req *r= eq) =20 dev =3D &wq->idxd->pdev->dev; =20 - if (!dma_map_sg(dev, req->src, 1, DMA_TO_DEVICE)) { + if (unlikely(use_bounce_src)) { + page =3D mempool_alloc(iaa_bounce_pool, GFP_ATOMIC); + if (!page) { + iaa_wq_put(wq); + return deflate_generic_decompress(req); + } + + if (sg_copy_to_buffer(req->src, sg_nents(req->src), + page_address(page), req->slen) !=3D req->slen) { + mempool_free(page, iaa_bounce_pool); + iaa_wq_put(wq); + return deflate_generic_decompress(req); + } + + sg_init_table(&req_ctx->bounce_sg, 1); + sg_set_page(&req_ctx->bounce_sg, page, req->slen, 0); + req_ctx->bounce_src =3D page; + src =3D &req_ctx->bounce_sg; + } + + if (!dma_map_sg(dev, src, 1, DMA_TO_DEVICE)) { dev_dbg(dev, "couldn't map src sg for iaa device %d, wq %d\n", iaa_wq->iaa_device->idxd->id, iaa_wq->wq->id); + if (req_ctx->bounce_src) { + mempool_free(req_ctx->bounce_src, iaa_bounce_pool); + req_ctx->bounce_src =3D NULL; + } iaa_wq_put(wq); return deflate_generic_decompress(req); } - src_addr =3D sg_dma_address(req->src); - dev_dbg(dev, "map src %llx req->src %p slen %d sg_len %d\n", src_addr, - req->src, req->slen, sg_dma_len(req->src)); + src_addr =3D sg_dma_address(src); + dev_dbg(dev, "dma_map_sg, src_addr %llx, src %p," + " req->slen %d, sg_dma_len(sg) %d\n", src_addr, + src, req->slen, sg_dma_len(src)); =20 if (!dma_map_sg(dev, req->dst, 1, DMA_FROM_DEVICE)) { dev_dbg(dev, "couldn't map dst sg for iaa device %d, wq %d\n", iaa_wq->iaa_device->idxd->id, iaa_wq->wq->id); - dma_unmap_sg(dev, req->src, 1, DMA_TO_DEVICE); + iaa_unmap_src(dev, req); iaa_wq_put(wq); return deflate_generic_decompress(req); } @@ -1666,7 +1729,7 @@ static int iaa_comp_adecompress(struct acomp_req *req) dev_dbg(dev, "asynchronous decompress failed ret=3D%d\n", ret); =20 dma_unmap_sg(dev, req->dst, 1, DMA_FROM_DEVICE); - dma_unmap_sg(dev, req->src, 1, DMA_TO_DEVICE); + iaa_unmap_src(dev, req); iaa_wq_put(wq); =20 return ret; @@ -1700,7 +1763,7 @@ static struct acomp_alg iaa_acomp_fixed_deflate =3D { .cra_driver_name =3D "deflate-iaa", .cra_flags =3D CRYPTO_ALG_ASYNC, .cra_ctxsize =3D sizeof(struct iaa_compression_ctx), - .cra_reqsize =3D sizeof(u32), + .cra_reqsize =3D sizeof(struct iaa_req_ctx), .cra_module =3D THIS_MODULE, .cra_priority =3D IAA_ALG_PRIORITY, } @@ -1899,6 +1962,12 @@ static int __init iaa_crypto_init_module(void) goto err_aecs_init; } =20 + iaa_bounce_pool =3D mempool_create_page_pool(IAA_BOUNCE_POOL_SIZE, 0); + if (!iaa_bounce_pool) { + ret =3D -ENOMEM; + goto err_bounce_pool; + } + ret =3D idxd_driver_register(&iaa_crypto_driver); if (ret) { pr_debug("IAA wq sub-driver registration failed\n"); @@ -1932,6 +2001,9 @@ static int __init iaa_crypto_init_module(void) err_verify_attr_create: idxd_driver_unregister(&iaa_crypto_driver); err_driver_reg: + mempool_destroy(iaa_bounce_pool); + iaa_bounce_pool =3D NULL; +err_bounce_pool: iaa_aecs_cleanup_fixed(); err_aecs_init: =20 @@ -1948,6 +2020,8 @@ static void __exit iaa_crypto_cleanup_module(void) driver_remove_file(&iaa_crypto_driver.drv, &driver_attr_verify_compress); idxd_driver_unregister(&iaa_crypto_driver); + mempool_destroy(iaa_bounce_pool); + iaa_bounce_pool =3D NULL; iaa_aecs_cleanup_fixed(); =20 pr_debug("cleaned up\n"); --=20 2.55.0