From nobody Thu Sep 24 12:50:59 2026 Received: from pdx-out-004.esa.us-west-2.outbound.mail-perimeter.amazon.com (pdx-out-004.esa.us-west-2.outbound.mail-perimeter.amazon.com [44.246.77.92]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0E5A03D0BED; Thu, 24 Sep 2026 07:58:57 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=44.246.77.92 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1790236739; cv=none; b=tcuyWNTtXlyqDK5XRhi+FwmM8RR4hGh3vP4JFsuI7lwL2LaVhIV3hxXoWXVXm/LbL3jySJdVaiQIVezFCQZhgmBqJ/TctYEhsIqajw0ziUazf2VWND+0TDyWxaAww6CWgqOVCZ5nj0uRSw7vXwPh+H7afNVUkVMUTmne0qWhlLk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1790236739; c=relaxed/simple; bh=uRLD90ofwdcRaXkT7+31WCE21OuL6RWJmucs1E7Ny7c=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=ik7vB1wUbthSmyc2ja+tgm7o1fazPRKAZJsu9dBBw3ZXT5WbnMD0Sh6rENTQjqqWRWQ7QibOwkYSOnf3p0QMxv420zgYnNHX4j1TJwJGCwimnwWsljHZiQlcHCehCANIU+JS4gnzXuhI/LVLpViVhA3Z4j0R+ZNf8wgeMPkmmF0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amazon.com; spf=pass smtp.mailfrom=amazon.com; dkim=pass (2048-bit key) header.d=amazon.com header.i=@amazon.com header.b=lfADdIiM; arc=none smtp.client-ip=44.246.77.92 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amazon.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=amazon.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=amazon.com header.i=@amazon.com header.b="lfADdIiM" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amazon.com; i=@amazon.com; q=dns/txt; s=amazoncorp2; t=1790236738; x=1821772738; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=xR6D4Fpn8Lfnsy+EDTn4FQrP1m5sYBEibA/5npG+Mc8=; b=lfADdIiMGXYwqaPGylVgHju07jHXSR+jfJsusSwmclF/9MPidutCLMSt OalhVkRRbYKQ9ViRKzY/UqXWMp2CBoOuGaXmlcvVWpx7t1vA+HGdc34ct bUmupoeB2adpn03T+vvO9PzRb7QdD72V9u4y+vWKPqCYJDlKK1zybGavg ZS9T8FwLz7Zb5+bh8KxiborLxgtt603YLNRaLtM8K1jT54ZOIzxQ3y5WA uCr6s1BkMg1o6Bh91dOmETn2bT4CrClaLCtvaX+CHm81HYV/df5asi0jw Tn+ILOBRKd2CiSm8xKjyKDIScNfoKlSIc4QjP1YiwIXlfQfaNrpHioGOH A==; X-CSE-ConnectionGUID: 3RL2gkWdRcCNEBEcKXg8rQ== X-CSE-MsgGUID: kVIwJgbDTQu99lBcCPEs2A== X-IronPort-AV: E=Sophos;i="6.27,120,1787011200"; d="scan'208";a="29512013" Received: from ip-10-5-6-203.us-west-2.compute.internal (HELO smtpout.naws.us-west-2.prod.farcaster.email.amazon.dev) ([10.5.6.203]) by internal-pdx-out-004.esa.us-west-2.outbound.mail-perimeter.amazon.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 24 Sep 2026 07:58:55 +0000 Received: from EX19MTAUWC001.ant.amazon.com [205.251.233.105:3528] by smtpin.naws.us-west-2.prod.farcaster.email.amazon.dev [10.0.16.27:2525] with esmtp (Farcaster) id 33ce8e2e-4efa-4114-afe2-7bbf48d6b324; Thu, 24 Sep 2026 07:58:55 +0000 (UTC) X-Farcaster-Flow-ID: 33ce8e2e-4efa-4114-afe2-7bbf48d6b324 Received: from EX19D001UWA001.ant.amazon.com (10.13.138.214) by EX19MTAUWC001.ant.amazon.com (10.250.64.174) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA) id 15.2.2562.49; Thu, 24 Sep 2026 07:58:55 +0000 Received: from dev-dsk-lravich-1b-7405803b.eu-west-1.amazon.com (10.13.225.95) by EX19D001UWA001.ant.amazon.com (10.13.138.214) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA) id 15.2.2562.49; Thu, 24 Sep 2026 07:58:53 +0000 From: Leonid Ravich To: , CC: , , , , , , , Subject: [PATCH v6 1/6] crypto: skcipher - add per-request unit_size Date: Thu, 24 Sep 2026 07:58:41 +0000 Message-ID: <20260924075846.28203-2-lravich@amazon.com> X-Mailer: git-send-email 2.47.3 In-Reply-To: <20260924075846.28203-1-lravich@amazon.com> References: <20260924075846.28203-1-lravich@amazon.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: EX19D040UWB002.ant.amazon.com (10.13.138.89) To EX19D001UWA001.ant.amazon.com (10.13.138.214) Content-Type: text/plain; charset="utf-8" Add a unit_size field to struct skcipher_request, mirroring the acomp unit_size (and its setter), so a caller can submit several data units in one request: cryptlen / unit_size units sharing one starting IV, with per-unit IVs derived from the IV as a 64-bit little-endian data-unit-number counter held in its low 8 bytes. unit_size =3D=3D 0 (the default) means a normal single-unit request; skcipher_request_set_tfm() and the on-stack request initializer zero the field, so existing callers and reused requests are unaffected. Suggested-by: Herbert Xu Signed-off-by: Leonid Ravich --- include/crypto/skcipher.h | 46 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 46 insertions(+) diff --git a/include/crypto/skcipher.h b/include/crypto/skcipher.h index 4efe2ca8c4d1..2d7805409140 100644 --- a/include/crypto/skcipher.h +++ b/include/crypto/skcipher.h @@ -31,6 +31,17 @@ struct scatterlist; /** * struct skcipher_request - Symmetric key cipher request * @cryptlen: Number of bytes to encrypt or decrypt + * @unit_size: Size in bytes of each data unit, or 0 for a + * single-unit request (the default). When non-zero, must be a + * multiple of the cipher block size and @cryptlen must be a + * positive multiple of it. The data-unit number is a 64-bit + * little-endian counter in the low 8 bytes of @iv, incremented + * once per unit and wrapping at 2^64; the remaining IV bytes are + * left unchanged (any other on-disk IV layout is produced by a + * template wrapping the algorithm). @iv itself is not modified + * by the request. An algorithm that advertises + * CRYPTO_ALG_REQ_SEG handles the whole request natively and must + * follow this same counter and @iv-preservation convention. * @iv: Initialisation Vector * @src: Source SG list * @dst: Destination SG list @@ -39,6 +50,7 @@ struct scatterlist; */ struct skcipher_request { unsigned int cryptlen; + unsigned int unit_size; =20 u8 *iv; =20 @@ -225,6 +237,7 @@ struct lskcipher_alg { struct skcipher_request *name =3D \ (((struct skcipher_request *)__##name##_desc)->base.tfm =3D \ crypto_sync_skcipher_tfm((_tfm)), \ + ((struct skcipher_request *)__##name##_desc)->unit_size =3D 0, \ (void *)__##name##_desc) =20 /** @@ -819,6 +832,8 @@ static inline void skcipher_request_set_tfm(struct skci= pher_request *req, struct crypto_skcipher *tfm) { req->base.tfm =3D crypto_skcipher_tfm(tfm); + /* New tfm, new request: default to single-unit. */ + req->unit_size =3D 0; } =20 static inline void skcipher_request_set_sync_tfm(struct skcipher_request *= req, @@ -908,6 +923,10 @@ static inline void skcipher_request_set_callback(struc= t skcipher_request *req, req->base.complete =3D compl; req->base.data =3D data; req->base.flags =3D flags; + /* Reset the per-op multi-unit control; a reused request defaults to + * single-unit until skcipher_request_set_unit_size() opts back in. + */ + req->unit_size =3D 0; } =20 /** @@ -937,5 +956,32 @@ static inline void skcipher_request_set_crypt( req->iv =3D iv; } =20 +/** + * skcipher_request_set_unit_size() - submit as multiple data units + * @req: request handle + * @unit_size: unit size in bytes (a multiple of the cipher block size), + * or 0 to disable + * + * Process @req as @cryptlen / @unit_size data units sharing one starting + * @iv, with per-unit IVs derived by treating @iv as a wide counter (the + * data-unit-number convention). @cryptlen must be a positive multiple of + * @unit_size. If the algorithm does not handle multiple units natively, + * the API transparently splits the request into one call per unit; that + * split additionally requires an ivsize that is a non-zero multiple of 8 + * and at most 32 bytes, and rejects a violating request with -EINVAL. An + * algorithm advertising CRYPTO_ALG_REQ_SEG receives the whole request and + * enforces its own constraints. + * + * This function must be called after skcipher_request_set_tfm() and + * skcipher_request_set_callback(), both of which reset @req->unit_size + * to 0. + */ +static inline void +skcipher_request_set_unit_size(struct skcipher_request *req, + unsigned int unit_size) +{ + req->unit_size =3D unit_size; +} + #endif /* _CRYPTO_SKCIPHER_H */ =20 --=20 2.47.3 From nobody Thu Sep 24 12:50:59 2026 Received: from pdx-out-013.esa.us-west-2.outbound.mail-perimeter.amazon.com (pdx-out-013.esa.us-west-2.outbound.mail-perimeter.amazon.com [34.218.115.239]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id BC66A43F4A9; Thu, 24 Sep 2026 07:59:00 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=34.218.115.239 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1790236742; cv=none; b=tNw5aqUn1fdJk/EDQRkBf3Mrt1rIL4pJ20dMCw1KpcR9XMAqOm3EkHWfN+GqJL5QgSP+gxQhtJUKaj2NBkerR2hRCkUwpIoMW2LvcW6n2iE4uOgiqpAC3iEMBuwUzKWosC41VxR1C4u2VCOIxJyXoJyO/zd3GTJGbPupwMamLqw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1790236742; c=relaxed/simple; bh=dLU021jZO83OkhPh3al7j7elX0OHMX6tDi+7AO+04XM=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=YtZAq+uYBHdflbeSB2hU8HtnvXLBGP4+PmFir4zj5hTQfGxqHHXq2XBd0uWBfbeDm8v9iHq71cCbTfCnIzH15F0gjRRdZ6zO2vBTWdJXvQFw2wBB8n4nk7IH6ZEBVzC7JWJFAe7Hw3pm/UQOhWtQeAv9tfgEnVxzwwGyEzphkQ8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amazon.com; spf=pass smtp.mailfrom=amazon.com; dkim=pass (2048-bit key) header.d=amazon.com header.i=@amazon.com header.b=BdgXubRK; arc=none smtp.client-ip=34.218.115.239 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amazon.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=amazon.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=amazon.com header.i=@amazon.com header.b="BdgXubRK" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amazon.com; i=@amazon.com; q=dns/txt; s=amazoncorp2; t=1790236740; x=1821772740; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=OvOvPDn/0aC+MRRS9yXhpVAo5rrw/Rz0Ky+brwv+uMw=; b=BdgXubRKAVMJdcbPjWrgpm+fJ3CV6uvLFfeC1kgA16TYpxkrMWtsf6RW zyr+wIJG0SZqdSn96p8AJB1XZkFJQFjHvuOa7NvYI6IjtCDW3Q1pvajFD PBwkHw7POyUzdtQ0YE7GE2BOMGR+9kb/odUBPfQKP1YW28kjUMW1naGzT Wk73hSxOn8yVLF0TfZtZg2c/8ZpHqCRjS0+z+DcBIoiS/lkh75tBg7YVw 1vxO5em26WP7I1cIkxw/WlQ6IXDs7Xy4RdtNxTC/x0BVb88esBMk+WNNG njFiUpJuGybzhREDLJQRkH/1aC/kqRZTlbO+IH43I1yGBU6tChKAkNeJf Q==; X-CSE-ConnectionGUID: asVA47ilRCW10AvV4nfqXQ== X-CSE-MsgGUID: XmBqy+pwQCq82ExdEvBs2Q== X-IronPort-AV: E=Sophos;i="6.27,120,1787011200"; d="scan'208";a="29302663" Received: from ip-10-5-12-219.us-west-2.compute.internal (HELO smtpout.naws.us-west-2.prod.farcaster.email.amazon.dev) ([10.5.12.219]) by internal-pdx-out-013.esa.us-west-2.outbound.mail-perimeter.amazon.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 24 Sep 2026 07:58:57 +0000 Received: from EX19MTAUWB002.ant.amazon.com [205.251.233.48:23556] by smtpin.naws.us-west-2.prod.farcaster.email.amazon.dev [10.0.58.237:2525] with esmtp (Farcaster) id a42c9bb3-f593-4707-8eff-48313dc89cf5; Thu, 24 Sep 2026 07:58:57 +0000 (UTC) X-Farcaster-Flow-ID: a42c9bb3-f593-4707-8eff-48313dc89cf5 Received: from EX19D001UWA001.ant.amazon.com (10.13.138.214) by EX19MTAUWB002.ant.amazon.com (10.250.64.231) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA) id 15.2.2562.49; Thu, 24 Sep 2026 07:58:56 +0000 Received: from dev-dsk-lravich-1b-7405803b.eu-west-1.amazon.com (10.13.225.95) by EX19D001UWA001.ant.amazon.com (10.13.138.214) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA) id 15.2.2562.49; Thu, 24 Sep 2026 07:58:55 +0000 From: Leonid Ravich To: , CC: , , , , , , , , Kanchana P Sridhar Subject: [PATCH v6 2/6] crypto: acomp - Add bit to indicate segmentation support Date: Thu, 24 Sep 2026 07:58:42 +0000 Message-ID: <20260924075846.28203-3-lravich@amazon.com> X-Mailer: git-send-email 2.47.3 In-Reply-To: <20260924075846.28203-1-lravich@amazon.com> References: <20260924075846.28203-1-lravich@amazon.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: EX19D040UWB002.ant.amazon.com (10.13.138.89) To EX19D001UWA001.ant.amazon.com (10.13.138.214) Content-Type: text/plain; charset="utf-8" From: Kanchana P Sridhar This patch adds segmentation support for compression. Add a bit to the crypto_alg flags to indicate support for segmentation. Also add a helper for acomp to test whether a given tfm supports segmentation. Signed-off-by: Herbert Xu (cherry picked from commit 9c16129795e20b055d5fa8ea7e3fdd35cf6a9a1a) Signed-off-by: Leonid Ravich --- include/crypto/algapi.h | 5 +++++ include/crypto/internal/acompress.h | 5 +++++ include/linux/crypto.h | 3 +++ 3 files changed, 13 insertions(+) diff --git a/include/crypto/algapi.h b/include/crypto/algapi.h index 05deea9dac5e..7d406cfe5751 100644 --- a/include/crypto/algapi.h +++ b/include/crypto/algapi.h @@ -280,6 +280,11 @@ static inline bool crypto_tfm_req_virt(struct crypto_t= fm *tfm) return tfm->__crt_alg->cra_flags & CRYPTO_ALG_REQ_VIRT; } =20 +static inline bool crypto_tfm_req_seg(struct crypto_tfm *tfm) +{ + return tfm->__crt_alg->cra_flags & CRYPTO_ALG_REQ_SEG; +} + static inline u32 crypto_request_flags(struct crypto_async_request *req) { return req->flags & ~CRYPTO_TFM_REQ_ON_STACK; diff --git a/include/crypto/internal/acompress.h b/include/crypto/internal/= acompress.h index 9cd37df32dc4..93ae59623f13 100644 --- a/include/crypto/internal/acompress.h +++ b/include/crypto/internal/acompress.h @@ -189,6 +189,11 @@ static inline bool crypto_acomp_req_virt(struct crypto= _acomp *tfm) return crypto_tfm_req_virt(&tfm->base); } =20 +static inline bool crypto_acomp_req_seg(struct crypto_acomp *tfm) +{ + return crypto_tfm_req_seg(&tfm->base); +} + void crypto_acomp_free_streams(struct crypto_acomp_streams *s); int crypto_acomp_alloc_streams(struct crypto_acomp_streams *s); =20 diff --git a/include/linux/crypto.h b/include/linux/crypto.h index a2137e19be7d..89b9c3f87f4d 100644 --- a/include/linux/crypto.h +++ b/include/linux/crypto.h @@ -139,6 +139,9 @@ /* Set if the algorithm cannot have a fallback (e.g., phmac). */ #define CRYPTO_ALG_NO_FALLBACK 0x00080000 =20 +/* Set if the algorithm supports segmentation. */ +#define CRYPTO_ALG_REQ_SEG 0x00100000 + /* The high bits 0xff000000 are reserved for type-specific flags. */ =20 /* --=20 2.47.3 From nobody Thu Sep 24 12:50:59 2026 Received: from pdx-out-015.esa.us-west-2.outbound.mail-perimeter.amazon.com (pdx-out-015.esa.us-west-2.outbound.mail-perimeter.amazon.com [50.112.246.219]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B042C43F8A1; Thu, 24 Sep 2026 07:59:01 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=50.112.246.219 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1790236743; cv=none; b=DF6YXb70Krb6PqZFi7SL+f/MNwx6MqyvtOoAjQS7X3ho/kVA2o/8WUeNgXMgd1yD4gQXJsvFIYqtfCXej3FUYilrDdgw8EnDFOXhXms9SMREQ1bnBtwXAsN0XXAIFHEKF1sKstixYDRszJdytradRF1n/OY9qaKdVjn+R88+Wg0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1790236743; c=relaxed/simple; bh=fdiZHr1WzWC8owD6HFDgNp2ni3RXDVaDb840dbxOpcQ=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=hsr3nF6XZQc7YoIZJmK1aIiLWMg0PUKIAdQ3Kg+Bac0xUbrgRl4FjQSd7vxEi7yWknNcs505c+YOSy75ilRC95VTne45m9hA5438f7jVMcMmxfTsskevs4Pv393l2E53XRknrRxF/2YGK1j2HwbLD5liWRXVy86ZWRgPqurf8yc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amazon.com; spf=pass smtp.mailfrom=amazon.com; dkim=pass (2048-bit key) header.d=amazon.com header.i=@amazon.com header.b=DyBBh3b7; arc=none smtp.client-ip=50.112.246.219 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amazon.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=amazon.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=amazon.com header.i=@amazon.com header.b="DyBBh3b7" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amazon.com; i=@amazon.com; q=dns/txt; s=amazoncorp2; t=1790236741; x=1821772741; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=uCiR4m3gEDIjBUSHTt1ExAl7BOUXXUa0MIVN5VZyFxQ=; b=DyBBh3b7sa+xQwc4TbhSHTGACw1DcLxal5zUm0hoD0rhT4zdu56NGWZF KMDlgqaPuHlS33JHyOAehIosITYY9QfEqw8lCRNqLZ5AqU3sa0TjZWcDx uYZPHd4WzhiXwHM3HUJey9CkgO/CRVQ/n/u1Kazw7KimUDy9tRJgX/5rY xQ8KyPAcfWMgsmV+L9sUQk2c+B1i7ZNbQsmmN2Mwx4vl+n06mRPtZqg4z bJ1W/aJAz3maLlnGbN+MomDQ7uOwK/9JHtwa8cq1+t9DnJldku2bzvLFO 3hN2Hd5Y6sR7zMygeNl5GCTwU2PXD3hTFZN3OBGPalr1L2+WvgRMXTQfn g==; X-CSE-ConnectionGUID: FaRZ70wiRzGLoMhx86vGSw== X-CSE-MsgGUID: tSPBc/H8SamLpMk8Bm/wsw== X-IronPort-AV: E=Sophos;i="6.27,120,1787011200"; d="scan'208";a="29295289" Received: from ip-10-5-6-203.us-west-2.compute.internal (HELO smtpout.naws.us-west-2.prod.farcaster.email.amazon.dev) ([10.5.6.203]) by internal-pdx-out-015.esa.us-west-2.outbound.mail-perimeter.amazon.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 24 Sep 2026 07:58:59 +0000 Received: from EX19MTAUWA002.ant.amazon.com [205.251.233.178:7317] by smtpin.naws.us-west-2.prod.farcaster.email.amazon.dev [10.0.16.27:2525] with esmtp (Farcaster) id 6e27a216-6ae1-48a6-9653-f87c6e32dfc9; Thu, 24 Sep 2026 07:58:59 +0000 (UTC) X-Farcaster-Flow-ID: 6e27a216-6ae1-48a6-9653-f87c6e32dfc9 Received: from EX19D001UWA001.ant.amazon.com (10.13.138.214) by EX19MTAUWA002.ant.amazon.com (10.250.64.202) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA) id 15.2.2562.49; Thu, 24 Sep 2026 07:58:58 +0000 Received: from dev-dsk-lravich-1b-7405803b.eu-west-1.amazon.com (10.13.225.95) by EX19D001UWA001.ant.amazon.com (10.13.138.214) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA) id 15.2.2562.49; Thu, 24 Sep 2026 07:58:56 +0000 From: Leonid Ravich To: , CC: , , , , , , , Subject: [PATCH v6 3/6] crypto: skcipher - add crypto_skcipher_req_seg() helper Date: Thu, 24 Sep 2026 07:58:43 +0000 Message-ID: <20260924075846.28203-4-lravich@amazon.com> X-Mailer: git-send-email 2.47.3 In-Reply-To: <20260924075846.28203-1-lravich@amazon.com> References: <20260924075846.28203-1-lravich@amazon.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: EX19D040UWB002.ant.amazon.com (10.13.138.89) To EX19D001UWA001.ant.amazon.com (10.13.138.214) Content-Type: text/plain; charset="utf-8" Thin skcipher wrapper over crypto_tfm_req_seg() (mirroring crypto_acomp_req_seg()), so the skcipher mid-layer can ask whether an algorithm advertises native multi-unit (CRYPTO_ALG_REQ_SEG) support. Signed-off-by: Leonid Ravich --- include/crypto/internal/skcipher.h | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/include/crypto/internal/skcipher.h b/include/crypto/internal/s= kcipher.h index a965b6aabf61..8e9353f4252e 100644 --- a/include/crypto/internal/skcipher.h +++ b/include/crypto/internal/skcipher.h @@ -249,6 +249,11 @@ static inline bool crypto_skcipher_tested(struct crypt= o_skcipher *tfm) return tfm_base->__crt_alg->cra_flags & CRYPTO_ALG_TESTED; } =20 +static inline bool crypto_skcipher_req_seg(struct crypto_skcipher *tfm) +{ + return crypto_tfm_req_seg(&tfm->base); +} + static inline void *skcipher_request_ctx(struct skcipher_request *req) { return req->__ctx; --=20 2.47.3 From nobody Thu Sep 24 12:50:59 2026 Received: from pdx-out-009.esa.us-west-2.outbound.mail-perimeter.amazon.com (pdx-out-009.esa.us-west-2.outbound.mail-perimeter.amazon.com [35.155.198.111]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 09B0E413789; Thu, 24 Sep 2026 07:59:26 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=35.155.198.111 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1790236768; cv=none; b=K6e6zUwRIPb5RR/QPupTmF6V5iH00XMEKgqi4p8KRqA+BtU0WHmqIxp6Z1x6ComOWt4X6omFwpy7LdAcRu9cO6UM+Te7cmHtmImbxfWLqJXK6OP18Pwd8gM5XfhDFImGN2nRqrNKHQZHbyE2krrsydUjXkWHwM5SCQqrQov0sQA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1790236768; c=relaxed/simple; bh=Ltn94xgSMghIzo/Jv3ryi2MCyLURItnDbmFwZTtvYPU=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=Aq0uDqWMjLJTnUzpCkmkQUHc8vBWqqsuME7FbI/TGJI4YFrAkAaD7D0N/jGONWDUdb+CqswnNbHRQf20UNe7rajFgP7cXAbysbCLBDWxFf7XtZp5FaF+INIpGMSeKdgj5bCjI3x9H8oxWowSzI3wkgCv/QYaUz0iuHzfjWSpsSk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amazon.com; spf=pass smtp.mailfrom=amazon.com; dkim=pass (2048-bit key) header.d=amazon.com header.i=@amazon.com header.b=oZ30hn9M; arc=none smtp.client-ip=35.155.198.111 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amazon.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=amazon.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=amazon.com header.i=@amazon.com header.b="oZ30hn9M" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amazon.com; i=@amazon.com; q=dns/txt; s=amazoncorp2; t=1790236767; x=1821772767; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=1LeNJJmXuOOwaF3g5L5jTNjFtBySa/mG9oiGNVHE4BE=; b=oZ30hn9MCm/DKwaoUn459Ikl2MSPTo64gc9x/hhSRLu8G446u7m0qZq4 Z8iSyRPGp5jggTJggaR9aCFONGSDo8KL49gfhzAoYCvoxkImJqcSuiguX avLPNWxkLuGpXWPuky1n2JLqM0PDfWD7pjIDVoDOKDvH482U5xpMmY3gn E4MhBlorP+VTEe4UQLC2ZvCC5B2dRMucfMM690FwbBmTg3oNN4Ll9z1Er C8xoTPtFAbGWVHvFbTaR6KpdrcfU2D/M4J7+qH2266wiKP+17tS1j7heo 0xEX1oucYnz+VRZb+MOvU+zImJPtMu839p+JuOuof+WJ1UcWlQzAc9fIV Q==; X-CSE-ConnectionGUID: jHiw0YDKTqapEofeic83mQ== X-CSE-MsgGUID: nGdbCIe3RpuhCbHRNkAcvw== X-IronPort-AV: E=Sophos;i="6.27,120,1787011200"; d="scan'208";a="29419260" Received: from ip-10-5-6-203.us-west-2.compute.internal (HELO smtpout.naws.us-west-2.prod.farcaster.email.amazon.dev) ([10.5.6.203]) by internal-pdx-out-009.esa.us-west-2.outbound.mail-perimeter.amazon.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 24 Sep 2026 07:59:24 +0000 Received: from EX19MTAUWB001.ant.amazon.com [205.251.233.104:22481] by smtpin.naws.us-west-2.prod.farcaster.email.amazon.dev [10.0.40.18:2525] with esmtp (Farcaster) id 17a31595-cb57-4caa-978d-4e8adbabfd3c; Thu, 24 Sep 2026 07:59:23 +0000 (UTC) X-Farcaster-Flow-ID: 17a31595-cb57-4caa-978d-4e8adbabfd3c Received: from EX19D001UWA001.ant.amazon.com (10.13.138.214) by EX19MTAUWB001.ant.amazon.com (10.250.64.248) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA) id 15.2.2562.49; Thu, 24 Sep 2026 07:59:23 +0000 Received: from dev-dsk-lravich-1b-7405803b.eu-west-1.amazon.com (10.13.225.95) by EX19D001UWA001.ant.amazon.com (10.13.138.214) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA) id 15.2.2562.49; Thu, 24 Sep 2026 07:59:21 +0000 From: Leonid Ravich To: , CC: , , , , , , , Subject: [PATCH v6 4/6] crypto: skcipher - split multi-unit requests in the API layer Date: Thu, 24 Sep 2026 07:58:44 +0000 Message-ID: <20260924075846.28203-5-lravich@amazon.com> X-Mailer: git-send-email 2.47.3 In-Reply-To: <20260924075846.28203-1-lravich@amazon.com> References: <20260924075846.28203-1-lravich@amazon.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: EX19D036UWC002.ant.amazon.com (10.13.139.242) To EX19D001UWA001.ant.amazon.com (10.13.138.214) Content-Type: text/plain; charset="utf-8" When a caller sets skcipher_request::unit_size and the algorithm does not advertise CRYPTO_ALG_REQ_SEG, transparently split the request in crypto_skcipher_encrypt/decrypt(): one call per data unit, advancing the IV between units as a 64-bit little-endian data-unit-number counter held in the low 8 bytes (the dm-crypt plain64/essiv convention). The counter wraps at 2^64 and never carries into the higher IV bytes, so the output is bit-identical to the per-unit path across the counter rollover; any other on-disk IV format is produced by a template wrapping the algorithm. An algorithm with native multi-unit support gets the whole request unchanged. The eventual goal is for underlying algorithms to gain native support so this path stops triggering. The split reuses the caller's request for each unit (same tfm, so the request context is already sized) and restores it before returning; req->iv is never modified -- each unit gets a private IV copy the algorithm may clobber. That copy is aligned to MAX_ALGAPI_ALIGNMASK so it keeps the alignment the caller's IV had. The split is synchronous, so a multi-unit request on an async non-native algorithm is rejected -EOPNOTSUPP; it reschedules between units when the caller allows sleeping (CRYPTO_TFM_REQ_MAY_SLEEP), since a large batch would otherwise run without a preemption point. Callers that never set unit_size pay one unlikely() test; the split runs before the lskcipher redirect so lskcipher-backed modes (e.g. cbc) are split correctly too. Suggested-by: Herbert Xu Signed-off-by: Leonid Ravich --- crypto/skcipher.c | 112 ++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 112 insertions(+) diff --git a/crypto/skcipher.c b/crypto/skcipher.c index 2b31d1d5d268..db1576f6bba0 100644 --- a/crypto/skcipher.c +++ b/crypto/skcipher.c @@ -432,6 +432,112 @@ int crypto_skcipher_setkey(struct crypto_skcipher *tf= m, const u8 *key, } EXPORT_SYMBOL_GPL(crypto_skcipher_setkey); =20 +/* Bounds the on-stack per-unit IV buffers: 16 covers xts, 32 Adiantum. */ +#define SKCIPHER_MAX_UNIT_IVSIZE 32 + +/* + * Advance the per-unit IV to the next data unit. The data-unit number is= a + * 64-bit little-endian counter held in the low 8 bytes of @iv (matching + * dm-crypt's plain64/essiv sector generators, which the caller feeds in a= s a + * little-endian sector number). It wraps at 2^64 and never carries into = the + * higher IV bytes, so batched output stays bit-identical to the per-unit = path + * across the counter rollover; any other on-disk IV format is produced by= a + * template wrapping the algorithm, not here. + */ +static void skcipher_unit_iv_next(u8 *iv) +{ + __le64 lo; + + memcpy(&lo, iv, sizeof(lo)); + lo =3D cpu_to_le64(le64_to_cpu(lo) + 1); + memcpy(iv, &lo, sizeof(lo)); +} + +/* + * Transparently split a multi-unit request for an algorithm with no native + * multi-unit support: one call per unit, walking the IV as a wide counter. + * The caller's request is reused for each unit (same tfm, so the request + * context is already correctly sized) and fully restored before returning. + * @req->iv is never modified; each unit gets a private copy the algorithm + * may write back in place (e.g. xts). + */ +static int skcipher_crypt_unit(struct skcipher_request *req, bool enc) +{ + struct crypto_skcipher *tfm =3D crypto_skcipher_reqtfm(req); + struct skcipher_alg *alg =3D crypto_skcipher_alg(tfm); + const unsigned int unit =3D req->unit_size; + const unsigned int total =3D req->cryptlen; + const unsigned int ivsize =3D crypto_skcipher_ivsize(tfm); + bool inplace =3D req->src =3D=3D req->dst; + struct scatterlist *o_src =3D req->src, *o_dst =3D req->dst; + struct scatter_walk src_walk, dst_walk; + struct scatterlist src_sg[2], dst_sg[2]; + u8 iv_ctr[SKCIPHER_MAX_UNIT_IVSIZE]; + /* Becomes req->iv: keep the alignment the caller's IV would have had. */ + u8 iv_unit[SKCIPHER_MAX_UNIT_IVSIZE] __aligned(MAX_ALGAPI_ALIGNMASK + 1); + u8 *o_iv =3D req->iv; + unsigned int off; + int err =3D 0; + + if (!total || !IS_ALIGNED(unit, crypto_skcipher_blocksize(tfm)) || + (total % unit) || !ivsize || + !IS_ALIGNED(ivsize, sizeof(__le64)) || + ivsize > SKCIPHER_MAX_UNIT_IVSIZE) + return -EINVAL; + + /* The split is synchronous; only a native (REQ_SEG) alg may be async. */ + if (alg->co.base.cra_flags & CRYPTO_ALG_ASYNC) + return -EOPNOTSUPP; + + /* iv_ctr is the counter; iv_unit is the per-unit copy. */ + memcpy(iv_ctr, req->iv, ivsize); + + sg_init_table(src_sg, 2); + scatterwalk_start(&src_walk, req->src); + if (!inplace) { + sg_init_table(dst_sg, 2); + scatterwalk_start(&dst_walk, req->dst); + } + + req->unit_size =3D 0; + req->cryptlen =3D unit; + + for (off =3D 0; off < total; off +=3D unit) { + scatterwalk_get_sglist(&src_walk, src_sg); + scatterwalk_skip(&src_walk, unit); + req->src =3D src_sg; + if (inplace) { + req->dst =3D src_sg; + } else { + scatterwalk_get_sglist(&dst_walk, dst_sg); + scatterwalk_skip(&dst_walk, unit); + req->dst =3D dst_sg; + } + + memcpy(iv_unit, iv_ctr, ivsize); + req->iv =3D iv_unit; + err =3D enc ? crypto_skcipher_encrypt(req) : + crypto_skcipher_decrypt(req); + if (err) + break; + + skcipher_unit_iv_next(iv_ctr); + /* + * Match dm-crypt's per-sector reschedule, but only when the + * caller allows sleeping (the split can run in atomic context). + */ + if (req->base.flags & CRYPTO_TFM_REQ_MAY_SLEEP) + cond_resched(); + } + + req->src =3D o_src; + req->dst =3D o_dst; + req->iv =3D o_iv; + req->cryptlen =3D total; + req->unit_size =3D unit; + return err; +} + int crypto_skcipher_encrypt(struct skcipher_request *req) { struct crypto_skcipher *tfm =3D crypto_skcipher_reqtfm(req); @@ -439,6 +545,9 @@ int crypto_skcipher_encrypt(struct skcipher_request *re= q) =20 if (crypto_skcipher_get_flags(tfm) & CRYPTO_TFM_NEED_KEY) return -ENOKEY; + /* Must precede the lskcipher redirect, which ignores unit_size. */ + if (unlikely(req->unit_size) && !crypto_skcipher_req_seg(tfm)) + return skcipher_crypt_unit(req, true); if (alg->co.base.cra_type !=3D &crypto_skcipher_type) return crypto_lskcipher_encrypt_sg(req); return alg->encrypt(req); @@ -452,6 +561,9 @@ int crypto_skcipher_decrypt(struct skcipher_request *re= q) =20 if (crypto_skcipher_get_flags(tfm) & CRYPTO_TFM_NEED_KEY) return -ENOKEY; + /* Must precede the lskcipher redirect, which ignores unit_size. */ + if (unlikely(req->unit_size) && !crypto_skcipher_req_seg(tfm)) + return skcipher_crypt_unit(req, false); if (alg->co.base.cra_type !=3D &crypto_skcipher_type) return crypto_lskcipher_decrypt_sg(req); return alg->decrypt(req); --=20 2.47.3 From nobody Thu Sep 24 12:50:59 2026 Received: from pdx-out-008.esa.us-west-2.outbound.mail-perimeter.amazon.com (pdx-out-008.esa.us-west-2.outbound.mail-perimeter.amazon.com [52.42.203.116]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A8A8E43C7C1; Thu, 24 Sep 2026 07:59:28 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=52.42.203.116 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1790236771; cv=none; b=IkpjvD6kbBhl7XW3qMvgXxzwwFLNQreCBDGUHf6AKFLjV/BAUsknBzWudSE6PUUpDc4O07t5FNTDUSFXrOzD4qL7zYRSAIg00VFTmB3jWFIMDUw1xI/7bVoYvsk1g0ovtaScRaVQPb0mYRu8At6RvptvILlkcmDSnGKIEN+B22k= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1790236771; c=relaxed/simple; bh=VWw8V4pGr0owLDf3UcPhPH2MT6Zo5PEddJ75ArtMtcs=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=sVHrnM+/hIjwp94WMdY5IVo1tIaHQ1gdW60JGHtvvxJTjK+kzvITiKOgcNBMR7HU3emQ58wtpFHuiAMCztX4UNsD9RFnTLbE/pUpTgmXQkALp1saH/gX5+rNP/4LKR7sqd72hbNvgtAB76LtMfOA8OkUkO87DeyXaWnfQhOSKHE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amazon.com; spf=pass smtp.mailfrom=amazon.com; dkim=pass (2048-bit key) header.d=amazon.com header.i=@amazon.com header.b=WTus7FZ8; arc=none smtp.client-ip=52.42.203.116 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amazon.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=amazon.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=amazon.com header.i=@amazon.com header.b="WTus7FZ8" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amazon.com; i=@amazon.com; q=dns/txt; s=amazoncorp2; t=1790236768; x=1821772768; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=YgwTr/ayWgmnL9MGFOXJHjDYbA9tdAiOKfUQa3KUotM=; b=WTus7FZ87xD7+ly31Nnsgx0KYLQ1UXiylyahToQ20JKNKZtG1UbYwpEQ 3PrmHUXh74F137LG35/TRfeRoijVhEtuw0cZGZx6t8B3z9oQcMf7NBZGo 4mSzflCyDsJmq0N7ASSczLyTTgxlM6SPEFej2hp6KwhzFUOlP7Qp/x2FJ FQMm0+9y5u+6HEuhP8fQrDjhF5LqRaz0quyVJrcXn3vgWv/sA8aTCowmH tnKP0I5IlDyGKCfyqOGDOy5MCTcYHx3zYxK+VCrKG9IOt6Up/djstTJz1 En/OQOsWhyrdh52pQG2C2VxkaBhdJODB1qTPbDABJo4cQwlfanWzndRGi g==; X-CSE-ConnectionGUID: Ixy4z4mfQYq9qvOJGla9oQ== X-CSE-MsgGUID: dHMnEMJ4T0WnAt+x5dhfNw== X-IronPort-AV: E=Sophos;i="6.27,120,1787011200"; d="scan'208";a="29546286" Received: from ip-10-5-12-219.us-west-2.compute.internal (HELO smtpout.naws.us-west-2.prod.farcaster.email.amazon.dev) ([10.5.12.219]) by internal-pdx-out-008.esa.us-west-2.outbound.mail-perimeter.amazon.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 24 Sep 2026 07:59:26 +0000 Received: from EX19MTAUWC002.ant.amazon.com [205.251.233.111:14576] by smtpin.naws.us-west-2.prod.farcaster.email.amazon.dev [10.0.16.27:2525] with esmtp (Farcaster) id 226f5852-3a50-4cd9-b029-5a58ef5087eb; Thu, 24 Sep 2026 07:59:25 +0000 (UTC) X-Farcaster-Flow-ID: 226f5852-3a50-4cd9-b029-5a58ef5087eb Received: from EX19D001UWA001.ant.amazon.com (10.13.138.214) by EX19MTAUWC002.ant.amazon.com (10.250.64.143) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA) id 15.2.2562.49; Thu, 24 Sep 2026 07:59:25 +0000 Received: from dev-dsk-lravich-1b-7405803b.eu-west-1.amazon.com (10.13.225.95) by EX19D001UWA001.ant.amazon.com (10.13.138.214) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA) id 15.2.2562.49; Thu, 24 Sep 2026 07:59:23 +0000 From: Leonid Ravich To: , CC: , , , , , , , Subject: [PATCH v6 5/6] crypto: testmgr - test multi-unit dispatch Date: Thu, 24 Sep 2026 07:58:45 +0000 Message-ID: <20260924075846.28203-6-lravich@amazon.com> X-Mailer: git-send-email 2.47.3 In-Reply-To: <20260924075846.28203-1-lravich@amazon.com> References: <20260924075846.28203-1-lravich@amazon.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: EX19D036UWC002.ant.amazon.com (10.13.139.242) To EX19D001UWA001.ant.amazon.com (10.13.138.214) Content-Type: text/plain; charset="utf-8" Cross-check multi-unit dispatch against an independent single-unit reference for every self-tested sync skcipher with an eligible IV: one batched request (unit_size set) over a deliberately fragmented scatterlist must produce ciphertext byte-identical to N single-unit requests with counter-walked IVs, then round-trip. The reference increments the IV as a 64-bit little-endian counter in the low 8 bytes -- independent of the API layer's implementation, so the two agree only if the carry-and-wrap is right -- and each unit size is additionally run with an IV seeded to force the counter to wrap back to zero. The caller's IV must come back unmodified. Covers ivsize 16 (xts) and 32 (Adiantum). Signed-off-by: Leonid Ravich --- crypto/testmgr.c | 272 +++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 272 insertions(+) diff --git a/crypto/testmgr.c b/crypto/testmgr.c index 4d86efae65b2..225c6a9806cb 100644 --- a/crypto/testmgr.c +++ b/crypto/testmgr.c @@ -3211,6 +3211,274 @@ static int test_skcipher(int enc, const struct ciph= er_test_suite *suite, return 0; } =20 +/* Upper bound on the IVs the multi-unit split accepts (16: xts; 32: Adian= tum). */ +#define TEST_MDU_MAX_IVSIZE 32 + +/* + * Increment the IV as a 64-bit little-endian data-unit-number counter in = the + * low 8 bytes (byte 0 the LSB), wrapping at 2^64 with no carry above -- t= he + * dm-crypt plain64/essiv convention. Deliberately independent of + * skcipher_unit_iv_next()'s implementation, so the two only agree if the + * carry-and-wrap is right. + */ +static void test_mdu_iv_inc(u8 *iv) +{ + int i; + + for (i =3D 0; i < 8; i++) + if (++iv[i]) + break; +} + +/* + * Seed @iv so the low 64-bit counter (bytes [0,8)) is all-ones but its + * least-significant byte: the 2nd increment wraps the counter back to zer= o, + * exercising the rollover. Bytes outside the low 8 keep their value. + */ +static void test_mdu_iv_boundary(u8 *iv) +{ + unsigned int i; + + for (i =3D 0; i < 8; i++) + iv[i] =3D 0xff; + iv[0] =3D 0xfe; +} + +/* Encrypt one du_size block with a plain single-DU request (the reference= ). */ +static int test_mdu_ref_encrypt(struct crypto_skcipher *tfm, const u8 *in, + u8 *out, unsigned int du_size, const u8 *iv, + unsigned int ivsize) +{ + struct skcipher_request *req; + struct scatterlist sg_in; + DECLARE_CRYPTO_WAIT(wait); + u8 ivbuf[TEST_MDU_MAX_IVSIZE]; + int err; + + req =3D skcipher_request_alloc(tfm, GFP_KERNEL); + if (!req) + return -ENOMEM; + memcpy(ivbuf, iv, ivsize); + memcpy(out, in, du_size); + sg_init_one(&sg_in, out, du_size); + skcipher_request_set_callback(req, CRYPTO_TFM_REQ_MAY_BACKLOG | + CRYPTO_TFM_REQ_MAY_SLEEP, + crypto_req_done, &wait); + skcipher_request_set_crypt(req, &sg_in, &sg_in, du_size, ivbuf); + err =3D crypto_wait_req(crypto_skcipher_encrypt(req), &wait); + skcipher_request_free(req); + return err; +} + +/* + * Build an SG over @buf with du_size-unaligned entries, so the splitter's + * per-DU views cross SG entries and exercise the scatter_walk cursor. + */ +static void test_mdu_sg_fragment(struct scatterlist *sg, unsigned int nent= s, + u8 *buf, unsigned int total) +{ + unsigned int chunk =3D total / nents; + unsigned int off =3D 0, i; + + sg_init_table(sg, nents); + for (i =3D 0; i < nents; i++) { + unsigned int len =3D (i =3D=3D nents - 1) ? total - off : chunk; + + sg_set_buf(&sg[i], buf + off, len); + off +=3D len; + } +} + +#define TEST_MDU_NR_UNITS 4 +#define TEST_MDU_NR_FRAGS 5 +/* + * Verify a batched (unit_size-set) request on @tfm is byte-equal to an + * independent N x single-unit reference on the same tfm with + * little-endian-walked IVs, over a fragmented SG, then round-trips. + * @iv_orig is the ivsize-byte starting IV (the caller varies it to exerci= se + * both a random IV and one seeded to cross a carry boundary). + */ +static int test_skcipher_multi_du_one(struct crypto_skcipher *tfm, + unsigned int du_size, + const u8 *iv_orig) +{ + const char *driver =3D crypto_skcipher_driver_name(tfm); + const unsigned int total =3D du_size * TEST_MDU_NR_UNITS; + const unsigned int ivsize =3D crypto_skcipher_ivsize(tfm); + const u32 flags =3D CRYPTO_TFM_REQ_MAY_BACKLOG | CRYPTO_TFM_REQ_MAY_SLEEP; + struct skcipher_request *req =3D NULL; + struct scatterlist sg[TEST_MDU_NR_FRAGS], sg_out[TEST_MDU_NR_FRAGS]; + DECLARE_CRYPTO_WAIT(wait); + u8 iv_work[TEST_MDU_MAX_IVSIZE], iv_ref[TEST_MDU_MAX_IVSIZE]; + u8 *plain =3D NULL, *buf =3D NULL, *ref =3D NULL, *obuf =3D NULL; + unsigned int u; + int err; + + plain =3D kmalloc(total, GFP_KERNEL); + buf =3D kmalloc(total, GFP_KERNEL); + ref =3D kmalloc(total, GFP_KERNEL); + obuf =3D kmalloc(total, GFP_KERNEL); + req =3D skcipher_request_alloc(tfm, GFP_KERNEL); + if (!plain || !buf || !ref || !obuf || !req) { + err =3D -ENOMEM; + goto out; + } + + get_random_bytes(plain, total); + + /* Reference: per-unit single requests, counter-walked IVs. */ + memcpy(iv_ref, iv_orig, ivsize); + for (u =3D 0; u < TEST_MDU_NR_UNITS; u++) { + err =3D test_mdu_ref_encrypt(tfm, plain + u * du_size, + ref + u * du_size, du_size, iv_ref, + ivsize); + if (err) { + pr_err("alg: skcipher: %s multi-DU ref encrypt failed (du=3D%u): %d\n", + driver, du_size, err); + goto out; + } + test_mdu_iv_inc(iv_ref); + } + + /* Batched: one request over a fragmented SG. */ + memcpy(buf, plain, total); + memcpy(iv_work, iv_orig, ivsize); + test_mdu_sg_fragment(sg, TEST_MDU_NR_FRAGS, buf, total); + skcipher_request_set_callback(req, flags, crypto_req_done, &wait); + skcipher_request_set_crypt(req, sg, sg, total, iv_work); + skcipher_request_set_unit_size(req, du_size); + err =3D crypto_wait_req(crypto_skcipher_encrypt(req), &wait); + if (err) { + pr_err("alg: skcipher: %s multi-DU encrypt failed (du=3D%u): %d\n", + driver, du_size, err); + goto out; + } + if (memcmp(buf, ref, total) !=3D 0) { + pr_err("alg: skcipher: %s multi-DU ciphertext differs from single-DU ref= erence (du=3D%u)\n", + driver, du_size); + err =3D -EBADMSG; + goto out; + } + /* req->iv must be unchanged after multi-DU dispatch. */ + if (memcmp(iv_work, iv_orig, ivsize) !=3D 0) { + pr_err("alg: skcipher: %s multi-DU encrypt mutated caller IV (du=3D%u)\n= ", + driver, du_size); + err =3D -EBADMSG; + goto out; + } + + /* Out-of-place: distinct dst SG, fragmented differently from src. */ + memcpy(buf, plain, total); + memset(obuf, 0, total); + test_mdu_sg_fragment(sg, TEST_MDU_NR_FRAGS, buf, total); + test_mdu_sg_fragment(sg_out, TEST_MDU_NR_FRAGS - 2, obuf, total); + skcipher_request_set_callback(req, flags, crypto_req_done, &wait); + skcipher_request_set_crypt(req, sg, sg_out, total, iv_work); + skcipher_request_set_unit_size(req, du_size); + err =3D crypto_wait_req(crypto_skcipher_encrypt(req), &wait); + if (err) { + pr_err("alg: skcipher: %s multi-DU out-of-place encrypt failed (du=3D%u)= : %d\n", + driver, du_size, err); + goto out; + } + if (memcmp(obuf, ref, total) !=3D 0) { + pr_err("alg: skcipher: %s multi-DU out-of-place ciphertext differs (du= =3D%u)\n", + driver, du_size); + err =3D -EBADMSG; + goto out; + } + + /* Round-trip the batched ciphertext back to plaintext. */ + memcpy(buf, ref, total); + test_mdu_sg_fragment(sg, TEST_MDU_NR_FRAGS, buf, total); + skcipher_request_set_callback(req, flags, crypto_req_done, &wait); + skcipher_request_set_crypt(req, sg, sg, total, iv_work); + skcipher_request_set_unit_size(req, du_size); + err =3D crypto_wait_req(crypto_skcipher_decrypt(req), &wait); + if (err) { + pr_err("alg: skcipher: %s multi-DU decrypt failed (du=3D%u): %d\n", + driver, du_size, err); + goto out; + } + if (memcmp(buf, plain, total) !=3D 0) { + pr_err("alg: skcipher: %s multi-DU round-trip mismatch (du=3D%u)\n", + driver, du_size); + err =3D -EBADMSG; + } + +out: + skcipher_request_free(req); + kfree(obuf); + kfree(ref); + kfree(buf); + kfree(plain); + return err; +} + +/* + * Cross-check multi-unit dispatch against a single-unit reference on @tfm + * over all unit sizes. Returns 0 on success or skip; -EBADMSG on a real + * mismatch. + */ +static int test_skcipher_multi_du_sizes(struct crypto_skcipher *tfm) +{ + static const unsigned int du_sizes[] =3D { 512, 1024, 2048, 4096 }; + unsigned int ivsize =3D crypto_skcipher_ivsize(tfm); + u8 iv[TEST_MDU_MAX_IVSIZE]; + unsigned int j; + int err =3D 0; + + for (j =3D 0; j < ARRAY_SIZE(du_sizes); j++) { + /* A random starting IV. */ + get_random_bytes(iv, ivsize); + err =3D test_skcipher_multi_du_one(tfm, du_sizes[j], iv); + if (err) + break; + /* And one seeded to carry across a 64-bit limb. */ + get_random_bytes(iv, ivsize); + test_mdu_iv_boundary(iv); + err =3D test_skcipher_multi_du_one(tfm, du_sizes[j], iv); + if (err) + break; + cond_resched(); + } + return err; +} + +/* + * Cross-check multi-unit dispatch against a single-unit reference for eve= ry + * eligible ivsize (16: xts; 32: Adiantum). + */ +static int test_skcipher_multi_du(struct crypto_skcipher *tfm) +{ + unsigned int ivsize =3D crypto_skcipher_ivsize(tfm); + unsigned int blocksize =3D crypto_skcipher_blocksize(tfm); + u8 keybuf[128]; + unsigned int keylen; + int err; + + if (noslowtests) + return 0; + + /* Mirror the API-layer split's eligibility; skip what it rejects. */ + if (!ivsize || ivsize % sizeof(__le64) || ivsize > TEST_MDU_MAX_IVSIZE) + return 0; + if (!blocksize || 512 % blocksize) + return 0; /* unit sizes below are 512-multiples */ + if (crypto_skcipher_alg(tfm)->co.base.cra_flags & CRYPTO_ALG_ASYNC) + return 0; /* the transparent split is sync-only */ + + keylen =3D crypto_skcipher_min_keysize(tfm); + if (keylen > sizeof(keybuf)) + return 0; /* unusually large key; skip rather than overflow */ + get_random_bytes(keybuf, keylen); + err =3D crypto_skcipher_setkey(tfm, keybuf, keylen); + if (err) + return 0; /* weak/rejected key (e.g. XTS equal halves): skip */ + + return test_skcipher_multi_du_sizes(tfm); +} + static int alg_test_skcipher(const struct alg_test_desc *desc, const char *driver, u32 type, u32 mask) { @@ -3259,6 +3527,10 @@ static int alg_test_skcipher(const struct alg_test_d= esc *desc, if (err) goto out; =20 + err =3D test_skcipher_multi_du(tfm); + if (err) + goto out; + err =3D test_skcipher_vs_generic_impl(desc->generic_driver, req, tsgls); out: free_cipher_test_sglists(tsgls); --=20 2.47.3 From nobody Thu Sep 24 12:50:59 2026 Received: from pdx-out-011.esa.us-west-2.outbound.mail-perimeter.amazon.com (pdx-out-011.esa.us-west-2.outbound.mail-perimeter.amazon.com [52.35.192.45]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B98CD43F4DA; Thu, 24 Sep 2026 07:59:31 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=52.35.192.45 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1790236774; cv=none; b=cCEE+4Oej5CdZJnlMkUeDOdURJ6/T8SkjDWiTCeqvixlaUZqKVRCIJ5HcEEM1R0waRgUHnHU5qeuCKA9fw8miWROWcK1iRljs1u5zbHL+LD986BDHqlfwMXzVqbXWMhWV8TurL4EqrrgqdDUcZalM7ksl7m823Gs/nOTBbQj+BI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1790236774; c=relaxed/simple; bh=BKCT1WDmDFal5x2/3WzCjapvuYD9S1cY0r8uhuKWZ6U=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=MTNUzVKz6uj0Y6H4VddBnLcTKWzttw4ZWHSFxsTKEc7aC01E1G8cYjXPw3OdcHsAKVTfz1yntCtBtRbrsbb83dHzBwI01k4YNIJMOFeaZPPigSrrvt9HnMZVXhEjGkI1osbo5lGEnEED6xZMpA/A/vh6SSOCx1asOV4BIp/iIhU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amazon.com; spf=pass smtp.mailfrom=amazon.com; dkim=pass (2048-bit key) header.d=amazon.com header.i=@amazon.com header.b=L6Hl5ijS; arc=none smtp.client-ip=52.35.192.45 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amazon.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=amazon.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=amazon.com header.i=@amazon.com header.b="L6Hl5ijS" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amazon.com; i=@amazon.com; q=dns/txt; s=amazoncorp2; t=1790236771; x=1821772771; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=d0A6yBEKB5pIlMDOPiNvgUgiwxwya8mWjEPJIwGfWGU=; b=L6Hl5ijSDJpiLuOm+BOXehxh50kzAMdgCNvn5GeO4yKkNLrHDYZkM9/k QINdljxIeTTySOBO3394ffyWJ6JnPuOOxeXzMNi4LhHKbFgffSNtFgZ0o RrWY+NMSgTuaR07/0LKZz+x4ILN3A08jf26RHmyisArYWbcJ2EDtNeSjp aHrzQmCxCviExoZrRhPIW3J1bORlTxthMknm+1l48ZOovT4Qmz2T9OJYV Y3ysumBjeVCw7Gw1U/X/nY3aynhfPa/694L07hDLcIsOsINiTvynRnGjj RvoCwzx53xR1iQBRBWJrz+fn+TgGcZzicJsDHhtkLHBXZyULiTQwT1mKa g==; X-CSE-ConnectionGUID: tcZ4UgvyQ6+TY6f8BaqESg== X-CSE-MsgGUID: Va2iXJLfRbqMFKjQMHhZ6A== X-IronPort-AV: E=Sophos;i="6.27,120,1787011200"; d="scan'208";a="29282609" Received: from ip-10-5-12-219.us-west-2.compute.internal (HELO smtpout.naws.us-west-2.prod.farcaster.email.amazon.dev) ([10.5.12.219]) by internal-pdx-out-011.esa.us-west-2.outbound.mail-perimeter.amazon.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 24 Sep 2026 07:59:27 +0000 Received: from EX19MTAUWA001.ant.amazon.com [205.251.233.182:18184] by smtpin.naws.us-west-2.prod.farcaster.email.amazon.dev [10.0.45.34:2525] with esmtp (Farcaster) id 24cf0b7e-0a37-47c9-8099-317faaf3a0cd; Thu, 24 Sep 2026 07:59:27 +0000 (UTC) X-Farcaster-Flow-ID: 24cf0b7e-0a37-47c9-8099-317faaf3a0cd Received: from EX19D001UWA001.ant.amazon.com (10.13.138.214) by EX19MTAUWA001.ant.amazon.com (10.250.64.217) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA) id 15.2.2562.49; Thu, 24 Sep 2026 07:59:27 +0000 Received: from dev-dsk-lravich-1b-7405803b.eu-west-1.amazon.com (10.13.225.95) by EX19D001UWA001.ant.amazon.com (10.13.138.214) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA) id 15.2.2562.49; Thu, 24 Sep 2026 07:59:25 +0000 From: Leonid Ravich To: , CC: , , , , , , , Subject: [PATCH v6 6/6] dm crypt: batch a bio segment's sectors via multi-unit requests Date: Thu, 24 Sep 2026 07:58:46 +0000 Message-ID: <20260924075846.28203-7-lravich@amazon.com> X-Mailer: git-send-email 2.47.3 In-Reply-To: <20260924075846.28203-1-lravich@amazon.com> References: <20260924075846.28203-1-lravich@amazon.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: EX19D036UWC002.ant.amazon.com (10.13.139.242) To EX19D001UWA001.ant.amazon.com (10.13.138.214) Content-Type: text/plain; charset="utf-8" For eligible configurations, submit one skcipher request per contiguous bio segment instead of one per sector: set skcipher_request::unit_size =3D cc->sector_size and hand the crypto API the whole segment (e.g. the default 512-byte sector with a 4 KiB bio_vec -> one request of 8 data units), using only the existing inline single-entry scatterlist -- no per-bio allocation. Eligible means the per-sector IV is a little-endian data-unit-number counter in the low 64 bits with a step of exactly one per unit: plain64 and essiv (essiv qualifies because its IV input is le64(sector) -- the salt encryption lives in the essiv() template), single-tfm, non-aead, sector_size 512 or iv_large_sectors, and no integrity metadata (per-sector tags/IVs need the per-sector loop). plain64be is not batched: its on-disk IV is a big-endian counter in the high 8 bytes, not the little-endian low-limb layout the API-layer split walks, so it keeps the per-sector path (batching it would need a template producing that layout). Everything else likewise keeps the existing one-sector-per-request path unchanged. Since the API-layer transparent split is synchronous, an async cipher batches only if it handles multi-unit requests natively (CRYPTO_ALG_REQ_SEG). Batching is byte-for-byte identical to the per-sector path: ciphertext verified bit-identical to an unpatched baseline for plain64 and essiv. Signed-off-by: Leonid Ravich --- drivers/md/dm-crypt.c | 136 +++++++++++++++++++++++++++++++++++------- 1 file changed, 114 insertions(+), 22 deletions(-) diff --git a/drivers/md/dm-crypt.c b/drivers/md/dm-crypt.c index 608b617fb817..ffb66c7c7a65 100644 --- a/drivers/md/dm-crypt.c +++ b/drivers/md/dm-crypt.c @@ -115,6 +115,15 @@ struct crypt_iv_operations { struct dm_crypt_request *dmreq); void (*post)(struct crypt_config *cc, u8 *iv, struct dm_crypt_request *dmreq); + + /* + * Set for IV modes whose per-sector IV is a little-endian + * data-unit-number counter (IV(s+i) =3D=3D IV(s)+i) placed in the low + * 64-bit limb, enabling multi-unit batching via the skcipher API-layer + * split. Clear for non-counter modes (lmk, tcw, ...) and for counter + * modes whose on-disk IV is not that layout (e.g. plain64be). + */ + bool unit_counter; }; =20 struct iv_benbi_private { @@ -151,6 +160,7 @@ enum cipher_flags { CRYPT_IV_LARGE_SECTORS, /* Calculate IV from sector_size, not 512B secto= rs */ CRYPT_ENCRYPT_PREPROCESS, /* Must preprocess data for encryption (elephan= t) */ CRYPT_KEY_MAC_SIZE_SET, /* The integrity_key_size option was used */ + CRYPT_MULTI_DATA_UNIT, /* Batch a bio segment's sectors per crypto reque= st */ }; =20 /* @@ -1018,15 +1028,23 @@ static const struct crypt_iv_operations crypt_iv_pl= ain_ops =3D { }; =20 static const struct crypt_iv_operations crypt_iv_plain64_ops =3D { - .generator =3D crypt_iv_plain64_gen + .generator =3D crypt_iv_plain64_gen, + .unit_counter =3D true, }; =20 static const struct crypt_iv_operations crypt_iv_plain64be_ops =3D { - .generator =3D crypt_iv_plain64be_gen + .generator =3D crypt_iv_plain64be_gen, + /* + * No unit_counter: the big-endian, high-limb on-disk layout is not the + * little-endian low-limb counter the API-layer split walks. Batching + * it needs a template producing this layout; unbatched for now. + */ }; =20 static const struct crypt_iv_operations crypt_iv_essiv_ops =3D { - .generator =3D crypt_iv_essiv_gen + .generator =3D crypt_iv_essiv_gen, + /* IV input is le64(sector); the salt-encrypt lives in essiv(). */ + .unit_counter =3D true, }; =20 static const struct crypt_iv_operations crypt_iv_benbi_ops =3D { @@ -1349,21 +1367,51 @@ static int crypt_convert_block_aead(struct crypt_co= nfig *cc, return r; } =20 +/* + * Bytes to process in one skcipher request: a whole contiguous segment wh= en + * batching (multi-data-unit), else one sector. 0 means an unusable + * (sub-sector / misaligned) segment. + */ +static unsigned int crypt_skcipher_len(struct crypt_config *cc, + const struct bio_vec *bv_in, + const struct bio_vec *bv_out) +{ + const unsigned int sector_size =3D cc->sector_size; + + if (test_bit(CRYPT_MULTI_DATA_UNIT, &cc->cipher_flags)) + return round_down(min(bv_in->bv_len, bv_out->bv_len), + sector_size); + + /* Reject unexpected unaligned bio. */ + if (unlikely(bv_in->bv_len & (sector_size - 1))) + return 0; + return sector_size; +} + +/* + * Encrypt/decrypt one bio segment (one sector, or a whole segment when + * batching) and report the bytes done in *out_processed. The integrity / + * preprocess / post handling is inert when batching (crypt_can_batch_unit= s() + * excludes those configs). + */ static int crypt_convert_block_skcipher(struct crypt_config *cc, struct convert_context *ctx, struct skcipher_request *req, - unsigned int tag_offset) + unsigned int tag_offset, + unsigned int *out_processed) { struct bio_vec bv_in =3D bio_iter_iovec(ctx->bio_in, ctx->iter_in); struct bio_vec bv_out =3D bio_iter_iovec(ctx->bio_out, ctx->iter_out); + const unsigned int sector_size =3D cc->sector_size; struct scatterlist *sg_in, *sg_out; struct dm_crypt_request *dmreq; u8 *iv, *org_iv, *tag_iv; __le64 *sector; + unsigned int len; int r =3D 0; =20 - /* Reject unexpected unaligned bio. */ - if (unlikely(bv_in.bv_len & (cc->sector_size - 1))) + len =3D crypt_skcipher_len(cc, &bv_in, &bv_out); + if (unlikely(!len)) return -EIO; =20 dmreq =3D dmreq_of_req(cc, req); @@ -1386,10 +1434,10 @@ static int crypt_convert_block_skcipher(struct cryp= t_config *cc, sg_out =3D &dmreq->sg_out[0]; =20 sg_init_table(sg_in, 1); - sg_set_page(sg_in, bv_in.bv_page, cc->sector_size, bv_in.bv_offset); + sg_set_page(sg_in, bv_in.bv_page, len, bv_in.bv_offset); =20 sg_init_table(sg_out, 1); - sg_set_page(sg_out, bv_out.bv_page, cc->sector_size, bv_out.bv_offset); + sg_set_page(sg_out, bv_out.bv_page, len, bv_out.bv_offset); =20 if (cc->iv_gen_ops) { /* For READs use IV stored in integrity metadata */ @@ -1410,7 +1458,9 @@ static int crypt_convert_block_skcipher(struct crypt_= config *cc, memcpy(iv, org_iv, cc->iv_size); } =20 - skcipher_request_set_crypt(req, sg_in, sg_out, cc->sector_size, iv); + skcipher_request_set_crypt(req, sg_in, sg_out, len, iv); + if (test_bit(CRYPT_MULTI_DATA_UNIT, &cc->cipher_flags)) + skcipher_request_set_unit_size(req, sector_size); =20 if (bio_data_dir(ctx->bio_in) =3D=3D WRITE) r =3D crypto_skcipher_encrypt(req); @@ -1420,9 +1470,10 @@ static int crypt_convert_block_skcipher(struct crypt= _config *cc, if (!r && cc->iv_gen_ops && cc->iv_gen_ops->post) cc->iv_gen_ops->post(cc, org_iv, dmreq); =20 - bio_advance_iter(ctx->bio_in, &ctx->iter_in, cc->sector_size); - bio_advance_iter(ctx->bio_out, &ctx->iter_out, cc->sector_size); + bio_advance_iter(ctx->bio_in, &ctx->iter_in, len); + bio_advance_iter(ctx->bio_out, &ctx->iter_out, len); =20 + *out_processed =3D len; return r; } =20 @@ -1509,13 +1560,25 @@ static void crypt_free_req(struct crypt_config *cc,= void *req, struct bio *base_ crypt_free_req_skcipher(cc, req, base_bio); } =20 +/* + * Advance the IV-sector and integrity-tag cursors by @processed bytes; the + * bio iterators are advanced by the per-block helpers themselves. + */ +static void crypt_convert_advance(struct crypt_config *cc, + struct convert_context *ctx, + unsigned int processed) +{ + ctx->cc_sector +=3D processed >> SECTOR_SHIFT; + ctx->tag_offset +=3D processed / cc->sector_size; +} + /* * Encrypt / decrypt data from one bio to another one (can be the same one) */ static blk_status_t crypt_convert(struct crypt_config *cc, struct convert_context *ctx, bool atomic, bool reset_pending) { - unsigned int sector_step =3D cc->sector_size >> SECTOR_SHIFT; + unsigned int processed; int r; =20 /* @@ -1536,10 +1599,12 @@ static blk_status_t crypt_convert(struct crypt_conf= ig *cc, =20 atomic_inc(&ctx->cc_pending); =20 + processed =3D cc->sector_size; if (crypt_integrity_aead(cc)) r =3D crypt_convert_block_aead(cc, ctx, ctx->r.req_aead, ctx->tag_offse= t); else - r =3D crypt_convert_block_skcipher(cc, ctx, ctx->r.req, ctx->tag_offset= ); + r =3D crypt_convert_block_skcipher(cc, ctx, ctx->r.req, + ctx->tag_offset, &processed); =20 switch (r) { /* @@ -1559,8 +1624,7 @@ static blk_status_t crypt_convert(struct crypt_config= *cc, * exit and continue processing in a workqueue */ ctx->r.req =3D NULL; - ctx->tag_offset++; - ctx->cc_sector +=3D sector_step; + crypt_convert_advance(cc, ctx, processed); return BLK_STS_DEV_RESOURCE; } } else { @@ -1574,16 +1638,14 @@ static blk_status_t crypt_convert(struct crypt_conf= ig *cc, */ case -EINPROGRESS: ctx->r.req =3D NULL; - ctx->tag_offset++; - ctx->cc_sector +=3D sector_step; + crypt_convert_advance(cc, ctx, processed); continue; /* * The request was already processed (synchronously). */ case 0: atomic_dec(&ctx->cc_pending); - ctx->cc_sector +=3D sector_step; - ctx->tag_offset++; + crypt_convert_advance(cc, ctx, processed); if (!atomic) cond_resched(); continue; @@ -2345,12 +2407,28 @@ static int crypt_alloc_tfms_aead(struct crypt_confi= g *cc, char *ciphermode) return 0; } =20 +/* + * Whether multi-unit batching applies: a counter IV mode (unit_counter se= t), + * single-tfm, non-aead, and a per-unit IV step of exactly one (512B secto= rs + * or iv_large_sectors). The IV must also satisfy the API split's counter + * constraints (non-zero multiple of 8, <=3D 32 bytes). Integrity is excl= uded + * in crypt_ctr_cipher(), which runs after integrity is configured. + */ +static bool crypt_can_batch_units(struct crypt_config *cc) +{ + return !crypt_integrity_aead(cc) && cc->tfms_count =3D=3D 1 && + cc->iv_gen_ops && cc->iv_gen_ops->unit_counter && + cc->iv_size && IS_ALIGNED(cc->iv_size, sizeof(__le64)) && + cc->iv_size <=3D 32 && + (cc->sector_size =3D=3D (1 << SECTOR_SHIFT) || + test_bit(CRYPT_IV_LARGE_SECTORS, &cc->cipher_flags)); +} + static int crypt_alloc_tfms(struct crypt_config *cc, char *ciphermode) { if (crypt_integrity_aead(cc)) return crypt_alloc_tfms_aead(cc, ciphermode); - else - return crypt_alloc_tfms_skcipher(cc, ciphermode); + return crypt_alloc_tfms_skcipher(cc, ciphermode); } =20 static unsigned int crypt_subkey_size(struct crypt_config *cc) @@ -2999,7 +3077,6 @@ static int crypt_ctr_cipher_old(struct dm_target *ti,= char *cipher_in, char *key goto bad_mem; } =20 - /* Allocate cipher */ ret =3D crypt_alloc_tfms(cc, cipher_api); if (ret < 0) { ti->error =3D "Error allocating crypto tfm"; @@ -3063,6 +3140,21 @@ static int crypt_ctr_cipher(struct dm_target *ti, ch= ar *cipher_in, char *key) } } =20 + /* + * Enable multi-unit batching for an eligible config with no integrity + * (integrity is set up after cipher alloc, hence the re-check here). + * The API layer's transparent split is synchronous, so an async cipher + * batches only if it handles multi-unit requests natively. + */ + if (crypt_can_batch_units(cc) && !cc->integrity_tag_size && + !cc->integrity_iv_size && + (crypto_skcipher_alg(any_tfm(cc))->co.base.cra_flags & + (CRYPTO_ALG_ASYNC | CRYPTO_ALG_REQ_SEG)) !=3D CRYPTO_ALG_ASYNC) { + set_bit(CRYPT_MULTI_DATA_UNIT, &cc->cipher_flags); + DMINFO("Using multi-data-unit crypto offload (du=3D%u)", + cc->sector_size); + } + /* wipe the kernel key payload copy */ if (cc->key_string) memset(cc->key, 0, cc->key_size * sizeof(u8)); --=20 2.47.3