From nobody Fri Dec 19 02:49:54 2025 Received: from mail-ej1-f73.google.com (mail-ej1-f73.google.com [209.85.218.73]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C762F3370EA for ; Fri, 31 Oct 2025 10:39:48 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.218.73 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1761907190; cv=none; b=Gs+IkUSK4RBGxApD8g22METaxI084SHGU4pRRME1kSEXfN2HVkSlC767fawT3mpjNV8dMEPBu4eTON48aG9L2ibJyxxFMUjarLadkZDDfLX77d+sfF2/jmikU/TMKSpeJdxuMgU1rkW30/sAft5YKyxi4v2EwNxAfL6DfrsIMH8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1761907190; c=relaxed/simple; bh=G5RRBNLWLHzW2CGf5F/p+HsgvhPSQr6XbmCWT5GCu/g=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=t1KmEB5cgV/Zkk6+LxR6TyFLbkL5Q7rFZQoyINXnx/8djYLk5s7JfVySaIwqHtrISZgf3AaNLYjFKa/LjWdqg30vp+BrFZ4sxcVBLeIit8PTpgFnGNMAY1e5Xu8nq6fLRl7xO1/S8pbmPoNgaF7kNeXMNjUeX9Rct0nRgM2EBJA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=DfD94dFX; arc=none smtp.client-ip=209.85.218.73 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="DfD94dFX" Received: by mail-ej1-f73.google.com with SMTP id a640c23a62f3a-b6d752b2891so186787666b.3 for ; Fri, 31 Oct 2025 03:39:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1761907187; x=1762511987; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=YcJUAK6wYbUnIItcXuhAD3+TuliFkqmiKq4znCgSGF0=; b=DfD94dFXFgvxIuMzP+eKj/PMnThYexq88yovWKJDJAYqf+itbPKGbglgm3UKBe4Ae1 hR/dhe9meXSKU1FESPfQDAVpwlCxPz+hf5cV8ilQAE56aMoqGK2BJ4f4poLOKuVxD0Zg isLbfwwv+anpjOtpJvoe0tsQ8Mo76WOdz0NfVXMItV132I3gr+mgP5sLO5G1UEcj2dWm iJcM/D2uz5EpI3sAZ1V0khwO8TUurOTeA+KlZVrKu3HJ5X/ZXwCFp2PsuSqFH3pgkwfS pv3hi9w7zyJvAJZoNgmSPjWA8tbQke9qwRTEtSucOmykdkoP8thPQlCtmfXQ/dhDe5Fr 4pWQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1761907187; x=1762511987; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=YcJUAK6wYbUnIItcXuhAD3+TuliFkqmiKq4znCgSGF0=; b=Y5xlP2LAp177GWecTCTQb4KgMExT1XnjbR8cLv/7KWcy+lWq6KwcYGpE6DKKYLDGjj U9FZeDo0elpTtrODvTzkAx6cfHJ/iUI5C1OhT13qBqmU0wEj1J8OpdTBS9iEc53ZLbdv LBU1Hqo709tsiE9XWdmvTwJuJQYVnqMoPw59Q7UTPzlaxSrvhhZl/jNXoowTkfLVz1x8 U9Rtwc/fvdhSmtDCFjfZWKQqKFC5f5hb5YIc3049lKE+8LLcWGxg5gEjrCTyX9ny5RJl zrEPZdT1StacVqdbQPaqGAduOM/HESrgxdfiIxuLejCNRLiy3ZUaUSRYzTcuGLmNsIQO sjzA== X-Gm-Message-State: AOJu0YyIkyb9YACg268CES6s4dzpUmz3LCE/tGzOpdZFDdl0rx0yGVdX O+Et1OPI1wGqzGMxds3LBB1uLT4/JwY0NkUbVpmoTy37xrBgmOh9Xv4XAWAnuzL9E991EFDifw= = X-Google-Smtp-Source: AGHT+IGZh2XVfTLLj0zhjt1BICvTQf4H+YXy15GPZEDU6T61fPt/5oBnkxDlk7NOR3Q1+6PHRVp+5rAW X-Received: from ejdao14.prod.google.com ([2002:a17:907:f48e:b0:b6d:5f6f:f88d]) (user=ardb job=prod-delivery.src-stubby-dispatcher) by 2002:a17:906:4fd3:b0:b2d:d7ba:8e7b with SMTP id a640c23a62f3a-b70701b183cmr329375466b.23.1761907187206; Fri, 31 Oct 2025 03:39:47 -0700 (PDT) Date: Fri, 31 Oct 2025 11:39:00 +0100 In-Reply-To: <20251031103858.529530-23-ardb+git@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20251031103858.529530-23-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=926; i=ardb@kernel.org; h=from:subject; bh=dO4a5jlY66z9w+gPMMFWeTG7Ku3rgSUkr2hPJWOMIU0=; b=owGbwMvMwCVmkMcZplerG8N4Wi2JIZNl4pHD/75JiC/9K2vH/lbtRfh3+QvnpDrMDjge27JVP HOJ/5G3HaUsDGJcDLJiiiwCs/++23l6olSt8yxZmDmsTCBDGLg4BWAihQEM/zSie2ONrfwLXk3+ t1VdNqr2RUzhcYsZcfEmO5qeTnM7/IXhn8H7NtmnN3p+XXt/sq5gYUf1EqtZvT3e2uK8kaf+XVX dzwAA X-Mailer: git-send-email 2.51.1.930.gacf6e81ea2-goog Message-ID: <20251031103858.529530-24-ardb+git@google.com> Subject: [PATCH v4 01/21] crypto/arm64: aes-ce-ccm - Avoid pointless yield of the NEON unit From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: linux-kernel@vger.kernel.org, linux-crypto@vger.kernel.org, herbert@gondor.apana.org.au, ebiggers@kernel.org, Ard Biesheuvel Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Ard Biesheuvel Kernel mode NEON sections are now preemptible on arm64, and so there is no need to yield it explicitly in order to prevent scheduling latency spikes. Reviewed-by: Eric Biggers Signed-off-by: Ard Biesheuvel Acked-by: Catalin Marinas Acked-by: Herbert Xu --- arch/arm64/crypto/aes-ce-ccm-glue.c | 5 +---- 1 file changed, 1 insertion(+), 4 deletions(-) diff --git a/arch/arm64/crypto/aes-ce-ccm-glue.c b/arch/arm64/crypto/aes-ce= -ccm-glue.c index 2d791d51891b..2eb4e76cabc3 100644 --- a/arch/arm64/crypto/aes-ce-ccm-glue.c +++ b/arch/arm64/crypto/aes-ce-ccm-glue.c @@ -114,11 +114,8 @@ static u32 ce_aes_ccm_auth_data(u8 mac[], u8 const in[= ], u32 abytes, in +=3D adv; abytes -=3D adv; =20 - if (unlikely(rem)) { - kernel_neon_end(); - kernel_neon_begin(); + if (unlikely(rem)) macp =3D 0; - } } else { u32 l =3D min(AES_BLOCK_SIZE - macp, abytes); =20 --=20 2.51.1.930.gacf6e81ea2-goog From nobody Fri Dec 19 02:49:54 2025 Received: from mail-wm1-f74.google.com (mail-wm1-f74.google.com [209.85.128.74]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 047C433890A for ; Fri, 31 Oct 2025 10:39:49 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.74 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1761907191; cv=none; b=kBEV3IH57lasTdwk6Mh9GQuOsgBjr5Rs0RY3y0s+ZNzBKW4CvvVWpVlbi8n7vplqiqvMqOXNftoFNCVKlimwHz3W1LaXBr3RK/mvkT+6bBlCxoLdIbvsKcPmdKv6ZlVdY+cGuLbDtOaM8vWfrNVdNKmTxwzQEgksO2T59d0mGj4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1761907191; c=relaxed/simple; bh=YgxDh8WWdIzSNRNYvsvKvpMfpn793r2yiLb1h9a45as=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=iJPCNYh+Qj0TXFnJ3KeZjzeZ9wXEdvJlw61lkz2Q5E/gDv0eoUulyPr4/bRS8qXw6s+K7uvIh9POcIbzMCZAFYEcARDD3kejrcx2d/zUsVfT8QyxLyYYdL9c61pzRk11stoo6K7xJs7DR7GHbXWx/Ft/ZiBsv4gq/aeyB9tqjIQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=vcX0loCr; arc=none smtp.client-ip=209.85.128.74 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="vcX0loCr" Received: by mail-wm1-f74.google.com with SMTP id 5b1f17b1804b1-475de1afec6so9779945e9.1 for ; Fri, 31 Oct 2025 03:39:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1761907188; x=1762511988; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=HN/+IzS5796w6R+9DpNyrnH5kTzGCEYUE8OX1y/TT0c=; b=vcX0loCrubb98yyHl/WY0Qc4yplD0zlAWkivbi2UVP0Nm3IcJN7itPNpwkXSurdTJi 1D0alpfN2qkKKINGWF+iiX9Ajp1mVUXb2yZolkHGCw9x6XEQXrC8KPdOKIJrUMnDeGK+ Ft+uVRsxGicrdB4fQHSGu6IOZCLCZbd4QB7tPopuec9FeiPpoHVxkmT0248yeWsWfNJF B1+8h1bLmsg9WihGRw10tDhJ2r1wxgf5D7XAdUw2yj6++pnnw4fbfkw56KVVZzGNHY65 YodUy/+oXPS97lqCu64RZs6MP2wuh2lu6+wxc2mzDKbXgeWw6aDgMgdbbDcgYWV0BLaH P1kQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1761907188; x=1762511988; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=HN/+IzS5796w6R+9DpNyrnH5kTzGCEYUE8OX1y/TT0c=; b=FhxF3PSUVe4QwzRHb0579XXTZNqo4uk0NqPbeu7vgiuqn/7obWlWroTRasIsUvKb4g 0NmJqM26jOSubzygSBK0Kl55PALcSeCy5TRVGuFw40yE+kogzrKw6wnpJkfH6yeh7ynK K9pQj/gFS67SpU7VJgVT3YEqLLIw5MXee2fi3YmsCzIrxECl2pwT84ic7KQEnp2Iu3zt fUrM0Jkbk+NCVvBVyUv2yDJTUzFDSHrUp/JOsTVZSHE/Zw6Si50RgZLPLmkdIyXvhp5Z M/tJ8xid5VrUYjKPLHaV2K9Xe+IjKbqCx0VhRAzaFXU+XGPIfmDgQw6MaLdJRO7Lb2Dr b7Og== X-Gm-Message-State: AOJu0Yx9gk1H2Eh0+zzW8ogOxyvMJk8uytjSRfIcR3vwuN73evW2YCci o+nUeJmUQR4/R5FpV8+EyhUAed1Lf2aPU2SEbs+x5jn8EFP1+LWcRXzFnBDQZzbNRPOjfTMgkQ= = X-Google-Smtp-Source: AGHT+IHlmFXxc1C8TcZ+HXf8DCZ7hZ6snCdGJ0O89HH9mn+babxzm78NbK82txSCG6oBizn12pHWQ7nS X-Received: from wma8.prod.google.com ([2002:a05:600c:8908:b0:46e:4c7c:5162]) (user=ardb job=prod-delivery.src-stubby-dispatcher) by 2002:a7b:ca54:0:b0:475:dc32:5600 with SMTP id 5b1f17b1804b1-477262e8fd6mr49406465e9.19.1761907188374; Fri, 31 Oct 2025 03:39:48 -0700 (PDT) Date: Fri, 31 Oct 2025 11:39:01 +0100 In-Reply-To: <20251031103858.529530-23-ardb+git@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20251031103858.529530-23-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=1825; i=ardb@kernel.org; h=from:subject; bh=yjdixCt9NaDfY8u4Fm681ZNm+FKMZ22mvY7LIiuXs68=; b=owGbwMvMwCVmkMcZplerG8N4Wi2JIZNl4rFLr9lSCrOW7tixbpb2jXSBA2sFQ/zV59hqMJ4UW BvJV8HVUcrCIMbFICumyCIw+++7nacnStU6z5KFmcPKBDKEgYtTACYy7wHD/+A1+gVed6+eSP31 oeHEnZUKayoiHTn2rFzct9j7W+9xWxeGP3y15UmKR/Odrh3eEr7Iwj51X+TpI87ad6Rlf32r4mF fwQEA X-Mailer: git-send-email 2.51.1.930.gacf6e81ea2-goog Message-ID: <20251031103858.529530-25-ardb+git@google.com> Subject: [PATCH v4 02/21] crypto/arm64: sm4-ce-ccm - Avoid pointless yield of the NEON unit From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: linux-kernel@vger.kernel.org, linux-crypto@vger.kernel.org, herbert@gondor.apana.org.au, ebiggers@kernel.org, Ard Biesheuvel Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Ard Biesheuvel Kernel mode NEON sections are now preemptible on arm64, and so there is no need to yield it when calling APIs that may sleep. Also, move the calls to kernel_neon_end() to the same scope as kernel_neon_begin(). This is needed for a subsequent change where a stack buffer is allocated transparently and passed to kernel_neon_begin(). Reviewed-by: Eric Biggers Signed-off-by: Ard Biesheuvel Acked-by: Catalin Marinas Acked-by: Herbert Xu --- arch/arm64/crypto/sm4-ce-ccm-glue.c | 25 +++++--------------- 1 file changed, 6 insertions(+), 19 deletions(-) diff --git a/arch/arm64/crypto/sm4-ce-ccm-glue.c b/arch/arm64/crypto/sm4-ce= -ccm-glue.c index e9cc1c1364ec..e92cbdf1aaee 100644 --- a/arch/arm64/crypto/sm4-ce-ccm-glue.c +++ b/arch/arm64/crypto/sm4-ce-ccm-glue.c @@ -172,35 +172,22 @@ static int ccm_crypt(struct aead_request *req, struct= skcipher_walk *walk, if (req->assoclen) ccm_calculate_auth_mac(req, mac); =20 - while (walk->nbytes && walk->nbytes !=3D walk->total) { + while (walk->nbytes) { unsigned int tail =3D walk->nbytes % SM4_BLOCK_SIZE; =20 + if (walk->nbytes =3D=3D walk->total) + tail =3D 0; + sm4_ce_ccm_crypt(rkey_enc, walk->dst.virt.addr, walk->src.virt.addr, walk->iv, walk->nbytes - tail, mac); =20 - kernel_neon_end(); - err =3D skcipher_walk_done(walk, tail); - - kernel_neon_begin(); } =20 - if (walk->nbytes) { - sm4_ce_ccm_crypt(rkey_enc, walk->dst.virt.addr, - walk->src.virt.addr, walk->iv, - walk->nbytes, mac); - - sm4_ce_ccm_final(rkey_enc, ctr0, mac); + sm4_ce_ccm_final(rkey_enc, ctr0, mac); =20 - kernel_neon_end(); - - err =3D skcipher_walk_done(walk, 0); - } else { - sm4_ce_ccm_final(rkey_enc, ctr0, mac); - - kernel_neon_end(); - } + kernel_neon_end(); =20 return err; } --=20 2.51.1.930.gacf6e81ea2-goog From nobody Fri Dec 19 02:49:54 2025 Received: from mail-ej1-f73.google.com (mail-ej1-f73.google.com [209.85.218.73]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id AFB1322B8B6 for ; Fri, 31 Oct 2025 10:39:50 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.218.73 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1761907192; cv=none; b=fi+fTzMGDm0hSBsXJiZN+09H0V3YDyw/fCf+qQSP3ITF5Q/qCYTnF3rExgNhdSF676waPwIAzPXsfqP6vbhpRi3T9SxLUukBXspCo5TLurbZQ3jYMCaLh3JONRAbhnpAYaKaN4dnCvmx0+jQHBFabowWbQ2sfA425OFoyO9Te08= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1761907192; c=relaxed/simple; bh=BJMZI7b2vwwoob2+Y/C61N/aS3PcsU16KD7zSWRIv08=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=i0jEJJFmKuGvNraUj4kL5ftdbmOAzDc03fukFX67+6pDaevamBYGLIJSbot6AvlDBKEvGPChcxixrsTne4QActBFg36pjF/ea3MmBGhrUz5FNd5mz5PTWTwI70g9bmQsWXoM8boIjuHxcdgO1JRyM0tIah9RHxz9zlcMzA0qmOA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=zmJOF90J; arc=none smtp.client-ip=209.85.218.73 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="zmJOF90J" Received: by mail-ej1-f73.google.com with SMTP id a640c23a62f3a-b704635310aso198045866b.3 for ; Fri, 31 Oct 2025 03:39:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1761907189; x=1762511989; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=ZwX5OSigI0JdbJe/ghC6Nz1Hun3Gyl+7nBcMkOknwv4=; b=zmJOF90J+kw76CagWPYlBFYUmEIXeEdDoZwtr5NZT8gS3XkzXcXH17rcFsR1GRGKIz LJGZfGEVH6vjL0OeuOGsY/eaNlP1TJhTiFU4snMFVQL3HebGhLNaJ/9aHJbKLupuVsW9 Y7ksQ36QYxjL5ST8jaRaTujbeVLWBMDtXIcqhqZOZ0CckiSk8Jy9naU2ESl+4OGlBP5c T99OdxY5MDwQbGQb+W0QNt1+zfz5s/sNS5yw2pJV7WB0WJR30pXI7PO6ohFH90Vaiemk L4qb34g7EXyj0kfID+iM2KgFXz1DGkzTLR7QYzFtLTO8k/1oJzaCr0qi/Eg9VuIlXnJv lGVg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1761907189; x=1762511989; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=ZwX5OSigI0JdbJe/ghC6Nz1Hun3Gyl+7nBcMkOknwv4=; b=UTUOfb5YFuhZ9CmFW3f1Qchug4L1qgBUlTNlGC63MbEijIjIZHr+D0WWLMvaNwf2hD JBsEIymBBqVZkDeijpkB/Z47FaKu3Hp7gUtonAx73aMl96XWwGqkBCAsoprAnlqQoeIt i1J+aMWctqoHsUVKAAECqrv3Mx0v5LuCmbj721/MbKpvnJRxhMnScBNTVz2775IxPuI5 jqAVGwJvHfOhnj42TMXJ8maMIJAjVcSLb15OG+8X+60gO/bHy7eBopU9CsfV2iwEKEUC 0Dy6xloJryHw3okvEB++fw16651WdwXoq5aKNMjoAuG9UTir3KdMuAJftBAveYyUrktR 0kBg== X-Gm-Message-State: AOJu0Yxo+JGjPqkwLqjRkyUddJzYxzcaq8CCrwJP5V4XL0XFINyAK428 e6VsrJk2niWFmHsb5gTnRVQr5aF3zh5dOJZwe5KRlpl7UprSD1sHHTXEPZH/zhVdtBpB4RN+mg= = X-Google-Smtp-Source: AGHT+IG2w9Vi0qrSdd2t9CIYM3Jx8PEua5OuP5pZQSuAoghiP0ZDJ01QHeEVRbAeOEHa+k//Ah65fYrH X-Received: from ejbmd20.prod.google.com ([2002:a17:906:ae94:b0:b6d:5fc2:65ea]) (user=ardb job=prod-delivery.src-stubby-dispatcher) by 2002:a17:907:7241:b0:b40:e7ee:b5ec with SMTP id a640c23a62f3a-b7070845975mr319361266b.59.1761907189198; Fri, 31 Oct 2025 03:39:49 -0700 (PDT) Date: Fri, 31 Oct 2025 11:39:02 +0100 In-Reply-To: <20251031103858.529530-23-ardb+git@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20251031103858.529530-23-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=2015; i=ardb@kernel.org; h=from:subject; bh=HLMA/w9qPUHfG22juJFC8tVZVVYrXXnkL8gtOvgDLHg=; b=owGbwMvMwCVmkMcZplerG8N4Wi2JIZNl4vGA/peW+wtXBbVv7Dpsf1jj5lGGQqNs5jdprn94z b5wBnzsKGVhEONikBVTZBGY/ffdztMTpWqdZ8nCzGFlAhnCwMUpABPJnMnI8LL7+4pN6kEvdOb1 bjocqLR//fHEVdcE4yWCrAU2iy2Yp8zIsCdvqadSueE7XuVPQucX/n3y1+LMi9Bv9y+1n33y/2X KbEYA X-Mailer: git-send-email 2.51.1.930.gacf6e81ea2-goog Message-ID: <20251031103858.529530-26-ardb+git@google.com> Subject: [PATCH v4 03/21] crypto/arm64: sm4-ce-gcm - Avoid pointless yield of the NEON unit From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: linux-kernel@vger.kernel.org, linux-crypto@vger.kernel.org, herbert@gondor.apana.org.au, ebiggers@kernel.org, Ard Biesheuvel Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Ard Biesheuvel Kernel mode NEON sections are now preemptible on arm64, and so there is no need to yield it when calling APIs that may sleep. Also, move the calls to kernel_neon_end() to the same scope as kernel_neon_begin(). This is needed for a subsequent change where a stack buffer is allocated transparently and passed to kernel_neon_begin(). While at it, simplify the logic. Reviewed-by: Eric Biggers Signed-off-by: Ard Biesheuvel Acked-by: Catalin Marinas Acked-by: Herbert Xu --- arch/arm64/crypto/sm4-ce-gcm-glue.c | 25 +++++--------------- 1 file changed, 6 insertions(+), 19 deletions(-) diff --git a/arch/arm64/crypto/sm4-ce-gcm-glue.c b/arch/arm64/crypto/sm4-ce= -gcm-glue.c index c2ea3d5f690b..8f6fc8c33c3f 100644 --- a/arch/arm64/crypto/sm4-ce-gcm-glue.c +++ b/arch/arm64/crypto/sm4-ce-gcm-glue.c @@ -154,36 +154,23 @@ static int gcm_crypt(struct aead_request *req, struct= skcipher_walk *walk, if (req->assoclen) gcm_calculate_auth_mac(req, ghash); =20 - while (walk->nbytes) { + do { unsigned int tail =3D walk->nbytes % SM4_BLOCK_SIZE; const u8 *src =3D walk->src.virt.addr; u8 *dst =3D walk->dst.virt.addr; + const u8 *l =3D NULL; =20 if (walk->nbytes =3D=3D walk->total) { - sm4_ce_pmull_gcm_crypt(ctx->key.rkey_enc, dst, src, iv, - walk->nbytes, ghash, - ctx->ghash_table, - (const u8 *)&lengths); - - kernel_neon_end(); - - return skcipher_walk_done(walk, 0); + l =3D (const u8 *)&lengths; + tail =3D 0; } =20 sm4_ce_pmull_gcm_crypt(ctx->key.rkey_enc, dst, src, iv, walk->nbytes - tail, ghash, - ctx->ghash_table, NULL); - - kernel_neon_end(); + ctx->ghash_table, l); =20 err =3D skcipher_walk_done(walk, tail); - - kernel_neon_begin(); - } - - sm4_ce_pmull_gcm_crypt(ctx->key.rkey_enc, NULL, NULL, iv, - walk->nbytes, ghash, ctx->ghash_table, - (const u8 *)&lengths); + } while (walk->nbytes); =20 kernel_neon_end(); =20 --=20 2.51.1.930.gacf6e81ea2-goog From nobody Fri Dec 19 02:49:54 2025 Received: from mail-wm1-f73.google.com (mail-wm1-f73.google.com [209.85.128.73]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 98E8D33E37C for ; Fri, 31 Oct 2025 10:39:51 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.73 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1761907194; cv=none; b=ofOBfotkSQ8PfgcHrYazUJU5lU1FGAkAJDOh0Do1QTyrrJBRFoPqO1658DPdYdniHC/gjmDEkd7uya/nil/ijqNeFjPSO0M44PP3lnVVqH1TjZHTwXZxfgeqrIp6Me+P9wqmM6/d8pesFISg2MOMtBfFzyuZnZ+9g4gxMTYWjzM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1761907194; c=relaxed/simple; bh=1sGjwL/f15WqDxjZ6B3QjAhT4bc84EG6TafOD55kaSc=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=BG7TbyzWQVdArdsnQFGS4ebtMMTnqP/h9oNzb3TrEguAIzsLXN2U4ErYsTEuzmR030tCMeHIDnmczaVK4OyUx3cLkuBAvIRfcAeE/86jQGSUUAh/TvmjozxzEb9vxLIbaT+FLOwIvLf6ZLxkg/bsyzB+27rMpBwCXYP3nnxXnd4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=jtPJQ/nQ; arc=none smtp.client-ip=209.85.128.73 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="jtPJQ/nQ" Received: by mail-wm1-f73.google.com with SMTP id 5b1f17b1804b1-470fd92ad57so23151375e9.3 for ; Fri, 31 Oct 2025 03:39:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1761907190; x=1762511990; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=AFCvTIOsXSjYhMkRl4dIW/P5UlNb2kn2VBhE2xHZEzQ=; b=jtPJQ/nQJD9y0aHUBdFQAhDSCy+SNzqehIQsnTIETl/qJc8I02ZYNKIXJcDDY2PSN3 v77OgSFb85mfDhFWeCh+/rsRy7RahXmQiPvgCvAfVqHF1DhB9ObFbO3mNq5Ng0JD2u6p wus7zDva2pjRUNWswp/KZCQ6jTSSSjJV3428YT+3sQf6cZm5ZOy4ghbQI1pGEhQGwGLH PJjr/yxg4f89KSO+mwowef94iPqUoDX8OGqgpAhlNxQTsiWmME4NCvHrsAglMJwq/f/b OPRSfVnemUaz3yqrCqZJN9oc/P1zSHvmgOLIMVQdbtSO91z1XtJOnWD1X5dE9lHJhLPn eSNA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1761907190; x=1762511990; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=AFCvTIOsXSjYhMkRl4dIW/P5UlNb2kn2VBhE2xHZEzQ=; b=AXida2WYnAb3rPCB848Co+jx1TYi1m8R9T3SB4xZBmwIFrCOHchsIob/NAwRQ+9a52 fzHHzLsobMBq5nknUJYftMLATt7Ym2f5+qXGl2PUUFAMuRSxgGeRlOFkYnSNVXrJVPZP N5OGKe5MkCnbopqYbbW+urbOJFVBGK5vODGr/7h/uJkyaphigutoLRKX0W1fXYcPc/VP o6IZMqW4GtGTdgMdjgLrFerawygEEzU2+vYITs9jNGPRKRS7EI5e/L0oQqxZQobjDy6X nhhJVHQ4Yafza6qC1iRA6/h/48i4pmGHmmpAnE/CT0iliH7F1YqaI1Zc11zwzce5Vl1c jDow== X-Gm-Message-State: AOJu0Yxv2rVtAIb7qywNF78E/YglF3oQtfSOp7hw20grHYCoxhwgvqbo bnFNWnjB4BEldg5nsqpO633S+72NWt8w2vyOA0Aa3KcAFq979lz+GlKhE+JUf5l+LWPpdvob1Q= = X-Google-Smtp-Source: AGHT+IHWFYs4QMEYXdlkbsATPtP+/gZ1YWrkitgykutzLvMorl43xTdKzAxhRGPUNTRTOeWGSSSeXYsY X-Received: from wmcn22.prod.google.com ([2002:a05:600c:c0d6:b0:46e:1f26:9212]) (user=ardb job=prod-delivery.src-stubby-dispatcher) by 2002:a05:600c:c16a:b0:46f:b42e:e39e with SMTP id 5b1f17b1804b1-477308c60aamr29196925e9.39.1761907189938; Fri, 31 Oct 2025 03:39:49 -0700 (PDT) Date: Fri, 31 Oct 2025 11:39:03 +0100 In-Reply-To: <20251031103858.529530-23-ardb+git@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20251031103858.529530-23-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=1521; i=ardb@kernel.org; h=from:subject; bh=94iSyJlcyEo5KVonam61v2/twjbge67xOTFdxdGB8QY=; b=owGbwMvMwCVmkMcZplerG8N4Wi2JIZNl4kmfrSk5Qpcfzl42UcROjqnaPjK96+1NWR7t6X8Zu kS0Lh/qKGVhEONikBVTZBGY/ffdztMTpWqdZ8nCzGFlAhnCwMUpABPpTGH4K3/s/uZKlaPzZ+Tu 2nPn9NXVy6bU6yrM+VGz+seS3o+/WE0YGbYbSc2X5uHX85V6cL03YfU0mwtcZ7oWJ8kw/G1SCn6 6nRsA X-Mailer: git-send-email 2.51.1.930.gacf6e81ea2-goog Message-ID: <20251031103858.529530-27-ardb+git@google.com> Subject: [PATCH v4 04/21] arm64/simd: Add scoped guard API for kernel mode SIMD From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: linux-kernel@vger.kernel.org, linux-crypto@vger.kernel.org, herbert@gondor.apana.org.au, ebiggers@kernel.org, Ard Biesheuvel , Kees Cook Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Ard Biesheuvel Encapsulate kernel_neon_begin() and kernel_neon_end() using a 'ksimd' cleanup guard. This hides the prototype of those functions, allowing them to be changed for arm64 but not ARM, without breaking code that is shared between those architectures (RAID6, AEGIS-128) It probably makes sense to expose this API more widely across architectures, as it affords more flexibility to the arch code to plumb it in, while imposing more rigid rules regarding the start/end bookends appearing in matched pairs. Reviewed-by: Kees Cook Reviewed-by: Eric Biggers Signed-off-by: Ard Biesheuvel Acked-by: Catalin Marinas Reviewed-by: Jonathan Cameron --- arch/arm64/include/asm/simd.h | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/arch/arm64/include/asm/simd.h b/arch/arm64/include/asm/simd.h index 8e86c9e70e48..d9f83c478736 100644 --- a/arch/arm64/include/asm/simd.h +++ b/arch/arm64/include/asm/simd.h @@ -6,12 +6,15 @@ #ifndef __ASM_SIMD_H #define __ASM_SIMD_H =20 +#include #include #include #include #include #include =20 +#include + #ifdef CONFIG_KERNEL_MODE_NEON =20 /* @@ -40,4 +43,8 @@ static __must_check inline bool may_use_simd(void) { =20 #endif /* ! CONFIG_KERNEL_MODE_NEON */ =20 +DEFINE_LOCK_GUARD_0(ksimd, kernel_neon_begin(), kernel_neon_end()) + +#define scoped_ksimd() scoped_guard(ksimd) + #endif --=20 2.51.1.930.gacf6e81ea2-goog From nobody Fri Dec 19 02:49:54 2025 Received: from mail-wm1-f74.google.com (mail-wm1-f74.google.com [209.85.128.74]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 49267340D84 for ; Fri, 31 Oct 2025 10:39:52 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.74 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1761907194; cv=none; b=GTu1QMuqChN+3JmKsSpUORpB2f22k+APSMhbZeEUoDAqkIvV+J0oZOnRGr1Z93OXKh4OOQafiPzCZPHmswf1Tk3yMUwBC61SsyACAeq7dBGPRbh5VlMr0vKjyjknGhJ5sD5sI0dcDAQpg8G90GAqUKODOXo2RtJz+7xds3egnxg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1761907194; c=relaxed/simple; bh=yBb3/AVwhx7HPzbiZpLBbPTW06U/K5ubRBiRHWSOhDg=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=Jtsh9qcHw/5Uk5ezzKgOfEkZ3PfAqCoF7hQOYCRwR0bnCFT3dEL6qMJD7hmJSBqnixNn4jnlOWCikpupYPYfWNsKZ/npiQ2kRX/+gNxXxM6qolxX60qgV/CE+zerbKZoHVa+ByEicpeO+xiT4/5/YMNSKZW/+ptQwcVmCzEYaFk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=jHxr/7pH; arc=none smtp.client-ip=209.85.128.74 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="jHxr/7pH" Received: by mail-wm1-f74.google.com with SMTP id 5b1f17b1804b1-47106720618so18831795e9.1 for ; Fri, 31 Oct 2025 03:39:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1761907191; x=1762511991; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=h/RaERATIracn7UtSwS5tFdTv/dEoSriJN9vedmIsxs=; b=jHxr/7pHlC7deWj6triksIHLQDXKJGZanrf+/iWeE1IDx4bjMK+DgByFBMCfPV7WvD GLGFqa44M853IiArTaN47M1kOZKjLB0XcgEg5hpIpCPBO5Fq2mCG7HQv3+jYWZxsNzig tOn171UzjmSsEEwcvj55bIHZLg6BKKaa8hB6xlEB7YtWHtuzxxuTAyZ6Jk+kjQSBlXQK +2XPUE5UB4IMUBlFSN3XAB9PLFM5Mrveb9VhwovNDhZ43JsKjol/vqoDIIO4nixQVw96 zKf7Cppzk5GOEqtiy7t5td0Zv6acr/kxqRGmumHzll+FBgJyjccI/yM2CkeVIBYNAUID VDPA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1761907191; x=1762511991; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=h/RaERATIracn7UtSwS5tFdTv/dEoSriJN9vedmIsxs=; b=h+b/+6fQmvDYzLGf/l9Oo3qrYTGcccsasPSLn/tkF2kuLy+Z6tTqeENNV/6DkLamJy kessqGWwblygZ3zb7AipeLFiYIbR6Metf4jSk6bbKuZBoEr4x+5EvqLL0q67av4mLgQR d3sT3/ejK9u90OiAnHnUOnJgAeI+opl4QfzAN9t/AOC2HPVjdCO8PyaMhEOH6bWNqFfH bpBbCsPqJi9AZg87omVSzdm81n5pPj677a+9k373KuB57F1WWW7gXpO6W2DpyodK2dtg hzPzaUuX7KOGkCppRNVcQ0JqD1FCucjWZHjpHAgHa0ahdrwadwIFXXEHlIPgw+RkKb0g akaQ== X-Gm-Message-State: AOJu0YwB6nZdPxSmxnagW889IZ5K92lL+ukw8Agv8pYA+6yVY2RLP4yv g3kLqs58BYmCsA77UMnP1lV53Obiyrrugdi6YPCxBVogrg7HsHx5mKm5DropS9I08Ima+WgfQQ= = X-Google-Smtp-Source: AGHT+IEhDOZqTYQN6JCsrWCZR21DjrXOlbXVaNqUWcCfWzjZpQ00/Tqc6oq9nYRZhpYSEno0TxL2X/Vx X-Received: from wmwp6.prod.google.com ([2002:a05:600d:8306:b0:477:9e6:88bf]) (user=ardb job=prod-delivery.src-stubby-dispatcher) by 2002:a05:600c:3146:b0:46f:c55a:5a8d with SMTP id 5b1f17b1804b1-47730802df6mr26621535e9.4.1761907190616; Fri, 31 Oct 2025 03:39:50 -0700 (PDT) Date: Fri, 31 Oct 2025 11:39:04 +0100 In-Reply-To: <20251031103858.529530-23-ardb+git@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20251031103858.529530-23-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=1073; i=ardb@kernel.org; h=from:subject; bh=M/7s7wcn0cdKhOhqASUBxonxhz5yy2SsKjo9N/ZpQ7w=; b=owGbwMvMwCVmkMcZplerG8N4Wi2JIZNl4qnyS8sPKb69IFY1+c0PRoPgI+G1QUYH3jPN7vrGE 6knvD2no5SFQYyLQVZMkUVg9t93O09PlKp1niULM4eVCWQIAxenAEykLJHhN0tBTKn6F6d2/Y0J FcET91VJ/rMuz/7h8zTBUu/D4nWvvzP8U13ef0Rt2iy/1D9zLa/cenzYpJVfaIUx26dve6cuSHw YxAQA X-Mailer: git-send-email 2.51.1.930.gacf6e81ea2-goog Message-ID: <20251031103858.529530-28-ardb+git@google.com> Subject: [PATCH v4 05/21] ARM/simd: Add scoped guard API for kernel mode SIMD From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: linux-kernel@vger.kernel.org, linux-crypto@vger.kernel.org, herbert@gondor.apana.org.au, ebiggers@kernel.org, Ard Biesheuvel , Kees Cook Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Ard Biesheuvel Implement the ksimd scoped guard API so that it can be used by code that supports both ARM and arm64. Reviewed-by: Kees Cook Reviewed-by: Eric Biggers Signed-off-by: Ard Biesheuvel Acked-by: Catalin Marinas --- arch/arm/include/asm/simd.h | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/arch/arm/include/asm/simd.h b/arch/arm/include/asm/simd.h index be08a8da046f..8549fa8b7253 100644 --- a/arch/arm/include/asm/simd.h +++ b/arch/arm/include/asm/simd.h @@ -2,14 +2,21 @@ #ifndef _ASM_SIMD_H #define _ASM_SIMD_H =20 +#include #include #include #include =20 +#include + static __must_check inline bool may_use_simd(void) { return IS_ENABLED(CONFIG_KERNEL_MODE_NEON) && !in_hardirq() && !irqs_disabled(); } =20 +DEFINE_LOCK_GUARD_0(ksimd, kernel_neon_begin(), kernel_neon_end()) + +#define scoped_ksimd() scoped_guard(ksimd) + #endif /* _ASM_SIMD_H */ --=20 2.51.1.930.gacf6e81ea2-goog From nobody Fri Dec 19 02:49:54 2025 Received: from mail-wr1-f74.google.com (mail-wr1-f74.google.com [209.85.221.74]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5165B342144 for ; Fri, 31 Oct 2025 10:39:53 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.221.74 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1761907195; cv=none; b=sVqB2NxaH6BkXMKGnM/jfp4/41gdFpK08xd+wCrYH5lssaNoFSEV5rKV6xxo4v2bQtlSXLSzCbPDL4vpP83VkwkRA/tYrpyY/Yx0ba9jBcoflew3UN4REGuVGHkrpx2r3lGghKHlG0BNMSUlcnQftzwV7Qqd2pxPzFBaYzzDrOA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1761907195; c=relaxed/simple; bh=vUXH7DcR4nX0nxx2PrHhmTkIgCH0oDmfEkfMRuToev4=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=QMoXEChpQ4zlGA1jkz1eM/+lqaJwgISiWYAVUtGBw0G+zKnG4h58VPuU+6ySjqihD+u3iTZnefPkBkAVAqIQ6f4zevrKr1FtTAJI32TNw8dNiE9c5m15nIvosBO9qMOtre3uHSrJvonDE6tb74qyK0nigccY2tD7jvboZHVjQ7M= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=dMc2SK5J; arc=none smtp.client-ip=209.85.221.74 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="dMc2SK5J" Received: by mail-wr1-f74.google.com with SMTP id ffacd0b85a97d-4298da9effcso2007652f8f.2 for ; Fri, 31 Oct 2025 03:39:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1761907192; x=1762511992; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=1zhuLLYlhoehauqY+PeyFZRU+wyZ+pjJLV9U014lhCY=; b=dMc2SK5JD4iYpHh9zw35kjvENtOANH+G1130Zj8dizM96rB6KyPmjj/FU6FMhpYpGs wvbhm9EpHomMbrRxHUSNnqCI+fImjIqi7EvTUPzceSK4WPNFTqPLtKCrGyXuUUwlsbha isjoNZsSVlDXehMyztsbQdD9i79mn9nplTI8BKRFBEH1pZnLrQLPCEKI4M9hcHyUeYeE h9Mdi7WLNPlQHO5Z6yVnIRQlMCXL34hoPit+3IulddAHZTBTCiDyEqyEb0+fTQsg/swg pEzAH5EpuZwXcT5u6qBdnzoh1P7rECdPnXndeM21h1+aQPIUVx6dEgWjl19DS/wXWrTt c5Cg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1761907192; x=1762511992; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=1zhuLLYlhoehauqY+PeyFZRU+wyZ+pjJLV9U014lhCY=; b=Af2FUDubl1yma6zXB4s5uF8ur8lKTQEIrWE12IZEjzzMyO591OkMx35f1zc0Rbepa4 p2LOXdx/zupkmysU56DWbjeG+Fq221nuS0dCJjbWo8NnNsYDI0DKH6QKet6CxC6et5GD Fo7EPYiDNu5we2Jwp1hSP3y21ZR26kuDfeffGlRRHTJsqM+sIW1xpLLiI6Zvdd0KKjM9 uEtmAANMtXKWegzn3lOV2jW0igsJ8tCJfyuMM8wV5KO/recw939y0KYo8BFmK1CuhHpM 3estF4dbgmsahjElAMb5Muy1LHZl/JvMhDjcmLAS/YES6HGARBxc6gtlILUtjzMbIe3q c53w== X-Gm-Message-State: AOJu0YwDzrYp9o8TU6y6aVeT+zgisoahRCAqOv0LsgMESrvmZqBKq3BV 39YOkxDOYgwq9oo2ItR06DID5SPlu8SISpyYOzmnYeHoEaskuVpeVQhqu46Jvl+NpKoWR6fmpA= = X-Google-Smtp-Source: AGHT+IG6mde3U+6HS4EOdxS/vl1RSdlu3ot1nl7+xIt1bPeQcqvsk9Gp/rEaM6G2Vof1hCgp+HjwgorY X-Received: from wmwd10.prod.google.com ([2002:a05:600d:62ea:b0:475:df4b:95a9]) (user=ardb job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6000:1aca:b0:428:3c66:a016 with SMTP id ffacd0b85a97d-429bd6ada02mr2121666f8f.30.1761907191819; Fri, 31 Oct 2025 03:39:51 -0700 (PDT) Date: Fri, 31 Oct 2025 11:39:05 +0100 In-Reply-To: <20251031103858.529530-23-ardb+git@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20251031103858.529530-23-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=2508; i=ardb@kernel.org; h=from:subject; bh=u5CQYJMAl0uvqTErnR5m9YSNYVrAtGuRx032f0bmEYY=; b=owGbwMvMwCVmkMcZplerG8N4Wi2JIZNl4pk+g01R4vvZplxhs3o/+6dfHc/NXfrTjWIU/s+tS lKyNlzdUcrCIMbFICumyCIw+++7nacnStU6z5KFmcPKBDKEgYtTACaioszwP+bq49duhVzc05/o 6NkGR2xf1dcgcuPS8osTiyUCo/03+zL8M2EzWHp57e95q5cvNfibOWFVobTthIJoz1VHZ+tu87J U4QYA X-Mailer: git-send-email 2.51.1.930.gacf6e81ea2-goog Message-ID: <20251031103858.529530-29-ardb+git@google.com> Subject: [PATCH v4 06/21] crypto: aegis128-neon - Move to more abstract 'ksimd' guard API From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: linux-kernel@vger.kernel.org, linux-crypto@vger.kernel.org, herbert@gondor.apana.org.au, ebiggers@kernel.org, Ard Biesheuvel Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Ard Biesheuvel Move away from calling kernel_neon_begin() and kernel_neon_end() directly, and instead, use the newly introduced scoped_ksimd() API. This permits arm64 to modify the kernel mode NEON API without affecting code that is shared between ARM and arm64. Reviewed-by: Eric Biggers Signed-off-by: Ard Biesheuvel Acked-by: Catalin Marinas --- crypto/aegis128-neon.c | 33 +++++++------------- 1 file changed, 12 insertions(+), 21 deletions(-) diff --git a/crypto/aegis128-neon.c b/crypto/aegis128-neon.c index 9ee50549e823..b41807e63bd3 100644 --- a/crypto/aegis128-neon.c +++ b/crypto/aegis128-neon.c @@ -4,7 +4,7 @@ */ =20 #include -#include +#include =20 #include "aegis.h" #include "aegis-neon.h" @@ -24,32 +24,28 @@ void crypto_aegis128_init_simd(struct aegis_state *stat= e, const union aegis_block *key, const u8 *iv) { - kernel_neon_begin(); - crypto_aegis128_init_neon(state, key, iv); - kernel_neon_end(); + scoped_ksimd() + crypto_aegis128_init_neon(state, key, iv); } =20 void crypto_aegis128_update_simd(struct aegis_state *state, const void *ms= g) { - kernel_neon_begin(); - crypto_aegis128_update_neon(state, msg); - kernel_neon_end(); + scoped_ksimd() + crypto_aegis128_update_neon(state, msg); } =20 void crypto_aegis128_encrypt_chunk_simd(struct aegis_state *state, u8 *dst, const u8 *src, unsigned int size) { - kernel_neon_begin(); - crypto_aegis128_encrypt_chunk_neon(state, dst, src, size); - kernel_neon_end(); + scoped_ksimd() + crypto_aegis128_encrypt_chunk_neon(state, dst, src, size); } =20 void crypto_aegis128_decrypt_chunk_simd(struct aegis_state *state, u8 *dst, const u8 *src, unsigned int size) { - kernel_neon_begin(); - crypto_aegis128_decrypt_chunk_neon(state, dst, src, size); - kernel_neon_end(); + scoped_ksimd() + crypto_aegis128_decrypt_chunk_neon(state, dst, src, size); } =20 int crypto_aegis128_final_simd(struct aegis_state *state, @@ -58,12 +54,7 @@ int crypto_aegis128_final_simd(struct aegis_state *state, unsigned int cryptlen, unsigned int authsize) { - int ret; - - kernel_neon_begin(); - ret =3D crypto_aegis128_final_neon(state, tag_xor, assoclen, cryptlen, - authsize); - kernel_neon_end(); - - return ret; + scoped_ksimd() + return crypto_aegis128_final_neon(state, tag_xor, assoclen, + cryptlen, authsize); } --=20 2.51.1.930.gacf6e81ea2-goog From nobody Fri Dec 19 02:49:54 2025 Received: from mail-wr1-f74.google.com (mail-wr1-f74.google.com [209.85.221.74]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 687EE3431F8 for ; Fri, 31 Oct 2025 10:39:54 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.221.74 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1761907197; cv=none; b=qidt/hjMxMov/u0dWr2ygVJ26OyPD6m/iGCDQ+SSaMX59hA/cJ45ALjo665XVh6FdP2McJH4RtPSpdjtTooJ2RZ1nZDsRKWKn2fLzYUvzJbS0TbPU26PfeDh6GQAUUn0/JvfoR2VgVuic85REx0mN8DCit3KmSB1MvTwXyhTXzo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1761907197; c=relaxed/simple; bh=WgYwOMvlr+zTqk1uI7p2SkyIbQb6yGgF7pl0TkawgU8=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=Yzp7EpZrA38beIYk/yhMCgIcQ5a1bdErcIUEHWnIXA6p5OKtHgR9ij9GyQsLZbDDOcR902p7n0eShBphEijrBgyJr40w+pJflZE/QoPxeqhljwr1zDjfdtOE9Ny2gfkKNRLIejR7RNdjWeUS4CksXsVBoXmbx8HwswgdeLKM7bk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=bxorp0jE; arc=none smtp.client-ip=209.85.221.74 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="bxorp0jE" Received: by mail-wr1-f74.google.com with SMTP id ffacd0b85a97d-427027a2095so1243226f8f.2 for ; Fri, 31 Oct 2025 03:39:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1761907193; x=1762511993; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=e4RjkuRbzhzq+DbHCUcTmZHT9d6t7fanSTph/M7FrwM=; b=bxorp0jEoyaSuqrX2Wt2htp89/+f/W5w5ZngpeHeTsh01pTANNzTMDogcf7B3yK00W JnvE94VuqTDnkWbSJeopKcKqISYlRhIBBBnwC3bmPHjeFn3ijc9/6vosWEMhE7ftllx1 DoJaK7cpTSedS+e2J6M7WZo82mKHGZMQSF8W5boONC2FTyELoWylX3DgKCvbv5S31nUg YPxfvusQmS0O8F1d/3i6I1TMeDTT3zVQhEGzE5zifySO4tVYR415R/SR0D81kZi/l4/j hrCpi5Cjzr7hQx4c0jdt8iKze/SrQFEQCT0MavMWwRolBSvGNi4wv6p4Nz6LeYLS7c5s c6Nw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1761907193; x=1762511993; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=e4RjkuRbzhzq+DbHCUcTmZHT9d6t7fanSTph/M7FrwM=; b=EFvA/89SsAHipW1qgNZbos5iDffsWwVohK7FI3d1FDnkFxKdWHG8P+iXC4c4gRhFsx SwAvA1cbreG2f/IbX2HO/t7Rmg0P8a2OMBLu1NpblF8PhcAPdPCTOxh6e9dl4X//L1dX IpzOpmK1kBqpoG5vlspYoZLCzCiBh3trgchYR7ZKAoYQ56OcOZ5PX/Tr0m+DV1+wfSJd arUNTKLyjRUYuTYxymFQ6DpYwKye0tLg7AP7I0Ypk+VqNLIrhfeRkpOTXRA4lXqUndHW EawJfUJxIOPaRJ+KH1gclnLzavhyAsMvhHUKHiTlVC2zBvgTlqWNPvICi0Eg1EXAVfq2 WZug== X-Gm-Message-State: AOJu0YzRhHg1xj+4ajjEH4U8U1afJTnb5e5sxgLJANgHeC5hdC7Mmp32 sreF85n6/Jf7WiNTMw67KtKdoF71JlyyQRZ0Bbk4BbSsHphOX3tvyDUl1xL8YskNMGo9/PrqXw= = X-Google-Smtp-Source: AGHT+IHVOvUG0Xu/U72ITBOpdefU2DBxMkwXD2QsdxXRJMDjdqSoWxzPqoXGu/4sUXEGYRcoh2Hw1odT X-Received: from wrbdz17.prod.google.com ([2002:a05:6000:e91:b0:429:8bd5:a979]) (user=ardb job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6000:4813:b0:429:89c2:d933 with SMTP id ffacd0b85a97d-429bd6a661bmr2703069f8f.36.1761907192812; Fri, 31 Oct 2025 03:39:52 -0700 (PDT) Date: Fri, 31 Oct 2025 11:39:06 +0100 In-Reply-To: <20251031103858.529530-23-ardb+git@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20251031103858.529530-23-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=3233; i=ardb@kernel.org; h=from:subject; bh=iSbN4ekVGltGzQmP5pLZ+bG2pN1WnvqNrOwI0kB4C8o=; b=kA0DAAoWMG4JVi59LVwByyZiAGkEkc3IqxaPccg6BBv5hnkGx/1bCaULY/XwigMLrW1nDGA+l Yh1BAAWCgAdFiEEEJv97rnLkRp9Q5odMG4JVi59LVwFAmkEkc0ACgkQMG4JVi59LVzG4QEAoaIi zu37MoBFQRoBZZ+yZD5/L9uJT55ZFas0kEqcoTgBANiP73S3Ww/wj+Epar2f+78g93ZAI/f4ta1 oEhX9xoUH X-Mailer: git-send-email 2.51.1.930.gacf6e81ea2-goog Message-ID: <20251031103858.529530-30-ardb+git@google.com> Subject: [PATCH v4 07/21] raid6: Move to more abstract 'ksimd' guard API From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: linux-kernel@vger.kernel.org, linux-crypto@vger.kernel.org, herbert@gondor.apana.org.au, ebiggers@kernel.org, Ard Biesheuvel Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Ard Biesheuvel Move away from calling kernel_neon_begin() and kernel_neon_end() directly, and instead, use the newly introduced scoped_ksimd() API. This permits arm64 to modify the kernel mode NEON API without affecting code that is shared between ARM and arm64. Reviewed-by: Eric Biggers Signed-off-by: Ard Biesheuvel Acked-by: Catalin Marinas --- lib/raid6/neon.c | 17 +++++++---------- lib/raid6/recov_neon.c | 15 ++++++--------- 2 files changed, 13 insertions(+), 19 deletions(-) diff --git a/lib/raid6/neon.c b/lib/raid6/neon.c index 0a2e76035ea9..6d9474ce6da9 100644 --- a/lib/raid6/neon.c +++ b/lib/raid6/neon.c @@ -8,10 +8,9 @@ #include =20 #ifdef __KERNEL__ -#include +#include #else -#define kernel_neon_begin() -#define kernel_neon_end() +#define scoped_ksimd() #define cpu_has_neon() (1) #endif =20 @@ -32,10 +31,9 @@ { \ void raid6_neon ## _n ## _gen_syndrome_real(int, \ unsigned long, void**); \ - kernel_neon_begin(); \ - raid6_neon ## _n ## _gen_syndrome_real(disks, \ + scoped_ksimd() \ + raid6_neon ## _n ## _gen_syndrome_real(disks, \ (unsigned long)bytes, ptrs); \ - kernel_neon_end(); \ } \ static void raid6_neon ## _n ## _xor_syndrome(int disks, \ int start, int stop, \ @@ -43,10 +41,9 @@ { \ void raid6_neon ## _n ## _xor_syndrome_real(int, \ int, int, unsigned long, void**); \ - kernel_neon_begin(); \ - raid6_neon ## _n ## _xor_syndrome_real(disks, \ - start, stop, (unsigned long)bytes, ptrs); \ - kernel_neon_end(); \ + scoped_ksimd() \ + raid6_neon ## _n ## _xor_syndrome_real(disks, \ + start, stop, (unsigned long)bytes, ptrs);\ } \ struct raid6_calls const raid6_neonx ## _n =3D { \ raid6_neon ## _n ## _gen_syndrome, \ diff --git a/lib/raid6/recov_neon.c b/lib/raid6/recov_neon.c index 70e1404c1512..9d99aeabd31a 100644 --- a/lib/raid6/recov_neon.c +++ b/lib/raid6/recov_neon.c @@ -7,11 +7,10 @@ #include =20 #ifdef __KERNEL__ -#include +#include #include "neon.h" #else -#define kernel_neon_begin() -#define kernel_neon_end() +#define scoped_ksimd() #define cpu_has_neon() (1) #endif =20 @@ -55,9 +54,8 @@ static void raid6_2data_recov_neon(int disks, size_t byte= s, int faila, qmul =3D raid6_vgfmul[raid6_gfinv[raid6_gfexp[faila] ^ raid6_gfexp[failb]]]; =20 - kernel_neon_begin(); - __raid6_2data_recov_neon(bytes, p, q, dp, dq, pbmul, qmul); - kernel_neon_end(); + scoped_ksimd() + __raid6_2data_recov_neon(bytes, p, q, dp, dq, pbmul, qmul); } =20 static void raid6_datap_recov_neon(int disks, size_t bytes, int faila, @@ -86,9 +84,8 @@ static void raid6_datap_recov_neon(int disks, size_t byte= s, int faila, /* Now, pick the proper data tables */ qmul =3D raid6_vgfmul[raid6_gfinv[raid6_gfexp[faila]]]; =20 - kernel_neon_begin(); - __raid6_datap_recov_neon(bytes, p, q, dq, qmul); - kernel_neon_end(); + scoped_ksimd() + __raid6_datap_recov_neon(bytes, p, q, dq, qmul); } =20 const struct raid6_recov_calls raid6_recov_neon =3D { --=20 2.51.1.930.gacf6e81ea2-goog From nobody Fri Dec 19 02:49:54 2025 Received: from mail-wm1-f73.google.com (mail-wm1-f73.google.com [209.85.128.73]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id AE44B344044 for ; Fri, 31 Oct 2025 10:39:55 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.73 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1761907197; cv=none; b=m+k0UsQIZO4XyvIDtN92eRWgsQvIYmhlWoWh4i3lPzSfHj+d7DesCjfes4MGpiowQmhH3aGmNUB1MOgS2b/tgRVV7MpMeNYg8vu2D7eVw3v2kjYAVPKhOgva0Qjnf8PgQK9ALwWzJcgQkweMe2f/cbLg2Z/UbQMKmKCRyXjMFsA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1761907197; c=relaxed/simple; bh=vcLYXmPPWTuVenr60nmlwXYSJ0xbKw2uaxTeUxzt5yA=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=o6FEyl4EqutopI5uKz2RW8rM4oKDZjxsRH6t8Zsd9DD9CUoU661vIxaUfL8d16/RZ5Ga+Oj7ZEssvCKEyJEpo+KEUIIG6nu2DqvDUI2RodmeMT8vEjCAw6Ld+nlsB5TGcx9jaoiiz3fybCFQCz8R8FNMoNW4fwYShL0Hwp0EVMI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=GRoYwdGT; arc=none smtp.client-ip=209.85.128.73 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="GRoYwdGT" Received: by mail-wm1-f73.google.com with SMTP id 5b1f17b1804b1-4710d174c31so17264005e9.0 for ; Fri, 31 Oct 2025 03:39:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1761907194; x=1762511994; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=2jLxU3doQ2B67TmmcpJW69nrs7h/yjzsvfGdFS0dWxA=; b=GRoYwdGTONhqnV9cDNcDlydHTdCHnO04OirmEKlnZIDt6jGjAZIuWiTEM/8khuRZrB ExgfNXHeIkq/oemWm1oX20V2z9KbdOx0u0Xrf78CmBQHO791qWmwgbwLiTGh3NzdkfUQ QKQmzjOjh9LwMaeAaGxhGy0jnuKg5LUKcj1XmoRYeZdzxtXQm2xNmd1RM0f6k9zRHKeZ 9fI5F/HPGZp/pqT0iW44ABlZe5mKAaPEX31kxEYA1psx9ApiCq+ahNYpQr9vBeAYDbEZ x90D+0lCHGFfkaX2POLEFEeqtPju9dBF15C4tk40F71zhTnS/v2qEENqTyHcx8FszTag +q3g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1761907194; x=1762511994; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=2jLxU3doQ2B67TmmcpJW69nrs7h/yjzsvfGdFS0dWxA=; b=tiFE6xe3hDOt/fWNbPkoR4TvBzxvmZu3kud7/CyPPkv+UZyLbMzavDzV8/iisldNQy N9Aurb7kbgAgRGvF6a2rSlk0Y9u7BY6vLZ4vlkLohMzYtxpxs8C9e6yopr1+Gmmwf3jq eU4jBkrux38gKzukQUVWOcb5t4fpeXgHuJzTEJqyote0PfIXmqjOCUwH9SPZgQBziQX4 aO2JZz5ZPshbdkgbTe6BnoGKWpkPu12wVn7YkUVeeSW4DqXZdDB3tpyXJnZWjrM//XPB 27E6C+4U2gpjzBoXPRSpvNxJZrXRYiDJAcPuvY9bRnFEZ38jUN18PXpEU8mwrUJDYIjn Q/Ng== X-Gm-Message-State: AOJu0Yw+GOhun1Am7yHAU1yYBZ0e4U+y88zk0+MufOB2+6NeZpAFdxyR abEIInaik7aTedZk3Yj/rwi1KkH6kDAaYmtQLGzTQZIEAznqT4NAkl7oB5zIA3QQRVNtb8673A= = X-Google-Smtp-Source: AGHT+IHzKlH+Xpl86AaLume6m7PB2ESbuE972ir2JDi9jrl4y4L+sgCVE3nBwQSicAm7jTR3l5P0IM/v X-Received: from wmga22.prod.google.com ([2002:a05:600c:2d56:b0:477:d9d:9b35]) (user=ardb job=prod-delivery.src-stubby-dispatcher) by 2002:a05:600c:6692:b0:45d:d1a3:ba6a with SMTP id 5b1f17b1804b1-477308a2814mr22900615e9.33.1761907193988; Fri, 31 Oct 2025 03:39:53 -0700 (PDT) Date: Fri, 31 Oct 2025 11:39:07 +0100 In-Reply-To: <20251031103858.529530-23-ardb+git@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20251031103858.529530-23-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=5823; i=ardb@kernel.org; h=from:subject; bh=Df6A5JOKcxjoUkwm445+Yocxyxyxk/Xv7orllYDUuOg=; b=owGbwMvMwCVmkMcZplerG8N4Wi2JIZNl4rmeezv/vbyRxyjA/UZKZO6jV6a7BH+Ws/Sobk+d9 lCNIWtNRykLgxgXg6yYIovA7L/vdp6eKFXrPEsWZg4rE8gQBi5OAZjIVQGG/74sD1zNeTrnckxU 1HlRfTPFeOZKtY2yAa+XTo1WfdHNEcbwP/+38E5zsUTHdy63dme/r7BV8et0s3usIb1mut1q5ZB GBgA= X-Mailer: git-send-email 2.51.1.930.gacf6e81ea2-goog Message-ID: <20251031103858.529530-31-ardb+git@google.com> Subject: [PATCH v4 08/21] lib/crc: Switch ARM and arm64 to 'ksimd' scoped guard API From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: linux-kernel@vger.kernel.org, linux-crypto@vger.kernel.org, herbert@gondor.apana.org.au, ebiggers@kernel.org, Ard Biesheuvel Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Ard Biesheuvel Before modifying the prototypes of kernel_neon_begin() and kernel_neon_end() to accommodate kernel mode FP/SIMD state buffers allocated on the stack, move arm64 to the new 'ksimd' scoped guard API, which encapsulates the calls to those functions. For symmetry, do the same for 32-bit ARM too. Reviewed-by: Eric Biggers Signed-off-by: Ard Biesheuvel Acked-by: Catalin Marinas --- lib/crc/arm/crc-t10dif.h | 16 +++++----------- lib/crc/arm/crc32.h | 11 ++++------- lib/crc/arm64/crc-t10dif.h | 16 +++++----------- lib/crc/arm64/crc32.h | 16 ++++++---------- 4 files changed, 20 insertions(+), 39 deletions(-) diff --git a/lib/crc/arm/crc-t10dif.h b/lib/crc/arm/crc-t10dif.h index 63441de5e3f1..aaeeab0defb5 100644 --- a/lib/crc/arm/crc-t10dif.h +++ b/lib/crc/arm/crc-t10dif.h @@ -5,7 +5,6 @@ * Copyright (C) 2016 Linaro Ltd */ =20 -#include #include =20 static __ro_after_init DEFINE_STATIC_KEY_FALSE(have_neon); @@ -20,21 +19,16 @@ asmlinkage void crc_t10dif_pmull8(u16 init_crc, const u= 8 *buf, size_t len, static inline u16 crc_t10dif_arch(u16 crc, const u8 *data, size_t length) { if (length >=3D CRC_T10DIF_PMULL_CHUNK_SIZE) { - if (static_branch_likely(&have_pmull)) { - if (likely(may_use_simd())) { - kernel_neon_begin(); - crc =3D crc_t10dif_pmull64(crc, data, length); - kernel_neon_end(); - return crc; - } + if (static_branch_likely(&have_pmull) && likely(may_use_simd())) { + scoped_ksimd() + return crc_t10dif_pmull64(crc, data, length); } else if (length > CRC_T10DIF_PMULL_CHUNK_SIZE && static_branch_likely(&have_neon) && likely(may_use_simd())) { u8 buf[16] __aligned(16); =20 - kernel_neon_begin(); - crc_t10dif_pmull8(crc, data, length, buf); - kernel_neon_end(); + scoped_ksimd() + crc_t10dif_pmull8(crc, data, length, buf); =20 return crc_t10dif_generic(0, buf, sizeof(buf)); } diff --git a/lib/crc/arm/crc32.h b/lib/crc/arm/crc32.h index 7b76f52f6907..f33de6b22cd4 100644 --- a/lib/crc/arm/crc32.h +++ b/lib/crc/arm/crc32.h @@ -8,7 +8,6 @@ #include =20 #include -#include #include =20 static __ro_after_init DEFINE_STATIC_KEY_FALSE(have_crc32); @@ -42,9 +41,8 @@ static inline u32 crc32_le_arch(u32 crc, const u8 *p, siz= e_t len) len -=3D n; } n =3D round_down(len, 16); - kernel_neon_begin(); - crc =3D crc32_pmull_le(p, n, crc); - kernel_neon_end(); + scoped_ksimd() + crc =3D crc32_pmull_le(p, n, crc); p +=3D n; len -=3D n; } @@ -71,9 +69,8 @@ static inline u32 crc32c_arch(u32 crc, const u8 *p, size_= t len) len -=3D n; } n =3D round_down(len, 16); - kernel_neon_begin(); - crc =3D crc32c_pmull_le(p, n, crc); - kernel_neon_end(); + scoped_ksimd() + crc =3D crc32c_pmull_le(p, n, crc); p +=3D n; len -=3D n; } diff --git a/lib/crc/arm64/crc-t10dif.h b/lib/crc/arm64/crc-t10dif.h index f88db2971805..0de03ab1aeab 100644 --- a/lib/crc/arm64/crc-t10dif.h +++ b/lib/crc/arm64/crc-t10dif.h @@ -7,7 +7,6 @@ =20 #include =20 -#include #include =20 static __ro_after_init DEFINE_STATIC_KEY_FALSE(have_asimd); @@ -22,21 +21,16 @@ asmlinkage u16 crc_t10dif_pmull_p64(u16 init_crc, const= u8 *buf, size_t len); static inline u16 crc_t10dif_arch(u16 crc, const u8 *data, size_t length) { if (length >=3D CRC_T10DIF_PMULL_CHUNK_SIZE) { - if (static_branch_likely(&have_pmull)) { - if (likely(may_use_simd())) { - kernel_neon_begin(); - crc =3D crc_t10dif_pmull_p64(crc, data, length); - kernel_neon_end(); - return crc; - } + if (static_branch_likely(&have_pmull) && likely(may_use_simd())) { + scoped_ksimd() + return crc_t10dif_pmull_p64(crc, data, length); } else if (length > CRC_T10DIF_PMULL_CHUNK_SIZE && static_branch_likely(&have_asimd) && likely(may_use_simd())) { u8 buf[16]; =20 - kernel_neon_begin(); - crc_t10dif_pmull_p8(crc, data, length, buf); - kernel_neon_end(); + scoped_ksimd() + crc_t10dif_pmull_p8(crc, data, length, buf); =20 return crc_t10dif_generic(0, buf, sizeof(buf)); } diff --git a/lib/crc/arm64/crc32.h b/lib/crc/arm64/crc32.h index 31e649cd40a2..1939a5dee477 100644 --- a/lib/crc/arm64/crc32.h +++ b/lib/crc/arm64/crc32.h @@ -2,7 +2,6 @@ =20 #include #include -#include #include =20 // The minimum input length to consider the 4-way interleaved code path @@ -23,9 +22,8 @@ static inline u32 crc32_le_arch(u32 crc, const u8 *p, siz= e_t len) =20 if (len >=3D min_len && cpu_have_named_feature(PMULL) && likely(may_use_simd())) { - kernel_neon_begin(); - crc =3D crc32_le_arm64_4way(crc, p, len); - kernel_neon_end(); + scoped_ksimd() + crc =3D crc32_le_arm64_4way(crc, p, len); =20 p +=3D round_down(len, 64); len %=3D 64; @@ -44,9 +42,8 @@ static inline u32 crc32c_arch(u32 crc, const u8 *p, size_= t len) =20 if (len >=3D min_len && cpu_have_named_feature(PMULL) && likely(may_use_simd())) { - kernel_neon_begin(); - crc =3D crc32c_le_arm64_4way(crc, p, len); - kernel_neon_end(); + scoped_ksimd() + crc =3D crc32c_le_arm64_4way(crc, p, len); =20 p +=3D round_down(len, 64); len %=3D 64; @@ -65,9 +62,8 @@ static inline u32 crc32_be_arch(u32 crc, const u8 *p, siz= e_t len) =20 if (len >=3D min_len && cpu_have_named_feature(PMULL) && likely(may_use_simd())) { - kernel_neon_begin(); - crc =3D crc32_be_arm64_4way(crc, p, len); - kernel_neon_end(); + scoped_ksimd() + crc =3D crc32_be_arm64_4way(crc, p, len); =20 p +=3D round_down(len, 64); len %=3D 64; --=20 2.51.1.930.gacf6e81ea2-goog From nobody Fri Dec 19 02:49:54 2025 Received: from mail-wr1-f74.google.com (mail-wr1-f74.google.com [209.85.221.74]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9078222B8B6 for ; Fri, 31 Oct 2025 10:39:56 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.221.74 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1761907199; cv=none; b=Blznqze7pNudQVxgLjxuAOITAldcfhKgMP1dzfx1vw4Cltpp964HqIfzJ3DmronkjG6NYVdQ8CCqK1ZP9/q5rCWqppVr8v+kmZjJejiuLKdDX7jGlRqVLnot9B2QksDxduwRAcWfGSAmJpH+CBCjHFEz4QRaFabfY79svoCaFkM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1761907199; c=relaxed/simple; bh=micg+uwnrglgHVal5EtYXqgPUjbxDWvBRDhUQ3SFgjA=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=WSrq0Niednu2W/9G6cYLLMWaqKTDkFtdgR4A84aTzs0TK/rCszImdTVcdLZI12Hn4hCcS2niAQ33km6rr09s8Sm36rqnAgIVNr3XvpHxWHuglBDd4FHGF2fWx7JOADGrd72m2AO8y3iIgGbQo8b0LbiTa5pYK70CZIetz38PRmQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=ZeYciOut; arc=none smtp.client-ip=209.85.221.74 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="ZeYciOut" Received: by mail-wr1-f74.google.com with SMTP id ffacd0b85a97d-4270a61ec48so2281116f8f.1 for ; Fri, 31 Oct 2025 03:39:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1761907195; x=1762511995; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=nlol8jlCrwU+dDQLpe8bM4Z+Y39WEsP00gTragHj4Rc=; b=ZeYciOut2+NKr9+n+xvSV14XcLOlxZEmxmHGAuDTQs3Tzx1I1UgsddPUG0W6GgdmMz AcXv50dr6uk8hGzQO0K1YNbzANTrPrLuIQ/7/aBDZwbvt1Ttp11tmgISgxZt0Rsglb6o QOHFboUhZHoAyBLutZz+KwAlRAtFUdi0tVircUlzcd+chJN6/xMhlJlugz9cxDYFXmTM 1mOQ6b0lWEqcCx3KYhYkiPxL1kNojJWJmmC3bzrZu49zB1XCsH14oGAmjdmgR92o84rJ hMYHk59UoSUHHVvEvpzpZsoPMAltkh3wJV/PsrBPNCLC/AAd0X6Ogl4rotoF3AYXVevT Q73w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1761907195; x=1762511995; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=nlol8jlCrwU+dDQLpe8bM4Z+Y39WEsP00gTragHj4Rc=; b=EsHNqhmWQDjhBGPQIL96/SeLAVxHWEXy0SHzpQs62ySAMUV6NKJrvfxZSapRh95HiJ i6Q/K2pQXMQZW18g8pQyV9Jp+lIW7RtIwvzZO74y1Ylk9SAhPwleLbZaYB91h2jdbPKC 8Ku8daAafgsugbmFKFxd4jRHOZavj9vitXUeYDkTJftAEvoSTqU7MKPr7xKRoOADDO87 WbA+bdl54f54K5cfpPpVSnAPeIBFncPqw260rNbm0mP9knTa5Il7fPbjOXG/ybfDNSY0 OmpfdCPwyncxkVDrWZ56wtW1v7OHoGhRIxpIGGi57/Bl+fjKaAqV1Dwg3LF+GnRAZdnG kjAw== X-Gm-Message-State: AOJu0YxwDOaLmJC/ED33LMJ2tekzIMOt+LpZIemjIAj+JJU2hkEbZuqQ JvpcLNjJxRCdop5ZiEMN9pk7E0RP/DjM4Jnp5QToww7Kjksth+hYyH+a+H0gmV6ew3+bzGM+NA= = X-Google-Smtp-Source: AGHT+IEQo+IgSEHdcMiYz77LsIcY8TwOhGlEenTve1eoVqnDIxVO+kbrC2+61YhJ3qT6ML4IQGiBjP00 X-Received: from wrme11.prod.google.com ([2002:adf:e38b:0:b0:3ec:e23c:988f]) (user=ardb job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6000:25c2:b0:429:b718:167c with SMTP id ffacd0b85a97d-429bd68847fmr2351214f8f.28.1761907194945; Fri, 31 Oct 2025 03:39:54 -0700 (PDT) Date: Fri, 31 Oct 2025 11:39:08 +0100 In-Reply-To: <20251031103858.529530-23-ardb+git@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20251031103858.529530-23-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=9584; i=ardb@kernel.org; h=from:subject; bh=NAEDE4qtKhFGVJRETAhS4vJcuLrplU7qZJjmEUjIxNE=; b=owGbwMvMwCVmkMcZplerG8N4Wi2JIZNl4oWTawUlGKbMPhxh+IFJa77kWuVTa6o+rtAO+rtTs e7AEoeGjlIWBjEuBlkxRRaB2X/f7Tw9UarWeZYszBxWJpAhDFycAjARPgmGf8YNBi18M3KZX10O 8i6cNtHgu9LSmtjyWB2/fdrqabI7TjD8lRJafc5n8ebYDs9JK0/mRgqtEGQ5c+DF2eU5fHfnNOw 3ZgYA X-Mailer: git-send-email 2.51.1.930.gacf6e81ea2-goog Message-ID: <20251031103858.529530-32-ardb+git@google.com> Subject: [PATCH v4 09/21] lib/crypto: Switch ARM and arm64 to 'ksimd' scoped guard API From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: linux-kernel@vger.kernel.org, linux-crypto@vger.kernel.org, herbert@gondor.apana.org.au, ebiggers@kernel.org, Ard Biesheuvel Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Ard Biesheuvel Before modifying the prototypes of kernel_neon_begin() and kernel_neon_end() to accommodate kernel mode FP/SIMD state buffers allocated on the stack, move arm64 to the new 'ksimd' scoped guard API, which encapsulates the calls to those functions. For symmetry, do the same for 32-bit ARM too. Reviewed-by: Eric Biggers Signed-off-by: Ard Biesheuvel Acked-by: Catalin Marinas --- lib/crypto/arm/chacha.h | 6 ++---- lib/crypto/arm/poly1305.h | 6 ++---- lib/crypto/arm/sha1.h | 13 ++++++------- lib/crypto/arm/sha256.h | 12 ++++++------ lib/crypto/arm/sha512.h | 5 ++--- lib/crypto/arm64/chacha.h | 11 ++++------- lib/crypto/arm64/poly1305.h | 6 ++---- lib/crypto/arm64/sha1.h | 7 +++---- lib/crypto/arm64/sha256.h | 19 ++++++++----------- lib/crypto/arm64/sha512.h | 8 ++++---- 10 files changed, 39 insertions(+), 54 deletions(-) diff --git a/lib/crypto/arm/chacha.h b/lib/crypto/arm/chacha.h index 0cae30f8ee5d..b27ba00b3b23 100644 --- a/lib/crypto/arm/chacha.h +++ b/lib/crypto/arm/chacha.h @@ -12,7 +12,6 @@ =20 #include #include -#include #include =20 asmlinkage void chacha_block_xor_neon(const struct chacha_state *state, @@ -87,9 +86,8 @@ static void chacha_crypt_arch(struct chacha_state *state,= u8 *dst, do { unsigned int todo =3D min_t(unsigned int, bytes, SZ_4K); =20 - kernel_neon_begin(); - chacha_doneon(state, dst, src, todo, nrounds); - kernel_neon_end(); + scoped_ksimd() + chacha_doneon(state, dst, src, todo, nrounds); =20 bytes -=3D todo; src +=3D todo; diff --git a/lib/crypto/arm/poly1305.h b/lib/crypto/arm/poly1305.h index 0021cf368307..0fe903d8de55 100644 --- a/lib/crypto/arm/poly1305.h +++ b/lib/crypto/arm/poly1305.h @@ -6,7 +6,6 @@ */ =20 #include -#include #include #include #include @@ -32,9 +31,8 @@ static void poly1305_blocks(struct poly1305_block_state *= state, const u8 *src, do { unsigned int todo =3D min_t(unsigned int, len, SZ_4K); =20 - kernel_neon_begin(); - poly1305_blocks_neon(state, src, todo, padbit); - kernel_neon_end(); + scoped_ksimd() + poly1305_blocks_neon(state, src, todo, padbit); =20 len -=3D todo; src +=3D todo; diff --git a/lib/crypto/arm/sha1.h b/lib/crypto/arm/sha1.h index 29f8bcad0447..3e2d8c7cab9f 100644 --- a/lib/crypto/arm/sha1.h +++ b/lib/crypto/arm/sha1.h @@ -4,7 +4,6 @@ * * Copyright 2025 Google LLC */ -#include #include =20 static __ro_after_init DEFINE_STATIC_KEY_FALSE(have_neon); @@ -22,12 +21,12 @@ static void sha1_blocks(struct sha1_block_state *state, { if (IS_ENABLED(CONFIG_KERNEL_MODE_NEON) && static_branch_likely(&have_neon) && likely(may_use_simd())) { - kernel_neon_begin(); - if (static_branch_likely(&have_ce)) - sha1_ce_transform(state, data, nblocks); - else - sha1_transform_neon(state, data, nblocks); - kernel_neon_end(); + scoped_ksimd() { + if (static_branch_likely(&have_ce)) + sha1_ce_transform(state, data, nblocks); + else + sha1_transform_neon(state, data, nblocks); + } } else { sha1_block_data_order(state, data, nblocks); } diff --git a/lib/crypto/arm/sha256.h b/lib/crypto/arm/sha256.h index 7556457b3094..ae7e52dd6e3b 100644 --- a/lib/crypto/arm/sha256.h +++ b/lib/crypto/arm/sha256.h @@ -22,12 +22,12 @@ static void sha256_blocks(struct sha256_block_state *st= ate, { if (IS_ENABLED(CONFIG_KERNEL_MODE_NEON) && static_branch_likely(&have_neon) && likely(may_use_simd())) { - kernel_neon_begin(); - if (static_branch_likely(&have_ce)) - sha256_ce_transform(state, data, nblocks); - else - sha256_block_data_order_neon(state, data, nblocks); - kernel_neon_end(); + scoped_ksimd() { + if (static_branch_likely(&have_ce)) + sha256_ce_transform(state, data, nblocks); + else + sha256_block_data_order_neon(state, data, nblocks); + } } else { sha256_block_data_order(state, data, nblocks); } diff --git a/lib/crypto/arm/sha512.h b/lib/crypto/arm/sha512.h index d1b485dd275d..ed9bd81d6d78 100644 --- a/lib/crypto/arm/sha512.h +++ b/lib/crypto/arm/sha512.h @@ -19,9 +19,8 @@ static void sha512_blocks(struct sha512_block_state *stat= e, { if (IS_ENABLED(CONFIG_KERNEL_MODE_NEON) && static_branch_likely(&have_neon) && likely(may_use_simd())) { - kernel_neon_begin(); - sha512_block_data_order_neon(state, data, nblocks); - kernel_neon_end(); + scoped_ksimd() + sha512_block_data_order_neon(state, data, nblocks); } else { sha512_block_data_order(state, data, nblocks); } diff --git a/lib/crypto/arm64/chacha.h b/lib/crypto/arm64/chacha.h index ba6c22d46086..ca8c6a8b0578 100644 --- a/lib/crypto/arm64/chacha.h +++ b/lib/crypto/arm64/chacha.h @@ -23,7 +23,6 @@ #include =20 #include -#include #include =20 asmlinkage void chacha_block_xor_neon(const struct chacha_state *state, @@ -65,9 +64,8 @@ static void hchacha_block_arch(const struct chacha_state = *state, if (!static_branch_likely(&have_neon) || !crypto_simd_usable()) { hchacha_block_generic(state, out, nrounds); } else { - kernel_neon_begin(); - hchacha_block_neon(state, out, nrounds); - kernel_neon_end(); + scoped_ksimd() + hchacha_block_neon(state, out, nrounds); } } =20 @@ -81,9 +79,8 @@ static void chacha_crypt_arch(struct chacha_state *state,= u8 *dst, do { unsigned int todo =3D min_t(unsigned int, bytes, SZ_4K); =20 - kernel_neon_begin(); - chacha_doneon(state, dst, src, todo, nrounds); - kernel_neon_end(); + scoped_ksimd() + chacha_doneon(state, dst, src, todo, nrounds); =20 bytes -=3D todo; src +=3D todo; diff --git a/lib/crypto/arm64/poly1305.h b/lib/crypto/arm64/poly1305.h index aed5921ccd9a..b77669767cd6 100644 --- a/lib/crypto/arm64/poly1305.h +++ b/lib/crypto/arm64/poly1305.h @@ -6,7 +6,6 @@ */ =20 #include -#include #include #include #include @@ -31,9 +30,8 @@ static void poly1305_blocks(struct poly1305_block_state *= state, const u8 *src, do { unsigned int todo =3D min_t(unsigned int, len, SZ_4K); =20 - kernel_neon_begin(); - poly1305_blocks_neon(state, src, todo, padbit); - kernel_neon_end(); + scoped_ksimd() + poly1305_blocks_neon(state, src, todo, padbit); =20 len -=3D todo; src +=3D todo; diff --git a/lib/crypto/arm64/sha1.h b/lib/crypto/arm64/sha1.h index aaef4ebfc5e3..bc7071f1be09 100644 --- a/lib/crypto/arm64/sha1.h +++ b/lib/crypto/arm64/sha1.h @@ -4,7 +4,6 @@ * * Copyright 2025 Google LLC */ -#include #include #include =20 @@ -20,9 +19,9 @@ static void sha1_blocks(struct sha1_block_state *state, do { size_t rem; =20 - kernel_neon_begin(); - rem =3D __sha1_ce_transform(state, data, nblocks); - kernel_neon_end(); + scoped_ksimd() + rem =3D __sha1_ce_transform(state, data, nblocks); + data +=3D (nblocks - rem) * SHA1_BLOCK_SIZE; nblocks =3D rem; } while (nblocks); diff --git a/lib/crypto/arm64/sha256.h b/lib/crypto/arm64/sha256.h index 80d06df27d3a..568dff0f276a 100644 --- a/lib/crypto/arm64/sha256.h +++ b/lib/crypto/arm64/sha256.h @@ -4,7 +4,6 @@ * * Copyright 2025 Google LLC */ -#include #include #include =20 @@ -27,17 +26,16 @@ static void sha256_blocks(struct sha256_block_state *st= ate, do { size_t rem; =20 - kernel_neon_begin(); - rem =3D __sha256_ce_transform(state, - data, nblocks); - kernel_neon_end(); + scoped_ksimd() + rem =3D __sha256_ce_transform(state, data, + nblocks); + data +=3D (nblocks - rem) * SHA256_BLOCK_SIZE; nblocks =3D rem; } while (nblocks); } else { - kernel_neon_begin(); - sha256_block_neon(state, data, nblocks); - kernel_neon_end(); + scoped_ksimd() + sha256_block_neon(state, data, nblocks); } } else { sha256_block_data_order(state, data, nblocks); @@ -66,9 +64,8 @@ static bool sha256_finup_2x_arch(const struct __sha256_ct= x *ctx, if (IS_ENABLED(CONFIG_KERNEL_MODE_NEON) && static_branch_likely(&have_ce) && len >=3D SHA256_BLOCK_SIZE && len <=3D 65536 && likely(may_use_simd())) { - kernel_neon_begin(); - sha256_ce_finup2x(ctx, data1, data2, len, out1, out2); - kernel_neon_end(); + scoped_ksimd() + sha256_ce_finup2x(ctx, data1, data2, len, out1, out2); kmsan_unpoison_memory(out1, SHA256_DIGEST_SIZE); kmsan_unpoison_memory(out2, SHA256_DIGEST_SIZE); return true; diff --git a/lib/crypto/arm64/sha512.h b/lib/crypto/arm64/sha512.h index ddb0d256f73a..7eb7ef04d268 100644 --- a/lib/crypto/arm64/sha512.h +++ b/lib/crypto/arm64/sha512.h @@ -4,7 +4,7 @@ * * Copyright 2025 Google LLC */ -#include + #include #include =20 @@ -24,9 +24,9 @@ static void sha512_blocks(struct sha512_block_state *stat= e, do { size_t rem; =20 - kernel_neon_begin(); - rem =3D __sha512_ce_transform(state, data, nblocks); - kernel_neon_end(); + scoped_ksimd() + rem =3D __sha512_ce_transform(state, data, nblocks); + data +=3D (nblocks - rem) * SHA512_BLOCK_SIZE; nblocks =3D rem; } while (nblocks); --=20 2.51.1.930.gacf6e81ea2-goog From nobody Fri Dec 19 02:49:54 2025 Received: from mail-wr1-f73.google.com (mail-wr1-f73.google.com [209.85.221.73]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5EE7A3446D9 for ; Fri, 31 Oct 2025 10:39:57 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.221.73 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1761907199; cv=none; b=HrKd0zCt1Em26Y1H1vTcvWoSs6dj97hmMR7tF75GVu76HktvnZCaJWs9O0JdJE9fKTARAhTIQT4ORjyzCtuStYQ211V6DhC1uWw2fJI51Ni3ykqyHWYV+lgRQO0OUlNaYBE4Aut63e5Ydkf9Zl1DRvW85MhTJFwnORRG99hlW50= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1761907199; c=relaxed/simple; bh=Mt817fVHl1ncEn/trrFurKYEXgvjGDzS5+4EPDmK7xo=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=QIH+l71WjHxE+sTLscKVRSg5YQynDKbA/mq7RRdOMsefpl7LfejpQiUQP9dB7FBCd+clb9xPAmqsQ85WAqZhRgCihYTFZLtRewQWFzUCrMZq8tKfQ0Q4y1ybXUEs98iRMOxKQKrTl8i5zRoXxWLRqw5CGp0i8HZRSb07P5qtRrE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=WBj9iP6Z; arc=none smtp.client-ip=209.85.221.73 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="WBj9iP6Z" Received: by mail-wr1-f73.google.com with SMTP id ffacd0b85a97d-429be5aeea2so380984f8f.2 for ; Fri, 31 Oct 2025 03:39:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1761907196; x=1762511996; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=7SwJrIr6U1+pVeLDI8b/Z90ePApGArR/MfORdKx+bA8=; b=WBj9iP6Z/aizR0R7QCX+5xx7Zs6sv4DCgzpczr3k/iiSK9tz48XqJGZ5IMc+DwmpWo 12XkiVA+zdz5YlVsAhFZSSpQJNpqhAnDk5irTzc/domazh2DWV+A7kgZ2B5NeH0o4UQv 11U5Q33Kk1qn8dADhOuNRFhuFg5v8sSPTZH/zus9vDn4UFX3pMzkTpnpHk0D+MrITcoT 65yxXXdmgmkHSGBmDSes4MckRio1izuSEs4T7LxVLOWBCxOj9sM2938HQV0YDf4YIlmo 5Ur6r3185g2HhhbJGh2XO+oYYv3tIQ0NegEulhW1A6gc2fwHpv0LJ4pKLKdEqUWt+Mvr SuYQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1761907196; x=1762511996; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=7SwJrIr6U1+pVeLDI8b/Z90ePApGArR/MfORdKx+bA8=; b=Vn6xsMiDptwa9RQ9GEr9a1gCD/zg77jhLG6I0tshhgVIRb/hxsHKXsMyeCIETdnaTz 8W11RChrJGdSZHfS4Jlz/EmGODT1wxWDEqKm3mYV3D+4GT3KvaoFotdiEX8KWqAIq9d3 o2R8dB8axE7ZBNbokh+Kwn0nDH5daSHcJEL8T15in39npkx53YFDj/IOUNfKymGL7lQM UzzRs2kiwlyDbc9u4WCKxVZ0YmPuV2xisTx7Na+e9dLmvyJp4P6v9Ky4bAcWPXq1et96 6Dkg9IMFZUadL3fQganefH5nY8agO4qNkgwbEt7fCFVNuQdlHoMM+JneH5v3kOK3YolP WBAQ== X-Gm-Message-State: AOJu0Yze1ZkAnLBQzrDS9mF9599xJWno8/wBJuTcxVbXAif4mVgZE6Vx cD0QsK7Nc5FU9+/KFjtoS0fQGlbDxUVqbhF/fDNiiQ5KtdaZ4IyFnBYSKa4/XMPsudUwR/RYjQ= = X-Google-Smtp-Source: AGHT+IFMY96nZKFh7zEZwtP6yze2D7QGtPE/U4QgTFRH0L770EQGt27hCpp9UjR+jUWQd0JcGD6pkpoY X-Received: from wrbdz17.prod.google.com ([2002:a05:6000:e91:b0:429:8bd5:a979]) (user=ardb job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6000:4382:b0:429:bd09:e7b6 with SMTP id ffacd0b85a97d-429bd67245dmr2637319f8f.10.1761907195978; Fri, 31 Oct 2025 03:39:55 -0700 (PDT) Date: Fri, 31 Oct 2025 11:39:09 +0100 In-Reply-To: <20251031103858.529530-23-ardb+git@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20251031103858.529530-23-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=4882; i=ardb@kernel.org; h=from:subject; bh=vWH74Y94585NgOqf0h7A64VGzkAz8Q+4d31ezTbmH+c=; b=owGbwMvMwCVmkMcZplerG8N4Wi2JIZNl4kWG1Tv6ni+dXPjVf5pkyIGK7c4rL1vsmm+dyxXTa KDjEHyvo5SFQYyLQVZMkUVg9t93O09PlKp1niULM4eVCWwIF6cATOS3MiPDj+roXWqWpraFilwP eON+TMnYP1Hi99MlH6tnMi+cf63/McMfjttR+uu+L2tYHTR989KnLsKr6/tUGZ/2V53O3Xvs7cM v3AA= X-Mailer: git-send-email 2.51.1.930.gacf6e81ea2-goog Message-ID: <20251031103858.529530-33-ardb+git@google.com> Subject: [PATCH v4 10/21] crypto/arm64: aes-ccm - Switch to 'ksimd' scoped guard API From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: linux-kernel@vger.kernel.org, linux-crypto@vger.kernel.org, herbert@gondor.apana.org.au, ebiggers@kernel.org, Ard Biesheuvel Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Ard Biesheuvel Switch to the more abstract 'scoped_ksimd()' API, which will be modified in a future patch to transparently allocate a kernel mode FP/SIMD state buffer on the stack, so that kernel mode FP/SIMD code remains preemptible in principe, but without the memory overhead that adds 528 bytes to the size of struct task_struct. Reviewed-by: Eric Biggers Signed-off-by: Ard Biesheuvel Acked-by: Catalin Marinas --- arch/arm64/crypto/aes-ce-ccm-glue.c | 135 ++++++++++---------- 1 file changed, 66 insertions(+), 69 deletions(-) diff --git a/arch/arm64/crypto/aes-ce-ccm-glue.c b/arch/arm64/crypto/aes-ce= -ccm-glue.c index 2eb4e76cabc3..c4fd648471f1 100644 --- a/arch/arm64/crypto/aes-ce-ccm-glue.c +++ b/arch/arm64/crypto/aes-ce-ccm-glue.c @@ -8,7 +8,6 @@ * Author: Ard Biesheuvel */ =20 -#include #include #include #include @@ -16,6 +15,8 @@ #include #include =20 +#include + #include "aes-ce-setkey.h" =20 MODULE_IMPORT_NS("CRYPTO_INTERNAL"); @@ -184,40 +185,38 @@ static int ccm_encrypt(struct aead_request *req) if (unlikely(err)) return err; =20 - kernel_neon_begin(); - - if (req->assoclen) - ccm_calculate_auth_mac(req, mac); - - do { - u32 tail =3D walk.nbytes % AES_BLOCK_SIZE; - const u8 *src =3D walk.src.virt.addr; - u8 *dst =3D walk.dst.virt.addr; - u8 buf[AES_BLOCK_SIZE]; - u8 *final_iv =3D NULL; - - if (walk.nbytes =3D=3D walk.total) { - tail =3D 0; - final_iv =3D orig_iv; - } - - if (unlikely(walk.nbytes < AES_BLOCK_SIZE)) - src =3D dst =3D memcpy(&buf[sizeof(buf) - walk.nbytes], - src, walk.nbytes); - - ce_aes_ccm_encrypt(dst, src, walk.nbytes - tail, - ctx->key_enc, num_rounds(ctx), - mac, walk.iv, final_iv); - - if (unlikely(walk.nbytes < AES_BLOCK_SIZE)) - memcpy(walk.dst.virt.addr, dst, walk.nbytes); - - if (walk.nbytes) { - err =3D skcipher_walk_done(&walk, tail); - } - } while (walk.nbytes); - - kernel_neon_end(); + scoped_ksimd() { + if (req->assoclen) + ccm_calculate_auth_mac(req, mac); + + do { + u32 tail =3D walk.nbytes % AES_BLOCK_SIZE; + const u8 *src =3D walk.src.virt.addr; + u8 *dst =3D walk.dst.virt.addr; + u8 buf[AES_BLOCK_SIZE]; + u8 *final_iv =3D NULL; + + if (walk.nbytes =3D=3D walk.total) { + tail =3D 0; + final_iv =3D orig_iv; + } + + if (unlikely(walk.nbytes < AES_BLOCK_SIZE)) + src =3D dst =3D memcpy(&buf[sizeof(buf) - walk.nbytes], + src, walk.nbytes); + + ce_aes_ccm_encrypt(dst, src, walk.nbytes - tail, + ctx->key_enc, num_rounds(ctx), + mac, walk.iv, final_iv); + + if (unlikely(walk.nbytes < AES_BLOCK_SIZE)) + memcpy(walk.dst.virt.addr, dst, walk.nbytes); + + if (walk.nbytes) { + err =3D skcipher_walk_done(&walk, tail); + } + } while (walk.nbytes); + } =20 if (unlikely(err)) return err; @@ -251,40 +250,38 @@ static int ccm_decrypt(struct aead_request *req) if (unlikely(err)) return err; =20 - kernel_neon_begin(); - - if (req->assoclen) - ccm_calculate_auth_mac(req, mac); - - do { - u32 tail =3D walk.nbytes % AES_BLOCK_SIZE; - const u8 *src =3D walk.src.virt.addr; - u8 *dst =3D walk.dst.virt.addr; - u8 buf[AES_BLOCK_SIZE]; - u8 *final_iv =3D NULL; - - if (walk.nbytes =3D=3D walk.total) { - tail =3D 0; - final_iv =3D orig_iv; - } - - if (unlikely(walk.nbytes < AES_BLOCK_SIZE)) - src =3D dst =3D memcpy(&buf[sizeof(buf) - walk.nbytes], - src, walk.nbytes); - - ce_aes_ccm_decrypt(dst, src, walk.nbytes - tail, - ctx->key_enc, num_rounds(ctx), - mac, walk.iv, final_iv); - - if (unlikely(walk.nbytes < AES_BLOCK_SIZE)) - memcpy(walk.dst.virt.addr, dst, walk.nbytes); - - if (walk.nbytes) { - err =3D skcipher_walk_done(&walk, tail); - } - } while (walk.nbytes); - - kernel_neon_end(); + scoped_ksimd() { + if (req->assoclen) + ccm_calculate_auth_mac(req, mac); + + do { + u32 tail =3D walk.nbytes % AES_BLOCK_SIZE; + const u8 *src =3D walk.src.virt.addr; + u8 *dst =3D walk.dst.virt.addr; + u8 buf[AES_BLOCK_SIZE]; + u8 *final_iv =3D NULL; + + if (walk.nbytes =3D=3D walk.total) { + tail =3D 0; + final_iv =3D orig_iv; + } + + if (unlikely(walk.nbytes < AES_BLOCK_SIZE)) + src =3D dst =3D memcpy(&buf[sizeof(buf) - walk.nbytes], + src, walk.nbytes); + + ce_aes_ccm_decrypt(dst, src, walk.nbytes - tail, + ctx->key_enc, num_rounds(ctx), + mac, walk.iv, final_iv); + + if (unlikely(walk.nbytes < AES_BLOCK_SIZE)) + memcpy(walk.dst.virt.addr, dst, walk.nbytes); + + if (walk.nbytes) { + err =3D skcipher_walk_done(&walk, tail); + } + } while (walk.nbytes); + } =20 if (unlikely(err)) return err; --=20 2.51.1.930.gacf6e81ea2-goog From nobody Fri Dec 19 02:49:54 2025 Received: from mail-wr1-f73.google.com (mail-wr1-f73.google.com [209.85.221.73]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6BE3233BBA0 for ; Fri, 31 Oct 2025 10:39:58 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.221.73 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1761907201; cv=none; b=nX5Y2MFUMq8FL3zBsOC5W9x7iHwituSKZYwPCS02F16ANLJWgapKy+1VhnGMntQMUB1CVBVHcYLsOHDRyi0/9BtDK59PxsdSakAdcbFEsIvPkY3TeGjE08AWctspaKp5pgToNxooBvDHiGX4jOHo0fmG6hHSsqLF7CMeQeIG1EI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1761907201; c=relaxed/simple; bh=LJep+y4ATnNVUskfDNp0lQuGBZD2EP4g979aUfeXIPk=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=ZH7A8XMTcpkTCq1zcujpsqNAkqxBwz3kGxAZ66HmUvAv0YB4n9E7PfpcqGZ/43ZBR1mnRd+fWasg3jyWcvX5E9JXp2UgJtCgbuTQv8LwUeKyX+Mdw+kg4WvWGRcg0Sor6n3Znm5yDL3xd8/C+cp1ir7av51REbTZhOdu0MDUJ7E= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=xJSWJmoi; arc=none smtp.client-ip=209.85.221.73 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="xJSWJmoi" Received: by mail-wr1-f73.google.com with SMTP id ffacd0b85a97d-429c2f6a5adso132254f8f.0 for ; Fri, 31 Oct 2025 03:39:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1761907197; x=1762511997; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=vW/4HMHRiwaJ6EVSFpt6OOUDjFOC/wmRBK5AoFPty4Y=; b=xJSWJmoicoxZsHSIvNqa/BNqq7L7ZItZ3HOABbn2wo5Q0V0aPY9G53HwOeWTuVFlz6 wGtTNR28YXkGepbFABYhpOzoIXJUVAYLYcCy1rHbGDy08siTnFxpJCnp1wumHHsEJLXW dDczrrOqqanI4do2rUzZKwavLIoFbL+uFsHsEu6wL+GlXVxIT8HGiTW+yrf1aEGQaNSM QWI0Xc5QNIMDmDWoQZBufqX6OftJxiDLmDnglSiDSjzDtZCZjP4n7UnPoa0gcdRVXXRY eHJqRJo4lLhZmXjswF+w6zOz8YOxYf5FjrZtRhmLGiJJgEROfnVhtMLZ2hZ4PkLvcl4x +X+w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1761907197; x=1762511997; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=vW/4HMHRiwaJ6EVSFpt6OOUDjFOC/wmRBK5AoFPty4Y=; b=m7qoVTqpwAAJiKdiobIQisPnF/54NmlE4Ca9Xj7IEFQ9kMXIbTSZkaJSkMdV0P7Y7c ovvDzc+S65P3D9NMw9QoZ301Ca/zsXC3UTurjPalm8//HnJA+ZX5vMuAiLzTkbNZ0J3P 9C2iqd2nwioixWMgljlwYrnJ+pqT3iQnFhM2uixQMqT6Jz7G0fop+p+ETaZP6hzXJ+g6 rX/3uEuOK6uYgb1Cr6liBZS4GCmaWV64VUMBtQw0JfERNoK3EIDlw1hpGweYQbdd16NX z2hFG+TFH4+dKmHbKgMCqgw5/N8AdJ0t2R+Iip1Hr0SyKlrobznDMLxfGjFwQfEoQf9U R+GA== X-Gm-Message-State: AOJu0YxLUttQmCCMMZZxAoy2yhsqVKp9nt61S4gY9sytSYSVG7FOwILn 10AfIFSrAnFy5mP06DEEZwd5LZN6qlGz7e3X0Mi2Q3sQCQ06y/AuJbG4oZ6z/tZC0bVzU7BFkg= = X-Google-Smtp-Source: AGHT+IGpykGfTzt+UMgehQVa+X/CIbWFhM4CD7YwyHd7ft0MjQCcnZ1F4YBYBw/qaiFCQGs74a7PjyhA X-Received: from wrsz9.prod.google.com ([2002:a5d:4c89:0:b0:427:53e:a421]) (user=ardb job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6000:2005:b0:428:4354:aa27 with SMTP id ffacd0b85a97d-429bd6a8df9mr2532656f8f.46.1761907196795; Fri, 31 Oct 2025 03:39:56 -0700 (PDT) Date: Fri, 31 Oct 2025 11:39:10 +0100 In-Reply-To: <20251031103858.529530-23-ardb+git@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20251031103858.529530-23-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=20750; i=ardb@kernel.org; h=from:subject; bh=GyvDP6RYmRurKRktBsz1r+elH0Fzi+2wFgZdad4GjgQ=; b=owGbwMvMwCVmkMcZplerG8N4Wi2JIZNl4uVdDmeN3vy/c4SPaV/1nBkhITkMEpqJV3WUHPSyk t6aLjzaUcrCIMbFICumyCIw+++7nacnStU6z5KFmcPKBDKEgYtTACbi58Pwz6xu0qz1tkULOlkS H+rOqlcx637eMuuxWAtffMin/XxWGxj+aaj94Qg+e+RdZ2zVId6Ac1uqn3To2M3/02Zvs6dyo+F iRgA= X-Mailer: git-send-email 2.51.1.930.gacf6e81ea2-goog Message-ID: <20251031103858.529530-34-ardb+git@google.com> Subject: [PATCH v4 11/21] crypto/arm64: aes-blk - Switch to 'ksimd' scoped guard API From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: linux-kernel@vger.kernel.org, linux-crypto@vger.kernel.org, herbert@gondor.apana.org.au, ebiggers@kernel.org, Ard Biesheuvel Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Ard Biesheuvel Switch to the more abstract 'scoped_ksimd()' API, which will be modified in a future patch to transparently allocate a kernel mode FP/SIMD state buffer on the stack, so that kernel mode FP/SIMD code remains preemptible in principe, but without the memory overhead that adds 528 bytes to the size of struct task_struct. Reviewed-by: Eric Biggers Signed-off-by: Ard Biesheuvel Acked-by: Catalin Marinas --- arch/arm64/crypto/aes-ce-glue.c | 87 ++++++------ arch/arm64/crypto/aes-glue.c | 139 ++++++++---------- arch/arm64/crypto/aes-neonbs-glue.c | 150 ++++++++++---------- 3 files changed, 181 insertions(+), 195 deletions(-) diff --git a/arch/arm64/crypto/aes-ce-glue.c b/arch/arm64/crypto/aes-ce-glu= e.c index 00b8749013c5..a4dad370991d 100644 --- a/arch/arm64/crypto/aes-ce-glue.c +++ b/arch/arm64/crypto/aes-ce-glue.c @@ -52,9 +52,8 @@ static void aes_cipher_encrypt(struct crypto_tfm *tfm, u8= dst[], u8 const src[]) return; } =20 - kernel_neon_begin(); - __aes_ce_encrypt(ctx->key_enc, dst, src, num_rounds(ctx)); - kernel_neon_end(); + scoped_ksimd() + __aes_ce_encrypt(ctx->key_enc, dst, src, num_rounds(ctx)); } =20 static void aes_cipher_decrypt(struct crypto_tfm *tfm, u8 dst[], u8 const = src[]) @@ -66,9 +65,8 @@ static void aes_cipher_decrypt(struct crypto_tfm *tfm, u8= dst[], u8 const src[]) return; } =20 - kernel_neon_begin(); - __aes_ce_decrypt(ctx->key_dec, dst, src, num_rounds(ctx)); - kernel_neon_end(); + scoped_ksimd() + __aes_ce_decrypt(ctx->key_dec, dst, src, num_rounds(ctx)); } =20 int ce_aes_expandkey(struct crypto_aes_ctx *ctx, const u8 *in_key, @@ -94,47 +92,48 @@ int ce_aes_expandkey(struct crypto_aes_ctx *ctx, const = u8 *in_key, for (i =3D 0; i < kwords; i++) ctx->key_enc[i] =3D get_unaligned_le32(in_key + i * sizeof(u32)); =20 - kernel_neon_begin(); - for (i =3D 0; i < sizeof(rcon); i++) { - u32 *rki =3D ctx->key_enc + (i * kwords); - u32 *rko =3D rki + kwords; - - rko[0] =3D ror32(__aes_ce_sub(rki[kwords - 1]), 8) ^ rcon[i] ^ rki[0]; - rko[1] =3D rko[0] ^ rki[1]; - rko[2] =3D rko[1] ^ rki[2]; - rko[3] =3D rko[2] ^ rki[3]; - - if (key_len =3D=3D AES_KEYSIZE_192) { - if (i >=3D 7) - break; - rko[4] =3D rko[3] ^ rki[4]; - rko[5] =3D rko[4] ^ rki[5]; - } else if (key_len =3D=3D AES_KEYSIZE_256) { - if (i >=3D 6) - break; - rko[4] =3D __aes_ce_sub(rko[3]) ^ rki[4]; - rko[5] =3D rko[4] ^ rki[5]; - rko[6] =3D rko[5] ^ rki[6]; - rko[7] =3D rko[6] ^ rki[7]; + scoped_ksimd() { + for (i =3D 0; i < sizeof(rcon); i++) { + u32 *rki =3D ctx->key_enc + (i * kwords); + u32 *rko =3D rki + kwords; + + rko[0] =3D ror32(__aes_ce_sub(rki[kwords - 1]), 8) ^ + rcon[i] ^ rki[0]; + rko[1] =3D rko[0] ^ rki[1]; + rko[2] =3D rko[1] ^ rki[2]; + rko[3] =3D rko[2] ^ rki[3]; + + if (key_len =3D=3D AES_KEYSIZE_192) { + if (i >=3D 7) + break; + rko[4] =3D rko[3] ^ rki[4]; + rko[5] =3D rko[4] ^ rki[5]; + } else if (key_len =3D=3D AES_KEYSIZE_256) { + if (i >=3D 6) + break; + rko[4] =3D __aes_ce_sub(rko[3]) ^ rki[4]; + rko[5] =3D rko[4] ^ rki[5]; + rko[6] =3D rko[5] ^ rki[6]; + rko[7] =3D rko[6] ^ rki[7]; + } } - } =20 - /* - * Generate the decryption keys for the Equivalent Inverse Cipher. - * This involves reversing the order of the round keys, and applying - * the Inverse Mix Columns transformation on all but the first and - * the last one. - */ - key_enc =3D (struct aes_block *)ctx->key_enc; - key_dec =3D (struct aes_block *)ctx->key_dec; - j =3D num_rounds(ctx); - - key_dec[0] =3D key_enc[j]; - for (i =3D 1, j--; j > 0; i++, j--) - __aes_ce_invert(key_dec + i, key_enc + j); - key_dec[i] =3D key_enc[0]; + /* + * Generate the decryption keys for the Equivalent Inverse + * Cipher. This involves reversing the order of the round + * keys, and applying the Inverse Mix Columns transformation on + * all but the first and the last one. + */ + key_enc =3D (struct aes_block *)ctx->key_enc; + key_dec =3D (struct aes_block *)ctx->key_dec; + j =3D num_rounds(ctx); + + key_dec[0] =3D key_enc[j]; + for (i =3D 1, j--; j > 0; i++, j--) + __aes_ce_invert(key_dec + i, key_enc + j); + key_dec[i] =3D key_enc[0]; + } =20 - kernel_neon_end(); return 0; } EXPORT_SYMBOL(ce_aes_expandkey); diff --git a/arch/arm64/crypto/aes-glue.c b/arch/arm64/crypto/aes-glue.c index 5e207ff34482..b087b900d279 100644 --- a/arch/arm64/crypto/aes-glue.c +++ b/arch/arm64/crypto/aes-glue.c @@ -5,8 +5,6 @@ * Copyright (C) 2013 - 2017 Linaro Ltd */ =20 -#include -#include #include #include #include @@ -20,6 +18,9 @@ #include #include =20 +#include +#include + #include "aes-ce-setkey.h" =20 #ifdef USE_V8_CRYPTO_EXTENSIONS @@ -186,10 +187,9 @@ static int __maybe_unused ecb_encrypt(struct skcipher_= request *req) err =3D skcipher_walk_virt(&walk, req, false); =20 while ((blocks =3D (walk.nbytes / AES_BLOCK_SIZE))) { - kernel_neon_begin(); - aes_ecb_encrypt(walk.dst.virt.addr, walk.src.virt.addr, - ctx->key_enc, rounds, blocks); - kernel_neon_end(); + scoped_ksimd() + aes_ecb_encrypt(walk.dst.virt.addr, walk.src.virt.addr, + ctx->key_enc, rounds, blocks); err =3D skcipher_walk_done(&walk, walk.nbytes % AES_BLOCK_SIZE); } return err; @@ -206,10 +206,9 @@ static int __maybe_unused ecb_decrypt(struct skcipher_= request *req) err =3D skcipher_walk_virt(&walk, req, false); =20 while ((blocks =3D (walk.nbytes / AES_BLOCK_SIZE))) { - kernel_neon_begin(); - aes_ecb_decrypt(walk.dst.virt.addr, walk.src.virt.addr, - ctx->key_dec, rounds, blocks); - kernel_neon_end(); + scoped_ksimd() + aes_ecb_decrypt(walk.dst.virt.addr, walk.src.virt.addr, + ctx->key_dec, rounds, blocks); err =3D skcipher_walk_done(&walk, walk.nbytes % AES_BLOCK_SIZE); } return err; @@ -224,10 +223,9 @@ static int cbc_encrypt_walk(struct skcipher_request *r= eq, unsigned int blocks; =20 while ((blocks =3D (walk->nbytes / AES_BLOCK_SIZE))) { - kernel_neon_begin(); - aes_cbc_encrypt(walk->dst.virt.addr, walk->src.virt.addr, - ctx->key_enc, rounds, blocks, walk->iv); - kernel_neon_end(); + scoped_ksimd() + aes_cbc_encrypt(walk->dst.virt.addr, walk->src.virt.addr, + ctx->key_enc, rounds, blocks, walk->iv); err =3D skcipher_walk_done(walk, walk->nbytes % AES_BLOCK_SIZE); } return err; @@ -253,10 +251,9 @@ static int cbc_decrypt_walk(struct skcipher_request *r= eq, unsigned int blocks; =20 while ((blocks =3D (walk->nbytes / AES_BLOCK_SIZE))) { - kernel_neon_begin(); - aes_cbc_decrypt(walk->dst.virt.addr, walk->src.virt.addr, - ctx->key_dec, rounds, blocks, walk->iv); - kernel_neon_end(); + scoped_ksimd() + aes_cbc_decrypt(walk->dst.virt.addr, walk->src.virt.addr, + ctx->key_dec, rounds, blocks, walk->iv); err =3D skcipher_walk_done(walk, walk->nbytes % AES_BLOCK_SIZE); } return err; @@ -322,10 +319,9 @@ static int cts_cbc_encrypt(struct skcipher_request *re= q) if (err) return err; =20 - kernel_neon_begin(); - aes_cbc_cts_encrypt(walk.dst.virt.addr, walk.src.virt.addr, - ctx->key_enc, rounds, walk.nbytes, walk.iv); - kernel_neon_end(); + scoped_ksimd() + aes_cbc_cts_encrypt(walk.dst.virt.addr, walk.src.virt.addr, + ctx->key_enc, rounds, walk.nbytes, walk.iv); =20 return skcipher_walk_done(&walk, 0); } @@ -379,10 +375,9 @@ static int cts_cbc_decrypt(struct skcipher_request *re= q) if (err) return err; =20 - kernel_neon_begin(); - aes_cbc_cts_decrypt(walk.dst.virt.addr, walk.src.virt.addr, - ctx->key_dec, rounds, walk.nbytes, walk.iv); - kernel_neon_end(); + scoped_ksimd() + aes_cbc_cts_decrypt(walk.dst.virt.addr, walk.src.virt.addr, + ctx->key_dec, rounds, walk.nbytes, walk.iv); =20 return skcipher_walk_done(&walk, 0); } @@ -399,11 +394,11 @@ static int __maybe_unused essiv_cbc_encrypt(struct sk= cipher_request *req) =20 blocks =3D walk.nbytes / AES_BLOCK_SIZE; if (blocks) { - kernel_neon_begin(); - aes_essiv_cbc_encrypt(walk.dst.virt.addr, walk.src.virt.addr, - ctx->key1.key_enc, rounds, blocks, - req->iv, ctx->key2.key_enc); - kernel_neon_end(); + scoped_ksimd() + aes_essiv_cbc_encrypt(walk.dst.virt.addr, + walk.src.virt.addr, + ctx->key1.key_enc, rounds, blocks, + req->iv, ctx->key2.key_enc); err =3D skcipher_walk_done(&walk, walk.nbytes % AES_BLOCK_SIZE); } return err ?: cbc_encrypt_walk(req, &walk); @@ -421,11 +416,11 @@ static int __maybe_unused essiv_cbc_decrypt(struct sk= cipher_request *req) =20 blocks =3D walk.nbytes / AES_BLOCK_SIZE; if (blocks) { - kernel_neon_begin(); - aes_essiv_cbc_decrypt(walk.dst.virt.addr, walk.src.virt.addr, - ctx->key1.key_dec, rounds, blocks, - req->iv, ctx->key2.key_enc); - kernel_neon_end(); + scoped_ksimd() + aes_essiv_cbc_decrypt(walk.dst.virt.addr, + walk.src.virt.addr, + ctx->key1.key_dec, rounds, blocks, + req->iv, ctx->key2.key_enc); err =3D skcipher_walk_done(&walk, walk.nbytes % AES_BLOCK_SIZE); } return err ?: cbc_decrypt_walk(req, &walk); @@ -461,10 +456,9 @@ static int __maybe_unused xctr_encrypt(struct skcipher= _request *req) else if (nbytes < walk.total) nbytes &=3D ~(AES_BLOCK_SIZE - 1); =20 - kernel_neon_begin(); - aes_xctr_encrypt(dst, src, ctx->key_enc, rounds, nbytes, - walk.iv, byte_ctr); - kernel_neon_end(); + scoped_ksimd() + aes_xctr_encrypt(dst, src, ctx->key_enc, rounds, nbytes, + walk.iv, byte_ctr); =20 if (unlikely(nbytes < AES_BLOCK_SIZE)) memcpy(walk.dst.virt.addr, @@ -506,10 +500,9 @@ static int __maybe_unused ctr_encrypt(struct skcipher_= request *req) else if (nbytes < walk.total) nbytes &=3D ~(AES_BLOCK_SIZE - 1); =20 - kernel_neon_begin(); - aes_ctr_encrypt(dst, src, ctx->key_enc, rounds, nbytes, - walk.iv); - kernel_neon_end(); + scoped_ksimd() + aes_ctr_encrypt(dst, src, ctx->key_enc, rounds, nbytes, + walk.iv); =20 if (unlikely(nbytes < AES_BLOCK_SIZE)) memcpy(walk.dst.virt.addr, @@ -562,11 +555,10 @@ static int __maybe_unused xts_encrypt(struct skcipher= _request *req) if (walk.nbytes < walk.total) nbytes &=3D ~(AES_BLOCK_SIZE - 1); =20 - kernel_neon_begin(); - aes_xts_encrypt(walk.dst.virt.addr, walk.src.virt.addr, - ctx->key1.key_enc, rounds, nbytes, - ctx->key2.key_enc, walk.iv, first); - kernel_neon_end(); + scoped_ksimd() + aes_xts_encrypt(walk.dst.virt.addr, walk.src.virt.addr, + ctx->key1.key_enc, rounds, nbytes, + ctx->key2.key_enc, walk.iv, first); err =3D skcipher_walk_done(&walk, walk.nbytes - nbytes); } =20 @@ -584,11 +576,10 @@ static int __maybe_unused xts_encrypt(struct skcipher= _request *req) if (err) return err; =20 - kernel_neon_begin(); - aes_xts_encrypt(walk.dst.virt.addr, walk.src.virt.addr, - ctx->key1.key_enc, rounds, walk.nbytes, - ctx->key2.key_enc, walk.iv, first); - kernel_neon_end(); + scoped_ksimd() + aes_xts_encrypt(walk.dst.virt.addr, walk.src.virt.addr, + ctx->key1.key_enc, rounds, walk.nbytes, + ctx->key2.key_enc, walk.iv, first); =20 return skcipher_walk_done(&walk, 0); } @@ -634,11 +625,10 @@ static int __maybe_unused xts_decrypt(struct skcipher= _request *req) if (walk.nbytes < walk.total) nbytes &=3D ~(AES_BLOCK_SIZE - 1); =20 - kernel_neon_begin(); - aes_xts_decrypt(walk.dst.virt.addr, walk.src.virt.addr, - ctx->key1.key_dec, rounds, nbytes, - ctx->key2.key_enc, walk.iv, first); - kernel_neon_end(); + scoped_ksimd() + aes_xts_decrypt(walk.dst.virt.addr, walk.src.virt.addr, + ctx->key1.key_dec, rounds, nbytes, + ctx->key2.key_enc, walk.iv, first); err =3D skcipher_walk_done(&walk, walk.nbytes - nbytes); } =20 @@ -657,11 +647,10 @@ static int __maybe_unused xts_decrypt(struct skcipher= _request *req) return err; =20 =20 - kernel_neon_begin(); - aes_xts_decrypt(walk.dst.virt.addr, walk.src.virt.addr, - ctx->key1.key_dec, rounds, walk.nbytes, - ctx->key2.key_enc, walk.iv, first); - kernel_neon_end(); + scoped_ksimd() + aes_xts_decrypt(walk.dst.virt.addr, walk.src.virt.addr, + ctx->key1.key_dec, rounds, walk.nbytes, + ctx->key2.key_enc, walk.iv, first); =20 return skcipher_walk_done(&walk, 0); } @@ -808,10 +797,9 @@ static int cmac_setkey(struct crypto_shash *tfm, const= u8 *in_key, return err; =20 /* encrypt the zero vector */ - kernel_neon_begin(); - aes_ecb_encrypt(ctx->consts, (u8[AES_BLOCK_SIZE]){}, ctx->key.key_enc, - rounds, 1); - kernel_neon_end(); + scoped_ksimd() + aes_ecb_encrypt(ctx->consts, (u8[AES_BLOCK_SIZE]){}, + ctx->key.key_enc, rounds, 1); =20 cmac_gf128_mul_by_x(consts, consts); cmac_gf128_mul_by_x(consts + 1, consts); @@ -837,10 +825,10 @@ static int xcbc_setkey(struct crypto_shash *tfm, cons= t u8 *in_key, if (err) return err; =20 - kernel_neon_begin(); - aes_ecb_encrypt(key, ks[0], ctx->key.key_enc, rounds, 1); - aes_ecb_encrypt(ctx->consts, ks[1], ctx->key.key_enc, rounds, 2); - kernel_neon_end(); + scoped_ksimd() { + aes_ecb_encrypt(key, ks[0], ctx->key.key_enc, rounds, 1); + aes_ecb_encrypt(ctx->consts, ks[1], ctx->key.key_enc, rounds, 2); + } =20 return cbcmac_setkey(tfm, key, sizeof(key)); } @@ -860,10 +848,9 @@ static void mac_do_update(struct crypto_aes_ctx *ctx, = u8 const in[], int blocks, int rem; =20 do { - kernel_neon_begin(); - rem =3D aes_mac_update(in, ctx->key_enc, rounds, blocks, - dg, enc_before, !enc_before); - kernel_neon_end(); + scoped_ksimd() + rem =3D aes_mac_update(in, ctx->key_enc, rounds, blocks, + dg, enc_before, !enc_before); in +=3D (blocks - rem) * AES_BLOCK_SIZE; blocks =3D rem; } while (blocks); diff --git a/arch/arm64/crypto/aes-neonbs-glue.c b/arch/arm64/crypto/aes-ne= onbs-glue.c index c4a623e86593..d496effb0a5b 100644 --- a/arch/arm64/crypto/aes-neonbs-glue.c +++ b/arch/arm64/crypto/aes-neonbs-glue.c @@ -85,9 +85,8 @@ static int aesbs_setkey(struct crypto_skcipher *tfm, cons= t u8 *in_key, =20 ctx->rounds =3D 6 + key_len / 4; =20 - kernel_neon_begin(); - aesbs_convert_key(ctx->rk, rk.key_enc, ctx->rounds); - kernel_neon_end(); + scoped_ksimd() + aesbs_convert_key(ctx->rk, rk.key_enc, ctx->rounds); =20 return 0; } @@ -110,10 +109,9 @@ static int __ecb_crypt(struct skcipher_request *req, blocks =3D round_down(blocks, walk.stride / AES_BLOCK_SIZE); =20 - kernel_neon_begin(); - fn(walk.dst.virt.addr, walk.src.virt.addr, ctx->rk, - ctx->rounds, blocks); - kernel_neon_end(); + scoped_ksimd() + fn(walk.dst.virt.addr, walk.src.virt.addr, ctx->rk, + ctx->rounds, blocks); err =3D skcipher_walk_done(&walk, walk.nbytes - blocks * AES_BLOCK_SIZE); } @@ -146,9 +144,8 @@ static int aesbs_cbc_ctr_setkey(struct crypto_skcipher = *tfm, const u8 *in_key, =20 memcpy(ctx->enc, rk.key_enc, sizeof(ctx->enc)); =20 - kernel_neon_begin(); - aesbs_convert_key(ctx->key.rk, rk.key_enc, ctx->key.rounds); - kernel_neon_end(); + scoped_ksimd() + aesbs_convert_key(ctx->key.rk, rk.key_enc, ctx->key.rounds); memzero_explicit(&rk, sizeof(rk)); =20 return 0; @@ -167,11 +164,11 @@ static int cbc_encrypt(struct skcipher_request *req) unsigned int blocks =3D walk.nbytes / AES_BLOCK_SIZE; =20 /* fall back to the non-bitsliced NEON implementation */ - kernel_neon_begin(); - neon_aes_cbc_encrypt(walk.dst.virt.addr, walk.src.virt.addr, - ctx->enc, ctx->key.rounds, blocks, - walk.iv); - kernel_neon_end(); + scoped_ksimd() + neon_aes_cbc_encrypt(walk.dst.virt.addr, + walk.src.virt.addr, + ctx->enc, ctx->key.rounds, blocks, + walk.iv); err =3D skcipher_walk_done(&walk, walk.nbytes % AES_BLOCK_SIZE); } return err; @@ -193,11 +190,10 @@ static int cbc_decrypt(struct skcipher_request *req) blocks =3D round_down(blocks, walk.stride / AES_BLOCK_SIZE); =20 - kernel_neon_begin(); - aesbs_cbc_decrypt(walk.dst.virt.addr, walk.src.virt.addr, - ctx->key.rk, ctx->key.rounds, blocks, - walk.iv); - kernel_neon_end(); + scoped_ksimd() + aesbs_cbc_decrypt(walk.dst.virt.addr, walk.src.virt.addr, + ctx->key.rk, ctx->key.rounds, blocks, + walk.iv); err =3D skcipher_walk_done(&walk, walk.nbytes - blocks * AES_BLOCK_SIZE); } @@ -220,30 +216,32 @@ static int ctr_encrypt(struct skcipher_request *req) const u8 *src =3D walk.src.virt.addr; u8 *dst =3D walk.dst.virt.addr; =20 - kernel_neon_begin(); - if (blocks >=3D 8) { - aesbs_ctr_encrypt(dst, src, ctx->key.rk, ctx->key.rounds, - blocks, walk.iv); - dst +=3D blocks * AES_BLOCK_SIZE; - src +=3D blocks * AES_BLOCK_SIZE; - } - if (nbytes && walk.nbytes =3D=3D walk.total) { - u8 buf[AES_BLOCK_SIZE]; - u8 *d =3D dst; - - if (unlikely(nbytes < AES_BLOCK_SIZE)) - src =3D dst =3D memcpy(buf + sizeof(buf) - nbytes, - src, nbytes); - - neon_aes_ctr_encrypt(dst, src, ctx->enc, ctx->key.rounds, - nbytes, walk.iv); + scoped_ksimd() { + if (blocks >=3D 8) { + aesbs_ctr_encrypt(dst, src, ctx->key.rk, + ctx->key.rounds, blocks, + walk.iv); + dst +=3D blocks * AES_BLOCK_SIZE; + src +=3D blocks * AES_BLOCK_SIZE; + } + if (nbytes && walk.nbytes =3D=3D walk.total) { + u8 buf[AES_BLOCK_SIZE]; + u8 *d =3D dst; + + if (unlikely(nbytes < AES_BLOCK_SIZE)) + src =3D dst =3D memcpy(buf + sizeof(buf) - + nbytes, src, nbytes); + + neon_aes_ctr_encrypt(dst, src, ctx->enc, + ctx->key.rounds, nbytes, + walk.iv); =20 - if (unlikely(nbytes < AES_BLOCK_SIZE)) - memcpy(d, dst, nbytes); + if (unlikely(nbytes < AES_BLOCK_SIZE)) + memcpy(d, dst, nbytes); =20 - nbytes =3D 0; + nbytes =3D 0; + } } - kernel_neon_end(); err =3D skcipher_walk_done(&walk, nbytes); } return err; @@ -320,33 +318,33 @@ static int __xts_crypt(struct skcipher_request *req, = bool encrypt, in =3D walk.src.virt.addr; nbytes =3D walk.nbytes; =20 - kernel_neon_begin(); - if (blocks >=3D 8) { - if (first =3D=3D 1) - neon_aes_ecb_encrypt(walk.iv, walk.iv, - ctx->twkey, - ctx->key.rounds, 1); - first =3D 2; - - fn(out, in, ctx->key.rk, ctx->key.rounds, blocks, - walk.iv); - - out +=3D blocks * AES_BLOCK_SIZE; - in +=3D blocks * AES_BLOCK_SIZE; - nbytes -=3D blocks * AES_BLOCK_SIZE; + scoped_ksimd() { + if (blocks >=3D 8) { + if (first =3D=3D 1) + neon_aes_ecb_encrypt(walk.iv, walk.iv, + ctx->twkey, + ctx->key.rounds, 1); + first =3D 2; + + fn(out, in, ctx->key.rk, ctx->key.rounds, blocks, + walk.iv); + + out +=3D blocks * AES_BLOCK_SIZE; + in +=3D blocks * AES_BLOCK_SIZE; + nbytes -=3D blocks * AES_BLOCK_SIZE; + } + if (walk.nbytes =3D=3D walk.total && nbytes > 0) { + if (encrypt) + neon_aes_xts_encrypt(out, in, ctx->cts.key_enc, + ctx->key.rounds, nbytes, + ctx->twkey, walk.iv, first); + else + neon_aes_xts_decrypt(out, in, ctx->cts.key_dec, + ctx->key.rounds, nbytes, + ctx->twkey, walk.iv, first); + nbytes =3D first =3D 0; + } } - if (walk.nbytes =3D=3D walk.total && nbytes > 0) { - if (encrypt) - neon_aes_xts_encrypt(out, in, ctx->cts.key_enc, - ctx->key.rounds, nbytes, - ctx->twkey, walk.iv, first); - else - neon_aes_xts_decrypt(out, in, ctx->cts.key_dec, - ctx->key.rounds, nbytes, - ctx->twkey, walk.iv, first); - nbytes =3D first =3D 0; - } - kernel_neon_end(); err =3D skcipher_walk_done(&walk, nbytes); } =20 @@ -369,14 +367,16 @@ static int __xts_crypt(struct skcipher_request *req, = bool encrypt, in =3D walk.src.virt.addr; nbytes =3D walk.nbytes; =20 - kernel_neon_begin(); - if (encrypt) - neon_aes_xts_encrypt(out, in, ctx->cts.key_enc, ctx->key.rounds, - nbytes, ctx->twkey, walk.iv, first); - else - neon_aes_xts_decrypt(out, in, ctx->cts.key_dec, ctx->key.rounds, - nbytes, ctx->twkey, walk.iv, first); - kernel_neon_end(); + scoped_ksimd() { + if (encrypt) + neon_aes_xts_encrypt(out, in, ctx->cts.key_enc, + ctx->key.rounds, nbytes, ctx->twkey, + walk.iv, first); + else + neon_aes_xts_decrypt(out, in, ctx->cts.key_dec, + ctx->key.rounds, nbytes, ctx->twkey, + walk.iv, first); + } =20 return skcipher_walk_done(&walk, 0); } --=20 2.51.1.930.gacf6e81ea2-goog From nobody Fri Dec 19 02:49:54 2025 Received: from mail-wm1-f73.google.com (mail-wm1-f73.google.com [209.85.128.73]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 356CE345CBA for ; Fri, 31 Oct 2025 10:39:58 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.73 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1761907201; cv=none; b=hIjzjgkD5wUDdzlO3ooGIgjAxI0bNkjlsZbVpyOfloEkByGlEuf5uzTggojE1K6UmrtxDvx1Fu4mi6s4tIqDRBbAbtlo6b2jZPmsOtWvA4fmVFa/PJpz95fJZ+3Rm+ATR0PWFCLK2qzrpKAKSX0GDZptKPiPYw5cLEJyMbUURgE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1761907201; c=relaxed/simple; bh=TVw+CnmJFmWZrxBaJk6VSGgwnVZfUbIi3slBCLmlOJo=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=n/6F3l7+dBYGjGwwut5Ajqq3b9P+tURkGYFunEqUJNw0a/DJM6zXQpMCtxnTD2VIt/AMiGWiPGkJQAoqPURHDHTBa+kfcc3x3X5K6rlIwXa5bDZVsXj6Q1iVUzeS2u0pW7sSINu1xdviY8/dJRL1soZINrwVPreHXcQ8V7NK1gM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=NN/ZPy/Y; arc=none smtp.client-ip=209.85.128.73 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="NN/ZPy/Y" Received: by mail-wm1-f73.google.com with SMTP id 5b1f17b1804b1-475db4369b1so11286745e9.3 for ; Fri, 31 Oct 2025 03:39:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1761907197; x=1762511997; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=fnEn/zNYGpYeH2kzoRoKsGDEwWWCiCH27FEc88gwnrM=; b=NN/ZPy/YHW05bB1f0x/k4KuwUKqqnGnzlJn+vIvkv07JNMGWdtK/jp8opg3tezW4zX CWFRpSPhpnlOEXrkMng9KNh3VfhBLatMNWloubj8eWPAtb5pjuDsMB2rn32nUi82zT3a QuEItZwS4PMZz3TUWrEll71C5vU1c7FbKFYZIuuUYyJxL4BQEWmJZXKQydHKllR6FDxe +7av51x+HXBTxHEgTjkbh9+molTAoN90YvTLPxSoZrnNX7bZ0ja37/BN5d0K0Lxs9s5c 522n8edQchcEVqcO6yMCO0B6sVANnxOEXs756k9A8fghCmFt3zDmCxnINUj1FMLTEzyN iI6w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1761907197; x=1762511997; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=fnEn/zNYGpYeH2kzoRoKsGDEwWWCiCH27FEc88gwnrM=; b=hXluNC2q3SQ51/zgLtJ5vL3mr4V9UvWBhdXhGKwd4qlSp+IPnXGTvq4UyB4rwMehXx 3BdjSPE2grKUK5zViVuq2eky1JqJ8juXNy0LkDH8XpJJ9dtNXJtyl4wvNiJNvFfie5xQ 0StqE4wwUKwemWbG3dYW1nzxytJXR/XiY9LnhjC6iB5kd+1Rz4D+eUQTM99MgTfo0rC2 YdDOTji5Am3QAcYREGpwAg/bkfXyAQ3GB3JyhNXAEmbj0EIrD5HDxGIxh29DNHNQMkjB u9gf+hc82gxm17iPpBSFWE2nEtfrdD41L1DMVqiZvKRw0uImN57K5/RVtjWu4QidhT+/ qZfA== X-Gm-Message-State: AOJu0Yww93nseE3mEUc5s8quAqAg2mdnIhvW++qpQSVT/dxypzkPo5zo omRGtuzA/AG49kH1vrfAKITzTV0aSXPnFkHA8NZLYNU5VWJFhV0tE8brZwUUb5XWwJ5atdHhBg= = X-Google-Smtp-Source: AGHT+IHCyP4IoqlcDxNTDlmIlfwEjZNFju0cobcumYETAExfBlb7/GHCab1JzpZ/nMQ3mxmWwEU6aIlU X-Received: from wmbz25.prod.google.com ([2002:a05:600c:c099:b0:477:1449:b093]) (user=ardb job=prod-delivery.src-stubby-dispatcher) by 2002:a05:600c:4e43:b0:477:e66:406e with SMTP id 5b1f17b1804b1-4773088be60mr32292295e9.29.1761907197646; Fri, 31 Oct 2025 03:39:57 -0700 (PDT) Date: Fri, 31 Oct 2025 11:39:11 +0100 In-Reply-To: <20251031103858.529530-23-ardb+git@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20251031103858.529530-23-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=2613; i=ardb@kernel.org; h=from:subject; bh=yue2NgA2zOFgSdHmsGEDM3WvkPWdV3V1tFnMJXN62q8=; b=owGbwMvMwCVmkMcZplerG8N4Wi2JIZNl4pXDLztqpIo6hLcsd73t7SATYpg2w/73MaM52isPZ 9sdCFrbUcrCIMbFICumyCIw+++7nacnStU6z5KFmcPKBDKEgYtTACayZCLD/+hF9jw8SjcfLluw 8djdsI+hFpuVv2Ue7+59fDNvX+yWHa8Z/vvtjXpyx0itOtrFvyVh5VYhrwPVWY/3XEn/972AqXr +Gx4A X-Mailer: git-send-email 2.51.1.930.gacf6e81ea2-goog Message-ID: <20251031103858.529530-35-ardb+git@google.com> Subject: [PATCH v4 12/21] crypto/arm64: aes-gcm - Switch to 'ksimd' scoped guard API From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: linux-kernel@vger.kernel.org, linux-crypto@vger.kernel.org, herbert@gondor.apana.org.au, ebiggers@kernel.org, Ard Biesheuvel Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Ard Biesheuvel Switch to the more abstract 'scoped_ksimd()' API, which will be modified in a future patch to transparently allocate a kernel mode FP/SIMD state buffer on the stack, so that kernel mode FP/SIMD code remains preemptible in principe, but without the memory overhead that adds 528 bytes to the size of struct task_struct. Reviewed-by: Eric Biggers Signed-off-by: Ard Biesheuvel Acked-by: Catalin Marinas --- arch/arm64/crypto/ghash-ce-glue.c | 27 ++++++++++---------- 1 file changed, 13 insertions(+), 14 deletions(-) diff --git a/arch/arm64/crypto/ghash-ce-glue.c b/arch/arm64/crypto/ghash-ce= -glue.c index 4995b6e22335..7951557a285a 100644 --- a/arch/arm64/crypto/ghash-ce-glue.c +++ b/arch/arm64/crypto/ghash-ce-glue.c @@ -5,7 +5,6 @@ * Copyright (C) 2014 - 2018 Linaro Ltd. */ =20 -#include #include #include #include @@ -22,6 +21,8 @@ #include #include =20 +#include + MODULE_DESCRIPTION("GHASH and AES-GCM using ARMv8 Crypto Extensions"); MODULE_AUTHOR("Ard Biesheuvel "); MODULE_LICENSE("GPL v2"); @@ -74,9 +75,8 @@ void ghash_do_simd_update(int blocks, u64 dg[], const cha= r *src, u64 const h[][2], const char *head)) { - kernel_neon_begin(); - simd_update(blocks, dg, src, key->h, head); - kernel_neon_end(); + scoped_ksimd() + simd_update(blocks, dg, src, key->h, head); } =20 /* avoid hogging the CPU for too long */ @@ -329,11 +329,10 @@ static int gcm_encrypt(struct aead_request *req, char= *iv, int assoclen) tag =3D NULL; } =20 - kernel_neon_begin(); - pmull_gcm_encrypt(nbytes, dst, src, ctx->ghash_key.h, - dg, iv, ctx->aes_key.key_enc, nrounds, - tag); - kernel_neon_end(); + scoped_ksimd() + pmull_gcm_encrypt(nbytes, dst, src, ctx->ghash_key.h, + dg, iv, ctx->aes_key.key_enc, nrounds, + tag); =20 if (unlikely(!nbytes)) break; @@ -399,11 +398,11 @@ static int gcm_decrypt(struct aead_request *req, char= *iv, int assoclen) tag =3D NULL; } =20 - kernel_neon_begin(); - ret =3D pmull_gcm_decrypt(nbytes, dst, src, ctx->ghash_key.h, - dg, iv, ctx->aes_key.key_enc, - nrounds, tag, otag, authsize); - kernel_neon_end(); + scoped_ksimd() + ret =3D pmull_gcm_decrypt(nbytes, dst, src, + ctx->ghash_key.h, + dg, iv, ctx->aes_key.key_enc, + nrounds, tag, otag, authsize); =20 if (unlikely(!nbytes)) break; --=20 2.51.1.930.gacf6e81ea2-goog From nobody Fri Dec 19 02:49:54 2025 Received: from mail-wm1-f73.google.com (mail-wm1-f73.google.com [209.85.128.73]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5F3B134678E for ; Fri, 31 Oct 2025 10:40:00 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.73 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1761907202; cv=none; b=bxM/FBAUO79MhEkvBkMvtoIbalBSBTLT32GsZCQjb4MQwj5MWH08i5iUbncA3V6nIQvZbblzzwn2X96FNV6+OlxBQSxocMk9YZ3K3yMx8qY9WXxQTltOgMeySQba5C2wfEvJI9KuSsWJ+noiTSg7gi4xcnMwwyXor4NY4WzhGVk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1761907202; c=relaxed/simple; bh=THSYwS2QbD5A2iM+nFKxv+UIHkNNGdsVXqfTT1Y2lNw=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=df3x0qwFQLfO2P9SkRnRh7NTxYyPlxoy4++YWiFdrRAQAXfKpm6kg23+sdC5vV5mwQLjh6IOFak5A7bG4jetbQVJobbgxUqAOxR5wKPj2Nhm5ax26ADL+MWadaIRStv/SxHnmK+afBCis7nXwVH6wJchx2GDgOkHTosJ5unZwtg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=oNGIc+RJ; arc=none smtp.client-ip=209.85.128.73 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="oNGIc+RJ" Received: by mail-wm1-f73.google.com with SMTP id 5b1f17b1804b1-475da25c4c4so15270305e9.0 for ; Fri, 31 Oct 2025 03:40:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1761907199; x=1762511999; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=4s6P+Feh6QTHXx+j4+pZbRiwHHcjHTLEIbaDo5nXNow=; b=oNGIc+RJeI3rTUXbpxF3Xvix2wKvdyhFE0OHbhRhD+kgyAhdl7xVW9xaGjRQnXdV69 clMpHi/i6EJxJQDLKJwqTgrEiSnEGX+iBlIWKxbB0cHwruqGUSV/Rm41zGpgTiqlqKBH LzHbJP/qa+gXJ1iR/AFgDvucspQXCCbvrMuYUreuWzE+3rpKss34rSxzq4yXOHhuVgT8 jZOE09zlpZW4n8+3jP8duyvJ6CzmDXscGTu/3lMskuDKCdpp+K24ZpQUsvkgR8LsCgd0 JkDHJuiPaNEQ4NOzgbHzDfK5oApTbpQgx/91joQTK+jEh1/wFDPn0sUPKiVHpfrAre7r Mb8g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1761907199; x=1762511999; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=4s6P+Feh6QTHXx+j4+pZbRiwHHcjHTLEIbaDo5nXNow=; b=l412cbgEWhKsETvB9s85cpb8NbEwLQRmsTF0OGCQTtbaApvyUrtVgdjw45Hn3dkgxI y9DHXwFJkR5Z2jI8TvmaJhcZswB93k8I5o0ZTbPyiRcF0MLWksl8Uc6/1XhPvTm5FzSw GcLp2dBP6Y81T2JbCC2pGLR5XeMuLMrdJk9UAYSIGNpjBLSAFLxflWGm1OZtFtRQHKnk Bzrc7jqGpU2EZiRO4y2YABdBiYpLr9fYCFnR2C88fdK89DyVRKpVqZAkDypdkGyFhqO6 82N0Uq4P/N5gmV35n8PpNJKmzyaPkXJY6fwBIykZRz7HZ7TG6sMjwn44kPouyebCw8zZ vyfw== X-Gm-Message-State: AOJu0Ywne8GzsRTb3Qn2Dd7UVHU3vM9pf5UJ7vZpQP+sALkcgu91Qe1K WMEC7jBwyNeyVITaiMFMYG9VbHS9fR17qG5Zi7at2fJToTuFwU50FkXGoZqOEEYeYsInj3f1Ew= = X-Google-Smtp-Source: AGHT+IEfIHxmt0Akn8ZOdGgYUmWAojffoRFxXG63QnY50c3838BNl1o+lCSGk7AdLGAKU776OVcmZYAl X-Received: from wmbf23.prod.google.com ([2002:a05:600c:5957:b0:475:d8ac:bbb0]) (user=ardb job=prod-delivery.src-stubby-dispatcher) by 2002:a05:600c:190e:b0:46e:3cd9:e56f with SMTP id 5b1f17b1804b1-47730802ca0mr29236185e9.6.1761907198711; Fri, 31 Oct 2025 03:39:58 -0700 (PDT) Date: Fri, 31 Oct 2025 11:39:12 +0100 In-Reply-To: <20251031103858.529530-23-ardb+git@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20251031103858.529530-23-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=1189; i=ardb@kernel.org; h=from:subject; bh=gI3hKx/eLJBwHcKg2Z09pXqMlir5s87xW/Yb/W5o4zk=; b=owGbwMvMwCVmkMcZplerG8N4Wi2JIZNl4jWpy2tS+Bo75wd5RN++5pC6flaDZOhdjaXyL29rv ptye2tARykLgxgXg6yYIovA7L/vdp6eKFXrPEsWZg4rE8gQBi5OAZhI7DtGho6N39KFNuivY32j GDljvd68v90bNjKtOyAX6WXkfGla2RxGhlksDTN+Lky+f+d0gcYX0Ud7VrsIxVS0X5aRuJh5ySK YgQ8A X-Mailer: git-send-email 2.51.1.930.gacf6e81ea2-goog Message-ID: <20251031103858.529530-36-ardb+git@google.com> Subject: [PATCH v4 13/21] crypto/arm64: nhpoly1305 - Switch to 'ksimd' scoped guard API From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: linux-kernel@vger.kernel.org, linux-crypto@vger.kernel.org, herbert@gondor.apana.org.au, ebiggers@kernel.org, Ard Biesheuvel Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Ard Biesheuvel Switch to the more abstract 'scoped_ksimd()' API, which will be modified in a future patch to transparently allocate a kernel mode FP/SIMD state buffer on the stack, so that kernel mode FP/SIMD code remains preemptible in principe, but without the memory overhead that adds 528 bytes to the size of struct task_struct. Reviewed-by: Eric Biggers Signed-off-by: Ard Biesheuvel Acked-by: Catalin Marinas --- arch/arm64/crypto/nhpoly1305-neon-glue.c | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/arch/arm64/crypto/nhpoly1305-neon-glue.c b/arch/arm64/crypto/n= hpoly1305-neon-glue.c index e4a0b463f080..013de6ac569a 100644 --- a/arch/arm64/crypto/nhpoly1305-neon-glue.c +++ b/arch/arm64/crypto/nhpoly1305-neon-glue.c @@ -25,9 +25,8 @@ static int nhpoly1305_neon_update(struct shash_desc *desc, do { unsigned int n =3D min_t(unsigned int, srclen, SZ_4K); =20 - kernel_neon_begin(); - crypto_nhpoly1305_update_helper(desc, src, n, nh_neon); - kernel_neon_end(); + scoped_ksimd() + crypto_nhpoly1305_update_helper(desc, src, n, nh_neon); src +=3D n; srclen -=3D n; } while (srclen); --=20 2.51.1.930.gacf6e81ea2-goog From nobody Fri Dec 19 02:49:54 2025 Received: from mail-wr1-f74.google.com (mail-wr1-f74.google.com [209.85.221.74]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E6E6D3469E9 for ; Fri, 31 Oct 2025 10:40:00 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.221.74 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1761907202; cv=none; b=CDVIra9ZbIZ6Kaci+1nvdkce2edtbqsvB05aDwpjVCLQclG6HPWYl4upcol6w6G8WOnYwCY4MjyZgzuU8Ea7+xRGlV0AFVbjtTVO+5QumIqTzQt+ioG5XKg5Gf11wSr6dx2KTAsU76hDQbW4ZrvGSDs7PG8QuLubD8nI8WRKgzQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1761907202; c=relaxed/simple; bh=OTHWBoff2PzsQfKx5tYIGytixloTvdhFhmJWQ7BVRmE=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=tkS19pwdoC9udFUBoH0whaWxdtEfBIeenJ6HElDp+C+FxwyY2D6dWye6ZH7ZKd181cn1nxCBYJ5Qn1pivxcJaJvCxtzOTrZzv9MMGWyS6Psfo/j301pQWA8SXsMJHlSU9wyGNN4LTgv7b307Gjlz7PykdyNqzgNhn6Pnux42Fig= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=JIEniFNI; arc=none smtp.client-ip=209.85.221.74 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="JIEniFNI" Received: by mail-wr1-f74.google.com with SMTP id ffacd0b85a97d-427015f63faso1237859f8f.0 for ; Fri, 31 Oct 2025 03:40:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1761907199; x=1762511999; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=8r3QIAGejqa8NPmAEDiNAoce3SvTXlMmEJ2PuS5pgns=; b=JIEniFNIVqR68or89bm9dzpdB/YUnieWja71JcAlqbhZM+s04A1G3mgDxXdiOPRaLq /TMXg3Ih5DmYoPm7ouK/6UyH7BgkTWNiaArCMFf8z+7FZ1ErMA/xS9f01UnYVCn/MAGI CSPC7IXd/laNZQOXxyP822mXWrlH261M5B22VKfM4cKGDH8WWnXsiYimEy8nESmQm2PV 5mD73nDG7dWiH8O1pZJKvsWwMb2B1JsvH6WfAr+77nHiHcqV3oDn62H50e0zpTPAkrAN UeopT1+Manbmubz0nslOACeIWICRg4/2WpYZpwCNTUU69/xOasjqb1cvDxUPnBMUY8tk V7YA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1761907199; x=1762511999; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=8r3QIAGejqa8NPmAEDiNAoce3SvTXlMmEJ2PuS5pgns=; b=DQXR9sFpRfI0+Vmbkq62JDwriRkWVuXif8crtkmdUdn18KDto5w7WKqH9eRz6LJSiV vJbamk0nVN9MeDCWP+avmu7u3QaEwtCLRZxzmEsPv4IUKZPgNpXMY815ciTTEIMFqWNn Jl9IzHwNHgy7gzhyuenCsl5URKYH7zDVS5mdDS6uNHaIlVxnLPTRYturESJF+c7QIole lJSFPThAqPVA1XMkibu/Qm7VjHwBZ9UGmTYN4eycsEi/U76Z+Z/98iLvnnFJ+wW4Eyv/ X3UVy80YVlAeokKpz4HhJy8WlfSOvFbxJPki/lUxoMnAWw/9gbo2GB/rByXfPdmdTMZl AR2A== X-Gm-Message-State: AOJu0Yw5MMr0TT2vnb8SVgiblywrcNKfl/RQLVuAQP6lGlEStUb5mi9o ElYBBn4j8X1eyoJQlI0U1D7szOAXbdQ1KQ/pfcC8/ker/NR79MCb8kuNzAI4ay/PxRiWcFBTJg= = X-Google-Smtp-Source: AGHT+IHA3+sT6jB9B/KN6RWBYWXqri0+R/04SATB2rPp1zUItDUhniCQ0O44BM/cLxoWHqbEStWH964U X-Received: from wmdd17.prod.google.com ([2002:a05:600c:a211:b0:475:da28:9004]) (user=ardb job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6000:3107:b0:3eb:5ff:cb2e with SMTP id ffacd0b85a97d-429bd6c18c6mr2345917f8f.29.1761907199634; Fri, 31 Oct 2025 03:39:59 -0700 (PDT) Date: Fri, 31 Oct 2025 11:39:13 +0100 In-Reply-To: <20251031103858.529530-23-ardb+git@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20251031103858.529530-23-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=1693; i=ardb@kernel.org; h=from:subject; bh=OwQNFyTd57ahcVrUrGlQonZMFcw8cHUaJECkttM55ZY=; b=owGbwMvMwCVmkMcZplerG8N4Wi2JIZNl4vX9e1blpTIus+YpZ055on81Z6uG6Kzv/3bW/3FVt VieGfW2o5SFQYyLQVZMkUVg9t93O09PlKp1niULM4eVCWQIAxenAEyE/TAjw+ZcN4knofcXpPy4 84prj+PpMyWG05oFHEREzzJYzPdvVWX4Hy/3sbJlSWGW+WfZ7pZugYuliV1rvwfdWbFudeXKb3f zmAA= X-Mailer: git-send-email 2.51.1.930.gacf6e81ea2-goog Message-ID: <20251031103858.529530-37-ardb+git@google.com> Subject: [PATCH v4 14/21] crypto/arm64: polyval - Switch to 'ksimd' scoped guard API From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: linux-kernel@vger.kernel.org, linux-crypto@vger.kernel.org, herbert@gondor.apana.org.au, ebiggers@kernel.org, Ard Biesheuvel Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Ard Biesheuvel Switch to the more abstract 'scoped_ksimd()' API, which will be modified in a future patch to transparently allocate a kernel mode FP/SIMD state buffer on the stack, so that kernel mode FP/SIMD code remains preemptible in principe, but without the memory overhead that adds 528 bytes to the size of struct task_struct. Reviewed-by: Eric Biggers Signed-off-by: Ard Biesheuvel Acked-by: Catalin Marinas --- arch/arm64/crypto/polyval-ce-glue.c | 12 +++++------- 1 file changed, 5 insertions(+), 7 deletions(-) diff --git a/arch/arm64/crypto/polyval-ce-glue.c b/arch/arm64/crypto/polyva= l-ce-glue.c index c4e653688ea0..51eefbe97885 100644 --- a/arch/arm64/crypto/polyval-ce-glue.c +++ b/arch/arm64/crypto/polyval-ce-glue.c @@ -15,7 +15,7 @@ * ARMv8 Crypto Extensions instructions to implement the finite field oper= ations. */ =20 -#include +#include #include #include #include @@ -45,16 +45,14 @@ asmlinkage void pmull_polyval_mul(u8 *op1, const u8 *op= 2); static void internal_polyval_update(const struct polyval_tfm_ctx *keys, const u8 *in, size_t nblocks, u8 *accumulator) { - kernel_neon_begin(); - pmull_polyval_update(keys, in, nblocks, accumulator); - kernel_neon_end(); + scoped_ksimd() + pmull_polyval_update(keys, in, nblocks, accumulator); } =20 static void internal_polyval_mul(u8 *op1, const u8 *op2) { - kernel_neon_begin(); - pmull_polyval_mul(op1, op2); - kernel_neon_end(); + scoped_ksimd() + pmull_polyval_mul(op1, op2); } =20 static int polyval_arm64_setkey(struct crypto_shash *tfm, --=20 2.51.1.930.gacf6e81ea2-goog From nobody Fri Dec 19 02:49:54 2025 Received: from mail-wr1-f74.google.com (mail-wr1-f74.google.com [209.85.221.74]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D83C6346E55 for ; Fri, 31 Oct 2025 10:40:01 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.221.74 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1761907203; cv=none; b=f4f8Zv3CRvf4JZaAKd549QlVfqSq6VcuZKf6TVJki9CmbqUT5BF3p98kt896rO6C0Q6DGhnaT/SeggUDqfjRE7ojUNdGi/d556L8w4Q4q8QsMXsl1B118qXFKigMn0tSMs7oomxBnv6x9V5qd2dt/SXuOc9Yc7aiu7zmBfHz6Zg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1761907203; c=relaxed/simple; bh=PquTLJC8FZmcZb9ob1Ts3zmY8+PS/ldlDEgQ/on24C8=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=Wt6Gx0u3BsscuyM+saCV+H4GzwQnji+pR1ikQyV/nDgjDlLtZLpwMpRGKOb6ZWDfUIanaUlKvXaURx/ZtfDIumShSJ8n7VLhZ9OIA3KBMxgGBtuI8L2FFj9VtIfgXPIsOBwOLGlhFQhRXGCjc4Zdr1Zj/dU50b3IG/tOce9LGco= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=uhzjIFGB; arc=none smtp.client-ip=209.85.221.74 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="uhzjIFGB" Received: by mail-wr1-f74.google.com with SMTP id ffacd0b85a97d-428566218c6so1092824f8f.0 for ; Fri, 31 Oct 2025 03:40:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1761907200; x=1762512000; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=6yWeicN1l2mf2js2fMAiSU2E7NZlQ/I1rzR2l36yFu0=; b=uhzjIFGBEyfs6KirzUqUX3MsSoaMn7oIoquclF7gtX11TA/+L4U0LnitU52khBV+pz et79ZAwaAWoqwc3tfnftBZmNTtbtzfwGgtOb2qLMy08zeQ0qh/ls7+Fztmx9hTckB+v1 N4dLRIh0ARAGR8HvzhBFIVFBBLsDnS07Xtvkcpwu3lF96FPxv/QiRdKOaUfQY+YECegJ UeQ1BehmzmAIPHLnxDQ4qyB2ezNalBIjQ1xrG2EJLpTAUckXWfRgx0Sjlpy877OEnuh4 EIbdblMVKOsnNrZUwQQw62STcn7epmYqQ1fFzi6/GfwAL0ADqO3392NF/FP1qf+awOZH PUYw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1761907200; x=1762512000; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=6yWeicN1l2mf2js2fMAiSU2E7NZlQ/I1rzR2l36yFu0=; b=eFKEEOueWp2EVhyqvbJvOMFDg8HxUQCo63515flVOJyYpZCn/TiozERcHbaBY6xMIU i2LVneSoW6Qa1XLqaJT9XJOP0VrzayKeUHX5nXXVte2HFvOvoTUPuJKYNNLuABLpzAOY 5Df5aw2Cje9wQHG9XdALSVTsO1XoD61Y+m4AWVmwQHhaf1PGSdOx5DT5iYQp3CT0TuM/ GPLOYTx+mwL6BD4gKHDBnVYw37ybRQ8HUqP9BvO3FyicbmobphmzOjWLNElpkptciG8l c/lQbRb3w4kJyovs4NJ6kKaUoezvEU3lXrmBxjFCdFEm9fTG5vlHgtMlElGTz35P4tFy M5/Q== X-Gm-Message-State: AOJu0YzVGzadM59uLW9DyIyx3Crg7XMI6Y+i5/H0aLEzNaWVYHbx1Xmm hapI4sDRtrs4MuVbnBElU/eoBM/wRhwL+cCgGGqBYAupgiNdfzYFNAl6m3nD6hMMPFykZES1og= = X-Google-Smtp-Source: AGHT+IGge7CDFO2OF9s+yRivgMJ3E9pJmbYtIoCyI0ADpE92mt1tOKjrtt06Hn0rVZeuCfWm13CM2Kef X-Received: from wmbh2.prod.google.com ([2002:a05:600c:a102:b0:477:172a:1020]) (user=ardb job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6000:290c:b0:427:9e6:3a76 with SMTP id ffacd0b85a97d-429bd69910emr2488587f8f.36.1761907200510; Fri, 31 Oct 2025 03:40:00 -0700 (PDT) Date: Fri, 31 Oct 2025 11:39:14 +0100 In-Reply-To: <20251031103858.529530-23-ardb+git@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20251031103858.529530-23-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=1531; i=ardb@kernel.org; h=from:subject; bh=HC3oKa6XX6UzW4T61UkwwEg9itjK9pSOgUvVasKYlBk=; b=owGbwMvMwCVmkMcZplerG8N4Wi2JIZNl4k3lqGSf0tTW9dM3lu6fuKnZar2H1FFe5v+PezZ0m Zeta/vfUcrCIMbFICumyCIw+++7nacnStU6z5KFmcPKBDKEgYtTACbytZ3hn21Qzpv8ZRv+Har8 9KmtbbbpjscfoguOVrx0LerWOarod5uRYUuDAgP/9kk7Mvbetlm0MWDyzq03jO+IzUisMv9cprf QmxkA X-Mailer: git-send-email 2.51.1.930.gacf6e81ea2-goog Message-ID: <20251031103858.529530-38-ardb+git@google.com> Subject: [PATCH v4 15/21] crypto/arm64: sha3 - Switch to 'ksimd' scoped guard API From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: linux-kernel@vger.kernel.org, linux-crypto@vger.kernel.org, herbert@gondor.apana.org.au, ebiggers@kernel.org, Ard Biesheuvel Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Ard Biesheuvel Switch to the more abstract 'scoped_ksimd()' API, which will be modified in a future patch to transparently allocate a kernel mode FP/SIMD state buffer on the stack, so that kernel mode FP/SIMD code remains preemptible in principe, but without the memory overhead that adds 528 bytes to the size of struct task_struct. Reviewed-by: Eric Biggers Signed-off-by: Ard Biesheuvel Acked-by: Catalin Marinas --- arch/arm64/crypto/sha3-ce-glue.c | 10 ++++------ 1 file changed, 4 insertions(+), 6 deletions(-) diff --git a/arch/arm64/crypto/sha3-ce-glue.c b/arch/arm64/crypto/sha3-ce-g= lue.c index b4f1001046c9..22732760edd3 100644 --- a/arch/arm64/crypto/sha3-ce-glue.c +++ b/arch/arm64/crypto/sha3-ce-glue.c @@ -46,9 +46,8 @@ static int sha3_update(struct shash_desc *desc, const u8 = *data, do { int rem; =20 - kernel_neon_begin(); - rem =3D sha3_ce_transform(sctx->st, data, blocks, ds); - kernel_neon_end(); + scoped_ksimd() + rem =3D sha3_ce_transform(sctx->st, data, blocks, ds); data +=3D (blocks - rem) * bs; blocks =3D rem; } while (blocks); @@ -73,9 +72,8 @@ static int sha3_finup(struct shash_desc *desc, const u8 *= src, unsigned int len, memset(block + len, 0, bs - len); block[bs - 1] |=3D 0x80; =20 - kernel_neon_begin(); - sha3_ce_transform(sctx->st, block, 1, ds); - kernel_neon_end(); + scoped_ksimd() + sha3_ce_transform(sctx->st, block, 1, ds); memzero_explicit(block , sizeof(block)); =20 for (i =3D 0; i < ds / 8; i++) --=20 2.51.1.930.gacf6e81ea2-goog From nobody Fri Dec 19 02:49:54 2025 Received: from mail-wm1-f73.google.com (mail-wm1-f73.google.com [209.85.128.73]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 08F37346FD3 for ; Fri, 31 Oct 2025 10:40:02 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.73 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1761907206; cv=none; b=Tm7HmzkxvtlusPleUhEIJpe/CjDhkpBT7O8+011UnKzDDvMLt61UVtdsRBo0VOxk12bAeaGxZTMFiupwHnn5Is7XF1nlXXmxcOrNw+bZqY8m2+D8kGX9z4dDxvFXXxe16y9THomtw7sxbS6ynC5G2Y4DRRvd99W6DL1ag9zEdhw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1761907206; c=relaxed/simple; bh=DWK9Vv9/mhwHlh+oHiA0jTxszZXG3zdb9vfDxzgDObw=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=k7Nsh2FUM9FDNbSBE1ZbynX68ztNWg5wAoSne2H0FtVMpocD6H8UGQChUkO5tGCcUobyFKJmjcwB6C0rysvlMtvlu+qLgr2eaB130SxpCgNd7890pq6bVIChvxNZnHHKf/7LjXokYEv0T0Zp2j13XX6pyNRcmmBedhpU/R3RzZI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=F5/zcSqb; arc=none smtp.client-ip=209.85.128.73 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="F5/zcSqb" Received: by mail-wm1-f73.google.com with SMTP id 5b1f17b1804b1-4770e0910e4so14170815e9.0 for ; Fri, 31 Oct 2025 03:40:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1761907201; x=1762512001; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=E137w2jyBQdKQzJ2QNCPA5foha+2UsFo+5XbWLAP07E=; b=F5/zcSqbsBx++a/QC6IolSWe+wqrCCJ3luySCoqPBX/RRqufX9LXserCjLij2dcC0H GPylAbrtYRygl8+LBTUtJuBi8M3RToAI/Sd9NJppYQYd8oWxOlHqu3rB9BaJhjozz/Qf IU9wnpEwCuhgbpw52oMUTcW/s/1AIC0i0qOPnAIxbcPhxX5S1Rhb3TYMV7mT0rftB9JK JhuKuo1NUvVvjO+JB8rRFNQTBzVpS2275Z8uorgZFK1A47fc1P0EJRhh54A+7N+7RSVF 8OfRj0gAz4vg/bQxDyZ1VBYf9sVqFRkC2VwRlw3C1VIjrdXDGbsGbpRSB3BYLtN+zmdS 433w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1761907201; x=1762512001; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=E137w2jyBQdKQzJ2QNCPA5foha+2UsFo+5XbWLAP07E=; b=cyzc3urQYMK1zNQ7VOKMgmV4Vciq6fITdd3eQt5O3gZA6n41N69UhnXlEwbV6kcGMb jaEPpc0W0q/sNhJtiRYbObzKoiMjs7wVPD4XTQqXT5TEMW830q4CgFDOd0FezaPGG3+s AcYviwD/0lYntvphkSnaGdblL6K7qDbc5Es/6MEJ5bmMs6Jpc3nkZ4Ur0DWDbaSD+E34 IVE6tonLqXPFjr3DdNjRBgkZGreCkFigy98FSj9O4WN6Zb1GvOOYYJbPajIO5o01pD0l she2K/eZObYPGtzhw1XjhkCdRc46quJQwTir4AxGfy9GjGJJaVY9BcDY+JALLDDMOmLE mR4Q== X-Gm-Message-State: AOJu0YyPGrDt2ykggotfhBp9oE2aPtBFBP6zLakkXNL0GiDLgUEs9x3I P6p97Ww3I8Gs7sciG7gQs2FtmBe6qAY2etLKSB4rU2NRRsAw5+T1hlZaw+NKV3BpwWrYwn8AZQ= = X-Google-Smtp-Source: AGHT+IGSC7lpglLVQXDhc+qETrFDrUbREN63HeHWglrMlVHBvofBu6xBm//uDsp9EQNcouOaMC5P8NO+ X-Received: from wmbgx15.prod.google.com ([2002:a05:600c:858f:b0:477:e0a:f99e]) (user=ardb job=prod-delivery.src-stubby-dispatcher) by 2002:a05:600c:35cc:b0:475:d8f3:71c1 with SMTP id 5b1f17b1804b1-477308be6f1mr25615455e9.27.1761907201250; Fri, 31 Oct 2025 03:40:01 -0700 (PDT) Date: Fri, 31 Oct 2025 11:39:15 +0100 In-Reply-To: <20251031103858.529530-23-ardb+git@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20251031103858.529530-23-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=3235; i=ardb@kernel.org; h=from:subject; bh=BmcyRj2En4D2bqTYOu3u59S/hVwg5l7rSd12ZpuUTBI=; b=owGbwMvMwCVmkMcZplerG8N4Wi2JIZNl4i3DFR/fNq1Xakt8FWTJdOhpvVsDwyGJ7E3WhxY8W eL7YfO/jlIWBjEuBlkxRRaB2X/f7Tw9UarWeZYszBxWJpAhDFycAjAREXVGhnVfdl04dOv5V9/U g4f3qN4+xbZ6/aX0qa92dd2dOcdIdd4bRoYprzoDmf2F1r7MzJdLrtM6/3EWn+/hW98LH/u82rz 40ixWAA== X-Mailer: git-send-email 2.51.1.930.gacf6e81ea2-goog Message-ID: <20251031103858.529530-39-ardb+git@google.com> Subject: [PATCH v4 16/21] crypto/arm64: sm3 - Switch to 'ksimd' scoped guard API From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: linux-kernel@vger.kernel.org, linux-crypto@vger.kernel.org, herbert@gondor.apana.org.au, ebiggers@kernel.org, Ard Biesheuvel Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Ard Biesheuvel Switch to the more abstract 'scoped_ksimd()' API, which will be modified in a future patch to transparently allocate a kernel mode FP/SIMD state buffer on the stack, so that kernel mode FP/SIMD code remains preemptible in principe, but without the memory overhead that adds 528 bytes to the size of struct task_struct. Reviewed-by: Eric Biggers Signed-off-by: Ard Biesheuvel Acked-by: Catalin Marinas --- arch/arm64/crypto/sm3-ce-glue.c | 15 ++++++++------- arch/arm64/crypto/sm3-neon-glue.c | 16 ++++++---------- 2 files changed, 14 insertions(+), 17 deletions(-) diff --git a/arch/arm64/crypto/sm3-ce-glue.c b/arch/arm64/crypto/sm3-ce-glu= e.c index eac6f5fa0abe..24c1fcfae072 100644 --- a/arch/arm64/crypto/sm3-ce-glue.c +++ b/arch/arm64/crypto/sm3-ce-glue.c @@ -5,7 +5,6 @@ * Copyright (C) 2018 Linaro Ltd */ =20 -#include #include #include #include @@ -13,6 +12,8 @@ #include #include =20 +#include + MODULE_DESCRIPTION("SM3 secure hash using ARMv8 Crypto Extensions"); MODULE_AUTHOR("Ard Biesheuvel "); MODULE_LICENSE("GPL v2"); @@ -25,18 +26,18 @@ static int sm3_ce_update(struct shash_desc *desc, const= u8 *data, { int remain; =20 - kernel_neon_begin(); - remain =3D sm3_base_do_update_blocks(desc, data, len, sm3_ce_transform); - kernel_neon_end(); + scoped_ksimd() { + remain =3D sm3_base_do_update_blocks(desc, data, len, sm3_ce_transform); + } return remain; } =20 static int sm3_ce_finup(struct shash_desc *desc, const u8 *data, unsigned int len, u8 *out) { - kernel_neon_begin(); - sm3_base_do_finup(desc, data, len, sm3_ce_transform); - kernel_neon_end(); + scoped_ksimd() { + sm3_base_do_finup(desc, data, len, sm3_ce_transform); + } return sm3_base_finish(desc, out); } =20 diff --git a/arch/arm64/crypto/sm3-neon-glue.c b/arch/arm64/crypto/sm3-neon= -glue.c index 6c4611a503a3..15f30cc24f32 100644 --- a/arch/arm64/crypto/sm3-neon-glue.c +++ b/arch/arm64/crypto/sm3-neon-glue.c @@ -5,7 +5,7 @@ * Copyright (C) 2022 Tianjia Zhang */ =20 -#include +#include #include #include #include @@ -20,20 +20,16 @@ asmlinkage void sm3_neon_transform(struct sm3_state *ss= t, u8 const *src, static int sm3_neon_update(struct shash_desc *desc, const u8 *data, unsigned int len) { - int remain; - - kernel_neon_begin(); - remain =3D sm3_base_do_update_blocks(desc, data, len, sm3_neon_transform); - kernel_neon_end(); - return remain; + scoped_ksimd() + return sm3_base_do_update_blocks(desc, data, len, + sm3_neon_transform); } =20 static int sm3_neon_finup(struct shash_desc *desc, const u8 *data, unsigned int len, u8 *out) { - kernel_neon_begin(); - sm3_base_do_finup(desc, data, len, sm3_neon_transform); - kernel_neon_end(); + scoped_ksimd() + sm3_base_do_finup(desc, data, len, sm3_neon_transform); return sm3_base_finish(desc, out); } =20 --=20 2.51.1.930.gacf6e81ea2-goog From nobody Fri Dec 19 02:49:54 2025 Received: from mail-wr1-f73.google.com (mail-wr1-f73.google.com [209.85.221.73]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C314A346FC1 for ; Fri, 31 Oct 2025 10:40:03 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.221.73 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1761907208; cv=none; b=cYGKNOmtWNz+IWMdR9q4mySlY4WcanZXokdSTU7LylCV4yBoscuA/kySe/2xLMKQ1CyeZCeuZQTuHWuH1UWuojAjrqB8WDT1+D3XOoIRjFaS+ZCbRNjlXHt3ybHwrEN40bfbiYVBPk7TK+nWJ5YuljM/y7hhImlutsMx8GKll74= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1761907208; c=relaxed/simple; bh=L6keGmWhuHKNO+Dmu93z1gbDsfl8pX3Yp2RLDWDq9Y8=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=IPiy1qateWG1Mp0lK2wI5E1wUh/YN54s7ntXgoXk7PT9fQ2NpPjAjASpI4y7h2BaAFRnw/aOQJ0LDSjwbChzUobnNmqFRhtYBEd/LKKmQFsN7RgWXc/PyILOXZ5EguOdTtSwAxkAzbMvtUCs8oAIEe3iivT7mGBTMounD7yU7bE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=4MAajkgP; arc=none smtp.client-ip=209.85.221.73 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="4MAajkgP" Received: by mail-wr1-f73.google.com with SMTP id ffacd0b85a97d-3f384f10762so2243002f8f.3 for ; Fri, 31 Oct 2025 03:40:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1761907202; x=1762512002; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=uPf4jH2Xq6YglmJt/Z/1H4uAG3AvOY53HgLQBHEzlps=; b=4MAajkgPegW8G7ldPwuNAvT81U7FBuWyd2HCinP8PncKlF+6au66Rjvz3lJUhiuZZe Bc6ww5xR6vZAB5kgJenOz/KQzrsfaUP8ongsIfeb3KDsQjgrZemkPH0USxJe+KJulU5r IIHXWnlrj7c31pKyr4p2DkphP//2QWfnuiLZ097r18RMip7Pw4f0fOQ3YX+3RavsRyYR yYl8e6jgJ8cG/XiDClFz6wH1OrDsltGAsD6mWVIVR3c30BW/Y8j5NYJMcuEibC0yG3rN 3exSnw9uKRcbs/uuHnAzDk72S76dy0dEFToWSjibm/yn9GUM2XdXOS9B10lqAdVd6i1/ uIWA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1761907202; x=1762512002; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=uPf4jH2Xq6YglmJt/Z/1H4uAG3AvOY53HgLQBHEzlps=; b=hTGNEdba3QCVLrs9c5c9Me60ShuoU9i3GkOfEfXRF4zKKzGQnb8tR7LIqWz0KrESOw UCmQeLXHib/F9jOn9goXEwjsb2EJNGUn5W7st2ZND5IRvwfDu0ZdJpFU2+sosjgtRz0o LYA+GcyG+ChHA3gsQCNQ3pArHxly8dZo3jMzxxKKm+NOQdsaBCfCIIvtVfOlltz4H+ju szhd+h76ior6sVyNoAM89JJrS/ufGzSVAJ6WZzOABuGgDcOnTA5J4DBLINtwVhrwzZS9 n3y0qIGytbkhI+hdtk9HwCjoP5HfeoExmemn5ZbsljYYyKiSUoxjthZ6/oX1YvYvx4zw F7JQ== X-Gm-Message-State: AOJu0YzdMxdYwtFBFP7bZHw2VeAYJMG3Yf7LUzph8QHAaK5s6bVKlNdh RQ5yUHNveU0gB2Bj3Ez/YrkhpR1U7HaHCnPuJ85G8AGSPFucqwCUr1pMIIl6goHcDMGT1LEdfQ= = X-Google-Smtp-Source: AGHT+IHBWOfDUYQF0RJYz8tdd5VLz5W14AhZr9qm6OU728YrhW2RvtEkgtdUgRsapicAbOxJrTA/wCjg X-Received: from wmcq27.prod.google.com ([2002:a05:600c:c11b:b0:46f:b32b:55f]) (user=ardb job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6000:430d:b0:3ea:e0fd:290a with SMTP id ffacd0b85a97d-429bd68322dmr2532065f8f.12.1761907202154; Fri, 31 Oct 2025 03:40:02 -0700 (PDT) Date: Fri, 31 Oct 2025 11:39:16 +0100 In-Reply-To: <20251031103858.529530-23-ardb+git@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20251031103858.529530-23-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=17484; i=ardb@kernel.org; h=from:subject; bh=ZhZahtwfNc9VdlqtoGJDsIW5NGC9QTCl2puzpufxwjY=; b=owGbwMvMwCVmkMcZplerG8N4Wi2JIZNl4u3XT5x0PzWdb+7d/Tb09vpLT49919h+/K62M1Op7 8NKNafjHaUsDGJcDLJiiiwCs/++23l6olSt8yxZmDmsTCBDGLg4BWAiuUGMDHum7S8KXSUqPGHK nUdfDm9e0iLT48HaFDtvqpfm7mQOu30M/1SYmJuO33YsY5V0dVDfeT0j/9Gt1Zv0PhadSrn2V4r hEyMA X-Mailer: git-send-email 2.51.1.930.gacf6e81ea2-goog Message-ID: <20251031103858.529530-40-ardb+git@google.com> Subject: [PATCH v4 17/21] crypto/arm64: sm4 - Switch to 'ksimd' scoped guard API From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: linux-kernel@vger.kernel.org, linux-crypto@vger.kernel.org, herbert@gondor.apana.org.au, ebiggers@kernel.org, Ard Biesheuvel Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Ard Biesheuvel Switch to the more abstract 'scoped_ksimd()' API, which will be modified in a future patch to transparently allocate a kernel mode FP/SIMD state buffer on the stack, so that kernel mode FP/SIMD code remains preemptible in principe, but without the memory overhead that adds 528 bytes to the size of struct task_struct. Reviewed-by: Eric Biggers Signed-off-by: Ard Biesheuvel Acked-by: Catalin Marinas --- arch/arm64/crypto/sm4-ce-ccm-glue.c | 38 ++-- arch/arm64/crypto/sm4-ce-cipher-glue.c | 10 +- arch/arm64/crypto/sm4-ce-gcm-glue.c | 53 +++-- arch/arm64/crypto/sm4-ce-glue.c | 214 +++++++++----------- arch/arm64/crypto/sm4-neon-glue.c | 25 +-- 5 files changed, 149 insertions(+), 191 deletions(-) diff --git a/arch/arm64/crypto/sm4-ce-ccm-glue.c b/arch/arm64/crypto/sm4-ce= -ccm-glue.c index e92cbdf1aaee..332f02167a96 100644 --- a/arch/arm64/crypto/sm4-ce-ccm-glue.c +++ b/arch/arm64/crypto/sm4-ce-ccm-glue.c @@ -11,7 +11,7 @@ #include #include #include -#include +#include #include #include #include @@ -35,10 +35,9 @@ static int ccm_setkey(struct crypto_aead *tfm, const u8 = *key, if (key_len !=3D SM4_KEY_SIZE) return -EINVAL; =20 - kernel_neon_begin(); - sm4_ce_expand_key(key, ctx->rkey_enc, ctx->rkey_dec, - crypto_sm4_fk, crypto_sm4_ck); - kernel_neon_end(); + scoped_ksimd() + sm4_ce_expand_key(key, ctx->rkey_enc, ctx->rkey_dec, + crypto_sm4_fk, crypto_sm4_ck); =20 return 0; } @@ -167,28 +166,25 @@ static int ccm_crypt(struct aead_request *req, struct= skcipher_walk *walk, memcpy(ctr0, walk->iv, SM4_BLOCK_SIZE); crypto_inc(walk->iv, SM4_BLOCK_SIZE); =20 - kernel_neon_begin(); + scoped_ksimd() { + if (req->assoclen) + ccm_calculate_auth_mac(req, mac); =20 - if (req->assoclen) - ccm_calculate_auth_mac(req, mac); - - while (walk->nbytes) { - unsigned int tail =3D walk->nbytes % SM4_BLOCK_SIZE; + while (walk->nbytes) { + unsigned int tail =3D walk->nbytes % SM4_BLOCK_SIZE; =20 - if (walk->nbytes =3D=3D walk->total) - tail =3D 0; + if (walk->nbytes =3D=3D walk->total) + tail =3D 0; =20 - sm4_ce_ccm_crypt(rkey_enc, walk->dst.virt.addr, - walk->src.virt.addr, walk->iv, - walk->nbytes - tail, mac); + sm4_ce_ccm_crypt(rkey_enc, walk->dst.virt.addr, + walk->src.virt.addr, walk->iv, + walk->nbytes - tail, mac); =20 - err =3D skcipher_walk_done(walk, tail); + err =3D skcipher_walk_done(walk, tail); + } + sm4_ce_ccm_final(rkey_enc, ctr0, mac); } =20 - sm4_ce_ccm_final(rkey_enc, ctr0, mac); - - kernel_neon_end(); - return err; } =20 diff --git a/arch/arm64/crypto/sm4-ce-cipher-glue.c b/arch/arm64/crypto/sm4= -ce-cipher-glue.c index c31d76fb5a17..bceec833ef4e 100644 --- a/arch/arm64/crypto/sm4-ce-cipher-glue.c +++ b/arch/arm64/crypto/sm4-ce-cipher-glue.c @@ -32,9 +32,8 @@ static void sm4_ce_encrypt(struct crypto_tfm *tfm, u8 *ou= t, const u8 *in) if (!crypto_simd_usable()) { sm4_crypt_block(ctx->rkey_enc, out, in); } else { - kernel_neon_begin(); - sm4_ce_do_crypt(ctx->rkey_enc, out, in); - kernel_neon_end(); + scoped_ksimd() + sm4_ce_do_crypt(ctx->rkey_enc, out, in); } } =20 @@ -45,9 +44,8 @@ static void sm4_ce_decrypt(struct crypto_tfm *tfm, u8 *ou= t, const u8 *in) if (!crypto_simd_usable()) { sm4_crypt_block(ctx->rkey_dec, out, in); } else { - kernel_neon_begin(); - sm4_ce_do_crypt(ctx->rkey_dec, out, in); - kernel_neon_end(); + scoped_ksimd() + sm4_ce_do_crypt(ctx->rkey_dec, out, in); } } =20 diff --git a/arch/arm64/crypto/sm4-ce-gcm-glue.c b/arch/arm64/crypto/sm4-ce= -gcm-glue.c index 8f6fc8c33c3f..ef06f4f768a1 100644 --- a/arch/arm64/crypto/sm4-ce-gcm-glue.c +++ b/arch/arm64/crypto/sm4-ce-gcm-glue.c @@ -11,7 +11,7 @@ #include #include #include -#include +#include #include #include #include @@ -48,13 +48,11 @@ static int gcm_setkey(struct crypto_aead *tfm, const u8= *key, if (key_len !=3D SM4_KEY_SIZE) return -EINVAL; =20 - kernel_neon_begin(); - - sm4_ce_expand_key(key, ctx->key.rkey_enc, ctx->key.rkey_dec, - crypto_sm4_fk, crypto_sm4_ck); - sm4_ce_pmull_ghash_setup(ctx->key.rkey_enc, ctx->ghash_table); - - kernel_neon_end(); + scoped_ksimd() { + sm4_ce_expand_key(key, ctx->key.rkey_enc, ctx->key.rkey_dec, + crypto_sm4_fk, crypto_sm4_ck); + sm4_ce_pmull_ghash_setup(ctx->key.rkey_enc, ctx->ghash_table); + } return 0; } =20 @@ -149,31 +147,28 @@ static int gcm_crypt(struct aead_request *req, struct= skcipher_walk *walk, memcpy(iv, req->iv, GCM_IV_SIZE); put_unaligned_be32(2, iv + GCM_IV_SIZE); =20 - kernel_neon_begin(); + scoped_ksimd() { + if (req->assoclen) + gcm_calculate_auth_mac(req, ghash); =20 - if (req->assoclen) - gcm_calculate_auth_mac(req, ghash); + do { + unsigned int tail =3D walk->nbytes % SM4_BLOCK_SIZE; + const u8 *src =3D walk->src.virt.addr; + u8 *dst =3D walk->dst.virt.addr; + const u8 *l =3D NULL; =20 - do { - unsigned int tail =3D walk->nbytes % SM4_BLOCK_SIZE; - const u8 *src =3D walk->src.virt.addr; - u8 *dst =3D walk->dst.virt.addr; - const u8 *l =3D NULL; - - if (walk->nbytes =3D=3D walk->total) { - l =3D (const u8 *)&lengths; - tail =3D 0; - } - - sm4_ce_pmull_gcm_crypt(ctx->key.rkey_enc, dst, src, iv, - walk->nbytes - tail, ghash, - ctx->ghash_table, l); - - err =3D skcipher_walk_done(walk, tail); - } while (walk->nbytes); + if (walk->nbytes =3D=3D walk->total) { + l =3D (const u8 *)&lengths; + tail =3D 0; + } =20 - kernel_neon_end(); + sm4_ce_pmull_gcm_crypt(ctx->key.rkey_enc, dst, src, iv, + walk->nbytes - tail, ghash, + ctx->ghash_table, l); =20 + err =3D skcipher_walk_done(walk, tail); + } while (walk->nbytes); + } return err; } =20 diff --git a/arch/arm64/crypto/sm4-ce-glue.c b/arch/arm64/crypto/sm4-ce-glu= e.c index 7a60e7b559dc..5569cece5a0b 100644 --- a/arch/arm64/crypto/sm4-ce-glue.c +++ b/arch/arm64/crypto/sm4-ce-glue.c @@ -8,7 +8,7 @@ * Copyright (C) 2022 Tianjia Zhang */ =20 -#include +#include #include #include #include @@ -74,10 +74,9 @@ static int sm4_setkey(struct crypto_skcipher *tfm, const= u8 *key, if (key_len !=3D SM4_KEY_SIZE) return -EINVAL; =20 - kernel_neon_begin(); - sm4_ce_expand_key(key, ctx->rkey_enc, ctx->rkey_dec, - crypto_sm4_fk, crypto_sm4_ck); - kernel_neon_end(); + scoped_ksimd() + sm4_ce_expand_key(key, ctx->rkey_enc, ctx->rkey_dec, + crypto_sm4_fk, crypto_sm4_ck); return 0; } =20 @@ -94,12 +93,12 @@ static int sm4_xts_setkey(struct crypto_skcipher *tfm, = const u8 *key, if (ret) return ret; =20 - kernel_neon_begin(); - sm4_ce_expand_key(key, ctx->key1.rkey_enc, - ctx->key1.rkey_dec, crypto_sm4_fk, crypto_sm4_ck); - sm4_ce_expand_key(&key[SM4_KEY_SIZE], ctx->key2.rkey_enc, - ctx->key2.rkey_dec, crypto_sm4_fk, crypto_sm4_ck); - kernel_neon_end(); + scoped_ksimd() { + sm4_ce_expand_key(key, ctx->key1.rkey_enc, + ctx->key1.rkey_dec, crypto_sm4_fk, crypto_sm4_ck); + sm4_ce_expand_key(&key[SM4_KEY_SIZE], ctx->key2.rkey_enc, + ctx->key2.rkey_dec, crypto_sm4_fk, crypto_sm4_ck); + } =20 return 0; } @@ -117,16 +116,14 @@ static int sm4_ecb_do_crypt(struct skcipher_request *= req, const u32 *rkey) u8 *dst =3D walk.dst.virt.addr; unsigned int nblks; =20 - kernel_neon_begin(); - - nblks =3D BYTES2BLKS(nbytes); - if (nblks) { - sm4_ce_crypt(rkey, dst, src, nblks); - nbytes -=3D nblks * SM4_BLOCK_SIZE; + scoped_ksimd() { + nblks =3D BYTES2BLKS(nbytes); + if (nblks) { + sm4_ce_crypt(rkey, dst, src, nblks); + nbytes -=3D nblks * SM4_BLOCK_SIZE; + } } =20 - kernel_neon_end(); - err =3D skcipher_walk_done(&walk, nbytes); } =20 @@ -167,16 +164,14 @@ static int sm4_cbc_crypt(struct skcipher_request *req, =20 nblocks =3D nbytes / SM4_BLOCK_SIZE; if (nblocks) { - kernel_neon_begin(); - - if (encrypt) - sm4_ce_cbc_enc(ctx->rkey_enc, dst, src, - walk.iv, nblocks); - else - sm4_ce_cbc_dec(ctx->rkey_dec, dst, src, - walk.iv, nblocks); - - kernel_neon_end(); + scoped_ksimd() { + if (encrypt) + sm4_ce_cbc_enc(ctx->rkey_enc, dst, src, + walk.iv, nblocks); + else + sm4_ce_cbc_dec(ctx->rkey_dec, dst, src, + walk.iv, nblocks); + } } =20 err =3D skcipher_walk_done(&walk, nbytes % SM4_BLOCK_SIZE); @@ -249,16 +244,14 @@ static int sm4_cbc_cts_crypt(struct skcipher_request = *req, bool encrypt) if (err) return err; =20 - kernel_neon_begin(); - - if (encrypt) - sm4_ce_cbc_cts_enc(ctx->rkey_enc, walk.dst.virt.addr, - walk.src.virt.addr, walk.iv, walk.nbytes); - else - sm4_ce_cbc_cts_dec(ctx->rkey_dec, walk.dst.virt.addr, - walk.src.virt.addr, walk.iv, walk.nbytes); - - kernel_neon_end(); + scoped_ksimd() { + if (encrypt) + sm4_ce_cbc_cts_enc(ctx->rkey_enc, walk.dst.virt.addr, + walk.src.virt.addr, walk.iv, walk.nbytes); + else + sm4_ce_cbc_cts_dec(ctx->rkey_dec, walk.dst.virt.addr, + walk.src.virt.addr, walk.iv, walk.nbytes); + } =20 return skcipher_walk_done(&walk, 0); } @@ -288,28 +281,26 @@ static int sm4_ctr_crypt(struct skcipher_request *req) u8 *dst =3D walk.dst.virt.addr; unsigned int nblks; =20 - kernel_neon_begin(); - - nblks =3D BYTES2BLKS(nbytes); - if (nblks) { - sm4_ce_ctr_enc(ctx->rkey_enc, dst, src, walk.iv, nblks); - dst +=3D nblks * SM4_BLOCK_SIZE; - src +=3D nblks * SM4_BLOCK_SIZE; - nbytes -=3D nblks * SM4_BLOCK_SIZE; - } - - /* tail */ - if (walk.nbytes =3D=3D walk.total && nbytes > 0) { - u8 keystream[SM4_BLOCK_SIZE]; - - sm4_ce_crypt_block(ctx->rkey_enc, keystream, walk.iv); - crypto_inc(walk.iv, SM4_BLOCK_SIZE); - crypto_xor_cpy(dst, src, keystream, nbytes); - nbytes =3D 0; + scoped_ksimd() { + nblks =3D BYTES2BLKS(nbytes); + if (nblks) { + sm4_ce_ctr_enc(ctx->rkey_enc, dst, src, walk.iv, nblks); + dst +=3D nblks * SM4_BLOCK_SIZE; + src +=3D nblks * SM4_BLOCK_SIZE; + nbytes -=3D nblks * SM4_BLOCK_SIZE; + } + + /* tail */ + if (walk.nbytes =3D=3D walk.total && nbytes > 0) { + u8 keystream[SM4_BLOCK_SIZE]; + + sm4_ce_crypt_block(ctx->rkey_enc, keystream, walk.iv); + crypto_inc(walk.iv, SM4_BLOCK_SIZE); + crypto_xor_cpy(dst, src, keystream, nbytes); + nbytes =3D 0; + } } =20 - kernel_neon_end(); - err =3D skcipher_walk_done(&walk, nbytes); } =20 @@ -359,18 +350,16 @@ static int sm4_xts_crypt(struct skcipher_request *req= , bool encrypt) if (nbytes < walk.total) nbytes &=3D ~(SM4_BLOCK_SIZE - 1); =20 - kernel_neon_begin(); - - if (encrypt) - sm4_ce_xts_enc(ctx->key1.rkey_enc, walk.dst.virt.addr, - walk.src.virt.addr, walk.iv, nbytes, - rkey2_enc); - else - sm4_ce_xts_dec(ctx->key1.rkey_dec, walk.dst.virt.addr, - walk.src.virt.addr, walk.iv, nbytes, - rkey2_enc); - - kernel_neon_end(); + scoped_ksimd() { + if (encrypt) + sm4_ce_xts_enc(ctx->key1.rkey_enc, walk.dst.virt.addr, + walk.src.virt.addr, walk.iv, nbytes, + rkey2_enc); + else + sm4_ce_xts_dec(ctx->key1.rkey_dec, walk.dst.virt.addr, + walk.src.virt.addr, walk.iv, nbytes, + rkey2_enc); + } =20 rkey2_enc =3D NULL; =20 @@ -395,18 +384,16 @@ static int sm4_xts_crypt(struct skcipher_request *req= , bool encrypt) if (err) return err; =20 - kernel_neon_begin(); - - if (encrypt) - sm4_ce_xts_enc(ctx->key1.rkey_enc, walk.dst.virt.addr, - walk.src.virt.addr, walk.iv, walk.nbytes, - rkey2_enc); - else - sm4_ce_xts_dec(ctx->key1.rkey_dec, walk.dst.virt.addr, - walk.src.virt.addr, walk.iv, walk.nbytes, - rkey2_enc); - - kernel_neon_end(); + scoped_ksimd() { + if (encrypt) + sm4_ce_xts_enc(ctx->key1.rkey_enc, walk.dst.virt.addr, + walk.src.virt.addr, walk.iv, walk.nbytes, + rkey2_enc); + else + sm4_ce_xts_dec(ctx->key1.rkey_dec, walk.dst.virt.addr, + walk.src.virt.addr, walk.iv, walk.nbytes, + rkey2_enc); + } =20 return skcipher_walk_done(&walk, 0); } @@ -510,11 +497,9 @@ static int sm4_cbcmac_setkey(struct crypto_shash *tfm,= const u8 *key, if (key_len !=3D SM4_KEY_SIZE) return -EINVAL; =20 - kernel_neon_begin(); - sm4_ce_expand_key(key, ctx->key.rkey_enc, ctx->key.rkey_dec, - crypto_sm4_fk, crypto_sm4_ck); - kernel_neon_end(); - + scoped_ksimd() + sm4_ce_expand_key(key, ctx->key.rkey_enc, ctx->key.rkey_dec, + crypto_sm4_fk, crypto_sm4_ck); return 0; } =20 @@ -530,15 +515,13 @@ static int sm4_cmac_setkey(struct crypto_shash *tfm, = const u8 *key, =20 memset(consts, 0, SM4_BLOCK_SIZE); =20 - kernel_neon_begin(); - - sm4_ce_expand_key(key, ctx->key.rkey_enc, ctx->key.rkey_dec, - crypto_sm4_fk, crypto_sm4_ck); + scoped_ksimd() { + sm4_ce_expand_key(key, ctx->key.rkey_enc, ctx->key.rkey_dec, + crypto_sm4_fk, crypto_sm4_ck); =20 - /* encrypt the zero block */ - sm4_ce_crypt_block(ctx->key.rkey_enc, (u8 *)consts, (const u8 *)consts); - - kernel_neon_end(); + /* encrypt the zero block */ + sm4_ce_crypt_block(ctx->key.rkey_enc, (u8 *)consts, (const u8 *)consts); + } =20 /* gf(2^128) multiply zero-ciphertext with u and u^2 */ a =3D be64_to_cpu(consts[0].a); @@ -568,18 +551,16 @@ static int sm4_xcbc_setkey(struct crypto_shash *tfm, = const u8 *key, if (key_len !=3D SM4_KEY_SIZE) return -EINVAL; =20 - kernel_neon_begin(); - - sm4_ce_expand_key(key, ctx->key.rkey_enc, ctx->key.rkey_dec, - crypto_sm4_fk, crypto_sm4_ck); + scoped_ksimd() { + sm4_ce_expand_key(key, ctx->key.rkey_enc, ctx->key.rkey_dec, + crypto_sm4_fk, crypto_sm4_ck); =20 - sm4_ce_crypt_block(ctx->key.rkey_enc, key2, ks[0]); - sm4_ce_crypt(ctx->key.rkey_enc, ctx->consts, ks[1], 2); + sm4_ce_crypt_block(ctx->key.rkey_enc, key2, ks[0]); + sm4_ce_crypt(ctx->key.rkey_enc, ctx->consts, ks[1], 2); =20 - sm4_ce_expand_key(key2, ctx->key.rkey_enc, ctx->key.rkey_dec, - crypto_sm4_fk, crypto_sm4_ck); - - kernel_neon_end(); + sm4_ce_expand_key(key2, ctx->key.rkey_enc, ctx->key.rkey_dec, + crypto_sm4_fk, crypto_sm4_ck); + } =20 return 0; } @@ -600,10 +581,9 @@ static int sm4_mac_update(struct shash_desc *desc, con= st u8 *p, unsigned int nblocks =3D len / SM4_BLOCK_SIZE; =20 len %=3D SM4_BLOCK_SIZE; - kernel_neon_begin(); - sm4_ce_mac_update(tctx->key.rkey_enc, ctx->digest, p, - nblocks, false, true); - kernel_neon_end(); + scoped_ksimd() + sm4_ce_mac_update(tctx->key.rkey_enc, ctx->digest, p, + nblocks, false, true); return len; } =20 @@ -619,10 +599,9 @@ static int sm4_cmac_finup(struct shash_desc *desc, con= st u8 *src, ctx->digest[len] ^=3D 0x80; consts +=3D SM4_BLOCK_SIZE; } - kernel_neon_begin(); - sm4_ce_mac_update(tctx->key.rkey_enc, ctx->digest, consts, 1, - false, true); - kernel_neon_end(); + scoped_ksimd() + sm4_ce_mac_update(tctx->key.rkey_enc, ctx->digest, consts, 1, + false, true); memcpy(out, ctx->digest, SM4_BLOCK_SIZE); return 0; } @@ -635,10 +614,9 @@ static int sm4_cbcmac_finup(struct shash_desc *desc, c= onst u8 *src, =20 if (len) { crypto_xor(ctx->digest, src, len); - kernel_neon_begin(); - sm4_ce_crypt_block(tctx->key.rkey_enc, ctx->digest, - ctx->digest); - kernel_neon_end(); + scoped_ksimd() + sm4_ce_crypt_block(tctx->key.rkey_enc, ctx->digest, + ctx->digest); } memcpy(out, ctx->digest, SM4_BLOCK_SIZE); return 0; diff --git a/arch/arm64/crypto/sm4-neon-glue.c b/arch/arm64/crypto/sm4-neon= -glue.c index e3500aca2d18..e944c2a2efb0 100644 --- a/arch/arm64/crypto/sm4-neon-glue.c +++ b/arch/arm64/crypto/sm4-neon-glue.c @@ -48,11 +48,8 @@ static int sm4_ecb_do_crypt(struct skcipher_request *req= , const u32 *rkey) =20 nblocks =3D nbytes / SM4_BLOCK_SIZE; if (nblocks) { - kernel_neon_begin(); - - sm4_neon_crypt(rkey, dst, src, nblocks); - - kernel_neon_end(); + scoped_ksimd() + sm4_neon_crypt(rkey, dst, src, nblocks); } =20 err =3D skcipher_walk_done(&walk, nbytes % SM4_BLOCK_SIZE); @@ -126,12 +123,9 @@ static int sm4_cbc_decrypt(struct skcipher_request *re= q) =20 nblocks =3D nbytes / SM4_BLOCK_SIZE; if (nblocks) { - kernel_neon_begin(); - - sm4_neon_cbc_dec(ctx->rkey_dec, dst, src, - walk.iv, nblocks); - - kernel_neon_end(); + scoped_ksimd() + sm4_neon_cbc_dec(ctx->rkey_dec, dst, src, + walk.iv, nblocks); } =20 err =3D skcipher_walk_done(&walk, nbytes % SM4_BLOCK_SIZE); @@ -157,12 +151,9 @@ static int sm4_ctr_crypt(struct skcipher_request *req) =20 nblocks =3D nbytes / SM4_BLOCK_SIZE; if (nblocks) { - kernel_neon_begin(); - - sm4_neon_ctr_crypt(ctx->rkey_enc, dst, src, - walk.iv, nblocks); - - kernel_neon_end(); + scoped_ksimd() + sm4_neon_ctr_crypt(ctx->rkey_enc, dst, src, + walk.iv, nblocks); =20 dst +=3D nblocks * SM4_BLOCK_SIZE; src +=3D nblocks * SM4_BLOCK_SIZE; --=20 2.51.1.930.gacf6e81ea2-goog From nobody Fri Dec 19 02:49:54 2025 Received: from mail-wm1-f73.google.com (mail-wm1-f73.google.com [209.85.128.73]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 91FCD346E5B for ; Fri, 31 Oct 2025 10:40:04 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.73 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1761907206; cv=none; b=OErCoxNDR7eg3XHpyslnAMyrjmakDN+25jwe8vrUNFCorHicCUZc/ItcMl/dGaPHe7FeJBodc3WKeWbkMtszkGwqIMfeKuOGEtiClfgQ+HjpxN/qKP2wf52fqKax7vcGl30kAjONbRoqjwksP3K9VZTNlZ7GUxdoMmdlhDyEEGA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1761907206; c=relaxed/simple; bh=72D7ld0QaLVbpLWswxFeBab7Xjr8fyBYpl2QMGIhIQU=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=pYMklsh/U8VlkkzhsCI66wMfkGlYx2WBoXyNahwtCZPA5IJo99i6vPrSQVtmLGtTdUcM0FG1kF0uXC5o/sfFjbdyCBWN6GtrYklSo5PnuywUTsOX1pMBaLe9JKug8BrnSmpatarL+sl978tJwUJkFcydgFwT/xKlhiexUgpm4hk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=ffrGY6hU; arc=none smtp.client-ip=209.85.128.73 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="ffrGY6hU" Received: by mail-wm1-f73.google.com with SMTP id 5b1f17b1804b1-475e032d81bso18335365e9.3 for ; Fri, 31 Oct 2025 03:40:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1761907203; x=1762512003; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=7EBCI2bCGa2uMBxuOl4BNLMr0TWQdRgwa7kpYQ+xRQo=; b=ffrGY6hUW6tdCwPBzv1UKROgJEdUkFtTfJTlX8YqDqyye8MuuiE9OKjy4IDKTMnQH+ 8d9nrZVm1bkdcZ+a5JgHxnFeS7QJb2D6MH8duEZ8UXM9AXzpt/x/QsKHwpnqf+AAajeY 6BmKp0wPPN/4gg0/+cubkFXFksIuPV0uI7NkAHi0rrtppjFZwr+3jghXiPbydH+Ch/FC dm2gC+4u35T7I3UxNiny9iDEXXi6cqxeVE0Gz/ofRz++/PJWeJDnCaLnksVjw34jD7dr Na3074kgQyYSAVGiCGyPFbrim7ylc6QnrQu7CMbRQ7KZJgA+Kqz5H5kSYKq2OiRyPnK+ pw9w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1761907203; x=1762512003; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=7EBCI2bCGa2uMBxuOl4BNLMr0TWQdRgwa7kpYQ+xRQo=; b=Ap1hfH/ACQ47daKXPJy3I+EmsYKJaLnlhpGiCi1wKyH4TNTAtdTwJ6irLOBRCpw5Y5 ldiBKATAwdiwM+WNN3z0K3OnvtcWZaai0Ak152SGMTf94KK/BB3lsHoDZl2xQ5J6fMTj sg94R2dRKaM3wGMOc61/Wf/xDOxMYxZjQEBcre9fciSHdTC5puu8Na770Du7oCgfGf0Z CGeXTUOKhVFRGQmKbigMVzYXDTELx9ZHEj2Sz8vxU1oNczC4iqo/DP0/NXkpJXw3P370 bOMR81b/DLCpawA0VUgqJWhWcr0VYfQF1bwUzCWWLAKpWvfo1QSiWda8ebIGnMXp2KDM U8xQ== X-Gm-Message-State: AOJu0YymC0x9VAvm8Kb0YtS/791M71lLejporUUXU/uEf9qawYLw0uwn vb2/9iyNQYJS6ZKJIyB1sN3i8ShsJwDdV7Ko6jP+SE497cOFq3mEe5Db8Lk792/h3AA0+FMdTg= = X-Google-Smtp-Source: AGHT+IEsXL7pay8yayVs9zynCOgQoK1Z8jXHUNfyuqH/t2/LwKAhQiBuGbloXMNB6A2V6Qfym6M16bpZ X-Received: from wmcq10.prod.google.com ([2002:a05:600c:c10a:b0:475:de83:8f42]) (user=ardb job=prod-delivery.src-stubby-dispatcher) by 2002:a05:600c:190e:b0:46e:2562:e7b8 with SMTP id 5b1f17b1804b1-4773086e447mr24492465e9.21.1761907203196; Fri, 31 Oct 2025 03:40:03 -0700 (PDT) Date: Fri, 31 Oct 2025 11:39:17 +0100 In-Reply-To: <20251031103858.529530-23-ardb+git@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20251031103858.529530-23-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=2401; i=ardb@kernel.org; h=from:subject; bh=+pRFDT2DZ8ceDoxhoXJcyM9jtfEdqdQN8vIsBlZZ/gU=; b=owGbwMvMwCVmkMcZplerG8N4Wi2JIZNl4t1Dt15MeJZtLZJ3Lzvy9fFThtcuKypWxDt61IdM+ cdQEmrYUcrCIMbFICumyCIw+++7nacnStU6z5KFmcPKBDKEgYtTACZy8iYjw6+vB+x4V7qGiTeI G9gKXzI9xNa6iaNu36esCEmj/TPP6TMyHGXxmszGJcf3Y+WEDUrhOzWLv/x3+6d9ImZ/w8awfis bfgA= X-Mailer: git-send-email 2.51.1.930.gacf6e81ea2-goog Message-ID: <20251031103858.529530-41-ardb+git@google.com> Subject: [PATCH v4 18/21] arm64/xorblocks: Switch to 'ksimd' scoped guard API From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: linux-kernel@vger.kernel.org, linux-crypto@vger.kernel.org, herbert@gondor.apana.org.au, ebiggers@kernel.org, Ard Biesheuvel Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Ard Biesheuvel Switch to the more abstract 'scoped_ksimd()' API, which will be modified in a future patch to transparently allocate a kernel mode FP/SIMD state buffer on the stack, so that kernel mode FP/SIMD code remains preemptible in principe, but without the memory overhead that adds 528 bytes to the size of struct task_struct. Reviewed-by: Eric Biggers Signed-off-by: Ard Biesheuvel Acked-by: Catalin Marinas --- arch/arm64/include/asm/xor.h | 22 ++++++++------------ 1 file changed, 9 insertions(+), 13 deletions(-) diff --git a/arch/arm64/include/asm/xor.h b/arch/arm64/include/asm/xor.h index befcd8a7abc9..c38e3d017a79 100644 --- a/arch/arm64/include/asm/xor.h +++ b/arch/arm64/include/asm/xor.h @@ -9,7 +9,7 @@ #include #include #include -#include +#include =20 #ifdef CONFIG_KERNEL_MODE_NEON =20 @@ -19,9 +19,8 @@ static void xor_neon_2(unsigned long bytes, unsigned long * __restrict p1, const unsigned long * __restrict p2) { - kernel_neon_begin(); - xor_block_inner_neon.do_2(bytes, p1, p2); - kernel_neon_end(); + scoped_ksimd() + xor_block_inner_neon.do_2(bytes, p1, p2); } =20 static void @@ -29,9 +28,8 @@ xor_neon_3(unsigned long bytes, unsigned long * __restric= t p1, const unsigned long * __restrict p2, const unsigned long * __restrict p3) { - kernel_neon_begin(); - xor_block_inner_neon.do_3(bytes, p1, p2, p3); - kernel_neon_end(); + scoped_ksimd() + xor_block_inner_neon.do_3(bytes, p1, p2, p3); } =20 static void @@ -40,9 +38,8 @@ xor_neon_4(unsigned long bytes, unsigned long * __restric= t p1, const unsigned long * __restrict p3, const unsigned long * __restrict p4) { - kernel_neon_begin(); - xor_block_inner_neon.do_4(bytes, p1, p2, p3, p4); - kernel_neon_end(); + scoped_ksimd() + xor_block_inner_neon.do_4(bytes, p1, p2, p3, p4); } =20 static void @@ -52,9 +49,8 @@ xor_neon_5(unsigned long bytes, unsigned long * __restric= t p1, const unsigned long * __restrict p4, const unsigned long * __restrict p5) { - kernel_neon_begin(); - xor_block_inner_neon.do_5(bytes, p1, p2, p3, p4, p5); - kernel_neon_end(); + scoped_ksimd() + xor_block_inner_neon.do_5(bytes, p1, p2, p3, p4, p5); } =20 static struct xor_block_template xor_block_arm64 =3D { --=20 2.51.1.930.gacf6e81ea2-goog From nobody Fri Dec 19 02:49:54 2025 Received: from mail-wm1-f74.google.com (mail-wm1-f74.google.com [209.85.128.74]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 729B9347FED for ; Fri, 31 Oct 2025 10:40:05 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.74 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1761907208; cv=none; b=h1sxj8E1OPQMFZTjGy/iwvceobZb9c4SVc9eszPYmZd+j2seDZwUc2Rw+vE9rE52p+4d4ULQshtjc8Wt5HHq/bNArpg+oUQi9/1ic7bC1By/PkE4ZXc26IvWVxH+TowN3lkcUR48g7/O+sSD7aS4dkJHyaMvElXMlyxW39FpC6o= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1761907208; c=relaxed/simple; bh=fX9clgG/AwrnKocsdMCV+xHB+ouI4s9+Hm3T3/663k4=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=Evpo7+9xXFNUFoDVeglFXIAfyZmmVG9kswZ0Le+MkcbrNUo/FcCa5l/KTO3eMGlAnR86GCkZwBmDkq0vYzzpb3MX65Z2m+OuHz77b4Vt+Vj+jvNz6AF0QWxhghtfgX7jdKMQXh01AOwaVpYEeQe95LNJdlgsklgUpO+y4IHI2mo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=YAVGLrKR; arc=none smtp.client-ip=209.85.128.74 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="YAVGLrKR" Received: by mail-wm1-f74.google.com with SMTP id 5b1f17b1804b1-475dd9906e1so12549305e9.0 for ; Fri, 31 Oct 2025 03:40:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1761907204; x=1762512004; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=0Xa3/Awesw6OMt3PSs2q2cBtqduLAjKaT7ji9cCCq90=; b=YAVGLrKRkzeQSotkiwpeCO+SSVOfJ0dUgCtrAbA0e6UUDwyhKykh4Y3INdlWhio/wj pAGyTUKkYM9kc0X4irzBaIfRHj0/cVISJRGxz6DJ1DPeaajmfvdTwkxHCUx5q45P//nN NyNdN8pssPmV1tZkLH69HZMhWkb6JA19leWs/RU0uExmWb9ets8NnAUErbL7cowCJTf3 RrE3kyU7dwyC1iYVkC/buEsVf5bPdYGM1tvY17LoSymp6QVm81HU3qKQVfkp+vKQnR6X WQa5dqMpikxdBu2S09g9jTYwOTL1d3UuE/WvbKtlzAqAZZ0deptuhBuXYx0wODSlPRSz I/vg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1761907204; x=1762512004; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=0Xa3/Awesw6OMt3PSs2q2cBtqduLAjKaT7ji9cCCq90=; b=Ld+Ns2HRSysaeg/fxaoC8MG0N2MWCkW3LK094+DPal3bPtiWF/vWLRiYrIhY3saVN+ JHu3Fpj3ngL1AxxD7lIM1tcqcUyix+U62lwOfKJfre+It7G+RE3x31qcBMYrLn13kSkV hEjib+RAXFxZCnRuIO9WmVH9LUKmFHSqeFffydPlkoUrbZ4I8TrsVwNvIVWRFqwXAVOw SF9X1cB1d8ZpgjuapHERwCWWrzYy9df2Pub5LQhBW0DZ8CyrrUoAp8mWLRt1yPmUm090 Pc9diBDddI5FPpqmGJM0SY/zSOWTK5khmSslfo5zQHEi9Fy5hEcEZK6Rk+t3OpyyIV3p 1dZw== X-Gm-Message-State: AOJu0YzwIX1hFeLIDoK9nDUOufWUGTCYoXHQboUKo7n1S/2Bmmnf8K5h 7DJIpYj+u7wddE/qU714IySGUdjFZWIIqn8llRifoDF8n8OPLw9XVkmBQaPfKNDWVs6anPv/eg= = X-Google-Smtp-Source: AGHT+IFG1Dv0rjVcEt2/0GVVIUFS+KzXMYU0UsEadz9yjstPJXyipRoXNOS4eHXBONiuPbDNnDbrnIrq X-Received: from wmwq8.prod.google.com ([2002:a05:600d:8348:b0:476:4b14:2edf]) (user=ardb job=prod-delivery.src-stubby-dispatcher) by 2002:a05:600c:538e:b0:46e:33b2:c8da with SMTP id 5b1f17b1804b1-47730890e99mr28895635e9.32.1761907204019; Fri, 31 Oct 2025 03:40:04 -0700 (PDT) Date: Fri, 31 Oct 2025 11:39:18 +0100 In-Reply-To: <20251031103858.529530-23-ardb+git@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20251031103858.529530-23-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=1914; i=ardb@kernel.org; h=from:subject; bh=JNFlhs5YlNOgtpbYMnFMguYKcFlrOPN7ItLaOkFoa/A=; b=owGbwMvMwCVmkMcZplerG8N4Wi2JIZNl4r3azVwKvxa+e+bzxvzhjJCoDf0pmzLzjxfveskbc 3b79c+sHaUsDGJcDLJiiiwCs/++23l6olSt8yxZmDmsTCBDGLg4BWAiL3IY/qmunjvvbXvjWee2 punuZxcfZXy6++W/uyz//fM8bu+WUnjF8D/61YbAlb+lv9pt6nmpE1ux7mTG++LWrclW3s33r5W svsEDAA== X-Mailer: git-send-email 2.51.1.930.gacf6e81ea2-goog Message-ID: <20251031103858.529530-42-ardb+git@google.com> Subject: [PATCH v4 19/21] net/mlx5: Switch to more abstract scoped ksimd guard API on arm64 From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: linux-kernel@vger.kernel.org, linux-crypto@vger.kernel.org, herbert@gondor.apana.org.au, ebiggers@kernel.org, Ard Biesheuvel , Saeed Mahameed , Leon Romanovsky , Tariq Toukan , Mark Bloch Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Ard Biesheuvel Instead of calling kernel_neon_begin/end directly, switch to the scoped guard API which encapsulates those calls. This is needed because the prototypes of those APIs are going to be modified and will require a kernel mode FP/SIMD buffer to be provided, which the scoped guard API will do transparently. Cc: Saeed Mahameed Cc: Leon Romanovsky Cc: Tariq Toukan Cc: Mark Bloch Signed-off-by: Ard Biesheuvel Acked-by: Catalin Marinas --- drivers/net/ethernet/mellanox/mlx5/core/wc.c | 19 ++++++++++--------- 1 file changed, 10 insertions(+), 9 deletions(-) diff --git a/drivers/net/ethernet/mellanox/mlx5/core/wc.c b/drivers/net/eth= ernet/mellanox/mlx5/core/wc.c index 05e5fd777d4f..815a7c97d6b0 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/wc.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/wc.c @@ -9,6 +9,7 @@ =20 #if IS_ENABLED(CONFIG_KERNEL_MODE_NEON) && IS_ENABLED(CONFIG_ARM64) #include +#include #endif =20 #define TEST_WC_NUM_WQES 255 @@ -264,15 +265,15 @@ static void mlx5_iowrite64_copy(struct mlx5_wc_sq *sq= , __be32 mmio_wqe[16], { #if IS_ENABLED(CONFIG_KERNEL_MODE_NEON) && IS_ENABLED(CONFIG_ARM64) if (cpu_has_neon()) { - kernel_neon_begin(); - asm volatile - (".arch_extension simd\n\t" - "ld1 {v0.16b, v1.16b, v2.16b, v3.16b}, [%0]\n\t" - "st1 {v0.16b, v1.16b, v2.16b, v3.16b}, [%1]" - : - : "r"(mmio_wqe), "r"(sq->bfreg.map + offset) - : "memory", "v0", "v1", "v2", "v3"); - kernel_neon_end(); + scoped_ksimd() { + asm volatile( + ".arch_extension simd\n\t" + "ld1 {v0.16b, v1.16b, v2.16b, v3.16b}, [%0]\n\t" + "st1 {v0.16b, v1.16b, v2.16b, v3.16b}, [%1]" + : + : "r"(mmio_wqe), "r"(sq->bfreg.map + offset) + : "memory", "v0", "v1", "v2", "v3"); + } return; } #endif --=20 2.51.1.930.gacf6e81ea2-goog From nobody Fri Dec 19 02:49:54 2025 Received: from mail-wm1-f74.google.com (mail-wm1-f74.google.com [209.85.128.74]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 72AE1347FEE for ; Fri, 31 Oct 2025 10:40:06 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.74 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1761907208; cv=none; b=WJQatA69OZzGkTtFUY8fFGaXG4W4QydBdwRc52OBp7eH+TxXHzgInw/jfIVurzyHev/nefhBzUsS0KhQd6yc75knF0vibs2S0zIOVWW56g/9dcUJC56I+OtFE9pkFyGQ6tpHKsvGVrUzglkhFhOikVDcT03fEbm1pS8sX5dpLqk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1761907208; c=relaxed/simple; bh=FR+N77IdiGrlCFWn63AwwcQrA42LGDK7Cq6yK+Fgkuo=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=TdTRlA+Mlxynsbe4zfQP/Tz8vLW5GRXpqcaBSPNQX+DXHAWZqExnuLQ+U0LvTI78A4iykBSxVlOnDPA9dy0yJbKgmYDjzs4enoXn+8Xi4Fkr951GK6GcTPYS4MmhERV5hQ0La5s6BFw7MYNQMhnr9g/OFejBlHr2PMtHjHG8pDE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=LStgMf4V; arc=none smtp.client-ip=209.85.128.74 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="LStgMf4V" Received: by mail-wm1-f74.google.com with SMTP id 5b1f17b1804b1-4770c37331fso13900505e9.3 for ; Fri, 31 Oct 2025 03:40:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1761907205; x=1762512005; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=SF8ncrqKGZBPqJiWodPZcZtzsyty0i7Yz17S72ramgE=; b=LStgMf4VHRGYxkZK8MLrbZkS9aaUCTEYp1rEk2RHYoQvz8mKp87ZQnNMsHxFqVqs+a Xd05Yv9CwLCvGtHmm/5dZpOC5cLycuuXR2+ttFHV7P5ZDQNcNF3q9Xh2r4zmUFdjmCAh 8/AYj3SyLRrgMWOR1qBUYfTcp7x2TmwPnlxHsa4mWEObrfPRsKtzSjgVZq7o2XnvSQZJ sPiaQfcchd8bKEUL7oWDWIIayp5sA/ouxRICSaTDdclluYC67fJqZlmzpM+zAR0XQLFT 3VWUUbU29cLb+e6/J4qKp1sWhFjbdWFULPLxqjEx4IpQuP5xLG9RIwRIE531VUTok4XR MbKg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1761907205; x=1762512005; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=SF8ncrqKGZBPqJiWodPZcZtzsyty0i7Yz17S72ramgE=; b=wf9Uy1px70GGijeMmGdelzLJoChnwdPsCjrHiiwG5rh9SNAovm4B5GyrGB4ztrO8/o SfVqU4LjuaiIT2pSXXLZ/5bCYFZqF1Zq1fsF5LV2oO6l03jsFMZViPebjjioJaeNI+HC 46VUybklJtZgJGrxaUuP2HzAdNHdlt2k787E5lWVY9afheFV8A6aXDk9DkSgIELpAV2h c+uatff1hGLjL2Kb2hv9kSM2L25obvUrq1iU2epUytpwNDka9wQ6rahV6wNbFe4V9/cb N/e8ak8jib5fSCrsf08I+YmXB0IDmSrga3myUZZ9zma/CguJADVn3wSS5cB/5H2ftOU8 zu9w== X-Gm-Message-State: AOJu0YyS/aKpJQZLWS01TK5HX9sEE8T2TNcVEY0ZNHVzWDsxh51Bamda utAQFQFzshF7EYWwTb/1IhlgwjmuKHvHEA3BNTr9iPCa15IlPDH0qoctxpNkwR5EJ8HuuoXg5A= = X-Google-Smtp-Source: AGHT+IEcqBxbY2AIAxAcJMsPotcWd8+1QqYVT1aVGm+VjwDxESgE8ZawF9/oIrAUJEx68xLqm7/XWK7L X-Received: from wma7.prod.google.com ([2002:a05:600c:8907:b0:477:14b8:19f6]) (user=ardb job=prod-delivery.src-stubby-dispatcher) by 2002:a05:600c:1f0e:b0:46e:4704:b01e with SMTP id 5b1f17b1804b1-477305a24bcmr31060015e9.8.1761907204696; Fri, 31 Oct 2025 03:40:04 -0700 (PDT) Date: Fri, 31 Oct 2025 11:39:19 +0100 In-Reply-To: <20251031103858.529530-23-ardb+git@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20251031103858.529530-23-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=1630; i=ardb@kernel.org; h=from:subject; bh=QCkmJ7w8pvMSbfD6OyFFYQvX8b+Mf1wyse4VMzYa6NQ=; b=owGbwMvMwCVmkMcZplerG8N4Wi2JIZNl4gP+H0eZF9crby3kuR/0NfFDu3admAK3vWychkKUR phch0lHKQuDGBeDrJgii8Dsv+92np4oVes8SxZmDisTyBAGLk4BmEhBC8P/nEtnfS3ZXiR9vDY9 pu4ZV2LeS23Rv09n3jGVaZy24fmjswz/Yy7EKLWZ/1XO9HrZ7r+lRMb6Z+SrY24TdVlFxV8Y6Kv xAgA= X-Mailer: git-send-email 2.51.1.930.gacf6e81ea2-goog Message-ID: <20251031103858.529530-43-ardb+git@google.com> Subject: [PATCH v4 20/21] arm64/fpu: Enforce task-context only for generic kernel mode FPU From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: linux-kernel@vger.kernel.org, linux-crypto@vger.kernel.org, herbert@gondor.apana.org.au, ebiggers@kernel.org, Ard Biesheuvel Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Ard Biesheuvel The generic kernel mode FPU API, which is used by the AMDGPU driver to perform floating point calculations, is modeled after the most restrictive architecture that supports it. This means it doesn't support preemption, and can only be used from task context. The arm64 implementation is a bit more flexible, but supporting that in the generic API complicates matters slightly, and for no good reason, given that the only user does not need it. So enforce that kernel_fpu_begin() can only be called from task context, and [redundantly] disable preemption. This removes the need for users of this API to provide a kernel mode FP/SIMD state after a future patch that makes that compulsory for preemptible task context. Signed-off-by: Ard Biesheuvel Acked-by: Catalin Marinas --- arch/arm64/include/asm/fpu.h | 16 ++++++++++++++-- 1 file changed, 14 insertions(+), 2 deletions(-) diff --git a/arch/arm64/include/asm/fpu.h b/arch/arm64/include/asm/fpu.h index 2ae50bdce59b..bdc4c6304c6a 100644 --- a/arch/arm64/include/asm/fpu.h +++ b/arch/arm64/include/asm/fpu.h @@ -6,10 +6,22 @@ #ifndef __ASM_FPU_H #define __ASM_FPU_H =20 +#include #include =20 #define kernel_fpu_available() cpu_has_neon() -#define kernel_fpu_begin() kernel_neon_begin() -#define kernel_fpu_end() kernel_neon_end() + +static inline void kernel_fpu_begin(void) +{ + BUG_ON(!in_task()); + preempt_disable(); + kernel_neon_begin(); +} + +static inline void kernel_fpu_end(void) +{ + kernel_neon_end(); + preempt_enable(); +} =20 #endif /* ! __ASM_FPU_H */ --=20 2.51.1.930.gacf6e81ea2-goog From nobody Fri Dec 19 02:49:54 2025 Received: from mail-ej1-f73.google.com (mail-ej1-f73.google.com [209.85.218.73]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 26CE734844F for ; Fri, 31 Oct 2025 10:40:07 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.218.73 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1761907209; cv=none; b=jBjkXAPoyo0oPy5jT942UbPf1pGmCZOfR/ywRjb+vbljW5tN3tnglTh0utFqdFB7BolwCBmT/XCgaEoZI1IUSBX4/YQfq60OXFo7PydBDRlMRZqwP0X9ENLn4rmTIgmmioztfwx00MAjVd+1sEAXF1g/BPv3S05UPi/Rjj5+dJA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1761907209; c=relaxed/simple; bh=HFZsj5SF9uZu3M555auUiRt8ZHbHBGx0ZJPNt87V03I=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=FUERHTvTgV386afomMTuVGHrnYITvjcVbvxnfpAFOQ4t5fdcxjWP7r0HhUgeMFfb8Pc2SdLkHO+Dw6qcuHW4ZvughZGp8SOlGgarIerthHhA+Yeq5Bh2fvopr/0l0DPU9SQ6mi5tV1+Ycqs0wZ2W4ngC52Wv7XDWGUiqx1PT7gQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=X/Onl3YO; arc=none smtp.client-ip=209.85.218.73 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="X/Onl3YO" Received: by mail-ej1-f73.google.com with SMTP id a640c23a62f3a-b448c864d45so246735266b.2 for ; Fri, 31 Oct 2025 03:40:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1761907205; x=1762512005; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=hXSBfWV8tpC2Nd3cljXrFyvXUntLEuS79VQ7cN3V6Kg=; b=X/Onl3YON8cYuFvn6SG11uP/41Ek4YHqv7gSldGGuhw6xKMWW/If+HwfsmYVbA67Mp OFiFGFeRFv0QuRPz2WunDFznV2wtRNoJwopwMnACB5yo2jPOCNqB+G0JLh6rORbiKB1M Zk+hps/CHskhwbdgGWlza/xQCWGpGfD1qMcL9DlO3KOZ8kEWAEyWxNBh2kr+gy3/e4tl QV1jdnYYczN418fUcbIRGy4KM5vzJ5xGIG8PJ5iGYISbq1XC360x/EmadunVe83ux5AH dI+Kieb7F6fSevAPuVB0Dg4X+wIuOHW0v1diS+bFpiqGEOch1ZQB3AibM7yk9cxPbBEn ClIQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1761907205; x=1762512005; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=hXSBfWV8tpC2Nd3cljXrFyvXUntLEuS79VQ7cN3V6Kg=; b=Bwh4MZsH7sy26Ay6shrXf+Qtj2ieTWilZz/WgRRyV46SonLl/foz+vj6n1g4gmLHnq NV345Vq3/r72CqbGTGzYjm66IJsLJimSzFF9oM7AbMIESNCsogUy37VgmNNlx/TAnx3L 3zj17QA/ZsuaXEtDgLEVYj9lmmbW+UVbjV8Frg39IB7RvTc9aofiguspKcWtLL/A01DM xBgL5KutW4eBumAwGPj76BVgyC2WZRo1lRmSKFoEqnqN45FpLWk3oX2D7FKRmRSXpoIO QIYHSWs9AlIKA5uUlnPFCOLFJXhm44BWlNfh/O2UjVgfQ637VRPB3pM5Ns7Ql0ewZkzN 6L4w== X-Gm-Message-State: AOJu0YyYebil5ZM91T4FbElYq+O8lqRL0+aB2BbNMa2pvG2nuJPxcBzP Mp+ok5NVLyKFGPdfz3NH9eCfDofleSY+5EvVzVoUVrXCYXXNAOH/N9fYG6C+rGxdx0Ifv/zw4w= = X-Google-Smtp-Source: AGHT+IHsCyufAFYruq64bIN1s13iRBPnlbTRcCcT0v8JxVpk2siEwC+tdjkWMK4wLBKHpRG6/wZkpb6/ X-Received: from ejnc2.prod.google.com ([2002:a17:906:7622:b0:b3a:f659:314b]) (user=ardb job=prod-delivery.src-stubby-dispatcher) by 2002:a17:906:354a:b0:b6d:4fe5:ead0 with SMTP id a640c23a62f3a-b70700b561amr213996666b.10.1761907205667; Fri, 31 Oct 2025 03:40:05 -0700 (PDT) Date: Fri, 31 Oct 2025 11:39:20 +0100 In-Reply-To: <20251031103858.529530-23-ardb+git@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20251031103858.529530-23-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=8906; i=ardb@kernel.org; h=from:subject; bh=cmnjLnh4iP86mnRISKNH+TGi8gCkbqyWu/Q2rv/wC+k=; b=owGbwMvMwCVmkMcZplerG8N4Wi2JIZNl4sOk8CMO764nGqfuqbwan9Nbczdctat9y2r203sCq 5hLDqzpKGVhEONikBVTZBGY/ffdztMTpWqdZ8nCzGFlAhnCwMUpABOJamRkWP3TQ39O6kPtPvfY KSY/lQ6tV1drLvtnylQwpy4+RuJxCCPDKj+rS784v1Smex89/9vBtz7O6AGP5AW9+44q13wW1+7 lBQA= X-Mailer: git-send-email 2.51.1.930.gacf6e81ea2-goog Message-ID: <20251031103858.529530-44-ardb+git@google.com> Subject: [PATCH v4 21/21] arm64/fpsimd: Allocate kernel mode FP/SIMD buffers on the stack From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: linux-kernel@vger.kernel.org, linux-crypto@vger.kernel.org, herbert@gondor.apana.org.au, ebiggers@kernel.org, Ard Biesheuvel Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Ard Biesheuvel Commit aefbab8e77eb16b5 ("arm64: fpsimd: Preserve/restore kernel mode NEON at context switch") added a 'kernel_fpsimd_state' field to struct thread_struct, which is the arch-specific portion of struct task_struct, and is allocated for each task in the system. The size of this field is 528 bytes, resulting in non-negligible bloat of task_struct, and the resulting memory overhead may impact performance on systems with many processes. This allocation is only used if the task is scheduled out or interrupted by a softirq while using the FP/SIMD unit in kernel mode, and so it is possible to transparently allocate this buffer on the caller's stack instead. So tweak the 'ksimd' scoped guard implementation so that a stack buffer is allocated and passed to both kernel_neon_begin() and kernel_neon_end(), and either record it in the task struct, or use it directly to preserve the task mode kernel FP/SIMD when running in softirq context. Passing the address to both functions, and checking the addresses for consistency ensures that callers of the updated bare begin/end API use it in a manner that is consistent with the new context switch semantics. Signed-off-by: Ard Biesheuvel Acked-by: Catalin Marinas --- arch/arm64/include/asm/fpu.h | 4 +- arch/arm64/include/asm/neon.h | 4 +- arch/arm64/include/asm/processor.h | 7 ++- arch/arm64/include/asm/simd.h | 7 ++- arch/arm64/kernel/fpsimd.c | 53 ++++++++++++++------ 5 files changed, 54 insertions(+), 21 deletions(-) diff --git a/arch/arm64/include/asm/fpu.h b/arch/arm64/include/asm/fpu.h index bdc4c6304c6a..751e88a96734 100644 --- a/arch/arm64/include/asm/fpu.h +++ b/arch/arm64/include/asm/fpu.h @@ -15,12 +15,12 @@ static inline void kernel_fpu_begin(void) { BUG_ON(!in_task()); preempt_disable(); - kernel_neon_begin(); + kernel_neon_begin(NULL); } =20 static inline void kernel_fpu_end(void) { - kernel_neon_end(); + kernel_neon_end(NULL); preempt_enable(); } =20 diff --git a/arch/arm64/include/asm/neon.h b/arch/arm64/include/asm/neon.h index d4b1d172a79b..acebee4605b5 100644 --- a/arch/arm64/include/asm/neon.h +++ b/arch/arm64/include/asm/neon.h @@ -13,7 +13,7 @@ =20 #define cpu_has_neon() system_supports_fpsimd() =20 -void kernel_neon_begin(void); -void kernel_neon_end(void); +void kernel_neon_begin(struct user_fpsimd_state *); +void kernel_neon_end(struct user_fpsimd_state *); =20 #endif /* ! __ASM_NEON_H */ diff --git a/arch/arm64/include/asm/processor.h b/arch/arm64/include/asm/pr= ocessor.h index 61d62bfd5a7b..de3c3b65461d 100644 --- a/arch/arm64/include/asm/processor.h +++ b/arch/arm64/include/asm/processor.h @@ -172,7 +172,12 @@ struct thread_struct { unsigned long fault_code; /* ESR_EL1 value */ struct debug_info debug; /* debugging */ =20 - struct user_fpsimd_state kernel_fpsimd_state; + /* + * Set [cleared] by kernel_neon_begin() [kernel_neon_end()] to the + * address of a caller provided buffer that will be used to preserve a + * task's kernel mode FPSIMD state while it is scheduled out. + */ + struct user_fpsimd_state *kernel_fpsimd_state; unsigned int kernel_fpsimd_cpu; #ifdef CONFIG_ARM64_PTR_AUTH struct ptrauth_keys_user keys_user; diff --git a/arch/arm64/include/asm/simd.h b/arch/arm64/include/asm/simd.h index d9f83c478736..7ddb25df5c98 100644 --- a/arch/arm64/include/asm/simd.h +++ b/arch/arm64/include/asm/simd.h @@ -43,8 +43,11 @@ static __must_check inline bool may_use_simd(void) { =20 #endif /* ! CONFIG_KERNEL_MODE_NEON */ =20 -DEFINE_LOCK_GUARD_0(ksimd, kernel_neon_begin(), kernel_neon_end()) +DEFINE_LOCK_GUARD_1(ksimd, + struct user_fpsimd_state, + kernel_neon_begin(_T->lock), + kernel_neon_end(_T->lock)) =20 -#define scoped_ksimd() scoped_guard(ksimd) +#define scoped_ksimd() scoped_guard(ksimd, &(struct user_fpsimd_state){}) =20 #endif diff --git a/arch/arm64/kernel/fpsimd.c b/arch/arm64/kernel/fpsimd.c index e3f8f51748bc..1c652ce4d40d 100644 --- a/arch/arm64/kernel/fpsimd.c +++ b/arch/arm64/kernel/fpsimd.c @@ -1489,21 +1489,23 @@ static void fpsimd_load_kernel_state(struct task_st= ruct *task) * Elide the load if this CPU holds the most recent kernel mode * FPSIMD context of the current task. */ - if (last->st =3D=3D &task->thread.kernel_fpsimd_state && + if (last->st =3D=3D task->thread.kernel_fpsimd_state && task->thread.kernel_fpsimd_cpu =3D=3D smp_processor_id()) return; =20 - fpsimd_load_state(&task->thread.kernel_fpsimd_state); + fpsimd_load_state(task->thread.kernel_fpsimd_state); } =20 static void fpsimd_save_kernel_state(struct task_struct *task) { struct cpu_fp_state cpu_fp_state =3D { - .st =3D &task->thread.kernel_fpsimd_state, + .st =3D task->thread.kernel_fpsimd_state, .to_save =3D FP_STATE_FPSIMD, }; =20 - fpsimd_save_state(&task->thread.kernel_fpsimd_state); + BUG_ON(!cpu_fp_state.st); + + fpsimd_save_state(task->thread.kernel_fpsimd_state); fpsimd_bind_state_to_cpu(&cpu_fp_state); =20 task->thread.kernel_fpsimd_cpu =3D smp_processor_id(); @@ -1774,6 +1776,7 @@ void fpsimd_update_current_state(struct user_fpsimd_s= tate const *state) void fpsimd_flush_task_state(struct task_struct *t) { t->thread.fpsimd_cpu =3D NR_CPUS; + t->thread.kernel_fpsimd_state =3D NULL; /* * If we don't support fpsimd, bail out after we have * reset the fpsimd_cpu for this task and clear the @@ -1833,8 +1836,13 @@ void fpsimd_save_and_flush_cpu_state(void) * * The caller may freely use the FPSIMD registers until kernel_neon_end() = is * called. + * + * Unless called from non-preemptible task context, @state must point to a + * caller provided buffer that will be used to preserve the task's kernel = mode + * FPSIMD context when it is scheduled out, or if it is interrupted by ker= nel + * mode FPSIMD occurring in softirq context. May be %NULL otherwise. */ -void kernel_neon_begin(void) +void kernel_neon_begin(struct user_fpsimd_state *state) { if (WARN_ON(!system_supports_fpsimd())) return; @@ -1846,7 +1854,7 @@ void kernel_neon_begin(void) /* Save unsaved fpsimd state, if any: */ if (test_thread_flag(TIF_KERNEL_FPSTATE)) { BUG_ON(IS_ENABLED(CONFIG_PREEMPT_RT) || !in_serving_softirq()); - fpsimd_save_kernel_state(current); + fpsimd_save_state(state); } else { fpsimd_save_user_state(); =20 @@ -1867,8 +1875,17 @@ void kernel_neon_begin(void) * mode in task context. So in this case, setting the flag here * is always appropriate. */ - if (IS_ENABLED(CONFIG_PREEMPT_RT) || !in_serving_softirq()) + if (IS_ENABLED(CONFIG_PREEMPT_RT) || !in_serving_softirq()) { + /* + * Record the caller provided buffer as the kernel mode + * FP/SIMD buffer for this task, so that the state can + * be preserved and restored on a context switch. + */ + WARN_ON(current->thread.kernel_fpsimd_state !=3D NULL); + WARN_ON(preemptible() && !state); + current->thread.kernel_fpsimd_state =3D state; set_thread_flag(TIF_KERNEL_FPSTATE); + } } =20 /* Invalidate any task state remaining in the fpsimd regs: */ @@ -1886,22 +1903,30 @@ EXPORT_SYMBOL_GPL(kernel_neon_begin); * * The caller must not use the FPSIMD registers after this function is cal= led, * unless kernel_neon_begin() is called again in the meantime. + * + * The value of @state must match the value passed to the preceding call to + * kernel_neon_begin(). */ -void kernel_neon_end(void) +void kernel_neon_end(struct user_fpsimd_state *state) { if (!system_supports_fpsimd()) return; =20 + if (!test_thread_flag(TIF_KERNEL_FPSTATE)) + return; + /* * If we are returning from a nested use of kernel mode FPSIMD, restore * the task context kernel mode FPSIMD state. This can only happen when * running in softirq context on non-PREEMPT_RT. */ - if (!IS_ENABLED(CONFIG_PREEMPT_RT) && in_serving_softirq() && - test_thread_flag(TIF_KERNEL_FPSTATE)) - fpsimd_load_kernel_state(current); - else + if (!IS_ENABLED(CONFIG_PREEMPT_RT) && in_serving_softirq()) { + fpsimd_load_state(state); + } else { clear_thread_flag(TIF_KERNEL_FPSTATE); + WARN_ON(current->thread.kernel_fpsimd_state !=3D state); + current->thread.kernel_fpsimd_state =3D NULL; + } } EXPORT_SYMBOL_GPL(kernel_neon_end); =20 @@ -1937,7 +1962,7 @@ void __efi_fpsimd_begin(void) WARN_ON(preemptible()); =20 if (may_use_simd()) { - kernel_neon_begin(); + kernel_neon_begin(&efi_fpsimd_state); } else { /* * If !efi_sve_state, SVE can't be in use yet and doesn't need @@ -1986,7 +2011,7 @@ void __efi_fpsimd_end(void) return; =20 if (!efi_fpsimd_state_used) { - kernel_neon_end(); + kernel_neon_end(&efi_fpsimd_state); } else { if (system_supports_sve() && efi_sve_state_used) { bool ffr =3D true; --=20 2.51.1.930.gacf6e81ea2-goog