From nobody Sat Feb 7 18:46:42 2026 Received: from mail-wm1-f73.google.com (mail-wm1-f73.google.com [209.85.128.73]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6CF9825B31D for ; Wed, 8 Oct 2025 15:46:47 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.73 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1759938409; cv=none; b=hBxuQSnQygSv4mLp3tPHXOKex5pEu9kySnu+T6fzJoMKTpH0HxFW8FIZy/hsGVU8I0wH3NORsZ34epNk1PjmQZEHOGiN4gLYBGzccCN1VnLV2pg92dHykxu0FsAJ8TruQBSNE1pFaUduaooG/IubRfiHs8CeFvhDGGB3sfdMOG4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1759938409; c=relaxed/simple; bh=12RXdlH8WUVCzzyh4vIFws4ja8Gr26ZMeYscmOm7oNE=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=jJfTPxOS6ebASY/YBQFzkHy9i7Z/ExkbdgeTKylLez0l/RZYptTdq5AzN7hECfgm7tn2g9WCCEE5UT6CS0Y0f9WZJE9zse0+WhGfsEf9tsPDfN9pqWco5Ebk4yVXQyl/2MejDvK1cdourqDUi0tFws9OdI9pXJyeFVtBFKQwbDg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=MkUEtHk4; arc=none smtp.client-ip=209.85.128.73 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="MkUEtHk4" Received: by mail-wm1-f73.google.com with SMTP id 5b1f17b1804b1-46e3af78819so35192775e9.1 for ; Wed, 08 Oct 2025 08:46:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1759938406; x=1760543206; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=0AhbZPensuqKG/yW0Qj7LE8IaoQDaRVlqpZyE8QDARA=; b=MkUEtHk4snIs5lRXjuc0RGebRrxYuNBdxlqNzkB5nZ/9E+9mGNa75OK+/LDVlerJyh 0CvnrK97nno0ptq7K2lk9McykRedmKZzhJDexco23c8L/xnDd0E/N2oNlVKqM+E4Vocp OFBeHsaRiwTE24y6gNMYzM78KmrsWTPcjgslga+cy4nVKvaeWtXVMNY/99/T+JPgitXS LJM/8qzIHQm2IPA6KFlSi1NIjQziIot/G7O1Vm7pFWa7ey/nn+LWh8gF3y42skE5msUk kXvJgcHg6zzKIaehoNaoMUvyjI41EJRfDtuZFKxp9LpjHz5XxapMq6j0/jk9wsskEDGs Hf1Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1759938406; x=1760543206; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=0AhbZPensuqKG/yW0Qj7LE8IaoQDaRVlqpZyE8QDARA=; b=pk7Ly/A4jn4q+vDsHOuXZTa+w+z6pf/xLwzVgAJMRpMIYcB+K7YF04k8L3sqidmfr5 h9eELU2+fmCchlxX4X6tG9I0XVYQCcTzo0x8Wi3vFYGhYcjwsZqMFcrRwNXh2WthLwEU dudaVoAsstLBeHFzO6GYrpjKPSXb9ex18tIThT/7OJuj3XPOGgM12HK5oAvGev6f62m5 vHUHE6LrVNiHWrkvievAcP212di8nH/96Z3bBe9cLVbLyzCn3hHQLpsZDmqRvNFuAGyA UZAHw01CVi9oeHL+J6/FrDZ3n8kf7Igg+TTx1sqgWIRS1MomMBYovSsLS2Pfd3PjY79L AuMg== X-Gm-Message-State: AOJu0YzMpowqs6x0f6MKTCM6x44GriGH9o24vvVC8Noogghrj+CDy5Qr EvzsjecIHR3xPkIMX4A488FCJFVYrl1MCJyImQCoMZx4wpKRf5I2cUGS7PCSUBBe4WAgF/E8lQ= = X-Google-Smtp-Source: AGHT+IFGiSWygFLP8i64IlU/Ujr5WlDyO+2EV/4Cl8eQXXpZQ1evYrz2O0ebGMwfs5ylyK1bfixkdj8Z X-Received: from wmbgx7.prod.google.com ([2002:a05:600c:8587:b0:46f:aa50:d70a]) (user=ardb job=prod-delivery.src-stubby-dispatcher) by 2002:a05:600c:4f08:b0:46e:3dcb:35b0 with SMTP id 5b1f17b1804b1-46fa9a94553mr27472645e9.2.1759938405883; Wed, 08 Oct 2025 08:46:45 -0700 (PDT) Date: Wed, 8 Oct 2025 17:45:35 +0200 In-Reply-To: <20251008154533.3089255-23-ardb+git@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20251008154533.3089255-23-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=926; i=ardb@kernel.org; h=from:subject; bh=dFhPhM339lpxbecIP5ywrFTMp3sezPQKRMrRY5eAeY4=; b=owGbwMvMwCVmkMcZplerG8N4Wi2JIeNZu1Lfp9XzPjod2dxh3FPPsqyyScbs3OtNeYcfX7Y+w Fm+oNm2o5SFQYyLQVZMkUVg9t93O09PlKp1niULM4eVCWQIAxenAEzE6QEjwxpeIaV+vccGIcs9 Dyc+unbs7fSJWqcyfXV2LPXdKGaT9ZmRYVXe3vsMtX7S1jcOe/o0fpbfGaJcmHeg7Mq7F+7nFwd d5gEA X-Mailer: git-send-email 2.51.0.710.ga91ca5db03-goog Message-ID: <20251008154533.3089255-24-ardb+git@google.com> Subject: [PATCH v3 01/21] crypto/arm64: aes-ce-ccm - Avoid pointless yield of the NEON unit From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: linux-kernel@vger.kernel.org, linux-crypto@vger.kernel.org, herbert@gondor.apana.org.au, ebiggers@kernel.org, Ard Biesheuvel Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Ard Biesheuvel Kernel mode NEON sections are now preemptible on arm64, and so there is no need to yield it explicitly in order to prevent scheduling latency spikes. Reviewed-by: Eric Biggers Signed-off-by: Ard Biesheuvel --- arch/arm64/crypto/aes-ce-ccm-glue.c | 5 +---- 1 file changed, 1 insertion(+), 4 deletions(-) diff --git a/arch/arm64/crypto/aes-ce-ccm-glue.c b/arch/arm64/crypto/aes-ce= -ccm-glue.c index 2d791d51891b..2eb4e76cabc3 100644 --- a/arch/arm64/crypto/aes-ce-ccm-glue.c +++ b/arch/arm64/crypto/aes-ce-ccm-glue.c @@ -114,11 +114,8 @@ static u32 ce_aes_ccm_auth_data(u8 mac[], u8 const in[= ], u32 abytes, in +=3D adv; abytes -=3D adv; =20 - if (unlikely(rem)) { - kernel_neon_end(); - kernel_neon_begin(); + if (unlikely(rem)) macp =3D 0; - } } else { u32 l =3D min(AES_BLOCK_SIZE - macp, abytes); =20 --=20 2.51.0.710.ga91ca5db03-goog From nobody Sat Feb 7 18:46:42 2026 Received: from mail-ed1-f73.google.com (mail-ed1-f73.google.com [209.85.208.73]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6365429DB9A for ; Wed, 8 Oct 2025 15:46:48 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.208.73 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1759938410; cv=none; b=sJ2rqAKrGcwRQvQ7XXyxuLX+RV0P1wX82M4NG14dNRp5tskdA6kRkM5+IT/JvRUTfZS67WP1ynnBMVjzep+OPNodh+vlBfFb+z+7mHQux1RysxHTfQrOdz6y00HTSmq+hM9YwtPgRonsTKpcB/tyrnPhqwwBJJvgA3bot5WH3qI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1759938410; c=relaxed/simple; bh=81RCAUhzxGVASsLN/Sj+xtc3cF0rIGFNTxe2ppMxs6A=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=uptxwk250W/W1lj0Ej2qyIB9Oh+7mDSyEJ88+s+cydvQoNX/MOUTij9rgNEkyR5Jek0T7t2SEm2MoclTFDcNgOQy44zEmmFF8/UErcDeq31irHq46ID0rSeXNJursZfn7p3NBHpnkfqeubWpQE3oQ5tiHUU1T0CFY7hw7zAw5MA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=OEwFH1Kx; arc=none smtp.client-ip=209.85.208.73 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="OEwFH1Kx" Received: by mail-ed1-f73.google.com with SMTP id 4fb4d7f45d1cf-6232f49fc79so8748385a12.2 for ; Wed, 08 Oct 2025 08:46:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1759938407; x=1760543207; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=0aOpjZgoFTagPU2U7+lhs7TWsUUBL3VjC18ylyP94Gs=; b=OEwFH1KxTvKNXc9bq+YswcwfENtIULETZ1TtiVs3cm+YpI895pHJ8rc6WCwYr5J5ts Xfr+cokOHdl61V7X9KujDAzdEN8NjLI8Jr1yfmdKbq9N5N6PmCFb31FxvcmX92b+Yh1A mo09X4Utq6DBsT/XTWpBDGXd9wbC/9TBhUcga9cgApie2B8c5ZdeGuG7tkeLzTcL5BM3 8xRnwxAEhmC7VKA/Y0vvfZB/6a9DaUgoXeVJlw4ZZl6f1x5XEzWNGhM0BWLOl2XQCBIx rkEj0PLYfJ4NnbPVMBFByMQFORrMEuTNUMuU5OTIc9dYC2A2NNTApea1w++3swzuQphR b6fA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1759938407; x=1760543207; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=0aOpjZgoFTagPU2U7+lhs7TWsUUBL3VjC18ylyP94Gs=; b=KQsXUQyQsNOjcZDzxaAnfYf7Cq6B0s4Ta4fAXDmuc6QOA/KHomAzGX3/qW4jiixlKs PZta6MWcOowNXYg+4jXh6sf7+LhXgW+ltrvzwjSdaVuWKrdto8nNXfBYmfQX1nWrjZG6 Jas5/II/8Mbwc3j4l1bmPlshbvO2aChlJJHJzYBSJ6FDc2KNle7jOHNdI3p0tgb5fmpF xG2xahVlg6L3PRyRB68FiN0RPYI2W0DDYYvfCugFDIjtL1dtvC3MiUG3tQTOWiWdh9Mn 5CV639J2Gczjk017ugoMiNmnsMVdiQ6co0cFQ2fN11IokzONAzCVAIYbP+0OoGaudUDK 3ymQ== X-Gm-Message-State: AOJu0YxiFzBzmhDwDpa0xh9N4RiHMVF/Ox5BWewxbn77GrxgOAnGd39Y bbQ8mpYPAcQ1DDhiDWLqexmGrd8tqr1JLnW7kCssuRVsVt8N87u8OEzb/IY+cYkfSYNH0yS9bA= = X-Google-Smtp-Source: AGHT+IEjXfSRLqs1LdB6T1FDKUwIpLPy5cDnenPLXjF2fx7wJgZU+J3eEdql/LMEKYHPQ3roG7kHp6XU X-Received: from edzh6.prod.google.com ([2002:a05:6402:946:b0:637:87fb:3b14]) (user=ardb job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6402:518a:b0:639:6bc8:c7bd with SMTP id 4fb4d7f45d1cf-639d5b89335mr3498965a12.15.1759938406837; Wed, 08 Oct 2025 08:46:46 -0700 (PDT) Date: Wed, 8 Oct 2025 17:45:36 +0200 In-Reply-To: <20251008154533.3089255-23-ardb+git@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20251008154533.3089255-23-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=1857; i=ardb@kernel.org; h=from:subject; bh=j9GSsyZWNvCHsZ+73xBchQmAL5ZAXeZZs4NWX80DqD8=; b=owGbwMvMwCVmkMcZplerG8N4Wi2JIeNZu8rfJAkfjdO/9vefY02Zc5ad2+qC2S4lbn8HkaMf6 6Lv5f/qKGVhEONikBVTZBGY/ffdztMTpWqdZ8nCzGFlAhnCwMUpABPZbcXwz5j/pePnsrXLRIR8 Hq0oNPfN4NuQPeX7guViSn0WLx5O92L4n2s8Y/Yhht5nll/sv2RFRi8UzXQ75xt2sXyZqpx+ilA yAwA= X-Mailer: git-send-email 2.51.0.710.ga91ca5db03-goog Message-ID: <20251008154533.3089255-25-ardb+git@google.com> Subject: [PATCH v3 02/21] crypto/arm64: sm4-ce-ccm - Avoid pointless yield of the NEON unit From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: linux-kernel@vger.kernel.org, linux-crypto@vger.kernel.org, herbert@gondor.apana.org.au, ebiggers@kernel.org, Ard Biesheuvel Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Ard Biesheuvel Kernel mode NEON sections are now preemptible on arm64, and so there is no need to yield it when calling APIs that may sleep. Also, move the calls to kernel_neon_end() to the same scope as kernel_neon_begin(). This is needed for a subsequent change where a stack buffer is allocated transparently and passed to kernel_neon_begin(). Acked-by: Eric Biggers [ardb: Simplify convoluted logic] Signed-off-by: Ard Biesheuvel Reviewed-by: Eric Biggers --- arch/arm64/crypto/sm4-ce-ccm-glue.c | 25 +++++--------------- 1 file changed, 6 insertions(+), 19 deletions(-) diff --git a/arch/arm64/crypto/sm4-ce-ccm-glue.c b/arch/arm64/crypto/sm4-ce= -ccm-glue.c index e9cc1c1364ec..e92cbdf1aaee 100644 --- a/arch/arm64/crypto/sm4-ce-ccm-glue.c +++ b/arch/arm64/crypto/sm4-ce-ccm-glue.c @@ -172,35 +172,22 @@ static int ccm_crypt(struct aead_request *req, struct= skcipher_walk *walk, if (req->assoclen) ccm_calculate_auth_mac(req, mac); =20 - while (walk->nbytes && walk->nbytes !=3D walk->total) { + while (walk->nbytes) { unsigned int tail =3D walk->nbytes % SM4_BLOCK_SIZE; =20 + if (walk->nbytes =3D=3D walk->total) + tail =3D 0; + sm4_ce_ccm_crypt(rkey_enc, walk->dst.virt.addr, walk->src.virt.addr, walk->iv, walk->nbytes - tail, mac); =20 - kernel_neon_end(); - err =3D skcipher_walk_done(walk, tail); - - kernel_neon_begin(); } =20 - if (walk->nbytes) { - sm4_ce_ccm_crypt(rkey_enc, walk->dst.virt.addr, - walk->src.virt.addr, walk->iv, - walk->nbytes, mac); - - sm4_ce_ccm_final(rkey_enc, ctr0, mac); + sm4_ce_ccm_final(rkey_enc, ctr0, mac); =20 - kernel_neon_end(); - - err =3D skcipher_walk_done(walk, 0); - } else { - sm4_ce_ccm_final(rkey_enc, ctr0, mac); - - kernel_neon_end(); - } + kernel_neon_end(); =20 return err; } --=20 2.51.0.710.ga91ca5db03-goog From nobody Sat Feb 7 18:46:42 2026 Received: from mail-wr1-f74.google.com (mail-wr1-f74.google.com [209.85.221.74]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 648E22BDC19 for ; Wed, 8 Oct 2025 15:46:49 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.221.74 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1759938411; cv=none; b=qkvBgHqllxOttmRFAL6mDcl2u6ffaWbJDLvfWbemp1cNOCuFp1Zs1r4ImMAGgfD/H7n1vRi8GNtv/U3HFriLgbNWTHtdnakUt+IdepmwsW6pm+uIT5b7fFseYBcyiSzV6sQ2xkSh+mI8B4O1ndLt/vsmi0gpg+hzI01QhpWVNLA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1759938411; c=relaxed/simple; bh=wEi73K9tOwaYQZRIL/sXTFyXCpFYqwJBcsVGx35P9jc=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=CH1V3rQxUHhCd2QQ6UwgwAJewWDTdP8m+iVjluBu4sYh5WdZjhlRWCnkqAv6Hu8uMNJ7TW06rR4N/f2QmQe8b38ensDacQWfKppqjIxM372NgOmHq1yGNZ/PObyoF0FLaPddXqs0rSBsJMLJ2h9sv0FNbAeXzxXrXP9q7KVOKXU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=FPLnnmU1; arc=none smtp.client-ip=209.85.221.74 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="FPLnnmU1" Received: by mail-wr1-f74.google.com with SMTP id ffacd0b85a97d-3ee12ab7f33so29529f8f.2 for ; Wed, 08 Oct 2025 08:46:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1759938408; x=1760543208; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=14A2zAAjYNWwlBslQvMqX7czpAp/zYAIyPXaDxUH4U4=; b=FPLnnmU1Sc0hZ1hknsBraciNCn+oecRKYZouOx8NIuPAlEosqcC150x1psWP86IOGn mIHLl/yhaOascHIOaq4q27vdOVsBIMatZEecQpsgejwMMwOKjDIwkGoCqvLdZz6W8KlN NzktwXIvUj9t6ikPrq12hF5oNp6yV8QfZWGmb4dYa3/O3W2YnOAAEwInDPL14N15VUGU 0YZjfvkoxNwJT/0ZdF43NGWMeIePS9aNjfLitYDY0LDVC45Ou3Axz+eai8dPTn32o9XB LvISuE2E2WTZtPjZw5ZhIjEYVNLYL+xAwCGh2S5vfiN37Y16v2TvMnEaeNsRuLU2UB/B nCNw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1759938408; x=1760543208; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=14A2zAAjYNWwlBslQvMqX7czpAp/zYAIyPXaDxUH4U4=; b=VomIpuTujxGGqBZigUfQBmDsScfi28+FrF93ghrJ610jM88rOZE6xZ+3a4yFAX/Vbc K1XdRuY37SkrW0Q213hYTx9vLTBSovuh2BVkltOTlvQnwjyJw/akr32ZRowSpX+jhV9O 3ERhMYOj6pevzm1N9eKx++TdZxmH0PRdz69/bv8Qe49u6HLCG/NJE2mnm1KTVWwBLb09 HFs4XQh2dSCbvyoWwSDfVaLJRIgRYsnerrNHTjwKtDgsq3XeUGBfnKAbnsUMWeg7wfqx 6awjKY49gU701ad9DiNbTYeTd4AWsBB+cX7NBv9u2YZRvqKS+UtE8BT/8x0quM78txoB w51Q== X-Gm-Message-State: AOJu0YyI9a/oETTLfjb1/9nAu2U0ClQh3fMnKfUmLBHgM8qXhZkVc303 IfaWCDYsouZAezygkNH4YbcLUzPrYBsfA+mChTTDKlsCU4Fe72jcXvin75V40p0T9BubVpWQiA= = X-Google-Smtp-Source: AGHT+IHEE1faiKfKH1tfDaLUrBFzle6tgkGUM1s8NmshfL2C3RktZfgqJpT+SXKKnzw1/mEsdDS3IKGM X-Received: from wrvz8.prod.google.com ([2002:a5d:6548:0:b0:40f:b976:8cba]) (user=ardb job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6000:22c5:b0:3fd:bf1d:15d1 with SMTP id ffacd0b85a97d-4266e8dde71mr2375302f8f.49.1759938407736; Wed, 08 Oct 2025 08:46:47 -0700 (PDT) Date: Wed, 8 Oct 2025 17:45:37 +0200 In-Reply-To: <20251008154533.3089255-23-ardb+git@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20251008154533.3089255-23-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=2047; i=ardb@kernel.org; h=from:subject; bh=V+R1u4QlJCauwRhK0blDYK+Qy9hXMWQp6E2hey/88as=; b=owGbwMvMwCVmkMcZplerG8N4Wi2JIeNZu7ozS8uvGUohmk1rtfp6zv1Wq5MrdQzzun9T8pyDv tKV+psdpSwMYlwMsmKKLAKz/77beXqiVK3zLFmYOaxMIEMYuDgFYCLTtzIyPPJdPPc1bxab9711 x85sfSLz8ybbgVdxHSd8Hs40uOCj5sTIcP/E5wlprVHtJsG9m7uYTWTlSo9EtVitL3wUfvTK0lp ZBgA= X-Mailer: git-send-email 2.51.0.710.ga91ca5db03-goog Message-ID: <20251008154533.3089255-26-ardb+git@google.com> Subject: [PATCH v3 03/21] crypto/arm64: sm4-ce-gcm - Avoid pointless yield of the NEON unit From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: linux-kernel@vger.kernel.org, linux-crypto@vger.kernel.org, herbert@gondor.apana.org.au, ebiggers@kernel.org, Ard Biesheuvel Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Ard Biesheuvel Kernel mode NEON sections are now preemptible on arm64, and so there is no need to yield it when calling APIs that may sleep. Also, move the calls to kernel_neon_end() to the same scope as kernel_neon_begin(). This is needed for a subsequent change where a stack buffer is allocated transparently and passed to kernel_neon_begin(). While at it, simplify the logic. Acked-by: Eric Biggers [ardb: Simplify convoluted logic] Signed-off-by: Ard Biesheuvel Reviewed-by: Eric Biggers --- arch/arm64/crypto/sm4-ce-gcm-glue.c | 25 +++++--------------- 1 file changed, 6 insertions(+), 19 deletions(-) diff --git a/arch/arm64/crypto/sm4-ce-gcm-glue.c b/arch/arm64/crypto/sm4-ce= -gcm-glue.c index c2ea3d5f690b..8f6fc8c33c3f 100644 --- a/arch/arm64/crypto/sm4-ce-gcm-glue.c +++ b/arch/arm64/crypto/sm4-ce-gcm-glue.c @@ -154,36 +154,23 @@ static int gcm_crypt(struct aead_request *req, struct= skcipher_walk *walk, if (req->assoclen) gcm_calculate_auth_mac(req, ghash); =20 - while (walk->nbytes) { + do { unsigned int tail =3D walk->nbytes % SM4_BLOCK_SIZE; const u8 *src =3D walk->src.virt.addr; u8 *dst =3D walk->dst.virt.addr; + const u8 *l =3D NULL; =20 if (walk->nbytes =3D=3D walk->total) { - sm4_ce_pmull_gcm_crypt(ctx->key.rkey_enc, dst, src, iv, - walk->nbytes, ghash, - ctx->ghash_table, - (const u8 *)&lengths); - - kernel_neon_end(); - - return skcipher_walk_done(walk, 0); + l =3D (const u8 *)&lengths; + tail =3D 0; } =20 sm4_ce_pmull_gcm_crypt(ctx->key.rkey_enc, dst, src, iv, walk->nbytes - tail, ghash, - ctx->ghash_table, NULL); - - kernel_neon_end(); + ctx->ghash_table, l); =20 err =3D skcipher_walk_done(walk, tail); - - kernel_neon_begin(); - } - - sm4_ce_pmull_gcm_crypt(ctx->key.rkey_enc, NULL, NULL, iv, - walk->nbytes, ghash, ctx->ghash_table, - (const u8 *)&lengths); + } while (walk->nbytes); =20 kernel_neon_end(); =20 --=20 2.51.0.710.ga91ca5db03-goog From nobody Sat Feb 7 18:46:42 2026 Received: from mail-wm1-f74.google.com (mail-wm1-f74.google.com [209.85.128.74]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B43F62EBB87 for ; Wed, 8 Oct 2025 15:46:50 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.74 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1759938412; cv=none; b=Mca+NS4UvJ+pK+lNey7uwIepX1tRQlvy8k2wyADSN2PYy/k7ggyVRj53hzjpWQ4b3cLA7qIqALnzNkRg9da3nT6S7bxzCZHR4a1DAZhXhBjO1O+zLs+N1sEHfv4vfzTO4KpH7vgMPrJNsQ/zJnb9+GfBXBE3gj6hvrqQ2w6WC2M= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1759938412; c=relaxed/simple; bh=9FmzEtdFM20sJTDcHY2CkIswmX83GWfybFiLYLap2Ag=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=udXkAm9HTQd1YuZY4aGDg03AMfZGylqpW0OcDMvic45vZuNbqiLIztcY0g02aLB0CZc7uolr+wU0pj2pHgSjDOfbwBd2bMU4H0PXP4Cuk5Rv1JDama0kzlDYA/2ep/fZQ/p2+mAGw9GgOXy65FzdSsRVcuLe8uliUPXWNlZizjU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=0LOzILw3; arc=none smtp.client-ip=209.85.128.74 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="0LOzILw3" Received: by mail-wm1-f74.google.com with SMTP id 5b1f17b1804b1-46e3ed6540fso516345e9.0 for ; Wed, 08 Oct 2025 08:46:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1759938409; x=1760543209; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=uXOxGZ/YNATafGtRkFO3eyEiqFtBfSwf8M5twzbqcUE=; b=0LOzILw34Y572FNhSbCe+tM2odhcs1iVGydR7iB7f5mZMnNxvejBUY6aHeDCGIQWlE wH0wLh41+EtEZKzEu4yjfK6Daa7hlIK2c9jJlQT6JbxQ/YtOgbdEPkiMzy1ivl+jF1jL g1DtZ1634rO2NtAIXYNt3AEWlWjT9hNoI8or9T4n+nTe7kX2BJNlWn2gjDQOrlUxfeYp x9mRHqLW/fDTK3sAyJMtnTdX61O/9xkDzTWFLCbV7MijhC3Q7VPVVXJrZdknQP1KMRbX FvhoTbFv2BMQ2g6PXD10QeINDxJRx+uyKzl8XShVvyLWBjG8fJkYuIx20vxcwVD3twQ5 8qIw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1759938409; x=1760543209; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=uXOxGZ/YNATafGtRkFO3eyEiqFtBfSwf8M5twzbqcUE=; b=NFiyb2cl1GNkoDlaNecxbCrmX2yngINhPtMv7NdE9Z2xu/WNWov0O7XGCX210M1rOe oxs7AIxKTT/kg4CEiJ2KUYdJkzEHZwf6i/x+AUanfnMb9WcdpDfwusxFeBU7ESYcVvxD YWPu4+hK6BxY64sPuXqNzaLw9BuWAcpZp1Dr6PK0OoL+m468DCY0QbDoBu5Qb4F3MU78 sN0fh5QLpSdB+iu9oJudCdErMbVvFwHGpJLXITcB/yXwTiwApDZSlM5AXO52a0Av/w8e 8N48K12x0CiYxHjbB6pHElItGYPvyVELEU/lYP350QvY+lnh3WsnsDqnS5UjfKwZQIMU 1Wrw== X-Gm-Message-State: AOJu0YwS/Gb9bFmL5/iput8tYW9ZwGwspgMdX6EpxSCCz0psotpgKriM iXQivH+f5R2G6C6RXJcZoVRUHf449BW6bw4BDYbEqDy3LKM2u4L/YPl6MV/ltVwfBW6FNuk+RA= = X-Google-Smtp-Source: AGHT+IHrU6L0LG/tG+7wPrq4CdnXtlNwP7FFFswqEly2EF3y8DHI9O5XvzzoSmEJYWWHooMBlG9Iu1ob X-Received: from wmwn26.prod.google.com ([2002:a05:600d:435a:b0:46d:ab31:6ece]) (user=ardb job=prod-delivery.src-stubby-dispatcher) by 2002:a05:600c:a202:b0:46f:a995:55f8 with SMTP id 5b1f17b1804b1-46fa995565bmr20733855e9.13.1759938409082; Wed, 08 Oct 2025 08:46:49 -0700 (PDT) Date: Wed, 8 Oct 2025 17:45:38 +0200 In-Reply-To: <20251008154533.3089255-23-ardb+git@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20251008154533.3089255-23-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=1521; i=ardb@kernel.org; h=from:subject; bh=sjBBfqHYpRbSdr0WNjBeTQONw5wjcd3c1yQxz5xj5oM=; b=owGbwMvMwCVmkMcZplerG8N4Wi2JIeNZu5ZqzFvNj+9Xti34033rQmJlb9VLuTw5mZLqRXlT7 +4+4nGgo5SFQYyLQVZMkUVg9t93O09PlKp1niULM4eVCWQIAxenAEykeDbD/9TlQevcHdwrDbTZ 1a4VZTKeqptx4ERN2/FmMyZ/vv8bOxkZLt2MCSvUsjctElfyuRjF8PHzGpndpaF6f7z4ys6eWPC bEwA= X-Mailer: git-send-email 2.51.0.710.ga91ca5db03-goog Message-ID: <20251008154533.3089255-27-ardb+git@google.com> Subject: [PATCH v3 04/21] arm64/simd: Add scoped guard API for kernel mode SIMD From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: linux-kernel@vger.kernel.org, linux-crypto@vger.kernel.org, herbert@gondor.apana.org.au, ebiggers@kernel.org, Ard Biesheuvel , Kees Cook Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Ard Biesheuvel Encapsulate kernel_neon_begin() and kernel_neon_end() using a 'ksimd' cleanup guard. This hides the prototype of those functions, allowing them to be changed for arm64 but not ARM, without breaking code that is shared between those architectures (RAID6, AEGIS-128) It probably makes sense to expose this API more widely across architectures, as it affords more flexibility to the arch code to plumb it in, while imposing more rigid rules regarding the start/end bookends appearing in matched pairs. Reviewed-by: Kees Cook Reviewed-by: Eric Biggers Signed-off-by: Ard Biesheuvel --- arch/arm64/include/asm/simd.h | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/arch/arm64/include/asm/simd.h b/arch/arm64/include/asm/simd.h index 8e86c9e70e48..d9f83c478736 100644 --- a/arch/arm64/include/asm/simd.h +++ b/arch/arm64/include/asm/simd.h @@ -6,12 +6,15 @@ #ifndef __ASM_SIMD_H #define __ASM_SIMD_H =20 +#include #include #include #include #include #include =20 +#include + #ifdef CONFIG_KERNEL_MODE_NEON =20 /* @@ -40,4 +43,8 @@ static __must_check inline bool may_use_simd(void) { =20 #endif /* ! CONFIG_KERNEL_MODE_NEON */ =20 +DEFINE_LOCK_GUARD_0(ksimd, kernel_neon_begin(), kernel_neon_end()) + +#define scoped_ksimd() scoped_guard(ksimd) + #endif --=20 2.51.0.710.ga91ca5db03-goog From nobody Sat Feb 7 18:46:42 2026 Received: from mail-wm1-f74.google.com (mail-wm1-f74.google.com [209.85.128.74]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0D77D2F83D2 for ; Wed, 8 Oct 2025 15:46:51 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.74 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1759938414; cv=none; b=lljs/yIH2B6Tqqr1BLPV6JvXMqwZparEyTt8FZb6fxTbZ7mcAXkXjgqj7eaRmQmm4i7tA/i/nrVywVw/2n9JRVfUiyGw/f31fmp1x6SBaBDBntBeC/YTzIREIL+U4Kk2C5tGH4xXBJVtjeCyzrwgcdlG+C+aA9le0NxTsu6NftM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1759938414; c=relaxed/simple; bh=KEvq8fCT/mr/NUx3/4WNfXk1AMS/oERIC+y51xl/09U=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=eABrQIrxY8KtmTritD/nkXpQ10WJWWe+uXhqrR/AGrWynPnO76st/LYF8xKyNnJKmZEAA2mF0vVpmJXJ+gYvcrjPP3BQXFcmeONJkZ9Eip6RW336j3ixottuAMI5ltS7NPpwQgGvjy+Lvyp53PSp/LPrVrsacHt7JBvsnl+oWJw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=CzflwAAQ; arc=none smtp.client-ip=209.85.128.74 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="CzflwAAQ" Received: by mail-wm1-f74.google.com with SMTP id 5b1f17b1804b1-46e3ed6540fso516455e9.0 for ; Wed, 08 Oct 2025 08:46:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1759938410; x=1760543210; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=r6qU4dk91tAr2/hjLbeesrEFLeALQh619b79NXK4nw4=; b=CzflwAAQzAxqgCUjQWwomtSIHV3T3BQwxQhKyFzP0XZIrtCWMNdpKeOB4WndK9sz/e GhEWP+AdMbyhPRpPgY4rYEpcaRQ2b4fqWDY8qTIRZzIlB+ijqoM8jq6/Pj9OEV5P8d3q a9GlbOxO8PPn8zlBZ3aknRRl0eS/Z6BfuGlGLeUmR6/YlTZMcblglsW1s9riYSjR9/Xs yoyp7TmbnkqMd5qMH+26xrq4x1nIDN/FqLqxaHqSihyyvJVKY7wO4Rhtcu7uprr8eLQo pe8r34MMGdXF33amy/At27anz8MF+QZXTzfWaqecL8j9yMg8oyl9j4Awpu4q9S0d3wvf E9Mw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1759938410; x=1760543210; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=r6qU4dk91tAr2/hjLbeesrEFLeALQh619b79NXK4nw4=; b=guyC90IS2T6wCrra+dR6EftIGyEYPBys/cuNshd8aFD/PL4RwxPp5sADfpg99r8F1s pAny0tiE3anUcPF3acR43XP0c/bAh/67BcL8gMs8mx+jcxrf/t8HruvYPSo5zOyNR/Zx CXV0eZa7rDsRE4xAe9Ow7/G8xwz8KQCJAsZEy19NwZdDGbufqqgREe+2RSnAcfBM2VGj NJPsfrqdBSIT7i7YpNG3aCppv1PY9r3yjQQnqTssT/ANG0uw2mN5q5M5fydqX7mRVjA4 NPcOtVZALvooq8phoJm0tlKGXPWyWRCOpYD1T/9Nz3Z1aoOGpFB3FHUs/2UbZ0glLC8U s83A== X-Gm-Message-State: AOJu0Yy5ofV3kywzLci8usFruOFgO1KgL7L0JZuAa0lnWWdOPXRATw8/ e7dKllfFvlAKbz4prarYGTidVxI9JgBGMQ2BJdtXf5JhRvj1mVP6O45s2QN2TqFhsc88UxDyYQ= = X-Google-Smtp-Source: AGHT+IFgQd+cxAsEksCTNK8/LMhLr7VTFVSm/ymPavO/ZmIgqUdl9TlX+DTaPdHycCZ8dWC8FxaOxcb1 X-Received: from wmby11-n1.prod.google.com ([2002:a05:600c:c04b:10b0:45f:2be0:b17e]) (user=ardb job=prod-delivery.src-stubby-dispatcher) by 2002:a05:600c:1d9e:b0:46e:2815:8568 with SMTP id 5b1f17b1804b1-46fa9ebe0dfmr26954295e9.10.1759938410501; Wed, 08 Oct 2025 08:46:50 -0700 (PDT) Date: Wed, 8 Oct 2025 17:45:39 +0200 In-Reply-To: <20251008154533.3089255-23-ardb+git@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20251008154533.3089255-23-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=1073; i=ardb@kernel.org; h=from:subject; bh=03C5PCob8A/RgOeAQHX/4ZCOM9qpxWhAP4TBfsL5DQ8=; b=owGbwMvMwCVmkMcZplerG8N4Wi2JIeNZu67vx5bzQuJBJ9VNz8z77964fOkps5iNBqHnH35iW FNVeKyro5SFQYyLQVZMkUVg9t93O09PlKp1niULM4eVCWQIAxenAEzEfSrDH/4Xk6WsfxU6r3kX 4aVd5JkRbTaTSd1xptHVuhknJe8+PMbwP9hcNNXGOvjmNMd69f/504wueGtfWLcnMH7vf2vbg56 K/AA= X-Mailer: git-send-email 2.51.0.710.ga91ca5db03-goog Message-ID: <20251008154533.3089255-28-ardb+git@google.com> Subject: [PATCH v3 05/21] ARM/simd: Add scoped guard API for kernel mode SIMD From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: linux-kernel@vger.kernel.org, linux-crypto@vger.kernel.org, herbert@gondor.apana.org.au, ebiggers@kernel.org, Ard Biesheuvel , Kees Cook Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Ard Biesheuvel Implement the ksimd scoped guard API so that it can be used by code that supports both ARM and arm64. Reviewed-by: Kees Cook Reviewed-by: Eric Biggers Signed-off-by: Ard Biesheuvel --- arch/arm/include/asm/simd.h | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/arch/arm/include/asm/simd.h b/arch/arm/include/asm/simd.h index be08a8da046f..8549fa8b7253 100644 --- a/arch/arm/include/asm/simd.h +++ b/arch/arm/include/asm/simd.h @@ -2,14 +2,21 @@ #ifndef _ASM_SIMD_H #define _ASM_SIMD_H =20 +#include #include #include #include =20 +#include + static __must_check inline bool may_use_simd(void) { return IS_ENABLED(CONFIG_KERNEL_MODE_NEON) && !in_hardirq() && !irqs_disabled(); } =20 +DEFINE_LOCK_GUARD_0(ksimd, kernel_neon_begin(), kernel_neon_end()) + +#define scoped_ksimd() scoped_guard(ksimd) + #endif /* _ASM_SIMD_H */ --=20 2.51.0.710.ga91ca5db03-goog From nobody Sat Feb 7 18:46:42 2026 Received: from mail-wm1-f74.google.com (mail-wm1-f74.google.com [209.85.128.74]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4DD692F8BC3 for ; Wed, 8 Oct 2025 15:46:53 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.74 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1759938415; cv=none; b=dBMQCdsQkItiC4BpYhYTz88vK3lHuQIqdBNkvFUoncQhe6gdC+kCealy3JB5F5MaCVW6Kzq0p7qpnmyYwXyQ1wKjNGsIxCHLKmunU621i0nGGktkdP4w5uSyjfcESX6KWeUQgU/iPDe/Njb98iTYziOb+qk01aLpkyZGrX+dZn0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1759938415; c=relaxed/simple; bh=DzdKmNybBzIYfFee5bWkHzTZXIhZDSbQmeTIrng6jjY=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=Be02QvMRr0UOLo7HbtJgvbDRl9AtUZW6Os0eBJeajZd2dYUIAI3yQvkae4++1uzmEtd49r3X3LWP7VoZ8VdQmSPlTDsM7jm+WrUmfv36CPIj8WiJvIylG+Xx8Rk8m7hKUYrIGvq6c962IZ7zd04ZeS+ArYbkVaNR48kKIDBtquE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=W8EuFrW9; arc=none smtp.client-ip=209.85.128.74 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="W8EuFrW9" Received: by mail-wm1-f74.google.com with SMTP id 5b1f17b1804b1-46e38bd6680so26784235e9.1 for ; Wed, 08 Oct 2025 08:46:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1759938411; x=1760543211; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=BwsGC9QKQTLtCs75FaDogWhQBHMqAIsOgOtZEqdLRzo=; b=W8EuFrW98OBGgB2R4aGNd3zikw79LhOTgUkAhquV+3aL/L5J5O6Y7rnISX4BleZNwS UtHHqjJXp1Iwv0kRWewJcQLGLzypz0xXbb9yvnvLX/JgMPpI1dMwhpGw61rPG0mVLr1c OHulCdEf5ZaHH/CiFHrWH6viLuuiSx2KhtaeTOZyuV6RYtT4FaPSAwhYprcM+gXPc9lc xhhEIGy1d/PTj+T2abzPnftJ+mNmIHHxoykpcbJ5lS20BS9jDK3TSXltJ+5s2lM7tGsu vSelEfVCg3hK3ssXkUx1j7xJhPwobmgFmkW6CiyHbIoVUzEm9rl+vc7h8dkDy9thcNf0 4gbg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1759938411; x=1760543211; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=BwsGC9QKQTLtCs75FaDogWhQBHMqAIsOgOtZEqdLRzo=; b=jPS6Zzl4uhOMNzlHse1fhRZq4xfDEg0dGd9MiM2CUmTubCOSc85pzmIEZJ+0BY5+K1 ej6G3Zhe4DfsQ8sghwg+ZJh3iXzEUXgGIkMOGSxnxCPurnCkKVISRKF1mfiRc2LK4z32 vaTHTpUWSA2wLBRt13fX8f8JHlsP1r2bMGREN6kUekIEzd+vztxjLQe1qxoWIbh0/4uq /9fxBJh+wI/FRBGn/3DItUnXiAsZYAr6//9jdBjTsmePOnNVYhgi4gdOH4hbpG5iej7v Vj/2r9belFNHJ8rfqz69WI2ISxRQGbWgWSkV5JX29GJmEOBD84vDH5sGHwzPwGyyecjw Qzxg== X-Gm-Message-State: AOJu0Yw4hNn9tSwnoe4XID7l5YcEUznI517nmF0U8tc2ydGG6nFbrPDY FisvIJGHqkQaxCoiMeuXFuIPl+vEyKcBZ++/o1A8RmjCDbydalmGl5FgKuV+W5M38gpIR6jgog= = X-Google-Smtp-Source: AGHT+IEOgrgEpk6rF4pwgydWRRxFmyjFZR66yjaGox58gQWW/rEuccbEtdMOOvaxvGbVBzuauWSm+SmC X-Received: from wmbd24.prod.google.com ([2002:a05:600c:58d8:b0:46e:4943:289b]) (user=ardb job=prod-delivery.src-stubby-dispatcher) by 2002:a05:600c:1e86:b0:46e:4784:cdf5 with SMTP id 5b1f17b1804b1-46fa9a9f254mr31390415e9.15.1759938411713; Wed, 08 Oct 2025 08:46:51 -0700 (PDT) Date: Wed, 8 Oct 2025 17:45:40 +0200 In-Reply-To: <20251008154533.3089255-23-ardb+git@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20251008154533.3089255-23-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=2508; i=ardb@kernel.org; h=from:subject; bh=F7nm70Zr8k/vjaUwWeyRiRk62Pm6f7C9Vg0TLVgfN6E=; b=owGbwMvMwCVmkMcZplerG8N4Wi2JIeNZu/6Ovtba/KtFh7eWac/6dvG0Vb6kQLLduz69rw3NT pfS+6w7SlkYxLgYZMUUWQRm/3238/REqVrnWbIwc1iZQIYwcHEKwEQ2XmT4K99b9HVftbJii5ZR c4K5hGTNl4tpiZ/2z5q7WVjz134VPoZfzAWPtm382KB1uL7+b8/u0pMrj5oeOh0rknX4Tqj5v2Q 3BgA= X-Mailer: git-send-email 2.51.0.710.ga91ca5db03-goog Message-ID: <20251008154533.3089255-29-ardb+git@google.com> Subject: [PATCH v3 06/21] crypto: aegis128-neon - Move to more abstract 'ksimd' guard API From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: linux-kernel@vger.kernel.org, linux-crypto@vger.kernel.org, herbert@gondor.apana.org.au, ebiggers@kernel.org, Ard Biesheuvel Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Ard Biesheuvel Move away from calling kernel_neon_begin() and kernel_neon_end() directly, and instead, use the newly introduced scoped_ksimd() API. This permits arm64 to modify the kernel mode NEON API without affecting code that is shared between ARM and arm64. Reviewed-by: Eric Biggers Signed-off-by: Ard Biesheuvel --- crypto/aegis128-neon.c | 33 +++++++------------- 1 file changed, 12 insertions(+), 21 deletions(-) diff --git a/crypto/aegis128-neon.c b/crypto/aegis128-neon.c index 9ee50549e823..b41807e63bd3 100644 --- a/crypto/aegis128-neon.c +++ b/crypto/aegis128-neon.c @@ -4,7 +4,7 @@ */ =20 #include -#include +#include =20 #include "aegis.h" #include "aegis-neon.h" @@ -24,32 +24,28 @@ void crypto_aegis128_init_simd(struct aegis_state *stat= e, const union aegis_block *key, const u8 *iv) { - kernel_neon_begin(); - crypto_aegis128_init_neon(state, key, iv); - kernel_neon_end(); + scoped_ksimd() + crypto_aegis128_init_neon(state, key, iv); } =20 void crypto_aegis128_update_simd(struct aegis_state *state, const void *ms= g) { - kernel_neon_begin(); - crypto_aegis128_update_neon(state, msg); - kernel_neon_end(); + scoped_ksimd() + crypto_aegis128_update_neon(state, msg); } =20 void crypto_aegis128_encrypt_chunk_simd(struct aegis_state *state, u8 *dst, const u8 *src, unsigned int size) { - kernel_neon_begin(); - crypto_aegis128_encrypt_chunk_neon(state, dst, src, size); - kernel_neon_end(); + scoped_ksimd() + crypto_aegis128_encrypt_chunk_neon(state, dst, src, size); } =20 void crypto_aegis128_decrypt_chunk_simd(struct aegis_state *state, u8 *dst, const u8 *src, unsigned int size) { - kernel_neon_begin(); - crypto_aegis128_decrypt_chunk_neon(state, dst, src, size); - kernel_neon_end(); + scoped_ksimd() + crypto_aegis128_decrypt_chunk_neon(state, dst, src, size); } =20 int crypto_aegis128_final_simd(struct aegis_state *state, @@ -58,12 +54,7 @@ int crypto_aegis128_final_simd(struct aegis_state *state, unsigned int cryptlen, unsigned int authsize) { - int ret; - - kernel_neon_begin(); - ret =3D crypto_aegis128_final_neon(state, tag_xor, assoclen, cryptlen, - authsize); - kernel_neon_end(); - - return ret; + scoped_ksimd() + return crypto_aegis128_final_neon(state, tag_xor, assoclen, + cryptlen, authsize); } --=20 2.51.0.710.ga91ca5db03-goog From nobody Sat Feb 7 18:46:42 2026 Received: from mail-wm1-f73.google.com (mail-wm1-f73.google.com [209.85.128.73]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9D2A72F9C23 for ; Wed, 8 Oct 2025 15:46:54 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.73 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1759938417; cv=none; b=cs41do5JFAJdyjdcOB/UKE55EuDCSo7HvY+xL6PKL8zYm2RbvFrDnwnMD2QuhPNI7JN/QEYH1IMZLckO7hoF5mLa2O7K6p8xz5jMLvrxQdIi18Yl3KIlnvs72/uFprgqCMBOSOUeAdwBJmcxvHwzB4PLvhKWS0v6HvTd7w7VVd8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1759938417; c=relaxed/simple; bh=OKa7e4VZ6Av+L894vcBvYIq2RBNlaMFytXrCMa0czDk=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=Gu5SBj+iJz2KMszGqQdMkZMmXs/yGsMhqQY7U0NwcuaZsjumOxgBm1Lg2qdsqZZkUFwjrb9yoCpi7xn8BjP6brSfzpjNnw9A3E85x/a76Zi0Sz8y3ly/pBOPDsXygxlySmgWcf8MPCp4cE/popk4Nt/S7ytafxnRdT0vKoSA1JE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=BPqc/rWW; arc=none smtp.client-ip=209.85.128.73 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="BPqc/rWW" Received: by mail-wm1-f73.google.com with SMTP id 5b1f17b1804b1-46e3e177893so41946845e9.2 for ; Wed, 08 Oct 2025 08:46:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1759938413; x=1760543213; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=AgOp7iaV/RaDAR+GphQp+dqI9mVhhh2Ps95uPVoCH/4=; b=BPqc/rWWeJBHtdbUue2iTisg3OlFcQRhWVjic2ZeyYukW/SlbTt8UtcnJfHQTpSF3W q+VNTCCcWepW6PxcK5LIE3xDsnSK36haqSzBwVOW8pUFOxvetRC3UP1qLpCxpGs2lOCy MQY/ZTTqiVmwrvgXhE1s08CRs76bqMDMfSgodoc+n+Fdd5dxBbJsieeOto/MmIB+/eIs lYV7AF8xaiHEqWHSZoSO+HEhL30kxwXx0eQXUFkY4VlOiXiiQFihWgqTPYJ2P8FPHuoM FsAW0jcO1Ee9DlzQ3B64aHQVmDs+j4vSCydDyOU9dvuM2ZymrccyBS2x2DquM8xnN9Bm Khug== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1759938413; x=1760543213; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=AgOp7iaV/RaDAR+GphQp+dqI9mVhhh2Ps95uPVoCH/4=; b=QJtsUdoP7kqTx5Xq/ftT6lmBvTcMx8gd8sAVzrbodtjt1JMb0/ieXAzuG2uD7qTF3C 87WaVraWu1DnDKlUUFzLWMjKsGpy8fb1O4RHn/OnQ2nEUgNJq3tAj0BaPpM81jSha3at RaZL1eGGHG7n/NrBupV9yZIq9Bqe/PVUho4t1RsXGRoaGba7d13K0LnsI+Se4yd31V9l XBP6VKGpKe5y5+UHmHkXkKay4mrCtQ9U5AT5Q0WXmRGEX7DiJ8yDHzoWMQIufjl7tUVu QhHU3DL8q8CqDzrjnXoPv2IC+UV+e8O8aplRBJw+7DoWPRWCRilxPndbQNwRiXI8Ib1r zTzw== X-Gm-Message-State: AOJu0Ywd7MwGMlC1/nyiziI3aDQ05DZJJr+KZFdItAQqGyp07o+Skg+A y7LPTETTiw+wdJ3FMl2UGZnyav8N1iM7g/GGYDV9KBA8aQnKF3biXqfguFtnKLwOgKbPfHLHNw= = X-Google-Smtp-Source: AGHT+IHnmMnb0RKyaZwccpLn7QM/pWxbFeikk4zy0bIG8pjPiMhw8Qcfv9JNIYTDNAkRKG1Td/av8bhJ X-Received: from wmbji6.prod.google.com ([2002:a05:600c:a346:b0:46e:1b05:b386]) (user=ardb job=prod-delivery.src-stubby-dispatcher) by 2002:a05:600c:4753:b0:46e:1d8d:cfb6 with SMTP id 5b1f17b1804b1-46fa9af0621mr24939175e9.19.1759938412947; Wed, 08 Oct 2025 08:46:52 -0700 (PDT) Date: Wed, 8 Oct 2025 17:45:41 +0200 In-Reply-To: <20251008154533.3089255-23-ardb+git@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20251008154533.3089255-23-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=3233; i=ardb@kernel.org; h=from:subject; bh=S7UkFILzssk2NqWBV2TXMFPj5Ekrt9RH75yk9Qm+UwE=; b=owGbwMvMwCVmkMcZplerG8N4Wi2JIeNZu1F00hLnn+/fFUZJ/WdX51zowZrIPy+rUaZ34U/hZ wYnLvzpKGVhEONikBVTZBGY/ffdztMTpWqdZ8nCzGFlAhnCwMUpABM5upSRYfEi+fKcM1LLv9Rb ZLq4XVr5tUNinm6e3b6kR9wpF7cdN2FkeC7925hr96WfEn2XFhRqveJla9PxyjRiXDKv52Pr498 HuAE= X-Mailer: git-send-email 2.51.0.710.ga91ca5db03-goog Message-ID: <20251008154533.3089255-30-ardb+git@google.com> Subject: [PATCH v3 07/21] raid6: Move to more abstract 'ksimd' guard API From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: linux-kernel@vger.kernel.org, linux-crypto@vger.kernel.org, herbert@gondor.apana.org.au, ebiggers@kernel.org, Ard Biesheuvel Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Ard Biesheuvel Move away from calling kernel_neon_begin() and kernel_neon_end() directly, and instead, use the newly introduced scoped_ksimd() API. This permits arm64 to modify the kernel mode NEON API without affecting code that is shared between ARM and arm64. Reviewed-by: Eric Biggers Signed-off-by: Ard Biesheuvel --- lib/raid6/neon.c | 17 +++++++---------- lib/raid6/recov_neon.c | 15 ++++++--------- 2 files changed, 13 insertions(+), 19 deletions(-) diff --git a/lib/raid6/neon.c b/lib/raid6/neon.c index 0a2e76035ea9..6d9474ce6da9 100644 --- a/lib/raid6/neon.c +++ b/lib/raid6/neon.c @@ -8,10 +8,9 @@ #include =20 #ifdef __KERNEL__ -#include +#include #else -#define kernel_neon_begin() -#define kernel_neon_end() +#define scoped_ksimd() #define cpu_has_neon() (1) #endif =20 @@ -32,10 +31,9 @@ { \ void raid6_neon ## _n ## _gen_syndrome_real(int, \ unsigned long, void**); \ - kernel_neon_begin(); \ - raid6_neon ## _n ## _gen_syndrome_real(disks, \ + scoped_ksimd() \ + raid6_neon ## _n ## _gen_syndrome_real(disks, \ (unsigned long)bytes, ptrs); \ - kernel_neon_end(); \ } \ static void raid6_neon ## _n ## _xor_syndrome(int disks, \ int start, int stop, \ @@ -43,10 +41,9 @@ { \ void raid6_neon ## _n ## _xor_syndrome_real(int, \ int, int, unsigned long, void**); \ - kernel_neon_begin(); \ - raid6_neon ## _n ## _xor_syndrome_real(disks, \ - start, stop, (unsigned long)bytes, ptrs); \ - kernel_neon_end(); \ + scoped_ksimd() \ + raid6_neon ## _n ## _xor_syndrome_real(disks, \ + start, stop, (unsigned long)bytes, ptrs);\ } \ struct raid6_calls const raid6_neonx ## _n =3D { \ raid6_neon ## _n ## _gen_syndrome, \ diff --git a/lib/raid6/recov_neon.c b/lib/raid6/recov_neon.c index 70e1404c1512..9d99aeabd31a 100644 --- a/lib/raid6/recov_neon.c +++ b/lib/raid6/recov_neon.c @@ -7,11 +7,10 @@ #include =20 #ifdef __KERNEL__ -#include +#include #include "neon.h" #else -#define kernel_neon_begin() -#define kernel_neon_end() +#define scoped_ksimd() #define cpu_has_neon() (1) #endif =20 @@ -55,9 +54,8 @@ static void raid6_2data_recov_neon(int disks, size_t byte= s, int faila, qmul =3D raid6_vgfmul[raid6_gfinv[raid6_gfexp[faila] ^ raid6_gfexp[failb]]]; =20 - kernel_neon_begin(); - __raid6_2data_recov_neon(bytes, p, q, dp, dq, pbmul, qmul); - kernel_neon_end(); + scoped_ksimd() + __raid6_2data_recov_neon(bytes, p, q, dp, dq, pbmul, qmul); } =20 static void raid6_datap_recov_neon(int disks, size_t bytes, int faila, @@ -86,9 +84,8 @@ static void raid6_datap_recov_neon(int disks, size_t byte= s, int faila, /* Now, pick the proper data tables */ qmul =3D raid6_vgfmul[raid6_gfinv[raid6_gfexp[faila]]]; =20 - kernel_neon_begin(); - __raid6_datap_recov_neon(bytes, p, q, dq, qmul); - kernel_neon_end(); + scoped_ksimd() + __raid6_datap_recov_neon(bytes, p, q, dq, qmul); } =20 const struct raid6_recov_calls raid6_recov_neon =3D { --=20 2.51.0.710.ga91ca5db03-goog From nobody Sat Feb 7 18:46:42 2026 Received: from mail-wr1-f74.google.com (mail-wr1-f74.google.com [209.85.221.74]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 97AE32F9DAE for ; Wed, 8 Oct 2025 15:46:55 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.221.74 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1759938417; cv=none; b=CduL76f58zA+byAFbgeN6o8nyrLqdpmHoQBj6K+3+U9sDYN/Fsu43S2JPoRksJQsU4RlKkJfC9crvXpCVQ6QLU9NKsqElwxU57WJ5aXOCp97T0OrpJWqFIxEFD//ykib9p23l83F1ErWybSSKwdUgH3/oeB+0sIWcB70nrcd2s4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1759938417; c=relaxed/simple; bh=tm0KHalsNoubgIAXcvxcEVr5CUh06XQoWaGdoTKv1YQ=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=XKvri9JEedTPCMDRdiZ8Ke4oHlCE461f40vJcz2LpLqocAg1RuKeO0P9pTlIt88FBiXIdzZlolC7k7T/uc3QlhTWsqNjj1noeFQb3utZOLsylSe7FZxIh6njQ3ca/78rHUYgfLuzFAcMDuZXyYIAHtBQL+3yPOjSOa7sQNZQx8A= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=e65mljDN; arc=none smtp.client-ip=209.85.221.74 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="e65mljDN" Received: by mail-wr1-f74.google.com with SMTP id ffacd0b85a97d-4255d025f41so52284f8f.1 for ; Wed, 08 Oct 2025 08:46:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1759938414; x=1760543214; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=8GJQeBDpX+7SHx9ZUHYa1E43jiFPF1VO+DzB2+02pPc=; b=e65mljDNjcHZhIPsS+wJrpmBOKZfjguJ9zUGO9Sug2vAeCGpJJqeIM2wkGi8AYAzQs oriZKYrECDvtVzFjZWjd5kchShMmYBwTWzJP8nl+Op3WiQ01uO4COEZdR/Z/nA1geJ+U esRQgdZcBC48y9weV7j27HjzJ943nnzJ9Wrzo9LXf/kojo0OcNdpL+z3vcKoL3pptq5n wJ27kY+5tuZUJs4Qn6bA8CJ4CjSeMkF6qFanxmY1bWZrsQfER1bPIGzOyOybXQEHr6c3 CgbWHwA0yNIkA0YigQjrVCe+/WDfKkvjoXwy05B2CMSwjGz4jg2sDAdaEgOf22G11u5l +cNQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1759938414; x=1760543214; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=8GJQeBDpX+7SHx9ZUHYa1E43jiFPF1VO+DzB2+02pPc=; b=NVMcZdSMmJNnIJKBuiuA+a/dnVkOMjlx5sBrchzOxSV0R3x1g7DzEGCJy0ysf+GpPQ DYH5QiKFuo0RNQwAigvVbunUzl2HQYRK2CX1gRoj4FkniGyahaOWxJ8yU0h1lP5Kl+WW EgcwcaIPUAoWd6g1Z4cmJJrE+CUmWvCGiEXrOkuZhHn+lB1L0RDOIsA8xEyjDEW72K7m X+ZbNPN+VTKNfBujwyk/WNlntKNsjNeQGjWXWg2gCyeByd/o0N7TCcmJ28l9WXkUgrP2 /HQRmixw6aF3Y3UWtu1C9CXcxFO8z9v0yX5cngMx2m4QTrrQox1LD7NaNsS2zuT+zBji DLZw== X-Gm-Message-State: AOJu0YxBjbUVydq8gfu88PypIeaHHyYoVnVgty/JfrC8UzsKsZsoxbdv /rlv/hp385fUjxIq/dIJuCIMpFFFRDD8r6ikAR+WtbZGccKGT3P+t4PHLLMNmaKbe6wixXiN5g= = X-Google-Smtp-Source: AGHT+IHyGaKna1+bfZgQfPNXrTuRy57DagmSjcMNtJQ4By+4hEW1MTsC+nnKDwLQxtbqhtrb+xnIDUJa X-Received: from wmdd12.prod.google.com ([2002:a05:600c:a20c:b0:45f:2b4d:3c2b]) (user=ardb job=prod-delivery.src-stubby-dispatcher) by 2002:a5d:64e8:0:b0:425:855c:5879 with SMTP id ffacd0b85a97d-4266e7d63f2mr2739411f8f.15.1759938414048; Wed, 08 Oct 2025 08:46:54 -0700 (PDT) Date: Wed, 8 Oct 2025 17:45:42 +0200 In-Reply-To: <20251008154533.3089255-23-ardb+git@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20251008154533.3089255-23-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=5822; i=ardb@kernel.org; h=from:subject; bh=O+ioOVuVxfCkNommld21t7f0rh0GyrQ0rc9ryRMpovY=; b=owGbwMvMwCVmkMcZplerG8N4Wi2JIeNZu2ncnGkrmH5Nlrmdd3pvbf+RzjMHxE+wpLz4L6FsI Fbx7lR4RykLgxgXg6yYIovA7L/vdp6eKFXrPEsWZg4rE8gQBi5OAZiIthAjwwPGupk2RlMYkrPX VGawxjzSnVZ2/rCH+xKLy2x/lCXdVBgZpn7h0nxfJPkwOzOMbZa/Am9JvbtFR870LfOPFtedK+/ nAgA= X-Mailer: git-send-email 2.51.0.710.ga91ca5db03-goog Message-ID: <20251008154533.3089255-31-ardb+git@google.com> Subject: [PATCH v3 08/21] lib/crc: Switch ARM and arm64 to 'ksimd' scoped guard API From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: linux-kernel@vger.kernel.org, linux-crypto@vger.kernel.org, herbert@gondor.apana.org.au, ebiggers@kernel.org, Ard Biesheuvel Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Ard Biesheuvel Before modifying the prototypes of kernel_neon_begin() and kernel_neon_end() to accommodate kernel mode FP/SIMD state buffers allocated on the stack, move arm64 to the new 'ksimd' scoped guard API, which encapsulates the calls to those functions. For symmetry, do the same for 32-bit ARM too. Reviewed-by: Eric Biggers Signed-off-by: Ard Biesheuvel --- lib/crc/arm/crc-t10dif.h | 16 +++++----------- lib/crc/arm/crc32.h | 11 ++++------- lib/crc/arm64/crc-t10dif.h | 16 +++++----------- lib/crc/arm64/crc32.h | 16 ++++++---------- 4 files changed, 20 insertions(+), 39 deletions(-) diff --git a/lib/crc/arm/crc-t10dif.h b/lib/crc/arm/crc-t10dif.h index 63441de5e3f1..7f02a767f675 100644 --- a/lib/crc/arm/crc-t10dif.h +++ b/lib/crc/arm/crc-t10dif.h @@ -5,7 +5,6 @@ * Copyright (C) 2016 Linaro Ltd */ =20 -#include #include =20 static __ro_after_init DEFINE_STATIC_KEY_FALSE(have_neon); @@ -20,21 +19,16 @@ asmlinkage void crc_t10dif_pmull8(u16 init_crc, const u= 8 *buf, size_t len, static inline u16 crc_t10dif_arch(u16 crc, const u8 *data, size_t length) { if (length >=3D CRC_T10DIF_PMULL_CHUNK_SIZE) { - if (static_branch_likely(&have_pmull)) { - if (likely(may_use_simd())) { - kernel_neon_begin(); - crc =3D crc_t10dif_pmull64(crc, data, length); - kernel_neon_end(); - return crc; - } + if (static_branch_likely(&have_pmull) && likely(may_use_simd()) { + scoped_ksimd() + return crc_t10dif_pmull64(crc, data, length); } else if (length > CRC_T10DIF_PMULL_CHUNK_SIZE && static_branch_likely(&have_neon) && likely(may_use_simd())) { u8 buf[16] __aligned(16); =20 - kernel_neon_begin(); - crc_t10dif_pmull8(crc, data, length, buf); - kernel_neon_end(); + scoped_ksimd() + crc_t10dif_pmull8(crc, data, length, buf); =20 return crc_t10dif_generic(0, buf, sizeof(buf)); } diff --git a/lib/crc/arm/crc32.h b/lib/crc/arm/crc32.h index 7b76f52f6907..f33de6b22cd4 100644 --- a/lib/crc/arm/crc32.h +++ b/lib/crc/arm/crc32.h @@ -8,7 +8,6 @@ #include =20 #include -#include #include =20 static __ro_after_init DEFINE_STATIC_KEY_FALSE(have_crc32); @@ -42,9 +41,8 @@ static inline u32 crc32_le_arch(u32 crc, const u8 *p, siz= e_t len) len -=3D n; } n =3D round_down(len, 16); - kernel_neon_begin(); - crc =3D crc32_pmull_le(p, n, crc); - kernel_neon_end(); + scoped_ksimd() + crc =3D crc32_pmull_le(p, n, crc); p +=3D n; len -=3D n; } @@ -71,9 +69,8 @@ static inline u32 crc32c_arch(u32 crc, const u8 *p, size_= t len) len -=3D n; } n =3D round_down(len, 16); - kernel_neon_begin(); - crc =3D crc32c_pmull_le(p, n, crc); - kernel_neon_end(); + scoped_ksimd() + crc =3D crc32c_pmull_le(p, n, crc); p +=3D n; len -=3D n; } diff --git a/lib/crc/arm64/crc-t10dif.h b/lib/crc/arm64/crc-t10dif.h index f88db2971805..0de03ab1aeab 100644 --- a/lib/crc/arm64/crc-t10dif.h +++ b/lib/crc/arm64/crc-t10dif.h @@ -7,7 +7,6 @@ =20 #include =20 -#include #include =20 static __ro_after_init DEFINE_STATIC_KEY_FALSE(have_asimd); @@ -22,21 +21,16 @@ asmlinkage u16 crc_t10dif_pmull_p64(u16 init_crc, const= u8 *buf, size_t len); static inline u16 crc_t10dif_arch(u16 crc, const u8 *data, size_t length) { if (length >=3D CRC_T10DIF_PMULL_CHUNK_SIZE) { - if (static_branch_likely(&have_pmull)) { - if (likely(may_use_simd())) { - kernel_neon_begin(); - crc =3D crc_t10dif_pmull_p64(crc, data, length); - kernel_neon_end(); - return crc; - } + if (static_branch_likely(&have_pmull) && likely(may_use_simd())) { + scoped_ksimd() + return crc_t10dif_pmull_p64(crc, data, length); } else if (length > CRC_T10DIF_PMULL_CHUNK_SIZE && static_branch_likely(&have_asimd) && likely(may_use_simd())) { u8 buf[16]; =20 - kernel_neon_begin(); - crc_t10dif_pmull_p8(crc, data, length, buf); - kernel_neon_end(); + scoped_ksimd() + crc_t10dif_pmull_p8(crc, data, length, buf); =20 return crc_t10dif_generic(0, buf, sizeof(buf)); } diff --git a/lib/crc/arm64/crc32.h b/lib/crc/arm64/crc32.h index 31e649cd40a2..1939a5dee477 100644 --- a/lib/crc/arm64/crc32.h +++ b/lib/crc/arm64/crc32.h @@ -2,7 +2,6 @@ =20 #include #include -#include #include =20 // The minimum input length to consider the 4-way interleaved code path @@ -23,9 +22,8 @@ static inline u32 crc32_le_arch(u32 crc, const u8 *p, siz= e_t len) =20 if (len >=3D min_len && cpu_have_named_feature(PMULL) && likely(may_use_simd())) { - kernel_neon_begin(); - crc =3D crc32_le_arm64_4way(crc, p, len); - kernel_neon_end(); + scoped_ksimd() + crc =3D crc32_le_arm64_4way(crc, p, len); =20 p +=3D round_down(len, 64); len %=3D 64; @@ -44,9 +42,8 @@ static inline u32 crc32c_arch(u32 crc, const u8 *p, size_= t len) =20 if (len >=3D min_len && cpu_have_named_feature(PMULL) && likely(may_use_simd())) { - kernel_neon_begin(); - crc =3D crc32c_le_arm64_4way(crc, p, len); - kernel_neon_end(); + scoped_ksimd() + crc =3D crc32c_le_arm64_4way(crc, p, len); =20 p +=3D round_down(len, 64); len %=3D 64; @@ -65,9 +62,8 @@ static inline u32 crc32_be_arch(u32 crc, const u8 *p, siz= e_t len) =20 if (len >=3D min_len && cpu_have_named_feature(PMULL) && likely(may_use_simd())) { - kernel_neon_begin(); - crc =3D crc32_be_arm64_4way(crc, p, len); - kernel_neon_end(); + scoped_ksimd() + crc =3D crc32_be_arm64_4way(crc, p, len); =20 p +=3D round_down(len, 64); len %=3D 64; --=20 2.51.0.710.ga91ca5db03-goog From nobody Sat Feb 7 18:46:42 2026 Received: from mail-wm1-f74.google.com (mail-wm1-f74.google.com [209.85.128.74]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id DD05D2FABE0 for ; Wed, 8 Oct 2025 15:46:56 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.74 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1759938419; cv=none; b=aWb4G4Whgy9rXHfAs9QZ2sEA7yq5zwjGnYQkFoTYP70jnRGIld6nvmryPldzlSUzyn6j+h3GrRJgcCFAko4BF/mMT+5SudHemKjQOzIhm0KlE+qJ5eCO7nX5C2cmH3phhwxuBkbeHzyC/uT2durkj5LtTr9Q+U87DhHTZGUHhD4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1759938419; c=relaxed/simple; bh=RtakoNllASHXrHooobUros6OMf2QXz1XhPt9rY7cMgM=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=Jy2jais9K+XFJEJMoqS+AvuHYlw+FyIzlvT8Nl7SCEg/dgJZuBWuuDzctKgsaPJ9j8S5zK/2CgSb6y/wcJysegN23ZOSewkqixqidIQJwrwOUFsQSBLD6Ki3ulwMqyESKbJYYkngh94EvyeHqObd1e/KskKcVVhisPCQ7QUmOEw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=Bu/dNmak; arc=none smtp.client-ip=209.85.128.74 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="Bu/dNmak" Received: by mail-wm1-f74.google.com with SMTP id 5b1f17b1804b1-46e3ef2dd66so446595e9.1 for ; Wed, 08 Oct 2025 08:46:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1759938415; x=1760543215; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=suglsX43a1PiFmk1IVTuyelqyhWxvfsGVtpIchz7pR4=; b=Bu/dNmakfKBr6y72p8WCXvjYuw10ljRmk9MDlFIkieSvIutsd4G2kFcqgCUxviK/U3 mq4bU0oo3tYNsEpytrtVUSzpP5oqQwL+PyuRVH5PYjBt8ZZ/uFIx4IAKYXDyer2tRH2C 9JRcI/DPge4lF+8FKo83rZqGHDxBpYvCz7r5yEB6Z/RmyvPSRmX3soSvIUtmKx0OtRXj uN97BdZcAR/GpAbv+06MHaMZ3OpZ0VqeC8aeLScVnfHY8KqgppK4g3ocHqniq8gqxTcV AeK39UeKKdQyGqdEUP3RNiNF8Lkd7WXwyjqU566HV50W/95Kxa3bcYvlGZ4rBIE537o0 APCQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1759938415; x=1760543215; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=suglsX43a1PiFmk1IVTuyelqyhWxvfsGVtpIchz7pR4=; b=v73RLfSdqtJjA1EjsrinT2EYWzoRxLg9Ss+rMPLpodfu7msA2Wn+U8sY+6zqdmexvx x8tl0SNndcIotDqW9HlQMtNWtzTMppm0kfMH4K8DkEhtIuQnXvuPBfWx4tTYjuLAKKDv 50OCa5Fl9piByLeX4cOFziiSrVwu4/2Lmysnoa0+MQiEYKtXQONjz56oVMatKpF7e9oU evozJQVVOKv5iUZcTmR0INOAb0INaMHD4BATrf6RhLfcBdxRFugCSiteuytzxII1SVU/ cNhbZWQZLaqmR6cEifdmH1Q+oYdsoNzolgOv+uDvhoOfZz1kyWgJPyOE0T/3nTEa1HMb KJtg== X-Gm-Message-State: AOJu0YwdLcCMrfI9bEo9SIIlOu7RWAIeFpwzzR71Al8eAuianxXjbnhi 1bdbSkclkIMgutynbhad8KAJ+twxVo+2RwsNqtU7v+qp7Ao+K/ZEZjZlrpcK9aSrPULHqx4plA= = X-Google-Smtp-Source: AGHT+IFdfbQL0RA5gdPU5ap+qfVyAmYhR0j1pMraOOaIT9NAX6C6fTnoeZL9e/lDaNNwsknXNRyIUjcW X-Received: from wmgg6.prod.google.com ([2002:a05:600d:6:b0:46f:aa50:d70d]) (user=ardb job=prod-delivery.src-stubby-dispatcher) by 2002:a05:600c:8a8:b0:46d:38c4:1ac9 with SMTP id 5b1f17b1804b1-46fa2952c99mr33990935e9.2.1759938415296; Wed, 08 Oct 2025 08:46:55 -0700 (PDT) Date: Wed, 8 Oct 2025 17:45:43 +0200 In-Reply-To: <20251008154533.3089255-23-ardb+git@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20251008154533.3089255-23-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=9625; i=ardb@kernel.org; h=from:subject; bh=MDqXRAAfE9gWZ5IvXlO00eYBvbAZGnLkU83dj+lis3o=; b=owGbwMvMwCVmkMcZplerG8N4Wi2JIeNZu8VNuWn3b4Qv2bbiAavdW1fxGuMb59l9g/wMFpcxV 02PKN7ZUcrCIMbFICumyCIw+++7nacnStU6z5KFmcPKBDKEgYtTACZyt56R4eWc55Ou282UaWtU Sv46YXN8eaq/iXnjtembi7PWC7s/yGJkaDIuWSs5xWIdx8XWkIveTAKxEdY5B2V8vyxzZNruktT CBQA= X-Mailer: git-send-email 2.51.0.710.ga91ca5db03-goog Message-ID: <20251008154533.3089255-32-ardb+git@google.com> Subject: [PATCH v3 09/21] lib/crypto: Switch ARM and arm64 to 'ksimd' scoped guard API From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: linux-kernel@vger.kernel.org, linux-crypto@vger.kernel.org, herbert@gondor.apana.org.au, ebiggers@kernel.org, Ard Biesheuvel Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Ard Biesheuvel Before modifying the prototypes of kernel_neon_begin() and kernel_neon_end() to accommodate kernel mode FP/SIMD state buffers allocated on the stack, move arm64 to the new 'ksimd' scoped guard API, which encapsulates the calls to those functions. For symmetry, do the same for 32-bit ARM too. Reviewed-by: Eric Biggers Signed-off-by: Ard Biesheuvel --- lib/crypto/arm/chacha.h | 6 ++---- lib/crypto/arm/poly1305.h | 6 ++---- lib/crypto/arm/sha1.h | 13 ++++++------- lib/crypto/arm/sha256.h | 12 ++++++------ lib/crypto/arm/sha512.h | 6 ++---- lib/crypto/arm64/chacha.h | 11 ++++------- lib/crypto/arm64/poly1305.h | 6 ++---- lib/crypto/arm64/sha1.h | 7 +++---- lib/crypto/arm64/sha256.h | 19 ++++++++----------- lib/crypto/arm64/sha512.h | 8 ++++---- 10 files changed, 39 insertions(+), 55 deletions(-) diff --git a/lib/crypto/arm/chacha.h b/lib/crypto/arm/chacha.h index 0cae30f8ee5d..b27ba00b3b23 100644 --- a/lib/crypto/arm/chacha.h +++ b/lib/crypto/arm/chacha.h @@ -12,7 +12,6 @@ =20 #include #include -#include #include =20 asmlinkage void chacha_block_xor_neon(const struct chacha_state *state, @@ -87,9 +86,8 @@ static void chacha_crypt_arch(struct chacha_state *state,= u8 *dst, do { unsigned int todo =3D min_t(unsigned int, bytes, SZ_4K); =20 - kernel_neon_begin(); - chacha_doneon(state, dst, src, todo, nrounds); - kernel_neon_end(); + scoped_ksimd() + chacha_doneon(state, dst, src, todo, nrounds); =20 bytes -=3D todo; src +=3D todo; diff --git a/lib/crypto/arm/poly1305.h b/lib/crypto/arm/poly1305.h index 0021cf368307..0fe903d8de55 100644 --- a/lib/crypto/arm/poly1305.h +++ b/lib/crypto/arm/poly1305.h @@ -6,7 +6,6 @@ */ =20 #include -#include #include #include #include @@ -32,9 +31,8 @@ static void poly1305_blocks(struct poly1305_block_state *= state, const u8 *src, do { unsigned int todo =3D min_t(unsigned int, len, SZ_4K); =20 - kernel_neon_begin(); - poly1305_blocks_neon(state, src, todo, padbit); - kernel_neon_end(); + scoped_ksimd() + poly1305_blocks_neon(state, src, todo, padbit); =20 len -=3D todo; src +=3D todo; diff --git a/lib/crypto/arm/sha1.h b/lib/crypto/arm/sha1.h index 29f8bcad0447..3e2d8c7cab9f 100644 --- a/lib/crypto/arm/sha1.h +++ b/lib/crypto/arm/sha1.h @@ -4,7 +4,6 @@ * * Copyright 2025 Google LLC */ -#include #include =20 static __ro_after_init DEFINE_STATIC_KEY_FALSE(have_neon); @@ -22,12 +21,12 @@ static void sha1_blocks(struct sha1_block_state *state, { if (IS_ENABLED(CONFIG_KERNEL_MODE_NEON) && static_branch_likely(&have_neon) && likely(may_use_simd())) { - kernel_neon_begin(); - if (static_branch_likely(&have_ce)) - sha1_ce_transform(state, data, nblocks); - else - sha1_transform_neon(state, data, nblocks); - kernel_neon_end(); + scoped_ksimd() { + if (static_branch_likely(&have_ce)) + sha1_ce_transform(state, data, nblocks); + else + sha1_transform_neon(state, data, nblocks); + } } else { sha1_block_data_order(state, data, nblocks); } diff --git a/lib/crypto/arm/sha256.h b/lib/crypto/arm/sha256.h index 7556457b3094..ae7e52dd6e3b 100644 --- a/lib/crypto/arm/sha256.h +++ b/lib/crypto/arm/sha256.h @@ -22,12 +22,12 @@ static void sha256_blocks(struct sha256_block_state *st= ate, { if (IS_ENABLED(CONFIG_KERNEL_MODE_NEON) && static_branch_likely(&have_neon) && likely(may_use_simd())) { - kernel_neon_begin(); - if (static_branch_likely(&have_ce)) - sha256_ce_transform(state, data, nblocks); - else - sha256_block_data_order_neon(state, data, nblocks); - kernel_neon_end(); + scoped_ksimd() { + if (static_branch_likely(&have_ce)) + sha256_ce_transform(state, data, nblocks); + else + sha256_block_data_order_neon(state, data, nblocks); + } } else { sha256_block_data_order(state, data, nblocks); } diff --git a/lib/crypto/arm/sha512.h b/lib/crypto/arm/sha512.h index d1b485dd275d..8d3af965d2b3 100644 --- a/lib/crypto/arm/sha512.h +++ b/lib/crypto/arm/sha512.h @@ -18,10 +18,8 @@ static void sha512_blocks(struct sha512_block_state *sta= te, const u8 *data, size_t nblocks) { if (IS_ENABLED(CONFIG_KERNEL_MODE_NEON) && - static_branch_likely(&have_neon) && likely(may_use_simd())) { - kernel_neon_begin(); - sha512_block_data_order_neon(state, data, nblocks); - kernel_neon_end(); + scoped_ksimd() + sha512_block_data_order_neon(state, data, nblocks); } else { sha512_block_data_order(state, data, nblocks); } diff --git a/lib/crypto/arm64/chacha.h b/lib/crypto/arm64/chacha.h index ba6c22d46086..ca8c6a8b0578 100644 --- a/lib/crypto/arm64/chacha.h +++ b/lib/crypto/arm64/chacha.h @@ -23,7 +23,6 @@ #include =20 #include -#include #include =20 asmlinkage void chacha_block_xor_neon(const struct chacha_state *state, @@ -65,9 +64,8 @@ static void hchacha_block_arch(const struct chacha_state = *state, if (!static_branch_likely(&have_neon) || !crypto_simd_usable()) { hchacha_block_generic(state, out, nrounds); } else { - kernel_neon_begin(); - hchacha_block_neon(state, out, nrounds); - kernel_neon_end(); + scoped_ksimd() + hchacha_block_neon(state, out, nrounds); } } =20 @@ -81,9 +79,8 @@ static void chacha_crypt_arch(struct chacha_state *state,= u8 *dst, do { unsigned int todo =3D min_t(unsigned int, bytes, SZ_4K); =20 - kernel_neon_begin(); - chacha_doneon(state, dst, src, todo, nrounds); - kernel_neon_end(); + scoped_ksimd() + chacha_doneon(state, dst, src, todo, nrounds); =20 bytes -=3D todo; src +=3D todo; diff --git a/lib/crypto/arm64/poly1305.h b/lib/crypto/arm64/poly1305.h index aed5921ccd9a..b77669767cd6 100644 --- a/lib/crypto/arm64/poly1305.h +++ b/lib/crypto/arm64/poly1305.h @@ -6,7 +6,6 @@ */ =20 #include -#include #include #include #include @@ -31,9 +30,8 @@ static void poly1305_blocks(struct poly1305_block_state *= state, const u8 *src, do { unsigned int todo =3D min_t(unsigned int, len, SZ_4K); =20 - kernel_neon_begin(); - poly1305_blocks_neon(state, src, todo, padbit); - kernel_neon_end(); + scoped_ksimd() + poly1305_blocks_neon(state, src, todo, padbit); =20 len -=3D todo; src +=3D todo; diff --git a/lib/crypto/arm64/sha1.h b/lib/crypto/arm64/sha1.h index aaef4ebfc5e3..bc7071f1be09 100644 --- a/lib/crypto/arm64/sha1.h +++ b/lib/crypto/arm64/sha1.h @@ -4,7 +4,6 @@ * * Copyright 2025 Google LLC */ -#include #include #include =20 @@ -20,9 +19,9 @@ static void sha1_blocks(struct sha1_block_state *state, do { size_t rem; =20 - kernel_neon_begin(); - rem =3D __sha1_ce_transform(state, data, nblocks); - kernel_neon_end(); + scoped_ksimd() + rem =3D __sha1_ce_transform(state, data, nblocks); + data +=3D (nblocks - rem) * SHA1_BLOCK_SIZE; nblocks =3D rem; } while (nblocks); diff --git a/lib/crypto/arm64/sha256.h b/lib/crypto/arm64/sha256.h index 80d06df27d3a..568dff0f276a 100644 --- a/lib/crypto/arm64/sha256.h +++ b/lib/crypto/arm64/sha256.h @@ -4,7 +4,6 @@ * * Copyright 2025 Google LLC */ -#include #include #include =20 @@ -27,17 +26,16 @@ static void sha256_blocks(struct sha256_block_state *st= ate, do { size_t rem; =20 - kernel_neon_begin(); - rem =3D __sha256_ce_transform(state, - data, nblocks); - kernel_neon_end(); + scoped_ksimd() + rem =3D __sha256_ce_transform(state, data, + nblocks); + data +=3D (nblocks - rem) * SHA256_BLOCK_SIZE; nblocks =3D rem; } while (nblocks); } else { - kernel_neon_begin(); - sha256_block_neon(state, data, nblocks); - kernel_neon_end(); + scoped_ksimd() + sha256_block_neon(state, data, nblocks); } } else { sha256_block_data_order(state, data, nblocks); @@ -66,9 +64,8 @@ static bool sha256_finup_2x_arch(const struct __sha256_ct= x *ctx, if (IS_ENABLED(CONFIG_KERNEL_MODE_NEON) && static_branch_likely(&have_ce) && len >=3D SHA256_BLOCK_SIZE && len <=3D 65536 && likely(may_use_simd())) { - kernel_neon_begin(); - sha256_ce_finup2x(ctx, data1, data2, len, out1, out2); - kernel_neon_end(); + scoped_ksimd() + sha256_ce_finup2x(ctx, data1, data2, len, out1, out2); kmsan_unpoison_memory(out1, SHA256_DIGEST_SIZE); kmsan_unpoison_memory(out2, SHA256_DIGEST_SIZE); return true; diff --git a/lib/crypto/arm64/sha512.h b/lib/crypto/arm64/sha512.h index ddb0d256f73a..7eb7ef04d268 100644 --- a/lib/crypto/arm64/sha512.h +++ b/lib/crypto/arm64/sha512.h @@ -4,7 +4,7 @@ * * Copyright 2025 Google LLC */ -#include + #include #include =20 @@ -24,9 +24,9 @@ static void sha512_blocks(struct sha512_block_state *stat= e, do { size_t rem; =20 - kernel_neon_begin(); - rem =3D __sha512_ce_transform(state, data, nblocks); - kernel_neon_end(); + scoped_ksimd() + rem =3D __sha512_ce_transform(state, data, nblocks); + data +=3D (nblocks - rem) * SHA512_BLOCK_SIZE; nblocks =3D rem; } while (nblocks); --=20 2.51.0.710.ga91ca5db03-goog From nobody Sat Feb 7 18:46:42 2026 Received: from mail-wr1-f74.google.com (mail-wr1-f74.google.com [209.85.221.74]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E66D22FAC15 for ; Wed, 8 Oct 2025 15:46:57 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.221.74 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1759938419; cv=none; b=TasbeBiCgG9BH9ewSNLdMYlH7xv2x9IZvVxKipj+rcj7yPud8QQko0F3JvBkacseXPoOPTewf6Dj+NVoC85/LOEn3MLZId9up5yOGNY3dsyUI2RqfKqjcKskx1LfBvopcnoDX9zpg0lkEBJ7t6lkYU/Z2EYJGwBUSq4sTUWxMco= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1759938419; c=relaxed/simple; bh=hZ7D2VhE6rmVLNXRjWCtqtkp2VdiXoZ1njBgdHXCPWw=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=NhPruCqYeokuHie7Mhga++azeS+OWDlLvShF2Rjp5c7D+1TGexwHLmAGpprq0mKRRAyaMcanArwEDLsMSDat5QDByuzTlRPlFoPM2exRFWHwTPWSmEudbP2ne8fcUXIYGXgeopvOLlmO3CQ+uPDQQvhKK45EfH5oJoKniuEuEbE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=hWpFLHvp; arc=none smtp.client-ip=209.85.221.74 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="hWpFLHvp" Received: by mail-wr1-f74.google.com with SMTP id ffacd0b85a97d-3ecdf7b5c46so40800f8f.2 for ; Wed, 08 Oct 2025 08:46:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1759938416; x=1760543216; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=7ZTif7/cIVHMtQ45TCGuP+cAI2k4HcRnORL17PDNXW0=; b=hWpFLHvp+Ffoed0ZNFgUNyzoK2l3bNY2ONNlkr7Xefzu2VZ7CpPrfiNQgb0eh4Uk7h RIplBkOvPLFicy2XDTgMy0Tp9W0suTIwYs+ctfdyPIl9lCU3KH7ecoDMBTsRgkFiiPK2 K9/9bVq/5uEV4UsHAOSEgtBhMb+ku9szYuMIoa8dhAdsI8lO4DSj5hAwG+iUULcqRgm7 LyHeGX0Keq2xGCrnSRWe9sKo0ed3Ch1BJVbA53COcGLZyHhZY3KhW1V1GN6BkHcm2XX0 IEwJvEZYOz8LLI714CgDJnnHkA2PH529zdaq71AUXxRtB3VoVr3iwzi/eWRnwGZ1jCwv oczw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1759938416; x=1760543216; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=7ZTif7/cIVHMtQ45TCGuP+cAI2k4HcRnORL17PDNXW0=; b=LRCbyOm2VNja/gAUnE1uiz8qenpIqntrFpGhztRZ0N3MP0gKnzf3Uo0leJHvk6Ot6F o+nsuNDUWXJjjDvTFZJ5z1O/UMCA5DZQgIto42w5NDfSIr4jmJPQcLfwx7kjQ73Bmm0L 6hoLwOST++bmNZX3M/kkuTtrAssfk7kk6zuTyAzfj+wYP8tlIya6p5eyrPxjNnlUZO2S IJW56caNHnXxmcDrSbYBYVkaLhO2kgX5aGyJ7qfCX7kJxXSInsH78+htB7i4J15g5R/J dfzPIxizR8oQfaevC6Ex1z6zSRA3mZJvbybo6uXMmBEwBBl5OCfzYtcPYPfo/6UZe57+ eIiQ== X-Gm-Message-State: AOJu0Ywli3aRxc7LL9A/tjoOdeaihTo/77l+pw4ajZjYAEi9icdiZaRa dOoscsyiXfkslW89P/pQlIzjnCDKMt+WCTpliQPi8jCgae/2QXiYRk96mSUNqjvpf7i39NkU+w= = X-Google-Smtp-Source: AGHT+IEyCmYra1GwWRRLmZZtQP0MhI0OXcPDxc0b177KYVx3KWOyy1Y3Egz1AfgFxQB41mo55vgeHUjH X-Received: from wmju4.prod.google.com ([2002:a7b:cb04:0:b0:46e:6302:1a80]) (user=ardb job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6000:290d:b0:405:ed47:b22b with SMTP id ffacd0b85a97d-42666abb50fmr2372305f8f.10.1759938416483; Wed, 08 Oct 2025 08:46:56 -0700 (PDT) Date: Wed, 8 Oct 2025 17:45:44 +0200 In-Reply-To: <20251008154533.3089255-23-ardb+git@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20251008154533.3089255-23-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=4882; i=ardb@kernel.org; h=from:subject; bh=4UlGg4Ju4YWc+Ic+YbEYvUCY8K/tyCDEsga2nZVeBfU=; b=owGbwMvMwCVmkMcZplerG8N4Wi2JIeNZu80txrdaByc5vmw2PiCQKtyW6Mmrzf6v6zGLx/Pzn xq/HH/RUcrCIMbFICumyCIw+++7nacnStU6z5KFmcPKBDKEgYtTACZik8zwP5VFsj/wi4xbkYbR mRViX4tDOiw+xyiw3a0/z/Rsju+bOwz/9F1VbI4HTag/VTjZ88Tt3tTLPgLJSzLmOdWpF71MWxj BAAA= X-Mailer: git-send-email 2.51.0.710.ga91ca5db03-goog Message-ID: <20251008154533.3089255-33-ardb+git@google.com> Subject: [PATCH v3 10/21] crypto/arm64: aes-ccm - Switch to 'ksimd' scoped guard API From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: linux-kernel@vger.kernel.org, linux-crypto@vger.kernel.org, herbert@gondor.apana.org.au, ebiggers@kernel.org, Ard Biesheuvel Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Ard Biesheuvel Switch to the more abstract 'scoped_ksimd()' API, which will be modified in a future patch to transparently allocate a kernel mode FP/SIMD state buffer on the stack, so that kernel mode FP/SIMD code remains preemptible in principe, but without the memory overhead that adds 528 bytes to the size of struct task_struct. Reviewed-by: Eric Biggers Signed-off-by: Ard Biesheuvel --- arch/arm64/crypto/aes-ce-ccm-glue.c | 135 ++++++++++---------- 1 file changed, 66 insertions(+), 69 deletions(-) diff --git a/arch/arm64/crypto/aes-ce-ccm-glue.c b/arch/arm64/crypto/aes-ce= -ccm-glue.c index 2eb4e76cabc3..c4fd648471f1 100644 --- a/arch/arm64/crypto/aes-ce-ccm-glue.c +++ b/arch/arm64/crypto/aes-ce-ccm-glue.c @@ -8,7 +8,6 @@ * Author: Ard Biesheuvel */ =20 -#include #include #include #include @@ -16,6 +15,8 @@ #include #include =20 +#include + #include "aes-ce-setkey.h" =20 MODULE_IMPORT_NS("CRYPTO_INTERNAL"); @@ -184,40 +185,38 @@ static int ccm_encrypt(struct aead_request *req) if (unlikely(err)) return err; =20 - kernel_neon_begin(); - - if (req->assoclen) - ccm_calculate_auth_mac(req, mac); - - do { - u32 tail =3D walk.nbytes % AES_BLOCK_SIZE; - const u8 *src =3D walk.src.virt.addr; - u8 *dst =3D walk.dst.virt.addr; - u8 buf[AES_BLOCK_SIZE]; - u8 *final_iv =3D NULL; - - if (walk.nbytes =3D=3D walk.total) { - tail =3D 0; - final_iv =3D orig_iv; - } - - if (unlikely(walk.nbytes < AES_BLOCK_SIZE)) - src =3D dst =3D memcpy(&buf[sizeof(buf) - walk.nbytes], - src, walk.nbytes); - - ce_aes_ccm_encrypt(dst, src, walk.nbytes - tail, - ctx->key_enc, num_rounds(ctx), - mac, walk.iv, final_iv); - - if (unlikely(walk.nbytes < AES_BLOCK_SIZE)) - memcpy(walk.dst.virt.addr, dst, walk.nbytes); - - if (walk.nbytes) { - err =3D skcipher_walk_done(&walk, tail); - } - } while (walk.nbytes); - - kernel_neon_end(); + scoped_ksimd() { + if (req->assoclen) + ccm_calculate_auth_mac(req, mac); + + do { + u32 tail =3D walk.nbytes % AES_BLOCK_SIZE; + const u8 *src =3D walk.src.virt.addr; + u8 *dst =3D walk.dst.virt.addr; + u8 buf[AES_BLOCK_SIZE]; + u8 *final_iv =3D NULL; + + if (walk.nbytes =3D=3D walk.total) { + tail =3D 0; + final_iv =3D orig_iv; + } + + if (unlikely(walk.nbytes < AES_BLOCK_SIZE)) + src =3D dst =3D memcpy(&buf[sizeof(buf) - walk.nbytes], + src, walk.nbytes); + + ce_aes_ccm_encrypt(dst, src, walk.nbytes - tail, + ctx->key_enc, num_rounds(ctx), + mac, walk.iv, final_iv); + + if (unlikely(walk.nbytes < AES_BLOCK_SIZE)) + memcpy(walk.dst.virt.addr, dst, walk.nbytes); + + if (walk.nbytes) { + err =3D skcipher_walk_done(&walk, tail); + } + } while (walk.nbytes); + } =20 if (unlikely(err)) return err; @@ -251,40 +250,38 @@ static int ccm_decrypt(struct aead_request *req) if (unlikely(err)) return err; =20 - kernel_neon_begin(); - - if (req->assoclen) - ccm_calculate_auth_mac(req, mac); - - do { - u32 tail =3D walk.nbytes % AES_BLOCK_SIZE; - const u8 *src =3D walk.src.virt.addr; - u8 *dst =3D walk.dst.virt.addr; - u8 buf[AES_BLOCK_SIZE]; - u8 *final_iv =3D NULL; - - if (walk.nbytes =3D=3D walk.total) { - tail =3D 0; - final_iv =3D orig_iv; - } - - if (unlikely(walk.nbytes < AES_BLOCK_SIZE)) - src =3D dst =3D memcpy(&buf[sizeof(buf) - walk.nbytes], - src, walk.nbytes); - - ce_aes_ccm_decrypt(dst, src, walk.nbytes - tail, - ctx->key_enc, num_rounds(ctx), - mac, walk.iv, final_iv); - - if (unlikely(walk.nbytes < AES_BLOCK_SIZE)) - memcpy(walk.dst.virt.addr, dst, walk.nbytes); - - if (walk.nbytes) { - err =3D skcipher_walk_done(&walk, tail); - } - } while (walk.nbytes); - - kernel_neon_end(); + scoped_ksimd() { + if (req->assoclen) + ccm_calculate_auth_mac(req, mac); + + do { + u32 tail =3D walk.nbytes % AES_BLOCK_SIZE; + const u8 *src =3D walk.src.virt.addr; + u8 *dst =3D walk.dst.virt.addr; + u8 buf[AES_BLOCK_SIZE]; + u8 *final_iv =3D NULL; + + if (walk.nbytes =3D=3D walk.total) { + tail =3D 0; + final_iv =3D orig_iv; + } + + if (unlikely(walk.nbytes < AES_BLOCK_SIZE)) + src =3D dst =3D memcpy(&buf[sizeof(buf) - walk.nbytes], + src, walk.nbytes); + + ce_aes_ccm_decrypt(dst, src, walk.nbytes - tail, + ctx->key_enc, num_rounds(ctx), + mac, walk.iv, final_iv); + + if (unlikely(walk.nbytes < AES_BLOCK_SIZE)) + memcpy(walk.dst.virt.addr, dst, walk.nbytes); + + if (walk.nbytes) { + err =3D skcipher_walk_done(&walk, tail); + } + } while (walk.nbytes); + } =20 if (unlikely(err)) return err; --=20 2.51.0.710.ga91ca5db03-goog From nobody Sat Feb 7 18:46:42 2026 Received: from mail-wm1-f73.google.com (mail-wm1-f73.google.com [209.85.128.73]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id CF4CD2FB09E for ; Wed, 8 Oct 2025 15:46:58 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.73 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1759938421; cv=none; b=nwH+54MiRiCI6xwT+Gw35LfZQOwNwnGl0OLumGGgjy4cMKW/xWOfiS1/hyw7ZYPAvwSWJ5z8d9ci1CY08xkBZ7OcbrnXmdtH2ejzzMiu+qK965ZovBu7MGaxcfsT0UjPW1DeBB/3AZO91u3kc+ZzE6+T8urAJCyfsFIN9Pt7HCo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1759938421; c=relaxed/simple; bh=FOOTK9XQfGI6AufrMLN9uATcjwmBWbtEtVy/UrFUqiE=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=ptm5pUVRwngdQp4gifK8L1vpVTAkZlEcHtx4Sgs9MukTEZ0jwC0czGMOScjJG8c9SAKNna/GQxA6E3XE7d5WBNJrJbTrkuhIoo3Jflpiy/k8WbgwEykcqevCXL0iEKZ856/F7hi3+y7NLrmGuuF3PZrmSNZWVUjmgn2858bM7mQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=GlYTzF2p; arc=none smtp.client-ip=209.85.128.73 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="GlYTzF2p" Received: by mail-wm1-f73.google.com with SMTP id 5b1f17b1804b1-46e2c11b94cso43143295e9.3 for ; Wed, 08 Oct 2025 08:46:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1759938417; x=1760543217; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=L46HkLw8a5J3i9kWdBIi5leMJXM6J86JLgXVE8xjgwQ=; b=GlYTzF2p45ujGzwrXDV7o9pmx8kXEy2A+aUwSB9+FDagK+wtKwJ35eUTscxCmBXGlD uN3DYuZqVVTeEYyRJ+vA2iMLSKAiegiS2NX5wntnzzJauIxlQ4Q9G7qWW6YL7+iTn2rz BCgUZP0YoxZLjfCzHZ74gq9AuPerEeHjdbYgsfk9HuQwKiYSmInZ1Yu4oH94vUB0+AKi 4aDVLtx7U2wNAGAV8aEAiIAcG4Obza2YRl5xyAOrp4y6Sxlk5H85Og2djnPAyVYaqja0 +0GOhnf2v2SUmiFUlxDOZk06QcmwIo4EJCxbqv1E9RLvW8dTluAI7Gk6WvqWqdnAbMzy 7e1A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1759938417; x=1760543217; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=L46HkLw8a5J3i9kWdBIi5leMJXM6J86JLgXVE8xjgwQ=; b=p7D/FHP1QhO6H+KG7+C0zsHKXOL6Rnv8nVY5gfJSXR3uiJltXEkCx0cz/7mFNx4Wbg YvknQ9sgU8OpmsPTTJG9ZiYNbBmUPIVd9DqDB5yIpUubHnuXQ3NCbT/oBGxjHBcNt+4m 4VrGqy+eo8RIQbHVWBfXdeagZujsLiiIEw5m1non1O9I9Vl8eiYBxnJ6V+79c9g3PxYR LbOn96g0o07MskFQvZx0NOJ1dWnFI+j1SBnMOqaq2PqFW/QziNpGdYoTLRjilx9s4EAp BnSOIqVhT0vwlGChkKKLq14SVrIDS0MHoZZ5u2rZEgonHw2R+1Fh096zyNt/nksK6foq Xf+g== X-Gm-Message-State: AOJu0YxO5T9lqaD0a06iUOEVjsU3KRzwjNBb6SK8dxxu+qAjTEk70AwP EhzAj5p9k0LMYCt7YRWLOVBHiqg9vgnOs2DUGAHEcyg3u8JGh7LV0WIsFIR4WAuomVG7jiLRWw= = X-Google-Smtp-Source: AGHT+IG2yDjvx8M5RHQJcFuja3CFrUxqijB7b3AYImn0xKywLHMP4fir5zv+TsiSTqzGBKF0TNrX6Pht X-Received: from wmbes23.prod.google.com ([2002:a05:600c:8117:b0:46f:aa50:d70b]) (user=ardb job=prod-delivery.src-stubby-dispatcher) by 2002:a05:600c:5488:b0:46e:45f7:34fc with SMTP id 5b1f17b1804b1-46fa9b105ffmr28052175e9.29.1759938417442; Wed, 08 Oct 2025 08:46:57 -0700 (PDT) Date: Wed, 8 Oct 2025 17:45:45 +0200 In-Reply-To: <20251008154533.3089255-23-ardb+git@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20251008154533.3089255-23-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=20750; i=ardb@kernel.org; h=from:subject; bh=hUfqbOTmnijDWPqp3qx6DVKxfJBhdKIamRCPmoGE/Iw=; b=owGbwMvMwCVmkMcZplerG8N4Wi2JIeNZu11J+LVp7s17N7IebdU9kLv6zPrZrfMU3yfZlPWbx Advt9DpKGVhEONikBVTZBGY/ffdztMTpWqdZ8nCzGFlAhnCwMUpABPJ/8HIsOw+59K06G9xh+d0 z3t/+dkaA72b+8N+rPZzXuC+dpdD6RZGhnuHH3SyJPmv/JS/MnFd3is3mYjQk0lTUhdeEzyi9ep FFyMA X-Mailer: git-send-email 2.51.0.710.ga91ca5db03-goog Message-ID: <20251008154533.3089255-34-ardb+git@google.com> Subject: [PATCH v3 11/21] crypto/arm64: aes-blk - Switch to 'ksimd' scoped guard API From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: linux-kernel@vger.kernel.org, linux-crypto@vger.kernel.org, herbert@gondor.apana.org.au, ebiggers@kernel.org, Ard Biesheuvel Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Ard Biesheuvel Switch to the more abstract 'scoped_ksimd()' API, which will be modified in a future patch to transparently allocate a kernel mode FP/SIMD state buffer on the stack, so that kernel mode FP/SIMD code remains preemptible in principe, but without the memory overhead that adds 528 bytes to the size of struct task_struct. Reviewed-by: Eric Biggers Signed-off-by: Ard Biesheuvel --- arch/arm64/crypto/aes-ce-glue.c | 87 ++++++------ arch/arm64/crypto/aes-glue.c | 139 ++++++++---------- arch/arm64/crypto/aes-neonbs-glue.c | 150 ++++++++++---------- 3 files changed, 181 insertions(+), 195 deletions(-) diff --git a/arch/arm64/crypto/aes-ce-glue.c b/arch/arm64/crypto/aes-ce-glu= e.c index 00b8749013c5..a4dad370991d 100644 --- a/arch/arm64/crypto/aes-ce-glue.c +++ b/arch/arm64/crypto/aes-ce-glue.c @@ -52,9 +52,8 @@ static void aes_cipher_encrypt(struct crypto_tfm *tfm, u8= dst[], u8 const src[]) return; } =20 - kernel_neon_begin(); - __aes_ce_encrypt(ctx->key_enc, dst, src, num_rounds(ctx)); - kernel_neon_end(); + scoped_ksimd() + __aes_ce_encrypt(ctx->key_enc, dst, src, num_rounds(ctx)); } =20 static void aes_cipher_decrypt(struct crypto_tfm *tfm, u8 dst[], u8 const = src[]) @@ -66,9 +65,8 @@ static void aes_cipher_decrypt(struct crypto_tfm *tfm, u8= dst[], u8 const src[]) return; } =20 - kernel_neon_begin(); - __aes_ce_decrypt(ctx->key_dec, dst, src, num_rounds(ctx)); - kernel_neon_end(); + scoped_ksimd() + __aes_ce_decrypt(ctx->key_dec, dst, src, num_rounds(ctx)); } =20 int ce_aes_expandkey(struct crypto_aes_ctx *ctx, const u8 *in_key, @@ -94,47 +92,48 @@ int ce_aes_expandkey(struct crypto_aes_ctx *ctx, const = u8 *in_key, for (i =3D 0; i < kwords; i++) ctx->key_enc[i] =3D get_unaligned_le32(in_key + i * sizeof(u32)); =20 - kernel_neon_begin(); - for (i =3D 0; i < sizeof(rcon); i++) { - u32 *rki =3D ctx->key_enc + (i * kwords); - u32 *rko =3D rki + kwords; - - rko[0] =3D ror32(__aes_ce_sub(rki[kwords - 1]), 8) ^ rcon[i] ^ rki[0]; - rko[1] =3D rko[0] ^ rki[1]; - rko[2] =3D rko[1] ^ rki[2]; - rko[3] =3D rko[2] ^ rki[3]; - - if (key_len =3D=3D AES_KEYSIZE_192) { - if (i >=3D 7) - break; - rko[4] =3D rko[3] ^ rki[4]; - rko[5] =3D rko[4] ^ rki[5]; - } else if (key_len =3D=3D AES_KEYSIZE_256) { - if (i >=3D 6) - break; - rko[4] =3D __aes_ce_sub(rko[3]) ^ rki[4]; - rko[5] =3D rko[4] ^ rki[5]; - rko[6] =3D rko[5] ^ rki[6]; - rko[7] =3D rko[6] ^ rki[7]; + scoped_ksimd() { + for (i =3D 0; i < sizeof(rcon); i++) { + u32 *rki =3D ctx->key_enc + (i * kwords); + u32 *rko =3D rki + kwords; + + rko[0] =3D ror32(__aes_ce_sub(rki[kwords - 1]), 8) ^ + rcon[i] ^ rki[0]; + rko[1] =3D rko[0] ^ rki[1]; + rko[2] =3D rko[1] ^ rki[2]; + rko[3] =3D rko[2] ^ rki[3]; + + if (key_len =3D=3D AES_KEYSIZE_192) { + if (i >=3D 7) + break; + rko[4] =3D rko[3] ^ rki[4]; + rko[5] =3D rko[4] ^ rki[5]; + } else if (key_len =3D=3D AES_KEYSIZE_256) { + if (i >=3D 6) + break; + rko[4] =3D __aes_ce_sub(rko[3]) ^ rki[4]; + rko[5] =3D rko[4] ^ rki[5]; + rko[6] =3D rko[5] ^ rki[6]; + rko[7] =3D rko[6] ^ rki[7]; + } } - } =20 - /* - * Generate the decryption keys for the Equivalent Inverse Cipher. - * This involves reversing the order of the round keys, and applying - * the Inverse Mix Columns transformation on all but the first and - * the last one. - */ - key_enc =3D (struct aes_block *)ctx->key_enc; - key_dec =3D (struct aes_block *)ctx->key_dec; - j =3D num_rounds(ctx); - - key_dec[0] =3D key_enc[j]; - for (i =3D 1, j--; j > 0; i++, j--) - __aes_ce_invert(key_dec + i, key_enc + j); - key_dec[i] =3D key_enc[0]; + /* + * Generate the decryption keys for the Equivalent Inverse + * Cipher. This involves reversing the order of the round + * keys, and applying the Inverse Mix Columns transformation on + * all but the first and the last one. + */ + key_enc =3D (struct aes_block *)ctx->key_enc; + key_dec =3D (struct aes_block *)ctx->key_dec; + j =3D num_rounds(ctx); + + key_dec[0] =3D key_enc[j]; + for (i =3D 1, j--; j > 0; i++, j--) + __aes_ce_invert(key_dec + i, key_enc + j); + key_dec[i] =3D key_enc[0]; + } =20 - kernel_neon_end(); return 0; } EXPORT_SYMBOL(ce_aes_expandkey); diff --git a/arch/arm64/crypto/aes-glue.c b/arch/arm64/crypto/aes-glue.c index 5e207ff34482..b087b900d279 100644 --- a/arch/arm64/crypto/aes-glue.c +++ b/arch/arm64/crypto/aes-glue.c @@ -5,8 +5,6 @@ * Copyright (C) 2013 - 2017 Linaro Ltd */ =20 -#include -#include #include #include #include @@ -20,6 +18,9 @@ #include #include =20 +#include +#include + #include "aes-ce-setkey.h" =20 #ifdef USE_V8_CRYPTO_EXTENSIONS @@ -186,10 +187,9 @@ static int __maybe_unused ecb_encrypt(struct skcipher_= request *req) err =3D skcipher_walk_virt(&walk, req, false); =20 while ((blocks =3D (walk.nbytes / AES_BLOCK_SIZE))) { - kernel_neon_begin(); - aes_ecb_encrypt(walk.dst.virt.addr, walk.src.virt.addr, - ctx->key_enc, rounds, blocks); - kernel_neon_end(); + scoped_ksimd() + aes_ecb_encrypt(walk.dst.virt.addr, walk.src.virt.addr, + ctx->key_enc, rounds, blocks); err =3D skcipher_walk_done(&walk, walk.nbytes % AES_BLOCK_SIZE); } return err; @@ -206,10 +206,9 @@ static int __maybe_unused ecb_decrypt(struct skcipher_= request *req) err =3D skcipher_walk_virt(&walk, req, false); =20 while ((blocks =3D (walk.nbytes / AES_BLOCK_SIZE))) { - kernel_neon_begin(); - aes_ecb_decrypt(walk.dst.virt.addr, walk.src.virt.addr, - ctx->key_dec, rounds, blocks); - kernel_neon_end(); + scoped_ksimd() + aes_ecb_decrypt(walk.dst.virt.addr, walk.src.virt.addr, + ctx->key_dec, rounds, blocks); err =3D skcipher_walk_done(&walk, walk.nbytes % AES_BLOCK_SIZE); } return err; @@ -224,10 +223,9 @@ static int cbc_encrypt_walk(struct skcipher_request *r= eq, unsigned int blocks; =20 while ((blocks =3D (walk->nbytes / AES_BLOCK_SIZE))) { - kernel_neon_begin(); - aes_cbc_encrypt(walk->dst.virt.addr, walk->src.virt.addr, - ctx->key_enc, rounds, blocks, walk->iv); - kernel_neon_end(); + scoped_ksimd() + aes_cbc_encrypt(walk->dst.virt.addr, walk->src.virt.addr, + ctx->key_enc, rounds, blocks, walk->iv); err =3D skcipher_walk_done(walk, walk->nbytes % AES_BLOCK_SIZE); } return err; @@ -253,10 +251,9 @@ static int cbc_decrypt_walk(struct skcipher_request *r= eq, unsigned int blocks; =20 while ((blocks =3D (walk->nbytes / AES_BLOCK_SIZE))) { - kernel_neon_begin(); - aes_cbc_decrypt(walk->dst.virt.addr, walk->src.virt.addr, - ctx->key_dec, rounds, blocks, walk->iv); - kernel_neon_end(); + scoped_ksimd() + aes_cbc_decrypt(walk->dst.virt.addr, walk->src.virt.addr, + ctx->key_dec, rounds, blocks, walk->iv); err =3D skcipher_walk_done(walk, walk->nbytes % AES_BLOCK_SIZE); } return err; @@ -322,10 +319,9 @@ static int cts_cbc_encrypt(struct skcipher_request *re= q) if (err) return err; =20 - kernel_neon_begin(); - aes_cbc_cts_encrypt(walk.dst.virt.addr, walk.src.virt.addr, - ctx->key_enc, rounds, walk.nbytes, walk.iv); - kernel_neon_end(); + scoped_ksimd() + aes_cbc_cts_encrypt(walk.dst.virt.addr, walk.src.virt.addr, + ctx->key_enc, rounds, walk.nbytes, walk.iv); =20 return skcipher_walk_done(&walk, 0); } @@ -379,10 +375,9 @@ static int cts_cbc_decrypt(struct skcipher_request *re= q) if (err) return err; =20 - kernel_neon_begin(); - aes_cbc_cts_decrypt(walk.dst.virt.addr, walk.src.virt.addr, - ctx->key_dec, rounds, walk.nbytes, walk.iv); - kernel_neon_end(); + scoped_ksimd() + aes_cbc_cts_decrypt(walk.dst.virt.addr, walk.src.virt.addr, + ctx->key_dec, rounds, walk.nbytes, walk.iv); =20 return skcipher_walk_done(&walk, 0); } @@ -399,11 +394,11 @@ static int __maybe_unused essiv_cbc_encrypt(struct sk= cipher_request *req) =20 blocks =3D walk.nbytes / AES_BLOCK_SIZE; if (blocks) { - kernel_neon_begin(); - aes_essiv_cbc_encrypt(walk.dst.virt.addr, walk.src.virt.addr, - ctx->key1.key_enc, rounds, blocks, - req->iv, ctx->key2.key_enc); - kernel_neon_end(); + scoped_ksimd() + aes_essiv_cbc_encrypt(walk.dst.virt.addr, + walk.src.virt.addr, + ctx->key1.key_enc, rounds, blocks, + req->iv, ctx->key2.key_enc); err =3D skcipher_walk_done(&walk, walk.nbytes % AES_BLOCK_SIZE); } return err ?: cbc_encrypt_walk(req, &walk); @@ -421,11 +416,11 @@ static int __maybe_unused essiv_cbc_decrypt(struct sk= cipher_request *req) =20 blocks =3D walk.nbytes / AES_BLOCK_SIZE; if (blocks) { - kernel_neon_begin(); - aes_essiv_cbc_decrypt(walk.dst.virt.addr, walk.src.virt.addr, - ctx->key1.key_dec, rounds, blocks, - req->iv, ctx->key2.key_enc); - kernel_neon_end(); + scoped_ksimd() + aes_essiv_cbc_decrypt(walk.dst.virt.addr, + walk.src.virt.addr, + ctx->key1.key_dec, rounds, blocks, + req->iv, ctx->key2.key_enc); err =3D skcipher_walk_done(&walk, walk.nbytes % AES_BLOCK_SIZE); } return err ?: cbc_decrypt_walk(req, &walk); @@ -461,10 +456,9 @@ static int __maybe_unused xctr_encrypt(struct skcipher= _request *req) else if (nbytes < walk.total) nbytes &=3D ~(AES_BLOCK_SIZE - 1); =20 - kernel_neon_begin(); - aes_xctr_encrypt(dst, src, ctx->key_enc, rounds, nbytes, - walk.iv, byte_ctr); - kernel_neon_end(); + scoped_ksimd() + aes_xctr_encrypt(dst, src, ctx->key_enc, rounds, nbytes, + walk.iv, byte_ctr); =20 if (unlikely(nbytes < AES_BLOCK_SIZE)) memcpy(walk.dst.virt.addr, @@ -506,10 +500,9 @@ static int __maybe_unused ctr_encrypt(struct skcipher_= request *req) else if (nbytes < walk.total) nbytes &=3D ~(AES_BLOCK_SIZE - 1); =20 - kernel_neon_begin(); - aes_ctr_encrypt(dst, src, ctx->key_enc, rounds, nbytes, - walk.iv); - kernel_neon_end(); + scoped_ksimd() + aes_ctr_encrypt(dst, src, ctx->key_enc, rounds, nbytes, + walk.iv); =20 if (unlikely(nbytes < AES_BLOCK_SIZE)) memcpy(walk.dst.virt.addr, @@ -562,11 +555,10 @@ static int __maybe_unused xts_encrypt(struct skcipher= _request *req) if (walk.nbytes < walk.total) nbytes &=3D ~(AES_BLOCK_SIZE - 1); =20 - kernel_neon_begin(); - aes_xts_encrypt(walk.dst.virt.addr, walk.src.virt.addr, - ctx->key1.key_enc, rounds, nbytes, - ctx->key2.key_enc, walk.iv, first); - kernel_neon_end(); + scoped_ksimd() + aes_xts_encrypt(walk.dst.virt.addr, walk.src.virt.addr, + ctx->key1.key_enc, rounds, nbytes, + ctx->key2.key_enc, walk.iv, first); err =3D skcipher_walk_done(&walk, walk.nbytes - nbytes); } =20 @@ -584,11 +576,10 @@ static int __maybe_unused xts_encrypt(struct skcipher= _request *req) if (err) return err; =20 - kernel_neon_begin(); - aes_xts_encrypt(walk.dst.virt.addr, walk.src.virt.addr, - ctx->key1.key_enc, rounds, walk.nbytes, - ctx->key2.key_enc, walk.iv, first); - kernel_neon_end(); + scoped_ksimd() + aes_xts_encrypt(walk.dst.virt.addr, walk.src.virt.addr, + ctx->key1.key_enc, rounds, walk.nbytes, + ctx->key2.key_enc, walk.iv, first); =20 return skcipher_walk_done(&walk, 0); } @@ -634,11 +625,10 @@ static int __maybe_unused xts_decrypt(struct skcipher= _request *req) if (walk.nbytes < walk.total) nbytes &=3D ~(AES_BLOCK_SIZE - 1); =20 - kernel_neon_begin(); - aes_xts_decrypt(walk.dst.virt.addr, walk.src.virt.addr, - ctx->key1.key_dec, rounds, nbytes, - ctx->key2.key_enc, walk.iv, first); - kernel_neon_end(); + scoped_ksimd() + aes_xts_decrypt(walk.dst.virt.addr, walk.src.virt.addr, + ctx->key1.key_dec, rounds, nbytes, + ctx->key2.key_enc, walk.iv, first); err =3D skcipher_walk_done(&walk, walk.nbytes - nbytes); } =20 @@ -657,11 +647,10 @@ static int __maybe_unused xts_decrypt(struct skcipher= _request *req) return err; =20 =20 - kernel_neon_begin(); - aes_xts_decrypt(walk.dst.virt.addr, walk.src.virt.addr, - ctx->key1.key_dec, rounds, walk.nbytes, - ctx->key2.key_enc, walk.iv, first); - kernel_neon_end(); + scoped_ksimd() + aes_xts_decrypt(walk.dst.virt.addr, walk.src.virt.addr, + ctx->key1.key_dec, rounds, walk.nbytes, + ctx->key2.key_enc, walk.iv, first); =20 return skcipher_walk_done(&walk, 0); } @@ -808,10 +797,9 @@ static int cmac_setkey(struct crypto_shash *tfm, const= u8 *in_key, return err; =20 /* encrypt the zero vector */ - kernel_neon_begin(); - aes_ecb_encrypt(ctx->consts, (u8[AES_BLOCK_SIZE]){}, ctx->key.key_enc, - rounds, 1); - kernel_neon_end(); + scoped_ksimd() + aes_ecb_encrypt(ctx->consts, (u8[AES_BLOCK_SIZE]){}, + ctx->key.key_enc, rounds, 1); =20 cmac_gf128_mul_by_x(consts, consts); cmac_gf128_mul_by_x(consts + 1, consts); @@ -837,10 +825,10 @@ static int xcbc_setkey(struct crypto_shash *tfm, cons= t u8 *in_key, if (err) return err; =20 - kernel_neon_begin(); - aes_ecb_encrypt(key, ks[0], ctx->key.key_enc, rounds, 1); - aes_ecb_encrypt(ctx->consts, ks[1], ctx->key.key_enc, rounds, 2); - kernel_neon_end(); + scoped_ksimd() { + aes_ecb_encrypt(key, ks[0], ctx->key.key_enc, rounds, 1); + aes_ecb_encrypt(ctx->consts, ks[1], ctx->key.key_enc, rounds, 2); + } =20 return cbcmac_setkey(tfm, key, sizeof(key)); } @@ -860,10 +848,9 @@ static void mac_do_update(struct crypto_aes_ctx *ctx, = u8 const in[], int blocks, int rem; =20 do { - kernel_neon_begin(); - rem =3D aes_mac_update(in, ctx->key_enc, rounds, blocks, - dg, enc_before, !enc_before); - kernel_neon_end(); + scoped_ksimd() + rem =3D aes_mac_update(in, ctx->key_enc, rounds, blocks, + dg, enc_before, !enc_before); in +=3D (blocks - rem) * AES_BLOCK_SIZE; blocks =3D rem; } while (blocks); diff --git a/arch/arm64/crypto/aes-neonbs-glue.c b/arch/arm64/crypto/aes-ne= onbs-glue.c index c4a623e86593..d496effb0a5b 100644 --- a/arch/arm64/crypto/aes-neonbs-glue.c +++ b/arch/arm64/crypto/aes-neonbs-glue.c @@ -85,9 +85,8 @@ static int aesbs_setkey(struct crypto_skcipher *tfm, cons= t u8 *in_key, =20 ctx->rounds =3D 6 + key_len / 4; =20 - kernel_neon_begin(); - aesbs_convert_key(ctx->rk, rk.key_enc, ctx->rounds); - kernel_neon_end(); + scoped_ksimd() + aesbs_convert_key(ctx->rk, rk.key_enc, ctx->rounds); =20 return 0; } @@ -110,10 +109,9 @@ static int __ecb_crypt(struct skcipher_request *req, blocks =3D round_down(blocks, walk.stride / AES_BLOCK_SIZE); =20 - kernel_neon_begin(); - fn(walk.dst.virt.addr, walk.src.virt.addr, ctx->rk, - ctx->rounds, blocks); - kernel_neon_end(); + scoped_ksimd() + fn(walk.dst.virt.addr, walk.src.virt.addr, ctx->rk, + ctx->rounds, blocks); err =3D skcipher_walk_done(&walk, walk.nbytes - blocks * AES_BLOCK_SIZE); } @@ -146,9 +144,8 @@ static int aesbs_cbc_ctr_setkey(struct crypto_skcipher = *tfm, const u8 *in_key, =20 memcpy(ctx->enc, rk.key_enc, sizeof(ctx->enc)); =20 - kernel_neon_begin(); - aesbs_convert_key(ctx->key.rk, rk.key_enc, ctx->key.rounds); - kernel_neon_end(); + scoped_ksimd() + aesbs_convert_key(ctx->key.rk, rk.key_enc, ctx->key.rounds); memzero_explicit(&rk, sizeof(rk)); =20 return 0; @@ -167,11 +164,11 @@ static int cbc_encrypt(struct skcipher_request *req) unsigned int blocks =3D walk.nbytes / AES_BLOCK_SIZE; =20 /* fall back to the non-bitsliced NEON implementation */ - kernel_neon_begin(); - neon_aes_cbc_encrypt(walk.dst.virt.addr, walk.src.virt.addr, - ctx->enc, ctx->key.rounds, blocks, - walk.iv); - kernel_neon_end(); + scoped_ksimd() + neon_aes_cbc_encrypt(walk.dst.virt.addr, + walk.src.virt.addr, + ctx->enc, ctx->key.rounds, blocks, + walk.iv); err =3D skcipher_walk_done(&walk, walk.nbytes % AES_BLOCK_SIZE); } return err; @@ -193,11 +190,10 @@ static int cbc_decrypt(struct skcipher_request *req) blocks =3D round_down(blocks, walk.stride / AES_BLOCK_SIZE); =20 - kernel_neon_begin(); - aesbs_cbc_decrypt(walk.dst.virt.addr, walk.src.virt.addr, - ctx->key.rk, ctx->key.rounds, blocks, - walk.iv); - kernel_neon_end(); + scoped_ksimd() + aesbs_cbc_decrypt(walk.dst.virt.addr, walk.src.virt.addr, + ctx->key.rk, ctx->key.rounds, blocks, + walk.iv); err =3D skcipher_walk_done(&walk, walk.nbytes - blocks * AES_BLOCK_SIZE); } @@ -220,30 +216,32 @@ static int ctr_encrypt(struct skcipher_request *req) const u8 *src =3D walk.src.virt.addr; u8 *dst =3D walk.dst.virt.addr; =20 - kernel_neon_begin(); - if (blocks >=3D 8) { - aesbs_ctr_encrypt(dst, src, ctx->key.rk, ctx->key.rounds, - blocks, walk.iv); - dst +=3D blocks * AES_BLOCK_SIZE; - src +=3D blocks * AES_BLOCK_SIZE; - } - if (nbytes && walk.nbytes =3D=3D walk.total) { - u8 buf[AES_BLOCK_SIZE]; - u8 *d =3D dst; - - if (unlikely(nbytes < AES_BLOCK_SIZE)) - src =3D dst =3D memcpy(buf + sizeof(buf) - nbytes, - src, nbytes); - - neon_aes_ctr_encrypt(dst, src, ctx->enc, ctx->key.rounds, - nbytes, walk.iv); + scoped_ksimd() { + if (blocks >=3D 8) { + aesbs_ctr_encrypt(dst, src, ctx->key.rk, + ctx->key.rounds, blocks, + walk.iv); + dst +=3D blocks * AES_BLOCK_SIZE; + src +=3D blocks * AES_BLOCK_SIZE; + } + if (nbytes && walk.nbytes =3D=3D walk.total) { + u8 buf[AES_BLOCK_SIZE]; + u8 *d =3D dst; + + if (unlikely(nbytes < AES_BLOCK_SIZE)) + src =3D dst =3D memcpy(buf + sizeof(buf) - + nbytes, src, nbytes); + + neon_aes_ctr_encrypt(dst, src, ctx->enc, + ctx->key.rounds, nbytes, + walk.iv); =20 - if (unlikely(nbytes < AES_BLOCK_SIZE)) - memcpy(d, dst, nbytes); + if (unlikely(nbytes < AES_BLOCK_SIZE)) + memcpy(d, dst, nbytes); =20 - nbytes =3D 0; + nbytes =3D 0; + } } - kernel_neon_end(); err =3D skcipher_walk_done(&walk, nbytes); } return err; @@ -320,33 +318,33 @@ static int __xts_crypt(struct skcipher_request *req, = bool encrypt, in =3D walk.src.virt.addr; nbytes =3D walk.nbytes; =20 - kernel_neon_begin(); - if (blocks >=3D 8) { - if (first =3D=3D 1) - neon_aes_ecb_encrypt(walk.iv, walk.iv, - ctx->twkey, - ctx->key.rounds, 1); - first =3D 2; - - fn(out, in, ctx->key.rk, ctx->key.rounds, blocks, - walk.iv); - - out +=3D blocks * AES_BLOCK_SIZE; - in +=3D blocks * AES_BLOCK_SIZE; - nbytes -=3D blocks * AES_BLOCK_SIZE; + scoped_ksimd() { + if (blocks >=3D 8) { + if (first =3D=3D 1) + neon_aes_ecb_encrypt(walk.iv, walk.iv, + ctx->twkey, + ctx->key.rounds, 1); + first =3D 2; + + fn(out, in, ctx->key.rk, ctx->key.rounds, blocks, + walk.iv); + + out +=3D blocks * AES_BLOCK_SIZE; + in +=3D blocks * AES_BLOCK_SIZE; + nbytes -=3D blocks * AES_BLOCK_SIZE; + } + if (walk.nbytes =3D=3D walk.total && nbytes > 0) { + if (encrypt) + neon_aes_xts_encrypt(out, in, ctx->cts.key_enc, + ctx->key.rounds, nbytes, + ctx->twkey, walk.iv, first); + else + neon_aes_xts_decrypt(out, in, ctx->cts.key_dec, + ctx->key.rounds, nbytes, + ctx->twkey, walk.iv, first); + nbytes =3D first =3D 0; + } } - if (walk.nbytes =3D=3D walk.total && nbytes > 0) { - if (encrypt) - neon_aes_xts_encrypt(out, in, ctx->cts.key_enc, - ctx->key.rounds, nbytes, - ctx->twkey, walk.iv, first); - else - neon_aes_xts_decrypt(out, in, ctx->cts.key_dec, - ctx->key.rounds, nbytes, - ctx->twkey, walk.iv, first); - nbytes =3D first =3D 0; - } - kernel_neon_end(); err =3D skcipher_walk_done(&walk, nbytes); } =20 @@ -369,14 +367,16 @@ static int __xts_crypt(struct skcipher_request *req, = bool encrypt, in =3D walk.src.virt.addr; nbytes =3D walk.nbytes; =20 - kernel_neon_begin(); - if (encrypt) - neon_aes_xts_encrypt(out, in, ctx->cts.key_enc, ctx->key.rounds, - nbytes, ctx->twkey, walk.iv, first); - else - neon_aes_xts_decrypt(out, in, ctx->cts.key_dec, ctx->key.rounds, - nbytes, ctx->twkey, walk.iv, first); - kernel_neon_end(); + scoped_ksimd() { + if (encrypt) + neon_aes_xts_encrypt(out, in, ctx->cts.key_enc, + ctx->key.rounds, nbytes, ctx->twkey, + walk.iv, first); + else + neon_aes_xts_decrypt(out, in, ctx->cts.key_dec, + ctx->key.rounds, nbytes, ctx->twkey, + walk.iv, first); + } =20 return skcipher_walk_done(&walk, 0); } --=20 2.51.0.710.ga91ca5db03-goog From nobody Sat Feb 7 18:46:42 2026 Received: from mail-ed1-f73.google.com (mail-ed1-f73.google.com [209.85.208.73]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4E1EF2FAC14 for ; Wed, 8 Oct 2025 15:47:00 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.208.73 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1759938422; cv=none; b=TRjmHY58tDqcalBExruQ4Zoa9F80kGRPLW4AxDQyiQVk6kVhx5ZGg7viIkSNh323glEXisK6iugXlGYGIhBP4YuHWuAS3hM/59D34XV6Ds/M+i3WK7ukFGkLz6OyxnqFSHzTP5tCsvH+lDRzyj7UxoWfbgDGzE9R9iarX5uU30A= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1759938422; c=relaxed/simple; bh=qCZByrWzrmyH3UqPPmysPCtXP60NqDPuaPfmddQNwAA=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=Uqy6L3EfZMNFzqjtbmCk7OM39tS+AU17PYNDFLkbVTP/qeeA7yd6H7Fb/ai62o6FIbr5e6fXlseo6BMsXCHCELrD/vKlQuZzcLE/37suPELLZlJppkjyId6LYrIxe3s3/8ymirfGQ6l3ZQ5tWvEDQCvgFG3A+zhz1FLjWSmoFFc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=jsL5e5A6; arc=none smtp.client-ip=209.85.208.73 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="jsL5e5A6" Received: by mail-ed1-f73.google.com with SMTP id 4fb4d7f45d1cf-6215a926e31so6866851a12.3 for ; Wed, 08 Oct 2025 08:47:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1759938419; x=1760543219; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=v0Zu7n3LaOo+I8CYEuQkDSSyekyMvHu55FmTnZPR1Fs=; b=jsL5e5A625yqUhXudNynVVwIpRANnyLusCkB+HGpAEYqnxOAQncddEDMQ5yUSJNcjW AnjgzN8d/m5nneQLiZdjPMnKJuQtdQW4v6FfDjtODvhBXylBuNWcoJc+GZhwH/QnS2j3 W7r8g44MiUu0gT0SbOxPbjUJivjJ9m6JF6CfXgVwZ5apl+lwBonrCGkkH9r17l38OKeR H28RoqDNqv+ILtNo5l/ixHmVdccNUnY4sUbfleUyHJJkYXaHJrwxfOFhyM5NwJrDmsJz OnSzB9WNax9MHtjb8Z3CXzIa/t8Dj5VfA5cd6ojDkXoqbII6woVnXYWvIFZXb5Huv12j 6Www== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1759938419; x=1760543219; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=v0Zu7n3LaOo+I8CYEuQkDSSyekyMvHu55FmTnZPR1Fs=; b=YpBK33mDoW8mWGFN0k99HnW2Re9wKj7RMQP45FvaR2Ff7WEyLSo5MlaOfMZhGRyM+h ATDlnmQ0VuZSE+CuXH7eo7OAhQmIOPrrw6RNHERBnmN3YHwWYxxsz1mzw2aaeJtnA+8L UK8cs8YKoy+JBVI3ymfPa90mZMeb5mFxYVDAVJd2eT9OiZ+nY2wAQ8NZG8iGR3O1Y7D+ AfDjP6i8PWrjNkkNWbdEjkW9N+EnieaHvsPIpuz8lCHi/ePRedQTVmN0Y2fN3Td2zmQX 0G5XrqsVQrraVt1OI4mZyx+kZiZD8VLkhl+zAa7fSbMk+NKerirCcrlNn6go1dR+FBN/ PD8Q== X-Gm-Message-State: AOJu0YxT57Gisi3oJQg48vB3kgWc/Tf9OzI/MHa+QA3Xz8gngCODFePB I9Ly792hl9CxpcXJRsLs7ZgMtl5GReTPWTKBm4y3mFWKoy44y07iLml46gcutnWixscgWhAxgQ= = X-Google-Smtp-Source: AGHT+IGav2hs8j+BD8Ez6wtFb9ryWmg+5Cqi/FdQRo4iR29OSB6a6/a1XkaaR/BdhniWetTsV5AzKnAF X-Received: from edg11.prod.google.com ([2002:a05:6402:23cb:b0:633:88ce:6ba1]) (user=ardb job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6402:1ed0:b0:639:e469:8ad1 with SMTP id 4fb4d7f45d1cf-639e469a2e6mr1875949a12.20.1759938418627; Wed, 08 Oct 2025 08:46:58 -0700 (PDT) Date: Wed, 8 Oct 2025 17:45:46 +0200 In-Reply-To: <20251008154533.3089255-23-ardb+git@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20251008154533.3089255-23-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=2613; i=ardb@kernel.org; h=from:subject; bh=5MqfiBK+zzoh8ZIeemHCwImcnXO2p3OW2OkltAZA92c=; b=owGbwMvMwCVmkMcZplerG8N4Wi2JIeNZu6PQv0s/3R6FHTWrvdppvE8hc+vPH+tbHDdp7mpvM V4evbWxo5SFQYyLQVZMkUVg9t93O09PlKp1niULM4eVCWQIAxenAEwkezEjw0qBPKbzws7Ls/O1 ouV3v2yt4VpzsfVj3Rr2CQcy7kcG6DD8s28KWpu+NjTuYv5NwwkVnJz71GZnX1lqLbpLJixgrcQ EFgA= X-Mailer: git-send-email 2.51.0.710.ga91ca5db03-goog Message-ID: <20251008154533.3089255-35-ardb+git@google.com> Subject: [PATCH v3 12/21] crypto/arm64: aes-gcm - Switch to 'ksimd' scoped guard API From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: linux-kernel@vger.kernel.org, linux-crypto@vger.kernel.org, herbert@gondor.apana.org.au, ebiggers@kernel.org, Ard Biesheuvel Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Ard Biesheuvel Switch to the more abstract 'scoped_ksimd()' API, which will be modified in a future patch to transparently allocate a kernel mode FP/SIMD state buffer on the stack, so that kernel mode FP/SIMD code remains preemptible in principe, but without the memory overhead that adds 528 bytes to the size of struct task_struct. Reviewed-by: Eric Biggers Signed-off-by: Ard Biesheuvel --- arch/arm64/crypto/ghash-ce-glue.c | 27 ++++++++++---------- 1 file changed, 13 insertions(+), 14 deletions(-) diff --git a/arch/arm64/crypto/ghash-ce-glue.c b/arch/arm64/crypto/ghash-ce= -glue.c index 4995b6e22335..7951557a285a 100644 --- a/arch/arm64/crypto/ghash-ce-glue.c +++ b/arch/arm64/crypto/ghash-ce-glue.c @@ -5,7 +5,6 @@ * Copyright (C) 2014 - 2018 Linaro Ltd. */ =20 -#include #include #include #include @@ -22,6 +21,8 @@ #include #include =20 +#include + MODULE_DESCRIPTION("GHASH and AES-GCM using ARMv8 Crypto Extensions"); MODULE_AUTHOR("Ard Biesheuvel "); MODULE_LICENSE("GPL v2"); @@ -74,9 +75,8 @@ void ghash_do_simd_update(int blocks, u64 dg[], const cha= r *src, u64 const h[][2], const char *head)) { - kernel_neon_begin(); - simd_update(blocks, dg, src, key->h, head); - kernel_neon_end(); + scoped_ksimd() + simd_update(blocks, dg, src, key->h, head); } =20 /* avoid hogging the CPU for too long */ @@ -329,11 +329,10 @@ static int gcm_encrypt(struct aead_request *req, char= *iv, int assoclen) tag =3D NULL; } =20 - kernel_neon_begin(); - pmull_gcm_encrypt(nbytes, dst, src, ctx->ghash_key.h, - dg, iv, ctx->aes_key.key_enc, nrounds, - tag); - kernel_neon_end(); + scoped_ksimd() + pmull_gcm_encrypt(nbytes, dst, src, ctx->ghash_key.h, + dg, iv, ctx->aes_key.key_enc, nrounds, + tag); =20 if (unlikely(!nbytes)) break; @@ -399,11 +398,11 @@ static int gcm_decrypt(struct aead_request *req, char= *iv, int assoclen) tag =3D NULL; } =20 - kernel_neon_begin(); - ret =3D pmull_gcm_decrypt(nbytes, dst, src, ctx->ghash_key.h, - dg, iv, ctx->aes_key.key_enc, - nrounds, tag, otag, authsize); - kernel_neon_end(); + scoped_ksimd() + ret =3D pmull_gcm_decrypt(nbytes, dst, src, + ctx->ghash_key.h, + dg, iv, ctx->aes_key.key_enc, + nrounds, tag, otag, authsize); =20 if (unlikely(!nbytes)) break; --=20 2.51.0.710.ga91ca5db03-goog From nobody Sat Feb 7 18:46:42 2026 Received: from mail-wm1-f74.google.com (mail-wm1-f74.google.com [209.85.128.74]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4A85429D29F for ; Wed, 8 Oct 2025 15:47:01 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.74 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1759938423; cv=none; b=bGtmHdgy3RLCpX7WipYZvnLvNfnnvV/vQ9syOu49pzFq4/AN7SyxJUPT8HgpUZuSsHGqO47FrJjaSrEN3hJLGnGhfG8P1j+xgqZiaNhgnu5bdf0noqEBBqc8OS/BTaryVHWUFK1mrIwWPo3brArqJJ7wZ7GC7/f0geHEvkelelo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1759938423; c=relaxed/simple; bh=8/2or9yCCRuAQ3rBPtjIaQu4DThn+Ww7LOmUhgaq6a4=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=Uru+noT2I4pBaFy4pE9AKfN4LT5NEPPbogjKA7k6mc0w/AU2AcG3P5sf1ZeGJEV/Fgp/1/AvI/VeQYXARIAp1EOpt6Y6kzd+7RLAQmhBzxk9+7aanNWqN0tV1JhaTl3hs6ocA0/wfHFugy3Mtgt887MzLCTQvHpHss0U0SXbXFQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=bFc2Hgb0; arc=none smtp.client-ip=209.85.128.74 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="bFc2Hgb0" Received: by mail-wm1-f74.google.com with SMTP id 5b1f17b1804b1-46e38bd6680so26784785e9.1 for ; Wed, 08 Oct 2025 08:47:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1759938420; x=1760543220; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=Ez+4eKtd+5d+DlN9/UyWopIVxOEbKik+aYqVxPQpTVQ=; b=bFc2Hgb0zvoq3jKqzFnDT6qW3Z69YmMOD1oVa9V7rqYs3VrG2vTBqWSrl0K8bp8S0U mqMXzhi5oH96/7ZNHvtJG/+Trimhcnok9gXL6ZCqi1dyXOZ49By+cXIT05TsEKORmwWX OvHxHSeHw72xKPDfFgZjzRS2uv7qCsCKcMo0bb/nsTsxDjFkcBFoDG4XNQuP+4y46Iei Rb2pWyjzt9ztizAwnkZ0iemkNunZx1Mn0cJiNa3pe1Kok4+1/NFXpn4ynjd85RpJUP1r SDaKEpLbbjvuNx59BGg+By+y16iUco1GRyDS14kJyxdGOYIO8DbMcOeFRDUk6TrP39bH fz7A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1759938420; x=1760543220; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=Ez+4eKtd+5d+DlN9/UyWopIVxOEbKik+aYqVxPQpTVQ=; b=ERKLDoCy1b06n8ucWlZaYoqwVqVjPRxZHXMPQZsgFQ3Ol0ryid/bA27yjSc7BXNPXM NZ5inz4Vt+ZpzxfPzwFP/dEINiJqyOufZ28r6ky+miR1QRt1amgS8th6kkAR+cwfRJoT eQISgPTnWWxkwYGQYSTcgb0xOzoOf+MThaeSCrpdYhdJz6zIlApss30V5TAy2SZuwaB7 AcsAR+ukV+LgveNiGaBT0U18SIUm1JM74NJ9V9CbQw2+sygdgWe4FJMbSFyiDp/Iy2kM YOZrkQsDEdCra8eAOTLcg9EFwkucOEQeujMgU6ypyUnLHLj5kGe94YFfIeZWYHnV9Sxe jPYg== X-Gm-Message-State: AOJu0YyGfmCXIOCzQzThP5wk+wU+GgbC5UJs4Su9361qh1OfhKoUOpNm O5zQ5ekw4RLSqEj3DfT0VMb24aXYce+/kWO65KEE+ighMUqpzt9pR3dnBYXPF7u7LFIcSymcTw= = X-Google-Smtp-Source: AGHT+IGTYM4Pf/38/W7UXSGOfa/Cz0TebUDvS0+woKJDjAs0iBAdrVv8aCM0ALs/qJZsRftZYcTgSa7a X-Received: from wmvx6.prod.google.com ([2002:a05:600d:42e6:b0:46e:6a75:2910]) (user=ardb job=prod-delivery.src-stubby-dispatcher) by 2002:a05:600c:c162:b0:46e:4a60:ea2c with SMTP id 5b1f17b1804b1-46fa9b17e57mr26033895e9.37.1759938419976; Wed, 08 Oct 2025 08:46:59 -0700 (PDT) Date: Wed, 8 Oct 2025 17:45:47 +0200 In-Reply-To: <20251008154533.3089255-23-ardb+git@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20251008154533.3089255-23-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=1189; i=ardb@kernel.org; h=from:subject; bh=8sN9JxLWvPrUJL2U51Eh+5qEYXWcAYa7eZ5mEETt20c=; b=owGbwMvMwCVmkMcZplerG8N4Wi2JIeNZu/PzMHauZxLCKuERrRdfzF5wf7kDA39FxuHFVmtm5 k9ZGbCyo5SFQYyLQVZMkUVg9t93O09PlKp1niULM4eVCWQIAxenAEzkiCIjw9wiF2ODjCNqLeKT 1y57HJa2ecbP6MIlHaFB7NPv73o48RIjw6d9u7u+haqfKfBVlgpP4bO+/M9SaGnMwn/JDUovz4U dZAIA X-Mailer: git-send-email 2.51.0.710.ga91ca5db03-goog Message-ID: <20251008154533.3089255-36-ardb+git@google.com> Subject: [PATCH v3 13/21] crypto/arm64: nhpoly1305 - Switch to 'ksimd' scoped guard API From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: linux-kernel@vger.kernel.org, linux-crypto@vger.kernel.org, herbert@gondor.apana.org.au, ebiggers@kernel.org, Ard Biesheuvel Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Ard Biesheuvel Switch to the more abstract 'scoped_ksimd()' API, which will be modified in a future patch to transparently allocate a kernel mode FP/SIMD state buffer on the stack, so that kernel mode FP/SIMD code remains preemptible in principe, but without the memory overhead that adds 528 bytes to the size of struct task_struct. Reviewed-by: Eric Biggers Signed-off-by: Ard Biesheuvel --- arch/arm64/crypto/nhpoly1305-neon-glue.c | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/arch/arm64/crypto/nhpoly1305-neon-glue.c b/arch/arm64/crypto/n= hpoly1305-neon-glue.c index e4a0b463f080..013de6ac569a 100644 --- a/arch/arm64/crypto/nhpoly1305-neon-glue.c +++ b/arch/arm64/crypto/nhpoly1305-neon-glue.c @@ -25,9 +25,8 @@ static int nhpoly1305_neon_update(struct shash_desc *desc, do { unsigned int n =3D min_t(unsigned int, srclen, SZ_4K); =20 - kernel_neon_begin(); - crypto_nhpoly1305_update_helper(desc, src, n, nh_neon); - kernel_neon_end(); + scoped_ksimd() + crypto_nhpoly1305_update_helper(desc, src, n, nh_neon); src +=3D n; srclen -=3D n; } while (srclen); --=20 2.51.0.710.ga91ca5db03-goog From nobody Sat Feb 7 18:46:42 2026 Received: from mail-wm1-f74.google.com (mail-wm1-f74.google.com [209.85.128.74]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6B0512FB09C for ; Wed, 8 Oct 2025 15:47:02 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.74 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1759938424; cv=none; b=JHHDVaxwv8JKuFmGCkYBJoniHZ22/r0EtQ2FvpNnww4pMSSHCpZXEU10HoFzqqc4C23vP61FpYV3PGpYFqlcYKGM1Tt5bdzwhzuqyFYr79KMYIaq63LiEH7DLFK4lUYsavucGiw07zMzcCoBpr8pNV9ogVatUifhtCtpyyQdlfE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1759938424; c=relaxed/simple; bh=CY8sASFNr+cydSaRmK+XCtThWiaAY5mURJItMU/mY60=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=QPb1VXYhTWW+//0cwJdwc3AZoplx54WKItj1tKa5v11tFKMyFLVI5h0GrihHa38DcYRr3j9ABMrQPEnLnIMueuArCAB+K66M9PfFjjNpsD75/1KQ95xD8z7wyAe1QhNIJZer9yryiH8JWskjbxxSEfk0tFOCZo1lAGMPpN2al7Q= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=sCu12P0x; arc=none smtp.client-ip=209.85.128.74 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="sCu12P0x" Received: by mail-wm1-f74.google.com with SMTP id 5b1f17b1804b1-46e36686ca1so71991885e9.2 for ; Wed, 08 Oct 2025 08:47:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1759938421; x=1760543221; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=ZX+FOTwqwmP67LIKWGzioSokTb9JzlGhSXeNKUrBT70=; b=sCu12P0xd+iMvFV591zeA+TvF5mSlnq5o2NA5Hl6YD9Tlp0wji6JUcpt9vn3SgUchA 73MWnc8WChsK9gvIZS/dobatwDaDqU83Vgb+UTsIjFf1QYVDKtFetfLtwxZ0S4v7lT9C Tovc/QCAvUWjUoRkkgzP/OdXGMd9hQSlDPY+hgpkmaclzw+6YGGvL6k0O7yal+IpDNrM T+YzVvVWTdIAPkVSL1kXn5BO90cbEv2Mo1euPa8IOins3KaT9mE170lW5bjXa3H7yXm4 g+VnW08Dy6ytjMaRIeSuZa4vLyXLS6kiO/XZ+QsUUdd7W7MnBgQZAnGRNZ0ybPW0xzTx nCTg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1759938421; x=1760543221; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=ZX+FOTwqwmP67LIKWGzioSokTb9JzlGhSXeNKUrBT70=; b=mvK78mKZpZbE1pG0wWH29z8cZ+KRi9s4EwKC5kTOwewkBAuUs0QSv9o1oofEfoYzC9 NWz7e3RK3nMT0jZvgYOuTKEQXHA/Hqf0wgWkVAyUoW8ANbRGPZzLaou2Qa6EkZG7Rl/I DJmEyzcDM/CrWfv90I/SzWOqg5lC+kd/WIcy9x6cQreTCwIUIWlKWsFwydtHc6+ewNBI kS7lWocgLnN5CaBj5NegtYL7EgCrxf5tK+7/845Iypd3O7Bu7hj0qL9LaFyd/u1P1I/L QXNc5xEF4QrZG8Kn7S5Q7NGu+GeIocIB+5tF+4sKpx5xCWV9UFfrauTz7tbaMaRkDPxy 7Z7g== X-Gm-Message-State: AOJu0YzLyTNFe4nkO0Uvdmp/MwUYYBpuOJ/isSGhaUtry5hLnjCaUvux nX7iRuMcDhfYJZ3Cg8EFRDexKfSSN5OoAXZiskunB29wummFii0GZL23UODkBj5eFC6MwFp8ZA= = X-Google-Smtp-Source: AGHT+IEjiKxPM3P1jo2NCQMtIkDGka2bla0TgaKSnshbu1vw6e4DEdLUwhej0pREcTGcshyShoBFv3ri X-Received: from wmbjx19.prod.google.com ([2002:a05:600c:5793:b0:45f:2306:167]) (user=ardb job=prod-delivery.src-stubby-dispatcher) by 2002:a05:600c:3b07:b0:46e:1fb9:5497 with SMTP id 5b1f17b1804b1-46fa9af84fdmr26813135e9.18.1759938420853; Wed, 08 Oct 2025 08:47:00 -0700 (PDT) Date: Wed, 8 Oct 2025 17:45:48 +0200 In-Reply-To: <20251008154533.3089255-23-ardb+git@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20251008154533.3089255-23-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=1693; i=ardb@kernel.org; h=from:subject; bh=dcVJ+FEDVLktPv+MhuXF3k3ccZKN6EYdh2F7GcW5Gk4=; b=owGbwMvMwCVmkMcZplerG8N4Wi2JIeNZu9u/hFcu0+49t26d836RcFnpng+zf7O881h/33/d6 eqyzCUGHaUsDGJcDLJiiiwCs/++23l6olSt8yxZmDmsTCBDGLg4BWAiotMY/qk+2FRhLna5uZcx 5KXyndDC70EpWYYruRLTvz9gOn1bTZSR4eo0pYydhffVZG+Xdm4++epYQY1OzPEfXx4v1Ezvu1s UzwcA X-Mailer: git-send-email 2.51.0.710.ga91ca5db03-goog Message-ID: <20251008154533.3089255-37-ardb+git@google.com> Subject: [PATCH v3 14/21] crypto/arm64: polyval - Switch to 'ksimd' scoped guard API From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: linux-kernel@vger.kernel.org, linux-crypto@vger.kernel.org, herbert@gondor.apana.org.au, ebiggers@kernel.org, Ard Biesheuvel Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Ard Biesheuvel Switch to the more abstract 'scoped_ksimd()' API, which will be modified in a future patch to transparently allocate a kernel mode FP/SIMD state buffer on the stack, so that kernel mode FP/SIMD code remains preemptible in principe, but without the memory overhead that adds 528 bytes to the size of struct task_struct. Reviewed-by: Eric Biggers Signed-off-by: Ard Biesheuvel --- arch/arm64/crypto/polyval-ce-glue.c | 12 +++++------- 1 file changed, 5 insertions(+), 7 deletions(-) diff --git a/arch/arm64/crypto/polyval-ce-glue.c b/arch/arm64/crypto/polyva= l-ce-glue.c index c4e653688ea0..51eefbe97885 100644 --- a/arch/arm64/crypto/polyval-ce-glue.c +++ b/arch/arm64/crypto/polyval-ce-glue.c @@ -15,7 +15,7 @@ * ARMv8 Crypto Extensions instructions to implement the finite field oper= ations. */ =20 -#include +#include #include #include #include @@ -45,16 +45,14 @@ asmlinkage void pmull_polyval_mul(u8 *op1, const u8 *op= 2); static void internal_polyval_update(const struct polyval_tfm_ctx *keys, const u8 *in, size_t nblocks, u8 *accumulator) { - kernel_neon_begin(); - pmull_polyval_update(keys, in, nblocks, accumulator); - kernel_neon_end(); + scoped_ksimd() + pmull_polyval_update(keys, in, nblocks, accumulator); } =20 static void internal_polyval_mul(u8 *op1, const u8 *op2) { - kernel_neon_begin(); - pmull_polyval_mul(op1, op2); - kernel_neon_end(); + scoped_ksimd() + pmull_polyval_mul(op1, op2); } =20 static int polyval_arm64_setkey(struct crypto_shash *tfm, --=20 2.51.0.710.ga91ca5db03-goog From nobody Sat Feb 7 18:46:42 2026 Received: from mail-wr1-f73.google.com (mail-wr1-f73.google.com [209.85.221.73]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6E9DC2FBDF1 for ; Wed, 8 Oct 2025 15:47:03 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.221.73 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1759938425; cv=none; b=pqcHYz/rUSNz4tJVWh2aSdvwdcU8ke4KKhZgU7JGVt1hDcZcqzeRnkfQjE2SUeFVjovJUI3qj9p0hI4R6XYlq8w00nhGSPCC95Jw1XUoRboDinCn26jZYsu4Cqv0vtTXPBr0G46jKbfFxlo0/yEOvH+u2xS0QwLWct5fhMJ7ZUY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1759938425; c=relaxed/simple; bh=FQD/z+6lXiIcL/7kTxCFE2DLvjr2aRXNJXgQtJfNEVM=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=fG1buJ5xVwSWGjUGjYVcZKowrXczaW+lKMylCxWOvpm2WbgB6NXYnIbKZZOrfzU9fuAFHpUEAyQGgcYOs5GjZzvYgBpjqlnu3yVS3dAnGFWOqSZ5m7ZTK4xP3vBFFhCDzF7kEi8c7GVqO454/G7Rk4P8ij98PWVHUF1p9uIz6uA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=QUv55FJ4; arc=none smtp.client-ip=209.85.221.73 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="QUv55FJ4" Received: by mail-wr1-f73.google.com with SMTP id ffacd0b85a97d-426c4d733c4so34942f8f.3 for ; Wed, 08 Oct 2025 08:47:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1759938422; x=1760543222; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=s9LVoYmYTtGptKcGaIqz4TtHmJpnpszkhgfxtgMZN1E=; b=QUv55FJ41cyPQSy1Go9/Fx3VS66mEDNWnKAcDNwDq2WMUilUm+w2uRu5CzwU7NWdYM 4X4ClEq1r7Xg1EJxQnMpPNElfXYTp3Kq8pkbC6N1jrW9xdfu/MqLsruvxONNz0MBucgM 9cGSlHfARvzdkAp2LIYvuD0cqEBn8p61Gp0YASB54kmvDY4ZVyr8jrgWrV3wx8RfU0l8 SV+VMFR49gnXviJxyHPzfT+bM2ZgNppSc5S0hMiusfIr9v9Cij/qkq7VesmQZ93HfCzL 10AMs86L3i0b/DIQolYfyCS9gxxK+SverYdiACrwHW0aHIKmrpxiK18kKygh6TsAtI7L W6Zg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1759938422; x=1760543222; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=s9LVoYmYTtGptKcGaIqz4TtHmJpnpszkhgfxtgMZN1E=; b=URj0H8AeaA166E81y/OhQk8M9jlndhxHhD7hyBwb8aQ1MVTT9ODpzpx3H5C1P0H4F4 PVLz7PN5+IC+WoqrNLw7w6zBCzvipDSShziKGOu9qQE7qrEDFjRjkhi+jKcBGgzuXvao +vLPujwwmKQ9dl2S0ntHArG0H5xFRrtoioCpZeEm4zUb17jpjZJEbxiPdwNdmY/O3Uom d9nxH9+nr2jdYDUrRETl0mimYEGn1cFP/AKXO16bKm98pJtMuXRRM/7Lejos5jNMenjq drloP7ZeHVVdcw9QfFkCVWTfLm6HkuMG7+boV5s3t/jfdZB5jLvEXzQweGO7H1TaQUIl m9UA== X-Gm-Message-State: AOJu0YwW8gHH4LJNqIj48gdIQ0EhngZPLjVJghvMOAFQmUf1aIByZg/Z SpoDqz8D7mf9Zer/rcUoWiQBE3UolJPGhkii8Pt3YwMy7LQ/zydw2Brl2UHUHYUjN655iw+l4g= = X-Google-Smtp-Source: AGHT+IHpvSRci2gpv7ZU1IAhZM+xVMFiXHJZypmfvURMXvnTa0pwgCk5eYUgWECkxlzPguEf0Tz1w3HD X-Received: from wmbd24.prod.google.com ([2002:a05:600c:58d8:b0:46e:4943:289b]) (user=ardb job=prod-delivery.src-stubby-dispatcher) by 2002:a5d:5d03:0:b0:3ec:e0d0:60e5 with SMTP id ffacd0b85a97d-42667177bd4mr2760140f8f.15.1759938421849; Wed, 08 Oct 2025 08:47:01 -0700 (PDT) Date: Wed, 8 Oct 2025 17:45:49 +0200 In-Reply-To: <20251008154533.3089255-23-ardb+git@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20251008154533.3089255-23-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=1531; i=ardb@kernel.org; h=from:subject; bh=KbLRn3K+fhgTbW1ynT/DmRKNIqX5FUyDJ4LA4JLPqDs=; b=kA0DAAoWMG4JVi59LVwByyZiAGjmh0mgHwxNRpFR0uLeyEM/7Ru0XZRuTjmbCFMD+pwVKCMPv Ih1BAAWCgAdFiEEEJv97rnLkRp9Q5odMG4JVi59LVwFAmjmh0kACgkQMG4JVi59LVyXkwD9GvEh GB4myAwRpn32iKqBOQuzHl9IIEadlLMcTujoQkgA/ie4fhaKFva43l0IM40cGryU1yaSMyBQQI8 ARZPeCzsG X-Mailer: git-send-email 2.51.0.710.ga91ca5db03-goog Message-ID: <20251008154533.3089255-38-ardb+git@google.com> Subject: [PATCH v3 15/21] crypto/arm64: sha3 - Switch to 'ksimd' scoped guard API From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: linux-kernel@vger.kernel.org, linux-crypto@vger.kernel.org, herbert@gondor.apana.org.au, ebiggers@kernel.org, Ard Biesheuvel Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Ard Biesheuvel Switch to the more abstract 'scoped_ksimd()' API, which will be modified in a future patch to transparently allocate a kernel mode FP/SIMD state buffer on the stack, so that kernel mode FP/SIMD code remains preemptible in principe, but without the memory overhead that adds 528 bytes to the size of struct task_struct. Reviewed-by: Eric Biggers Signed-off-by: Ard Biesheuvel --- arch/arm64/crypto/sha3-ce-glue.c | 10 ++++------ 1 file changed, 4 insertions(+), 6 deletions(-) diff --git a/arch/arm64/crypto/sha3-ce-glue.c b/arch/arm64/crypto/sha3-ce-g= lue.c index b4f1001046c9..22732760edd3 100644 --- a/arch/arm64/crypto/sha3-ce-glue.c +++ b/arch/arm64/crypto/sha3-ce-glue.c @@ -46,9 +46,8 @@ static int sha3_update(struct shash_desc *desc, const u8 = *data, do { int rem; =20 - kernel_neon_begin(); - rem =3D sha3_ce_transform(sctx->st, data, blocks, ds); - kernel_neon_end(); + scoped_ksimd() + rem =3D sha3_ce_transform(sctx->st, data, blocks, ds); data +=3D (blocks - rem) * bs; blocks =3D rem; } while (blocks); @@ -73,9 +72,8 @@ static int sha3_finup(struct shash_desc *desc, const u8 *= src, unsigned int len, memset(block + len, 0, bs - len); block[bs - 1] |=3D 0x80; =20 - kernel_neon_begin(); - sha3_ce_transform(sctx->st, block, 1, ds); - kernel_neon_end(); + scoped_ksimd() + sha3_ce_transform(sctx->st, block, 1, ds); memzero_explicit(block , sizeof(block)); =20 for (i =3D 0; i < ds / 8; i++) --=20 2.51.0.710.ga91ca5db03-goog From nobody Sat Feb 7 18:46:42 2026 Received: from mail-wm1-f73.google.com (mail-wm1-f73.google.com [209.85.128.73]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 86C452FC032 for ; Wed, 8 Oct 2025 15:47:04 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.73 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1759938427; cv=none; b=piS/GUUtca/gfJi02JMqyD3wGpYOynVCS35CFkabipSistHKhjPaU+DTX0vLR6DVEeZ0rtcSVQivTMN7FTlLS0sdadJcs4DXuuv2liZzzfmwyFHfpi09CSN8EwtmneaLI7+3zsCuNXkxnCnXIoRxrrh0A9jFH4cNm3jCb8+n7+o= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1759938427; c=relaxed/simple; bh=2Y4ovck1VNEDJT/eJTVGRHTacMKXujIutu/8KsCMop0=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=YywoeDC3CkzhO4ZXtBn111gDVERNRQSlPZ4dqCv+2JhcS7l1ZKSEk1APAcfQ2k246hJiHERxRDCKERTpTUwI6FdQjeTv+emgvOhKHpXko3iuoXxxFRj23R9RgZ+UYwSYQkXUNHGkh1KRnfaGtsa1OhgQZq24vJ+7vUZ55OpfxAQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=pxg5SCVK; arc=none smtp.client-ip=209.85.128.73 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="pxg5SCVK" Received: by mail-wm1-f73.google.com with SMTP id 5b1f17b1804b1-46e407c600eso40374935e9.3 for ; Wed, 08 Oct 2025 08:47:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1759938423; x=1760543223; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=rx/I2mYB3NrCk/vXv2cfgvi3kC2D2Tey/+p9VMElDFg=; b=pxg5SCVKNNqh8TDdVWkQPvvXN7OooA1Js2WevrLevaGOPsFFxAPViEhZXfzNiP7CKp fK1MNg5VRRQWQ1fvsr8rP/rWPPq3M1tiBCnDVhOXScIgJm+mjPV8ZbhKwVGGUIsSHp/q L8hEzUVODdv4b+r0DLhpI6EKiuCD4tEGSW44De/bOo5/i9eOBGUOXEi37sKng8WnkIDN jwwwkqPsRt6Mv/JS4sOLvk18bxdAdpjebwLk9P7iRgw0/OZpUtQ0zcdrefLjT0t91SZV eFr0QcS+PuEmyP5xcIKbSQD5wKb57yiZ8wsLWEvFaFy8BsmIZPfI1B4c9gxfz98ejxV+ +L/w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1759938423; x=1760543223; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=rx/I2mYB3NrCk/vXv2cfgvi3kC2D2Tey/+p9VMElDFg=; b=rwxUSepbuPdkf6AoffVDhodJ5TlgiGEbYscSFowDkBDjtO0WQOZUFFu/OJJmjNFYqF V9L0HGFuF/ZjTbJaSeIqfFS5/HNzwtfVmnRKHxJeulMx2FlnHQNMfAm+yx95gHQFmYyX NLe0/L0OoGGnO4Nkvv6Fvi2RVQfwV0LFscxIMd4NMkKynrM/JEZcLM9xce7oTgpI4iBT 4jQ9K1M7+xgku/TLHT7doU749udHBxrPSpQthDATD1B4W1JASofGt5CBlee3z0GXBIuV auHpQFT5MqVPvoUVrtgzVIa6llhQddD7XPQAYMkkbNXRWbxoTaA045P0E/x4/YXu4CZv zeRQ== X-Gm-Message-State: AOJu0YxV4P1IhdFyy3tbpxpiVx262xANHqWru91c25gB6cImGnfCwpZQ y/ucJT4rzlDTNSCUcMEv3eZ8FtgRNPne9iG/xugf//jyDvs4BPoEcV/8ONHxu9L0dAMgMAZNkQ= = X-Google-Smtp-Source: AGHT+IGgGXd0YwsnNbBmOjS8U85+nouh42Fv0MQo/IxganC0g4CIjlugauCG5GDY/nU4c6gjRWBTho+b X-Received: from wmvy7.prod.google.com ([2002:a05:600d:4307:b0:46f:aa50:d706]) (user=ardb job=prod-delivery.src-stubby-dispatcher) by 2002:a05:600c:5028:b0:46e:428a:b4c7 with SMTP id 5b1f17b1804b1-46fa9af2ff1mr27394355e9.23.1759938422874; Wed, 08 Oct 2025 08:47:02 -0700 (PDT) Date: Wed, 8 Oct 2025 17:45:50 +0200 In-Reply-To: <20251008154533.3089255-23-ardb+git@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20251008154533.3089255-23-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=3235; i=ardb@kernel.org; h=from:subject; bh=jIptOtZmKOCZwDiEu+JvA9/u7XRZR8J3wVrBatzcfYQ=; b=kA0DAAoWMG4JVi59LVwByyZiAGjmh0uhjRILxkNErSHRqZl7bCOk8F2GbmbuEihaJS52zdYRB 4h1BAAWCgAdFiEEEJv97rnLkRp9Q5odMG4JVi59LVwFAmjmh0sACgkQMG4JVi59LVwfRQD9FKpH Q9qKX3oWwmzrLHOlLSPFAhIUtqFZBlk/vuE50v4BAPJ2xQQvHPUUD1P6GRLlf3WTDIgb0IKpfQw jPHxyFbcH X-Mailer: git-send-email 2.51.0.710.ga91ca5db03-goog Message-ID: <20251008154533.3089255-39-ardb+git@google.com> Subject: [PATCH v3 16/21] crypto/arm64: sm3 - Switch to 'ksimd' scoped guard API From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: linux-kernel@vger.kernel.org, linux-crypto@vger.kernel.org, herbert@gondor.apana.org.au, ebiggers@kernel.org, Ard Biesheuvel Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Ard Biesheuvel Switch to the more abstract 'scoped_ksimd()' API, which will be modified in a future patch to transparently allocate a kernel mode FP/SIMD state buffer on the stack, so that kernel mode FP/SIMD code remains preemptible in principe, but without the memory overhead that adds 528 bytes to the size of struct task_struct. Reviewed-by: Eric Biggers Signed-off-by: Ard Biesheuvel --- arch/arm64/crypto/sm3-ce-glue.c | 15 ++++++++------- arch/arm64/crypto/sm3-neon-glue.c | 16 ++++++---------- 2 files changed, 14 insertions(+), 17 deletions(-) diff --git a/arch/arm64/crypto/sm3-ce-glue.c b/arch/arm64/crypto/sm3-ce-glu= e.c index eac6f5fa0abe..24c1fcfae072 100644 --- a/arch/arm64/crypto/sm3-ce-glue.c +++ b/arch/arm64/crypto/sm3-ce-glue.c @@ -5,7 +5,6 @@ * Copyright (C) 2018 Linaro Ltd */ =20 -#include #include #include #include @@ -13,6 +12,8 @@ #include #include =20 +#include + MODULE_DESCRIPTION("SM3 secure hash using ARMv8 Crypto Extensions"); MODULE_AUTHOR("Ard Biesheuvel "); MODULE_LICENSE("GPL v2"); @@ -25,18 +26,18 @@ static int sm3_ce_update(struct shash_desc *desc, const= u8 *data, { int remain; =20 - kernel_neon_begin(); - remain =3D sm3_base_do_update_blocks(desc, data, len, sm3_ce_transform); - kernel_neon_end(); + scoped_ksimd() { + remain =3D sm3_base_do_update_blocks(desc, data, len, sm3_ce_transform); + } return remain; } =20 static int sm3_ce_finup(struct shash_desc *desc, const u8 *data, unsigned int len, u8 *out) { - kernel_neon_begin(); - sm3_base_do_finup(desc, data, len, sm3_ce_transform); - kernel_neon_end(); + scoped_ksimd() { + sm3_base_do_finup(desc, data, len, sm3_ce_transform); + } return sm3_base_finish(desc, out); } =20 diff --git a/arch/arm64/crypto/sm3-neon-glue.c b/arch/arm64/crypto/sm3-neon= -glue.c index 6c4611a503a3..15f30cc24f32 100644 --- a/arch/arm64/crypto/sm3-neon-glue.c +++ b/arch/arm64/crypto/sm3-neon-glue.c @@ -5,7 +5,7 @@ * Copyright (C) 2022 Tianjia Zhang */ =20 -#include +#include #include #include #include @@ -20,20 +20,16 @@ asmlinkage void sm3_neon_transform(struct sm3_state *ss= t, u8 const *src, static int sm3_neon_update(struct shash_desc *desc, const u8 *data, unsigned int len) { - int remain; - - kernel_neon_begin(); - remain =3D sm3_base_do_update_blocks(desc, data, len, sm3_neon_transform); - kernel_neon_end(); - return remain; + scoped_ksimd() + return sm3_base_do_update_blocks(desc, data, len, + sm3_neon_transform); } =20 static int sm3_neon_finup(struct shash_desc *desc, const u8 *data, unsigned int len, u8 *out) { - kernel_neon_begin(); - sm3_base_do_finup(desc, data, len, sm3_neon_transform); - kernel_neon_end(); + scoped_ksimd() + sm3_base_do_finup(desc, data, len, sm3_neon_transform); return sm3_base_finish(desc, out); } =20 --=20 2.51.0.710.ga91ca5db03-goog From nobody Sat Feb 7 18:46:42 2026 Received: from mail-wm1-f73.google.com (mail-wm1-f73.google.com [209.85.128.73]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 86BCD2FC02E for ; Wed, 8 Oct 2025 15:47:05 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.73 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1759938428; cv=none; b=tJzegV4vwPP4yMvL+dD1ipiJRPya8suf4TSL5ify/409gASQisskhlFsPEDt5GephBOt/G7R1jqmQ3krBkA4ka6Yc0/x1fm7NJ/pk1BI2rsdini5ohCeS4MTmTeGU6hM89QC7lT6qHKTVMFhFn0sX41gzDg2jMDDfHdXMB228fs= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1759938428; c=relaxed/simple; bh=MtZS01j2WfludD2fnJ5sd0Wl0jM+Sxa7nGKfIhIUi1A=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=ixM7vpaqVczMmHjNP5965551CM1uJW/Qckj9xS3GA9tHJuM1c9W6s7+kg2lKm/aRFCXjpvxIX7neXUk7WLDq8C5uZVFezjZdwFmPSumY6IDkh1ntI37ePcNsQQVwTD/Qq56hwlyyg1c8hvTfxNEYcfB6Vm5hsVLgChW7CxD9myE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=BoCEnnNP; arc=none smtp.client-ip=209.85.128.73 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="BoCEnnNP" Received: by mail-wm1-f73.google.com with SMTP id 5b1f17b1804b1-46e3a049abaso81735e9.0 for ; Wed, 08 Oct 2025 08:47:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1759938424; x=1760543224; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=8ZT9tgOoTUzUgeWgcfcZEHsGVCngyDkvjIERfXOEyEU=; b=BoCEnnNPwfwBomg/Bof0bn0slaDiv86SAT4/MycP+p7Ncej6jhLuhxNhyQ6OZWii/S 6lXxPJn4KOGQdHAkrx8B0rrF6LGLw7yHKN9iRewoSdLyIveXyNX82yoU8F8yNgUqAc23 P3J0CUJswMV62Xxq21wM3ztJZhaqiHNVrVHguPpQJW1THJxBaZZgGMpmfB33McVyQztN 55Hog1QSMV2/zHMQj2t/bxoFfmA8d+M0lZg/R9hiWk/Psm7EKnfiMesLtMTR4bnMaV1T 1ZOi6BNZs/R+9FLOhm9L7IRUcQit8FztVA6eiPUU50jecwWInEAE+YFoiDyAhjqOs/kW nkqA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1759938424; x=1760543224; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=8ZT9tgOoTUzUgeWgcfcZEHsGVCngyDkvjIERfXOEyEU=; b=CQVBLO0uBpdlmQ+qpJGN+T4WlFdPfkabW+MxVSXS6jmOa+CgF0iILwt84bzeSXQ59u R1FwPvngU4IDvVs5rcKlVp8tf/uYpKVln13ue4O4+mntGEL/xNrVzRw1gWwWLuPXYsL/ cJQYNWV6YEbQLJ/B+KghlDfRhhjGBd8AdNteDWkIO4ZHjExkUkHYGr9+PFXWGjdKSeBL FMiiXE27X9LS8z9ojxUno8o/ewA99sW4gIUUf2GLizpJ5/ANAMhGveuiEl92kJHyCpMN Cd6ZSf4i6RU5C951iS93fk4J6m+EA6lzCcYqdXc+pDE9qO6ZZcoqI6Jzpog01Jl2zuxc 1Q+g== X-Gm-Message-State: AOJu0YweZvrkb5MqlsjJEQH3+5JzHJPv8IW5CU46pBIhbvoLyk++aJgW M6SJYaNCwpKsVE+5xN3GwVmhgCNasgdjgN3RTwK83aOhMv9a1rAwQbAmRJ8MF7NHwYptWfS7sA= = X-Google-Smtp-Source: AGHT+IEC3/ADIk3U2uUD8KKtjU1+4hdim3vbK4dfDLBhgiS25kKiN4BPtS5PBsG6WxrUE6u65KzHoVVT X-Received: from wmhf4.prod.google.com ([2002:a7b:cc04:0:b0:45f:28be:a3aa]) (user=ardb job=prod-delivery.src-stubby-dispatcher) by 2002:a05:600c:4ec8:b0:46d:27b7:e7e5 with SMTP id 5b1f17b1804b1-46fa9b08d79mr24052095e9.32.1759938423837; Wed, 08 Oct 2025 08:47:03 -0700 (PDT) Date: Wed, 8 Oct 2025 17:45:51 +0200 In-Reply-To: <20251008154533.3089255-23-ardb+git@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20251008154533.3089255-23-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=17484; i=ardb@kernel.org; h=from:subject; bh=GGuWmKmmCk1mbfGpeGHXwnYhEo3fZJDcJH39NGQosfM=; b=owGbwMvMwCVmkMcZplerG8N4Wi2JIeNZu1/I2aj2R5OmtBg1b2XTEIzNvWJTaJovuqZy/oQy6 SuJd2Q6SlkYxLgYZMUUWQRm/3238/REqVrnWbIwc1iZQIYwcHEKwEQaHBn+BzT+3X5SouNR2HHN Fokz3F7rjzFXM069MfnjlV/Pu6atMGD4Z7XCuNh/I+8k9Z+fmpgmsS651Sk1/0P48ZaZKYqTMru /cAIA X-Mailer: git-send-email 2.51.0.710.ga91ca5db03-goog Message-ID: <20251008154533.3089255-40-ardb+git@google.com> Subject: [PATCH v3 17/21] crypto/arm64: sm4 - Switch to 'ksimd' scoped guard API From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: linux-kernel@vger.kernel.org, linux-crypto@vger.kernel.org, herbert@gondor.apana.org.au, ebiggers@kernel.org, Ard Biesheuvel Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Ard Biesheuvel Switch to the more abstract 'scoped_ksimd()' API, which will be modified in a future patch to transparently allocate a kernel mode FP/SIMD state buffer on the stack, so that kernel mode FP/SIMD code remains preemptible in principe, but without the memory overhead that adds 528 bytes to the size of struct task_struct. Reviewed-by: Eric Biggers Signed-off-by: Ard Biesheuvel --- arch/arm64/crypto/sm4-ce-ccm-glue.c | 38 ++-- arch/arm64/crypto/sm4-ce-cipher-glue.c | 10 +- arch/arm64/crypto/sm4-ce-gcm-glue.c | 53 +++-- arch/arm64/crypto/sm4-ce-glue.c | 214 +++++++++----------- arch/arm64/crypto/sm4-neon-glue.c | 25 +-- 5 files changed, 149 insertions(+), 191 deletions(-) diff --git a/arch/arm64/crypto/sm4-ce-ccm-glue.c b/arch/arm64/crypto/sm4-ce= -ccm-glue.c index e92cbdf1aaee..332f02167a96 100644 --- a/arch/arm64/crypto/sm4-ce-ccm-glue.c +++ b/arch/arm64/crypto/sm4-ce-ccm-glue.c @@ -11,7 +11,7 @@ #include #include #include -#include +#include #include #include #include @@ -35,10 +35,9 @@ static int ccm_setkey(struct crypto_aead *tfm, const u8 = *key, if (key_len !=3D SM4_KEY_SIZE) return -EINVAL; =20 - kernel_neon_begin(); - sm4_ce_expand_key(key, ctx->rkey_enc, ctx->rkey_dec, - crypto_sm4_fk, crypto_sm4_ck); - kernel_neon_end(); + scoped_ksimd() + sm4_ce_expand_key(key, ctx->rkey_enc, ctx->rkey_dec, + crypto_sm4_fk, crypto_sm4_ck); =20 return 0; } @@ -167,28 +166,25 @@ static int ccm_crypt(struct aead_request *req, struct= skcipher_walk *walk, memcpy(ctr0, walk->iv, SM4_BLOCK_SIZE); crypto_inc(walk->iv, SM4_BLOCK_SIZE); =20 - kernel_neon_begin(); + scoped_ksimd() { + if (req->assoclen) + ccm_calculate_auth_mac(req, mac); =20 - if (req->assoclen) - ccm_calculate_auth_mac(req, mac); - - while (walk->nbytes) { - unsigned int tail =3D walk->nbytes % SM4_BLOCK_SIZE; + while (walk->nbytes) { + unsigned int tail =3D walk->nbytes % SM4_BLOCK_SIZE; =20 - if (walk->nbytes =3D=3D walk->total) - tail =3D 0; + if (walk->nbytes =3D=3D walk->total) + tail =3D 0; =20 - sm4_ce_ccm_crypt(rkey_enc, walk->dst.virt.addr, - walk->src.virt.addr, walk->iv, - walk->nbytes - tail, mac); + sm4_ce_ccm_crypt(rkey_enc, walk->dst.virt.addr, + walk->src.virt.addr, walk->iv, + walk->nbytes - tail, mac); =20 - err =3D skcipher_walk_done(walk, tail); + err =3D skcipher_walk_done(walk, tail); + } + sm4_ce_ccm_final(rkey_enc, ctr0, mac); } =20 - sm4_ce_ccm_final(rkey_enc, ctr0, mac); - - kernel_neon_end(); - return err; } =20 diff --git a/arch/arm64/crypto/sm4-ce-cipher-glue.c b/arch/arm64/crypto/sm4= -ce-cipher-glue.c index c31d76fb5a17..bceec833ef4e 100644 --- a/arch/arm64/crypto/sm4-ce-cipher-glue.c +++ b/arch/arm64/crypto/sm4-ce-cipher-glue.c @@ -32,9 +32,8 @@ static void sm4_ce_encrypt(struct crypto_tfm *tfm, u8 *ou= t, const u8 *in) if (!crypto_simd_usable()) { sm4_crypt_block(ctx->rkey_enc, out, in); } else { - kernel_neon_begin(); - sm4_ce_do_crypt(ctx->rkey_enc, out, in); - kernel_neon_end(); + scoped_ksimd() + sm4_ce_do_crypt(ctx->rkey_enc, out, in); } } =20 @@ -45,9 +44,8 @@ static void sm4_ce_decrypt(struct crypto_tfm *tfm, u8 *ou= t, const u8 *in) if (!crypto_simd_usable()) { sm4_crypt_block(ctx->rkey_dec, out, in); } else { - kernel_neon_begin(); - sm4_ce_do_crypt(ctx->rkey_dec, out, in); - kernel_neon_end(); + scoped_ksimd() + sm4_ce_do_crypt(ctx->rkey_dec, out, in); } } =20 diff --git a/arch/arm64/crypto/sm4-ce-gcm-glue.c b/arch/arm64/crypto/sm4-ce= -gcm-glue.c index 8f6fc8c33c3f..ef06f4f768a1 100644 --- a/arch/arm64/crypto/sm4-ce-gcm-glue.c +++ b/arch/arm64/crypto/sm4-ce-gcm-glue.c @@ -11,7 +11,7 @@ #include #include #include -#include +#include #include #include #include @@ -48,13 +48,11 @@ static int gcm_setkey(struct crypto_aead *tfm, const u8= *key, if (key_len !=3D SM4_KEY_SIZE) return -EINVAL; =20 - kernel_neon_begin(); - - sm4_ce_expand_key(key, ctx->key.rkey_enc, ctx->key.rkey_dec, - crypto_sm4_fk, crypto_sm4_ck); - sm4_ce_pmull_ghash_setup(ctx->key.rkey_enc, ctx->ghash_table); - - kernel_neon_end(); + scoped_ksimd() { + sm4_ce_expand_key(key, ctx->key.rkey_enc, ctx->key.rkey_dec, + crypto_sm4_fk, crypto_sm4_ck); + sm4_ce_pmull_ghash_setup(ctx->key.rkey_enc, ctx->ghash_table); + } return 0; } =20 @@ -149,31 +147,28 @@ static int gcm_crypt(struct aead_request *req, struct= skcipher_walk *walk, memcpy(iv, req->iv, GCM_IV_SIZE); put_unaligned_be32(2, iv + GCM_IV_SIZE); =20 - kernel_neon_begin(); + scoped_ksimd() { + if (req->assoclen) + gcm_calculate_auth_mac(req, ghash); =20 - if (req->assoclen) - gcm_calculate_auth_mac(req, ghash); + do { + unsigned int tail =3D walk->nbytes % SM4_BLOCK_SIZE; + const u8 *src =3D walk->src.virt.addr; + u8 *dst =3D walk->dst.virt.addr; + const u8 *l =3D NULL; =20 - do { - unsigned int tail =3D walk->nbytes % SM4_BLOCK_SIZE; - const u8 *src =3D walk->src.virt.addr; - u8 *dst =3D walk->dst.virt.addr; - const u8 *l =3D NULL; - - if (walk->nbytes =3D=3D walk->total) { - l =3D (const u8 *)&lengths; - tail =3D 0; - } - - sm4_ce_pmull_gcm_crypt(ctx->key.rkey_enc, dst, src, iv, - walk->nbytes - tail, ghash, - ctx->ghash_table, l); - - err =3D skcipher_walk_done(walk, tail); - } while (walk->nbytes); + if (walk->nbytes =3D=3D walk->total) { + l =3D (const u8 *)&lengths; + tail =3D 0; + } =20 - kernel_neon_end(); + sm4_ce_pmull_gcm_crypt(ctx->key.rkey_enc, dst, src, iv, + walk->nbytes - tail, ghash, + ctx->ghash_table, l); =20 + err =3D skcipher_walk_done(walk, tail); + } while (walk->nbytes); + } return err; } =20 diff --git a/arch/arm64/crypto/sm4-ce-glue.c b/arch/arm64/crypto/sm4-ce-glu= e.c index 7a60e7b559dc..5569cece5a0b 100644 --- a/arch/arm64/crypto/sm4-ce-glue.c +++ b/arch/arm64/crypto/sm4-ce-glue.c @@ -8,7 +8,7 @@ * Copyright (C) 2022 Tianjia Zhang */ =20 -#include +#include #include #include #include @@ -74,10 +74,9 @@ static int sm4_setkey(struct crypto_skcipher *tfm, const= u8 *key, if (key_len !=3D SM4_KEY_SIZE) return -EINVAL; =20 - kernel_neon_begin(); - sm4_ce_expand_key(key, ctx->rkey_enc, ctx->rkey_dec, - crypto_sm4_fk, crypto_sm4_ck); - kernel_neon_end(); + scoped_ksimd() + sm4_ce_expand_key(key, ctx->rkey_enc, ctx->rkey_dec, + crypto_sm4_fk, crypto_sm4_ck); return 0; } =20 @@ -94,12 +93,12 @@ static int sm4_xts_setkey(struct crypto_skcipher *tfm, = const u8 *key, if (ret) return ret; =20 - kernel_neon_begin(); - sm4_ce_expand_key(key, ctx->key1.rkey_enc, - ctx->key1.rkey_dec, crypto_sm4_fk, crypto_sm4_ck); - sm4_ce_expand_key(&key[SM4_KEY_SIZE], ctx->key2.rkey_enc, - ctx->key2.rkey_dec, crypto_sm4_fk, crypto_sm4_ck); - kernel_neon_end(); + scoped_ksimd() { + sm4_ce_expand_key(key, ctx->key1.rkey_enc, + ctx->key1.rkey_dec, crypto_sm4_fk, crypto_sm4_ck); + sm4_ce_expand_key(&key[SM4_KEY_SIZE], ctx->key2.rkey_enc, + ctx->key2.rkey_dec, crypto_sm4_fk, crypto_sm4_ck); + } =20 return 0; } @@ -117,16 +116,14 @@ static int sm4_ecb_do_crypt(struct skcipher_request *= req, const u32 *rkey) u8 *dst =3D walk.dst.virt.addr; unsigned int nblks; =20 - kernel_neon_begin(); - - nblks =3D BYTES2BLKS(nbytes); - if (nblks) { - sm4_ce_crypt(rkey, dst, src, nblks); - nbytes -=3D nblks * SM4_BLOCK_SIZE; + scoped_ksimd() { + nblks =3D BYTES2BLKS(nbytes); + if (nblks) { + sm4_ce_crypt(rkey, dst, src, nblks); + nbytes -=3D nblks * SM4_BLOCK_SIZE; + } } =20 - kernel_neon_end(); - err =3D skcipher_walk_done(&walk, nbytes); } =20 @@ -167,16 +164,14 @@ static int sm4_cbc_crypt(struct skcipher_request *req, =20 nblocks =3D nbytes / SM4_BLOCK_SIZE; if (nblocks) { - kernel_neon_begin(); - - if (encrypt) - sm4_ce_cbc_enc(ctx->rkey_enc, dst, src, - walk.iv, nblocks); - else - sm4_ce_cbc_dec(ctx->rkey_dec, dst, src, - walk.iv, nblocks); - - kernel_neon_end(); + scoped_ksimd() { + if (encrypt) + sm4_ce_cbc_enc(ctx->rkey_enc, dst, src, + walk.iv, nblocks); + else + sm4_ce_cbc_dec(ctx->rkey_dec, dst, src, + walk.iv, nblocks); + } } =20 err =3D skcipher_walk_done(&walk, nbytes % SM4_BLOCK_SIZE); @@ -249,16 +244,14 @@ static int sm4_cbc_cts_crypt(struct skcipher_request = *req, bool encrypt) if (err) return err; =20 - kernel_neon_begin(); - - if (encrypt) - sm4_ce_cbc_cts_enc(ctx->rkey_enc, walk.dst.virt.addr, - walk.src.virt.addr, walk.iv, walk.nbytes); - else - sm4_ce_cbc_cts_dec(ctx->rkey_dec, walk.dst.virt.addr, - walk.src.virt.addr, walk.iv, walk.nbytes); - - kernel_neon_end(); + scoped_ksimd() { + if (encrypt) + sm4_ce_cbc_cts_enc(ctx->rkey_enc, walk.dst.virt.addr, + walk.src.virt.addr, walk.iv, walk.nbytes); + else + sm4_ce_cbc_cts_dec(ctx->rkey_dec, walk.dst.virt.addr, + walk.src.virt.addr, walk.iv, walk.nbytes); + } =20 return skcipher_walk_done(&walk, 0); } @@ -288,28 +281,26 @@ static int sm4_ctr_crypt(struct skcipher_request *req) u8 *dst =3D walk.dst.virt.addr; unsigned int nblks; =20 - kernel_neon_begin(); - - nblks =3D BYTES2BLKS(nbytes); - if (nblks) { - sm4_ce_ctr_enc(ctx->rkey_enc, dst, src, walk.iv, nblks); - dst +=3D nblks * SM4_BLOCK_SIZE; - src +=3D nblks * SM4_BLOCK_SIZE; - nbytes -=3D nblks * SM4_BLOCK_SIZE; - } - - /* tail */ - if (walk.nbytes =3D=3D walk.total && nbytes > 0) { - u8 keystream[SM4_BLOCK_SIZE]; - - sm4_ce_crypt_block(ctx->rkey_enc, keystream, walk.iv); - crypto_inc(walk.iv, SM4_BLOCK_SIZE); - crypto_xor_cpy(dst, src, keystream, nbytes); - nbytes =3D 0; + scoped_ksimd() { + nblks =3D BYTES2BLKS(nbytes); + if (nblks) { + sm4_ce_ctr_enc(ctx->rkey_enc, dst, src, walk.iv, nblks); + dst +=3D nblks * SM4_BLOCK_SIZE; + src +=3D nblks * SM4_BLOCK_SIZE; + nbytes -=3D nblks * SM4_BLOCK_SIZE; + } + + /* tail */ + if (walk.nbytes =3D=3D walk.total && nbytes > 0) { + u8 keystream[SM4_BLOCK_SIZE]; + + sm4_ce_crypt_block(ctx->rkey_enc, keystream, walk.iv); + crypto_inc(walk.iv, SM4_BLOCK_SIZE); + crypto_xor_cpy(dst, src, keystream, nbytes); + nbytes =3D 0; + } } =20 - kernel_neon_end(); - err =3D skcipher_walk_done(&walk, nbytes); } =20 @@ -359,18 +350,16 @@ static int sm4_xts_crypt(struct skcipher_request *req= , bool encrypt) if (nbytes < walk.total) nbytes &=3D ~(SM4_BLOCK_SIZE - 1); =20 - kernel_neon_begin(); - - if (encrypt) - sm4_ce_xts_enc(ctx->key1.rkey_enc, walk.dst.virt.addr, - walk.src.virt.addr, walk.iv, nbytes, - rkey2_enc); - else - sm4_ce_xts_dec(ctx->key1.rkey_dec, walk.dst.virt.addr, - walk.src.virt.addr, walk.iv, nbytes, - rkey2_enc); - - kernel_neon_end(); + scoped_ksimd() { + if (encrypt) + sm4_ce_xts_enc(ctx->key1.rkey_enc, walk.dst.virt.addr, + walk.src.virt.addr, walk.iv, nbytes, + rkey2_enc); + else + sm4_ce_xts_dec(ctx->key1.rkey_dec, walk.dst.virt.addr, + walk.src.virt.addr, walk.iv, nbytes, + rkey2_enc); + } =20 rkey2_enc =3D NULL; =20 @@ -395,18 +384,16 @@ static int sm4_xts_crypt(struct skcipher_request *req= , bool encrypt) if (err) return err; =20 - kernel_neon_begin(); - - if (encrypt) - sm4_ce_xts_enc(ctx->key1.rkey_enc, walk.dst.virt.addr, - walk.src.virt.addr, walk.iv, walk.nbytes, - rkey2_enc); - else - sm4_ce_xts_dec(ctx->key1.rkey_dec, walk.dst.virt.addr, - walk.src.virt.addr, walk.iv, walk.nbytes, - rkey2_enc); - - kernel_neon_end(); + scoped_ksimd() { + if (encrypt) + sm4_ce_xts_enc(ctx->key1.rkey_enc, walk.dst.virt.addr, + walk.src.virt.addr, walk.iv, walk.nbytes, + rkey2_enc); + else + sm4_ce_xts_dec(ctx->key1.rkey_dec, walk.dst.virt.addr, + walk.src.virt.addr, walk.iv, walk.nbytes, + rkey2_enc); + } =20 return skcipher_walk_done(&walk, 0); } @@ -510,11 +497,9 @@ static int sm4_cbcmac_setkey(struct crypto_shash *tfm,= const u8 *key, if (key_len !=3D SM4_KEY_SIZE) return -EINVAL; =20 - kernel_neon_begin(); - sm4_ce_expand_key(key, ctx->key.rkey_enc, ctx->key.rkey_dec, - crypto_sm4_fk, crypto_sm4_ck); - kernel_neon_end(); - + scoped_ksimd() + sm4_ce_expand_key(key, ctx->key.rkey_enc, ctx->key.rkey_dec, + crypto_sm4_fk, crypto_sm4_ck); return 0; } =20 @@ -530,15 +515,13 @@ static int sm4_cmac_setkey(struct crypto_shash *tfm, = const u8 *key, =20 memset(consts, 0, SM4_BLOCK_SIZE); =20 - kernel_neon_begin(); - - sm4_ce_expand_key(key, ctx->key.rkey_enc, ctx->key.rkey_dec, - crypto_sm4_fk, crypto_sm4_ck); + scoped_ksimd() { + sm4_ce_expand_key(key, ctx->key.rkey_enc, ctx->key.rkey_dec, + crypto_sm4_fk, crypto_sm4_ck); =20 - /* encrypt the zero block */ - sm4_ce_crypt_block(ctx->key.rkey_enc, (u8 *)consts, (const u8 *)consts); - - kernel_neon_end(); + /* encrypt the zero block */ + sm4_ce_crypt_block(ctx->key.rkey_enc, (u8 *)consts, (const u8 *)consts); + } =20 /* gf(2^128) multiply zero-ciphertext with u and u^2 */ a =3D be64_to_cpu(consts[0].a); @@ -568,18 +551,16 @@ static int sm4_xcbc_setkey(struct crypto_shash *tfm, = const u8 *key, if (key_len !=3D SM4_KEY_SIZE) return -EINVAL; =20 - kernel_neon_begin(); - - sm4_ce_expand_key(key, ctx->key.rkey_enc, ctx->key.rkey_dec, - crypto_sm4_fk, crypto_sm4_ck); + scoped_ksimd() { + sm4_ce_expand_key(key, ctx->key.rkey_enc, ctx->key.rkey_dec, + crypto_sm4_fk, crypto_sm4_ck); =20 - sm4_ce_crypt_block(ctx->key.rkey_enc, key2, ks[0]); - sm4_ce_crypt(ctx->key.rkey_enc, ctx->consts, ks[1], 2); + sm4_ce_crypt_block(ctx->key.rkey_enc, key2, ks[0]); + sm4_ce_crypt(ctx->key.rkey_enc, ctx->consts, ks[1], 2); =20 - sm4_ce_expand_key(key2, ctx->key.rkey_enc, ctx->key.rkey_dec, - crypto_sm4_fk, crypto_sm4_ck); - - kernel_neon_end(); + sm4_ce_expand_key(key2, ctx->key.rkey_enc, ctx->key.rkey_dec, + crypto_sm4_fk, crypto_sm4_ck); + } =20 return 0; } @@ -600,10 +581,9 @@ static int sm4_mac_update(struct shash_desc *desc, con= st u8 *p, unsigned int nblocks =3D len / SM4_BLOCK_SIZE; =20 len %=3D SM4_BLOCK_SIZE; - kernel_neon_begin(); - sm4_ce_mac_update(tctx->key.rkey_enc, ctx->digest, p, - nblocks, false, true); - kernel_neon_end(); + scoped_ksimd() + sm4_ce_mac_update(tctx->key.rkey_enc, ctx->digest, p, + nblocks, false, true); return len; } =20 @@ -619,10 +599,9 @@ static int sm4_cmac_finup(struct shash_desc *desc, con= st u8 *src, ctx->digest[len] ^=3D 0x80; consts +=3D SM4_BLOCK_SIZE; } - kernel_neon_begin(); - sm4_ce_mac_update(tctx->key.rkey_enc, ctx->digest, consts, 1, - false, true); - kernel_neon_end(); + scoped_ksimd() + sm4_ce_mac_update(tctx->key.rkey_enc, ctx->digest, consts, 1, + false, true); memcpy(out, ctx->digest, SM4_BLOCK_SIZE); return 0; } @@ -635,10 +614,9 @@ static int sm4_cbcmac_finup(struct shash_desc *desc, c= onst u8 *src, =20 if (len) { crypto_xor(ctx->digest, src, len); - kernel_neon_begin(); - sm4_ce_crypt_block(tctx->key.rkey_enc, ctx->digest, - ctx->digest); - kernel_neon_end(); + scoped_ksimd() + sm4_ce_crypt_block(tctx->key.rkey_enc, ctx->digest, + ctx->digest); } memcpy(out, ctx->digest, SM4_BLOCK_SIZE); return 0; diff --git a/arch/arm64/crypto/sm4-neon-glue.c b/arch/arm64/crypto/sm4-neon= -glue.c index e3500aca2d18..e944c2a2efb0 100644 --- a/arch/arm64/crypto/sm4-neon-glue.c +++ b/arch/arm64/crypto/sm4-neon-glue.c @@ -48,11 +48,8 @@ static int sm4_ecb_do_crypt(struct skcipher_request *req= , const u32 *rkey) =20 nblocks =3D nbytes / SM4_BLOCK_SIZE; if (nblocks) { - kernel_neon_begin(); - - sm4_neon_crypt(rkey, dst, src, nblocks); - - kernel_neon_end(); + scoped_ksimd() + sm4_neon_crypt(rkey, dst, src, nblocks); } =20 err =3D skcipher_walk_done(&walk, nbytes % SM4_BLOCK_SIZE); @@ -126,12 +123,9 @@ static int sm4_cbc_decrypt(struct skcipher_request *re= q) =20 nblocks =3D nbytes / SM4_BLOCK_SIZE; if (nblocks) { - kernel_neon_begin(); - - sm4_neon_cbc_dec(ctx->rkey_dec, dst, src, - walk.iv, nblocks); - - kernel_neon_end(); + scoped_ksimd() + sm4_neon_cbc_dec(ctx->rkey_dec, dst, src, + walk.iv, nblocks); } =20 err =3D skcipher_walk_done(&walk, nbytes % SM4_BLOCK_SIZE); @@ -157,12 +151,9 @@ static int sm4_ctr_crypt(struct skcipher_request *req) =20 nblocks =3D nbytes / SM4_BLOCK_SIZE; if (nblocks) { - kernel_neon_begin(); - - sm4_neon_ctr_crypt(ctx->rkey_enc, dst, src, - walk.iv, nblocks); - - kernel_neon_end(); + scoped_ksimd() + sm4_neon_ctr_crypt(ctx->rkey_enc, dst, src, + walk.iv, nblocks); =20 dst +=3D nblocks * SM4_BLOCK_SIZE; src +=3D nblocks * SM4_BLOCK_SIZE; --=20 2.51.0.710.ga91ca5db03-goog From nobody Sat Feb 7 18:46:42 2026 Received: from mail-wm1-f74.google.com (mail-wm1-f74.google.com [209.85.128.74]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B66E52FCBED for ; Wed, 8 Oct 2025 15:47:06 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.74 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1759938428; cv=none; b=VCYsK26nQ3KlSHWYJ+BUmaWZEfCxb22Wqesd99WVdNBrx4+ek3SPm81E9J/4oZt/mSHxuDePkER8Kx/65jzzzAYWRIKTazohebPQ7lZYn++c+E36fhLVWxZx8ZCZ+jrgPXyMzU/VlGc71qVdulbfr/MkfnTg7O3G0YyyY8i1UeE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1759938428; c=relaxed/simple; bh=e1uCNgvDUPL6Q8IGrP4fE1tjT9jxmUxE+KdCoB9JZM4=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=ZZB2ALfb1f3zrFSoGgPZu22NklCWQcM4UpTTXY1fvT3HbSViwlNoU0oXBevbmFsk4gHGJ8xdVQWQMVZor9A30er8mM6xnY1NWb5ENCFPtLuvJGTCqpb4/dTSjrTwct+4aSJQ5BA2CZYSItJpdS4DskcilFiW3NVycABVnOA4BOQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=jkr9eYNL; arc=none smtp.client-ip=209.85.128.74 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="jkr9eYNL" Received: by mail-wm1-f74.google.com with SMTP id 5b1f17b1804b1-46e36686ca1so71992295e9.2 for ; Wed, 08 Oct 2025 08:47:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1759938425; x=1760543225; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=pCqfiDMNXwNp3eFCLlUtfPhbgD3LgIedHtp2uSYpz9o=; b=jkr9eYNLbk7wVyDI6UHYi+5oKhLyYsw+OAeiiWWIFWyV5yvpSWfSZBSDHo+RKdGlP+ /NyCCmyoxiZt/jGygIUbRKurDRf2frvj5PfpuhjgnMlTmcOE18dPlVwhP8e5LDC1YRhS xfoCcUbbOchJIBv7Qkcv3PbllSQNnhdUvD/qBbAnKv1T9pZKvyNuCL2LxFf3+ZLGpWNH BwPmlJg74Rvp+SYSaoL9VyyHBCi0F4lIf8wdHTLPtShc1Wy2+QTepdbmc8ExbYFM5AyU t5wgrgOQhvDOO+pPNkonT55FBjrvuWVIc2fibVTRZfQCVY3pvbVlmlTIKUGfeLJ1pfkx 8mBA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1759938425; x=1760543225; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=pCqfiDMNXwNp3eFCLlUtfPhbgD3LgIedHtp2uSYpz9o=; b=At5ilA6Z+o2jDeaT+O14FEpojR1BHctZVoUzsX8esR4Qd0iZnuRC6RqTdfXDXeo9+i zhaTQNeMNJo3eSkyI3seJnndDm/YBBY1mztSwhn2OAA4uGtJxTczI0Q6Pu03kBLChtjl tOr8MkyjSKCmLwLv68OAQl3wPhMfrNKhCjBgFoFwvPsypAkqJHwexjXDkqM/VjGAuw9c bqDAdJenYp3cVhO6wFu3QDxPK9xiu/wTJ0OcGgCQFHRJzY13cFDzOe1seaj/YnYJZ4br OaxffCYdaNVZiNpUF8pQFEALeqIiTgbtoeequdz1zLeeQdbKZEwixjY5i0/gQEATCTia a/gg== X-Gm-Message-State: AOJu0YxXJbUewxrSY7rJqKslcGRQUJtT+TvKI07Ypt8sb124FZhGEEkZ Hd9VzSGWy/+R5t/EGx4zT1PU6HWHdIhWum9Y6jdiYA4lAEDmpao4fQdmIjY9dRZR39FRDIN5NQ= = X-Google-Smtp-Source: AGHT+IEstc5WbaKR984lcE73yhlUqY4mnQpun5Em/Dltp0yYfhzejBDQJd+oDTcw6HoRIbI7W/BkGtQg X-Received: from wmjf4.prod.google.com ([2002:a7b:cd04:0:b0:46e:38c2:1a34]) (user=ardb job=prod-delivery.src-stubby-dispatcher) by 2002:a05:600c:468e:b0:46e:4b79:551 with SMTP id 5b1f17b1804b1-46fa9b092femr34512895e9.31.1759938425183; Wed, 08 Oct 2025 08:47:05 -0700 (PDT) Date: Wed, 8 Oct 2025 17:45:52 +0200 In-Reply-To: <20251008154533.3089255-23-ardb+git@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20251008154533.3089255-23-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=2401; i=ardb@kernel.org; h=from:subject; bh=5nhtgxb3dS2zVbgouam5e+M4Gmu5K56lN9xZqYXBI3s=; b=owGbwMvMwCVmkMcZplerG8N4Wi2JIeNZe6DOTtbCqP/MvEdnNO/+dDayK/WX4sFtL+4vEJj/I +l81yPtjlIWBjEuBlkxRRaB2X/f7Tw9UarWeZYszBxWJpAhDFycAjCRyBUM/6z/Z+Z7rrJVsj3o NKFtdgfTrePKpyWn38q/ubvk6uoTq6UZGebIWslEbD0dYrpZ+BV3WHF94Ya0X7/LTFfJp3Rny69 v4AMA X-Mailer: git-send-email 2.51.0.710.ga91ca5db03-goog Message-ID: <20251008154533.3089255-41-ardb+git@google.com> Subject: [PATCH v3 18/21] arm64/xorblocks: Switch to 'ksimd' scoped guard API From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: linux-kernel@vger.kernel.org, linux-crypto@vger.kernel.org, herbert@gondor.apana.org.au, ebiggers@kernel.org, Ard Biesheuvel Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Ard Biesheuvel Switch to the more abstract 'scoped_ksimd()' API, which will be modified in a future patch to transparently allocate a kernel mode FP/SIMD state buffer on the stack, so that kernel mode FP/SIMD code remains preemptible in principe, but without the memory overhead that adds 528 bytes to the size of struct task_struct. Reviewed-by: Eric Biggers Signed-off-by: Ard Biesheuvel --- arch/arm64/include/asm/xor.h | 22 ++++++++------------ 1 file changed, 9 insertions(+), 13 deletions(-) diff --git a/arch/arm64/include/asm/xor.h b/arch/arm64/include/asm/xor.h index befcd8a7abc9..c38e3d017a79 100644 --- a/arch/arm64/include/asm/xor.h +++ b/arch/arm64/include/asm/xor.h @@ -9,7 +9,7 @@ #include #include #include -#include +#include =20 #ifdef CONFIG_KERNEL_MODE_NEON =20 @@ -19,9 +19,8 @@ static void xor_neon_2(unsigned long bytes, unsigned long * __restrict p1, const unsigned long * __restrict p2) { - kernel_neon_begin(); - xor_block_inner_neon.do_2(bytes, p1, p2); - kernel_neon_end(); + scoped_ksimd() + xor_block_inner_neon.do_2(bytes, p1, p2); } =20 static void @@ -29,9 +28,8 @@ xor_neon_3(unsigned long bytes, unsigned long * __restric= t p1, const unsigned long * __restrict p2, const unsigned long * __restrict p3) { - kernel_neon_begin(); - xor_block_inner_neon.do_3(bytes, p1, p2, p3); - kernel_neon_end(); + scoped_ksimd() + xor_block_inner_neon.do_3(bytes, p1, p2, p3); } =20 static void @@ -40,9 +38,8 @@ xor_neon_4(unsigned long bytes, unsigned long * __restric= t p1, const unsigned long * __restrict p3, const unsigned long * __restrict p4) { - kernel_neon_begin(); - xor_block_inner_neon.do_4(bytes, p1, p2, p3, p4); - kernel_neon_end(); + scoped_ksimd() + xor_block_inner_neon.do_4(bytes, p1, p2, p3, p4); } =20 static void @@ -52,9 +49,8 @@ xor_neon_5(unsigned long bytes, unsigned long * __restric= t p1, const unsigned long * __restrict p4, const unsigned long * __restrict p5) { - kernel_neon_begin(); - xor_block_inner_neon.do_5(bytes, p1, p2, p3, p4, p5); - kernel_neon_end(); + scoped_ksimd() + xor_block_inner_neon.do_5(bytes, p1, p2, p3, p4, p5); } =20 static struct xor_block_template xor_block_arm64 =3D { --=20 2.51.0.710.ga91ca5db03-goog From nobody Sat Feb 7 18:46:42 2026 Received: from mail-wm1-f74.google.com (mail-wm1-f74.google.com [209.85.128.74]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5D8792FD1BC for ; Wed, 8 Oct 2025 15:47:08 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.74 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1759938430; cv=none; b=Vh8NRsuGPmNftg+j1GyYb79TM+fl3I0sXBbNwe7MQfBbjtnfze/fdMA8qEflIdO9+m6Yl8AQtXihQHx23XMuxeLuhesDmMFeLoyE80ka7BNHgc/7II+wM+n5HMBShSW/XPqMyKwORglrOnFKnMEwQpmbTsAtN4ET7NqvGDUjf2U= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1759938430; c=relaxed/simple; bh=T2a3v4NeDZkwfqj85MbBcAieCeChFKaNVae0DBFB4FA=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=XIMlBN7rOEy+Yus0KvcfG/n9l/xVXL92JzXvInDT47tm8ExqKuhTTzTGPTzTGoC+OgPTsGf96iNPrPfpCwYy1X4jt/jLq6GdrZc1VymFTrqSak0bgefhuV6JtXTsxXkwD4aZIcKJrBQOQYw8XPOP2vGsfU8DxJpNgeTq1JP/qpw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=BySEllG6; arc=none smtp.client-ip=209.85.128.74 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="BySEllG6" Received: by mail-wm1-f74.google.com with SMTP id 5b1f17b1804b1-46e2c11b94cso43144435e9.3 for ; Wed, 08 Oct 2025 08:47:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1759938426; x=1760543226; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=XyDqe41TOpzGrced2VguyWuuhzoYjsOPlRjpRYq84qU=; b=BySEllG6wojB+KZsMULQMHRXoFN4DXyJU68ZCAFMYCafvtg2Wj6W89tuOfZep6bDhT 9HwCRChfqn/SCZRqPRbQ3q9/trngZw6xtk2NVRtPnhfCKUHrQvJTf8gWmF1mRD+yKW75 GrQUaZiCremDtp3ZyKD+ZDTtM66w8JumHL6strMkxp7TlhgDY4DjG5QeI4l1b0ufY5fn f1A0kl7t9wbu2zUoyt1BoXZNAlZ6WS0pHOUZP8u93cccxACo3uQHcre17YTqmDoOuCpY JiJN1ShCk021hRcBGQl2+w08vPjxeRyZrORUvkQyLdqlQT0JOeEAwQouYs8vKKOZO0NX zjWw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1759938426; x=1760543226; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=XyDqe41TOpzGrced2VguyWuuhzoYjsOPlRjpRYq84qU=; b=PTMdhlzGtOIJa1ewk5DkB/eVgczL4UQb7wkFqQDIlOIgm1PUA9heRnagDDouJjlCbe dG34zO1im1gfFTjNuFjhnor7sFghVsM9Q801jmrrthA9yux3iqdTvsx3dckf7EJvtIXC lvhR1Vy1kS7N1I9G2C3DfcnXh9h9gCz5zVBb1P2yyocoaKxRm7Ori64qGxewec0u6vaD lwPBFASFbae0rLWyiF5WW9Ga0liWQ85EGtDvEgQG3J7EnC+OoVDLY8hEvHpWkpDxQYyr IbG0m+I4G0KHArKRm4eBA/T7hvULKM7v5NL/U1+XX93F5blaLYasMaxBOZIU+9GL+l5r /DVw== X-Gm-Message-State: AOJu0YyvWBvxa7aGlzqYwxlGDHj5500PW7SJtsHf+6rLfI2zI8S8uNFa psObqKj/pAm0hPbH5AWjH8pph0mun6ihweWsUiTqhsiU9M1OMKcN7eBtmO0/v6MOyBJXdxSyDQ= = X-Google-Smtp-Source: AGHT+IGd+xyyENe4mDMwHsq1XM4LCrOZp0Y90ZGldhJ1LzDJeD7pXJXrRWavBZZTJfyTw+krJ8EAav3c X-Received: from wmjs21.prod.google.com ([2002:a7b:c395:0:b0:46f:aafc:e6d4]) (user=ardb job=prod-delivery.src-stubby-dispatcher) by 2002:a05:600c:c4a8:b0:46e:4921:9443 with SMTP id 5b1f17b1804b1-46fa9b1b277mr26777065e9.37.1759938426503; Wed, 08 Oct 2025 08:47:06 -0700 (PDT) Date: Wed, 8 Oct 2025 17:45:53 +0200 In-Reply-To: <20251008154533.3089255-23-ardb+git@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20251008154533.3089255-23-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=1762; i=ardb@kernel.org; h=from:subject; bh=LscVi7axqbts3F3wdJxDdLq4C5QXUNo3h535UnzCneo=; b=owGbwMvMwCVmkMcZplerG8N4Wi2JIeNZe/By5YdKnzyPLFhZtfvlkqQJPIGL7q0zd1kx5X1Om 6tMZcC8jlIWBjEuBlkxRRaB2X/f7Tw9UarWeZYszBxWJpAhDFycAjCRGUEM/0Mi6397J6ztaDx5 4LfWJuaoVCufzUVPp960Sz/e4/wj6h4jw8XefzPlvzT+lC6UnnrnMc+9e7+mPeQ7s8nt6e9pZ/O Zr/MAAA== X-Mailer: git-send-email 2.51.0.710.ga91ca5db03-goog Message-ID: <20251008154533.3089255-42-ardb+git@google.com> Subject: [PATCH v3 19/21] net/mlx5: Switch to more abstract scoped ksimd guard API on arm64 From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: linux-kernel@vger.kernel.org, linux-crypto@vger.kernel.org, herbert@gondor.apana.org.au, ebiggers@kernel.org, Ard Biesheuvel Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Ard Biesheuvel Instead of calling kernel_neon_begin/end directly, switch to the scoped guard API which encapsulates those calls. This is needed because the prototypes of those APIs are going to be modified and will require a kernel mode FP/SIMD buffer to be provided, which the scoped guard API will do transparently. Signed-off-by: Ard Biesheuvel --- drivers/net/ethernet/mellanox/mlx5/core/wc.c | 19 ++++++++++--------- 1 file changed, 10 insertions(+), 9 deletions(-) diff --git a/drivers/net/ethernet/mellanox/mlx5/core/wc.c b/drivers/net/eth= ernet/mellanox/mlx5/core/wc.c index c281153bd411..815a7c97d6b0 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/wc.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/wc.c @@ -9,6 +9,7 @@ =20 #if IS_ENABLED(CONFIG_KERNEL_MODE_NEON) && IS_ENABLED(CONFIG_ARM64) #include +#include #endif =20 #define TEST_WC_NUM_WQES 255 @@ -264,15 +265,15 @@ static void mlx5_iowrite64_copy(struct mlx5_wc_sq *sq= , __be32 mmio_wqe[16], { #if IS_ENABLED(CONFIG_KERNEL_MODE_NEON) && IS_ENABLED(CONFIG_ARM64) if (cpu_has_neon()) { - kernel_neon_begin(); - asm volatile - (".arch_extension simd;\n\t" - "ld1 {v0.16b, v1.16b, v2.16b, v3.16b}, [%0]\n\t" - "st1 {v0.16b, v1.16b, v2.16b, v3.16b}, [%1]" - : - : "r"(mmio_wqe), "r"(sq->bfreg.map + offset) - : "memory", "v0", "v1", "v2", "v3"); - kernel_neon_end(); + scoped_ksimd() { + asm volatile( + ".arch_extension simd\n\t" + "ld1 {v0.16b, v1.16b, v2.16b, v3.16b}, [%0]\n\t" + "st1 {v0.16b, v1.16b, v2.16b, v3.16b}, [%1]" + : + : "r"(mmio_wqe), "r"(sq->bfreg.map + offset) + : "memory", "v0", "v1", "v2", "v3"); + } return; } #endif --=20 2.51.0.710.ga91ca5db03-goog From nobody Sat Feb 7 18:46:42 2026 Received: from mail-wr1-f74.google.com (mail-wr1-f74.google.com [209.85.221.74]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4EA642FD7AA for ; Wed, 8 Oct 2025 15:47:09 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.221.74 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1759938431; cv=none; b=IXjJ1bcZuGim86GnlBUOYMPKNM+GIzo4qkTubGmXS6a6O9BjDqDPDrg4c3Io1X2NVB80ts+ij+OfQBoqBzPwpihcGHUW6xVGEeKW4Qf8sKfMgabgCGIkDXp164L+HSoZlh/1maqlVfZX8prwo4wTrJT1iFaWSNT5yagODmCKtrM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1759938431; c=relaxed/simple; bh=5G4nFD2pp89m/e0LNcJVsGUkoysspkzidmC0Tj5QN0U=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=j6X84I1nsl6T697ANWiKZ095YdMKO84v37f3zXMTTAq1sOdk+2l6uZaRV48DQ5z8BKkBjopguKHeDct8QsaT61iHfjG1iwgNZqLbKgxf22vcssTlKxUJ+ZB3bWyEZ4yuSu+sC5Av3SItqpbh/K9tvG/tYl3pZmnkYfcjbdQby14= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=aEWOMFG6; arc=none smtp.client-ip=209.85.221.74 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="aEWOMFG6" Received: by mail-wr1-f74.google.com with SMTP id ffacd0b85a97d-3efa77de998so62516f8f.0 for ; Wed, 08 Oct 2025 08:47:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1759938428; x=1760543228; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=4d430to7F+bJwYs62ijVveUJQwYq/Bl3TbNr2srG04A=; b=aEWOMFG6R9p/ynfLbkMBJn96ejp7MPOkQWaetVSliuln2Jhgi9SRP9MbIagqrBYdlh oSzESMOYfsbRkIoNvsrAkD0hgTdyRoLYDtF5pK22ew7yIHIs3za3pEFwGM8CjF8Wjyd1 3hvBvLN86g0OvHGFYRHaOnPl2WuXSExEwdVKdJftToYSNcKqc7fH5SiiW6SVF64KginQ u6J5xnEZpG/sB9B5iZvXBjh+mRpG7CvA7X01ppZqcawDox0px6CvJpRZi6j6GnCmP6nx QHwFD7D/5mmzc2+emUcYEsAAJyaJr4i+j4ujXxT2leoJdH/V8hRPAVycaNEm4IZSC69Y ObXg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1759938428; x=1760543228; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=4d430to7F+bJwYs62ijVveUJQwYq/Bl3TbNr2srG04A=; b=vn1+esGLbSHxuJ6qYltw6OS+FcusJuk1pMX47Xwxf08B6CQEnfAIuBAlTxu1gjwlJ4 RHccbQWnNRuoOgTKQJlbn/GNtjMt45SMOUwccY/NZs5hR6Avedc09PcvqWqhPRIeipLI gZNGjcod4/6Rz+IyvfqJpHYbtqiv+8bg8HquxIhlcYSgmqFGaObpMA/ox0X4mbDHhxIN BQWsC1NivNE0lD554/kOmf19pMYFqeuvHscq+sZ0aPHHuYfArxV3dsofBjn5Ksa0rRYJ GOatsfxd7fF4CKtDBtMAbW5OfYHeVQyORhkJIH+AjCgcOY/Zbqf+uCYvSvKC6n4gxeKP uXiQ== X-Gm-Message-State: AOJu0Yw3JqC6DNzKiAX9r7Jd1RAtYbkZ0yzqZzz4t9ulbt8p6VIQYEG4 R7eUMwraUuiT/fGax3yU3yqRZW+FyDHmHiOe12lB99nZnPX3IkpIi5+P5j1PzaerC0/PLUjeQA= = X-Google-Smtp-Source: AGHT+IFxuPNcvfnI61ScnGz/kEaKdHt8OBEaJHISLPH795AUUxCAL4+HW8dxd6WipTwVX9VIvhQ1077r X-Received: from wmlm12.prod.google.com ([2002:a7b:ca4c:0:b0:45f:29fc:83d]) (user=ardb job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6000:2389:b0:3fd:eb15:77a with SMTP id ffacd0b85a97d-42666ac2da4mr2403184f8f.6.1759938427683; Wed, 08 Oct 2025 08:47:07 -0700 (PDT) Date: Wed, 8 Oct 2025 17:45:54 +0200 In-Reply-To: <20251008154533.3089255-23-ardb+git@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20251008154533.3089255-23-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=1631; i=ardb@kernel.org; h=from:subject; bh=jCkHQXSdf8VNipopESRILGcuCgPWNMlfZdyre0v/uHk=; b=owGbwMvMwCVmkMcZplerG8N4Wi2JIeNZe1iU7eedkh0ZXVG1KaJNn85s2LNxVn6X6HFXzp2nI 6vrlfg6SlkYxLgYZMUUWQRm/3238/REqVrnWbIwc1iZQIYwcHEKwERm7mVkWPDjweI7tbdrJQPY eIJjdzxemCDdfydySVTtoncHsld9F2JkOOuxUuhE0XeuS1f/dqy3qu09MXn3XCM9na+rHyR9evz pOzcA X-Mailer: git-send-email 2.51.0.710.ga91ca5db03-goog Message-ID: <20251008154533.3089255-43-ardb+git@google.com> Subject: [PATCH v3 20/21] arm64/fpu: Enforce task-context only for generic kernel mode FPU From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: linux-kernel@vger.kernel.org, linux-crypto@vger.kernel.org, herbert@gondor.apana.org.au, ebiggers@kernel.org, Ard Biesheuvel Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Ard Biesheuvel The generic kernel mode FPU API, which is used by the AMDGPU driver to perform floating point calculations, is modeled after the most restrictive architecture that supports it. This means it doesn't support preemption, and can only be used from task context. The arm64 implementation is a bit more flexible, but supporting that in the generic API complicates matters slightly, and for no good reason, given that the only user does not need it. So enforce that kernel_fpu_begin() can only be called from task context, and [redundantly] disable preemption. This removes the need for users of this API to provide a kernel mode FP/SIMD state after a future patch that makes that compulsory for preemptible task context. Signed-off-by: Ard Biesheuvel --- arch/arm64/include/asm/fpu.h | 16 ++++++++++++++-- 1 file changed, 14 insertions(+), 2 deletions(-) diff --git a/arch/arm64/include/asm/fpu.h b/arch/arm64/include/asm/fpu.h index 2ae50bdce59b..52a1c513bdf3 100644 --- a/arch/arm64/include/asm/fpu.h +++ b/arch/arm64/include/asm/fpu.h @@ -6,10 +6,22 @@ #ifndef __ASM_FPU_H #define __ASM_FPU_H =20 +#include #include =20 #define kernel_fpu_available() cpu_has_neon() -#define kernel_fpu_begin() kernel_neon_begin() -#define kernel_fpu_end() kernel_neon_end() + +static inline void kernel_fpu_begin(void) +{ + BUG_ON(!in_task()); + preempt_disable(); + kernel_neon_begin(); +} + +static inline void kernel_fpu_end(void) +{ + kernel_neon_end(); + preempt_disable(); +} =20 #endif /* ! __ASM_FPU_H */ --=20 2.51.0.710.ga91ca5db03-goog From nobody Sat Feb 7 18:46:42 2026 Received: from mail-wr1-f73.google.com (mail-wr1-f73.google.com [209.85.221.73]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D167E2BD034 for ; Wed, 8 Oct 2025 15:47:10 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.221.73 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1759938432; cv=none; b=sgaQE6vO5wE2aFr0nSDX6YOBN0uWZR1gILCWd2E4eavAImMokPr141E1NlGajl/Z/XAGvQ4SyZ1tgDOX0HsR7AcWKqFhp5w7VlTNFPHGJUFoiLE651nC4MyQ1ycwUoSWPRZ4mn+uzEB/v2YiB0J1B7dEzvFl8bwsswz9d8HjKbk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1759938432; c=relaxed/simple; bh=dcsoTMvU6BQfqDU8Ebpep4KV5tTYSoNmzT6aKXdDKSo=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=bZnunxqWEU/jgZS1psGASlglgJURcWwBscZY8rJrMLvGmgjhO3+Yld5KN/7KV5gnf7C01JkVpJd2JOJRhoDDM7pQBXyb0gj8R/jGejncoPZ4MZTiLHJd/wMGNZsE7OtAD8ZvNb/KAXkEHNoJisfI27DIFwvW0MM2LgTH4eZ9iVg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=3rrIPHws; arc=none smtp.client-ip=209.85.221.73 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="3rrIPHws" Received: by mail-wr1-f73.google.com with SMTP id ffacd0b85a97d-3ee13baf21dso67702f8f.0 for ; Wed, 08 Oct 2025 08:47:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1759938429; x=1760543229; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=P5EC4bnWWk+Vzr+mFCriKnKa8JCHW95jn5qMkKvnHLI=; b=3rrIPHwsNJ4BUIKdC7qAiAwC+0vZ2xAb9dI8iG3MOV5mek8CmAzSYHB1J76PTeC3hS ih2zDJM+cX29YJTMWTQqC8cvEapCgFvQsBmE/u/L5lbdop4fIKX+nPmMRkjR2cOBUUi0 bbHJQstoPTd+Qr/lGN3JwjR/jaj7F+yYgAXXga56Z21oq+BBkEmAHAxwYnPvpxtzJnnK c2U+2NEoaUZj6wCogyLODuCjv7p9SBP7qmeTlfk6zucjOzlG2gOsVQybmEKa087SgM64 sSRvd/wAVizP/5PETwMZMUgUWe+LXKYjYlA7m6Fs6FNKoefjaInkVCJkql2NfgkY0GZv WvEg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1759938429; x=1760543229; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=P5EC4bnWWk+Vzr+mFCriKnKa8JCHW95jn5qMkKvnHLI=; b=m0Dct6d64xRHeMvZ+fw3AF8Ia6N5UwE9J7vLTirOGUUysg5UNqibsUYXfPRzJA+HU/ 062NfRAQo4jk+KzTRhGK/DnEaJ6hURPDvUP4WVuxZwYbK+3wnOE7k/jBGVsWMkKCWLpO osDvIaf24jJPO6Tdh/G0WxmyWooaVNuCFTSWA+31GnJZTHY+cF3rr81sDanY3USa2fhu DxJq1yIRXzAoV1Eub6rN1REXwZ2pUVZtSX3q6+c04t/iNXAQNs8hvE9+8+sLSgFOcrXA /sFNYkd4gZdvMFw4foYr4wOYKtlppq5zQAawMVfztLbawxsrjy5SAOlwOIxHFpokTBq4 cBMQ== X-Gm-Message-State: AOJu0YzMCMcqZLZjsaz0IFghKX6jo5k5kGjJSRQzVVO6dz6cq8yfNfTF 6zS03hsvFFtmJA6cjbA0+rwhihi9yLT2wICrj7x6ZnpOAYxnKXhwwTQm/tyqlvqMLHYkBJX1UQ= = X-Google-Smtp-Source: AGHT+IFKBOFngZ4tgCIe5stSYpbVjOkdovhSMBx+KSFMPmr5szglET/KMW92mtXFPvgDpIR9UTC1qiLy X-Received: from wrpb3.prod.google.com ([2002:adf:f243:0:b0:408:941a:35ac]) (user=ardb job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6000:186c:b0:3ec:c50c:715b with SMTP id ffacd0b85a97d-4266726d746mr2480961f8f.19.1759938428971; Wed, 08 Oct 2025 08:47:08 -0700 (PDT) Date: Wed, 8 Oct 2025 17:45:55 +0200 In-Reply-To: <20251008154533.3089255-23-ardb+git@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20251008154533.3089255-23-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=8211; i=ardb@kernel.org; h=from:subject; bh=qFYb9c8NrpZLi3t0kBZMhSZQnTeMa7rAW/va1fm5iKE=; b=owGbwMvMwCVmkMcZplerG8N4Wi2JIeNZe6RrMdOP7WWaXEZtnOWrti149D/yfIGH75Sc8K5pV 7iPvpHuKGVhEONikBVTZBGY/ffdztMTpWqdZ8nCzGFlAhnCwMUpABMJiWP4X7jkwKn7H/dZuW4x ODbr7eZbykva9wkk/ljVKvx577T7C/gY/mmdXv3TQeCF1AfldQYdfqv08sQkbmlNt3dnzp139Kh lNhsA X-Mailer: git-send-email 2.51.0.710.ga91ca5db03-goog Message-ID: <20251008154533.3089255-44-ardb+git@google.com> Subject: [PATCH v3 21/21] arm64/fpsimd: Allocate kernel mode FP/SIMD buffers on the stack From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: linux-kernel@vger.kernel.org, linux-crypto@vger.kernel.org, herbert@gondor.apana.org.au, ebiggers@kernel.org, Ard Biesheuvel Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Ard Biesheuvel Commit aefbab8e77eb16b5 ("arm64: fpsimd: Preserve/restore kernel mode NEON at context switch") added a 'kernel_fpsimd_state' field to struct thread_struct, which is the arch-specific portion of struct task_struct, and is allocated for each task in the system. The size of this field is 528 bytes, resulting in non-negligible bloat of task_struct, and the resulting memory overhead may impact performance on systems with many processes. This allocation is only used if the task is scheduled out or interrupted by a softirq while using the FP/SIMD unit in kernel mode, and so it is possible to transparently allocate this buffer on the caller's stack instead. So tweak the 'ksimd' scoped guard implementation so that a stack buffer is allocated and passed to both kernel_neon_begin() and kernel_neon_end(), and either record it in the task struct, or use it directly to preserve the task mode kernel FP/SIMD when running in softirq context. Passing the address to both functions, and checking the addresses for consistency ensures that callers of the updated bare begin/end API use it in a manner that is consistent with the new context switch semantics. Signed-off-by: Ard Biesheuvel --- arch/arm64/include/asm/fpu.h | 4 +- arch/arm64/include/asm/neon.h | 4 +- arch/arm64/include/asm/processor.h | 2 +- arch/arm64/include/asm/simd.h | 7 +++- arch/arm64/kernel/fpsimd.c | 44 +++++++++++++------- 5 files changed, 40 insertions(+), 21 deletions(-) diff --git a/arch/arm64/include/asm/fpu.h b/arch/arm64/include/asm/fpu.h index 52a1c513bdf3..e252c518de05 100644 --- a/arch/arm64/include/asm/fpu.h +++ b/arch/arm64/include/asm/fpu.h @@ -15,12 +15,12 @@ static inline void kernel_fpu_begin(void) { BUG_ON(!in_task()); preempt_disable(); - kernel_neon_begin(); + kernel_neon_begin(NULL); } =20 static inline void kernel_fpu_end(void) { - kernel_neon_end(); + kernel_neon_end(NULL); preempt_disable(); } =20 diff --git a/arch/arm64/include/asm/neon.h b/arch/arm64/include/asm/neon.h index d4b1d172a79b..acebee4605b5 100644 --- a/arch/arm64/include/asm/neon.h +++ b/arch/arm64/include/asm/neon.h @@ -13,7 +13,7 @@ =20 #define cpu_has_neon() system_supports_fpsimd() =20 -void kernel_neon_begin(void); -void kernel_neon_end(void); +void kernel_neon_begin(struct user_fpsimd_state *); +void kernel_neon_end(struct user_fpsimd_state *); =20 #endif /* ! __ASM_NEON_H */ diff --git a/arch/arm64/include/asm/processor.h b/arch/arm64/include/asm/pr= ocessor.h index 61d62bfd5a7b..226e635c53d9 100644 --- a/arch/arm64/include/asm/processor.h +++ b/arch/arm64/include/asm/processor.h @@ -172,7 +172,7 @@ struct thread_struct { unsigned long fault_code; /* ESR_EL1 value */ struct debug_info debug; /* debugging */ =20 - struct user_fpsimd_state kernel_fpsimd_state; + struct user_fpsimd_state *kernel_fpsimd_state; unsigned int kernel_fpsimd_cpu; #ifdef CONFIG_ARM64_PTR_AUTH struct ptrauth_keys_user keys_user; diff --git a/arch/arm64/include/asm/simd.h b/arch/arm64/include/asm/simd.h index d9f83c478736..7ddb25df5c98 100644 --- a/arch/arm64/include/asm/simd.h +++ b/arch/arm64/include/asm/simd.h @@ -43,8 +43,11 @@ static __must_check inline bool may_use_simd(void) { =20 #endif /* ! CONFIG_KERNEL_MODE_NEON */ =20 -DEFINE_LOCK_GUARD_0(ksimd, kernel_neon_begin(), kernel_neon_end()) +DEFINE_LOCK_GUARD_1(ksimd, + struct user_fpsimd_state, + kernel_neon_begin(_T->lock), + kernel_neon_end(_T->lock)) =20 -#define scoped_ksimd() scoped_guard(ksimd) +#define scoped_ksimd() scoped_guard(ksimd, &(struct user_fpsimd_state){}) =20 #endif diff --git a/arch/arm64/kernel/fpsimd.c b/arch/arm64/kernel/fpsimd.c index e3f8f51748bc..cab866d52bb7 100644 --- a/arch/arm64/kernel/fpsimd.c +++ b/arch/arm64/kernel/fpsimd.c @@ -1489,21 +1489,23 @@ static void fpsimd_load_kernel_state(struct task_st= ruct *task) * Elide the load if this CPU holds the most recent kernel mode * FPSIMD context of the current task. */ - if (last->st =3D=3D &task->thread.kernel_fpsimd_state && + if (last->st =3D=3D task->thread.kernel_fpsimd_state && task->thread.kernel_fpsimd_cpu =3D=3D smp_processor_id()) return; =20 - fpsimd_load_state(&task->thread.kernel_fpsimd_state); + fpsimd_load_state(task->thread.kernel_fpsimd_state); } =20 static void fpsimd_save_kernel_state(struct task_struct *task) { struct cpu_fp_state cpu_fp_state =3D { - .st =3D &task->thread.kernel_fpsimd_state, + .st =3D task->thread.kernel_fpsimd_state, .to_save =3D FP_STATE_FPSIMD, }; =20 - fpsimd_save_state(&task->thread.kernel_fpsimd_state); + BUG_ON(!cpu_fp_state.st); + + fpsimd_save_state(task->thread.kernel_fpsimd_state); fpsimd_bind_state_to_cpu(&cpu_fp_state); =20 task->thread.kernel_fpsimd_cpu =3D smp_processor_id(); @@ -1774,6 +1776,7 @@ void fpsimd_update_current_state(struct user_fpsimd_s= tate const *state) void fpsimd_flush_task_state(struct task_struct *t) { t->thread.fpsimd_cpu =3D NR_CPUS; + t->thread.kernel_fpsimd_state =3D NULL; /* * If we don't support fpsimd, bail out after we have * reset the fpsimd_cpu for this task and clear the @@ -1834,7 +1837,7 @@ void fpsimd_save_and_flush_cpu_state(void) * The caller may freely use the FPSIMD registers until kernel_neon_end() = is * called. */ -void kernel_neon_begin(void) +void kernel_neon_begin(struct user_fpsimd_state *state) { if (WARN_ON(!system_supports_fpsimd())) return; @@ -1846,7 +1849,7 @@ void kernel_neon_begin(void) /* Save unsaved fpsimd state, if any: */ if (test_thread_flag(TIF_KERNEL_FPSTATE)) { BUG_ON(IS_ENABLED(CONFIG_PREEMPT_RT) || !in_serving_softirq()); - fpsimd_save_kernel_state(current); + fpsimd_save_state(state); } else { fpsimd_save_user_state(); =20 @@ -1867,8 +1870,16 @@ void kernel_neon_begin(void) * mode in task context. So in this case, setting the flag here * is always appropriate. */ - if (IS_ENABLED(CONFIG_PREEMPT_RT) || !in_serving_softirq()) + if (IS_ENABLED(CONFIG_PREEMPT_RT) || !in_serving_softirq()) { + /* + * Record the caller provided buffer as the kernel mode + * FP/SIMD buffer for this task, so that the state can + * be preserved and restored on a context switch. + */ + WARN_ON(current->thread.kernel_fpsimd_state !=3D NULL); + current->thread.kernel_fpsimd_state =3D state; set_thread_flag(TIF_KERNEL_FPSTATE); + } } =20 /* Invalidate any task state remaining in the fpsimd regs: */ @@ -1887,21 +1898,26 @@ EXPORT_SYMBOL_GPL(kernel_neon_begin); * The caller must not use the FPSIMD registers after this function is cal= led, * unless kernel_neon_begin() is called again in the meantime. */ -void kernel_neon_end(void) +void kernel_neon_end(struct user_fpsimd_state *state) { if (!system_supports_fpsimd()) return; =20 + if (!test_thread_flag(TIF_KERNEL_FPSTATE)) + return; + /* * If we are returning from a nested use of kernel mode FPSIMD, restore * the task context kernel mode FPSIMD state. This can only happen when * running in softirq context on non-PREEMPT_RT. */ - if (!IS_ENABLED(CONFIG_PREEMPT_RT) && in_serving_softirq() && - test_thread_flag(TIF_KERNEL_FPSTATE)) - fpsimd_load_kernel_state(current); - else + if (!IS_ENABLED(CONFIG_PREEMPT_RT) && in_serving_softirq()) { + fpsimd_load_state(state); + } else { clear_thread_flag(TIF_KERNEL_FPSTATE); + WARN_ON(current->thread.kernel_fpsimd_state !=3D state); + current->thread.kernel_fpsimd_state =3D NULL; + } } EXPORT_SYMBOL_GPL(kernel_neon_end); =20 @@ -1937,7 +1953,7 @@ void __efi_fpsimd_begin(void) WARN_ON(preemptible()); =20 if (may_use_simd()) { - kernel_neon_begin(); + kernel_neon_begin(&efi_fpsimd_state); } else { /* * If !efi_sve_state, SVE can't be in use yet and doesn't need @@ -1986,7 +2002,7 @@ void __efi_fpsimd_end(void) return; =20 if (!efi_fpsimd_state_used) { - kernel_neon_end(); + kernel_neon_end(&efi_fpsimd_state); } else { if (system_supports_sve() && efi_sve_state_used) { bool ffr =3D true; --=20 2.51.0.710.ga91ca5db03-goog