From nobody Thu Oct 2 10:38:59 2025 Received: from mail-wm1-f74.google.com (mail-wm1-f74.google.com [209.85.128.74]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4F75E2D77ED for ; Thu, 18 Sep 2025 06:36:10 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.74 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758177372; cv=none; b=ukR7neVadxrfgsObgJ9C8UbSPLDWxxVVj+aUjKDEGUhd6dR7vLB7nKB59aWT8MuMMgGPWqTKtFpjkKyUAlJ8jNQmdNO/uvLvKwspunWbRC4PclvwwVCN1ostgYh1Gt3wiU2wXlYBNKQEymNMDeWG0/hPwkUptSiGExYq/7ZuZnA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758177372; c=relaxed/simple; bh=mLsK2P9o/PVsSH+ZSFN+0IASgJ72ISRGleKWYu9sCSg=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=OeMfoY0gSpQ/N+IKUSoB4sNb8BQFrZPO3heAX2OISJNjkId07foY5PNQEcI30s1nAAQW0gKQPFsaTphP8ET4dLDSIkLZfFml9czQMD1VbsD/d8fvl5xBlv11PMtkobOI6G/cm9UIdM/Im3BOhuUzMn/HhywdM8tu5yKiPZNbW98= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=D+7e8irB; arc=none smtp.client-ip=209.85.128.74 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="D+7e8irB" Received: by mail-wm1-f74.google.com with SMTP id 5b1f17b1804b1-45df9e11fc6so3461015e9.3 for ; Wed, 17 Sep 2025 23:36:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1758177369; x=1758782169; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=pD1GqEfStiaojYrQnQeMsM18jIp83N4vklBUGeXEk0Y=; b=D+7e8irBiDdNYI7S5xxbpyUYvaBLJCXgUhIrPHm1crl45UieccFahRubnYAfXAnmPi +wKtVsIUbnI+Ae8sBnj39Xno1JPh5+9jg4kMrKaBbfEDIJ2mLUKzsFRsVctxLvpLe2af djb1G5M+NtG+3hBtkGDB/BTqwJziZ9mBPF1B1IJNuuTXTJkj9N+BZhhjb1idtIsRpwVH SSf2hYftLzyO2rG6UdAkx8CkVYJ0OyrA2xsjeyJEH6auY6uGbX3R57FIW94/r29yJ0jP KZvV3uaO9wkyryhWbaInumUEjmyHr6Wm8rcOupH7icM21pfv3j2j97UqiSoSGpTocol5 MRpw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1758177369; x=1758782169; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=pD1GqEfStiaojYrQnQeMsM18jIp83N4vklBUGeXEk0Y=; b=PpsT4gy7bDqqutSxi3JzcJmmrHUVmQQ+K9GF6GdRXSGJXVYimKhV9EuQYKWLah6kZ9 uz7om7DSFTfYbVqhcUMmhuPr1DtuuwEuO6yQJv0Yn9/XPhSWKZLl3vPraOiYQL4TaqTV i4jrtDk3g25ipmGnTI6s0zYoK7NsPmDAgY3Dpl+Sz8hor8VbIjV8kbgFDP4bB+D0Wk0o PKlD9X86AvUlcScCQzG3Ih/7auEjBBmta9lnV+VQ6Mq/BBPsKz9drbOo0lF9ENF8P+mk Uy8B643giHuvboXEMa63fxaF4KE3fNXzURGGMmsDCUUnwqJ9Bn8RA6fuN0UPz6IOC6Bs fKMA== X-Forwarded-Encrypted: i=1; AJvYcCUDx7P03lRRm0ckrbVzAEjRrUAdC4itkwmbsk+AjCvhHLdPaqSTXzBIqa6SIc7+LMR34l+cna31bDPgda4=@vger.kernel.org X-Gm-Message-State: AOJu0YzorMFa4uo82nAiPiSkXIHMvMXlTkAgMChy3wt6UQaEyB4Pnp98 0D/ad9R77/EQvMp8mrauybucEO38SRIoBa9EwRxLckWBazgghK/KxPfZIlUW6rV+Nv+S0sWCuw= = X-Google-Smtp-Source: AGHT+IFBX7q+olV3TxKCMLPGDOXeRlUO54aOaLC2GWe3WCaoWqZDVDYjlTr0slmxpCXGLHTZDbGyUhSR X-Received: from wmrn43.prod.google.com ([2002:a05:600c:502b:b0:45d:eb3b:54a0]) (user=ardb job=prod-delivery.src-stubby-dispatcher) by 2002:a05:600c:3509:b0:45d:f817:c44c with SMTP id 5b1f17b1804b1-46201f8b105mr44939285e9.4.1758177368875; Wed, 17 Sep 2025 23:36:08 -0700 (PDT) Date: Thu, 18 Sep 2025 08:35:41 +0200 In-Reply-To: <20250918063539.2640512-7-ardb+git@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20250918063539.2640512-7-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=877; i=ardb@kernel.org; h=from:subject; bh=XBh8pq2KsPALfmaJFm04pRdLlOab2MUmglzVdmLzRgY=; b=owGbwMvMwCVmkMcZplerG8N4Wi2JIeP0CtusBs1FB0Ilkqf/6511vMyv3m3WTW7p2cK6mvmG7 y7Ky/R1lLIwiHExyIopsgjM/vtu5+mJUrXOs2Rh5rAygQxh4OIUgInwLWFkeLtZV0eJ14IlZdrh Z49bnoeq/lx+6g2z4IXFWotOFsbOT2Zk2Mr280O4kl3o6UN71aymdmzevvzOMfesD5F33m26171 7HSsA X-Mailer: git-send-email 2.51.0.384.g4c02a37b29-goog Message-ID: <20250918063539.2640512-8-ardb+git@google.com> Subject: [PATCH 1/5] crypto/arm64: aes-ce-ccm - Avoid pointless yield of the NEON unit From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: linux-crypto@vger.kernel.org, linux-kernel@vger.kernel.org, herbert@gondor.apana.org.au, ebiggers@kernel.org, Ard Biesheuvel , Marc Zyngier , Will Deacon , Mark Rutland , Kees Cook , Catalin Marinas , Mark Brown Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Ard Biesheuvel Kernel mode NEON sections are now preemptible on arm64, and so there is no need to yield it explicitly in order to prevent scheduling latency spikes. Signed-off-by: Ard Biesheuvel Reviewed-by: Mark Brown --- arch/arm64/crypto/aes-ce-ccm-glue.c | 5 +---- 1 file changed, 1 insertion(+), 4 deletions(-) diff --git a/arch/arm64/crypto/aes-ce-ccm-glue.c b/arch/arm64/crypto/aes-ce= -ccm-glue.c index 2d791d51891b..2eb4e76cabc3 100644 --- a/arch/arm64/crypto/aes-ce-ccm-glue.c +++ b/arch/arm64/crypto/aes-ce-ccm-glue.c @@ -114,11 +114,8 @@ static u32 ce_aes_ccm_auth_data(u8 mac[], u8 const in[= ], u32 abytes, in +=3D adv; abytes -=3D adv; =20 - if (unlikely(rem)) { - kernel_neon_end(); - kernel_neon_begin(); + if (unlikely(rem)) macp =3D 0; - } } else { u32 l =3D min(AES_BLOCK_SIZE - macp, abytes); =20 --=20 2.51.0.384.g4c02a37b29-goog From nobody Thu Oct 2 10:38:59 2025 Received: from mail-wr1-f73.google.com (mail-wr1-f73.google.com [209.85.221.73]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5E49C2D7DD3 for ; Thu, 18 Sep 2025 06:36:11 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.221.73 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758177373; cv=none; b=bmRLpC1IAb/31jdiZScF9NruDqNKecTmWbAnaF2EmLfKl8dbYdny1WO9eryTsqqoE78SlpqJfUflH8dTsd2xIxN26mCTUVso27ew+0VO5hYM0X/d2hL4oiNUViSR+r95GTS7F1IiVk7qU0WIYdw0XaIfnmH6vKLHkn+oxGWH4/0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758177373; c=relaxed/simple; bh=n5KE+fSMMf16oy+VMZN+yuIACUKup3MoEPGdDmhinn0=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=qjoA85/erkebIoCkydm2LkxqRcNPAhiTOuzSRLj6dqmub6/7RSpbqRCjoX81ubP60lU4U/6NSFY3bwjNFMzkKLU63OHSKh4Kqi8SNSB6+tlKSP8SPS/TOH2OMaISVGOGmqy7PhBdom3DVBKH5zddNfTPPglFwtnjNtEkuXLV+tU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=xwHaQ6be; arc=none smtp.client-ip=209.85.221.73 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="xwHaQ6be" Received: by mail-wr1-f73.google.com with SMTP id ffacd0b85a97d-3ece0fd841cso391584f8f.0 for ; Wed, 17 Sep 2025 23:36:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1758177370; x=1758782170; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=Nyu3sJfTvBzaaF8NRzdol7D0KlXiLUJYuDTKFu5v89g=; b=xwHaQ6be6j7B01zuWRzY8+2K1iNQTcociGeIri69gmnvWQ+0C/qDhCHE+hAjuOhUJZ CYHZcCLlzweTBS66K0L65dHRpK7AIHnv/iwYZcnAJGOyNCp6Yi/e7TTyX9LTyIevHpX7 EmI8nh14By10ioLzlOqcrJWhDIOMpYOSjICFCgpYnvyQ883UVKYHUkmb4LavbZHxrDiV hhsqNo3sY32CvVKbFuPO/EwCrMs0zuSAq0o3O/U15Dr6XHJQ98Uw+TKZHxZ14SZ+iuKn 44b4vtbQMRMRVnbWexiB7aULfBrAEGDT2ErW6KNrt4bIm7YFvdrCPn4SHgFTD/xx//35 a/1A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1758177370; x=1758782170; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=Nyu3sJfTvBzaaF8NRzdol7D0KlXiLUJYuDTKFu5v89g=; b=YQCkP4MrJoE/rwGvZc0WZnwWniuzFRLrQRJgB4fNtRAcN/eD0jWefoj6CIVF33NmL5 PWVSurwmT5p0yleHomSmAhT7eiJaKZeHJoFAsXFNn4cJ9j9zvl6L81uNon2zLDGizBcB Vfsw2W7MqvopzpaZ7m/QvfosTZTzLYEvVYEZLo1xwAnB/wiIGOC/zCNiVKoj0D9N1RiZ zy5nqFGJRZZ/+nNAgyR2WhfyC0qHXfT2QzHgTqR1ycj8MvsgmmxTYN7fdFvnkJ8zdi7h 3ICR3AhnuDsAdJR4y3bHFmqYB0f/He/4NIJqC85CQS7Xv4niIjdxCRTVQmo9R/fsqZrT H1PQ== X-Forwarded-Encrypted: i=1; AJvYcCVJHaQDq9tCXwjT+PlXisJibQyxDrnpxms5lohpeo/LmohyMj0yQcn2Z3p8US1Ynf2TAVHX9jB1RT/QeJw=@vger.kernel.org X-Gm-Message-State: AOJu0Yx17C8aqPhsUSVmLzXaz0NbSQwjHP8mnivwdGPUXTGDruaXfDt/ OaiLSpfdBXvmhPiGfAC41zAcNUhlIxb0oCYa+sJfEw50gqEg/d67UzXV5GunBNoZjFDggO/z0w= = X-Google-Smtp-Source: AGHT+IGb4uPsjREwonnmwlL62vl4gaBXll8+OUpYZ766bJ2GqzrjEt+LTSzqc6rAb48NKjy2ajlwiyBq X-Received: from wmbdr18.prod.google.com ([2002:a05:600c:6092:b0:45c:b59f:8805]) (user=ardb job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6000:2084:b0:3df:c5e3:55fe with SMTP id ffacd0b85a97d-3ecdfa3fa97mr3557665f8f.29.1758177369823; Wed, 17 Sep 2025 23:36:09 -0700 (PDT) Date: Thu, 18 Sep 2025 08:35:42 +0200 In-Reply-To: <20250918063539.2640512-7-ardb+git@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20250918063539.2640512-7-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=1398; i=ardb@kernel.org; h=from:subject; bh=dTXA4xH7T53eWCY0KfUSuWwFdpHaVa8zubCJjHScVKg=; b=owGbwMvMwCVmkMcZplerG8N4Wi2JIeP0CvtShrSnrH/mf+9Kv+QanbfrxfSq7ceW7dWUPJX7Z uePpHtzOkpZGMS4GGTFFFkEZv99t/P0RKla51myMHNYmUCGMHBxCsBEyp8z/FN8zda/WnfdybOd 5a+eKDxrZT32aL2f9z2dhL0vHqxhNrrFyDBlU3Lq+R/eWoxC/6T+nLY9obtbbtc1+dtqsxfmnHn xMp8BAA== X-Mailer: git-send-email 2.51.0.384.g4c02a37b29-goog Message-ID: <20250918063539.2640512-9-ardb+git@google.com> Subject: [PATCH 2/5] crypto/arm64: sm4-ce-ccm - Avoid pointless yield of the NEON unit From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: linux-crypto@vger.kernel.org, linux-kernel@vger.kernel.org, herbert@gondor.apana.org.au, ebiggers@kernel.org, Ard Biesheuvel , Marc Zyngier , Will Deacon , Mark Rutland , Kees Cook , Catalin Marinas , Mark Brown Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Ard Biesheuvel Kernel mode NEON sections are now preemptible on arm64, and so there is no need to yield it when calling APIs that may sleep. Also, move the calls to kernel_neon_end() to the same scope as kernel_neon_begin(). This is needed for a subsequent change where a stack buffer is allocated transparently and passed to kernel_neon_begin(). Signed-off-by: Ard Biesheuvel Reviewed-by: Mark Brown --- arch/arm64/crypto/sm4-ce-ccm-glue.c | 10 ++-------- 1 file changed, 2 insertions(+), 8 deletions(-) diff --git a/arch/arm64/crypto/sm4-ce-ccm-glue.c b/arch/arm64/crypto/sm4-ce= -ccm-glue.c index e9cc1c1364ec..f9771ab2a05f 100644 --- a/arch/arm64/crypto/sm4-ce-ccm-glue.c +++ b/arch/arm64/crypto/sm4-ce-ccm-glue.c @@ -179,11 +179,7 @@ static int ccm_crypt(struct aead_request *req, struct = skcipher_walk *walk, walk->src.virt.addr, walk->iv, walk->nbytes - tail, mac); =20 - kernel_neon_end(); - err =3D skcipher_walk_done(walk, tail); - - kernel_neon_begin(); } =20 if (walk->nbytes) { @@ -193,15 +189,13 @@ static int ccm_crypt(struct aead_request *req, struct= skcipher_walk *walk, =20 sm4_ce_ccm_final(rkey_enc, ctr0, mac); =20 - kernel_neon_end(); - err =3D skcipher_walk_done(walk, 0); } else { sm4_ce_ccm_final(rkey_enc, ctr0, mac); - - kernel_neon_end(); } =20 + kernel_neon_end(); + return err; } =20 --=20 2.51.0.384.g4c02a37b29-goog From nobody Thu Oct 2 10:38:59 2025 Received: from mail-wm1-f74.google.com (mail-wm1-f74.google.com [209.85.128.74]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id AA6992DAFB5 for ; Thu, 18 Sep 2025 06:36:12 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.74 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758177374; cv=none; b=gcUhwOR+vsLf3Xv//oZBjLUsArzLwzZWkJ3VTxSxh4fLbu52EOTXx7OZJEZG/zE8Z1gORkYi3MX0B9PIHEaIUcIdG8IeLeuGf5joMGtxC/YMchPydyLgp8HyTYolmQ3lcdPNT9xul7hDFk8YC0wTH6CDsPz2iyl3HqdMA3d8kUA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758177374; c=relaxed/simple; bh=2l669OT1VVHx2cj5ILqnp92BauRxbLKh1Xc2uacpqlI=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=ZIz6iVufC8f+utcfWjf/j7NoMyQdE8fvTw+D8cDJAjXk99jywWOt4emDJGNpb0JfYbIuurnVaQwwqZEPvhJykH8kHcIJ34CT5ZmhA2kV2vnXkCuAj5EGXQJIjH9gUvAM8s52feCvEVLvxtn8PMK91nUM/HMrNwbBDyHec0Jk5sY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=Ua7Zczlj; arc=none smtp.client-ip=209.85.128.74 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="Ua7Zczlj" Received: by mail-wm1-f74.google.com with SMTP id 5b1f17b1804b1-45de5fdda1aso2730595e9.1 for ; Wed, 17 Sep 2025 23:36:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1758177371; x=1758782171; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=z8Z9iG9ZM/mOKO9yo1YmJm8M5BUfmaBw3dfvTInzj84=; b=Ua7ZczljHktTBPnjoWO2gbrsfbn6g2xTTb/V6XtxUYr7ILpGzphXlIEgysWli823v2 bJ1ak5e99cmxbC7DhXHe9Ypjp+Avw2cjSnu7XLKXuvXyxMjNPdd14z68kSMAp6qJhWyO bazyAyJZ76ChJIvdi1X1Uuew5xnboAwzc5agbnv2RL2nJdWgxaYdBUh0EwGKoEKWh2JG w/JQQ1c/DB+DS1J21w+Urj4DTU8o2bsH4509S9ifYTwYLNG6YGe0fkjGCcyq3zVrON4y Ys2HMEFMbSPqETfLgKhLd2jWzuA8/kDG/gnSqvLymUBTBkiOso+8jKdtic6AIv4c0iIw qVrw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1758177371; x=1758782171; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=z8Z9iG9ZM/mOKO9yo1YmJm8M5BUfmaBw3dfvTInzj84=; b=RB1qtm7WM5uDud7tiBbyriLWxnaFQpEQjgPRbndP2EJW67lbkShVy5Qtu5XPHbDz0J zs7qgvWdFf7hKpeQDZwjq2CLNwEyXc6SXRiKKlGFrjmSiEbBH7/qU6ljsDTRxrUx8p+x rELdvROiUeX/Fdq1uxmklWosGGLDwc2UdawFGMWf51+IsIF1gSAvfXTJjQejmu+tBjru xyIpprVqUOVfxqrRgZzqYz1866eIUTsSgWqZgDvQbb9kpsq1ycFT620edzbEwKEshAZS cSnkiEBKpZdO0vs+79u5AcDqZD9EUvRIwuAcWVFv6/VooprVSMY7kH0tJgNlJIDwx+ij zMeQ== X-Forwarded-Encrypted: i=1; AJvYcCWgywcEEQob52l+GQYOPGlAmKn46wk9GmJbO5RkIQuwS4pPt+kOQ0hX/jkidq96BmlNuPPn2HRh2QB5scU=@vger.kernel.org X-Gm-Message-State: AOJu0YwsAHogZGs/s+iYjVwx7nZ+u0TqRRGkCvAzNLY91rPGgiO5J8Ct 6Jc/2bbKsadC+hz3Rm9Jw4VP2mZVu2ndYSkkT3VElAwfO2PSTkz08NmDE7lRndm91pdBLITq2A= = X-Google-Smtp-Source: AGHT+IHxBAKGYkR61gqu3o/eZtme4C7mY2byQeaaeuydCfPZf2Zi8C+SYZvLGQxjBgBDO7fwzsRPKVL5 X-Received: from wmbei6.prod.google.com ([2002:a05:600c:3f06:b0:45f:2b4d:3c2b]) (user=ardb job=prod-delivery.src-stubby-dispatcher) by 2002:adf:fcd0:0:b0:3ed:e1d8:bd73 with SMTP id ffacd0b85a97d-3ede1d8c23amr1247837f8f.57.1758177370889; Wed, 17 Sep 2025 23:36:10 -0700 (PDT) Date: Thu, 18 Sep 2025 08:35:43 +0200 In-Reply-To: <20250918063539.2640512-7-ardb+git@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20250918063539.2640512-7-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=1496; i=ardb@kernel.org; h=from:subject; bh=NpGBicW7ecuYgImi2Z4C0oMoJDuRSNpKAibFeYN1DS4=; b=owGbwMvMwCVmkMcZplerG8N4Wi2JIeP0Codcfb8dR3+tWDl7z4OeX7N+7mBZfqpUzlRu0Z2Eg N0RjsfjOkpZGMS4GGTFFFkEZv99t/P0RKla51myMHNYmUCGMHBxCsBE1rkyMqxy3Z108I3FPy+T 55nvPez/shUk+fnvlZowfxqX4PIFXEAVz7wF1pwVaTiwZrrgxZ7Zynsi8h5NFTE72c94dlHL5Y9 R7AA= X-Mailer: git-send-email 2.51.0.384.g4c02a37b29-goog Message-ID: <20250918063539.2640512-10-ardb+git@google.com> Subject: [PATCH 3/5] crypto/arm64: sm4-ce-gcm - Avoid pointless yield of the NEON unit From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: linux-crypto@vger.kernel.org, linux-kernel@vger.kernel.org, herbert@gondor.apana.org.au, ebiggers@kernel.org, Ard Biesheuvel , Marc Zyngier , Will Deacon , Mark Rutland , Kees Cook , Catalin Marinas , Mark Brown Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Ard Biesheuvel Kernel mode NEON sections are now preemptible on arm64, and so there is no need to yield it when calling APIs that may sleep. Also, move the calls to kernel_neon_end() to the same scope as kernel_neon_begin(). This is needed for a subsequent change where a stack buffer is allocated transparently and passed to kernel_neon_begin(). Signed-off-by: Ard Biesheuvel Reviewed-by: Mark Brown --- arch/arm64/crypto/sm4-ce-gcm-glue.c | 10 +++------- 1 file changed, 3 insertions(+), 7 deletions(-) diff --git a/arch/arm64/crypto/sm4-ce-gcm-glue.c b/arch/arm64/crypto/sm4-ce= -gcm-glue.c index c2ea3d5f690b..170cd0151385 100644 --- a/arch/arm64/crypto/sm4-ce-gcm-glue.c +++ b/arch/arm64/crypto/sm4-ce-gcm-glue.c @@ -165,26 +165,22 @@ static int gcm_crypt(struct aead_request *req, struct= skcipher_walk *walk, ctx->ghash_table, (const u8 *)&lengths); =20 - kernel_neon_end(); - - return skcipher_walk_done(walk, 0); + err =3D skcipher_walk_done(walk, 0); + goto out; } =20 sm4_ce_pmull_gcm_crypt(ctx->key.rkey_enc, dst, src, iv, walk->nbytes - tail, ghash, ctx->ghash_table, NULL); =20 - kernel_neon_end(); - err =3D skcipher_walk_done(walk, tail); - - kernel_neon_begin(); } =20 sm4_ce_pmull_gcm_crypt(ctx->key.rkey_enc, NULL, NULL, iv, walk->nbytes, ghash, ctx->ghash_table, (const u8 *)&lengths); =20 +out: kernel_neon_end(); =20 return err; --=20 2.51.0.384.g4c02a37b29-goog From nobody Thu Oct 2 10:38:59 2025 Received: from mail-wr1-f74.google.com (mail-wr1-f74.google.com [209.85.221.74]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 74F842DCBFC for ; Thu, 18 Sep 2025 06:36:13 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.221.74 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758177375; cv=none; b=DmidChlOE0LG0sp4w7yo3koSytVzeKPN+irSYw8Ue+tUGmZCmbZPpGdwX/PRxB0/hQQEaLMJKW1r/Bue+d36myxUKguB7Op+HZycFGuOFpXesHmq/iSjFfc24Z6zlZR/fzxE0t46jTG0CDH1iU6Bi4aLQZ4rek+OIHtWNPH9Zx4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758177375; c=relaxed/simple; bh=vWcZ1QI8Z0VaPchJBkch+vuuwu97oCktK301998ooq0=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=JwfeIwMI8Rt6TN+VLYBF42Lkn1AAQ0xvzgCBuL7J2I7ePgBYGPHN4E24ztnd2gRYDf9TH+QzWOj+IBB40XY5E3UVy1M6mySTewFqAwMIHXlxyVZMtXahse2YiVN1U55kGc39rrfuFfHOYc9y5nVenLDhjis6Mjp3gXcZqjkAxAI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=VWOp21Sh; arc=none smtp.client-ip=209.85.221.74 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="VWOp21Sh" Received: by mail-wr1-f74.google.com with SMTP id ffacd0b85a97d-3e40057d90aso261084f8f.0 for ; Wed, 17 Sep 2025 23:36:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1758177372; x=1758782172; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=nd/aa+Cyw0z/Ul0/qN6NY/ef41tVXmadJuCUPt5tdFQ=; b=VWOp21ShvkbTAf3SDWezj96iSzA2qVWgyYbTsajuxJAqc2dX1HpgdCAjhWb85ceLqZ fRBNAsdN6PHRSuMjLxIJ59M8u4W4iRCdPi+3HBvVgByoqTXvWVtnGZPPxSYNvxyxvFNr DqnCXcQtzWHuu8tizxSMChSqK8vEoQLl7KG4Ox8Jsq6FYZc2yKRhsHScXsoyeGNAX2qD n8a8fupMZuAJyVh82YC3LIUpp0xvQtQGvjUYggKovowfBqdrNXf6/bQcI5mpcnjjYMDb gjh8Lfw7Ix7q/Rui3YEvppPf80OgZgzWhcay4Ek443flvG0WhlbIzlwfhij6T9SFHpw8 +Hcg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1758177372; x=1758782172; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=nd/aa+Cyw0z/Ul0/qN6NY/ef41tVXmadJuCUPt5tdFQ=; b=uKmmG0laf5D1q7f/qIaX/CoNSqVXsSFHTrBUU1WYtU5WnPzSxJucDyoHntc1za7wkl 8EHpWoUt0CaA7653/kzHNIMujXLsFC09k/a54NdaJ2h7IAQb0rAD2exOrC+Km96U8OrJ zCrGmWWRgH91WTG06SxCRxFx2H3af2I+V8Q9HJ6vL7XYBCHQ877m1sj8191QYfqm/KhE RCda/XEJpNsCiIGIjLIoUsTZMAx/p4B/x/slRyytskv4x65uNisT+t2zuPvSkFIGKFiL C3gVySLE2ESH6JUbxZyw7qCfatceOv9Jy90GmNru1hHpvICXJP+stBXjXCPYhaLjsINi UYeA== X-Forwarded-Encrypted: i=1; AJvYcCWvQ0ZcVgnuTBDidxSrVW5GixfBHH1JnkwzhBmLYiu0ZhfNyUUYOs6CqPSfxrkexHsWhJb4Ix9f13SuRcg=@vger.kernel.org X-Gm-Message-State: AOJu0Ywpy2qXwM8kqo85pSObLS6a/oa28TwUPsj3Beypc7JVMNRXP5fR 1wWA4X8lJv152L/2ji/GAJ29juknZ4CHIUms1V3xHjACTVDm1xXb4XHjBTUCApuEUK0AmQ/oUg= = X-Google-Smtp-Source: AGHT+IEaYfwr01ZjRAssGnsJ1gUGd1rN8g5Gj0e5CjahKJgyizByZHhPk8uuPViunThk6UOOlyd5IgyT X-Received: from wrbbs28.prod.google.com ([2002:a05:6000:71c:b0:3eb:6104:3eb]) (user=ardb job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6000:2284:b0:3ea:124c:8fc7 with SMTP id ffacd0b85a97d-3ecdfa1434dmr3783151f8f.39.1758177371854; Wed, 17 Sep 2025 23:36:11 -0700 (PDT) Date: Thu, 18 Sep 2025 08:35:44 +0200 In-Reply-To: <20250918063539.2640512-7-ardb+git@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20250918063539.2640512-7-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=2845; i=ardb@kernel.org; h=from:subject; bh=fbTmP4aG6tbBHGWrU82oUT6Reb9I9nyA/9uuGWnWHsA=; b=owGbwMvMwCVmkMcZplerG8N4Wi2JIeP0Cqcpcmo3s9lOdcud6TTT/3/e+Haah27JoTmrPn36k qtkYXWuo5SFQYyLQVZMkUVg9t93O09PlKp1niULM4eVCWQIAxenAEykPJyRYc9Ujxln1/M8UN0U srST+emhX57lN5vdFz69bBItdNUs9xHDP1OjzLYvG7QqeQXWeDWfFnstxjJ5Zu7xFo5XF1s+H1P byAcA X-Mailer: git-send-email 2.51.0.384.g4c02a37b29-goog Message-ID: <20250918063539.2640512-11-ardb+git@google.com> Subject: [PATCH 4/5] arm64/fpsimd: Require kernel NEON begin/end calls from the same scope From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: linux-crypto@vger.kernel.org, linux-kernel@vger.kernel.org, herbert@gondor.apana.org.au, ebiggers@kernel.org, Ard Biesheuvel , Marc Zyngier , Will Deacon , Mark Rutland , Kees Cook , Catalin Marinas , Mark Brown Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Ard Biesheuvel In preparation for making kernel_neon_begin() allocate a stack based buffer for stashing the kernel mode FP/SIMD state on a context switch, redefine kernel_neon_begin() and kernel_neon_end() as macros, in a way that requires them to be called from the same scope, and therefore, running from the same stack frame. Signed-off-by: Ard Biesheuvel --- arch/arm64/include/asm/neon.h | 7 +++++-- arch/arm64/kernel/fpsimd.c | 12 ++++++------ 2 files changed, 11 insertions(+), 8 deletions(-) diff --git a/arch/arm64/include/asm/neon.h b/arch/arm64/include/asm/neon.h index d4b1d172a79b..4e24f1058b55 100644 --- a/arch/arm64/include/asm/neon.h +++ b/arch/arm64/include/asm/neon.h @@ -13,7 +13,10 @@ =20 #define cpu_has_neon() system_supports_fpsimd() =20 -void kernel_neon_begin(void); -void kernel_neon_end(void); +void __kernel_neon_begin(void); +void __kernel_neon_end(void); + +#define kernel_neon_begin() do { __kernel_neon_begin() +#define kernel_neon_end() __kernel_neon_end(); } while (0) =20 #endif /* ! __ASM_NEON_H */ diff --git a/arch/arm64/kernel/fpsimd.c b/arch/arm64/kernel/fpsimd.c index c37f02d7194e..d7eb073d1366 100644 --- a/arch/arm64/kernel/fpsimd.c +++ b/arch/arm64/kernel/fpsimd.c @@ -1833,7 +1833,7 @@ void fpsimd_save_and_flush_cpu_state(void) * The caller may freely use the FPSIMD registers until kernel_neon_end() = is * called. */ -void kernel_neon_begin(void) +void __kernel_neon_begin(void) { if (WARN_ON(!system_supports_fpsimd())) return; @@ -1875,7 +1875,7 @@ void kernel_neon_begin(void) =20 put_cpu_fpsimd_context(); } -EXPORT_SYMBOL_GPL(kernel_neon_begin); +EXPORT_SYMBOL_GPL(__kernel_neon_begin); =20 /* * kernel_neon_end(): give the CPU FPSIMD registers back to the current ta= sk @@ -1886,7 +1886,7 @@ EXPORT_SYMBOL_GPL(kernel_neon_begin); * The caller must not use the FPSIMD registers after this function is cal= led, * unless kernel_neon_begin() is called again in the meantime. */ -void kernel_neon_end(void) +void __kernel_neon_end(void) { if (!system_supports_fpsimd()) return; @@ -1902,7 +1902,7 @@ void kernel_neon_end(void) else clear_thread_flag(TIF_KERNEL_FPSTATE); } -EXPORT_SYMBOL_GPL(kernel_neon_end); +EXPORT_SYMBOL_GPL(__kernel_neon_end); =20 #ifdef CONFIG_EFI =20 @@ -1936,7 +1936,7 @@ void __efi_fpsimd_begin(void) WARN_ON(preemptible()); =20 if (may_use_simd()) { - kernel_neon_begin(); + __kernel_neon_begin(); } else { /* * If !efi_sve_state, SVE can't be in use yet and doesn't need @@ -1985,7 +1985,7 @@ void __efi_fpsimd_end(void) return; =20 if (!efi_fpsimd_state_used) { - kernel_neon_end(); + __kernel_neon_end(); } else { if (system_supports_sve() && efi_sve_state_used) { bool ffr =3D true; --=20 2.51.0.384.g4c02a37b29-goog From nobody Thu Oct 2 10:38:59 2025 Received: from mail-wm1-f74.google.com (mail-wm1-f74.google.com [209.85.128.74]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5E5DD2D97B6 for ; Thu, 18 Sep 2025 06:36:14 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.74 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758177376; cv=none; b=LXCrF13qsfnZv/k5761UZZEbGvXBMkFJcEIdxx3vOffEiB1UKDJNyParvW0hiFqDEnQe3urUv4VlUdCDv1JSl5xnGw4gBZkQVLV6k3vTEqTREbvXjHn06DBexdV1RdzrpK46p+zZgF4leyX3dK+y1lQkMCWSJpom+lIV70Tqdh8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758177376; c=relaxed/simple; bh=yZilTx72GF8qqKvNUbLyRsBvojfMOoic5NxQVYR1doQ=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=h78E27mXPpJrT53JEgIamyz7EMKD0BLhIRht5XmkikD7R+kBa4mB/kdZwImyCFGcVnfTVMfbxkV2NgyrTTkdYFbn/GWLwLAm8QsGOV3OROGX1cltVOvNwSIwvnLiQw/MLhy93QIPJDYLoN6tt01BOzK8Prpuw7/4DrXAHCMQt98= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=gCdRHsIt; arc=none smtp.client-ip=209.85.128.74 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="gCdRHsIt" Received: by mail-wm1-f74.google.com with SMTP id 5b1f17b1804b1-45b990eb77cso4265025e9.0 for ; Wed, 17 Sep 2025 23:36:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1758177373; x=1758782173; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=A1ta9KttEF8sJmGrgUmmN67wbPwjz6EsYqXVUKnq6iI=; b=gCdRHsItTVNUcyqQGCJDAlulH7ER5iVlFUaZTwNwI4eMZ2XDpV1v4GCbIoTT5G0qGt R6bNas8vIzA2zVfnuHKk0s8JwBM1G5wIMRhkHpZpA3JIGSR3PCw+qMbMjAVnXC9CYQSO zpPvkP72QHOImC5H2/wcIqjsRLz9Z/CSHM8Wvuyj/CCWHF6tPnSGM9sbkis76GFW4D0n c6F7Yf1U3+aMygwyBY67Hk/ZyrXNMzW0/OMq16xHUOgv/MMuo3QkelOxPuupKgttarGv js5/vmTnJgmC1jg9014BSiC9YTyNaYfx0sEw1Gs3NlgUCJcOGxMKq+m4D38rYZn6aSCj KawQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1758177373; x=1758782173; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=A1ta9KttEF8sJmGrgUmmN67wbPwjz6EsYqXVUKnq6iI=; b=THmX1osgbpbiEFE0z04o0vCFtElATVPFkVrbz+gM1FQIkTymGBGum0Zm/saeYgVYB3 /z0ei5WrIAW8r5GOezAl/JONTwe9CdgBEXaPoODGZT754sZp6Ahs7BDwoZ/gZymdR0AB Jec0y5jFgoBtlGA9u1fhR/gBQtuwl6tVXsyALUYOoMt2R0F/n4RnNIDTwTszYu9+cvZR RneZaU5M/8soQ+HZzNo3HhntkhtnNFP/xtzzda5jz7ZNCOG5TMckKOnnkqNLnTdTjQv9 1qHDA6DAaQHmVW2EKd8o7gzf8iLQrop7ptONql1gnMiqOGMqyx9PyGxfMIjVtX5kt9DV ezeQ== X-Forwarded-Encrypted: i=1; AJvYcCV+GKbP0hkLy90yO48YzAGytEAnpmo9rC/9Vxds+iE4i15xC0xBTJwRQIrZKK9zZiSjVrQz6EXaN+mFe2k=@vger.kernel.org X-Gm-Message-State: AOJu0YzKbEvyBiaMcU/3Us2II5MrVz2s0US4JPxTqgogXqHyumysZ56T fIIp3ttMcY1lespPSTKFCboh4QKzLdDhCZ4V2dKFKCp3tCdaN4N0ijtlXCiXVJN9IjG0gwbYKw= = X-Google-Smtp-Source: AGHT+IHwDdNQKSpulh2BKmpGNYTuQzvpQWSRjIlrRJfiSKNGg3YFQhrNwINDqw6AqWcxEWBgOnrglZXe X-Received: from wmbhc26.prod.google.com ([2002:a05:600c:871a:b0:45f:2859:5428]) (user=ardb job=prod-delivery.src-stubby-dispatcher) by 2002:a05:600c:c0d7:b0:45b:9b9f:88cb with SMTP id 5b1f17b1804b1-464fdf4425emr15559625e9.16.1758177372848; Wed, 17 Sep 2025 23:36:12 -0700 (PDT) Date: Thu, 18 Sep 2025 08:35:45 +0200 In-Reply-To: <20250918063539.2640512-7-ardb+git@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20250918063539.2640512-7-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=5265; i=ardb@kernel.org; h=from:subject; bh=Dk5FXoMF39rDUZtpDj8iIP76qecw7z+7kErWyEtMPzo=; b=owGbwMvMwCVmkMcZplerG8N4Wi2JIeP0CufThftPBJ5ss7Q590bVUEs2Ter/i0lV+lcLvH5sO OEvxvmxo5SFQYyLQVZMkUVg9t93O09PlKp1niULM4eVCWQIAxenAExE8A0jQ1dfScM10ctyTn+n ty7RPK65fneCi7xyrPjMkFVnk3OWVDH8r2H2uPotW6BoL8e7uzOTizP+SbnZ3nFqrFr5/s6HEP6 VLAA= X-Mailer: git-send-email 2.51.0.384.g4c02a37b29-goog Message-ID: <20250918063539.2640512-12-ardb+git@google.com> Subject: [PATCH 5/5] arm64/fpsimd: Allocate kernel mode FP/SIMD buffers on the stack From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: linux-crypto@vger.kernel.org, linux-kernel@vger.kernel.org, herbert@gondor.apana.org.au, ebiggers@kernel.org, Ard Biesheuvel , Marc Zyngier , Will Deacon , Mark Rutland , Kees Cook , Catalin Marinas , Mark Brown Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Ard Biesheuvel Commit aefbab8e77eb16b5 ("arm64: fpsimd: Preserve/restore kernel mode NEON at context switch") added a 'kernel_fpsimd_state' field to struct thread_struct, which is the arch-specific portion of struct task_struct, and is allocated for each task in the system. The size of this field is 528 bytes, resulting in non-trivial bloat of task_struct, and the resulting memory overhead may impact performance on systems with many processes. This allocation is only used if the task is scheduled out or interrupted by a softirq while using the FP/SIMD unit in kernel mode, and given that calls to kernel_neon_begin() and kernel_neon_end() are now guaranteed to originate from the same lexical scope, it is possible to transparently allocate this buffer on the caller's stack instead. Signed-off-by: Ard Biesheuvel --- arch/arm64/include/asm/neon.h | 4 +-- arch/arm64/include/asm/processor.h | 2 +- arch/arm64/kernel/fpsimd.c | 26 ++++++++++++++------ 3 files changed, 21 insertions(+), 11 deletions(-) diff --git a/arch/arm64/include/asm/neon.h b/arch/arm64/include/asm/neon.h index 4e24f1058b55..acaac98ff449 100644 --- a/arch/arm64/include/asm/neon.h +++ b/arch/arm64/include/asm/neon.h @@ -13,10 +13,10 @@ =20 #define cpu_has_neon() system_supports_fpsimd() =20 -void __kernel_neon_begin(void); +void __kernel_neon_begin(struct user_fpsimd_state *); void __kernel_neon_end(void); =20 -#define kernel_neon_begin() do { __kernel_neon_begin() +#define kernel_neon_begin() do { __kernel_neon_begin(&(struct user_fpsimd_= state){}) #define kernel_neon_end() __kernel_neon_end(); } while (0) =20 #endif /* ! __ASM_NEON_H */ diff --git a/arch/arm64/include/asm/processor.h b/arch/arm64/include/asm/pr= ocessor.h index 61d62bfd5a7b..226e635c53d9 100644 --- a/arch/arm64/include/asm/processor.h +++ b/arch/arm64/include/asm/processor.h @@ -172,7 +172,7 @@ struct thread_struct { unsigned long fault_code; /* ESR_EL1 value */ struct debug_info debug; /* debugging */ =20 - struct user_fpsimd_state kernel_fpsimd_state; + struct user_fpsimd_state *kernel_fpsimd_state; unsigned int kernel_fpsimd_cpu; #ifdef CONFIG_ARM64_PTR_AUTH struct ptrauth_keys_user keys_user; diff --git a/arch/arm64/kernel/fpsimd.c b/arch/arm64/kernel/fpsimd.c index d7eb073d1366..919c53a26484 100644 --- a/arch/arm64/kernel/fpsimd.c +++ b/arch/arm64/kernel/fpsimd.c @@ -1488,21 +1488,23 @@ static void fpsimd_load_kernel_state(struct task_st= ruct *task) * Elide the load if this CPU holds the most recent kernel mode * FPSIMD context of the current task. */ - if (last->st =3D=3D &task->thread.kernel_fpsimd_state && + if (last->st =3D=3D task->thread.kernel_fpsimd_state && task->thread.kernel_fpsimd_cpu =3D=3D smp_processor_id()) return; =20 - fpsimd_load_state(&task->thread.kernel_fpsimd_state); + fpsimd_load_state(task->thread.kernel_fpsimd_state); } =20 static void fpsimd_save_kernel_state(struct task_struct *task) { struct cpu_fp_state cpu_fp_state =3D { - .st =3D &task->thread.kernel_fpsimd_state, + .st =3D task->thread.kernel_fpsimd_state, .to_save =3D FP_STATE_FPSIMD, }; =20 - fpsimd_save_state(&task->thread.kernel_fpsimd_state); + BUG_ON(!cpu_fp_state.st); + + fpsimd_save_state(task->thread.kernel_fpsimd_state); fpsimd_bind_state_to_cpu(&cpu_fp_state); =20 task->thread.kernel_fpsimd_cpu =3D smp_processor_id(); @@ -1773,6 +1775,7 @@ void fpsimd_update_current_state(struct user_fpsimd_s= tate const *state) void fpsimd_flush_task_state(struct task_struct *t) { t->thread.fpsimd_cpu =3D NR_CPUS; + t->thread.kernel_fpsimd_state =3D NULL; /* * If we don't support fpsimd, bail out after we have * reset the fpsimd_cpu for this task and clear the @@ -1833,7 +1836,7 @@ void fpsimd_save_and_flush_cpu_state(void) * The caller may freely use the FPSIMD registers until kernel_neon_end() = is * called. */ -void __kernel_neon_begin(void) +void __kernel_neon_begin(struct user_fpsimd_state *s) { if (WARN_ON(!system_supports_fpsimd())) return; @@ -1849,6 +1852,13 @@ void __kernel_neon_begin(void) } else { fpsimd_save_user_state(); =20 + /* + * Record the caller provided buffer as the kernel mode FP/SIMD + * buffer for this task, so that the state can be preserved and + * restored on a context switch. + */ + current->thread.kernel_fpsimd_state =3D s; + /* * Set the thread flag so that the kernel mode FPSIMD state * will be context switched along with the rest of the task @@ -1899,8 +1909,8 @@ void __kernel_neon_end(void) if (!IS_ENABLED(CONFIG_PREEMPT_RT) && in_serving_softirq() && test_thread_flag(TIF_KERNEL_FPSTATE)) fpsimd_load_kernel_state(current); - else - clear_thread_flag(TIF_KERNEL_FPSTATE); + else if (test_and_clear_thread_flag(TIF_KERNEL_FPSTATE)) + current->thread.kernel_fpsimd_state =3D NULL; } EXPORT_SYMBOL_GPL(__kernel_neon_end); =20 @@ -1936,7 +1946,7 @@ void __efi_fpsimd_begin(void) WARN_ON(preemptible()); =20 if (may_use_simd()) { - __kernel_neon_begin(); + __kernel_neon_begin(&efi_fpsimd_state); } else { /* * If !efi_sve_state, SVE can't be in use yet and doesn't need --=20 2.51.0.384.g4c02a37b29-goog