From nobody Mon Dec 1 22:03:59 2025 Received: from mail-wr1-f74.google.com (mail-wr1-f74.google.com [209.85.221.74]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id CA1DA30F7F2 for ; Thu, 27 Nov 2025 09:22:55 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.221.74 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1764235377; cv=none; b=oJdx/giHaykqsA9Txa1BhSaUaVG9laefqIii2bi7cOHqgveSsRKzFcBb4Fs/1OE88zwGJV3M/FXzZWbXL0vwTYTZ8L27tdQUpPWhP4gDDE6oKzbFFyJXP4fhnGYPz6MFbF7JpZlfvt0/a2qNZi0aETDyrz2FMa/ggr2RT4GXqrA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1764235377; c=relaxed/simple; bh=5pU8T3R8EfCbCbbm9FQnjNIKnM4Oc0VA2KxjEL7tDKI=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=av1ppr4Rd99hYrRcMIw+fDJL+HvC04mdDRoUIaeSyA3unHT0NgHD1oDs9d1mPUh3ansSOit6SHElU5PpO/sd3leuFJtTZlu6FMN+zDj3XtzEydxjVitQCce6SXM22DHntMF77ivbCEEvsebdwuCDISWKOWexRmdE4eG92EZkODo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=UkA7+Gvy; arc=none smtp.client-ip=209.85.221.74 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="UkA7+Gvy" Received: by mail-wr1-f74.google.com with SMTP id ffacd0b85a97d-429cdb0706aso497459f8f.0 for ; Thu, 27 Nov 2025 01:22:55 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1764235374; x=1764840174; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=lm8/PsD24JkqtJEBb/JHL05JmcK80K+YBsbDIDkSPDM=; b=UkA7+GvyMvUjv3iPLCdYg8o+5CUVRL1ncq5gdLMjiOtbOthuiglXgfK+ydxFcJd3AC lo6ER/rjiLjPGA07DsHTLzhAcZMc5tJsotFoJGEBHbun4e5AR1WwmdeagkfJ8OKDdUZe CbCvC+lQdbg3oj2VJ/RHRFwmlupxKeqgOS/tCCm+uTTGBEFTkgLNlPPCySgrtXX/D4gZ 1Ohj3jhJv1/teyIpaGM0e4XtANKbQi7q0GhKia+eepsUIVvxL6zuheDHAxzZZ4QHJfvD rKGdlEniApnNQ62L+LHpW+HwgUgJmdMx5sZcwcdYVVMirys0cqq37OgKnerKYZYwyVUu PwSw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1764235374; x=1764840174; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=lm8/PsD24JkqtJEBb/JHL05JmcK80K+YBsbDIDkSPDM=; b=kGokUecMKeyk80cy5W8h1i8wAKTNTqftZX8kjQEio5U+vjnLj01fhADViLtLap+igO JQBd6TQ34XslyDmDYlVqSMZlv9TrLF7bL+YHIXKF9wojkyqA3Iroy3CvBVQN7uKzz+yB PzyWM1WVsUHB6HLnSnf+S0gZr/wayaFT+SJN0rhCgVfa3WpbAVQswjsvXkBM9HV3pFVb kX2z3ULSCvh12t1sZkjDaZXT8NGBMTnISeqNc63Hb1owuFC6J8c9UfKS+9IyZbdwFjFb nJdUEcGoaohbPnffkJ4rGo95n2q+zJNJX0AwTTTrV6gRiaiAzDnnR30+QAuXiZD925EC Czrg== X-Forwarded-Encrypted: i=1; AJvYcCUpNrlWhfPSb94PjmDi7xS46iJNbX3UwjwOv433dlDKBY5o52h121gGFmOdmuQYsKTbluXzevKQI+2KrWk=@vger.kernel.org X-Gm-Message-State: AOJu0Yzkk4xG2YAB/VT0muOTdw47J29WdHCyXdjDV010RZ2XPyVOIwBB vPCq/fbo7NwefA75enaiho+ItAtmLGuSAnjvj9fL1atTsIvYYrrCpxg0OR5bgII7kjemY1EzCw= = X-Google-Smtp-Source: AGHT+IFiRUSTUyUmmDCfp9DRVB+ITplmlr8U1Vk596xaInsS5h9GpzAVehE1Ma2+bZO9lWJEhGAZ3DXo X-Received: from wrwt2.prod.google.com ([2002:a5d:6a42:0:b0:429:de3f:8280]) (user=ardb job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6000:1447:b0:42b:32c3:3948 with SMTP id ffacd0b85a97d-42cc1d0ce41mr23885901f8f.31.1764235374004; Thu, 27 Nov 2025 01:22:54 -0800 (PST) Date: Thu, 27 Nov 2025 10:22:28 +0100 In-Reply-To: <20251127092226.1439196-8-ardb+git@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20251127092226.1439196-8-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=875; i=ardb@kernel.org; h=from:subject; bh=LXaoefJgP6t3V697yrG8QR6lNpOK4nbDq1CishfTlx0=; b=owGbwMvMwCVmkMcZplerG8N4Wi2JIVNDItRQuip5xYWJK0Jad7HG9b2/1b1Fii33VdirLhdnB tc7m+w6SlkYxLgYZMUUWQRm/3238/REqVrnWbIwc1iZQIYwcHEKwEQ4mRn+aa9dL7tAoXn35YWy QSLC104ZXX9x+MnM33/3+icp7VJvc2dk6Fganr6zP8NI1tT1wnbPW+tMZMK2Xolc17SUM+bHreX qDAA= X-Mailer: git-send-email 2.52.0.107.ga0afd4fd5b-goog Message-ID: <20251127092226.1439196-9-ardb+git@google.com> Subject: [RFC/RFT PATCH 1/6] hexagon: Wire up cmpxchg64_local() to generic implementation From: Ard Biesheuvel To: linux-hardening@vger.kernel.org Cc: linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, Ard Biesheuvel , Kees Cook , Ryan Roberts , Will Deacon , Arnd Bergmann , Jeremy Linton , Catalin Marinas , Mark Rutland , "Jason A. Donenfeld" Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Ard Biesheuvel Provide cmpxchg64_local() for hexagon so we can start using it in generic code. Signed-off-by: Ard Biesheuvel --- arch/hexagon/include/asm/cmpxchg.h | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/arch/hexagon/include/asm/cmpxchg.h b/arch/hexagon/include/asm/= cmpxchg.h index 9c58fb81f7fd..05e426475d2a 100644 --- a/arch/hexagon/include/asm/cmpxchg.h +++ b/arch/hexagon/include/asm/cmpxchg.h @@ -8,6 +8,8 @@ #ifndef _ASM_CMPXCHG_H #define _ASM_CMPXCHG_H =20 +#include + /* * __arch_xchg - atomically exchange a register and a memory location * @x: value to swap @@ -72,4 +74,6 @@ __arch_xchg(unsigned long x, volatile void *ptr, int size) __oldval; \ }) =20 +#define arch_cmpxchg64_local __generic_cmpxchg64_local + #endif /* _ASM_CMPXCHG_H */ --=20 2.52.0.107.ga0afd4fd5b-goog From nobody Mon Dec 1 22:03:59 2025 Received: from mail-ej1-f74.google.com (mail-ej1-f74.google.com [209.85.218.74]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A50DA328241 for ; Thu, 27 Nov 2025 09:22:56 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.218.74 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1764235378; cv=none; b=XK9w2nkDGW1C5MPGXxLEqDSGKv8FCT5eWI3F7oH7F4OCKQCkT8rUhiMOqSpGXe/g7Yw1S+SkUYKcS4cDdLz63IWsLuqxgN3HClkDL65RR9FGQeI9zus1y1EDXcmwTB/YkoGuDp2FH34+XOB3fwa4D0x5inlptqxQVw5PzVmhnrs= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1764235378; c=relaxed/simple; bh=Ue2OWN8dxl4S1oCDpiOXA5V1r7n4OPJH5dF2+7O1GlI=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=u1jlzQSf9J6+yh5xp2elvGOVsdfngqONdmpGixxFungb5l0eT3U3HFIF0Z+WROkcX/2dqkpp78tnUAqcUVMK20JPPr1Guf6btW0oDmv3QAW0VdVL+tptnfUBqxCaVvNC+RRv6E0EZPbC8tooSD2TaEMfiXWfnclD3eZVVKn/jkU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=A93bpXOa; arc=none smtp.client-ip=209.85.218.74 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="A93bpXOa" Received: by mail-ej1-f74.google.com with SMTP id a640c23a62f3a-b72a95dc686so53156966b.3 for ; Thu, 27 Nov 2025 01:22:56 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1764235375; x=1764840175; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=nxY6K3VXBOtHUoqxNWrBAsAGxnWGhHRmDGILp9vXmzY=; b=A93bpXOahsQgW+Vy8fvUntMEmJfgeMYfjDpj+zPji5/hYLF1B3XkncUIrWXZ/OSa6A hwzyWDyNHeWssLS9f93I/QJmU8CDUtwVhFutQBgw0BttV4bY1L6ljGRI/MrdG/mM3bXV w/TDhl5HrYCU5ciW3/fHfASaHPiiCfGFBThWs0uXx1DG6Si61xCi4akLXKs74yioD470 9oxd2+Zuyj0aZwP8C67bFUuu1RkEbaGNCotjqrXDem2XeYRabfwcGjuBrmO21dTMMIRm 4qV8UZRUEfjExMj/hIDF7WjMY7KZjReWFZrg9RufodtIza0v6gQu1ztT8NXImXDuIuJG qnSg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1764235375; x=1764840175; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=nxY6K3VXBOtHUoqxNWrBAsAGxnWGhHRmDGILp9vXmzY=; b=Gzt05x7fAIE6R+zEqWPNcRx+Z1bseh2eEiUv/c9kqt2xVGF60AntEG3U6NdyaCR4US iBcQUMjz07ksWLBUqIm+vBEVI5G41ovxbcdgfZNlzM37N5Xv7TFQ1K8KqRkKMJWZCQxk 5rbxZaVbsj0VPuBeKBQkBJ5+7qbDmKFslRbEv7PjZc98AMkijXFNd09BVK9DcvDo0X5u Sci2adp9Qq9K1lK/RLz68z8pWHqGuhynGDtsy5ao73B5l6Kvvx4OIYMTNes9DK4h+XCD 4oUwZRKJmK89120fuLdtXa1d4IAYneEruWTkWR7kRMpLwm9GHDqMq3oXoxCXpni+08RT qqyw== X-Forwarded-Encrypted: i=1; AJvYcCVBaqq8zoB/H3zOUo4wkP0pysO0UFxbnXzI2GYALHXXZwZje08zWCUNKnD1MAOh1OgyMwZ6HdZExB4ClB4=@vger.kernel.org X-Gm-Message-State: AOJu0Yw2f8mwNOq5AdfBHS33jruykI0U3cbHGE0toWkt+AX+kAK4ng2R NRAB9EPAVEpl5+I8sefVj4erUTiHtx/UjHZeeGiO9lXKKvWhpq3Z6AWFFBNzks9ZAfIjdYFzdQ= = X-Google-Smtp-Source: AGHT+IGFdDZxAqRvQLpZk6/eWvniWko12T2iQL0bdLzORuqHked+ojY3yfI07AtzD9o2ck5cQMXXIq6+ X-Received: from ejcvv1.prod.google.com ([2002:a17:907:a681:b0:b73:36a1:4995]) (user=ardb job=prod-delivery.src-stubby-dispatcher) by 2002:a17:907:3d90:b0:b0e:d477:4972 with SMTP id a640c23a62f3a-b76c54090f0mr967515866b.25.1764235375046; Thu, 27 Nov 2025 01:22:55 -0800 (PST) Date: Thu, 27 Nov 2025 10:22:29 +0100 In-Reply-To: <20251127092226.1439196-8-ardb+git@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20251127092226.1439196-8-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=697; i=ardb@kernel.org; h=from:subject; bh=bPOIFl7qSFo8fyyP7vM3iVB+S1w+UZidjo3tNBxTCOI=; b=owGbwMvMwCVmkMcZplerG8N4Wi2JIVNDIkz65cQ92/8IPf/2+Omz95cLdJSj//5xu/T6vgODl 7BcJ49nRykLgxgXg6yYIovA7L/vdp6eKFXrPEsWZg4rE8gQBi5OAZjIxyyGfxY7e1+ncsRubd0+ r/dYC9upMAneH5cXs1vfeyHuyNHMVc7IMOWVVNRJEeuexw86Pyaei/4Wy1X763/6xVc6DttPhKw q5gQA X-Mailer: git-send-email 2.52.0.107.ga0afd4fd5b-goog Message-ID: <20251127092226.1439196-10-ardb+git@google.com> Subject: [RFC/RFT PATCH 2/6] arc: Wire up cmpxchg64_local() to generic implementation From: Ard Biesheuvel To: linux-hardening@vger.kernel.org Cc: linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, Ard Biesheuvel , Kees Cook , Ryan Roberts , Will Deacon , Arnd Bergmann , Jeremy Linton , Catalin Marinas , Mark Rutland , "Jason A. Donenfeld" Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Ard Biesheuvel Provide cmpxchg64_local() for hexagon so we can start using it in generic code. Signed-off-by: Ard Biesheuvel --- arch/arc/include/asm/cmpxchg.h | 3 +++ 1 file changed, 3 insertions(+) diff --git a/arch/arc/include/asm/cmpxchg.h b/arch/arc/include/asm/cmpxchg.h index 76f43db0890f..f2d55823645c 100644 --- a/arch/arc/include/asm/cmpxchg.h +++ b/arch/arc/include/asm/cmpxchg.h @@ -12,6 +12,7 @@ =20 #include #include +#include =20 #ifdef CONFIG_ARC_HAS_LLSC =20 @@ -142,4 +143,6 @@ =20 #endif =20 +#define arch_cmpxchg64_local __generic_cmpxchg64_local + #endif --=20 2.52.0.107.ga0afd4fd5b-goog From nobody Mon Dec 1 22:03:59 2025 Received: from mail-wm1-f74.google.com (mail-wm1-f74.google.com [209.85.128.74]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B9740329366 for ; Thu, 27 Nov 2025 09:22:57 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.74 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1764235380; cv=none; b=V0AwiNCKFFiSxBOMljUjUwgwZsw912rWNlT7LNkUXGyKk3TWSW66hP4iruNtWrn34t84f1WvLBuPvbD1oput3/HCUO8wSQziqjotmFQeZOK5fSlwrSGnxjHZLIz/ipNnX+C9s+nUSchgqdzyD5/YhUcx/B5Lkrkw8HZ/idZi6cM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1764235380; c=relaxed/simple; bh=ICrszCMtJ1LPXNd7C0Ih5tX9jm2ajTy+fULG6068eLw=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=ZY/Fel9nJNqCPvgGSlbSYr8zNZvBOZvAreFaFFpykVYSf1aATaIJWXZ5vyPgsyh4jGAeiylXUDCnQuOhfRpGiRgyO3bRNxg6O++kwnu/rje6JZxZNJT4jrOI2TtM3UrWm5994ptV4UA58gM4Qd5QyV2b086ExCwBkNg51eecIaY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=s2ejCHsB; arc=none smtp.client-ip=209.85.128.74 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="s2ejCHsB" Received: by mail-wm1-f74.google.com with SMTP id 5b1f17b1804b1-4779b432aecso3242855e9.0 for ; Thu, 27 Nov 2025 01:22:57 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1764235376; x=1764840176; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=G8PUYE65+35CiVgD1jp+ggIop/9/H10ihf2g4jePZBk=; b=s2ejCHsBoK0nHEX0MpvjncNiNr7/MoflMgFd+uukCn9mClhqfERv5hqpbY85yejr0m 3z2by1Sp+1ng3A3mDm4HWKqtsG87lE002JTUCNbsQq7ESvuLspsAY1twreWya4ENid6R r1HXcPQ8r9sjI0MOQrq9FXEoEme68dn8cpjjkPgP2Au4Xg+tLtRvZkwMDjQ03gwNh0+m qHgms78BcXA7Ixopb3EpeLjEqddnhZxha7hLp2yrauONE9kFh/obOTxbZfQkDd+pB4oo aM9my8px53pKdv+3u1Kqa76OBpPkxSSUGjJY9BCinXc9loS+oM46/N2ys3Pfmxoa93GT xkcw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1764235376; x=1764840176; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=G8PUYE65+35CiVgD1jp+ggIop/9/H10ihf2g4jePZBk=; b=GXwTzw32JU6odp4HE343mHuhcSVDfje62ShrPvhdqUGlpeX7FUEkQwmT3qPjrNk7SE L675Y2rmuwrQLRJ6t/xXPIV5/7/qi+w73UmhSKzXWnDSyIe7jq9Whu4I8vll3+7iG6oe n9iWbPtPfRzLE7rZooQTLzyHLHC6gGJePcNrnvTHcdxuj+pDmbRpPJBYZeUvnhaKNCs6 LoE+esUeuLmiiFqk0b2TGLMRmQNoVmdFj4Ym7CuYgUJcQOzssSSNoJ2G5EDjVw03t3XG oV48L1Zc7qhKQo7/YALi2KB2NuFz1A7f/pH85ia0outKGJuTEfyvtQVjxgFBWeuNSKr4 sKsg== X-Forwarded-Encrypted: i=1; AJvYcCXT2nJ6cJPpAkW4yi/EjM2LPbgkB+3KScwvc1Na8T4QFG4r4aM8G5P1rp0eBFsxs0PUib+2xGAg314jhiI=@vger.kernel.org X-Gm-Message-State: AOJu0YyGmUKOsXlahoESqItGcJ2NlBkiAUEGWszo5S0s5lxIQoaENENk ykxEgE6QtpTYAQukm5UrBE3xyhKGFt9Uu8zAra6N2ANb8lpFSYvWKv03HANL7FYwbE/DlooFyA= = X-Google-Smtp-Source: AGHT+IHGmStaBw0kw+zywe3YOqYeTN7JkzcsA1+VS2rjrhQpPUmECbRNT13sfo28k0MCm5UyvTanyD0N X-Received: from wmbh15.prod.google.com ([2002:a05:600c:a10f:b0:477:988a:7675]) (user=ardb job=prod-delivery.src-stubby-dispatcher) by 2002:a05:600c:1382:b0:477:a977:b8a7 with SMTP id 5b1f17b1804b1-477c1117956mr205200185e9.19.1764235376129; Thu, 27 Nov 2025 01:22:56 -0800 (PST) Date: Thu, 27 Nov 2025 10:22:30 +0100 In-Reply-To: <20251127092226.1439196-8-ardb+git@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20251127092226.1439196-8-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=1946; i=ardb@kernel.org; h=from:subject; bh=39kBqCzlZF8SLfFJgWsjj400jCEDRYRkq2STbPbbenE=; b=owGbwMvMwCVmkMcZplerG8N4Wi2JIVNDIqI3ZHbml073v/KHdtpnHfpRm/Py3LMdqh9Sljd+f rhn84cNHaUsDGJcDLJiiiwCs/++23l6olSt8yxZmDmsTCBDGLg4BWAi7oYM/0M7zlcnzJvAe846 80bGvn+b3uxjbs5YvCouYfXjnXIBr/4yMixpPnD96cT+y7kXZlzTaJTelrqtrYVD1/PozHcXtNf yb2UEAA== X-Mailer: git-send-email 2.52.0.107.ga0afd4fd5b-goog Message-ID: <20251127092226.1439196-11-ardb+git@google.com> Subject: [RFC/RFT PATCH 3/6] random: Use u32 to keep track of batched entropy generation From: Ard Biesheuvel To: linux-hardening@vger.kernel.org Cc: linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, Ard Biesheuvel , Kees Cook , Ryan Roberts , Will Deacon , Arnd Bergmann , Jeremy Linton , Catalin Marinas , Mark Rutland , "Jason A. Donenfeld" Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Ard Biesheuvel The batched entropy containers each have a generation field, to keep track of the base_crng generation from which it was last reseeded. This use case does not require all bits of the unsigned long to be stored: storing only 32 bits is sufficient to determine whether or not we're at most 4 billion generations behind, which seems ample. So use an unsigned int instead: this will allow a future patch to treat the generation and position as a single 64-bit quantity, which can be used locklessly in a compare-and-exchange() operation. Signed-off-by: Ard Biesheuvel --- drivers/char/random.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/drivers/char/random.c b/drivers/char/random.c index b8b24b6ed3fe..0e04bc60d034 100644 --- a/drivers/char/random.c +++ b/drivers/char/random.c @@ -507,7 +507,7 @@ struct batch_ ##type { \ */ \ type entropy[CHACHA_BLOCK_SIZE * 3 / (2 * sizeof(type))]; \ local_lock_t lock; \ - unsigned long generation; \ + unsigned int generation; \ unsigned int position; \ }; \ \ @@ -521,7 +521,7 @@ type get_random_ ##type(void) \ type ret; \ unsigned long flags; \ struct batch_ ##type *batch; \ - unsigned long next_gen; \ + unsigned int next_gen; \ \ warn_unseeded_randomness(); \ \ @@ -533,7 +533,7 @@ type get_random_ ##type(void) \ local_lock_irqsave(&batched_entropy_ ##type.lock, flags); \ batch =3D raw_cpu_ptr(&batched_entropy_##type); \ \ - next_gen =3D READ_ONCE(base_crng.generation); \ + next_gen =3D (unsigned int)READ_ONCE(base_crng.generation); \ if (batch->position >=3D ARRAY_SIZE(batch->entropy) || \ next_gen !=3D batch->generation) { \ _get_random_bytes(batch->entropy, sizeof(batch->entropy)); \ --=20 2.52.0.107.ga0afd4fd5b-goog From nobody Mon Dec 1 22:03:59 2025 Received: from mail-wr1-f74.google.com (mail-wr1-f74.google.com [209.85.221.74]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id DBF18329C53 for ; Thu, 27 Nov 2025 09:22:58 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.221.74 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1764235382; cv=none; b=f1LVI+BjcHTL1DR62aKWEGdUV/mr9BoYG2eP4uwXQxLeQdla3kRR1wAeBv6ojn9lvwZi3Q1WAIZ0wemUL0Qwz3B8HtvTWFGBW3EH/FyXBWeYwckEajC0BLsgSSRVg6YCs+J3qEq/5Wo3ib4P60QduvPPx1+qkI7IOFs0xA8CxDY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1764235382; c=relaxed/simple; bh=F2UdfCjShiQuHh3oQQRYHB4xtYfaoIt5uvb/4ByzDno=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=GUrLXn3LEnEESW7RZjqREjxRc2HZkRjGeqNSLxz4n/uaKjUDUH0mv5qDEqNturysTRq6g4fk2Lqbn4U1aq5v1eC0s2iEK8jObjO7hk0QBoZIL/mr+shmxf9WuX1bhx/gtNaVaQWCDeq4rrsJbfYGF8a+ylbkfElCjGukyKlkLqc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=pZJ1u51H; arc=none smtp.client-ip=209.85.221.74 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="pZJ1u51H" Received: by mail-wr1-f74.google.com with SMTP id ffacd0b85a97d-429c7b0ae36so445338f8f.0 for ; Thu, 27 Nov 2025 01:22:58 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1764235377; x=1764840177; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=wzVFhaCJ4tPvEzjzVh2/FLJsOENJe5tM82eV69ICp3w=; b=pZJ1u51HYn6Z3mdEJPllCXyHTdraMpcNmsUTdxW5UpdYzALZw1y2IIvsfLONEd4/By uDl87otuD/09t6oCtKMHYvurg7XLzJ31NICBwmXCrpf/x7LKBeRtVy9bcVLr2yEIZc+Y ac50xRUpzj7piw3ipk/PQGLtsX+e6yuWaNTfFBnOvwPJM5h5b9FKdoj4dYZzujx5zje6 kV1WvglHSomVB+9liVn+3R1l3HPJRbt2/8qDO9lw49o5RFnFllLPfC+B5U4znHNJ00LH it1a6FJAilaNT6OEkVo4wJnbX031MEbascrDIbPoH3AZDHqXCXfh4R7MLcn/k9TE5nyL eWwA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1764235377; x=1764840177; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=wzVFhaCJ4tPvEzjzVh2/FLJsOENJe5tM82eV69ICp3w=; b=qyKcO+Yv551+TgEMGoINq0/yWQrkCxgOo+GEXHngkYuk+wKuieaKG7VQ6Kb0Yy1Ihu /2pPcaNqf/CF8w3AW1qnK0EY9Pbe0Oir9dUapFbp8HQJnNUbwO+GrsGkUIa8mVxH6TLA 7mUjd07RCebbP13/Dffsi+2Isdlbh8ks3kzLLsRjmIPQQwbY/HtgdsBxp0pGKL1mxkG0 NlFmUPua4Fsdpb5Xr8LdlhTKWlOmE1gR/uVRObeO7zSZZ11wt8YljKMd9MzmyCzHP/xZ o6ycCkDwkUtUVSL8hrMzkZ01Y68LsIug8OqItb9g68JR8RwjszN+Wxfrl4ocOHR70X9X n66w== X-Forwarded-Encrypted: i=1; AJvYcCUDPYgdMDMdUfQKhy/1MLqm83d2dVKvfTsokhBOQF1Hwkv1aLewLGNyH+9CGWzIyLNGarwMD5NpjSLCK0M=@vger.kernel.org X-Gm-Message-State: AOJu0Yw6hN+5VPlyoiGf93s7ddoq+Yvu8Rp5VuWFrMVnZxb2TcnA2A2G +itG+Jm7p8euDCI2h5E9zGL6wx48Oby4pvtpoUQfSmwtDBh7we3bggmSlMTJOBIASsNEWBosZQ= = X-Google-Smtp-Source: AGHT+IGd+iJrVyefWu9xcpdYdKEXoxro/lG9n2ewRU/WJbCL+3bUle2bLSXZHjm1+EN52LuTQiG20BfQ X-Received: from wrqr7.prod.google.com ([2002:a5d:4987:0:b0:42b:b28a:6746]) (user=ardb job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6000:4010:b0:42b:3dbe:3a37 with SMTP id ffacd0b85a97d-42cc1302285mr26345067f8f.10.1764235376919; Thu, 27 Nov 2025 01:22:56 -0800 (PST) Date: Thu, 27 Nov 2025 10:22:31 +0100 In-Reply-To: <20251127092226.1439196-8-ardb+git@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20251127092226.1439196-8-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=4605; i=ardb@kernel.org; h=from:subject; bh=2kdRtRT+6jrXGVJpTsN6JRllZWlsEfiMzJMbIKIwLig=; b=owGbwMvMwCVmkMcZplerG8N4Wi2JIVNDItKrV2/nxk2T1FL2Pt6aq6pVO+2PRuinE8KPNTo6a 3nj3j3uKGVhEONikBVTZBGY/ffdztMTpWqdZ8nCzGFlAhnCwMUpABPp8mFkuDohVFskVsdiR1zu wmdCi5KUPl1VU7Fe/dFZccnmV/sc3jEynJLJaUjTW3hYn2F3RubPpNRt29cIn9B6cfPQJ+dKTzk BPgA= X-Mailer: git-send-email 2.52.0.107.ga0afd4fd5b-goog Message-ID: <20251127092226.1439196-12-ardb+git@google.com> Subject: [RFC/RFT PATCH 4/6] random: Use a lockless fast path for get_random_uXX() From: Ard Biesheuvel To: linux-hardening@vger.kernel.org Cc: linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, Ard Biesheuvel , Kees Cook , Ryan Roberts , Will Deacon , Arnd Bergmann , Jeremy Linton , Catalin Marinas , Mark Rutland , "Jason A. Donenfeld" Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Ard Biesheuvel Currently, the implementations of the get_random_uXX() API protect their critical section with a local lock and disabling interrupts, to ensure that the code does not race with itself when called from interrupt context. Given that the fast path does nothing more than read a single uXX quantity from a linear buffer and bump the position pointer, poking the hardware registers to disable and re-enable interrupts is disproportionately costly, and best avoided. There are two conditions under which the batched entropy buffer is replenished, which is what forms the critical section: - the buffer is exhausted - the base_crng generation counter has incremented. By combining the position and generation counters into a single u64, we can use compare and exchange to implement the fast path without taking the local lock or disabling interrupts. By constructing the expected and next values carefully, the compare and exchange will only succeed if - we did not race with ourselves, i.e., the compare and exchange increments the position counter by exactly 1; - the buffer is not exhausted - the generation counter equals the base_crng generation counter. Only if the compare and exchange fails is the original slow path taken, and only in that case do we take the local lock. This results in a considerable speedup (3-5x) when benchmarking get_random_u8() in a tight loop. Signed-off-by: Ard Biesheuvel --- drivers/char/random.c | 44 ++++++++++++++------ 1 file changed, 31 insertions(+), 13 deletions(-) diff --git a/drivers/char/random.c b/drivers/char/random.c index 0e04bc60d034..71bd74871540 100644 --- a/drivers/char/random.c +++ b/drivers/char/random.c @@ -496,6 +496,12 @@ static ssize_t get_random_bytes_user(struct iov_iter *= iter) * should be called and return 0 at least once at any point prior. */ =20 +#ifdef __LITTLE_ENDIAN +#define LOHI(lo, hi) lo, hi +#else +#define LOHI(lo, hi) hi, lo +#endif + #define DEFINE_BATCHED_ENTROPY(type) \ struct batch_ ##type { \ /* \ @@ -507,8 +513,12 @@ struct batch_ ##type { \ */ \ type entropy[CHACHA_BLOCK_SIZE * 3 / (2 * sizeof(type))]; \ local_lock_t lock; \ - unsigned int generation; \ - unsigned int position; \ + union { \ + struct { \ + unsigned int LOHI(position, generation); \ + }; \ + u64 posgen; \ + }; \ }; \ \ static DEFINE_PER_CPU(struct batch_ ##type, batched_entropy_ ##type) =3D {= \ @@ -522,6 +532,7 @@ type get_random_ ##type(void) \ unsigned long flags; \ struct batch_ ##type *batch; \ unsigned int next_gen; \ + u64 next; \ \ warn_unseeded_randomness(); \ \ @@ -530,21 +541,28 @@ type get_random_ ##type(void) \ return ret; \ } \ \ - local_lock_irqsave(&batched_entropy_ ##type.lock, flags); \ - batch =3D raw_cpu_ptr(&batched_entropy_##type); \ + batch =3D &get_cpu_var(batched_entropy_##type); \ \ next_gen =3D (unsigned int)READ_ONCE(base_crng.generation); \ - if (batch->position >=3D ARRAY_SIZE(batch->entropy) || \ - next_gen !=3D batch->generation) { \ - _get_random_bytes(batch->entropy, sizeof(batch->entropy)); \ - batch->position =3D 0; \ - batch->generation =3D next_gen; \ + next =3D (u64)next_gen << 32; \ + if (likely(batch->position < ARRAY_SIZE(batch->entropy))) { \ + next |=3D batch->position + 1; /* next-1 is bogus otherwise */ \ + ret =3D batch->entropy[batch->position]; \ + } \ + if (cmpxchg64_local(&batch->posgen, next, next - 1) !=3D next - 1) { \ + local_lock_irqsave(&batched_entropy_ ##type.lock, flags); \ + if (batch->position >=3D ARRAY_SIZE(batch->entropy) || \ + next_gen !=3D batch->generation) { \ + _get_random_bytes(batch->entropy, sizeof(batch->entropy));\ + batch->position =3D 0; \ + batch->generation =3D next_gen; \ + } \ + ret =3D batch->entropy[batch->position++]; \ + local_unlock_irqrestore(&batched_entropy_ ##type.lock, flags); \ } \ \ - ret =3D batch->entropy[batch->position]; \ - batch->entropy[batch->position] =3D 0; \ - ++batch->position; \ - local_unlock_irqrestore(&batched_entropy_ ##type.lock, flags); \ + batch->entropy[batch->position - 1] =3D 0; \ + put_cpu_var(batched_entropy_##type); \ return ret; \ } \ EXPORT_SYMBOL(get_random_ ##type); --=20 2.52.0.107.ga0afd4fd5b-goog From nobody Mon Dec 1 22:03:59 2025 Received: from mail-wm1-f74.google.com (mail-wm1-f74.google.com [209.85.128.74]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5733E329C71 for ; Thu, 27 Nov 2025 09:22:59 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.74 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1764235383; cv=none; b=p1RCQlNqLzu7W+dmw7B7AA2TkN9QIBKa1uUwEaTx/JmzkOKiPR+nBFGlC6q7urudKPXXU0zulz1ATphKnL1JWPoj7ydo8iK0VI88jN2N807no24Mi30pX85ZkmtMSQUgvNj9iYmGw7z/VP0kE4++54Q4RsFO1tBuTTizxtP2kkY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1764235383; c=relaxed/simple; bh=kldLPMrabkFH+b+H9XhnBTB2n/8hF1gS8m0DCzqmMtI=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=ebmx847xsvEBhGSUJXnoPfKB9FkWV3YFryTBWIM8cdvP70X9sn8Xm16Yf4jHcnbwcGzHObYsIMDCF2U0swWcZviuo5+VqiTL2qYhVBiNSJ1G0hQjg3+ntcvfcINPlx4bbIlb+cAv74t8QsicgrrLrMWWHj/2iF6vs08lqtewfDI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=AZsWkVQf; arc=none smtp.client-ip=209.85.128.74 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="AZsWkVQf" Received: by mail-wm1-f74.google.com with SMTP id 5b1f17b1804b1-47775585257so3819815e9.1 for ; Thu, 27 Nov 2025 01:22:59 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1764235378; x=1764840178; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=JB581+q6d8JNHOaG4/CmEZh7pybRjg32PuHi0tZmZIA=; b=AZsWkVQfb3HM5MwRffzi9JM6066sFJpmiENOFnZcTqI/K4bJtYzS7Gu5xCItNytTq6 DCPNvQ+KSeKVJSRvHBijtxdHJRxkWuqUrin74LSFvuGtWhxXKoME6ZgptM1LuGfixBTF yg8o5U1hWaj4uAgRI3ExGDhFMNwuvoqPsryjX8mjsZ0p0HKZGf1xj3iSNCeI8GHfBe7N sTrf2snT0Ek82qQKNEfVBI0gyqCaI1CMrc5KtBHLyHDFhrADVosSszM4dU9TktouyuFC urbREQAaT+nQikRG0pzpEbxzl7zp90WH8uMcAMCrrE5037p+/HtkYqBt1RV9lcaD8628 frbw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1764235378; x=1764840178; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=JB581+q6d8JNHOaG4/CmEZh7pybRjg32PuHi0tZmZIA=; b=CYAj3QMeTbwV0C24b2rKLD88sx1n157g27jOB3KM5kDaRXlnek8E53bhnqlcYbBXGL YmshCP04kuhNVP5RmeQpvzSySSJBFb9pa6h3QM8GpqratuH3NEdkZUCK4Lmgp4GWZt+i qR8u7dEOExombydBsfptIgTshReFj2RSL6pYjVE/fyKQrGt/5XPGrJVXHrZzdFRko8mx NHf9+i1bCGxWl3xHb4+GtznYl2tgAgIdRmygT6z96ah9ZR+Hc+MCpOG1uMnGkpgRAf/H +w1o6s3mnMGimVBNM8Q1GIDwpkAWGXHIfX6algcwa1ZllABWsb6qi7ncuSyuYqgHiiyt ge3A== X-Forwarded-Encrypted: i=1; AJvYcCWKy/ffi3Gg5nkwBZKj9eNycbvfCcgqtgA9EwSHuf7d8vdGuzc6Gmd6Uhxv+fR3KMK6k5Nne7XHdu/iaVM=@vger.kernel.org X-Gm-Message-State: AOJu0YwtbxSeRyWyFSIDU+Z9qjjcaKuKOLHhgUda2H+iMxyoE7fYTFkI +rotKSHsc+HYy1E6uLwPU5q5S3PSIA3qhFlWi6Blhd5yuw/oKZMI38YsN3MrS5x59PAoNl+zuw= = X-Google-Smtp-Source: AGHT+IFYv0kogWxmcupk5m1hglVZfqkLWIq7UCyp+YAuGkotWUr1R/Sn1oljNF0wCvZqVj/2EKsscbj3 X-Received: from wmcn10.prod.google.com ([2002:a05:600c:c0ca:b0:477:afa:d217]) (user=ardb job=prod-delivery.src-stubby-dispatcher) by 2002:a05:600c:5489:b0:477:b0b8:4dd0 with SMTP id 5b1f17b1804b1-477c1119c31mr210109105e9.17.1764235378167; Thu, 27 Nov 2025 01:22:58 -0800 (PST) Date: Thu, 27 Nov 2025 10:22:32 +0100 In-Reply-To: <20251127092226.1439196-8-ardb+git@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20251127092226.1439196-8-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=1326; i=ardb@kernel.org; h=from:subject; bh=RU3u+df4cRqtBp/zfgIcQz76smq5xWuDnV7q7Ms5D20=; b=owGbwMvMwCVmkMcZplerG8N4Wi2JIVNDIrr465FjPcfCXNcVfxHj4EvKbXd52VbuPb/gSv/TG XXn23d0lLIwiHExyIopsgjM/vtu5+mJUrXOs2Rh5rAygQxh4OIUgImUFTH8j/4n/GKfxT4FdwFR 3ZtT2XVftd3pMVD/b7vd89W/09tYChkZzgTY/PHoy9xc3v9lheV8neWin38fWBntofx47Zr7DX8 +MgIA X-Mailer: git-send-email 2.52.0.107.ga0afd4fd5b-goog Message-ID: <20251127092226.1439196-13-ardb+git@google.com> Subject: [RFC/RFT PATCH 5/6] random: Plug race in preceding patch From: Ard Biesheuvel To: linux-hardening@vger.kernel.org Cc: linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, Ard Biesheuvel , Kees Cook , Ryan Roberts , Will Deacon , Arnd Bergmann , Jeremy Linton , Catalin Marinas , Mark Rutland , "Jason A. Donenfeld" Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Ard Biesheuvel The lockless get_random_uXX() reads the next value from the linear buffer and then overwrites it with a 0x0 value. This is racy, as the code might be re-entered by an interrupt handler, and so the store might redundantly wipe the location accessed by the interrupt context rather than the interrupted context. To plug this race, wipe the preceding location when reading the next value from the linear buffer. Given that the position is always non-zero outside of the critical section, this is guaranteed to be safe, and ensures that the produced values are always wiped from the buffer. Signed-off-by: Ard Biesheuvel --- drivers/char/random.c | 1 + 1 file changed, 1 insertion(+) diff --git a/drivers/char/random.c b/drivers/char/random.c index 71bd74871540..e8ba460c5c9c 100644 --- a/drivers/char/random.c +++ b/drivers/char/random.c @@ -547,6 +547,7 @@ type get_random_ ##type(void) \ next =3D (u64)next_gen << 32; \ if (likely(batch->position < ARRAY_SIZE(batch->entropy))) { \ next |=3D batch->position + 1; /* next-1 is bogus otherwise */ \ + batch->entropy[batch->position - 1] =3D 0; \ ret =3D batch->entropy[batch->position]; \ } \ if (cmpxchg64_local(&batch->posgen, next, next - 1) !=3D next - 1) { \ --=20 2.52.0.107.ga0afd4fd5b-goog From nobody Mon Dec 1 22:03:59 2025 Received: from mail-wm1-f73.google.com (mail-wm1-f73.google.com [209.85.128.73]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8DEB532936E for ; Thu, 27 Nov 2025 09:23:01 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.73 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1764235384; cv=none; b=I+cZuRkfo+/rEmN+l2bXG2ti4pr791cPGPkRK4p8TappTGKE9Joox5zV3ySYw/Nfr2MHEGsCISGuT5IcMDElFMKkieZ3fxdDz7op4bQs3Nz2vDUS5OPxjxjLoJH3qED5UVVqX9Pv5Fh59pHBbo7K1CIumLKodnQ6aySwLu96LCs= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1764235384; c=relaxed/simple; bh=9G3bFsH+JBVhTmjQAmKXBaP5hiXW4+Cp+p5cAJyH2SI=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=g/+8kHZ+Ylkpwe7yM7AGh2bPA9jHnqc20sBRSt/s33Ac4S5JzTEobjq7Qdc0w+/6UI3cKbwdVSmlY3w9y8ZhY0K9hkBVQ5rXw3M25utjX4caDGMsnMl6drqxOLkJFcxDi9Vt8W903uSD9IdWuAsAtn9u8lW3EzNKn3hhKbMHkIo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=NWVZqUsO; arc=none smtp.client-ip=209.85.128.73 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="NWVZqUsO" Received: by mail-wm1-f73.google.com with SMTP id 5b1f17b1804b1-477563a0c75so2951215e9.1 for ; Thu, 27 Nov 2025 01:23:01 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1764235379; x=1764840179; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=EUL0bXq8chbknjAsMuUQFU4e6lGXKM8WYzXlQLV3Sa4=; b=NWVZqUsOBszDPPbkTSBUgPoVErlgHtq24SjH36cDGhcgDzJspm0ObX4fipQFprwJ2u qmRyXRzd7++ENr+MJGV9FAiO7aPQFvNWw9UPDxijZxFudDO0pPL/LJLL3VX5Q6oTI8dv ftxwDDXzhclruHzFIs0ZSNrpfMBax2LgeY6FCjzhKl+v0s8Jw95fgWJaoWyz2t6i+ThP TaUXwexS2BvuItkI7q5S4RJ7LLb/DxvB9vbQ9lIo43NLUaDdvP4AUi+XvXtj96xQTF9/ y/pvDcdCmAJJyfhm+gnCVa2JAlBbv7krSxZw3fw4INVADWmA1aiC70BiKbJGf7aLtqHv hVUg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1764235379; x=1764840179; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=EUL0bXq8chbknjAsMuUQFU4e6lGXKM8WYzXlQLV3Sa4=; b=wv+5BfD4Y3uWWrJ5sLwo1q/s4e6H0sCEzg+p3lGUTuVyZYQVre5DDtk1oHQFva69ev jJwsFmhyT1wR18t6ZEE5FyKQw50aW+053QjNCowlwO777qo6zNgwnnD5FHhSYK/asysv misTaz+F3zMLqlfOk9B/jFv3bl+RZ737dov05DLBZg1s++/jl2YIxKaymtwW1DC/GNrH BEx2I9AnxqsjIm7fTOUUMiUOvGUn/0nVeSXvbtofPri4am1EmsiEWERJ3Y/30TDa+Aq8 gNLR7x42afDuNLudBKS2F931lUJ9F3xz+fenGJeVIk+DkIzvk8xKxeq/YWwT3CxHtrnS OxHQ== X-Forwarded-Encrypted: i=1; AJvYcCWEXrxrMGCeCMs0wWsGTAWEF8Bayr90crK9utEYhCaG9rT3pUvXaQsJnfWNLa5ruSUCegfslxK39Q0JmjY=@vger.kernel.org X-Gm-Message-State: AOJu0YxiGilPY2fZD2DQCfrcYbEylO5WHhs4RU21F67Tq3eVgwYqEcf9 j2Lra+Ax7DFAA0+vsn0FbnfHS6CBsx8SGsIDgUkQbfD34g7c1/8Ok5uIxcfv1Uj7XV65UgP8SA= = X-Google-Smtp-Source: AGHT+IEj3dusT3MDvpMgsbHZIfe/XZF/uYEQTBHV/zXh1M/+kIcSqUep12/hQYGAypuqMk42bH+W9Glh X-Received: from wmri12-n2.prod.google.com ([2002:a05:600c:8a0c:20b0:477:9a28:9ea7]) (user=ardb job=prod-delivery.src-stubby-dispatcher) by 2002:a05:600c:4ed3:b0:477:b734:8c22 with SMTP id 5b1f17b1804b1-47904ad05bbmr93756915e9.8.1764235379306; Thu, 27 Nov 2025 01:22:59 -0800 (PST) Date: Thu, 27 Nov 2025 10:22:33 +0100 In-Reply-To: <20251127092226.1439196-8-ardb+git@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20251127092226.1439196-8-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=5832; i=ardb@kernel.org; h=from:subject; bh=8GaMUTgdagAMtzY89sbLAS19s1WTGVeC8Lgqk53nbS8=; b=owGbwMvMwCVmkMcZplerG8N4Wi2JIVNDImbL21N8hQ7vpnw54rsvfi1f31pm9QU1E1ScxBx79 eeeOqXQUcrCIMbFICumyCIw+++7nacnStU6z5KFmcPKBDKEgYtTACai6cHwTz1Igemqx8v3z7Zv 10zom521SvxatNkqR5cSmfPekY8+nmFkOKutlWi8pZ79TQHDv40xK48GnP8TPXNRmZZEU2Ppcpt tHAA= X-Mailer: git-send-email 2.52.0.107.ga0afd4fd5b-goog Message-ID: <20251127092226.1439196-14-ardb+git@google.com> Subject: [RFC/RFT PATCH 6/6] randomize_kstack: Use get_random_u8() at entry for entropy From: Ard Biesheuvel To: linux-hardening@vger.kernel.org Cc: linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, Ard Biesheuvel , Kees Cook , Ryan Roberts , Will Deacon , Arnd Bergmann , Jeremy Linton , Catalin Marinas , Mark Rutland , "Jason A. Donenfeld" Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Ard Biesheuvel The randomize kstack code has a couple of sharp edges, which are due to the fact that a counter was considered more suitable for providing entropy than the kernel's RNG. Given that kstack randomization only requires 6-8 bits of entropy, get_random_u8() is more suitable here, and its fast path has been made lockless so that there is no overhead beyond a preempt_dis/enable and a local 64-bit cmpxchg() to update the RNG's internal state. This means there is no need to defer sampling the counter until syscall exit, which also removes the need for a per-CPU variable to carry that value over to the next syscall on the same CPU. That works around some issues identified by Ryan in [0]. So replace the use of the per-CPU variable in add_random_kstack_offset() with a direct use of get_random_u8() for all users, and turn choose_kstack_random_offset() into an empty stub so that each arch can get rid of it individually. [0] https://lore.kernel.org/all/dd8c37bc-795f-4c7a-9086-69e584d8ab24@arm.co= m/ Signed-off-by: Ard Biesheuvel --- arch/Kconfig | 9 +++-- include/linux/randomize_kstack.h | 36 +++----------------- init/main.c | 1 - 3 files changed, 8 insertions(+), 38 deletions(-) diff --git a/arch/Kconfig b/arch/Kconfig index 61130b88964b..baea154f833e 100644 --- a/arch/Kconfig +++ b/arch/Kconfig @@ -1520,12 +1520,11 @@ config HAVE_ARCH_RANDOMIZE_KSTACK_OFFSET help An arch should select this symbol if it can support kernel stack offset randomization with calls to add_random_kstack_offset() - during syscall entry and choose_random_kstack_offset() during - syscall exit. Careful removal of -fstack-protector-strong and + during syscall entry. Careful removal of -fstack-protector-strong and -fstack-protector should also be applied to the entry code and - closely examined, as the artificial stack bump looks like an array - to the compiler, so it will attempt to add canary checks regardless - of the static branch state. + closely examined, as the artificial stack bump looks like an array to + the compiler, so it will attempt to add canary checks regardless of + the static branch state. =20 config RANDOMIZE_KSTACK_OFFSET bool "Support for randomizing kernel stack offset on syscall entry" if EX= PERT diff --git a/include/linux/randomize_kstack.h b/include/linux/randomize_kst= ack.h index 1d982dbdd0d0..db1f056e61ec 100644 --- a/include/linux/randomize_kstack.h +++ b/include/linux/randomize_kstack.h @@ -9,7 +9,6 @@ =20 DECLARE_STATIC_KEY_MAYBE(CONFIG_RANDOMIZE_KSTACK_OFFSET_DEFAULT, randomize_kstack_offset); -DECLARE_PER_CPU(u32, kstack_offset); =20 /* * Do not use this anywhere else in the kernel. This is used here because @@ -50,49 +49,22 @@ DECLARE_PER_CPU(u32, kstack_offset); * add_random_kstack_offset - Increase stack utilization by previously * chosen random offset * - * This should be used in the syscall entry path when interrupts and - * preempt are disabled, and after user registers have been stored to - * the stack. For testing the resulting entropy, please see: + * This should be used in the syscall entry path after user registers have= been + * stored to the stack. For testing the resulting entropy, please see: * tools/testing/selftests/lkdtm/stack-entropy.sh */ #define add_random_kstack_offset() do { \ if (static_branch_maybe(CONFIG_RANDOMIZE_KSTACK_OFFSET_DEFAULT, \ &randomize_kstack_offset)) { \ - u32 offset =3D raw_cpu_read(kstack_offset); \ + u32 offset =3D get_random_u8() << 2; \ u8 *ptr =3D __kstack_alloca(KSTACK_OFFSET_MAX(offset)); \ /* Keep allocation even after "ptr" loses scope. */ \ asm volatile("" :: "r"(ptr) : "memory"); \ } \ } while (0) - -/** - * choose_random_kstack_offset - Choose the random offset for the next - * add_random_kstack_offset() - * - * This should only be used during syscall exit when interrupts and - * preempt are disabled. This position in the syscall flow is done to - * frustrate attacks from userspace attempting to learn the next offset: - * - Maximize the timing uncertainty visible from userspace: if the - * offset is chosen at syscall entry, userspace has much more control - * over the timing between choosing offsets. "How long will we be in - * kernel mode?" tends to be more difficult to predict than "how long - * will we be in user mode?" - * - Reduce the lifetime of the new offset sitting in memory during - * kernel mode execution. Exposure of "thread-local" memory content - * (e.g. current, percpu, etc) tends to be easier than arbitrary - * location memory exposure. - */ -#define choose_random_kstack_offset(rand) do { \ - if (static_branch_maybe(CONFIG_RANDOMIZE_KSTACK_OFFSET_DEFAULT, \ - &randomize_kstack_offset)) { \ - u32 offset =3D raw_cpu_read(kstack_offset); \ - offset =3D ror32(offset, 5) ^ (rand); \ - raw_cpu_write(kstack_offset, offset); \ - } \ -} while (0) #else /* CONFIG_RANDOMIZE_KSTACK_OFFSET */ #define add_random_kstack_offset() do { } while (0) -#define choose_random_kstack_offset(rand) do { } while (0) #endif /* CONFIG_RANDOMIZE_KSTACK_OFFSET */ +#define choose_random_kstack_offset(rand) do { } while (0) =20 #endif diff --git a/init/main.c b/init/main.c index 07a3116811c5..048a62538242 100644 --- a/init/main.c +++ b/init/main.c @@ -830,7 +830,6 @@ static inline void initcall_debug_enable(void) #ifdef CONFIG_RANDOMIZE_KSTACK_OFFSET DEFINE_STATIC_KEY_MAYBE_RO(CONFIG_RANDOMIZE_KSTACK_OFFSET_DEFAULT, randomize_kstack_offset); -DEFINE_PER_CPU(u32, kstack_offset); =20 static int __init early_randomize_kstack_offset(char *buf) { --=20 2.52.0.107.ga0afd4fd5b-goog