From nobody Sun Jun 28 02:48:46 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 42069C433F5 for ; Tue, 15 Feb 2022 21:23:13 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S241021AbiBOVXW (ORCPT ); Tue, 15 Feb 2022 16:23:22 -0500 Received: from mxb-00190b01.gslb.pphosted.com ([23.128.96.19]:45616 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230057AbiBOVXT (ORCPT ); Tue, 15 Feb 2022 16:23:19 -0500 Received: from ams.source.kernel.org (ams.source.kernel.org [145.40.68.75]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A0A42DAAFD for ; Tue, 15 Feb 2022 13:23:08 -0800 (PST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 61750B81C26 for ; Tue, 15 Feb 2022 21:23:07 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 6321DC340EB; Tue, 15 Feb 2022 21:23:05 +0000 (UTC) Authentication-Results: smtp.kernel.org; dkim=pass (1024-bit key) header.d=zx2c4.com header.i=@zx2c4.com header.b="CTEYBH8x" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=zx2c4.com; s=20210105; t=1644960184; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=nS+oXcZ2pN2lYjrkN1g4qSCJevoLs3pQQOe/+hYsyoY=; b=CTEYBH8x4bXf5K2MNWmw3jehA2slD8U3HWhanzb8wmKnMmHiv93xtyHkIGhouPrjhA7Y+6 K3YYYxoA+dCu/9rtVQC7G76ygVqn3C+URjcuS3ldpzF433g9K1CfRK4b4DOon7P65T8LBd h4cVQvwRLz8fhRhDT2VJr8LHcSljm80= Received: by mail.zx2c4.com (ZX2C4 Mail Server) with ESMTPSA id b8f126ed (TLSv1.3:AEAD-AES256-GCM-SHA384:256:NO); Tue, 15 Feb 2022 21:23:03 +0000 (UTC) From: "Jason A. Donenfeld" To: linux-kernel@vger.kernel.org, bigeasy@linutronix.de, linux@dominikbrodowski.net, sultan@kerneltoast.com Cc: "Jason A. Donenfeld" , Thomas Gleixner , Peter Zijlstra , Theodore Ts'o Subject: [PATCH v4 1/2] random: set fast pool count to zero in cpuhp prepare Date: Tue, 15 Feb 2022 22:22:54 +0100 Message-Id: <20220215212255.273253-2-Jason@zx2c4.com> In-Reply-To: <20220215212255.273253-1-Jason@zx2c4.com> References: <20220215212255.273253-1-Jason@zx2c4.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Rather than having to use expensive atomics, which were visibly the most expensive thing in the entire irq handler, simply take care of the extreme edge case of resetting count to 0 in the cpuhp prepare handler, before irqs have started. This simplifies the code a bit and lets us use vanilla variables rather than atomics, and performance should be improved. Cc: Thomas Gleixner Cc: Peter Zijlstra Cc: Theodore Ts'o Cc: Sebastian Andrzej Siewior Cc: Sultan Alsawaf Cc: Dominik Brodowski Signed-off-by: Jason A. Donenfeld --- drivers/char/random.c | 39 +++++++++++++++++++++++--------------- include/linux/cpuhotplug.h | 1 + include/linux/random.h | 4 ++++ kernel/cpu.c | 6 ++++++ 4 files changed, 35 insertions(+), 15 deletions(-) diff --git a/drivers/char/random.c b/drivers/char/random.c index ee21f301ff16..a3cc147406b0 100644 --- a/drivers/char/random.c +++ b/drivers/char/random.c @@ -1182,7 +1182,7 @@ struct fast_pool { }; struct work_struct mix; unsigned long last; - atomic_t count; + unsigned int count; u16 reg_idx; }; =20 @@ -1218,6 +1218,25 @@ static void fast_mix(u32 pool[4]) =20 static DEFINE_PER_CPU(struct fast_pool, irq_randomness); =20 +#ifdef CONFIG_SMP +int random_prepare_cpu(unsigned int cpu) +{ + /* + * We want to reset a few things when a CPU comes online, in case + * it was previously offlined and therefore has stale information. + */ + + /* + * Set irq randomness count to zero so that new accumulated + * irqs are fresh, and more importantly, so that its worker + * is permitted to schedule again when it comes back online, + * since the MIX_INFLIGHT flag will be cleared. + */ + per_cpu_ptr(&irq_randomness, cpu)->count =3D 0; + return 0; +} +#endif + static u32 get_reg(struct fast_pool *f, struct pt_regs *regs) { u32 *ptr =3D (u32 *)regs; @@ -1242,15 +1261,6 @@ static void mix_interrupt_randomness(struct work_str= uct *work) local_irq_disable(); if (fast_pool !=3D this_cpu_ptr(&irq_randomness)) { local_irq_enable(); - /* - * If we are unlucky enough to have been moved to another CPU, - * during CPU hotplug while the CPU was shutdown then we set - * our count to zero atomically so that when the CPU comes - * back online, it can enqueue work again. The _release here - * pairs with the atomic_inc_return_acquire in - * add_interrupt_randomness(). - */ - atomic_set_release(&fast_pool->count, 0); return; } =20 @@ -1259,7 +1269,7 @@ static void mix_interrupt_randomness(struct work_stru= ct *work) * consistent view, before we reenable irqs again. */ memcpy(pool, fast_pool->pool32, sizeof(pool)); - atomic_set(&fast_pool->count, 0); + fast_pool->count =3D 0; fast_pool->last =3D jiffies; local_irq_enable(); =20 @@ -1295,14 +1305,13 @@ void add_interrupt_randomness(int irq) } =20 fast_mix(fast_pool->pool32); - /* The _acquire here pairs with the atomic_set_release in mix_interrupt_r= andomness(). */ - new_count =3D (unsigned int)atomic_inc_return_acquire(&fast_pool->count); + new_count =3D ++fast_pool->count; =20 if (unlikely(crng_init =3D=3D 0)) { if (new_count >=3D 64 && crng_pre_init_inject(fast_pool->pool32, sizeof(fast_pool->pool32), true, true) > 0) { - atomic_set(&fast_pool->count, 0); + fast_pool->count =3D 0; fast_pool->last =3D now; if (spin_trylock(&input_pool.lock)) { _mix_pool_bytes(&fast_pool->pool32, sizeof(fast_pool->pool32)); @@ -1320,7 +1329,7 @@ void add_interrupt_randomness(int irq) =20 if (unlikely(!fast_pool->mix.func)) INIT_WORK(&fast_pool->mix, mix_interrupt_randomness); - atomic_or(MIX_INFLIGHT, &fast_pool->count); + fast_pool->count |=3D MIX_INFLIGHT; queue_work_on(raw_smp_processor_id(), system_highpri_wq, &fast_pool->mix); } EXPORT_SYMBOL_GPL(add_interrupt_randomness); diff --git a/include/linux/cpuhotplug.h b/include/linux/cpuhotplug.h index 411a428ace4d..38294af566e4 100644 --- a/include/linux/cpuhotplug.h +++ b/include/linux/cpuhotplug.h @@ -127,6 +127,7 @@ enum cpuhp_state { CPUHP_MM_ZSWP_POOL_PREPARE, CPUHP_KVM_PPC_BOOK3S_PREPARE, CPUHP_ZCOMP_PREPARE, + CPUHP_RANDOM_PREPARE, CPUHP_TIMERS_PREPARE, CPUHP_MIPS_SOC_PREPARE, CPUHP_BP_PREPARE_DYN, diff --git a/include/linux/random.h b/include/linux/random.h index d7354de9351e..41da6628e838 100644 --- a/include/linux/random.h +++ b/include/linux/random.h @@ -35,6 +35,10 @@ extern void add_interrupt_randomness(int irq) __latent_e= ntropy; extern void add_hwgenerator_randomness(const void *buffer, size_t count, size_t entropy); =20 +#ifdef CONFIG_SMP +extern int random_prepare_cpu(unsigned int cpu); +#endif + extern void get_random_bytes(void *buf, size_t nbytes); extern int wait_for_random_bytes(void); extern int __init rand_initialize(void); diff --git a/kernel/cpu.c b/kernel/cpu.c index 407a2568f35e..8da392a6dc80 100644 --- a/kernel/cpu.c +++ b/kernel/cpu.c @@ -34,6 +34,7 @@ #include #include #include +#include =20 #include #define CREATE_TRACE_POINTS @@ -1689,6 +1690,11 @@ static struct cpuhp_step cpuhp_hp_states[] =3D { .startup.single =3D rcutree_prepare_cpu, .teardown.single =3D rcutree_dead_cpu, }, + [CPUHP_RANDOM_PREPARE] =3D { + .name =3D "random:prepare", + .startup.single =3D random_prepare_cpu, + .teardown.single =3D NULL, + }, /* * On the tear-down path, timers_dead_cpu() must be invoked * before blk_mq_queue_reinit_notify() from notify_dead(), --=20 2.35.0 From nobody Sun Jun 28 02:48:46 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id EDC3AC433F5 for ; Tue, 15 Feb 2022 21:23:16 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S243391AbiBOVXZ (ORCPT ); Tue, 15 Feb 2022 16:23:25 -0500 Received: from mxb-00190b01.gslb.pphosted.com ([23.128.96.19]:45758 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S239244AbiBOVXV (ORCPT ); Tue, 15 Feb 2022 16:23:21 -0500 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id EE7ECDB84D for ; Tue, 15 Feb 2022 13:23:10 -0800 (PST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 93E9B61917 for ; Tue, 15 Feb 2022 21:23:10 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 6D1CEC340EB; Tue, 15 Feb 2022 21:23:09 +0000 (UTC) Authentication-Results: smtp.kernel.org; dkim=pass (1024-bit key) header.d=zx2c4.com header.i=@zx2c4.com header.b="m8HIgyBa" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=zx2c4.com; s=20210105; t=1644960188; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=BBa9wueJoGb7LxuMzbvrV62ilts8LkBCNQfxIqDAz2s=; b=m8HIgyBalfmKrwyVHWHZVY/qWQW9jNtjaj4jJY43LvVEakS6gV2vbr84BHwcM70hQQMTpp DJvLkiU5z8veBKfa1a8kkLSxoEtGtjMLHZY2BtYaN+AYjbhEFj82BRTRYfujrfWZ/I1IE/ M96uJ7Kbn27f+I1+4tikCSKPXwAMifg= Received: by mail.zx2c4.com (ZX2C4 Mail Server) with ESMTPSA id c92f5e54 (TLSv1.3:AEAD-AES256-GCM-SHA384:256:NO); Tue, 15 Feb 2022 21:23:08 +0000 (UTC) From: "Jason A. Donenfeld" To: linux-kernel@vger.kernel.org, bigeasy@linutronix.de, linux@dominikbrodowski.net, sultan@kerneltoast.com Cc: "Jason A. Donenfeld" , Theodore Ts'o Subject: [PATCH v4 2/2] random: invalidate crngs and batches in cpuhp prepare Date: Tue, 15 Feb 2022 22:22:55 +0100 Message-Id: <20220215212255.273253-3-Jason@zx2c4.com> In-Reply-To: <20220215212255.273253-1-Jason@zx2c4.com> References: <20220215212255.273253-1-Jason@zx2c4.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Now that we have a cpuhp prepare notifier, we can invalidate the keys used by the per-cpu crngs and the batches used by per-cpu batched entropy, so that if the cpus come back online, and the generation counter happens to have cycled all the way around to where it was before, it doesn't mistakenly use the old data. The chances of this happening are exceedingly rare, but since we now have the notifier setup, doing this is basically free. Cc: Sebastian Andrzej Siewior Cc: Sultan Alsawaf Cc: Dominik Brodowski Cc: Theodore Ts'o Signed-off-by: Jason A. Donenfeld --- drivers/char/random.c | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/drivers/char/random.c b/drivers/char/random.c index a3cc147406b0..41188a49d43e 100644 --- a/drivers/char/random.c +++ b/drivers/char/random.c @@ -1233,6 +1233,14 @@ int random_prepare_cpu(unsigned int cpu) * since the MIX_INFLIGHT flag will be cleared. */ per_cpu_ptr(&irq_randomness, cpu)->count =3D 0; + + /* + * We also want to invalidate per-cpu crngs and batches, so + * that we always use fresh entropy. + */ + per_cpu_ptr(&crngs, cpu)->generation =3D ULONG_MAX; + per_cpu_ptr(&batched_entropy_u32, cpu)->position =3D UINT_MAX; + per_cpu_ptr(&batched_entropy_u64, cpu)->position =3D UINT_MAX; return 0; } #endif --=20 2.35.0