From nobody Tue Apr 7 14:03:40 2026 Received: from mail-ot1-f49.google.com (mail-ot1-f49.google.com [209.85.210.49]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id CC6B11B4244 for ; Fri, 3 Apr 2026 14:24:09 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.49 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1775226251; cv=none; b=btOxEwhu4m/MgugKK8HQu9kjIuhBfjA3deNf/CdwwpeYtCPyoA6m6HYZFPdJCGwQNLyqmxoTmtpeF0s66k06SThJMFJD4bd5W0xexqRmhjhMDFi1yHwQcBRca8uFVFm6TjEDc+p1V3h9QWnIfPM/v9LAAKyTy4LeYpW/lJlIHy0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1775226251; c=relaxed/simple; bh=JEbOGTbCGqklP/cu1D/wFazp39cERX3Y6A2nBWEIH1w=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=ZpW0Ef1kolgNaeCt93EDKTxj3iQbiGfj3an45fDxh+5Fp/vKcAnFV3X/xB4MfSkd0Lr3WO/byTYJYprINzDchXS4E0nX9pouFgJvbPvzY/EAOeCkuclrZsHb3JdDAtzay5QOZSlIA5OrlEatPLNfKhqPaL8BbHPMp4evzzyt76A= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=Q378Xafm; arc=none smtp.client-ip=209.85.210.49 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="Q378Xafm" Received: by mail-ot1-f49.google.com with SMTP id 46e09a7af769-7d750eeaec3so878685a34.0 for ; Fri, 03 Apr 2026 07:24:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1775226249; x=1775831049; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=MSvAMBdnCWX1J3zGwiw7xvxdZPUTObrMMVmHYUBfq3Q=; b=Q378XafmGn7OWO0vn2QVZgBFVnHmcXHoWRQFt86EobZyIWP0X5ES8LhmKz3Cy6Zfo2 9iQuMXbtBjC41XuYJ+Z/1q39Ttz+3Uw1zPvpXxKfoZfRHxMPKHS0UggPwieWgR0UOrMQ 0T49W5nuDNAQ4XYI8GTrUJ1rLAYBLaE17kzrvOlQJ05OWTBM51+lEtzQw6LqYBvbKFBT h/yRvipuF1j1nVgySkFUICnOFWab+rrG9N5AGvba+aLC7eGhiSXVAihUCY5oyIvWU4Fv zufywE0z6xCuqbfg3YykucRCmFF/ZmBVTZjzIkL10W9rhg6YoHawyo4wovt0AbH5s7W5 AGiQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1775226249; x=1775831049; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=MSvAMBdnCWX1J3zGwiw7xvxdZPUTObrMMVmHYUBfq3Q=; b=psg2j5XvVZQjAoJskjw3LuHn5LgMA2EF6KVMONCkmssDDJcc2QjjiC+xqAa534sG+9 A8LF7pW0B+vR4D2K/s+6HGa8GRCAvbtu/ZOsD8y0Pb/ZFmyHXmS916c+XCDJzqO1w3Gi adsyLxVmsDs+/vB+MOY7Zs59HWdB9DyOQQJdZvLSr/FKQEgYvKFmbMg2JqURa0U3HOMV QFGQlWZuL9CRrHRn+P6bLQhhvQ/W1PJJaC+9TmJO7Tsfk2HP1wCvJd5ITscKgGxpv8RA k0VxnjVYqsUgIOOrrXriC4KZdh9Vf9rpT80eBFGHyyarJPfhY4+kP8jURXGhh67g8NYt uLkQ== X-Forwarded-Encrypted: i=1; AJvYcCWBS/kPBXKx2EFeMkj9rdpSRWIaMeEUl3r6puRCSdLKVKZffk5Omub5Aa6gmPvCQlNJ+eomYBkQIr6DLZc=@vger.kernel.org X-Gm-Message-State: AOJu0Yw0lydgVdPdZ2qTDStZyDMjKtUW2t3Ubry5pWtXqjYg+UXuJcEJ V0ztye6TtHZCdkmbsGb2z6jCXQ6qMAtdW0/zqJg8V+Kud0Vh6O6uPjy0 X-Gm-Gg: ATEYQzznYUmyxC7uU3P6pOALOScp7ojl5kC87aPIGDhQFi0BE14IQUuDYmhNdCeIfbB cz8RNQGVkaWLYV5ySnANQpeixwL9ufyB+VTtCcbq4sbVStuCO+mffFwgD5paTztZA6GWDaiRqmp nGU8r3eXMaK0l4L4Vqpg2u4oDyWzBfXc1uV0B348815HD/mOwAis21H+COfBSTNDB7GsEk4uqQV KIQiF8vArGQMPEv/Lp8HuCtjhfEWgZVnPfQHQ4WxtBlxa3PG8kSAQnH3mYnL9G7KyUXAGE4kOGX xwhltz8nGJflNHZLAkUGW+IuKRnYgFRlooPYISjnB6vYDgB1/TSWUavDBAe5PN/JtHxsOsGP9bd HMZvnRMnom4f9BrmKtNCu7bEmoZuBhbg2ytnTfwG0LI+V5F/1jol9NfUWv8s1l4fjRTeeG6uKJ6 AGlZoQLib1F8nwj/kcqp06AmroGCtge4Jp3NLl1QFcSNNyNC6uhGehUvu5H1HpEW0EKsQHK7Rv4 ZWvgagnDvaFlg== X-Received: by 2002:a05:6830:6084:b0:7d7:cb13:3fe9 with SMTP id 46e09a7af769-7dbb7112eaamr2165259a34.19.1775226248589; Fri, 03 Apr 2026 07:24:08 -0700 (PDT) Received: from frodo.raven-morpho.ts.net (c-98-38-17-99.hsd1.co.comcast.net. [98.38.17.99]) by smtp.googlemail.com with ESMTPSA id 46e09a7af769-7dbb7c3a0c8sm1806671a34.12.2026.04.03.07.24.07 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 03 Apr 2026 07:24:07 -0700 (PDT) From: Jim Cromie To: peterz@infradead.org, gregkh@linuxfoundation.org Cc: jpoimboe@kernel.org, jbaron@akamai.com, aliceryhl@google.com, rostedt@goodmis.org, ardb@kernel.org, x86@kernel.org, linux-kernel@vger.kernel.org, Jim Cromie Subject: [PATCH 1/5] jump_label: add queueing API for batched static key updates Date: Fri, 3 Apr 2026 08:23:57 -0600 Message-ID: <20260403142401.1387033-2-jim.cromie@gmail.com> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260403142401.1387033-1-jim.cromie@gmail.com> References: <20260403142401.1387033-1-jim.cromie@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Introduce a new queueing API for static keys to allow batching of instruction updates. This is provided for subsystems like dynamic-debug, which may toggle thousands of static keys in a single operation. Currently, this suffers from (N)$ overhead due to repeated machine-wide `stop_machine` or `text_poke_sync` synchronizations. The new API includes: - static_key_enable_queued(key) - static_key_disable_queued(key) - static_key_apply_queued() (the global synchronization barrier) - Corresponding static_branch_*_queued(x) macros. The _queued functionality is achieved by adding a 'bool apply' parameter to 3 existing midlayer static functions, and using it to decide whether or not to call arch_jump_label_transform_apply(). By not explicitly calling it, we allow the transform queue to fill 1st. This amortizes the cost of the IPI sync to 1/N (256 on x86). 1. __jump_label_update(struct static_key *key, bool apply) this implements the if (apply) arch_jump_label_transform_apply(); 2. __jump_label_mod_update(struct static_key *key, bool apply) extends _queued feature to modules' static-keys 3. rename jump_label_update(struct static_key *key) to __jump_label_update_key(struct static_key *key, bool apply) and wrap it with 1-liners: jump_label_update() & jump_label_update_queued() On architectures supporting HAVE_JUMP_LABEL_BATCH, the _queued variants lazily fill the architecture's patch queue. It is safe to mix queued and immediate updates; any non-queued static_key_enable/disable call will effectively "flush" any previously queued changes as a side effect. Or just call static_key_apply_queued(). On architectures that do not need a patch queue (e.g., those with atomic patching) or that do not support batching, the API transparently falls back to immediate patching. Signed-off-by: Jim Cromie --- include/linux/jump_label.h | 18 +++++++ kernel/jump_label.c | 104 +++++++++++++++++++++++++++++-------- 2 files changed, 100 insertions(+), 22 deletions(-) diff --git a/include/linux/jump_label.h b/include/linux/jump_label.h index fdb79dd1ebd8..283ee9360026 100644 --- a/include/linux/jump_label.h +++ b/include/linux/jump_label.h @@ -234,6 +234,9 @@ extern void static_key_slow_dec_cpuslocked(struct stati= c_key *key); extern int static_key_count(struct static_key *key); extern void static_key_enable(struct static_key *key); extern void static_key_disable(struct static_key *key); +extern void static_key_enable_queued(struct static_key *key); +extern void static_key_disable_queued(struct static_key *key); +extern void static_key_apply_queued(void); extern void static_key_enable_cpuslocked(struct static_key *key); extern void static_key_disable_cpuslocked(struct static_key *key); extern enum jump_label_type jump_label_init_type(struct jump_entry *entry); @@ -340,6 +343,18 @@ static inline void static_key_disable(struct static_ke= y *key) atomic_set(&key->enabled, 0); } =20 +static inline void static_key_enable_queued(struct static_key *key) +{ + static_key_enable(key); +} + +static inline void static_key_disable_queued(struct static_key *key) +{ + static_key_disable(key); +} + +static inline void static_key_apply_queued(void) {} + #define static_key_enable_cpuslocked(k) static_key_enable((k)) #define static_key_disable_cpuslocked(k) static_key_disable((k)) =20 @@ -535,6 +550,9 @@ extern bool ____wrong_branch_error(void); =20 #define static_branch_enable(x) static_key_enable(&(x)->key) #define static_branch_disable(x) static_key_disable(&(x)->key) +#define static_branch_enable_queued(x) static_key_enable_queued(&(x)->key) +#define static_branch_disable_queued(x) static_key_disable_queued(&(x)->k= ey) +#define static_branch_apply_queued() static_key_apply_queued() #define static_branch_enable_cpuslocked(x) static_key_enable_cpuslocked(&(= x)->key) #define static_branch_disable_cpuslocked(x) static_key_disable_cpuslocked(= &(x)->key) =20 diff --git a/kernel/jump_label.c b/kernel/jump_label.c index 7cb19e601426..9826aefd77e5 100644 --- a/kernel/jump_label.c +++ b/kernel/jump_label.c @@ -91,6 +91,7 @@ jump_label_sort_entries(struct jump_entry *start, struct = jump_entry *stop) } =20 static void jump_label_update(struct static_key *key); +static void jump_label_update_queued(struct static_key *key); =20 /* * There are similar definitions for the !CONFIG_JUMP_LABEL case in jump_l= abel.h. @@ -250,10 +251,57 @@ void static_key_disable(struct static_key *key) } EXPORT_SYMBOL_GPL(static_key_disable); =20 +void static_key_enable_queued(struct static_key *key) +{ + STATIC_KEY_CHECK_USE(key); + + if (atomic_read(&key->enabled) > 0) { + /* + * already enabled, don't act. + * warn if static-key-inc was used. + */ + WARN_ON_ONCE(atomic_read(&key->enabled) !=3D 1); + return; + } + + scoped_guard(jump_label_lock) { + if (atomic_read(&key->enabled) =3D=3D 0) { + /* + * set transitional value, update, + * then set stable enabled state + */ + atomic_set(&key->enabled, -1); + jump_label_update_queued(key); + atomic_set_release(&key->enabled, 1); + } + } +} + +void static_key_disable_queued(struct static_key *key) +{ + STATIC_KEY_CHECK_USE(key); + + if (atomic_read(&key->enabled) !=3D 1) { + /* + * not simply enabled; is disabled, inc-enabled, or + * in transition. don't act. warn if inc-enabled. + */ + WARN_ON_ONCE(atomic_read(&key->enabled) !=3D 0); + return; + } + + scoped_guard(jump_label_lock) { + /* + * update if simply enabled. dont act if in transition. + */ + if (atomic_cmpxchg(&key->enabled, 1, 0) =3D=3D 1) + jump_label_update_queued(key); + } +} + static bool static_key_dec_not_one(struct static_key *key) { int v; - /* * Go into the slow path if key::enabled is less than or equal than * one. One is valid to shut down the key, anything less than one @@ -488,28 +536,16 @@ static bool jump_label_can_update(struct jump_entry *= entry, bool init) return true; } =20 -#ifndef HAVE_JUMP_LABEL_BATCH static void __jump_label_update(struct static_key *key, struct jump_entry *entry, struct jump_entry *stop, - bool init) + bool init, bool apply) { for (; (entry < stop) && (jump_entry_key(entry) =3D=3D key); entry++) { - if (jump_label_can_update(entry, init)) - arch_jump_label_transform(entry, jump_label_type(entry)); - } -} -#else -static void __jump_label_update(struct static_key *key, - struct jump_entry *entry, - struct jump_entry *stop, - bool init) -{ - for (; (entry < stop) && (jump_entry_key(entry) =3D=3D key); entry++) { - if (!jump_label_can_update(entry, init)) continue; =20 +#ifdef HAVE_JUMP_LABEL_BATCH if (!arch_jump_label_transform_queue(entry, jump_label_type(entry))) { /* * Queue is full: Apply the current queue and try again. @@ -517,10 +553,24 @@ static void __jump_label_update(struct static_key *ke= y, arch_jump_label_transform_apply(); BUG_ON(!arch_jump_label_transform_queue(entry, jump_label_type(entry))); } +#else + arch_jump_label_transform(entry, jump_label_type(entry)); +#endif } - arch_jump_label_transform_apply(); +#ifdef HAVE_JUMP_LABEL_BATCH + if (apply) + arch_jump_label_transform_apply(); +#endif } + +void static_key_apply_queued(void) +{ +#ifdef HAVE_JUMP_LABEL_BATCH + jump_label_lock(); + arch_jump_label_transform_apply(); + jump_label_unlock(); #endif +} =20 void __init jump_label_init(void) { @@ -671,7 +721,7 @@ static int __jump_label_mod_text_reserved(void *start, = void *end) return ret; } =20 -static void __jump_label_mod_update(struct static_key *key) +static void __jump_label_mod_update(struct static_key *key, bool apply) { struct static_key_mod *mod; =20 @@ -692,7 +742,7 @@ static void __jump_label_mod_update(struct static_key *= key) else stop =3D m->jump_entries + m->num_jump_entries; __jump_label_update(key, mod->entries, stop, - m && m->state =3D=3D MODULE_STATE_COMING); + m && m->state =3D=3D MODULE_STATE_COMING, apply); } } =20 @@ -762,7 +812,7 @@ static int jump_label_add_module(struct module *mod) /* Only update if we've changed from our initial state */ do_poke: if (jump_label_type(iter) !=3D jump_label_init_type(iter)) - __jump_label_update(key, iter, iter_stop, true); + __jump_label_update(key, iter, iter_stop, true, true); } =20 return 0; @@ -892,7 +942,7 @@ int jump_label_text_reserved(void *start, void *end) return ret; } =20 -static void jump_label_update(struct static_key *key) +static void jump_label_update_key(struct static_key *key, bool apply) { struct jump_entry *stop =3D __stop___jump_table; bool init =3D system_state < SYSTEM_RUNNING; @@ -901,7 +951,7 @@ static void jump_label_update(struct static_key *key) struct module *mod; =20 if (static_key_linked(key)) { - __jump_label_mod_update(key); + __jump_label_mod_update(key, apply); return; } =20 @@ -916,7 +966,17 @@ static void jump_label_update(struct static_key *key) entry =3D static_key_entries(key); /* if there are no users, entry can be NULL */ if (entry) - __jump_label_update(key, entry, stop, init); + __jump_label_update(key, entry, stop, init, apply); +} + +static void jump_label_update(struct static_key *key) +{ + jump_label_update_key(key, true); +} + +static void jump_label_update_queued(struct static_key *key) +{ + jump_label_update_key(key, false); } =20 #ifdef CONFIG_STATIC_KEYS_SELFTEST --=20 2.53.0