From nobody Sun Feb 8 08:27:32 2026 Received: from mail-wr1-f74.google.com (mail-wr1-f74.google.com [209.85.221.74]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9AF423164D6 for ; Wed, 17 Dec 2025 11:21:44 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.221.74 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1765970507; cv=none; b=tFyrUPvnJzzqI/E9wkG61MAnbTiuBn8AncuIF1pAUUwo2HxQIw/FzWHOASs06ay/8vwZVBsUxn3gwfHn7en8e9vgLkyC9jmDiZEw/fl0frnyMSz3mXy5wBcpljXbhCImtsMBOnNnfYZJ9m/+HqOJAKLtA93+b1iVtnHI1n++RaA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1765970507; c=relaxed/simple; bh=LNHhM2s0FNoFuYALUvYHahtwRolnny8egm21AsJ1tag=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=kVhojsIGLU5nJnfpVHww76OFfT1JbyDCY1+DS9kWuxgO0xo4+G6ZU47jSUqWiFatCQ9UiM7fItnaNWcV/iZOIOBXHUIu19rSP+CIT/lmKsNUrk7k4xgikSjKF/pSzRkfmzVmrLfe1n9+91k4WqO0Wx4WRcXsnwJ0ebiwZ8d3VZA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--lrizzo.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=DJpyjTcV; arc=none smtp.client-ip=209.85.221.74 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--lrizzo.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="DJpyjTcV" Received: by mail-wr1-f74.google.com with SMTP id ffacd0b85a97d-430ffa9fd7fso1654594f8f.0 for ; Wed, 17 Dec 2025 03:21:44 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1765970503; x=1766575303; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=ZmCBnSY5i2PWNS+mgQi0gxQxxA/UKBXwwvGRJL++VOE=; b=DJpyjTcVdv5CEB5VqjJim6EFKmIkHOJjMHegnNA4MPHzWMdVfeGE2vtwBfV9sSkhh1 6Kv/ZhlRQ53gUsqNZ4TCoeur1FES5OqoG3hMsjHnv1daS/PQf3+GfTU8kM5tUfllk+4u A/cUxO7wr5uw1tpp+RYzt60LsRD4zVNd61qNiVkw2nWutHtnXyufaS4qmPwJ7EjXh26Q x9MdczYupgl2d7+Nn+mLNTIGMJGnWH1D61IzKxQOMXXooLfwhPnw8ZRmYdhN5V00H96S asx6xVPVarzFhX/LvoCA8h2VLZCmiHJ5OsC5EweLgkigR4tDhf23dR6SWZNO5icv70SJ HVwg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1765970503; x=1766575303; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=ZmCBnSY5i2PWNS+mgQi0gxQxxA/UKBXwwvGRJL++VOE=; b=EGKJUK+iMLHcAyKBlTDP6kEHin4Q827o9dkQTSFAo9EqQyHZQMdA0OSEL6l3lfSEDp aWtyUcnacMrZ6B2yxjnnRULttso9iT8BEIVRPanXGPa/b0HiKHxTUAHzgkPKFJkoT+Mq bOwDu+dLhOWSjleXW2EBGpv6WELw2yMBpLIOW7+jY3ADcIh8L+a9xd1HSDsY9HN0F1x9 aoq8g87hJxU+9iLXV0SaE8+tV2OjZM7NYZCXZwoF/eOdlPZ/yeijXvOd/jJdbzgBun0b UyTRUrAlUAx/qP/lhu23H5sH5ai6skxEQzWBM06qrPkyA7DCy49QN3lcIENa0Oe84XTD ukjg== X-Gm-Message-State: AOJu0Yy4htYoUYsiC1hU4AsBwCom6tyqrHf/C1pBuA/EuMSmSrZfG+oh ExVju3Xs4+zv1BH2C7Ylw2cUqpz6288Q8SrYUdeoupgrN6PKf0MzwnSiYigJYxoTZaJR7bvkI9V +IKaEJw== X-Google-Smtp-Source: AGHT+IFeDH71QaXLwk/WRNgz5Fe1tzmbIXg7oPH7+C+a/ggvJu3bPkrJ3dJKywl1FzvNPOpIdCaQ1G3J5pg= X-Received: from wmby5.prod.google.com ([2002:a05:600c:c045:b0:477:a0a1:eae8]) (user=lrizzo job=prod-delivery.src-stubby-dispatcher) by 2002:a05:600c:699a:b0:479:2651:3f9c with SMTP id 5b1f17b1804b1-47bda6e58d4mr40141695e9.14.1765970502991; Wed, 17 Dec 2025 03:21:42 -0800 (PST) Date: Wed, 17 Dec 2025 11:21:26 +0000 In-Reply-To: <20251217112128.1401896-1-lrizzo@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20251217112128.1401896-1-lrizzo@google.com> X-Mailer: git-send-email 2.52.0.305.g3fc767764a-goog Message-ID: <20251217112128.1401896-2-lrizzo@google.com> Subject: [PATCH-v3 1/3] genirq: Fixed Global Software Interrupt Moderation (GSIM) From: Luigi Rizzo To: Thomas Gleixner , Marc Zyngier , Luigi Rizzo , Paolo Abeni , Andrew Morton , Sean Christopherson , Jacob Pan Cc: linux-kernel@vger.kernel.org, linux-arch@vger.kernel.org, Bjorn Helgaas , Willem de Bruijn , Luigi Rizzo Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Some platforms show reduced I/O performance when the total device interrupt rate across the entire platform becomes too high. Interrupt moderation can be used to rate-limit individual interrupts, and as a consequence also the total rate. Not all devices implement moderation in hardware, or they may have impractically coarse granularity (e.g. NVME specifies 100us). GSIM implements interrupt moderation in software, allowing control of interrupt rates even without hardware support. It als provides a building block for more sophisticated, adaptive mechanisms. Moderation is enabled/disabled per interrupt source with echo 1 > /proc/irq/NN/soft_moderation # use 0 to disable In this version the delay is fixed, and equal for all interrupts with moderation enabled. It is set at boot via module parameter irq_moderation.delay_us=3DXX # range 0-500, 0 means disabled and can be updated at runtime with echo delay_us=3DXX > /proc/irq/soft_moderation Per CPU statistics on how often moderation is used are available via cat /proc/irq/soft_moderation GSIM is limited to edge interrupts using handle_edge_irq() or handle_fasteoi_irq(). It has been tested on Intel (including with intremap=3Dposted_msi), AMD, ARM with NIC, NVME and VFIO. PERFORMANCE BENEFITS: Below are some experimental results under high load (before/after rates are measured with conventional moderation or with this change): - 100Gbps NIC, 32 queues: rx goes from 50 Gbps to 92.8 Gbps (line rate). - 200Gbps NIC, 10 VMs (total 160 queues): rx goes from 30 Gbps to 190 Gbps = (line rate). - 12 SSD, 96 queues: 4K random read goes from 6M to 20.5M IOPS (device max). In all cases, latency up to p95 is unaffected at low/moderate load, even if compared with no moderation at all On Intel, intremap=3Dposted_msi requires the fix in https://lore.kernel.org/lkml/20251125214631.044440658@linutronix.de/ Change-Id: I7c321ef5d295cad6ab4a87f2fa79ce8c85c7bf9a Signed-off-by: Luigi Rizzo --- include/linux/irqdesc.h | 23 ++ kernel/irq/Kconfig | 12 + kernel/irq/Makefile | 1 + kernel/irq/chip.c | 16 +- kernel/irq/irq_moderation.c | 414 +++++++++++++++++++++++++++++++ kernel/irq/irq_moderation.h | 93 +++++++ kernel/irq/irq_moderation_hook.h | 104 ++++++++ kernel/irq/irqdesc.c | 1 + kernel/irq/proc.c | 3 + 9 files changed, 666 insertions(+), 1 deletion(-) create mode 100644 kernel/irq/irq_moderation.c create mode 100644 kernel/irq/irq_moderation.h create mode 100644 kernel/irq/irq_moderation_hook.h diff --git a/include/linux/irqdesc.h b/include/linux/irqdesc.h index 17902861de76d..d42ce57ff5406 100644 --- a/include/linux/irqdesc.h +++ b/include/linux/irqdesc.h @@ -17,6 +17,27 @@ struct irq_desc; struct irq_domain; struct pt_regs; =20 +#ifdef CONFIG_IRQ_SOFT_MODERATION + +/** + * struct irq_desc_mod - interrupt moderation information + * @ms_node: per-CPU list of moderated irq_desc + * @enable: enable moderation on this descriptor + */ +struct irq_desc_mod { + struct list_head ms_node; + bool enable; +}; + +void irq_moderation_init_fields(struct irq_desc_mod *mod); + +#else + +struct irq_desc_mod {}; +static inline void irq_moderation_init_fields(struct irq_desc_mod *mod) {} + +#endif + /** * struct irqstat - interrupt statistics * @cnt: real-time interrupt count @@ -46,6 +67,7 @@ struct irqstat { * @threads_handled: stats field for deferred spurious detection of thread= ed handlers * @threads_handled_last: comparator field for deferred spurious detection= of threaded handlers * @lock: locking for SMP + * @mod: moderation state * @affinity_hint: hint to user space for preferred irq affinity * @affinity_notify: context for notification of affinity changes * @pending_mask: pending rebalanced interrupts @@ -81,6 +103,7 @@ struct irq_desc { atomic_t threads_handled; int threads_handled_last; raw_spinlock_t lock; + struct irq_desc_mod mod; struct cpumask *percpu_enabled; #ifdef CONFIG_SMP const struct cpumask *affinity_hint; diff --git a/kernel/irq/Kconfig b/kernel/irq/Kconfig index 1b4254d19a73e..c258973b63459 100644 --- a/kernel/irq/Kconfig +++ b/kernel/irq/Kconfig @@ -155,6 +155,18 @@ config IRQ_KUNIT_TEST =20 endmenu =20 +# Support platform wide software interrupt moderation +config IRQ_SOFT_MODERATION + bool "Enable platform wide software interrupt moderation" + depends on PROC_FS=3Dy + help + Enable platform wide software interrupt moderation. + Uses a local timer to delay interrupts in configurable ways + and depending on various global system load indicators + and targets. + + If you don't know what to do here, say N. + config GENERIC_IRQ_MULTI_HANDLER bool help diff --git a/kernel/irq/Makefile b/kernel/irq/Makefile index 6ab3a40556670..c06da43d644f2 100644 --- a/kernel/irq/Makefile +++ b/kernel/irq/Makefile @@ -9,6 +9,7 @@ obj-$(CONFIG_GENERIC_IRQ_CHIP) +=3D generic-chip.o obj-$(CONFIG_GENERIC_IRQ_PROBE) +=3D autoprobe.o obj-$(CONFIG_IRQ_DOMAIN) +=3D irqdomain.o obj-$(CONFIG_IRQ_SIM) +=3D irq_sim.o +obj-$(CONFIG_IRQ_SOFT_MODERATION) +=3D irq_moderation.o obj-$(CONFIG_PROC_FS) +=3D proc.o obj-$(CONFIG_GENERIC_PENDING_IRQ) +=3D migration.o obj-$(CONFIG_GENERIC_IRQ_MIGRATION) +=3D cpuhotplug.o diff --git a/kernel/irq/chip.c b/kernel/irq/chip.c index 678f094d261a7..56faa5260b858 100644 --- a/kernel/irq/chip.c +++ b/kernel/irq/chip.c @@ -18,6 +18,7 @@ #include =20 #include "internals.h" +#include "irq_moderation_hook.h" =20 static irqreturn_t bad_chained_irq(int irq, void *dev_id) { @@ -739,6 +740,13 @@ void handle_fasteoi_irq(struct irq_desc *desc) =20 guard(raw_spinlock)(&desc->lock); =20 + if (irq_moderation_skip_moderated(desc)) { + mask_irq(desc); + desc->istate |=3D IRQS_PENDING; + cond_eoi_irq(chip, &desc->irq_data); + return; + } + /* * When an affinity change races with IRQ handling, the next interrupt * can arrive on the new CPU before the original CPU has completed @@ -765,6 +773,9 @@ void handle_fasteoi_irq(struct irq_desc *desc) =20 cond_unmask_eoi_irq(desc, chip); =20 + if (irq_moderation_hook(desc)) + return; + /* * When the race described above happens this will resend the interrupt. */ @@ -824,7 +835,7 @@ void handle_edge_irq(struct irq_desc *desc) { guard(raw_spinlock)(&desc->lock); =20 - if (!irq_can_handle(desc)) { + if (irq_moderation_skip_moderated(desc) || !irq_can_handle(desc)) { desc->istate |=3D IRQS_PENDING; mask_ack_irq(desc); return; @@ -854,6 +865,9 @@ void handle_edge_irq(struct irq_desc *desc) =20 handle_irq_event(desc); =20 + if (irq_moderation_hook(desc)) + break; + } while ((desc->istate & IRQS_PENDING) && !irqd_irq_disabled(&desc->irq_d= ata)); } EXPORT_SYMBOL(handle_edge_irq); diff --git a/kernel/irq/irq_moderation.c b/kernel/irq/irq_moderation.c new file mode 100644 index 0000000000000..2a7401bc51da2 --- /dev/null +++ b/kernel/irq/irq_moderation.c @@ -0,0 +1,414 @@ +// SPDX-License-Identifier: GPL-2.0 OR BSD-2-Clause + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include "internals.h" +#include "irq_moderation.h" + +/* + * Management code for Global Software Interrupt Moderation (GSIM) + * + * Some platforms show reduced I/O performance when the total device inter= rupt + * rate across the entire platform becomes too high. To address the proble= m, + * GSIM uses a hook after running the handler to implement software interr= upt + * moderation with programmable delay. + * + * Configuration is controlled at boot time via module parameters + * + * irq_moderation.${NAME}=3D${VALUE} + * + * or at runtime via /proc/irq/soft_moderation + * + * echo "${NAME}=3D${VALUE}" > /proc/irq/soft_moderation + * + * Supported parameters: + * + * delay_us (default 0, suggested 100, 0 off, range 0-500) + * Maximum moderation delay. A reasonable range is 20-100. Higher va= lues + * can be useful if the hardirq handler has long runtimes. + * + * Moderation can be enabled/disabled dynamically for individual interrupt= s with + * + * echo 1 > /proc/irq/NN/soft_moderation # use 0 to disable + * + * =3D=3D=3D MONITORING =3D=3D=3D + * + * cat /proc/irq/soft_moderation shows configuration and per-CPU/global st= atistics. + * + * =3D=3D=3D ARCHITECTURE =3D=3D=3D + * + * INTERRUPT HANDLING: + * - irq_moderation_hook() runs under desc->lock right after the interrupt= handler. + * If the interrupt must be moderated, it sets IRQD_IRQ_INPROGRESS, calls + * __disable_irq() adds the irq_desc to a per-CPU list of moderated inte= rrupts, + * and starts a moderation timer if not yet active. + * - desc->handler is modified so that when called on a moderated irq_desc= it + * calls mask_irq(), sets IRQS_PENDING and returns immediately. + * - the timer callback drains the moderation list: on each irq_desc it ac= quires + * desc->lock, and if desc->action !=3D NULL calls __enable_irq(), possi= bly calling + * the handler if IRQS_PENDING is set. + * + * INTERRUPT TEARDOWN + * is protected by IRQD_IRQ_INPROGRESS and checking desc->action !=3D NULL. + * This works because free_irq() runs in two steps: + * - first clear desc->action (under lock), + * - then call synchronize_irq(), which blocks on IRQD_IRQ_INPROGRESS + * before freeing resources. + * When the moderation timer races with free_irq() we can have two cases: + * 1. timer runs before clearing desc->action. In this case __enable_irq() + * is valid and the subsequent free_irq() will complete as intended + * 2. desc->action is cleared before the timer runs. In this case synchron= ize_irq() + * will block until the timer expires (remember moderation delays are v= ery short, + * comparable to C-state exit times), __enable_irq() will not be run, + * and free_irq() will complete successfully. + * + * INTERRUPT MIGRATION + * is protected by IRQD_IRQ_INPROGRESS that prevents running the handler o= n the + * new CPU while an interrupt is moderated. + * + * HOTPLUG + * During CPU shutdown, the kernel moves timers and reassigns interrupt af= finity + * to a new CPU. The easiest way and most robust way to guarantee that pen= ding + * events are handled correctly is to use a per-CPU "moderation_on" flag a= nd + * hotplug callbacks on CPUHP_AP_ONLINE_DYN: + * - on setup, set the flag. That will allow interrupts to be moderated. + * - on shutdown, with interrupts disabled, 1. clear the flag thus prevent= ing + * more interrupts to be moderated on that CPU, 2. flush the list of mod= erated + * interrupts (as if the timer had fired), and 3. cancel the timer. + * This avoids depending with the internals of the up/down sequence. + * + * SUSPEND + * Register a PM notifier to handle PM_SUSPEND_PREPARE and PM_POST_RESTORE= as + * hotplug shutdown and setup events. The hotplug callbacks are also invok= ed + * during suspend to/resume from disk. + * + * BOOT PROCESSOR + * Hotplug callbacks are not invoked for the boot processor. + * However the boot processor is the last one to go, and since there is + * no other place to run the timer callbacks, they will be run where they + * are supposed to. + */ + +/* Recommended values. */ +struct irq_mod_info irq_mod_info ____cacheline_aligned =3D { + .update_ms =3D 5, +}; + +module_param_named(delay_us, irq_mod_info.delay_us, uint, 0444); +MODULE_PARM_DESC(delay_us, "Max moderation delay us, 0 =3D moderation off,= range 0-500."); + +DEFINE_PER_CPU_ALIGNED(struct irq_mod_state, irq_mod_state); + +DEFINE_STATIC_KEY_FALSE(irq_moderation_enabled_key); + +/* Initialize moderation state, used in desc_set_defaults() */ +void irq_moderation_init_fields(struct irq_desc_mod *mod) +{ + INIT_LIST_HEAD(&mod->ms_node); + mod->enable =3D false; +} + +static int set_mode(struct irq_desc *desc, bool enable) +{ + lockdep_assert_held(&desc->lock); + if (enable) { + struct irq_data *irqd =3D &desc->irq_data; + struct irq_chip *chip =3D irqd->chip; + + /* Moderation is only supported in specific cases. */ + enable &=3D !irqd_is_level_type(irqd); + enable &=3D irqd_is_single_target(irqd); + enable &=3D !chip->irq_bus_lock && !chip->irq_bus_sync_unlock; + enable &=3D chip->irq_mask && chip->irq_unmask; + enable &=3D desc->handle_irq =3D=3D handle_edge_irq || + desc->handle_irq =3D=3D handle_fasteoi_irq; + if (!enable) + return -EOPNOTSUPP; + } + desc->mod.enable =3D enable; + return 0; +} + +#pragma clang diagnostic error "-Wformat" +/* Print statistics */ +static int moderation_show(struct seq_file *p, void *v) +{ + uint delay_us =3D irq_mod_info.delay_us; + int j; + +#define HEAD_FMT "%5s %8s %11s %11s %9s\n" +#define BODY_FMT "%5u %8u %11u %11u %9u\n" + + seq_printf(p, HEAD_FMT, + "# CPU", "delay_ns", "timer_set", "enqueue", "stray_irq"); + + for_each_possible_cpu(j) { + struct irq_mod_state *ms =3D per_cpu_ptr(&irq_mod_state, j); + + seq_printf(p, BODY_FMT, + j, ms->mod_ns, ms->timer_set, ms->enqueue, ms->stray_irq); + } + + seq_printf(p, "\n" + "enabled %s\n" + "delay_us %u\n", + str_yes_no(delay_us > 0), + delay_us); + + return 0; +} + +static int moderation_open(struct inode *inode, struct file *file) +{ + return single_open(file, moderation_show, pde_data(inode)); +} + +/* Helpers to set and clamp values from procfs or at init. */ +struct param_names { + const char *name; + uint *val; + uint min; + uint max; +}; + +static struct param_names param_names[] =3D { + { "delay_us", &irq_mod_info.delay_us, 0, 500 }, + /* Entries with no name cannot be set at runtime. */ + { "", &irq_mod_info.update_ms, 1, 100 }, +}; + +static void update_enable_key(void) +{ + struct static_key_false *key =3D &irq_moderation_enabled_key; + + if (irq_mod_info.delay_us !=3D 0) + static_branch_enable(key); + else + static_branch_disable(key); +} + +static ssize_t moderation_write(struct file *f, const char __user *buf, si= ze_t count, loff_t *ppos) +{ + uint i, val, copy_len, name_len; + struct param_names *n; + char cmd[40]; + + copy_len =3D min(sizeof(cmd) - 1, count); + if (count =3D=3D 0) + return -EINVAL; + if (copy_from_user(cmd, buf, copy_len)) + return -EFAULT; + cmd[copy_len] =3D '\0'; + for (i =3D 0, n =3D param_names; i < ARRAY_SIZE(param_names); i++, n++) { + name_len =3D strlen(n->name); + if (name_len < 1 || copy_len < name_len + 2 || strncmp(cmd, n->name, nam= e_len) || + cmd[name_len] !=3D '=3D') + continue; + if (kstrtouint(cmd + name_len + 1, 0, &val)) + return -EINVAL; + WRITE_ONCE(*(n->val), clamp(val, n->min, n->max)); + if (n->val =3D=3D &irq_mod_info.delay_us) + update_enable_key(); + /* Notify parameter update. */ + irq_mod_info.version++; + return count; + } + return -EINVAL; +} + +static const struct proc_ops proc_ops =3D { + .proc_open =3D moderation_open, + .proc_read =3D seq_read, + .proc_lseek =3D seq_lseek, + .proc_release =3D single_release, + .proc_write =3D moderation_write, +}; + +/* Handlers for /proc/irq/NN/soft_moderation */ +static int mode_show(struct seq_file *p, void *v) +{ + struct irq_desc *desc =3D p->private; + + if (!desc) + return -ENOENT; + + seq_puts(p, desc->mod.enable ? "on\n" : "off\n"); + return 0; +} + +static ssize_t mode_write(struct file *f, const char __user *buf, size_t c= ount, loff_t *ppos) +{ + struct irq_desc *desc =3D (struct irq_desc *)pde_data(file_inode(f)); + char cmd[40]; + bool enable; + int ret; + + if (!desc) + return -ENOENT; + if (count =3D=3D 0 || count + 1 > sizeof(cmd)) + return -EINVAL; + if (copy_from_user(cmd, buf, count)) + return -EFAULT; + cmd[count] =3D '\0'; + + ret =3D kstrtobool(cmd, &enable); + if (!ret) { + guard(raw_spinlock_irqsave)(&desc->lock); + ret =3D set_mode(desc, enable); + } + return ret ? : count; +} + +static int mode_open(struct inode *inode, struct file *file) +{ + return single_open(file, mode_show, pde_data(inode)); +} + +static const struct proc_ops mode_ops =3D { + .proc_open =3D mode_open, + .proc_read =3D seq_read, + .proc_lseek =3D seq_lseek, + .proc_release =3D single_release, + .proc_write =3D mode_write, +}; + +void irq_moderation_procfs_add(struct irq_desc *desc, umode_t umode) +{ + proc_create_data("soft_moderation", umode, desc->dir, &mode_ops, desc); +} + +void irq_moderation_procfs_remove(struct irq_desc *desc) +{ + remove_proc_entry("soft_moderation", desc->dir); +} + +/* Used on timer expiration or CPU shutdown. */ +static void drain_desc_list(struct irq_mod_state *ms) +{ + struct irq_desc *desc, *next; + + /* Remove from list and enable interrupts back. */ + list_for_each_entry_safe(desc, next, &ms->descs, mod.ms_node) { + guard(raw_spinlock)(&desc->lock); + list_del_init(&desc->mod.ms_node); + irqd_clear(&desc->irq_data, IRQD_IRQ_INPROGRESS); + /* Protect against competing free_irq(). */ + if (desc->action) + __enable_irq(desc); + } +} + +static enum hrtimer_restart timer_callback(struct hrtimer *timer) +{ + struct irq_mod_state *ms =3D this_cpu_ptr(&irq_mod_state); + + drain_desc_list(ms); + /* Prepare to accumulate next moderation delay. */ + ms->sleep_ns =3D 0; + return HRTIMER_NORESTART; +} + +/* Hotplug callback for setup. */ +static int cpu_setup_cb(uint cpu) +{ + struct irq_mod_state *ms =3D this_cpu_ptr(&irq_mod_state); + + if (ms->moderation_on) + return 0; + hrtimer_setup(&ms->timer, timer_callback, + CLOCK_MONOTONIC, HRTIMER_MODE_REL_PINNED_HARD); + INIT_LIST_HEAD(&ms->descs); + smp_mb(); + ms->moderation_on =3D true; + return 0; +} + +/* + * Hotplug callback for shutdown. + * Mark the CPU as offline for moderation, and drain the list of masked + * interrupts. Any subsequent interrupt on this CPU will not be + * moderated, but they will be on the new target. + */ +static int cpu_remove_cb(uint cpu) +{ + struct irq_mod_state *ms =3D this_cpu_ptr(&irq_mod_state); + unsigned long flags; + + local_irq_save(flags); + ms->moderation_on =3D false; + drain_desc_list(ms); + hrtimer_cancel(&ms->timer); + local_irq_restore(flags); + return 0; +} + +static void(mod_pm_prepare_cb)(void *arg) +{ + cpu_remove_cb(smp_processor_id()); +} + +static void(mod_pm_resume_cb)(void *arg) +{ + cpu_setup_cb(smp_processor_id()); +} + +static int mod_pm_notifier_cb( struct notifier_block *nb, unsigned long ev= ent, void *unused) +{ + switch (event) { + case PM_SUSPEND_PREPARE: + on_each_cpu(mod_pm_prepare_cb, NULL, 1); + break; + case PM_POST_SUSPEND: + on_each_cpu(mod_pm_resume_cb, NULL, 1); + break; + } + return NOTIFY_OK; +} + +struct notifier_block mod_nb =3D { + .notifier_call =3D mod_pm_notifier_cb, + .priority =3D 100, +}; + +static void clamp_parameter(uint *dst, uint val) +{ + struct param_names *n =3D param_names; + + for (int i =3D 0; i < ARRAY_SIZE(param_names); i++, n++) { + if (dst =3D=3D n->val) { + WRITE_ONCE(*dst, clamp(val, n->min, n->max)); + return; + } + } +} + +static int __init init_irq_moderation(void) +{ + /* Clamp all initial values to the allowed range, update version. */ + for (uint *cur =3D &irq_mod_info.delay_us; cur < irq_mod_info.pad; cur++) + clamp_parameter(cur, *cur); + irq_mod_info.version++; + + cpuhp_setup_state(CPUHP_AP_ONLINE_DYN, "irq_moderation", cpu_setup_cb, cp= u_remove_cb); + register_pm_notifier(&mod_nb); + + update_enable_key(); + + proc_create_data("irq/soft_moderation", 0644, NULL, &proc_ops, NULL); + return 0; +} + +MODULE_LICENSE("Dual BSD/GPL"); +MODULE_VERSION("1.0"); +MODULE_AUTHOR("Luigi Rizzo "); +MODULE_DESCRIPTION("Global Software Interrupt Moderation"); +module_init(init_irq_moderation); diff --git a/kernel/irq/irq_moderation.h b/kernel/irq/irq_moderation.h new file mode 100644 index 0000000000000..8fe1cf30bdf91 --- /dev/null +++ b/kernel/irq/irq_moderation.h @@ -0,0 +1,93 @@ +/* SPDX-License-Identifier: GPL-2.0 OR BSD-2-Clause */ + +#ifndef _LINUX_IRQ_MODERATION_H +#define _LINUX_IRQ_MODERATION_H + +/* Common data structures for Global Software Interrupt Moderation, GSIM */ + +#include +#include +#include +#include + +#ifdef CONFIG_IRQ_SOFT_MODERATION + +/** + * struct irq_mod_info - global configuration parameters and state + * @version: increment on writes to /proc/irq/soft_moderation + * @delay_us: maximum delay + * @update_ms: how often to update delay (epoch duration) + */ +struct irq_mod_info { + u32 version; + u32 delay_us; + u32 update_ms; + u32 pad[]; +}; + +extern struct irq_mod_info irq_mod_info; + +/** + * struct irq_mod_state - per-CPU moderation state + * + * Used on every interrupt: + * @timer: moderation timer + * @moderation_on: per-CPU enable, toggled during hotplug/suspend events + * @intr_count: interrupts in the last epoch + * @sleep_ns: accumulated time for actual delay + * @mod_ns: nominal moderation delay, recomputed every epoch + * + * Used less frequently, every few interrupts: + * @epoch_start_ns: start of current epoch + * @update_ns: update_ms from irq_mod_info, converted to ns + * @stray_irq: how many stray interrupts (almost never used) + * + * Used once per epoch per interrupt source: + * @descs: list of moderated irq_desc on this CPU + * @enqueue: how many enqueue on the list + * + * Used once per epoch: + * @version: version fetched from irq_mod_info + * + * Statistics + * @timer_set: how many timer_set calls + */ +struct irq_mod_state { + /* Used on every interrupt. */ + struct hrtimer timer; + bool moderation_on; + u32 intr_count; + u32 sleep_ns; + u32 mod_ns; + + /* Used less frequently. */ + u64 epoch_start_ns; + u32 update_ns; + u32 stray_irq; + + /* Used once per epoch per source. */ + struct list_head descs; + u32 enqueue; + + /* Used once per epoch. */ + u32 version; + + /* Statistics */ + u32 timer_set; +}; + +DECLARE_PER_CPU_ALIGNED(struct irq_mod_state, irq_mod_state); + +extern struct static_key_false irq_moderation_enabled_key; + +void irq_moderation_procfs_add(struct irq_desc *desc, umode_t umode); +void irq_moderation_procfs_remove(struct irq_desc *desc); + +#else /* CONFIG_IRQ_SOFT_MODERATION */ + +static inline void irq_moderation_procfs_add(struct irq_desc *desc, umode_= t umode) {} +static inline void irq_moderation_procfs_remove(struct irq_desc *desc) {} + +#endif /* !CONFIG_IRQ_SOFT_MODERATION */ + +#endif /* _LINUX_IRQ_MODERATION_H */ diff --git a/kernel/irq/irq_moderation_hook.h b/kernel/irq/irq_moderation_h= ook.h new file mode 100644 index 0000000000000..f9ac3005f69f4 --- /dev/null +++ b/kernel/irq/irq_moderation_hook.h @@ -0,0 +1,104 @@ +/* SPDX-License-Identifier: GPL-2.0 OR BSD-2-Clause */ + +#ifndef _LINUX_IRQ_MODERATION_HOOK_H +#define _LINUX_IRQ_MODERATION_HOOK_H + +/* Interrupt hooks for Global Software Interrupt Moderation, GSIM */ + +#include +#include +#include +#include + +#include "irq_moderation.h" + +#ifdef CONFIG_IRQ_SOFT_MODERATION + +static inline void __maybe_new_epoch(struct irq_mod_state *ms) +{ + const u64 now =3D ktime_get_ns(), epoch_ns =3D now - ms->epoch_start_ns; + const u32 slack_ns =3D 5000; + u32 version; + + /* Run approximately every update_ns, a little bit early is ok. */ + if (epoch_ns < ms->update_ns - slack_ns) + return; + ms->epoch_start_ns =3D now; + /* Fetch updated parameters. */ + while ((version =3D READ_ONCE(irq_mod_info.version)) !=3D ms->vers= ion) { + ms->update_ns =3D READ_ONCE(irq_mod_info.update_ms) * NSEC_PER_MSEC; + ms->mod_ns =3D READ_ONCE(irq_mod_info.delay_us) * NSEC_PER_USEC; + ms->version =3D version; + } +} + +static inline bool irq_moderation_needed(struct irq_desc *desc, struct irq= _mod_state *ms) +{ + if (!hrtimer_is_queued(&ms->timer)) { + const u32 min_delay_ns =3D 10000; + const u64 slack_ns =3D 2000; + + /* Accumulate sleep time, no moderation if too small. */ + ms->sleep_ns +=3D ms->mod_ns; + if (ms->sleep_ns < min_delay_ns) + return false; + /* We need moderation, start the timer. */ + ms->timer_set++; + hrtimer_start_range_ns(&ms->timer, ns_to_ktime(ms->sleep_ns), + slack_ns, HRTIMER_MODE_REL_PINNED_HARD); + } + + /* + * Add to the timer list and __disable_irq() to prevent serving subsequent + * interrupts. + */ + if (!list_empty(&desc->mod.ms_node)) { + /* Very unlikely, stray interrupt while moderated. */ + ms->stray_irq++; + } else { + ms->enqueue++; + list_add(&desc->mod.ms_node, &ms->descs); + __disable_irq(desc); + } + irqd_set(&desc->irq_data, IRQD_IRQ_INPROGRESS); + return true; +} + +/* + * Use after running the handler, with lock held. If this source should be + * moderated, disable it, add to the timer list for this CPU and return tr= ue. + */ +static inline bool irq_moderation_hook(struct irq_desc *desc) +{ + struct irq_mod_state *ms =3D this_cpu_ptr(&irq_mod_state); + + if (!static_branch_unlikely(&irq_moderation_enabled_key)) + return false; + if (!desc->mod.enable) + return false; + if (!ms->moderation_on) + return false; + + ms->intr_count++; + + /* Is this a new epoch? ktime_get_ns() is expensive, don't check too ofte= n. */ + if ((ms->intr_count & 0xf) =3D=3D 0) + __maybe_new_epoch(ms); + + return irq_moderation_needed(desc, ms); +} + +/* On entry of desc->irq_handler() tell handler to skip moderated interrup= ts. */ +static inline bool irq_moderation_skip_moderated(struct irq_desc *desc) +{ + return !list_empty(&desc->mod.ms_node); +} + +#else /* CONFIG_IRQ_SOFT_MODERATION */ + +static inline bool irq_moderation_hook(struct irq_desc *desc) { return fal= se; } +static inline bool irq_moderation_skip_moderated(struct irq_desc *desc) { = return false; } + +#endif /* !CONFIG_IRQ_SOFT_MODERATION */ + +#endif /* _LINUX_IRQ_MODERATION_HOOK_H */ diff --git a/kernel/irq/irqdesc.c b/kernel/irq/irqdesc.c index f8e4e13dbe339..bfc5802033383 100644 --- a/kernel/irq/irqdesc.c +++ b/kernel/irq/irqdesc.c @@ -134,6 +134,7 @@ static void desc_set_defaults(unsigned int irq, struct = irq_desc *desc, int node, desc->tot_count =3D 0; desc->name =3D NULL; desc->owner =3D owner; + irq_moderation_init_fields(&desc->mod); for_each_possible_cpu(cpu) *per_cpu_ptr(desc->kstat_irqs, cpu) =3D (struct irqstat) { }; desc_smp_init(desc, node, affinity); diff --git a/kernel/irq/proc.c b/kernel/irq/proc.c index 77258eafbf632..462e63af5d4d1 100644 --- a/kernel/irq/proc.c +++ b/kernel/irq/proc.c @@ -14,6 +14,7 @@ #include =20 #include "internals.h" +#include "irq_moderation.h" =20 /* * Access rules: @@ -376,6 +377,7 @@ void register_irq_proc(unsigned int irq, struct irq_des= c *desc) proc_create_single_data("effective_affinity_list", 0444, desc->dir, irq_effective_aff_list_proc_show, irqp); # endif + irq_moderation_procfs_add(desc, 0644); #endif proc_create_single_data("spurious", 0444, desc->dir, irq_spurious_proc_show, (void *)(long)irq); @@ -397,6 +399,7 @@ void unregister_irq_proc(unsigned int irq, struct irq_d= esc *desc) remove_proc_entry("effective_affinity", desc->dir); remove_proc_entry("effective_affinity_list", desc->dir); # endif + irq_moderation_procfs_remove(desc); #endif remove_proc_entry("spurious", desc->dir); =20 --=20 2.52.0.305.g3fc767764a-goog From nobody Sun Feb 8 08:27:32 2026 Received: from mail-ed1-f74.google.com (mail-ed1-f74.google.com [209.85.208.74]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9D12F3242D9 for ; Wed, 17 Dec 2025 11:21:46 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.208.74 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1765970510; cv=none; b=s8MArkvPyVINbhWET0nh5ta34fjpN+zyZ9DZoilx8Qm5C2GhCHLerXT12jkYPq725d5U8cpXj1UTGzfBoLCN+zCV9mU9uOVAHjoQyrMQ/oY/c7vBEli19Yclwa7Q4mEo/YinGebva3bFdo8HG8IAsh8Xox22RwcqQsXdonLBAOs= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1765970510; c=relaxed/simple; bh=PbT5sZfmDr+/nxVY4pWWN+h6fehfMdGovGMLtwFj30I=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=IPzAAOXH29o8V6wDLuNNx404lI7nww/Ho3LM4RbJLcwcKcLzHrImv6LVRYJIe383u60Le8JFWlJHmJ+zZJi09ek1y2Wvx0QwZ/CHBcNbv4VqNLlpDxvuCspypnJI5wfFIVorIA6LtyirjR3R8d7jHuliWEXfuljJISV54nv3NC8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--lrizzo.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=dbh7XPNI; arc=none smtp.client-ip=209.85.208.74 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--lrizzo.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="dbh7XPNI" Received: by mail-ed1-f74.google.com with SMTP id 4fb4d7f45d1cf-6409f6d6800so7790449a12.0 for ; Wed, 17 Dec 2025 03:21:46 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1765970505; x=1766575305; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=ttwkH5J3h4njr7aVXheWd9p5UAgulJqqOnMFy47kAAI=; b=dbh7XPNIBwltX4D8LD7Nv26S1V9vXWyk/d3/pxvFFpZLSL6adpVNEHDAVtYPLYGQ+u /Nczr/2HPw6SeZxGa/EyMU+mUKuKU/jYDUtfw6XwlrdYQIxfL4aWJ2itzIrnJGKBI+wp mNkljGhXkRDXfjyCA56mYSe8EqlYrpDA/pHkMH84+y2ilX3cN+HPJrLSKmClZBrkX7N6 xK8i5MavN9Et20Sb+CKmFYhXoTN6DsLcTYBCT4a4z1/x4rqlStAAhnlDB4MX++aOfeIF 1pXErjrx7kRLMHlddBE1q9NvLBCUsEhkgw3EJtjg/puHYf11mce2N4b9rrDUT7J8lkEU tNew== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1765970505; x=1766575305; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=ttwkH5J3h4njr7aVXheWd9p5UAgulJqqOnMFy47kAAI=; b=Y7aVn86bvX1FWZ5tkLslt+Xb/UDwDAJ1geHlb41S61B0V/pIo4KQsu+Y2W0XF8Z2Mu gLskTp2cf55rRFBzIcPVnqBYcTQTVypDZDffv4Ru1zVuprooOj7ym+VYPheX1ESEji7G vSKsEWlnNMz11PvOYbeUhtq0S284CeWjxpHuANs44RBcRmXbhIdknewmPf+I5sAyh+lU 8UlTHQ15HnRIT9AVYIGNKckSYDTzqXG6bc57KbFh51xr3Mx8snQLbayU+JMj6aukDh1O 55KdTe5WkouyDDfZfTvglf4i4l+/VXzSlvp3WJ54C9FV4KYGB3Hnv4xx6VsYWiq8Wb+G wT1g== X-Gm-Message-State: AOJu0YwJeHzvlwBQmmAiWKtqfq1VlYrs0E3R/I/O2MTpBTqQFx4DrX9J Uu0OEqML60+uigI1Spx0F3SNVuT2F+kNKObE5F4eOchJCUsmYS7x5KQGk0KZF/WH/jwk4sTcZGK rAvQJmA== X-Google-Smtp-Source: AGHT+IFDfOsMOee0hfiQvTUpX1IE1LLzHt3SPKjbuu0LgAk88CvrnY2u6emV5Ojh7/e0nVpY6PDW9rUud6s= X-Received: from edbco9.prod.google.com ([2002:a05:6402:c09:b0:641:a64d:bead]) (user=lrizzo job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6402:5209:b0:649:9251:1115 with SMTP id 4fb4d7f45d1cf-6499b1f369bmr15648762a12.29.1765970504895; Wed, 17 Dec 2025 03:21:44 -0800 (PST) Date: Wed, 17 Dec 2025 11:21:27 +0000 In-Reply-To: <20251217112128.1401896-1-lrizzo@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20251217112128.1401896-1-lrizzo@google.com> X-Mailer: git-send-email 2.52.0.305.g3fc767764a-goog Message-ID: <20251217112128.1401896-3-lrizzo@google.com> Subject: [PATCH-v3 2/3] genirq: Adaptive Global Software Interrupt Moderation (GSIM) From: Luigi Rizzo To: Thomas Gleixner , Marc Zyngier , Luigi Rizzo , Paolo Abeni , Andrew Morton , Sean Christopherson , Jacob Pan Cc: linux-kernel@vger.kernel.org, linux-arch@vger.kernel.org, Bjorn Helgaas , Willem de Bruijn , Luigi Rizzo Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Make GSIM adaptive by measuring total and per-CPU interrupt rates, as well as time spent in hardirq on each CPU. The metrics are compared to configurable targets, and each CPU adjusts the moderation delay up or down depending on the result, using multiplicative increase/decrease. Configuration is done at boot time via module parameters irq_moderation.${NAME}=3D${VALUE} or at runtime via /proc/irq/soft_moderation echo ${NAME}=3D${VALUE} > /proc/irq/soft_moderation Parameters are: delay_us (0 off, range 0-500) Maximum moderation delay in microseconds. target_intr_rate (0 off, range 0-50000000) Total rate above which moderation should trigger. hardirq_percent (0 off,range 0-100) Percentage of time spent in hardirq above which moderation should trigger. update_ms (range 1-100) How often the metrics should be computed and moderation delay updated. When target_intr_rate and hardirq_percent are both 0, GSIM uses a fixed moderation delay equal to delay_us. Otherwise, the delay is dynamically adjusted up or down, independently on each CPU, based on how the total and per-CPU metrics compare with the targets. Provided that delay_us suffices to bring the metrics within the target, the control loop will dynamically converge to the minimum actual moderation delay to stay within the target. Change-Id: I19b15d98e214a90fadee1c6e5bce6c8aac7a709a Signed-off-by: Luigi Rizzo --- kernel/irq/Makefile | 2 +- kernel/irq/irq_moderation.c | 94 ++++++++++++++++-- kernel/irq/irq_moderation.h | 46 ++++++++- kernel/irq/irq_moderation_hook.c | 157 +++++++++++++++++++++++++++++++ kernel/irq/irq_moderation_hook.h | 12 +-- 5 files changed, 291 insertions(+), 20 deletions(-) create mode 100644 kernel/irq/irq_moderation_hook.c diff --git a/kernel/irq/Makefile b/kernel/irq/Makefile index c06da43d644f2..d41cca6a5da76 100644 --- a/kernel/irq/Makefile +++ b/kernel/irq/Makefile @@ -9,7 +9,7 @@ obj-$(CONFIG_GENERIC_IRQ_CHIP) +=3D generic-chip.o obj-$(CONFIG_GENERIC_IRQ_PROBE) +=3D autoprobe.o obj-$(CONFIG_IRQ_DOMAIN) +=3D irqdomain.o obj-$(CONFIG_IRQ_SIM) +=3D irq_sim.o -obj-$(CONFIG_IRQ_SOFT_MODERATION) +=3D irq_moderation.o +obj-$(CONFIG_IRQ_SOFT_MODERATION) +=3D irq_moderation.o irq_moderation_hoo= k.o obj-$(CONFIG_PROC_FS) +=3D proc.o obj-$(CONFIG_GENERIC_PENDING_IRQ) +=3D migration.o obj-$(CONFIG_GENERIC_IRQ_MIGRATION) +=3D cpuhotplug.o diff --git a/kernel/irq/irq_moderation.c b/kernel/irq/irq_moderation.c index 2a7401bc51da2..be7d10404a24a 100644 --- a/kernel/irq/irq_moderation.c +++ b/kernel/irq/irq_moderation.c @@ -19,8 +19,9 @@ * * Some platforms show reduced I/O performance when the total device inter= rupt * rate across the entire platform becomes too high. To address the proble= m, - * GSIM uses a hook after running the handler to implement software interr= upt - * moderation with programmable delay. + * GSIM uses a hook after running the handler to measure global and per-CPU + * interrupt rates, compare them with configurable targets, and implements + * independent, per-CPU software moderation delays. * * Configuration is controlled at boot time via module parameters * @@ -36,6 +37,19 @@ * Maximum moderation delay. A reasonable range is 20-100. Higher va= lues * can be useful if the hardirq handler has long runtimes. * + * target_intr_rate (default 0, suggested 1000000, 0 off, range 0-500000= 00) + * The total interrupt rate above which moderation kicks in. + * Not particularly critical, a value in the 500K-1M range is usuall= y ok. + * + * hardirq_percent (default 0, suggested 70, 0 off, range 0-100) + * The hardirq percentage above which moderation kicks in. + * 50-90 is a reasonable range. + * + * FIXED MODERATION mode requires target_intr_rate=3D0, hardirq_perc= ent=3D0 + * + * update_ms (default 5, range 1-100) + * How often the load is measured and moderation delay updated. + * * Moderation can be enabled/disabled dynamically for individual interrupt= s with * * echo 1 > /proc/irq/NN/soft_moderation # use 0 to disable @@ -101,11 +115,22 @@ /* Recommended values. */ struct irq_mod_info irq_mod_info ____cacheline_aligned =3D { .update_ms =3D 5, + .increase_factor =3D MIN_SCALING_FACTOR, + .scale_cpus =3D 200, }; =20 module_param_named(delay_us, irq_mod_info.delay_us, uint, 0444); MODULE_PARM_DESC(delay_us, "Max moderation delay us, 0 =3D moderation off,= range 0-500."); =20 +module_param_named(hardirq_percent, irq_mod_info.hardirq_percent, uint, 04= 44); +MODULE_PARM_DESC(hardirq_percent, "Target max hardirq percentage, 0 off, r= ange 0-100."); + +module_param_named(target_intr_rate, irq_mod_info.target_intr_rate, uint, = 0444); +MODULE_PARM_DESC(target_intr_rate, "Target max interrupt rate, 0 off, rang= e 0-50000000."); + +module_param_named(update_ms, irq_mod_info.update_ms, uint, 0444); +MODULE_PARM_DESC(update_ms, "Update interval in milliseconds, range 1-100"= ); + DEFINE_PER_CPU_ALIGNED(struct irq_mod_state, irq_mod_state); =20 DEFINE_STATIC_KEY_FALSE(irq_moderation_enabled_key); @@ -142,27 +167,72 @@ static int set_mode(struct irq_desc *desc, bool enabl= e) /* Print statistics */ static int moderation_show(struct seq_file *p, void *v) { + ulong intr_rate =3D 0, irq_high =3D 0, my_irq_high =3D 0, hardirq_high = =3D 0; uint delay_us =3D irq_mod_info.delay_us; - int j; + u64 now =3D ktime_get_ns(); + int j, active_cpus =3D 0; =20 -#define HEAD_FMT "%5s %8s %11s %11s %9s\n" -#define BODY_FMT "%5u %8u %11u %11u %9u\n" +#define HEAD_FMT "%5s %8s %8s %4s %8s %11s %11s %11s %11s %11s %= 9s\n" +#define BODY_FMT "%5u %8u %8u %4u %8u %11u %11u %11u %11u %11u %= 9u\n" =20 seq_printf(p, HEAD_FMT, - "# CPU", "delay_ns", "timer_set", "enqueue", "stray_irq"); + "# CPU", "irq/s", "my_irq/s", "cpus", "delay_ns", + "irq_hi", "my_irq_hi", "hardirq_hi", "timer_set", + "enqueue", "stray_irq"); =20 for_each_possible_cpu(j) { struct irq_mod_state *ms =3D per_cpu_ptr(&irq_mod_state, j); + /* Watch out, epoch start_ns is 64 bits. */ + u64 epoch_start_ns =3D atomic64_read((atomic64_t *)&ms->epoch_start_ns); + s64 age_ms =3D min((now - epoch_start_ns) / NSEC_PER_MSEC, (u64)999999); + + if (age_ms < 10000) { + /* Average intr_rate over recently active CPUs. */ + active_cpus++; + intr_rate +=3D ms->intr_rate; + } else { + age_ms =3D -1; + ms->intr_rate =3D 0; + ms->my_intr_rate =3D 0; + ms->scaled_cpu_count =3D 0; + ms->mod_ns =3D 0; + } + + irq_high +=3D ms->irq_high; + my_irq_high +=3D ms->my_irq_high; + hardirq_high +=3D ms->hardirq_high; =20 seq_printf(p, BODY_FMT, - j, ms->mod_ns, ms->timer_set, ms->enqueue, ms->stray_irq); + j, ms->intr_rate, ms->my_intr_rate, + (ms->scaled_cpu_count + 128) / 256, + ms->mod_ns, ms->irq_high, ms->my_irq_high, + ms->hardirq_high, ms->timer_set, ms->enqueue, + ms->stray_irq); } =20 seq_printf(p, "\n" "enabled %s\n" - "delay_us %u\n", + "delay_us %u\n" + "target_intr_rate %u\n" + "hardirq_percent %u\n" + "update_ms %u\n" + "scale_cpus %u\n", str_yes_no(delay_us > 0), - delay_us); + delay_us, + irq_mod_info.target_intr_rate, irq_mod_info.hardirq_percent, + irq_mod_info.update_ms, irq_mod_info.scale_cpus); + + seq_printf(p, + "intr_rate %lu\n" + "irq_high %lu\n" + "my_irq_high %lu\n" + "hardirq_percent_high %lu\n" + "total_interrupts %u\n" + "total_cpus %u\n", + active_cpus ? intr_rate / active_cpus : 0, + irq_high, my_irq_high, hardirq_high, + READ_ONCE(*((u32 *)&irq_mod_info.total_intrs)), + READ_ONCE(*((u32 *)&irq_mod_info.total_cpus))); =20 return 0; } @@ -182,8 +252,12 @@ struct param_names { =20 static struct param_names param_names[] =3D { { "delay_us", &irq_mod_info.delay_us, 0, 500 }, + { "target_intr_rate", &irq_mod_info.target_intr_rate, 0, 50000000 }, + { "hardirq_percent", &irq_mod_info.hardirq_percent, 0, 100 }, + { "update_ms", &irq_mod_info.update_ms, 1, 100 }, /* Entries with no name cannot be set at runtime. */ - { "", &irq_mod_info.update_ms, 1, 100 }, + { "", &irq_mod_info.increase_factor, MIN_SCALING_FACTOR, 128 }, + { "", &irq_mod_info.scale_cpus, 50, 1000 }, }; =20 static void update_enable_key(void) diff --git a/kernel/irq/irq_moderation.h b/kernel/irq/irq_moderation.h index 8fe1cf30bdf91..bd8edb9524613 100644 --- a/kernel/irq/irq_moderation.h +++ b/kernel/irq/irq_moderation.h @@ -14,14 +14,32 @@ =20 /** * struct irq_mod_info - global configuration parameters and state + * @total_intrs: running count of total interrupts + * @total_cpus: running count of total active CPUs + * totals are updated every update_ms ("epoch") * @version: increment on writes to /proc/irq/soft_moderation - * @delay_us: maximum delay - * @update_ms: how often to update delay (epoch duration) + * @delay_us: fixed delay, or maximum for adaptive + * @target_intr_rate: target maximum interrupt rate + * @hardirq_percent: target maximum hardirq percentage + * @update_ms: how often to update delay/rate/fraction (epoch duration) + * @increase_factor: constant for exponential increase/decrease of delay + * @scale_cpus: (percent) scale factor to estimate active CPUs */ struct irq_mod_info { + /* These fields are written to by all CPUs every epoch. */ + ____cacheline_aligned + atomic_t total_intrs; + atomic_t total_cpus; + + /* These are mostly read (frequently), so use a different cacheline. */ + ____cacheline_aligned u32 version; u32 delay_us; + u32 target_intr_rate; + u32 hardirq_percent; u32 update_ms; + u32 increase_factor; + u32 scale_cpus; u32 pad[]; }; =20 @@ -48,9 +66,20 @@ extern struct irq_mod_info irq_mod_info; * * Used once per epoch: * @version: version fetched from irq_mod_info + * @delay_ns: fetched from irq_mod_info + * @epoch_ns: duration of last epoch + * @last_total_intrs: from irq_mod_info + * @last_total_cpus: from irq_mod_info + * @last_irqtime: from cpustat[CPUTIME_IRQ] * * Statistics + * @intr_rate: smoothed global interrupt rate + * @my_intr_rate: smoothed interrupt rate for this CPU * @timer_set: how many timer_set calls + * @scaled_cpu_count: smoothed CPU count (scaled) + * @irq_high: how many times global irq above threshold + * @my_irq_high: how many times local irq above threshold + * @hardirq_high: how many times local hardirq_percent above threshold */ struct irq_mod_state { /* Used on every interrupt. */ @@ -71,13 +100,26 @@ struct irq_mod_state { =20 /* Used once per epoch. */ u32 version; + u32 delay_ns; + u32 epoch_ns; + u32 last_total_intrs; + u32 last_total_cpus; + u64 last_irqtime; =20 /* Statistics */ + u32 intr_rate; + u32 my_intr_rate; u32 timer_set; + u32 timer_fire; + u32 scaled_cpu_count; + u32 irq_high; + u32 my_irq_high; + u32 hardirq_high; }; =20 DECLARE_PER_CPU_ALIGNED(struct irq_mod_state, irq_mod_state); =20 +#define MIN_SCALING_FACTOR 8u extern struct static_key_false irq_moderation_enabled_key; =20 void irq_moderation_procfs_add(struct irq_desc *desc, umode_t umode); diff --git a/kernel/irq/irq_moderation_hook.c b/kernel/irq/irq_moderation_h= ook.c new file mode 100644 index 0000000000000..f04ed25a482a0 --- /dev/null +++ b/kernel/irq/irq_moderation_hook.c @@ -0,0 +1,157 @@ +// SPDX-License-Identifier: GPL-2.0 OR BSD-2-Clause + +#include +#include +#include + +#include "internals.h" +#include "irq_moderation_hook.h" + +/* Slow path functions for interrupt moderation. */ + +static inline u32 smooth_avg(u32 old, u32 cur, u32 steps) +{ + const u32 smooth_factor =3D 64; + + steps =3D min(steps, smooth_factor - 1); + return ((smooth_factor - steps) * old + steps * cur) / smooth_factor; +} + +/* Measure and assess time spent in hardirq. */ +static inline bool hardirq_high(struct irq_mod_state *ms, u32 hardirq_perc= ent) +{ + bool above_threshold =3D false; + + if (IS_ENABLED(CONFIG_IRQ_TIME_ACCOUNTING)) { + u64 irqtime, cur =3D kcpustat_this_cpu->cpustat[CPUTIME_IRQ]; + + irqtime =3D cur - ms->last_irqtime; + ms->last_irqtime =3D cur; + + above_threshold =3D irqtime * 100 > (u64)ms->epoch_ns * hardirq_percent; + ms->hardirq_high +=3D above_threshold; + } + return above_threshold; +} + +/* Measure and assess total and per-CPU interrupt rates. */ +static inline bool irqrate_high(struct irq_mod_state *ms, u32 target_rate,= u32 steps) +{ + u32 intr_rate, my_intr_rate, delta_intrs, active_cpus, tmp; + bool my_rate_high, global_rate_high; + + my_intr_rate =3D ((u64)ms->intr_count * NSEC_PER_SEC) / ms->epoch_ns; + + /* Accumulate global counter and compute global interrupt rate. */ + tmp =3D atomic_add_return(ms->intr_count, &irq_mod_info.total_intrs); + ms->intr_count =3D 1; + delta_intrs =3D tmp - ms->last_total_intrs; + ms->last_total_intrs =3D tmp; + intr_rate =3D ((u64)delta_intrs * NSEC_PER_SEC) / ms->epoch_ns; + + /* + * Count how many CPUs handled interrupts in the last epoch, needed + * to determine the per-CPU target (target_rate / active_cpus). + * Each active CPU increments the global counter approximately every + * update_ns. Scale the value by (update_ns / ms->epoch_ns) to get the + * correct value. Also apply rounding and make sure active_cpus > 0. + */ + tmp =3D atomic_add_return(1, &irq_mod_info.total_cpus); + active_cpus =3D tmp - ms->last_total_cpus; + ms->last_total_cpus =3D tmp; + active_cpus =3D (active_cpus * ms->update_ns + ms->epoch_ns / 2) / ms->ep= och_ns; + if (active_cpus < 1) + active_cpus =3D 1; + + /* Compare with global and per-CPU targets. */ + global_rate_high =3D intr_rate > target_rate; + + /* + * Short epochs may lead to underestimate the number of active CPUs. + * Apply a scaling factor to compensate. This may make the controller + * a bit more aggressive but does not harm system throughput. + */ + my_rate_high =3D my_intr_rate * active_cpus * irq_mod_info.scale_cpus > t= arget_rate * 100; + + /* Statistics. */ + ms->intr_rate =3D smooth_avg(ms->intr_rate, intr_rate, steps); + ms->my_intr_rate =3D smooth_avg(ms->my_intr_rate, my_intr_rate, steps); + ms->scaled_cpu_count =3D smooth_avg(ms->scaled_cpu_count, active_cpus * 2= 56, steps); + ms->my_irq_high +=3D my_rate_high; + ms->irq_high +=3D global_rate_high; + + /* Moderate on this CPU only if both global and local rates are high. */ + return global_rate_high && my_rate_high; +} + +/* Periodic adjustment, called once per epoch. */ +void irq_moderation_update_epoch(struct irq_mod_state *ms) +{ + const u32 hardirq_percent =3D READ_ONCE(irq_mod_info.hardirq_percent); + const u32 target_rate =3D READ_ONCE(irq_mod_info.target_intr_rate); + const u32 min_delay_ns =3D 500; + bool above_target =3D false; + u32 version; + u32 steps; + + /* Fetch updated parameters. */ + while ((version =3D READ_ONCE(irq_mod_info.version)) !=3D ms->version) { + ms->update_ns =3D READ_ONCE(irq_mod_info.update_ms) * NSEC_PER_MSEC; + ms->mod_ns =3D READ_ONCE(irq_mod_info.delay_us) * NSEC_PER_USEC; + ms->delay_ns =3D ms->mod_ns; + ms->version =3D version; + } + + if (target_rate =3D=3D 0 && hardirq_percent =3D=3D 0) { + /* Use fixed moderation delay. */ + ms->mod_ns =3D ms->delay_ns; + ms->intr_rate =3D 0; + ms->my_intr_rate =3D 0; + ms->scaled_cpu_count =3D 0; + return; + } + + /* + * To scale values X by a factor (1 +/- 1/F) every "update_ns" we do + * X :=3D X * (1 +/- 1/F) + * If the interval is N times longer, applying the formula N times gives + * X :=3D X * ((1 +/- 1/F) ** N) + * We don't want to deal floating point or exponentials, and we cap N + * to some small value < F . This leads to an approximated formula + * X :=3D X * (1 +/- N/F) + * The variable steps below is the number N of steps. + */ + steps =3D clamp(ms->epoch_ns / ms->update_ns, 1u, MIN_SCALING_FACTOR - 1u= ); + + if (target_rate > 0 && irqrate_high(ms, target_rate, steps)) + above_target =3D true; + + if (hardirq_percent > 0 && hardirq_high(ms, hardirq_percent)) + above_target =3D true; + + /* + * Controller: adjust delay with exponential increase or decrease. + * + * Note the different constants: we increase fast (smaller factor) + * to aggressively slow down when the interrupt rate goes up, + * but decrease slowly (larger factor) because reducing the delay can + * drive up the interrupt rate and we don't want to create load spikes. + */ + if (above_target) { + const u32 increase_factor =3D READ_ONCE(irq_mod_info.increase_factor); + + /* Make sure the value is large enough for the exponential to grow. */ + if (ms->mod_ns < min_delay_ns) + ms->mod_ns =3D min_delay_ns; + ms->mod_ns +=3D ms->mod_ns * steps / increase_factor; + if (ms->mod_ns > ms->delay_ns) + ms->mod_ns =3D ms->delay_ns; + } else { + const u32 decrease_factor =3D 2 * READ_ONCE(irq_mod_info.increase_factor= ); + + ms->mod_ns -=3D ms->mod_ns * steps / decrease_factor; + /* Round down to 0 values that are too small to bother. */ + if (ms->mod_ns < min_delay_ns) + ms->mod_ns =3D 0; + } +} diff --git a/kernel/irq/irq_moderation_hook.h b/kernel/irq/irq_moderation_h= ook.h index f9ac3005f69f4..e96127923a04b 100644 --- a/kernel/irq/irq_moderation_hook.h +++ b/kernel/irq/irq_moderation_hook.h @@ -14,22 +14,20 @@ =20 #ifdef CONFIG_IRQ_SOFT_MODERATION =20 +void irq_moderation_update_epoch(struct irq_mod_state *ms); + static inline void __maybe_new_epoch(struct irq_mod_state *ms) { const u64 now =3D ktime_get_ns(), epoch_ns =3D now - ms->epoch_start_ns; const u32 slack_ns =3D 5000; - u32 version; =20 /* Run approximately every update_ns, a little bit early is ok. */ if (epoch_ns < ms->update_ns - slack_ns) return; + ms->epoch_ns =3D min(epoch_ns, (u64)U32_MAX); ms->epoch_start_ns =3D now; - /* Fetch updated parameters. */ - while ((version =3D READ_ONCE(irq_mod_info.version)) !=3D ms->vers= ion) { - ms->update_ns =3D READ_ONCE(irq_mod_info.update_ms) * NSEC_PER_MSEC; - ms->mod_ns =3D READ_ONCE(irq_mod_info.delay_us) * NSEC_PER_USEC; - ms->version =3D version; - } + /* Do the expensive processing */ + irq_moderation_update_epoch(ms); } =20 static inline bool irq_moderation_needed(struct irq_desc *desc, struct irq= _mod_state *ms) --=20 2.52.0.305.g3fc767764a-goog From nobody Sun Feb 8 08:27:32 2026 Received: from mail-ed1-f74.google.com (mail-ed1-f74.google.com [209.85.208.74]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 2E553328606 for ; Wed, 17 Dec 2025 11:21:49 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.208.74 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1765970511; cv=none; b=D0NlcLZjid21yI9f3W+tq6A+D2an6Zr/8/qWdAjn2w+KouWAU0ClB8KUtzh9f6pXkJmRmrVaFsW67Ci3uLovPTgdHUZaefohRool8coX0mWPhmygLgnYdtBAkUqR4mrZ4RitC+xJgSwEj9OttfMfLTrF5Noc+BkAppnscN4s7Sw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1765970511; c=relaxed/simple; bh=djGK9a/PbIEvwH7fMw191k8tLn5bmy64ldWAKNGBEIg=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=jnCNnelhWjLTHuZA4OCFZHHNAXbPwSr/lds5u2qtWrXB/8EA88x7ynR+os+nz6uDeAfyyZlZhYOaKZk34hFlSlRumd2ognDduELQOFGG81hUtQqAWfQCZg/eURcs7WhcB2/NPC5bw3HXv5yu9Wky2OtprDghZY+BEqAMclOIatA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--lrizzo.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=JLjZaawv; arc=none smtp.client-ip=209.85.208.74 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--lrizzo.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="JLjZaawv" Received: by mail-ed1-f74.google.com with SMTP id 4fb4d7f45d1cf-64b3eaf9bc4so987100a12.1 for ; Wed, 17 Dec 2025 03:21:49 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1765970508; x=1766575308; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=XtTq+yFtULAr3S4+lmRLp6BZLceWbGit2YFFxC7qk9w=; b=JLjZaawvzfShJkQdD2BxKLnkn9DruQo8Bq7b7MEYPWAxoq1UytLibWndspyD7kpy0t TVoUbbhfeyi+HnJCBk9MTVyoRJWfbkmLlylF//yxvx2pW8Pa5JLCpznKALWqwD8NT5j9 qsVFn7Kjd+wOwu+SVpLFqNGjnzEDS4VpzFdSsvGb9KDmnfYK/ck+ybXH5ABbs8v6Cj9x 3e63d7C1K60d5h8q5Kv1ZB7uChCTPkcyEeBmbVvM1VMCvUtCs5BfKjXOahMxSFRONbZA 45MPHcNxpaiTy9Vt1zlOlvxbD/xKe13ThZVQF8Z5Ji7P5AhBtBKESbINQQ5R/To2vRLg dEKw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1765970508; x=1766575308; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=XtTq+yFtULAr3S4+lmRLp6BZLceWbGit2YFFxC7qk9w=; b=caVEeTELgRU+qSLnkS6j05MTJVlejmZeuDor/GZUBeU2Hie+a/BMpOOlndCQDx6Rad ttPs8+2zDPp6f/PxLKfMlbYr80TFo6HSuxBMXMet3KNNb9Ia5m+Rv+XL8r/6TEcW6pkE EUchR8bxYWez2rZsaPF8+gatJSGvldqZh8/qbAChnDjrdFlcXr1CKFk3b0vW5FS1kx9i uF7ZJ3zbvLUwEwAv1p+La9K7WRAYHUBOGm+hJ57h5cL/YgLRfCbuKgI4Sqy1WWqr89pB s4r1CUCo0sY+BBFWc5YWxUhWq2s/9hDwnWBioNJjqsN2XjETtMW0fgJ8CS4bKTr9rdLz QSkw== X-Gm-Message-State: AOJu0Yz/VGbFrsVGJIKP/ThOcy76F5P+nQ7zWZR9eQ32WMALlJpp9hbi 3Vnq/CrHkY1mEyih0wZhbuU5LF/OQiMtxqdbSwCJdr7eK6GSM1XVS4ATIWlPDd9nrQf2O+5v9Cb MzgxgKQ== X-Google-Smtp-Source: AGHT+IHf617U3EOpdp52V/RhXFK5v3RSOZhwgvcp0C9fXsJFLpQ1TQTfSiq2gZ9LKhWsDSYcQTVE98fnY+g= X-Received: from edsx15.prod.google.com ([2002:aa7:dacf:0:b0:649:93b9:195f]) (user=lrizzo job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6402:2792:b0:649:a84c:745b with SMTP id 4fb4d7f45d1cf-649a84c7534mr15485990a12.3.1765970507467; Wed, 17 Dec 2025 03:21:47 -0800 (PST) Date: Wed, 17 Dec 2025 11:21:28 +0000 In-Reply-To: <20251217112128.1401896-1-lrizzo@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20251217112128.1401896-1-lrizzo@google.com> X-Mailer: git-send-email 2.52.0.305.g3fc767764a-goog Message-ID: <20251217112128.1401896-4-lrizzo@google.com> Subject: [PATCH-v3 3/3] genirq: Configurable default mode for GSIM From: Luigi Rizzo To: Thomas Gleixner , Marc Zyngier , Luigi Rizzo , Paolo Abeni , Andrew Morton , Sean Christopherson , Jacob Pan Cc: linux-kernel@vger.kernel.org, linux-arch@vger.kernel.org, Bjorn Helgaas , Willem de Bruijn , Luigi Rizzo Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" GSIM (Global Software Interrupt Moderation) can be enabled only after the interrupt is created by writing to /proc/irq/NN/soft_moderation. This is impractical when devices that are dynamically created or reconfigured. Add a module parameter irq_moderation.enable_list to specify whether moderation should be enabled at interrupt creation time. This is done with a comma-separated list of patterns (enable_list) matched against interrupt or handler names when the interrupt is created. This allows very flexible control without having to modify every single driver. As an example, one can limit to specific drivers by specifying the handler functions (using parentheses) as below irq_moderation.enable_list=3D"nvme_irq(),vfio_msihandler()" ora apply it to certain interrupt names irq_moderation.enable_list=3D"eth*,vfio*" Change-Id: Id5f7aba5b21ad478e4fb7edd0f00eeb2b83e07d2 Signed-off-by: Luigi Rizzo --- include/linux/irqdesc.h | 5 ++ kernel/irq/irq_moderation.c | 125 ++++++++++++++++++++++++++++++++++++ kernel/irq/manage.c | 4 ++ 3 files changed, 134 insertions(+) diff --git a/include/linux/irqdesc.h b/include/linux/irqdesc.h index d42ce57ff5406..e0785c709d013 100644 --- a/include/linux/irqdesc.h +++ b/include/linux/irqdesc.h @@ -16,6 +16,7 @@ struct module; struct irq_desc; struct irq_domain; struct pt_regs; +struct irqaction; =20 #ifdef CONFIG_IRQ_SOFT_MODERATION =20 @@ -30,11 +31,15 @@ struct irq_desc_mod { }; =20 void irq_moderation_init_fields(struct irq_desc_mod *mod); +bool irq_moderation_get_default(const struct irqaction *action); +void irq_moderation_enable(struct irq_desc *desc); =20 #else =20 struct irq_desc_mod {}; static inline void irq_moderation_init_fields(struct irq_desc_mod *mod) {} +static inline bool irq_moderation_get_default(const struct irqaction *acti= on) { return false; } +static inline void irq_moderation_enable(struct irq_desc *desc) {} =20 #endif =20 diff --git a/kernel/irq/irq_moderation.c b/kernel/irq/irq_moderation.c index be7d10404a24a..216bd1f83d23e 100644 --- a/kernel/irq/irq_moderation.c +++ b/kernel/irq/irq_moderation.c @@ -50,6 +50,11 @@ * update_ms (default 5, range 1-100) * How often the load is measured and moderation delay updated. * + * enable_list (comma-separated list of handlers or interrupt names) + * Enable moderation at creation for interrupts whose name or handler + * functions match patterns in this list. Example: + * "nvme_irq(),eth*,vfio_msihandler()" + * * Moderation can be enabled/disabled dynamically for individual interrupt= s with * * echo 1 > /proc/irq/NN/soft_moderation # use 0 to disable @@ -142,6 +147,90 @@ void irq_moderation_init_fields(struct irq_desc_mod *m= od) mod->enable =3D false; } =20 +/* + * irq_moderation.enable_list is a comma-separated list of patterns to mat= ch + * against the name eg "eth*" or handler e.g "nvme_irq()" of the interrupt. + * There is one active buffer and one used for updates, so replacement just + * needs to swap the pointer. + */ +#define MAX_CONFIG_LEN 4096 +static struct enable_list { + char buf[MAX_CONFIG_LEN]; + char buf2[MAX_CONFIG_LEN]; + char *cur; + uint len; + struct rw_semaphore rwsem; +} patterns =3D { .cur =3D patterns.buf, .rwsem =3D __RWSEM_INITIALIZER(pat= terns.rwsem) }; + +static int set_enable_list(const char *val, const struct kernel_param *kp) +{ + uint i, len =3D strlen(val); + char *dst; + + if (len >=3D MAX_CONFIG_LEN - 1) + return -E2BIG; + if (val[len - 1] =3D=3D '\n') + len--; + + /* Serialized by procfs. */ + dst =3D patterns.cur =3D=3D patterns.buf ? patterns.buf2 : patterns.buf; + /* Copy and split the string on commas. */ + for (i =3D 0; i < len; i++) + dst[i] =3D val[i] =3D=3D ',' ? '\0' : val[i]; + dst[i] =3D '\0'; + guard(rwsem_write)(&patterns.rwsem); + patterns.cur =3D dst; + patterns.len =3D len; + return 0; +} + +static int get_enable_list(char *buf, const struct kernel_param *kp) +{ + int i; + + /* Join the patterns with commas. */ + guard(rwsem_read)(&patterns.rwsem); + for (i =3D 0; i < patterns.len; i++) + buf[i] =3D patterns.cur[i] =3D=3D '\0' ? ',' : patterns.cur[i]; + buf[i] =3D '\0'; + return patterns.len; +} + +static const struct kernel_param_ops enable_list_ops =3D { + .set =3D set_enable_list, + .get =3D get_enable_list, +}; + +module_param_cb(enable_list, &enable_list_ops, NULL, 0444); + +/* Called early in __setup_irq() without desc->lock. */ +bool irq_moderation_get_default(const struct irqaction *act) +{ + guard(rwsem_read)(&patterns.rwsem); + for (int arg =3D 0; arg < 3; arg++) { + /* buf includes room for "()". */ + char buf[KSYM_SYMBOL_LEN + 3]; + const char *name; + + if (arg =3D=3D 0) { + name =3D act->name; + if (!name) + continue; + } else { + irq_handler_t fn =3D arg =3D=3D 1 ? act->handler : act->thread_fn; + + if (lookup_symbol_name((ulong)fn, buf)) + continue; + memcpy(buf + strlen(buf), "()", 3); + name =3D buf; + } + for (int i =3D 0; i < patterns.len; i +=3D 1 + strlen(patterns.cur + i)) + if (glob_match(patterns.cur + i, name)) + return true; + } + return false; +} + static int set_mode(struct irq_desc *desc, bool enable) { lockdep_assert_held(&desc->lock); @@ -163,6 +252,12 @@ static int set_mode(struct irq_desc *desc, bool enable) return 0; } =20 +/* Called with desc->lock held in __setup_irq(). */ +void irq_moderation_enable(struct irq_desc *desc) +{ + set_mode(desc, true); +} + #pragma clang diagnostic error "-Wformat" /* Print statistics */ static int moderation_show(struct seq_file *p, void *v) @@ -171,6 +266,7 @@ static int moderation_show(struct seq_file *p, void *v) uint delay_us =3D irq_mod_info.delay_us; u64 now =3D ktime_get_ns(); int j, active_cpus =3D 0; + char *buf; =20 #define HEAD_FMT "%5s %8s %8s %4s %8s %11s %11s %11s %11s %11s %= 9s\n" #define BODY_FMT "%5u %8u %8u %4u %8u %11u %11u %11u %11u %11u %= 9u\n" @@ -210,17 +306,24 @@ static int moderation_show(struct seq_file *p, void *= v) ms->stray_irq); } =20 + buf =3D kmalloc(MAX_CONFIG_LEN, GFP_KERNEL); + if (buf) + get_enable_list(buf, NULL); seq_printf(p, "\n" "enabled %s\n" + "enable_list '%s'\n" "delay_us %u\n" "target_intr_rate %u\n" "hardirq_percent %u\n" "update_ms %u\n" "scale_cpus %u\n", str_yes_no(delay_us > 0), + buf ? : "", delay_us, irq_mod_info.target_intr_rate, irq_mod_info.hardirq_percent, irq_mod_info.update_ms, irq_mod_info.scale_cpus); + if (buf) + kfree(buf); =20 seq_printf(p, "intr_rate %lu\n" @@ -258,6 +361,8 @@ static struct param_names param_names[] =3D { /* Entries with no name cannot be set at runtime. */ { "", &irq_mod_info.increase_factor, MIN_SCALING_FACTOR, 128 }, { "", &irq_mod_info.scale_cpus, 50, 1000 }, + /* val =3D NULL indicates a special entry for enable_list. */ + { "enable_list", NULL }, }; =20 static void update_enable_key(void) @@ -270,6 +375,24 @@ static void update_enable_key(void) static_branch_disable(key); } =20 +static ssize_t set_enable_list_from_proc(const char __user *buf, int count= , int ofs) +{ + char *cmd; + + if (count >=3D MAX_CONFIG_LEN) + return -EINVAL; + cmd =3D kmalloc(count + 1, GFP_KERNEL); + if (!cmd) + return -ENOMEM; + if (copy_from_user(cmd, buf, count)) { + kfree(cmd); + return -EFAULT; + } + set_enable_list(cmd + ofs, NULL); + kfree(cmd); + return count; +} + static ssize_t moderation_write(struct file *f, const char __user *buf, si= ze_t count, loff_t *ppos) { uint i, val, copy_len, name_len; @@ -287,6 +410,8 @@ static ssize_t moderation_write(struct file *f, const c= har __user *buf, size_t c if (name_len < 1 || copy_len < name_len + 2 || strncmp(cmd, n->name, nam= e_len) || cmd[name_len] !=3D '=3D') continue; + if (n->val =3D=3D NULL) + return set_enable_list_from_proc(buf, count, name_len + 1); if (kstrtouint(cmd + name_len + 1, 0, &val)) return -EINVAL; WRITE_ONCE(*(n->val), clamp(val, n->min, n->max)); diff --git a/kernel/irq/manage.c b/kernel/irq/manage.c index 8b1b4c8a4f54c..41b6e6ec2b09e 100644 --- a/kernel/irq/manage.c +++ b/kernel/irq/manage.c @@ -1449,6 +1449,7 @@ static bool valid_percpu_irqaction(struct irqaction *= old, struct irqaction *new) static int __setup_irq(unsigned int irq, struct irq_desc *desc, struct irqaction *new) { + bool use_moderation =3D irq_moderation_get_default(new); struct irqaction *old, **old_ptr; unsigned long flags, thread_mask =3D 0; int ret, nested, shared =3D 0; @@ -1761,6 +1762,9 @@ __setup_irq(unsigned int irq, struct irq_desc *desc, = struct irqaction *new) =20 irq_pm_install_action(desc, new); =20 + if (use_moderation) + irq_moderation_enable(desc); + /* Reset broken irq detection when installing new handler */ desc->irq_count =3D 0; desc->irqs_unhandled =3D 0; --=20 2.52.0.305.g3fc767764a-goog