From nobody Sat Feb 7 16:39:31 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id A693DC001E0 for ; Sat, 29 Jul 2023 08:39:26 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231288AbjG2IjZ (ORCPT ); Sat, 29 Jul 2023 04:39:25 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:34450 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229615AbjG2IjW (ORCPT ); Sat, 29 Jul 2023 04:39:22 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1C1F22733 for ; Sat, 29 Jul 2023 01:38:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1690619917; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=wj6iLXmkjbQ4r1Vww+IqOBjAiKyNj197Tvn6ogknTfo=; b=Lh3J3iRkR4iloH2nXn/8R/qZMN9f8TKoLk3cNoEUNi7JcOg5yWRrEvaq9MGvJx/5IyOqSK ffjD9TjHPi3sot9H7QhbRumKjYMUIoc/wd/xI0xBppxmnaLk+mrwSu8v1zkH4JWQVEwjvv JKfbSz/j+S9kEqmDSgcAA5XAxUNu0g0= Received: from mail-oi1-f198.google.com (mail-oi1-f198.google.com [209.85.167.198]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-288-lIv3Tpd_MSGrncRMW8Elig-1; Sat, 29 Jul 2023 04:38:35 -0400 X-MC-Unique: lIv3Tpd_MSGrncRMW8Elig-1 Received: by mail-oi1-f198.google.com with SMTP id 5614622812f47-3a3b8b73cbfso5920053b6e.2 for ; Sat, 29 Jul 2023 01:38:35 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1690619915; x=1691224715; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=wj6iLXmkjbQ4r1Vww+IqOBjAiKyNj197Tvn6ogknTfo=; b=VcL02wXUCmtouDMil/ve+2+mak1+khf46Jyc6N4JMCmXuj9b7fi+ZpdXZxJtwvsx8Z NtMjXLh6JtYhUWHEtvVx1CgHNOQwtqsRBUCYX7EefQDyw4s4P+tUIuF1WBwiXjhMo7l2 WJdgk2kedKCmNUdU0yjJI9OPf99YalY7YSJpsHq+Kluit10fUYP49y4lYXtPNnJIalcq z9aTE8PT6B0XOThxotkuvxWpkHQS3CkgvVQFJGYZ1+FNYOTPz3+Z+YnOp4hETDKbHZ46 V05hmeXuVyBNLS4vtWRNY/5RhNTSFxmAMNNHMs9seg1ktColAVyQp1sov+DCIDc3QMCx 4Q/A== X-Gm-Message-State: ABy/qLbP6YT0ynK8593AWg7v9dQrgvgqrz5YpsxKU3OMpzT0GbppqTOm mTeFZX5MWAX96k2fYVZWm6AshPg76nSHs2VzPwVrxRYC47bqKd/l0cLNP6d+O+oHuIMeNkzeDcS MYyvxz76ZSoAf+o/CdMM24PiG X-Received: by 2002:a54:4406:0:b0:3a3:9c50:8fed with SMTP id k6-20020a544406000000b003a39c508fedmr4987024oiw.27.1690619914910; Sat, 29 Jul 2023 01:38:34 -0700 (PDT) X-Google-Smtp-Source: APBJJlGJlpDEFi09nRzGsBj0F5286Z7QuumsI+XAEu/fL0Yu23OB45bAXyvNt8gjOo4AbJDHE1qHag== X-Received: by 2002:a54:4406:0:b0:3a3:9c50:8fed with SMTP id k6-20020a544406000000b003a39c508fedmr4987013oiw.27.1690619914726; Sat, 29 Jul 2023 01:38:34 -0700 (PDT) Received: from localhost.localdomain ([2804:1b3:a801:93ca:d713:4eb3:968e:3b54]) by smtp.gmail.com with ESMTPSA id be24-20020a056808219800b003a3f922207asm2467184oib.1.2023.07.29.01.38.33 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 29 Jul 2023 01:38:34 -0700 (PDT) From: Leonardo Bras To: Thomas Gleixner , Marcelo Tosatti , linux-kernel@vger.kernel.org Cc: Leonardo Bras Subject: [RFC PATCH 1/4] Introducing local_lock_n() and local queue & flush Date: Sat, 29 Jul 2023 05:37:32 -0300 Message-ID: <20230729083737.38699-3-leobras@redhat.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230729083737.38699-2-leobras@redhat.com> References: <20230729083737.38699-2-leobras@redhat.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Some places in the kernel implement a parallel programming strategy consisting on local_locks() for most of the work, and some rare remote operations are scheduled on target cpu. This keeps the overhead low since cacheline tends to be mostly local (and have no locks in non-RT kernels), and the few remote operations will be more costly due to scheduling. On the other hand, for RT workloads this can represent a problem: getting an important workload scheduled out to deal with some unrelated task is sure to introduce unexpected deadline misses. It's interesting, though, that local_lock()s in RT kernels become an spinlock(), so we can use this locking cost (that is already being paid) in order to avoid scheduling work on a remote cpu, and updating another cpu's per_cpu structure from the current cpu, while holding it's spinlock(). In order to do that, it's necessary to introduce a new set of functions to make it possible to get another cpu's local lock (local_*lock_n*()), and also the corresponding local_queue_work_on() and local_flush_work() helpers. On non-RT kernels, every local*_n*() works the exactly same as the non-n functions (the extra parameter is ignored), and both local_queue_work_on() and local_flush_work() call their non-local versions. For RT kernels, though, local_*lock_n*() will use the extra cpu parameter to select the correct per-cpu structure to work on, and acquire the spinlock for that cpu. local_queue_work_on() will just call the requested function in the current cpu: since the local_locks() are spinlocks() we are safe. local_flush_work() then becomes a no-op since no work is actually scheduled on a remote cpu. Some minimal code rework is needed in order to make this mechanism work: The calls for local_*lock*() on the functions that are currently scheduled on remote cpus need to be replaced my local_*lock_n*(), so in RT kernels they can reference a different cpu. This should have almost no impact on non-RT kernels: few this_cpu_ptr() will become per_cpu_ptr(,smp_processor_id()). On RT kernels, this should improve performance and reduces latency by removing scheduling noise. Signed-off-by: Leonardo Bras --- include/linux/local_lock.h | 18 ++++++++++ include/linux/local_lock_internal.h | 52 +++++++++++++++++++++++++++++ 2 files changed, 70 insertions(+) diff --git a/include/linux/local_lock.h b/include/linux/local_lock.h index e55010fa7329..f1fa1e8e3fbc 100644 --- a/include/linux/local_lock.h +++ b/include/linux/local_lock.h @@ -51,4 +51,22 @@ #define local_unlock_irqrestore(lock, flags) \ __local_unlock_irqrestore(lock, flags) =20 +#define local_lock_n(lock, cpu) \ + __local_lock_n(lock, cpu) + +#define local_unlock_n(lock, cpu) \ + __local_unlock_n(lock, cpu) + +#define local_lock_irqsave_n(lock, flags, cpu) \ + __local_lock_irqsave_n(lock, flags, cpu) + +#define local_unlock_irqrestore_n(lock, flags, cpu) \ + __local_unlock_irqrestore_n(lock, flags, cpu) + +#define local_queue_work_on(cpu, wq, work) \ + __local_queue_work_on(cpu, wq, work) + +#define local_flush_work(work) \ + __local_flush_work(work) + #endif diff --git a/include/linux/local_lock_internal.h b/include/linux/local_lock= _internal.h index 975e33b793a7..df064149fff8 100644 --- a/include/linux/local_lock_internal.h +++ b/include/linux/local_lock_internal.h @@ -98,6 +98,25 @@ do { \ local_irq_restore(flags); \ } while (0) =20 +#define __local_lock_n(lock, cpu) __local_lock(lock) +#define __local_unlock_n(lock, cpu) __local_unlock(lock) + +#define __local_lock_irqsave_n(lock, flags, cpu) \ + __local_lock_irqsave(lock, flags) + +#define __local_unlock_irqrestore_n(lock, flags, cpu) \ + __local_unlock_irqrestore(lock, flags) + +#define __local_queue_work_on(cpu, wq, work) \ + do { \ + typeof(cpu) __cpu =3D cpu; \ + typeof(work) __work =3D work; \ + __work->data.counter =3D __cpu; \ + queue_work_on(__cpu, wq, __work); \ + } while (0) + +#define __local_flush_work(work) flush_work(work) + #else /* !CONFIG_PREEMPT_RT */ =20 /* @@ -138,4 +157,37 @@ typedef spinlock_t local_lock_t; =20 #define __local_unlock_irqrestore(lock, flags) __local_unlock(lock) =20 +#define __local_lock_n(__lock, cpu) \ + do { \ + migrate_disable(); \ + spin_lock(per_cpu_ptr((__lock)), cpu); \ + } while (0) + +#define __local_unlock_n(__lock, cpu) \ + do { \ + spin_unlock(per_cpu_ptr((__lock)), cpu); \ + migrate_enable(); \ + } while (0) + +#define __local_lock_irqsave_n(lock, flags, cpu) \ + do { \ + typecheck(unsigned long, flags); \ + flags =3D 0; \ + __local_lock_n(lock, cpu); \ + } while (0) + +#define __local_unlock_irqrestore_n(lock, flags, cpu) \ + __local_unlock_n(lock, cpu) + +#define __local_queue_work_on(cpu, wq, work) \ + do { \ + typeof(cpu) __cpu =3D cpu; \ + typeof(work) __work =3D work; \ + __work->data =3D (typeof(__work->data))__cpu; \ + __work->func(__work); \ + } while (0) + +#define __local_flush_work(work) \ + do {} while (0) + #endif /* CONFIG_PREEMPT_RT */ --=20 2.41.0 From nobody Sat Feb 7 16:39:31 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id ACC94C001DC for ; Sat, 29 Jul 2023 08:39:39 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229753AbjG2Iji (ORCPT ); Sat, 29 Jul 2023 04:39:38 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:34460 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231279AbjG2Ij2 (ORCPT ); Sat, 29 Jul 2023 04:39:28 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A47FD3AB3 for ; Sat, 29 Jul 2023 01:38:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1690619919; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=tXaw8xh9/uqiELcQ6XYGjWI9gHB6EvF8rWATqtNVPNw=; b=ExDxyVtKLWF9bepxhJd1qIXwHC/jx5doc9PuUWieDZY6mQOG2B9g4sFbGAQgyWcZoFK3M7 QXNpdmWPFDrJropbD0gH+bqQsi1Ds4TsQ+vTRRzEghZDid4FKmDhNCAmziEUXsJurSunMF CKCEzzMsIqbOttVdr7pIVIVCA0zevCA= Received: from mail-oo1-f70.google.com (mail-oo1-f70.google.com [209.85.161.70]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-219-Gqqt_6UAMWqMonAAkokKVg-1; Sat, 29 Jul 2023 04:38:37 -0400 X-MC-Unique: Gqqt_6UAMWqMonAAkokKVg-1 Received: by mail-oo1-f70.google.com with SMTP id 006d021491bc7-565893ef956so4083244eaf.0 for ; Sat, 29 Jul 2023 01:38:37 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1690619916; x=1691224716; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=tXaw8xh9/uqiELcQ6XYGjWI9gHB6EvF8rWATqtNVPNw=; b=e2S7o2XR//N+WuJ1DMkedl0QG6FMqB86odNkzMtkcooJ32hw3XebzBMdZlmyw0nqD3 BPJKsa2gaXaQzoIX2cIW8Y4PoKNvbkZzWy72+OHsF2HyUfv/goKDV/Oi0PJCiHSbLBrA tza/GZ7dA1zFvljtqYFklBGJvi6tYIeBgIMLVM6fKTQaHSYrMlAsdhfEJUeY+tqnx17m kIuaMiRxmwM+sf4V8Kb41uyAHTIL8SegFh01Pan9DvgFJod+ALXzo8MBCdyfcNs9ktQW I/hSwBbf6SojzRpV1kwUZ/WIBjyYrfgK+AoG6UwuQYuNe++nEAd9APypxaGhhqolcvBD Ckww== X-Gm-Message-State: ABy/qLZ2dZ+ka12E/T5avVqvoG6NiGyxYVQWmMlEZO8/FLaXwHdrMcil MkRv3b6L/BsUo/QDxECe9yrswMY4K4Y82GnDtnAOn+0uP/Fex1sx95zHi6vnrOWZJ8xcI8iJT1e uT2GxmrsUy6E0KyO0/ICpflkCaB/Za2Rn X-Received: by 2002:a05:6808:309b:b0:3a5:ae6b:135e with SMTP id bl27-20020a056808309b00b003a5ae6b135emr5655239oib.53.1690619916684; Sat, 29 Jul 2023 01:38:36 -0700 (PDT) X-Google-Smtp-Source: APBJJlFmxI05UVL41BW0bFY5niMhRBlN3Ug2o75+FrvPm/cfGRczsrKhvhswW5Oyitt0V1cDN7u7Ug== X-Received: by 2002:a05:6808:309b:b0:3a5:ae6b:135e with SMTP id bl27-20020a056808309b00b003a5ae6b135emr5655230oib.53.1690619916470; Sat, 29 Jul 2023 01:38:36 -0700 (PDT) Received: from localhost.localdomain ([2804:1b3:a801:93ca:d713:4eb3:968e:3b54]) by smtp.gmail.com with ESMTPSA id be24-20020a056808219800b003a3f922207asm2467184oib.1.2023.07.29.01.38.34 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 29 Jul 2023 01:38:36 -0700 (PDT) From: Leonardo Bras To: Thomas Gleixner , Marcelo Tosatti , linux-kernel@vger.kernel.org Cc: Leonardo Bras Subject: [RFC PATCH 2/4] swap: apply new local_schedule_work_on() interface Date: Sat, 29 Jul 2023 05:37:33 -0300 Message-ID: <20230729083737.38699-4-leobras@redhat.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230729083737.38699-2-leobras@redhat.com> References: <20230729083737.38699-2-leobras@redhat.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Make use of the new local_*lock_n*() and local_schedule_work_on() interface to improve performance & latency on PREEMTP_RT kernels. For functions that may be scheduled in a different cpu, replace local_*lock*() by local_lock_n*(), and replace schedule_work_on() by local_schedule_work_on(). The same happens for flush_work() and local_flush_work(). This should bring no relevant performance impact on non-RT kernels: For functions that may be scheduled in a different cpu, the local_*lock's this_cpu_ptr() becomes a per_cpu_ptr(smp_processor_id()). Signed-off-by: Leonardo Bras --- mm/swap.c | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/mm/swap.c b/mm/swap.c index cd8f0150ba3a..a79f2091eae5 100644 --- a/mm/swap.c +++ b/mm/swap.c @@ -760,11 +760,11 @@ void lru_add_drain(void) * the same cpu. It shouldn't be a problem in !SMP case since * the core is only one and the locks will disable preemption. */ -static void lru_add_and_bh_lrus_drain(void) +static void lru_add_and_bh_lrus_drain(int cpu) { - local_lock(&cpu_fbatches.lock); - lru_add_drain_cpu(smp_processor_id()); - local_unlock(&cpu_fbatches.lock); + local_lock_n(&cpu_fbatches.lock, cpu); + lru_add_drain_cpu(cpu); + local_unlock_n(&cpu_fbatches.lock, cpu); invalidate_bh_lrus_cpu(); mlock_drain_local(); } @@ -782,9 +782,9 @@ void lru_add_drain_cpu_zone(struct zone *zone) =20 static DEFINE_PER_CPU(struct work_struct, lru_add_drain_work); =20 -static void lru_add_drain_per_cpu(struct work_struct *dummy) +static void lru_add_drain_per_cpu(struct work_struct *w) { - lru_add_and_bh_lrus_drain(); + lru_add_and_bh_lrus_drain(w->data.counter); } =20 static bool cpu_needs_drain(unsigned int cpu) @@ -888,13 +888,13 @@ static inline void __lru_add_drain_all(bool force_all= _cpus) =20 if (cpu_needs_drain(cpu)) { INIT_WORK(work, lru_add_drain_per_cpu); - queue_work_on(cpu, mm_percpu_wq, work); + local_queue_work_on(cpu, mm_percpu_wq, work); __cpumask_set_cpu(cpu, &has_work); } } =20 for_each_cpu(cpu, &has_work) - flush_work(&per_cpu(lru_add_drain_work, cpu)); + local_flush_work(&per_cpu(lru_add_drain_work, cpu)); =20 done: mutex_unlock(&lock); @@ -941,7 +941,7 @@ void lru_cache_disable(void) #ifdef CONFIG_SMP __lru_add_drain_all(true); #else - lru_add_and_bh_lrus_drain(); + lru_add_and_bh_lrus_drain(smp_processor_id()); #endif } =20 --=20 2.41.0 From nobody Sat Feb 7 16:39:31 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id E5208C001DC for ; Sat, 29 Jul 2023 08:39:43 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231379AbjG2Ijl (ORCPT ); Sat, 29 Jul 2023 04:39:41 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:34466 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231283AbjG2Ij2 (ORCPT ); Sat, 29 Jul 2023 04:39:28 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E434E448A for ; Sat, 29 Jul 2023 01:38:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1690619921; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=o8rIpeBRze9Zu+72Qm9sEpSig7svqTG6Kuh9DJYcSFE=; b=OB6KvnTc7VsgJS22BjKYm+UVkvttOWJ2rQK5E2YAImOg3rDP/O9BRvu72t0venSXo84uHk /aF0Mmtc0RRaPD6eCP+bIgQgwADcUo8xH4eM3c3P666M+gcLNxgtlint4nzXZ4dwhq165y C/cDRbbd7v9xoGNWbjlH/Z/XH8WpGkM= Received: from mail-oi1-f200.google.com (mail-oi1-f200.google.com [209.85.167.200]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-392-N9FgJUSPP0qhLCobfTOKaw-1; Sat, 29 Jul 2023 04:38:39 -0400 X-MC-Unique: N9FgJUSPP0qhLCobfTOKaw-1 Received: by mail-oi1-f200.google.com with SMTP id 5614622812f47-3a5ab57036fso5783864b6e.3 for ; Sat, 29 Jul 2023 01:38:39 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1690619918; x=1691224718; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=o8rIpeBRze9Zu+72Qm9sEpSig7svqTG6Kuh9DJYcSFE=; b=Y2rW7570eu3xyqX0qbBsXULwQ//cV6VY8BZbL7DSV3n+DzBzibBDDQ99waRk5Zbp34 qrtAdKbJtVcPp122XIeoCsCG03yDnH6rD6Y5DT9XiCbj0h5yClFfQSw3l7OeHiAF9jYt ywsR22NQurLhSkRIp0a8a2fCHWOCrV0re5Xt/v660XAp9zG8eYGelxmrs2hZw8+r1Iep CucJX+Ncj3AGLE2+e5WLTpHChDSEJowJ0gF8OxkWnelQNS6Roe1FLxwGsqb23Em70U0g Ju9XBqFvBBZoIfhZRlgma4r/BQ7M9Z7pHTtZG8kXQLkwnmU4bDka1It+jBau77sl13rD XoNA== X-Gm-Message-State: ABy/qLYe2AKavotAAGzJOJWpoayvk8xq2YB/3V1qYQUq7Kv7nGz18mdW FQbKf+GH0QBSPfj8zL1PUuelPDhsJlMGGURIpQNVhLiuEM4Clj1lGaGPHEHL8/BQGg/xxU4imcm 0ZDvIRJpQp5Hw7mNNs5VcVuVY X-Received: by 2002:a05:6808:1509:b0:3a3:6382:b67d with SMTP id u9-20020a056808150900b003a36382b67dmr5871263oiw.41.1690619918506; Sat, 29 Jul 2023 01:38:38 -0700 (PDT) X-Google-Smtp-Source: APBJJlE72np9M7VBc9ea9R1wrVx2dR1OmQ/wiPBtvU58KUGekGEzrx+xog2FSog4LO47TmsFk4c88A== X-Received: by 2002:a05:6808:1509:b0:3a3:6382:b67d with SMTP id u9-20020a056808150900b003a36382b67dmr5871253oiw.41.1690619918256; Sat, 29 Jul 2023 01:38:38 -0700 (PDT) Received: from localhost.localdomain ([2804:1b3:a801:93ca:d713:4eb3:968e:3b54]) by smtp.gmail.com with ESMTPSA id be24-20020a056808219800b003a3f922207asm2467184oib.1.2023.07.29.01.38.36 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 29 Jul 2023 01:38:37 -0700 (PDT) From: Leonardo Bras To: Thomas Gleixner , Marcelo Tosatti , linux-kernel@vger.kernel.org Cc: Leonardo Bras Subject: [RFC PATCH 3/4] memcontrol: apply new local_schedule_work_on() interface Date: Sat, 29 Jul 2023 05:37:34 -0300 Message-ID: <20230729083737.38699-5-leobras@redhat.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230729083737.38699-2-leobras@redhat.com> References: <20230729083737.38699-2-leobras@redhat.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Make use of the new local_*lock_n*() and local_schedule_work_on() interface to improve performance & latency on PREEMTP_RT kernels. For functions that may be scheduled in a different cpu, replace local_*lock*() by local_lock_n*(), and replace schedule_work_on() by local_schedule_work_on(). This should bring no relevant performance impact on non-RT kernels: For functions that may be scheduled in a different cpu, the local_*lock's this_cpu_ptr() becomes a per_cpu_ptr(smp_processor_id()). Signed-off-by: Leonardo Bras --- mm/memcontrol.c | 17 +++++++++++------ 1 file changed, 11 insertions(+), 6 deletions(-) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index e8ca4bdcb03c..6d4fa48d75e3 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -2277,7 +2277,7 @@ static void drain_stock(struct memcg_stock_pcp *stock) WRITE_ONCE(stock->cached, NULL); } =20 -static void drain_local_stock(struct work_struct *dummy) +static void _drain_local_stock(int cpu) { struct memcg_stock_pcp *stock; struct obj_cgroup *old =3D NULL; @@ -2288,18 +2288,23 @@ static void drain_local_stock(struct work_struct *d= ummy) * drain_stock races is that we always operate on local CPU stock * here with IRQ disabled */ - local_lock_irqsave(&memcg_stock.stock_lock, flags); + local_lock_irqsave_n(&memcg_stock.stock_lock, flags, cpu); =20 - stock =3D this_cpu_ptr(&memcg_stock); + stock =3D per_cpu_ptr(&memcg_stock, cpu); old =3D drain_obj_stock(stock); drain_stock(stock); clear_bit(FLUSHING_CACHED_CHARGE, &stock->flags); =20 - local_unlock_irqrestore(&memcg_stock.stock_lock, flags); + local_unlock_irqrestore_n(&memcg_stock.stock_lock, flags, cpu); if (old) obj_cgroup_put(old); } =20 +static void drain_local_stock(struct work_struct *w) +{ + _drain_local_stock((int)w->data.counter); +} + /* * Cache charges(val) to local per_cpu area. * This will be consumed by consume_stock() function, later. @@ -2365,9 +2370,9 @@ static void drain_all_stock(struct mem_cgroup *root_m= emcg) if (flush && !test_and_set_bit(FLUSHING_CACHED_CHARGE, &stock->flags)) { if (cpu =3D=3D curcpu) - drain_local_stock(&stock->work); + _drain_local_stock(cpu); else if (!cpu_is_isolated(cpu)) - schedule_work_on(cpu, &stock->work); + local_queue_work_on(cpu, system_wq, &stock->work); } } migrate_enable(); --=20 2.41.0 From nobody Sat Feb 7 16:39:31 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6659CC001DC for ; Sat, 29 Jul 2023 08:39:36 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231330AbjG2Ije (ORCPT ); Sat, 29 Jul 2023 04:39:34 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:34472 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229615AbjG2IjZ (ORCPT ); Sat, 29 Jul 2023 04:39:25 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2BFC8448C for ; Sat, 29 Jul 2023 01:38:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1690619922; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=RMopzQPIrgY6C9bjYkGrv8CSEJs16BIQJryaMHu6prw=; b=cJMaSYnwkbA2xCi73XGWqg1AUVDpVpk+uvz87p6LcVs24iBZPZf0nEg7WRhh5INIMSQsp6 EPWO+8gYvzkPnStIyTqx86GjIunR83tj9AS8P/oXT9R39H8703YGBh64iZcWFT8c4JUbGz bjywCWPvumJMdWOlQS3O3If0/58gOAY= Received: from mail-oi1-f199.google.com (mail-oi1-f199.google.com [209.85.167.199]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-83-SoX0zxJdOXSRFHExcnszMw-1; Sat, 29 Jul 2023 04:38:41 -0400 X-MC-Unique: SoX0zxJdOXSRFHExcnszMw-1 Received: by mail-oi1-f199.google.com with SMTP id 5614622812f47-3a5a7e981ddso5438478b6e.2 for ; Sat, 29 Jul 2023 01:38:40 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1690619920; x=1691224720; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=RMopzQPIrgY6C9bjYkGrv8CSEJs16BIQJryaMHu6prw=; b=E7DdzocmCanoFh8xyQ7lfRq6PTpzJOMwob2yhI/fH6/fovmm2JgC0y0Xyb6GCL/zMD n7g2XOYdvzAthyYWC4S7hcPNQW0FoTb2U7dVS9fczMZ5Xo5uo33YiDHActWTVnk+4WaA YYDfHpN/2BtWy4oj/pFElWaE9rBcOytOZ/Otp8BnIg2j+0grgl1dJ857AAUo++hCfCHw s0zekJMnDY/ypRlfQ3MuuR4XqtkOoNY/MyQj8iz8b0BBZoNDDlg8B2nsagAd7WKGF0Ds bPnTy6oTN65Af6uadMdtVFj4mBq6G934VdDvDKQ+ZDBjAzRTR2MrEcSpcadyPaZL4M6S d97w== X-Gm-Message-State: ABy/qLa8UyHpAMkVzf3Ex+74YtjfZq+QmjMW1ejeeACuf03GTWNYSmuT 8i/It2U7dd7klPKcRstmbNHVG7USYK0i1mt9MyJP/jwhGbRFqLN2jLiAqtHShQdWV5UGX2BA0bc B0VAyPQgDq7pK2oHpT/GbAAM1 X-Received: by 2002:a05:6808:138c:b0:3a5:98b4:c139 with SMTP id c12-20020a056808138c00b003a598b4c139mr6261452oiw.20.1690619920282; Sat, 29 Jul 2023 01:38:40 -0700 (PDT) X-Google-Smtp-Source: APBJJlEH5b27Ayet4VNcR1aftjuqWvQlRQfSXupw28ms9fmuNADQMu4IwIHNRnkpakCGJQDp6PAAdw== X-Received: by 2002:a05:6808:138c:b0:3a5:98b4:c139 with SMTP id c12-20020a056808138c00b003a598b4c139mr6261449oiw.20.1690619920040; Sat, 29 Jul 2023 01:38:40 -0700 (PDT) Received: from localhost.localdomain ([2804:1b3:a801:93ca:d713:4eb3:968e:3b54]) by smtp.gmail.com with ESMTPSA id be24-20020a056808219800b003a3f922207asm2467184oib.1.2023.07.29.01.38.38 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 29 Jul 2023 01:38:39 -0700 (PDT) From: Leonardo Bras To: Thomas Gleixner , Marcelo Tosatti , linux-kernel@vger.kernel.org Cc: Leonardo Bras Subject: [RFC PATCH 4/4] slub: apply new local_schedule_work_on() interface Date: Sat, 29 Jul 2023 05:37:35 -0300 Message-ID: <20230729083737.38699-6-leobras@redhat.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230729083737.38699-2-leobras@redhat.com> References: <20230729083737.38699-2-leobras@redhat.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Make use of the new local_*lock_n*() and local_schedule_work_on() interface to improve performance & latency on PREEMTP_RT kernels. For functions that may be scheduled in a different cpu, replace local_*lock*() by local_lock_n*(), and replace schedule_work_on() by local_schedule_work_on(). The same happens for flush_work() and local_flush_work(). This should bring no relevant performance impact on non-RT kernels: For functions that may be scheduled in a different cpu, the local_*lock's this_cpu_ptr() becomes a per_cpu_ptr(smp_processor_id()). Signed-off-by: Leonardo Bras --- mm/slub.c | 17 ++++++++++------- 1 file changed, 10 insertions(+), 7 deletions(-) diff --git a/mm/slub.c b/mm/slub.c index e3b5d5c0eb3a..feb4a502d9a8 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -2733,13 +2733,14 @@ static inline void unfreeze_partials_cpu(struct kme= m_cache *s, =20 #endif /* CONFIG_SLUB_CPU_PARTIAL */ =20 -static inline void flush_slab(struct kmem_cache *s, struct kmem_cache_cpu = *c) +static inline void flush_slab(struct kmem_cache *s, struct kmem_cache_cpu = *c, + int cpu) { unsigned long flags; struct slab *slab; void *freelist; =20 - local_lock_irqsave(&s->cpu_slab->lock, flags); + local_lock_irqsave_n(&s->cpu_slab->lock, flags, cpu); =20 slab =3D c->slab; freelist =3D c->freelist; @@ -2748,7 +2749,7 @@ static inline void flush_slab(struct kmem_cache *s, s= truct kmem_cache_cpu *c) c->freelist =3D NULL; c->tid =3D next_tid(c->tid); =20 - local_unlock_irqrestore(&s->cpu_slab->lock, flags); + local_unlock_irqrestore_n(&s->cpu_slab->lock, flags, cpu); =20 if (slab) { deactivate_slab(s, slab, freelist); @@ -2790,14 +2791,16 @@ static void flush_cpu_slab(struct work_struct *w) struct kmem_cache *s; struct kmem_cache_cpu *c; struct slub_flush_work *sfw; + int cpu; =20 + cpu =3D w->data.counter; sfw =3D container_of(w, struct slub_flush_work, work); =20 s =3D sfw->s; - c =3D this_cpu_ptr(s->cpu_slab); + c =3D per_cpu_ptr(s->cpu_slab, cpu); =20 if (c->slab) - flush_slab(s, c); + flush_slab(s, c, cpu); =20 unfreeze_partials(s); } @@ -2829,14 +2832,14 @@ static void flush_all_cpus_locked(struct kmem_cache= *s) INIT_WORK(&sfw->work, flush_cpu_slab); sfw->skip =3D false; sfw->s =3D s; - queue_work_on(cpu, flushwq, &sfw->work); + local_queue_work_on(cpu, flushwq, &sfw->work); } =20 for_each_online_cpu(cpu) { sfw =3D &per_cpu(slub_flush, cpu); if (sfw->skip) continue; - flush_work(&sfw->work); + local_flush_work(&sfw->work); } =20 mutex_unlock(&flush_lock); --=20 2.41.0