From nobody Sat Apr 11 00:44:32 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 74259ECAA25 for ; Thu, 25 Aug 2022 16:42:17 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S242050AbiHYQmQ (ORCPT ); Thu, 25 Aug 2022 12:42:16 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42440 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S243079AbiHYQl6 (ORCPT ); Thu, 25 Aug 2022 12:41:58 -0400 Received: from galois.linutronix.de (Galois.linutronix.de [193.142.43.55]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6054BBA9C7 for ; Thu, 25 Aug 2022 09:41:44 -0700 (PDT) From: Sebastian Andrzej Siewior DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1661445700; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=hw+yDpzfGkG1IUct7na3HhO4w8TPB3zBJd3VHyO99as=; b=dbPT4eGnse3NNDLfOalh1LKAYXUqHVvjCjmm2rcW9P7UvT3C8vag7j26Hv5s3jK9b4waE9 xi+fjLRo2tQgSfuQrHXyJCObz8jrlSaUP/IcnY8Ugv+HufxWFvQV9/L74kRuckfrS46zAf 9M2Krd+AXituPbCVC+R92IOmZ7Avt3/hlHs3A6pLZAebDDNLl9bPuOhzFO0JFS8d/nqDz/ jvf/4/bPNS0nIWAOPLB2xjLDWXgUhg48YCZZx1K4/pZ41Cxr23lMW4tyKDr/2dhqPXDEz5 kIQ+cwffKR30OxmiqKl+SrqF++zVch+SotnOk68W5i/pduWG9E+3iAu9EyZATQ== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1661445700; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=hw+yDpzfGkG1IUct7na3HhO4w8TPB3zBJd3VHyO99as=; b=1OJdxiQpOJoQjMRrEp8zMazoDCttBoJZv/t6Zd1FwRNy72ugvjVB3eAwC6DDSMLa6UnTSJ Vc35ptht3THDp7Ag== To: linux-kernel@vger.kernel.org Cc: Thomas Gleixner , Peter Zijlstra , Steven Rostedt , Linus Torvalds , Matthew Wilcox , Ben Segall , Daniel Bristot de Oliveira , Dietmar Eggemann , Ingo Molnar , Juri Lelli , Mel Gorman , Valentin Schneider , Vincent Guittot , Sebastian Andrzej Siewior Subject: [PATCH v2 1/8] preempt: Provide preempt_[dis|en]able_nested() Date: Thu, 25 Aug 2022 18:41:24 +0200 Message-Id: <20220825164131.402717-2-bigeasy@linutronix.de> In-Reply-To: <20220825164131.402717-1-bigeasy@linutronix.de> References: <20220825164131.402717-1-bigeasy@linutronix.de> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Thomas Gleixner On PREEMPT_RT enabled kernels, spinlocks and rwlocks are neither disabling preemption nor interrupts. Though there are a few places which depend on the implicit preemption/interrupt disable of those locks, e.g. seqcount write sections, per CPU statistics updates etc. To avoid sprinkling CONFIG_PREEMPT_RT conditionals all over the place, add preempt_disable_nested() and preempt_enable_nested() which should be descriptive enough. Add a lockdep assertion for the !PREEMPT_RT case to catch callers which do not have preemption disabled. Cc: Ben Segall Cc: Daniel Bristot de Oliveira Cc: Dietmar Eggemann Cc: Ingo Molnar Cc: Juri Lelli Cc: Mel Gorman Cc: Peter Zijlstra Cc: Steven Rostedt Cc: Valentin Schneider Cc: Vincent Guittot Suggested-by: Linus Torvalds Signed-off-by: Thomas Gleixner Signed-off-by: Sebastian Andrzej Siewior Acked-by: Peter Zijlstra (Intel) --- include/linux/preempt.h | 42 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 42 insertions(+) diff --git a/include/linux/preempt.h b/include/linux/preempt.h index b4381f255a5ca..0df425bf9bd75 100644 --- a/include/linux/preempt.h +++ b/include/linux/preempt.h @@ -421,4 +421,46 @@ static inline void migrate_enable(void) { } =20 #endif /* CONFIG_SMP */ =20 +/** + * preempt_disable_nested - Disable preemption inside a normally preempt d= isabled section + * + * Use for code which requires preemption protection inside a critical + * section which has preemption disabled implicitly on non-PREEMPT_RT + * enabled kernels, by e.g.: + * - holding a spinlock/rwlock + * - soft interrupt context + * - regular interrupt handlers + * + * On PREEMPT_RT enabled kernels spinlock/rwlock held sections, soft + * interrupt context and regular interrupt handlers are preemptible and + * only prevent migration. preempt_disable_nested() ensures that preemption + * is disabled for cases which require CPU local serialization even on + * PREEMPT_RT. For non-PREEMPT_RT kernels this is a NOP. + * + * The use cases are code sequences which are not serialized by a + * particular lock instance, e.g.: + * - seqcount write side critical sections where the seqcount is not + * associated to a particular lock and therefore the automatic + * protection mechanism does not work. This prevents a live lock + * against a preempting high priority reader. + * - RMW per CPU variable updates like vmstat. + */ +/* Macro to avoid header recursion hell vs. lockdep */ +#define preempt_disable_nested() \ +do { \ + if (IS_ENABLED(CONFIG_PREEMPT_RT)) \ + preempt_disable(); \ + else \ + lockdep_assert_preemption_disabled(); \ +} while (0) + +/** + * preempt_enable_nested - Undo the effect of preempt_disable_nested() + */ +static __always_inline void preempt_enable_nested(void) +{ + if (IS_ENABLED(CONFIG_PREEMPT_RT)) + preempt_enable(); +} + #endif /* __LINUX_PREEMPT_H */ --=20 2.37.2 From nobody Sat Apr 11 00:44:32 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 11403ECAA25 for ; Thu, 25 Aug 2022 16:42:28 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S242810AbiHYQm0 (ORCPT ); Thu, 25 Aug 2022 12:42:26 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42150 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S243139AbiHYQmC (ORCPT ); Thu, 25 Aug 2022 12:42:02 -0400 Received: from galois.linutronix.de (Galois.linutronix.de [IPv6:2a0a:51c0:0:12e:550::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 263F7BA161; Thu, 25 Aug 2022 09:41:54 -0700 (PDT) From: Sebastian Andrzej Siewior DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1661445700; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=1SZrqW9HXmPmjeRTWLOFIPzv0CDV88cfUXj8V+IhumA=; b=3aoPzsiddfr47zJqtlRQLXYAwPLIHSoLxMe/Nk47fEMwS6kj/gcI5cwRJs5BEuqm/bw9JU A7czQVdmEdkZ6JDGlC15B2mcrLEajs6svacjOk/5eYV/l9nHsCVBnlP4Y08VEAk5u7uqoH 8remR50kIC3O2l2rsHcpmtRNmM4wvQCz7KcSDXgpoIh+twXN65DR1PKMyPPpOKIolxaNxv mSMIBOmntF1TNdDLCxtdMPV98hVyX73aPqQwqEwZKRvjZD2lTvmMW/dB+IwPZm6nI9XllU yb1XydP2ZE+b5iizS7M3SipG30DcqJ6oIc12WiKG1WCRqMLjGSUn6pKps2M4xA== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1661445700; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=1SZrqW9HXmPmjeRTWLOFIPzv0CDV88cfUXj8V+IhumA=; b=yPSOGceGxVZ8dRB1hmldox0oTV9Bh5mC4tmfz4S4xReQF2fiuNDuW0qS/tLYNsYe8XCapI VNGfxPWh9Ejs0dDQ== To: linux-kernel@vger.kernel.org Cc: Thomas Gleixner , Peter Zijlstra , Steven Rostedt , Linus Torvalds , Matthew Wilcox , Alexander Viro , linux-fsdevel@vger.kernel.org, Sebastian Andrzej Siewior Subject: [PATCH v2 2/8] dentry: Use preempt_[dis|en]able_nested() Date: Thu, 25 Aug 2022 18:41:25 +0200 Message-Id: <20220825164131.402717-3-bigeasy@linutronix.de> In-Reply-To: <20220825164131.402717-1-bigeasy@linutronix.de> References: <20220825164131.402717-1-bigeasy@linutronix.de> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Thomas Gleixner Replace the open coded CONFIG_PREEMPT_RT conditional preempt_disable/enable() with the new helper. Signed-off-by: Thomas Gleixner Cc: Alexander Viro Cc: linux-fsdevel@vger.kernel.org Signed-off-by: Sebastian Andrzej Siewior Acked-by: Peter Zijlstra (Intel) Acked-by: Christian Brauner (Microsoft) --- fs/dcache.c | 13 ++----------- 1 file changed, 2 insertions(+), 11 deletions(-) diff --git a/fs/dcache.c b/fs/dcache.c index bb0c4d0038dbd..2ee8636016ee9 100644 --- a/fs/dcache.c +++ b/fs/dcache.c @@ -2597,15 +2597,7 @@ EXPORT_SYMBOL(d_rehash); =20 static inline unsigned start_dir_add(struct inode *dir) { - /* - * The caller holds a spinlock (dentry::d_lock). On !PREEMPT_RT - * kernels spin_lock() implicitly disables preemption, but not on - * PREEMPT_RT. So for RT it has to be done explicitly to protect - * the sequence count write side critical section against a reader - * or another writer preempting, which would result in a live lock. - */ - if (IS_ENABLED(CONFIG_PREEMPT_RT)) - preempt_disable(); + preempt_disable_nested(); for (;;) { unsigned n =3D dir->i_dir_seq; if (!(n & 1) && cmpxchg(&dir->i_dir_seq, n, n + 1) =3D=3D n) @@ -2618,8 +2610,7 @@ static inline void end_dir_add(struct inode *dir, uns= igned int n, wait_queue_head_t *d_wait) { smp_store_release(&dir->i_dir_seq, n + 2); - if (IS_ENABLED(CONFIG_PREEMPT_RT)) - preempt_enable(); + preempt_enable_nested(); wake_up_all(d_wait); } =20 --=20 2.37.2 From nobody Sat Apr 11 00:44:32 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7FA60ECAA25 for ; Thu, 25 Aug 2022 16:42:31 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S243196AbiHYQm3 (ORCPT ); Thu, 25 Aug 2022 12:42:29 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42520 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S243116AbiHYQmD (ORCPT ); Thu, 25 Aug 2022 12:42:03 -0400 Received: from galois.linutronix.de (Galois.linutronix.de [IPv6:2a0a:51c0:0:12e:550::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 47EF9BB012 for ; Thu, 25 Aug 2022 09:41:54 -0700 (PDT) From: Sebastian Andrzej Siewior DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1661445701; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=3a2uwD3ePfGP6QOLusJldJ8Cx5w8uOIWpoMcUghBNEI=; b=fAoAi6GN6WY3LeygB/lU0oNfEevDLSlEyWydv2qLC8qJtAwmaJKfLqO4P16YjHMIUo0Je9 llrAhoYh2vZSnpVGam3C5lt5iT7SkmCn51yjLk++hlqRWDKbRPigRr+Dc56Fpb8OYFzSe7 DnlzDN20dDV1rYDLSq3rpXJOlaXcaUMMB5MMNQFUokVaixRlqPj3UFwL9lQjHuBaaXpY67 HVdPQDhN7wH4uc9PcPmbOBplXAZUd9DEUEz4jPfqyQgdpRk0lgPTcxpzkq+V5x55YEH9gj +/KOfkwwtXkRGDvRjHAlLiKvucUPP/2n3UEDqCQOO+U6s4fLgWmQYRcw60KBTw== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1661445701; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=3a2uwD3ePfGP6QOLusJldJ8Cx5w8uOIWpoMcUghBNEI=; b=YGZc1zl3hETI58auazyzwWtrsPvOEkOwmmHlHQaf1c4aLojGysK6XG0Bv0ZbXFdkioOVw0 bEi0iQIIzmj2gYBA== To: linux-kernel@vger.kernel.org Cc: Thomas Gleixner , Peter Zijlstra , Steven Rostedt , Linus Torvalds , Matthew Wilcox , Andrew Morton , linux-mm@kvack.org, Sebastian Andrzej Siewior Subject: [PATCH v2 3/8] mm/vmstat: Use preempt_[dis|en]able_nested() Date: Thu, 25 Aug 2022 18:41:26 +0200 Message-Id: <20220825164131.402717-4-bigeasy@linutronix.de> In-Reply-To: <20220825164131.402717-1-bigeasy@linutronix.de> References: <20220825164131.402717-1-bigeasy@linutronix.de> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Thomas Gleixner Replace the open coded CONFIG_PREEMPT_RT conditional preempt_enable/disable() pairs with the new helper functions which hide the underlying implementation details. Signed-off-by: Thomas Gleixner Cc: Andrew Morton Cc: linux-mm@kvack.org Signed-off-by: Sebastian Andrzej Siewior Acked-by: Peter Zijlstra (Intel) Acked-by: Michal Hocko --- mm/vmstat.c | 36 ++++++++++++------------------------ 1 file changed, 12 insertions(+), 24 deletions(-) diff --git a/mm/vmstat.c b/mm/vmstat.c index 373d2730fcf21..d514fe7f90af0 100644 --- a/mm/vmstat.c +++ b/mm/vmstat.c @@ -355,8 +355,7 @@ void __mod_zone_page_state(struct zone *zone, enum zone= _stat_item item, * CPU migrations and preemption potentially corrupts a counter so * disable preemption. */ - if (IS_ENABLED(CONFIG_PREEMPT_RT)) - preempt_disable(); + preempt_disable_nested(); =20 x =3D delta + __this_cpu_read(*p); =20 @@ -368,8 +367,7 @@ void __mod_zone_page_state(struct zone *zone, enum zone= _stat_item item, } __this_cpu_write(*p, x); =20 - if (IS_ENABLED(CONFIG_PREEMPT_RT)) - preempt_enable(); + preempt_enable_nested(); } EXPORT_SYMBOL(__mod_zone_page_state); =20 @@ -393,8 +391,7 @@ void __mod_node_page_state(struct pglist_data *pgdat, e= num node_stat_item item, } =20 /* See __mod_node_page_state */ - if (IS_ENABLED(CONFIG_PREEMPT_RT)) - preempt_disable(); + preempt_disable_nested(); =20 x =3D delta + __this_cpu_read(*p); =20 @@ -406,8 +403,7 @@ void __mod_node_page_state(struct pglist_data *pgdat, e= num node_stat_item item, } __this_cpu_write(*p, x); =20 - if (IS_ENABLED(CONFIG_PREEMPT_RT)) - preempt_enable(); + preempt_enable_nested(); } EXPORT_SYMBOL(__mod_node_page_state); =20 @@ -441,8 +437,7 @@ void __inc_zone_state(struct zone *zone, enum zone_stat= _item item) s8 v, t; =20 /* See __mod_node_page_state */ - if (IS_ENABLED(CONFIG_PREEMPT_RT)) - preempt_disable(); + preempt_disable_nested(); =20 v =3D __this_cpu_inc_return(*p); t =3D __this_cpu_read(pcp->stat_threshold); @@ -453,8 +448,7 @@ void __inc_zone_state(struct zone *zone, enum zone_stat= _item item) __this_cpu_write(*p, -overstep); } =20 - if (IS_ENABLED(CONFIG_PREEMPT_RT)) - preempt_enable(); + preempt_enable_nested(); } =20 void __inc_node_state(struct pglist_data *pgdat, enum node_stat_item item) @@ -466,8 +460,7 @@ void __inc_node_state(struct pglist_data *pgdat, enum n= ode_stat_item item) VM_WARN_ON_ONCE(vmstat_item_in_bytes(item)); =20 /* See __mod_node_page_state */ - if (IS_ENABLED(CONFIG_PREEMPT_RT)) - preempt_disable(); + preempt_disable_nested(); =20 v =3D __this_cpu_inc_return(*p); t =3D __this_cpu_read(pcp->stat_threshold); @@ -478,8 +471,7 @@ void __inc_node_state(struct pglist_data *pgdat, enum n= ode_stat_item item) __this_cpu_write(*p, -overstep); } =20 - if (IS_ENABLED(CONFIG_PREEMPT_RT)) - preempt_enable(); + preempt_enable_nested(); } =20 void __inc_zone_page_state(struct page *page, enum zone_stat_item item) @@ -501,8 +493,7 @@ void __dec_zone_state(struct zone *zone, enum zone_stat= _item item) s8 v, t; =20 /* See __mod_node_page_state */ - if (IS_ENABLED(CONFIG_PREEMPT_RT)) - preempt_disable(); + preempt_disable_nested(); =20 v =3D __this_cpu_dec_return(*p); t =3D __this_cpu_read(pcp->stat_threshold); @@ -513,8 +504,7 @@ void __dec_zone_state(struct zone *zone, enum zone_stat= _item item) __this_cpu_write(*p, overstep); } =20 - if (IS_ENABLED(CONFIG_PREEMPT_RT)) - preempt_enable(); + preempt_enable_nested(); } =20 void __dec_node_state(struct pglist_data *pgdat, enum node_stat_item item) @@ -526,8 +516,7 @@ void __dec_node_state(struct pglist_data *pgdat, enum n= ode_stat_item item) VM_WARN_ON_ONCE(vmstat_item_in_bytes(item)); =20 /* See __mod_node_page_state */ - if (IS_ENABLED(CONFIG_PREEMPT_RT)) - preempt_disable(); + preempt_disable_nested(); =20 v =3D __this_cpu_dec_return(*p); t =3D __this_cpu_read(pcp->stat_threshold); @@ -538,8 +527,7 @@ void __dec_node_state(struct pglist_data *pgdat, enum n= ode_stat_item item) __this_cpu_write(*p, overstep); } =20 - if (IS_ENABLED(CONFIG_PREEMPT_RT)) - preempt_enable(); + preempt_enable_nested(); } =20 void __dec_zone_page_state(struct page *page, enum zone_stat_item item) --=20 2.37.2 From nobody Sat Apr 11 00:44:32 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9DE40ECAA25 for ; Thu, 25 Aug 2022 16:42:24 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S239604AbiHYQmW (ORCPT ); Thu, 25 Aug 2022 12:42:22 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41538 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S242849AbiHYQmC (ORCPT ); Thu, 25 Aug 2022 12:42:02 -0400 Received: from galois.linutronix.de (Galois.linutronix.de [IPv6:2a0a:51c0:0:12e:550::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id BAEB4BA175 for ; Thu, 25 Aug 2022 09:41:54 -0700 (PDT) From: Sebastian Andrzej Siewior DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1661445701; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=/sTDmikhifvT+jyCYSjHJibnGmKic2QnilMTiV7/DbU=; b=obMubdUd6EYGlMUE+9dZPkyrahV0KRgisk2845KsBngVKkRtiS/Msafs4/lAf5+xakKTZ/ GF/JrVvH839LGCfDXeW7jRQH1sCUYlyLY9jRMxBcmm8Dejy2OsJQrmodMkwVQcftSEiRkW EEOvcB3OI/zRussbsWUnJgWhcmoAGZPbEi8kDl9CbpK3DIX4/nnk5IRaY1IEp0MA/+aRS7 WUnCopuwdPLuK8XTQH+j2Sgxsx+MnhGCAXz4I3H8y+aEF054nxFtl/wIFeoeK8RmFMXGLI v0eiMX71mbgaS6WpTMLEcDqeu0mTrd2ekCrmJTPJVcX4OVj17fWIGAMkQah5kQ== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1661445701; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=/sTDmikhifvT+jyCYSjHJibnGmKic2QnilMTiV7/DbU=; b=HuR+XllO9XxgzG30p0gBGiEYr0BVeZQinpJziXEn5D8nhFALUt02pQEhbjGpt6BqBFKpLY 2uMIOQAaGXq31vDA== To: linux-kernel@vger.kernel.org Cc: Thomas Gleixner , Peter Zijlstra , Steven Rostedt , Linus Torvalds , Matthew Wilcox , Andrew Morton , linux-mm@kvack.org, Sebastian Andrzej Siewior Subject: [PATCH v2 4/8] mm/debug: Provide VM_WARN_ON_IRQS_ENABLED() Date: Thu, 25 Aug 2022 18:41:27 +0200 Message-Id: <20220825164131.402717-5-bigeasy@linutronix.de> In-Reply-To: <20220825164131.402717-1-bigeasy@linutronix.de> References: <20220825164131.402717-1-bigeasy@linutronix.de> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Thomas Gleixner Some places in the VM code expect interrupts disabled, which is a valid expectation on non-PREEMPT_RT kernels, but does not hold on RT kernels in some places because the RT spinlock substitution does not disable interrupts. To avoid sprinkling CONFIG_PREEMPT_RT conditionals into those places, provide VM_WARN_ON_IRQS_ENABLED() which is only enabled when VM_DEBUG=3Dy a= nd PREEMPT_RT=3Dn. Signed-off-by: Thomas Gleixner Cc: Andrew Morton Cc: linux-mm@kvack.org Signed-off-by: Sebastian Andrzej Siewior Acked-by: Peter Zijlstra (Intel) Acked-by: Michal Hocko --- include/linux/mmdebug.h | 6 ++++++ lib/Kconfig.debug | 3 +++ 2 files changed, 9 insertions(+) diff --git a/include/linux/mmdebug.h b/include/linux/mmdebug.h index 15ae78cd28536..b8728d11c9490 100644 --- a/include/linux/mmdebug.h +++ b/include/linux/mmdebug.h @@ -94,6 +94,12 @@ void dump_mm(const struct mm_struct *mm); #define VM_WARN(cond, format...) BUILD_BUG_ON_INVALID(cond) #endif =20 +#ifdef CONFIG_DEBUG_VM_IRQSOFF +#define VM_WARN_ON_IRQS_ENABLED() WARN_ON_ONCE(!irqs_disabled()) +#else +#define VM_WARN_ON_IRQS_ENABLED() do { } while (0) +#endif + #ifdef CONFIG_DEBUG_VIRTUAL #define VIRTUAL_BUG_ON(cond) BUG_ON(cond) #else diff --git a/lib/Kconfig.debug b/lib/Kconfig.debug index 072e4b289c13e..c96fc6820544c 100644 --- a/lib/Kconfig.debug +++ b/lib/Kconfig.debug @@ -803,6 +803,9 @@ config ARCH_HAS_DEBUG_VM_PGTABLE An architecture should select this when it can successfully build and run DEBUG_VM_PGTABLE. =20 +config DEBUG_VM_IRQSOFF + def_bool DEBUG_VM && !PREEMPT_RT + config DEBUG_VM bool "Debug VM" depends on DEBUG_KERNEL --=20 2.37.2 From nobody Sat Apr 11 00:44:32 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4CEFDECAA25 for ; Thu, 25 Aug 2022 16:42:35 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S243203AbiHYQmd (ORCPT ); Thu, 25 Aug 2022 12:42:33 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42276 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S242791AbiHYQmF (ORCPT ); Thu, 25 Aug 2022 12:42:05 -0400 Received: from galois.linutronix.de (Galois.linutronix.de [IPv6:2a0a:51c0:0:12e:550::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C3585B9FBA; Thu, 25 Aug 2022 09:42:00 -0700 (PDT) From: Sebastian Andrzej Siewior DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1661445701; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=rBDIufscYHVsHmPufaPAl/Vqv2LLB/GPVe+GUwmXUq8=; b=dwt4gAjuVE7mnQU8NsSY1OdNQ0rEZBIOo7TJBp+JnuKFR/KIJGimsOnnoMrAmxV7Ig+JT7 O249Q23IW3M+loKcYaWWeCCbHsG9iBRknQ60mgpEXPa4N+Eode7esLr9CsWoLpckIV1KzI ayi8JaUVrbuLHKvfTbkZw3UEWBw409RJLoheWD5IzDo/LyyVLw6mJMQH2W1WsMcD3Dh2VU y0AoX+D8MvMAnXdwKQNdlX2Uq1K1KYhECqVDrC126boDaX4wUq6U9kgnOZqzw/EZZ46Kw8 ahdSsDX24EgjlJ4uN9GjqVIcF/W2re4OGiNrAhu80st4X6zaYkG+ENS6mRRj9g== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1661445701; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=rBDIufscYHVsHmPufaPAl/Vqv2LLB/GPVe+GUwmXUq8=; b=/xWS01iGeCi7ocDyLNA4wcA1jGESk+49fcLFDvz6piEFCTHy/Cap5+B9ZdmwZa5+akYuxh YMeWgbw07cXCjbBw== To: linux-kernel@vger.kernel.org Cc: Thomas Gleixner , Peter Zijlstra , Steven Rostedt , Linus Torvalds , Matthew Wilcox , Johannes Weiner , Michal Hocko , Roman Gushchin , Shakeel Butt , Muchun Song , cgroups@vger.kernel.org, linux-mm@kvack.org, Sebastian Andrzej Siewior Subject: [PATCH v2 5/8] mm/memcontrol: Replace the PREEMPT_RT conditionals Date: Thu, 25 Aug 2022 18:41:28 +0200 Message-Id: <20220825164131.402717-6-bigeasy@linutronix.de> In-Reply-To: <20220825164131.402717-1-bigeasy@linutronix.de> References: <20220825164131.402717-1-bigeasy@linutronix.de> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Thomas Gleixner Use VM_WARN_ON_IRQS_ENABLED() and preempt_disable/enable_nested() to replace the CONFIG_PREEMPT_RT #ifdeffery. Signed-off-by: Thomas Gleixner Cc: Johannes Weiner Cc: Michal Hocko Cc: Roman Gushchin Cc: Shakeel Butt Cc: Muchun Song Cc: cgroups@vger.kernel.org Cc: linux-mm@kvack.org Signed-off-by: Sebastian Andrzej Siewior Acked-by: Johannes Weiner Reviewed-by: Muchun Song Acked-by: Peter Zijlstra (Intel) --- mm/memcontrol.c | 19 ++++++------------- 1 file changed, 6 insertions(+), 13 deletions(-) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index b69979c9ced5c..d35b6fa560f0a 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -597,25 +597,18 @@ static u64 flush_next_time; */ static void memcg_stats_lock(void) { -#ifdef CONFIG_PREEMPT_RT - preempt_disable(); -#else - VM_BUG_ON(!irqs_disabled()); -#endif + preempt_disable_nested(); + VM_WARN_ON_IRQS_ENABLED(); } =20 static void __memcg_stats_lock(void) { -#ifdef CONFIG_PREEMPT_RT - preempt_disable(); -#endif + preempt_disable_nested(); } =20 static void memcg_stats_unlock(void) { -#ifdef CONFIG_PREEMPT_RT - preempt_enable(); -#endif + preempt_enable_nested(); } =20 static inline void memcg_rstat_updated(struct mem_cgroup *memcg, int val) @@ -715,7 +708,7 @@ void __mod_memcg_lruvec_state(struct lruvec *lruvec, en= um node_stat_item idx, * interrupt context while other caller need to have disabled interrupt. */ __memcg_stats_lock(); - if (IS_ENABLED(CONFIG_DEBUG_VM) && !IS_ENABLED(CONFIG_PREEMPT_RT)) { + if (IS_ENABLED(CONFIG_DEBUG_VM)) { switch (idx) { case NR_ANON_MAPPED: case NR_FILE_MAPPED: @@ -725,7 +718,7 @@ void __mod_memcg_lruvec_state(struct lruvec *lruvec, en= um node_stat_item idx, WARN_ON_ONCE(!in_task()); break; default: - WARN_ON_ONCE(!irqs_disabled()); + VM_WARN_ON_IRQS_ENABLED(); } } =20 --=20 2.37.2 From nobody Sat Apr 11 00:44:32 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id A8E4CECAA25 for ; Thu, 25 Aug 2022 16:42:38 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S243208AbiHYQmh (ORCPT ); Thu, 25 Aug 2022 12:42:37 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42536 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230072AbiHYQmF (ORCPT ); Thu, 25 Aug 2022 12:42:05 -0400 Received: from galois.linutronix.de (Galois.linutronix.de [IPv6:2a0a:51c0:0:12e:550::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C3752BB026 for ; Thu, 25 Aug 2022 09:42:00 -0700 (PDT) From: Sebastian Andrzej Siewior DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1661445702; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=vNtMvikVP3A7JqyI64O+1a12M8t0vOpcITdU5U0TGnE=; b=1tbpx1sbmfdpwCzS9EIDtkWSVCGguf5I2hLDQGB3gqxy3NXTtgqxr/OyUHEVyLoWq3g+P+ uLfKJpOAWO7ON0OCmzt/gOS53iP8p88HEzAMeM0nfm02bLRVp3g1wSqC6DGK639HTZVca/ JgYApAw08cAzR11s+MES0bwsTZ0DDw6Tms9QX2ubEloCEvIkqQzXebuMe6ZSiYIsWJTwrK BoWu325TadyR5PkTVpwGarSfJURzf76C+VqcMoVVxUWCcgTKt3xE1cz/0uwZTvgUPYY7nT syGRgrGaTPLKYVFR1zfTNxCgXFsCiV0tf1g+P2nyTtUlEOHbZajGvExAn9TcFg== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1661445702; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=vNtMvikVP3A7JqyI64O+1a12M8t0vOpcITdU5U0TGnE=; b=05zUufWP4OOvhby/9h51rilXjm5mAeXZOKAZ+CURv0RQdj2qbxL8qBxvKw0Kr/LwMwpHpa 6RN8dg8XyVjM5qCg== To: linux-kernel@vger.kernel.org Cc: Thomas Gleixner , Peter Zijlstra , Steven Rostedt , Linus Torvalds , Matthew Wilcox , Andrew Morton , Nick Terrell , linux-mm@kvack.org, Sebastian Andrzej Siewior Subject: [PATCH v2 6/8] mm/compaction: Get rid of RT ifdeffery Date: Thu, 25 Aug 2022 18:41:29 +0200 Message-Id: <20220825164131.402717-7-bigeasy@linutronix.de> In-Reply-To: <20220825164131.402717-1-bigeasy@linutronix.de> References: <20220825164131.402717-1-bigeasy@linutronix.de> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Thomas Gleixner Move the RT dependency for the initial value of sysctl_compact_unevictable_allowed into Kconfig. Signed-off-by: Thomas Gleixner Cc: Andrew Morton Cc: Nick Terrell Cc: linux-mm@kvack.org Signed-off-by: Sebastian Andrzej Siewior Acked-by: Peter Zijlstra (Intel) --- mm/Kconfig | 6 ++++++ mm/compaction.c | 6 +----- 2 files changed, 7 insertions(+), 5 deletions(-) diff --git a/mm/Kconfig b/mm/Kconfig index 0331f1461f81c..3897e924e40f2 100644 --- a/mm/Kconfig +++ b/mm/Kconfig @@ -579,6 +579,12 @@ config COMPACTION it and then we would be really interested to hear about that at linux-mm@kvack.org. =20 +config COMPACT_UNEVICTABLE_DEFAULT + int + depends on COMPACTION + default 0 if PREEMPT_RT + default 1 + # # support for free page reporting config PAGE_REPORTING diff --git a/mm/compaction.c b/mm/compaction.c index 640fa76228dd9..10561cb1aaad9 100644 --- a/mm/compaction.c +++ b/mm/compaction.c @@ -1727,11 +1727,7 @@ typedef enum { * Allow userspace to control policy on scanning the unevictable LRU for * compactable pages. */ -#ifdef CONFIG_PREEMPT_RT -int sysctl_compact_unevictable_allowed __read_mostly =3D 0; -#else -int sysctl_compact_unevictable_allowed __read_mostly =3D 1; -#endif +int sysctl_compact_unevictable_allowed __read_mostly =3D CONFIG_COMPACT_UN= EVICTABLE_DEFAULT; =20 static inline void update_fast_start_pfn(struct compact_control *cc, unsigned long pfn) --=20 2.37.2 From nobody Sat Apr 11 00:44:32 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 971A8ECAA25 for ; Thu, 25 Aug 2022 16:42:49 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S242944AbiHYQms (ORCPT ); Thu, 25 Aug 2022 12:42:48 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42186 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234722AbiHYQmK (ORCPT ); Thu, 25 Aug 2022 12:42:10 -0400 Received: from galois.linutronix.de (Galois.linutronix.de [IPv6:2a0a:51c0:0:12e:550::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 90163BB038 for ; Thu, 25 Aug 2022 09:42:05 -0700 (PDT) From: Sebastian Andrzej Siewior DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1661445702; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=UZ0gImFj9vNb9oCghFEFyZK2gmeHgP1Q4vEv6+ZBTOo=; b=iZPGCTy46sVqGQTmbG+4Y+5JUk2LCMCns27UxDt2NcGkCvR86mDvUT5mf7UHORjsDHXnEb dbSvanfsvEh8sNGDjLVFm7HDYcPbErT/madlxlWk94fWVSWGm78tu2jqpj2Aa0UGhD7FNb bYt9NFkgYpQQrfzexIL3SOlbm0f12LRD/wcZCMXB92dUWi38W7918IYoxza7wUKVUr3gE/ mkR1SQObdAsGtoqcu+Yf5u8rZN0ZqxSafA7PORNYXClXKeoZq1BldrO9Mf0bFCilK+k8IZ oY8mBeN10xdLin4VnWWIGEai4rZEo7+gZfV6wl+7UmNCZEfE0tvQQJm3Zdkpcw== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1661445702; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=UZ0gImFj9vNb9oCghFEFyZK2gmeHgP1Q4vEv6+ZBTOo=; b=uzVcIXdcn/xNA2DNNwNkBxw35ISWhO/M9DgMSdVRoz5tw0i3FBmz9D7aRxiTgMNLUN8Z/5 kUWqWgKJerzA5UDg== To: linux-kernel@vger.kernel.org Cc: Thomas Gleixner , Peter Zijlstra , Steven Rostedt , Linus Torvalds , Matthew Wilcox , Sebastian Andrzej Siewior Subject: [PATCH v2 7/8] flex_proportions: Disable preemption entering the write section. Date: Thu, 25 Aug 2022 18:41:30 +0200 Message-Id: <20220825164131.402717-8-bigeasy@linutronix.de> In-Reply-To: <20220825164131.402717-1-bigeasy@linutronix.de> References: <20220825164131.402717-1-bigeasy@linutronix.de> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" The seqcount fprop_global::sequence is not associated with a lock. The write section (fprop_new_period()) is invoked from a timer and since the softirq is preemptible on PREEMPT_RT it is possible to preempt the write section which is not desited. Disable preemption around the write section on PREEMPT_RT. Signed-off-by: Sebastian Andrzej Siewior --- lib/flex_proportions.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/lib/flex_proportions.c b/lib/flex_proportions.c index 05cccbcf1661a..83332fefa6f42 100644 --- a/lib/flex_proportions.c +++ b/lib/flex_proportions.c @@ -70,6 +70,7 @@ bool fprop_new_period(struct fprop_global *p, int periods) */ if (events <=3D 1) return false; + preempt_disable_nested(); write_seqcount_begin(&p->sequence); if (periods < 64) events -=3D events >> periods; @@ -77,6 +78,7 @@ bool fprop_new_period(struct fprop_global *p, int periods) percpu_counter_add(&p->events, -events); p->period +=3D periods; write_seqcount_end(&p->sequence); + preempt_enable_nested(); =20 return true; } --=20 2.37.2 From nobody Sat Apr 11 00:44:32 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id F34CEECAA24 for ; Thu, 25 Aug 2022 16:42:52 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S243174AbiHYQmv (ORCPT ); Thu, 25 Aug 2022 12:42:51 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41446 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S243126AbiHYQmL (ORCPT ); Thu, 25 Aug 2022 12:42:11 -0400 Received: from galois.linutronix.de (Galois.linutronix.de [193.142.43.55]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id CB34FBA9EF; Thu, 25 Aug 2022 09:42:05 -0700 (PDT) From: Sebastian Andrzej Siewior DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1661445703; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=H7Xgde3Wpvu3ATr43T07ctyIs66Oc/APVwlDj/xUkBg=; b=XjkBzXKjrNu+Yx3g3d9gPf7KU3zOTua1gYU3Al/TTA53koMgM+lASUasrviGClpWVNHU7Q aHGfnDp+QCzOAiGsYViFdRhlSxQiLAlIvnbhTIrVm9wgzHwFdb6Qo4PiO3WHgPIOjnRxzC 4S9ScW6sRblKEfnTpcJtW8f7P/K3Dcwjyk6aGJGgy0hJfvv4axycOyMclcTcoylyA7hBsJ UCFUj4DoTYT30yfwnoQ5ieKEvQj8uvMotuwwfg52waihZVSI+bUp0Ovok7QaUv1jB4eQ4+ HYAHuvANiZICVQ/M2GGAWZl/+/59iKtjX5XtqbElDgdZP/S0/pp2VfP+16IT6Q== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1661445703; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=H7Xgde3Wpvu3ATr43T07ctyIs66Oc/APVwlDj/xUkBg=; b=mDa9KqiZ0A3+AN3G8BFBxOohOqFy83OuqmHn5CZmo9HOyUJC0PdO4BxQHfRdcbWaQZTzzy DEuquLju2SDg2oCA== To: linux-kernel@vger.kernel.org Cc: Thomas Gleixner , Peter Zijlstra , Steven Rostedt , Linus Torvalds , Matthew Wilcox , netdev@vger.kernel.org, Sebastian Andrzej Siewior Subject: [PATCH v2 8/8] u64_stats: Streamline the implementation Date: Thu, 25 Aug 2022 18:41:31 +0200 Message-Id: <20220825164131.402717-9-bigeasy@linutronix.de> In-Reply-To: <20220825164131.402717-1-bigeasy@linutronix.de> References: <20220825164131.402717-1-bigeasy@linutronix.de> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Thomas Gleixner The u64 stats code handles 3 different cases: - 32bit UP - 32bit SMP - 64bit with an unreadable #ifdef maze, which was recently expanded with PREEMPT_RT conditionals. Reduce it to two cases (32bit and 64bit) and drop the optimization for 32bit UP as suggested by Linus. Use the new preempt_disable/enable_nested() helpers to get rid of the CONFIG_PREEMPT_RT conditionals. Signed-off-by: Thomas Gleixner Cc: netdev@vger.kernel.org Signed-off-by: Sebastian Andrzej Siewior Acked-by: Peter Zijlstra (Intel) --- include/linux/u64_stats_sync.h | 147 +++++++++++++++------------------ 1 file changed, 65 insertions(+), 82 deletions(-) diff --git a/include/linux/u64_stats_sync.h b/include/linux/u64_stats_sync.h index 6ad4e9032d538..46040d66334a8 100644 --- a/include/linux/u64_stats_sync.h +++ b/include/linux/u64_stats_sync.h @@ -8,7 +8,7 @@ * * Key points : * - * - Use a seqcount on 32-bit SMP, only disable preemption for 32-bit UP. + * - Use a seqcount on 32-bit * - The whole thing is a no-op on 64-bit architectures. * * Usage constraints: @@ -20,7 +20,8 @@ * writer and also spin forever. * * 3) Write side must use the _irqsave() variant if other writers, or a re= ader, - * can be invoked from an IRQ context. + * can be invoked from an IRQ context. On 64bit systems this variant do= es not + * disable interrupts. * * 4) If reader fetches several counters, there is no guarantee the whole = values * are consistent w.r.t. each other (remember point #2: seqcounts are n= ot @@ -29,11 +30,6 @@ * 5) Readers are allowed to sleep or be preempted/interrupted: they perfo= rm * pure reads. * - * 6) Readers must use both u64_stats_fetch_{begin,retry}_irq() if the sta= ts - * might be updated from a hardirq or softirq context (remember point #= 1: - * seqcounts are not used for UP kernels). 32-bit UP stat readers could= read - * corrupted 64-bit values otherwise. - * * Usage : * * Stats producer (writer) should use following template granted it alread= y got @@ -66,7 +62,7 @@ #include =20 struct u64_stats_sync { -#if BITS_PER_LONG =3D=3D 32 && (defined(CONFIG_SMP) || defined(CONFIG_PREE= MPT_RT)) +#if BITS_PER_LONG =3D=3D 32 seqcount_t seq; #endif }; @@ -98,7 +94,22 @@ static inline void u64_stats_inc(u64_stats_t *p) local64_inc(&p->v); } =20 -#else +static inline void u64_stats_init(struct u64_stats_sync *syncp) { } +static inline void __u64_stats_update_begin(struct u64_stats_sync *syncp) = { } +static inline void __u64_stats_update_end(struct u64_stats_sync *syncp) { } +static inline unsigned long __u64_stats_irqsave(void) { return 0; } +static inline void __u64_stats_irqrestore(unsigned long flags) { } +static inline unsigned int __u64_stats_fetch_begin(const struct u64_stats_= sync *syncp) +{ + return 0; +} +static inline bool __u64_stats_fetch_retry(const struct u64_stats_sync *sy= ncp, + unsigned int start) +{ + return false; +} + +#else /* 64 bit */ =20 typedef struct { u64 v; @@ -123,123 +134,95 @@ static inline void u64_stats_inc(u64_stats_t *p) { p->v++; } -#endif =20 -#if BITS_PER_LONG =3D=3D 32 && (defined(CONFIG_SMP) || defined(CONFIG_PREE= MPT_RT)) -#define u64_stats_init(syncp) seqcount_init(&(syncp)->seq) -#else static inline void u64_stats_init(struct u64_stats_sync *syncp) { + seqcount_init(&syncp->seq); } -#endif =20 -static inline void u64_stats_update_begin(struct u64_stats_sync *syncp) +static inline void __u64_stats_update_begin(struct u64_stats_sync *syncp) { -#if BITS_PER_LONG =3D=3D 32 && (defined(CONFIG_SMP) || defined(CONFIG_PREE= MPT_RT)) - if (IS_ENABLED(CONFIG_PREEMPT_RT)) - preempt_disable(); + preempt_disable_nested(); write_seqcount_begin(&syncp->seq); -#endif } =20 -static inline void u64_stats_update_end(struct u64_stats_sync *syncp) +static inline void __u64_stats_update_end(struct u64_stats_sync *syncp) { -#if BITS_PER_LONG =3D=3D 32 && (defined(CONFIG_SMP) || defined(CONFIG_PREE= MPT_RT)) write_seqcount_end(&syncp->seq); - if (IS_ENABLED(CONFIG_PREEMPT_RT)) - preempt_enable(); -#endif + preempt_enable_nested(); } =20 -static inline unsigned long -u64_stats_update_begin_irqsave(struct u64_stats_sync *syncp) +static inline unsigned long __u64_stats_irqsave(void) { - unsigned long flags =3D 0; + unsigned long flags; =20 -#if BITS_PER_LONG =3D=3D 32 && (defined(CONFIG_SMP) || defined(CONFIG_PREE= MPT_RT)) - if (IS_ENABLED(CONFIG_PREEMPT_RT)) - preempt_disable(); - else - local_irq_save(flags); - write_seqcount_begin(&syncp->seq); -#endif + local_irq_save(flags); return flags; } =20 -static inline void -u64_stats_update_end_irqrestore(struct u64_stats_sync *syncp, - unsigned long flags) +static inline void __u64_stats_irqrestore(unsigned long flags) { -#if BITS_PER_LONG =3D=3D 32 && (defined(CONFIG_SMP) || defined(CONFIG_PREE= MPT_RT)) - write_seqcount_end(&syncp->seq); - if (IS_ENABLED(CONFIG_PREEMPT_RT)) - preempt_enable(); - else - local_irq_restore(flags); -#endif + local_irq_restore(flags); } =20 static inline unsigned int __u64_stats_fetch_begin(const struct u64_stats_= sync *syncp) { -#if BITS_PER_LONG =3D=3D 32 && (defined(CONFIG_SMP) || defined(CONFIG_PREE= MPT_RT)) return read_seqcount_begin(&syncp->seq); -#else - return 0; -#endif +} + +static inline bool __u64_stats_fetch_retry(const struct u64_stats_sync *sy= ncp, + unsigned int start) +{ + return read_seqcount_retry(&syncp->seq, start); +} +#endif /* !64 bit */ + +static inline void u64_stats_update_begin(struct u64_stats_sync *syncp) +{ + __u64_stats_update_begin(syncp); +} + +static inline void u64_stats_update_end(struct u64_stats_sync *syncp) +{ + __u64_stats_update_end(syncp); +} + +static inline unsigned long u64_stats_update_begin_irqsave(struct u64_stat= s_sync *syncp) +{ + unsigned long flags =3D __u64_stats_irqsave(); + + __u64_stats_update_begin(syncp); + return flags; +} + +static inline void u64_stats_update_end_irqrestore(struct u64_stats_sync *= syncp, + unsigned long flags) +{ + __u64_stats_update_end(syncp); + __u64_stats_irqrestore(flags); } =20 static inline unsigned int u64_stats_fetch_begin(const struct u64_stats_sy= nc *syncp) { -#if BITS_PER_LONG =3D=3D 32 && (!defined(CONFIG_SMP) && !defined(CONFIG_PR= EEMPT_RT)) - preempt_disable(); -#endif return __u64_stats_fetch_begin(syncp); } =20 -static inline bool __u64_stats_fetch_retry(const struct u64_stats_sync *sy= ncp, - unsigned int start) -{ -#if BITS_PER_LONG =3D=3D 32 && (defined(CONFIG_SMP) || defined(CONFIG_PREE= MPT_RT)) - return read_seqcount_retry(&syncp->seq, start); -#else - return false; -#endif -} - static inline bool u64_stats_fetch_retry(const struct u64_stats_sync *sync= p, unsigned int start) { -#if BITS_PER_LONG =3D=3D 32 && (!defined(CONFIG_SMP) && !defined(CONFIG_PR= EEMPT_RT)) - preempt_enable(); -#endif return __u64_stats_fetch_retry(syncp, start); } =20 -/* - * In case irq handlers can update u64 counters, readers can use following= helpers - * - SMP 32bit arches use seqcount protection, irq safe. - * - UP 32bit must disable irqs. - * - 64bit have no problem atomically reading u64 values, irq safe. - */ +/* Obsolete interfaces */ static inline unsigned int u64_stats_fetch_begin_irq(const struct u64_stat= s_sync *syncp) { -#if BITS_PER_LONG =3D=3D 32 && defined(CONFIG_PREEMPT_RT) - preempt_disable(); -#elif BITS_PER_LONG =3D=3D 32 && !defined(CONFIG_SMP) - local_irq_disable(); -#endif - return __u64_stats_fetch_begin(syncp); + return u64_stats_fetch_begin(syncp); } =20 static inline bool u64_stats_fetch_retry_irq(const struct u64_stats_sync *= syncp, unsigned int start) { -#if BITS_PER_LONG =3D=3D 32 && defined(CONFIG_PREEMPT_RT) - preempt_enable(); -#elif BITS_PER_LONG =3D=3D 32 && !defined(CONFIG_SMP) - local_irq_enable(); -#endif - return __u64_stats_fetch_retry(syncp, start); + return u64_stats_fetch_retry(syncp, start); } =20 #endif /* _LINUX_U64_STATS_SYNC_H */ --=20 2.37.2