From nobody Thu Sep 24 20:20:59 2026 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) client-ip=192.237.175.120; envelope-from=xen-devel-bounces@lists.xenproject.org; helo=lists.xenproject.org; Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=pass(p=none dis=none) header.from=xenproject.org ARC-Seal: i=1; a=rsa-sha256; t=1788542084; cv=none; d=zohomail.com; s=zohoarc; b=B16a09hu1GUvxWGJT1iclRIF5DejDDPJ3EYRNtKLENRWrwreRJg/vkft3HkXEO2kwM3Mh3feKXtWh29Eog3CPBxzhPeCaczlsZCXoKRXMae6xCxF+wWrnmiUvAoP7d/AjKzMOpGLWBi6jCf/fRyGLVvK4TKE/jrcMqanqRbRduE= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1788542084; h=Content-Type:Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=5FAZdrD/CSlhzCnqj5dZaUZDZ9GfXXAxevwlxaPWsI4=; b=je3JXI0u5/lQ76XBIfdlLh+XDHtsNh0GX9QEDrwyQAkrEQywR4PZLsq+kp7u/K23mVI6UImYrxGOyB/jBt0ff5/C1COoy9X3wFmGMjhhrxAVUuk0NYifArpwVQ6K/1jPMCJP1MPzc95qZicqIOq7gcf+wezP0plE8PlSrTDBhD0= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) by mx.zohomail.com with SMTPS id 1788542084336409.41516723468067; Fri, 4 Sep 2026 10:14:44 -0700 (PDT) Received: from list by lists.xenproject.org with outflank-mailman.1408902.1641173 (Exim 4.92) (envelope-from ) id 1x2XUe-0003BP-UA; Fri, 04 Sep 2026 17:14:24 +0000 Received: by outflank-mailman (output) from mailman id 1408902.1641173; Fri, 04 Sep 2026 17:14:24 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1x2XUe-0003BI-RO; Fri, 04 Sep 2026 17:14:24 +0000 Received: by outflank-mailman (input) for mailman id 1408902; Fri, 04 Sep 2026 17:14:23 +0000 Received: from mail.xenproject.org ([104.130.215.37]) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1x2XUd-0003B4-L4 for xen-devel@lists.xenproject.org; Fri, 04 Sep 2026 17:14:23 +0000 Received: from xenbits.xenproject.org ([104.239.192.120]) by mail.xenproject.org with esmtp (Exim 4.96) (envelope-from ) id 1x2XUc-005Kk8-1v; Fri, 04 Sep 2026 17:14:22 +0000 Received: from 224.pool85-54-217.dynamic.orange.es ([85.54.217.224] helo=localhost) by xenbits.xenproject.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.96) (envelope-from ) id 1x2XUc-000PXA-02; Fri, 04 Sep 2026 17:14:22 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=xenproject.org; s=20200302mail; h=Content-Transfer-Encoding:Content-Type: MIME-Version:References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From; bh=5FAZdrD/CSlhzCnqj5dZaUZDZ9GfXXAxevwlxaPWsI4=; b=uzdZ741gH0rcrN+RgRlzc39Q8t z6dnLjO5DoDWktcrogGnvXLbTTU/kjCv0A3dXJeQD3EIr5h8JjaeBNnpugHP33AMcCh/O9/YxUe1i FzYl9WJ70WuFLJL62foQJqXPRaJGZ6ocmia1T86q0LhWNQviCFLMn1BV6ijKlYPy2aBg=; From: Roger Pau Monne To: xen-devel@lists.xenproject.org Cc: =?UTF-8?q?J=C3=BCrgen=20Gro=C3=9F?= , Roger Pau Monne , Andrew Cooper , Anthony PERARD , Michal Orzel , Jan Beulich , Julien Grall , Stefano Stabellini Subject: [PATCH 1/5] xen/rcu: fix types Date: Fri, 4 Sep 2026 19:11:17 +0200 Message-ID: <20260904171121.65300-2-roger@xenproject.org> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260904171121.65300-1-roger@xenproject.org> References: <20260904171121.65300-1-roger@xenproject.org> MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-ZohoMail-DKIM: pass (identity @xenproject.org) X-ZM-MESSAGEID: 1788542086159158500 Adjust some types: int -> bool, int -> unsigned int. Also fix a couple of typos plus trailing white space. No functional change intended. Signed-off-by: Roger Pau Monn=C3=A9 Acked-by: Andrew Cooper --- xen/common/rcupdate.c | 12 ++++++------ xen/include/xen/rcupdate.h | 8 ++++---- 2 files changed, 10 insertions(+), 10 deletions(-) diff --git a/xen/common/rcupdate.c b/xen/common/rcupdate.c index fd5d3d7484a5..3b96f829c87c 100644 --- a/xen/common/rcupdate.c +++ b/xen/common/rcupdate.c @@ -509,18 +509,18 @@ static int __rcu_pending(struct rcu_ctrlblk *rcp, str= uct rcu_data *rdp) return 0; } =20 -int rcu_pending(int cpu) +bool rcu_pending(unsigned int cpu) { - return __rcu_pending(&rcu_ctrlblk, &per_cpu(rcu_data, cpu)); + return !!__rcu_pending(&rcu_ctrlblk, &per_cpu(rcu_data, cpu)); } =20 /* * Check to see if any future RCU-related work will need to be done * by the current CPU, even if none need be done immediately, returning - * 1 if so. This function is part of the RCU implementation; it is -not- + * true if so. This function is part of the RCU implementation; it is -no= t- * an exported member of the RCU API. */ -int rcu_needs_cpu(int cpu) +bool rcu_needs_cpu(unsigned int cpu) { struct rcu_data *rdp =3D &per_cpu(rcu_data, cpu); =20 @@ -529,7 +529,7 @@ int rcu_needs_cpu(int cpu) =20 /* * Timer for making sure the CPU where a callback is queued does - * periodically poke rcu_pedning(), so that it will invoke the callback + * periodically poke rcu_pending(), so that it will invoke the callback * not too late after the end of the grace period. */ static void rcu_idle_timer_start(void) @@ -588,7 +588,7 @@ static void cf_check rcu_idle_timer_handler(void* data) IDLE_TIMER_PERIOD_MIN); } =20 -void rcu_check_callbacks(int cpu) +void rcu_check_callbacks(unsigned int cpu) { struct rcu_data *rdp =3D &this_cpu(rcu_data); =20 diff --git a/xen/include/xen/rcupdate.h b/xen/include/xen/rcupdate.h index 95f4ad81c4a8..c57f628107cf 100644 --- a/xen/include/xen/rcupdate.h +++ b/xen/include/xen/rcupdate.h @@ -77,8 +77,8 @@ struct rcu_head { } while (0) =20 =20 -int rcu_pending(int cpu); -int rcu_needs_cpu(int cpu); +bool rcu_pending(unsigned int cpu); +bool rcu_needs_cpu(unsigned int cpu); =20 /* * Dummy lock type for passing to rcu_read_{lock,unlock}. Currently exists @@ -168,10 +168,10 @@ static inline void rcu_read_unlock(rcu_read_lock_t *l= ock) #define rcu_assign_pointer(p, v) ({ smp_wmb(); (p) =3D (v); }) =20 void rcu_init(void); -void rcu_check_callbacks(int cpu); +void rcu_check_callbacks(unsigned int cpu); =20 /* Exported interfaces */ -void call_rcu(struct rcu_head *head,=20 +void call_rcu(struct rcu_head *head, void (*func)(struct rcu_head *head)); =20 void rcu_barrier(void); --=20 2.53.0 From nobody Thu Sep 24 20:20:59 2026 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) client-ip=192.237.175.120; envelope-from=xen-devel-bounces@lists.xenproject.org; helo=lists.xenproject.org; Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=pass(p=none dis=none) header.from=xenproject.org ARC-Seal: i=1; a=rsa-sha256; t=1788542086; cv=none; d=zohomail.com; s=zohoarc; b=bP5E7zjuIlWOaNzE/Koc4fHpoIR8pFKmvNYDQLz6XdoVjpCxoSxFJmk4v9KVzSr8Uky+xu2uQJquCBubCwgfyODPVDx0BIcM1xayc9/ZRf4q6d4kZeWQrDTr6Im12zTrGwsHku67j9s+EeoQgzZv7cJZ6MSDA8+1jocSFTahr6M= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1788542086; h=Content-Type:Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=2HiqIR5wDrd1IuylDmsDPpeiruy+pLJWmkc28OV2WQ8=; b=YF0i3T7C9MdFUy0UMOY0y3ohm94ZRuhCDm/m7BuB00mtj8/0fu9r7GJ4i9gZmfACHoZcWm+bhQYo7TEzWPdJUWZ2rZDLEZ0x3e8ng9Mm29LFLvjkPspJMKcqctmfTZkEe25OXZuSEv6mW8b6XA8lKjtY7amp34hZYACaGx0tCvI= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) by mx.zohomail.com with SMTPS id 1788542086059488.8469434158751; Fri, 4 Sep 2026 10:14:46 -0700 (PDT) Received: from list by lists.xenproject.org with outflank-mailman.1408903.1641182 (Exim 4.92) (envelope-from ) id 1x2XUh-0003PO-8X; Fri, 04 Sep 2026 17:14:27 +0000 Received: by outflank-mailman (output) from mailman id 1408903.1641182; Fri, 04 Sep 2026 17:14:27 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1x2XUh-0003PE-2j; Fri, 04 Sep 2026 17:14:27 +0000 Received: by outflank-mailman (input) for mailman id 1408903; Fri, 04 Sep 2026 17:14:25 +0000 Received: from mail.xenproject.org ([104.130.215.37]) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1x2XUf-0003MH-Oo for xen-devel@lists.xenproject.org; Fri, 04 Sep 2026 17:14:25 +0000 Received: from xenbits.xenproject.org ([104.239.192.120]) by mail.xenproject.org with esmtp (Exim 4.96) (envelope-from ) id 1x2XUe-005KkI-2V; Fri, 04 Sep 2026 17:14:24 +0000 Received: from 224.pool85-54-217.dynamic.orange.es ([85.54.217.224] helo=localhost) by xenbits.xenproject.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.96) (envelope-from ) id 1x2XUe-000PhT-0i; Fri, 04 Sep 2026 17:14:24 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=xenproject.org; s=20200302mail; h=Content-Transfer-Encoding:Content-Type: MIME-Version:References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From; bh=2HiqIR5wDrd1IuylDmsDPpeiruy+pLJWmkc28OV2WQ8=; b=hp6HIua5hoLvQLeIHvXYsd+av9 X8SygqolTaBWogbx/59xIJKuNQPpWBqxlV/e4Faow3cwsOYnxtgRqf9HbE0NPOUplm+jsBPLsp8Vw 6kyCWj6WVzDBdvB0oVVz43SfhiifRbRnc/iOrnOGXIYbzSQJx5qMFnyIlqpjTBK3shMg=; From: Roger Pau Monne To: xen-devel@lists.xenproject.org Cc: =?UTF-8?q?J=C3=BCrgen=20Gro=C3=9F?= , Roger Pau Monne , Andrew Cooper , Anthony PERARD , Michal Orzel , Jan Beulich , Julien Grall , Stefano Stabellini Subject: [PATCH 2/5] xen/rcu: sort includes Date: Fri, 4 Sep 2026 19:11:18 +0200 Message-ID: <20260904171121.65300-3-roger@xenproject.org> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260904171121.65300-1-roger@xenproject.org> References: <20260904171121.65300-1-roger@xenproject.org> MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-ZohoMail-DKIM: pass (identity @xenproject.org) X-ZM-MESSAGEID: 1788542088011158500 Signed-off-by: Roger Pau Monn=C3=A9 Acked-by: Andrew Cooper --- xen/common/rcupdate.c | 19 ++++++++++--------- 1 file changed, 10 insertions(+), 9 deletions(-) diff --git a/xen/common/rcupdate.c b/xen/common/rcupdate.c index 3b96f829c87c..c1b6b2ae768b 100644 --- a/xen/common/rcupdate.c +++ b/xen/common/rcupdate.c @@ -31,21 +31,22 @@ * For detailed explanation of Read-Copy Update mechanism see - * http://lse.sourceforge.net/locking/rcupdate.html */ -#include -#include +#include +#include #include +#include #include -#include -#include -#include +#include #include #include -#include -#include -#include +#include +#include #include -#include +#include #include +#include + +#include =20 DEFINE_PER_CPU(unsigned int, rcu_lock_cnt); =20 --=20 2.53.0 From nobody Thu Sep 24 20:20:59 2026 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) client-ip=192.237.175.120; envelope-from=xen-devel-bounces@lists.xenproject.org; helo=lists.xenproject.org; Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=pass(p=none dis=none) header.from=xenproject.org ARC-Seal: i=1; a=rsa-sha256; t=1788542088; cv=none; d=zohomail.com; s=zohoarc; b=KzlRQuO5DWXa2qRF7gsk+QY+pVbiN1sbHODTtPDxubIWpAifGbNqWQV44RosmU50hcT7vzrjuUi6RUrw4Ia79nSdshY4WqM80un6hHtbgCd4g2qQAvctBQM7jdsdn0KIg7TNFr9WMiQeVE9vEd3t1xu+p4ZCCEZpefLtkMDMxnU= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1788542088; h=Content-Type:Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=8QytEU0lmc6HwZdMirOP0o0H/nj+OB9HGqWTN4Ozvyo=; b=BfzkkDAipUSVC7ZSaHspMk63bUu66B0ZpmoIKH6gaTrH3Xs3DY9DHh6AD85w3UFPoY1wyZeVcGzXYnp5iDzLB4LQv5oVH6lWIatr0Q5T6pPfmW5aQ1ttmgNribYrqruVnlc677uFI7Cw4Xx2gjV2E6yAkhmnpzvRS2QI8oXT0Xg= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) by mx.zohomail.com with SMTPS id 1788542088923236.01888893325747; Fri, 4 Sep 2026 10:14:48 -0700 (PDT) Received: from list by lists.xenproject.org with outflank-mailman.1408904.1641192 (Exim 4.92) (envelope-from ) id 1x2XUj-0003f0-Et; Fri, 04 Sep 2026 17:14:29 +0000 Received: by outflank-mailman (output) from mailman id 1408904.1641192; Fri, 04 Sep 2026 17:14:29 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1x2XUj-0003et-9J; Fri, 04 Sep 2026 17:14:29 +0000 Received: by outflank-mailman (input) for mailman id 1408904; Fri, 04 Sep 2026 17:14:27 +0000 Received: from mail.xenproject.org ([104.130.215.37]) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1x2XUh-0003QO-Ae for xen-devel@lists.xenproject.org; Fri, 04 Sep 2026 17:14:27 +0000 Received: from xenbits.xenproject.org ([104.239.192.120]) by mail.xenproject.org with esmtp (Exim 4.96) (envelope-from ) id 1x2XUg-005KkT-1w; Fri, 04 Sep 2026 17:14:26 +0000 Received: from 224.pool85-54-217.dynamic.orange.es ([85.54.217.224] helo=localhost) by xenbits.xenproject.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.96) (envelope-from ) id 1x2XUg-000Pv2-07; Fri, 04 Sep 2026 17:14:26 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=xenproject.org; s=20200302mail; h=Content-Transfer-Encoding:Content-Type: MIME-Version:References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From; bh=8QytEU0lmc6HwZdMirOP0o0H/nj+OB9HGqWTN4Ozvyo=; b=ErK5gLZmPi/Pj+Sm/QFByhluxM jBJnkHR45oUMKGQ59xflf5wJBimyQemm5/ekZAyBtdMW5UZOGpCcibAJD7+C6+tjO8rkHGS3HNCf7 uyUz1CfKKrcueNNN2dLdjYIri5aSDrnmjB5vIEh3AeNBHqpIATWpKxSsPdSjn1oJcqR4=; From: Roger Pau Monne To: xen-devel@lists.xenproject.org Cc: =?UTF-8?q?J=C3=BCrgen=20Gro=C3=9F?= , Roger Pau Monne , Andrew Cooper , Anthony PERARD , Michal Orzel , Jan Beulich , Julien Grall , Stefano Stabellini Subject: [PATCH 3/5] xen/rcu: introduce the concept of RCU epoch Date: Fri, 4 Sep 2026 19:11:19 +0200 Message-ID: <20260904171121.65300-4-roger@xenproject.org> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260904171121.65300-1-roger@xenproject.org> References: <20260904171121.65300-1-roger@xenproject.org> MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-ZohoMail-DKIM: pass (identity @xenproject.org) X-ZM-MESSAGEID: 1788542090015158500 An RCU epoch signals the lifetime of RCU references. Each CPU keeps track of the epoch when an RCU critical section is entered. When a RCU callback is added the current epoch is recorded in the callback, and increased, as a way to know when all CPUs have moved past a specific epoch, and thus there are no longer active references to objects fetched during that epoch. The compiler barrier is switched to a full memory barrier, as future uses of rcu_lock_cnt must ensure the count is increased before taking a reference to any RCU protected object. Use ACCESS_ONCE() avoid the compiler from shattering accesses to the variables. The reordering prevention aspect of ACCESS_ONCE() is not relevant here, but we must ensure accesses are not shattered, as there will be remote consumers of those variables. Signed-off-by: Roger Pau Monn=C3=A9 --- Can possibly be folded into the next patch, as it's lacking context on its own to understand the need to introduce the logic. --- xen/common/rcupdate.c | 6 ++++++ xen/include/xen/rcupdate.h | 19 +++++++++++++++---- 2 files changed, 21 insertions(+), 4 deletions(-) diff --git a/xen/common/rcupdate.c b/xen/common/rcupdate.c index c1b6b2ae768b..bd63280fd63c 100644 --- a/xen/common/rcupdate.c +++ b/xen/common/rcupdate.c @@ -49,6 +49,11 @@ #include =20 DEFINE_PER_CPU(unsigned int, rcu_lock_cnt); +/* Store epoch when CPU entered the RCU critical section. */ +DEFINE_PER_CPU(unsigned int, rcu_lock_epoch); + +/* Current RCU epoch, bumped every time a new callback is queued. */ +unsigned int rcu_epoch; =20 /* Global control variables for rcupdate callback mechanism. */ static struct rcu_ctrlblk { @@ -282,6 +287,7 @@ void call_rcu(struct rcu_head *head, =20 head->func =3D func; head->next =3D NULL; + head->added =3D arch_fetch_and_add(&rcu_epoch, 1); local_irq_save(flags); rdp =3D &this_cpu(rcu_data); *rdp->nxttail =3D head; diff --git a/xen/include/xen/rcupdate.h b/xen/include/xen/rcupdate.h index c57f628107cf..6c265c672c14 100644 --- a/xen/include/xen/rcupdate.h +++ b/xen/include/xen/rcupdate.h @@ -34,24 +34,34 @@ #include #include #include +#include #include #include =20 #define __rcu =20 DECLARE_PER_CPU(unsigned int, rcu_lock_cnt); +DECLARE_PER_CPU(unsigned int, rcu_lock_epoch); + +extern unsigned int rcu_epoch; =20 static inline void rcu_quiesce_disable(void) { + unsigned int cpu =3D smp_processor_id(); + preempt_disable(); - this_cpu(rcu_lock_cnt)++; - barrier(); + if ( !ACCESS_ONCE(per_cpu(rcu_lock_cnt, cpu))++ ) + { + ACCESS_ONCE(per_cpu(rcu_lock_epoch, cpu)) =3D ACCESS_ONCE(rcu_epoc= h); + smp_mb(); + } } =20 static inline void rcu_quiesce_enable(void) { - barrier(); - this_cpu(rcu_lock_cnt)--; + if ( this_cpu(rcu_lock_cnt) =3D=3D 1 ) + smp_mb(); + ACCESS_ONCE(this_cpu(rcu_lock_cnt))--; preempt_enable(); } =20 @@ -68,6 +78,7 @@ static inline bool rcu_quiesce_allowed(void) struct rcu_head { struct rcu_head *next; void (*func)(struct rcu_head *head); + unsigned int added; }; =20 #define RCU_HEAD_INIT { .next =3D NULL, .func =3D NULL } --=20 2.53.0 From nobody Thu Sep 24 20:20:59 2026 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) client-ip=192.237.175.120; envelope-from=xen-devel-bounces@lists.xenproject.org; helo=lists.xenproject.org; Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=pass(p=none dis=none) header.from=xenproject.org ARC-Seal: i=1; a=rsa-sha256; t=1788542091; cv=none; d=zohomail.com; s=zohoarc; b=mEib5Hcm4cZGF8jgYYPX73YObwpAXYYyu6K0gMnALE5LdENFvhsXCQt6vZGPi3EqLRjmEEOPqb8Ks77O44zXJw9hzosZRLG63MKkbEsYMUgFuNOcRx2+cOsuNwQDTcUu4vzIacmKRH09F8QIe+BmC06C59ROg10c+2guP1R1DN4= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1788542091; h=Content-Type:Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=LZhALLn0ovpkb2bRdZsHBI4v5WiQPyVT/P5+n0/4JSI=; b=esaYd72adWfvXrXwWFEq2D3FkZ/WJmXTRU6XTD4Y5nBKVM1O3VjY6DLP5Zbb6aWhi217dbDDDil5yy3mKjYOS63Q599KT12vmINDJMIm7w/GPNHscoWhwu1fcUJYTTgC16kZTux2RkW2LTjcuRpwC8Sn2idxXnCenYNwi11wgw0= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) by mx.zohomail.com with SMTPS id 1788542090810766.2634296964089; Fri, 4 Sep 2026 10:14:50 -0700 (PDT) Received: from list by lists.xenproject.org with outflank-mailman.1408905.1641200 (Exim 4.92) (envelope-from ) id 1x2XUk-0003tb-Oa; Fri, 04 Sep 2026 17:14:30 +0000 Received: by outflank-mailman (output) from mailman id 1408905.1641200; Fri, 04 Sep 2026 17:14:30 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1x2XUk-0003tU-KY; Fri, 04 Sep 2026 17:14:30 +0000 Received: by outflank-mailman (input) for mailman id 1408905; Fri, 04 Sep 2026 17:14:29 +0000 Received: from mail.xenproject.org ([104.130.215.37]) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1x2XUj-0003f6-Bo for xen-devel@lists.xenproject.org; Fri, 04 Sep 2026 17:14:29 +0000 Received: from xenbits.xenproject.org ([104.239.192.120]) by mail.xenproject.org with esmtp (Exim 4.96) (envelope-from ) id 1x2XUi-005Kkf-23; Fri, 04 Sep 2026 17:14:28 +0000 Received: from 224.pool85-54-217.dynamic.orange.es ([85.54.217.224] helo=localhost) by xenbits.xenproject.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.96) (envelope-from ) id 1x2XUi-000Q0G-0I; Fri, 04 Sep 2026 17:14:28 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=xenproject.org; s=20200302mail; h=Content-Transfer-Encoding:Content-Type: MIME-Version:References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From; bh=LZhALLn0ovpkb2bRdZsHBI4v5WiQPyVT/P5+n0/4JSI=; b=GA0ucdRsDlYpOMJX5g+REXqjui 8nRAl10euTpuzFxzt4cTjjUuS8m8OFybg20rxzHpraqmgkGr9mpmOvel1iHzai9WV5a1hOxtI1T9J +UTWil2l39Hsp0GpWpvVtB3S6RP6km2VDG3k9dIQsItoo1w5kvB7ylCI28tSkDiMEJNc=; From: Roger Pau Monne To: xen-devel@lists.xenproject.org Cc: =?UTF-8?q?J=C3=BCrgen=20Gro=C3=9F?= , Roger Pau Monne , Andrew Cooper , Anthony PERARD , Michal Orzel , Jan Beulich , Julien Grall , Stefano Stabellini Subject: [PATCH 4/5] xen/rcu: simplify RCU implementation Date: Fri, 4 Sep 2026 19:11:20 +0200 Message-ID: <20260904171121.65300-5-roger@xenproject.org> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260904171121.65300-1-roger@xenproject.org> References: <20260904171121.65300-1-roger@xenproject.org> MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-ZohoMail-DKIM: pass (identity @xenproject.org) X-ZM-MESSAGEID: 1788542092122158500 The current implementation has two shortcomings for certain Xen usages: * When using the null scheduler it's possible for a CPU to never enter Xen context. A CPU not entering Xen context can block other CPUs from executing RCU callbacks, as there will be no quiescent state observed if the CPU doesn't enter Xen context. * If a certain amount of callbacks are pending, RCU will try to force a quiescent state, by sending an IPI to remote CPUs. This causes unwanted interference. Keep track of the RCU epoch when a callback was added, and only execute it once all CPUs are either outside of RCU critical regions, or any CPUs inside of RCU critical regions have entered such past the epoch when the callback was queued. Knowing whether a CPU is inside a RCU critical region is done based on the CPU rcu_lock_cnt value. There's an additional cost introduced in rcu_quiesce_{disable,enable}(), as we now need to use a full memory barrier on the outermost critical section entry/exit to make sure changes to rcu_lock_cnt cannot be reordered with accesses to RCU protected objects. This simplifies the current RCU implementation, as we get rid of grace/quiescent periods and a fair amount of logic to manage the state tracking. Signed-off-by: Roger Pau Monn=C3=A9 --- The maximum batch of callbacks processed is limited to 10, this is bit arbitrary, but I think matches what the current logic attempts does. There's possibly some logic missing that rate-limits the amount of callbacks to process during a certain period. I can add those in v2 if the current approach is considered sane. Possibly there's a bit more pruning to do regarding the usage of grace and quiesce in comments or functions names - I leave that to either v2 or a different change. --- xen/common/rcupdate.c | 464 +++++++++---------------------------- xen/include/xen/rcupdate.h | 13 +- 2 files changed, 112 insertions(+), 365 deletions(-) diff --git a/xen/common/rcupdate.c b/xen/common/rcupdate.c index bd63280fd63c..d3c11f45bfa0 100644 --- a/xen/common/rcupdate.c +++ b/xen/common/rcupdate.c @@ -35,6 +35,7 @@ #include #include #include +#include #include #include #include @@ -55,74 +56,28 @@ DEFINE_PER_CPU(unsigned int, rcu_lock_epoch); /* Current RCU epoch, bumped every time a new callback is queued. */ unsigned int rcu_epoch; =20 -/* Global control variables for rcupdate callback mechanism. */ -static struct rcu_ctrlblk { - long cur; /* Current batch number. */ - long completed; /* Number of the last completed batch */ - int next_pending; /* Is the next batch already waiting? */ - - spinlock_t lock __cacheline_aligned; - cpumask_t cpumask; /* CPUs that need to switch in order ... */ - cpumask_t idle_cpumask; /* ... unless they are already idle */ - /* for current batch to proceed. */ -} __cacheline_aligned rcu_ctrlblk =3D { - .cur =3D -300, - .completed =3D -300, - .lock =3D SPIN_LOCK_UNLOCKED, -}; - -/* - * Per-CPU data for Read-Copy Update. - * nxtlist - new callbacks are added here - * curlist - current batch for which quiescent cycle started if any - */ +/* Per-CPU data for Read-Copy Update. */ struct rcu_data { - /* 1) quiescent state handling : */ - long quiescbatch; /* Batch # for grace period */ - int qs_pending; /* core waits for quiesc state */ - - /* 2) batch handling */ - long batch; /* Batch # for current RCU batch */ - struct rcu_head *nxtlist; - struct rcu_head **nxttail; - long qlen; /* # of queued callbacks */ - struct rcu_head *curlist; - struct rcu_head **curtail; - struct rcu_head *donelist; - struct rcu_head **donetail; - long blimit; /* Upper limit on a processed batch = */ - int cpu; - long last_rs_qlen; /* qlen during the last resched */ - - /* 3) idle CPUs handling */ + /* + * List of pending callbacks, sorted by ascending epoch. Use a thresh= old + * value to raise an RCU softirq if the queue exceeds a given length. + */ + struct list_head pending; + unsigned int nr; +#define RCU_QUEUE_THRESHOLD 100 + + /* Idle CPU handling */ struct timer idle_timer; bool idle_timer_active; =20 - bool process_callbacks; + /* Barrier handling. */ bool barrier_active; }; =20 /* - * If a CPU with RCU callbacks queued goes idle, when the grace period is - * not finished yet, how can we make sure that the callbacks will eventual= ly - * be executed? In Linux (2.6.21, the first "tickless idle" Linux kernel), - * the periodic timer tick would not be stopped for such CPU. Here in Xen, - * we (may) don't even have a periodic timer tick, so we need to use a - * special purpose timer. - * - * Such timer: - * 1) is armed only when a CPU with an RCU callback(s) queued goes idle - * before the end of the current grace period (_not_ for any CPUs that - * go idle!); - * 2) when it fires, it is only re-armed if the grace period is still - * running; - * 3) it is stopped immediately, if the CPU wakes up from idle and - * resumes 'normal' execution. - * - * About how far in the future the timer should be programmed each time, - * it's hard to tell (guess!!). Since this mimics Linux's periodic timer - * tick, take values used there as an indication. In Linux 2.6.21, tick - * period can be 10ms, 4ms, 3.33ms or 1ms. + * If a CPU with RCU callbacks queued goes idle before the callbacks can be + * drained use a timer to ensure the CPU is woken up to process the remain= ing + * callback queue. * * By default, we use 10ms, to enable at least some power saving on the * CPU that is going idle. The user can change this, via a boot time @@ -137,21 +92,18 @@ static s_time_t __read_mostly idle_timer_period; /* * Increment and decrement values for the idle timer handler. The algorithm * works as follows: - * - if the timer actually fires, and it finds out that the grace period i= sn't - * over yet, we add IDLE_TIMER_PERIOD_INCR to the timer's period; - * - if the timer actually fires and it finds the grace period over, we - * subtract IDLE_TIMER_PERIOD_DECR from the timer's period. + * - If the timer actually fires, and it finds out there are CPUs still in= RCU + * critical regions, we add IDLE_TIMER_PERIOD_INCR to the timer's period. + * Note this is not very accurate, as the CPUs in those RCU critical reg= ions + * might not be holding back the execution of the local callbacks. + * - If the timer actually fires and it finds no CPUs in critical RCU regi= ons, + * we subtract IDLE_TIMER_PERIOD_DECR from the timer's period. */ #define IDLE_TIMER_PERIOD_INCR MILLISECS(10) #define IDLE_TIMER_PERIOD_DECR MICROSECS(100) =20 static DEFINE_PER_CPU(struct rcu_data, rcu_data); =20 -static int blimit =3D 10; -static int qhimark =3D 10000; -static int qlowmark =3D 100; -static int rsinterval =3D 1000; - /* * rcu_barrier() handling: * Two counters are used to synchronize rcu_barrier() work: @@ -246,35 +198,12 @@ void rcu_barrier(void) put_cpu_maps(); } =20 -/* Is batch a before batch b ? */ -static inline int rcu_batch_before(long a, long b) -{ - return (a - b) < 0; -} - -static void force_quiescent_state(struct rcu_data *rdp, - struct rcu_ctrlblk *rcp) -{ - cpumask_t cpumask; - raise_softirq(RCU_SOFTIRQ); - if (unlikely(rdp->qlen - rdp->last_rs_qlen > rsinterval)) { - rdp->last_rs_qlen =3D rdp->qlen; - /* - * Don't send IPI to itself. With irqs disabled, - * rdp->cpu is the current cpu. - */ - cpumask_andnot(&cpumask, &rcp->cpumask, cpumask_of(rdp->cpu)); - cpumask_raise_softirq(&cpumask, RCU_SOFTIRQ); - } -} - /** * call_rcu - Queue an RCU callback for invocation after a grace period. * @head: structure to be used for queueing the RCU updates. * @func: actual update function to be invoked after the grace period * - * The update function will be invoked some time after a full grace - * period elapses, in other words after all currently executing RCU + * The update function will be invoked after all currently executing RCU * read-side critical sections have completed. RCU read-side critical * sections are delimited by rcu_read_lock() and rcu_read_unlock(), * and may be nested. @@ -283,205 +212,77 @@ void call_rcu(struct rcu_head *head, void (*func)(struct rcu_head *rcu)) { unsigned long flags; - struct rcu_data *rdp; + struct rcu_data *rdp =3D &this_cpu(rcu_data); =20 head->func =3D func; - head->next =3D NULL; head->added =3D arch_fetch_and_add(&rcu_epoch, 1); local_irq_save(flags); - rdp =3D &this_cpu(rcu_data); - *rdp->nxttail =3D head; - rdp->nxttail =3D &head->next; - if (unlikely(++rdp->qlen > qhimark)) { - rdp->blimit =3D INT_MAX; - force_quiescent_state(rdp, &rcu_ctrlblk); - } - local_irq_restore(flags); -} - -/* - * Invoke the completed RCU callbacks. They are expected to be in - * a per-cpu list. - */ -static void rcu_do_batch(struct rcu_data *rdp) -{ - struct rcu_head *next, *list; - int count =3D 0; - - list =3D rdp->donelist; - while (list) { - next =3D rdp->donelist =3D list->next; - list->func(list); - list =3D next; - rdp->qlen--; - if (++count >=3D rdp->blimit) - break; - } - if (rdp->blimit =3D=3D INT_MAX && rdp->qlen <=3D qlowmark) - rdp->blimit =3D blimit; - if (!rdp->donelist) - rdp->donetail =3D &rdp->donelist; - else - { - rdp->process_callbacks =3D true; - raise_softirq(RCU_SOFTIRQ); - } -} - -/* - * Grace period handling: - * The grace period handling consists out of two steps: - * - A new grace period is started. - * This is done by rcu_start_batch. The start is not broadcasted to - * all cpus, they must pick this up by comparing rcp->cur with - * rdp->quiescbatch. All cpus are recorded in the - * rcu_ctrlblk.cpumask bitmap. - * - All cpus must go through a quiescent state. - * Since the start of the grace period is not broadcasted, at least two - * calls to rcu_check_quiescent_state are required: - * The first call just notices that a new grace period is running. The - * following calls check if there was a quiescent state since the beginn= ing - * of the grace period. If so, it updates rcu_ctrlblk.cpumask. If - * the bitmap is empty, then the grace period is completed. - * rcu_check_quiescent_state calls rcu_start_batch(0) to start the next = grace - * period (if necessary). - */ -/* - * Register a new batch of callbacks, and start it up if there is currentl= y no - * active batch and the batch to be registered has not already occurred. - * Caller must hold rcu_ctrlblk.lock. - */ -static void rcu_start_batch(struct rcu_ctrlblk *rcp) -{ - if (rcp->next_pending && - rcp->completed =3D=3D rcp->cur) { - rcp->next_pending =3D 0; + list_add_tail(&head->list, &rdp->pending); + if ( ++rdp->nr > RCU_QUEUE_THRESHOLD ) /* - * next_pending =3D=3D 0 must be visible in - * __rcu_process_callbacks() before it can see new value of cur. + * Raise a softirq to attempt to force draining the queue, albeit + * there's no guarantee. */ - smp_wmb(); - rcp->cur++; - - /* - * Make sure the increment of rcp->cur is visible so, even if a - * CPU that is about to go idle, is captured inside rcp->cpumask, - * rcu_pending() will return false, which then means cpu_quiet() - * will be invoked, before the CPU would actually enter idle. - * - * This barrier is paired with the one in rcu_idle_enter(). - */ - smp_mb(); - cpumask_andnot(&rcp->cpumask, &cpu_online_map, &rcp->idle_cpumask); - } -} - -/* - * cpu went through a quiescent state since the beginning of the grace per= iod. - * Clear it from the cpu mask and complete the grace period if it was the = last - * cpu. Start another grace period if someone has further entries pending - */ -static void cpu_quiet(int cpu, struct rcu_ctrlblk *rcp) -{ - cpumask_clear_cpu(cpu, &rcp->cpumask); - if (cpumask_empty(&rcp->cpumask)) { - /* batch completed ! */ - rcp->completed =3D rcp->cur; - rcu_start_batch(rcp); - } + raise_softirq(RCU_SOFTIRQ); + local_irq_restore(flags); } =20 -/* - * Check if the cpu has gone through a quiescent state (say context - * switch). If so and if it already hasn't done so in this RCU - * quiescent cycle, then indicate that it has done so. - */ -static void rcu_check_quiescent_state(struct rcu_ctrlblk *rcp, - struct rcu_data *rdp) +#define RCU_MAX_BATCH 10 +static void cf_check rcu_process_callbacks(void) { - if (rdp->quiescbatch !=3D rcp->cur) { - /* start new grace period: */ - rdp->qs_pending =3D 1; - rdp->quiescbatch =3D rcp->cur; - return; - } - - /* Grace period already completed for this cpu? - * qs_pending is checked instead of the actual bitmap to avoid - * cacheline trashing. - */ - if (!rdp->qs_pending) - return; - - rdp->qs_pending =3D 0; + static DEFINE_PER_CPU(cpumask_t, rcu_scratch); + cpumask_t *in_rcu =3D &this_cpu(rcu_scratch); + struct rcu_data *rdp =3D &this_cpu(rcu_data); + unsigned int queued =3D 0, cpu; + LIST_HEAD(expired); + struct rcu_head *rcu; =20 - spin_lock(&rcp->lock); /* - * rdp->quiescbatch/rcp->cur and the cpu bitmap can come out of sync - * during cpu startup. Ignore the quiescent state. + * Populate a cpumask with any CPUs inside RCU critical regions. Note= that + * CPUs entering past this point are of no interest, they will certain= ly + * use an epoch past any queued callbacks here. */ - if (likely(rdp->quiescbatch =3D=3D rcp->cur)) - cpu_quiet(rdp->cpu, rcp); - - spin_unlock(&rcp->lock); -} - - -/* - * This does the RCU processing work from softirq context.=20 - */ -static void __rcu_process_callbacks(struct rcu_ctrlblk *rcp, - struct rcu_data *rdp) -{ - if (rdp->curlist && !rcu_batch_before(rcp->completed, rdp->batch)) { - *rdp->donetail =3D rdp->curlist; - rdp->donetail =3D rdp->curtail; - rdp->curlist =3D NULL; - rdp->curtail =3D &rdp->curlist; - } - - local_irq_disable(); - if (rdp->nxtlist && !rdp->curlist) { - rdp->curlist =3D rdp->nxtlist; - rdp->curtail =3D rdp->nxttail; - rdp->nxtlist =3D NULL; - rdp->nxttail =3D &rdp->nxtlist; - local_irq_enable(); - + cpumask_clear(in_rcu); + for_each_online_cpu ( cpu ) + if ( ACCESS_ONCE(per_cpu(rcu_lock_cnt, cpu)) ) + __cpumask_set_cpu(cpu, in_rcu); + + while ( queued < RCU_MAX_BATCH && + (rcu =3D list_first_entry_or_null(&rdp->pending, struct rcu_he= ad, + list)) ) + { /* - * start the next batch of callbacks - */ - - /* determine batch number */ - rdp->batch =3D rcp->cur + 1; - /* see the comment and corresponding wmb() in - * the rcu_start_batch() + * Fetching rcu_lock_epoch out of order is not a concern here: in = the + * worst case it's going to result in an older more restrictive ep= och + * being checked against. Note the adding of a callback issues a + * arch_fetch_and_add() which is a barrier on itself, and guarante= es + * remote changes to the CPU mask to be visible here. */ - smp_rmb(); - - if (!rcp->next_pending) { - /* and start it/schedule start if it's a new batch */ - spin_lock(&rcp->lock); - rcp->next_pending =3D 1; - rcu_start_batch(rcp); - spin_unlock(&rcp->lock); - } - } else { - local_irq_enable(); + for_each_cpu ( cpu, in_rcu ) + if ( (int)(ACCESS_ONCE(per_cpu(rcu_lock_epoch, cpu)) - + rcu->added) <=3D 0 ) + /* + * Callbacks are sorted, exit loop as soon as we find one = that + * can't be processed yet. + */ + goto process; + + list_del(&rcu->list); + list_add_tail(&rcu->list, &expired); + ASSERT(rdp->nr); + rdp->nr--; + queued++; } - rcu_check_quiescent_state(rcp, rdp); - if (rdp->donelist) - rcu_do_batch(rdp); -} =20 -static void cf_check rcu_process_callbacks(void) -{ - struct rcu_data *rdp =3D &this_cpu(rcu_data); + if ( queued =3D=3D RCU_MAX_BATCH && rdp->nr ) + /* There's more work to do, yield and raise a softirq to come back= . */ + raise_softirq(RCU_SOFTIRQ); =20 - if ( rdp->process_callbacks ) + process: + while ( (rcu =3D list_first_entry_or_null(&expired, struct rcu_head, l= ist)) ) { - rdp->process_callbacks =3D false; - __rcu_process_callbacks(&rcu_ctrlblk, rdp); + list_del(&rcu->list); + rcu->func(rcu); } =20 if ( atomic_read(&cpu_count) && !rdp->barrier_active ) @@ -492,33 +293,9 @@ static void cf_check rcu_process_callbacks(void) } } =20 -static int __rcu_pending(struct rcu_ctrlblk *rcp, struct rcu_data *rdp) -{ - /* This cpu has pending rcu entries and the grace period - * for them has completed. - */ - if (rdp->curlist && !rcu_batch_before(rcp->completed, rdp->batch)) - return 1; - - /* This cpu has no pending entries, but there are new entries */ - if (!rdp->curlist && rdp->nxtlist) - return 1; - - /* This cpu has finished callbacks to invoke */ - if (rdp->donelist) - return 1; - - /* The rcu core waits for a quiescent state from the cpu */ - if (rdp->quiescbatch !=3D rcp->cur || rdp->qs_pending) - return 1; - - /* nothing to do */ - return 0; -} - bool rcu_pending(unsigned int cpu) { - return !!__rcu_pending(&rcu_ctrlblk, &per_cpu(rcu_data, cpu)); + return !!per_cpu(rcu_data, cpu).nr; } =20 /* @@ -529,15 +306,13 @@ bool rcu_pending(unsigned int cpu) */ bool rcu_needs_cpu(unsigned int cpu) { - struct rcu_data *rdp =3D &per_cpu(rcu_data, cpu); - - return (rdp->curlist && !rdp->idle_timer_active) || rcu_pending(cpu); + return rcu_pending(cpu); } =20 /* * Timer for making sure the CPU where a callback is queued does * periodically poke rcu_pending(), so that it will invoke the callback - * not too late after the end of the grace period. + * not too late. */ static void rcu_idle_timer_start(void) { @@ -545,10 +320,9 @@ static void rcu_idle_timer_start(void) =20 /* * Note that we don't check rcu_pending() here. In fact, we don't want - * the timer armed on CPUs that are in the process of quiescing while - * going idle, unless they really are the ones with a queued callback. + * the timer armed on CPUs that don't have pending callbacks. */ - if (likely(!rdp->curlist)) + if (likely(!rdp->nr)) return; =20 set_timer(&rdp->idle_timer, NOW() + idle_timer_period); @@ -587,7 +361,7 @@ static void cf_check rcu_idle_timer_handler(void* data) { perfc_incr(rcu_idle_timer); =20 - if ( !cpumask_empty(&rcu_ctrlblk.cpumask) ) + if ( this_cpu(rcu_data).nr ) idle_timer_period =3D min(idle_timer_period + IDLE_TIMER_PERIOD_IN= CR, IDLE_TIMER_PERIOD_MAX); else @@ -597,55 +371,43 @@ static void cf_check rcu_idle_timer_handler(void* dat= a) =20 void rcu_check_callbacks(unsigned int cpu) { - struct rcu_data *rdp =3D &this_cpu(rcu_data); - - rdp->process_callbacks =3D true; raise_softirq(RCU_SOFTIRQ); } =20 -static void rcu_move_batch(struct rcu_data *this_rdp, struct rcu_head *lis= t, - struct rcu_head **tail) +/* Sorting functions for RCU list concatenation when a CPU goes offline. */ +static int cmp_rcu(void *priv, struct list_head *a, struct list_head *b) { - local_irq_disable(); - *this_rdp->nxttail =3D list; - if (list) - this_rdp->nxttail =3D tail; - local_irq_enable(); + const struct rcu_head *l =3D container_of(a, struct rcu_head, list), + *r =3D container_of(b, struct rcu_head, list); + + return (int)(l->added - r->added); } =20 static void rcu_offline_cpu(struct rcu_data *this_rdp, - struct rcu_ctrlblk *rcp, struct rcu_data *rdp) + struct rcu_data *rdp) { kill_timer(&rdp->idle_timer); =20 - /* If the cpu going offline owns the grace period we can block - * indefinitely waiting for it, so flush it here. - */ - spin_lock(&rcp->lock); - if (rcp->cur !=3D rcp->completed) - cpu_quiet(rdp->cpu, rcp); - spin_unlock(&rcp->lock); - - rcu_move_batch(this_rdp, rdp->donelist, rdp->donetail); - rcu_move_batch(this_rdp, rdp->curlist, rdp->curtail); - rcu_move_batch(this_rdp, rdp->nxtlist, rdp->nxttail); + if ( !rdp->nr ) + return; =20 + /* + * Append pending callbacks to the current CPU. By the time this is + * executed the CPU going offline cannot be in any RCU critical sectio= n or + * queue any more RCU work. + */ local_irq_disable(); - this_rdp->qlen +=3D rdp->qlen; + list_splice(&rdp->pending, &this_rdp->pending); + this_rdp->nr +=3D rdp->nr; + INIT_LIST_HEAD(&rdp->pending); + list_sort(NULL, &this_rdp->pending, cmp_rcu); local_irq_enable(); } =20 -static void rcu_init_percpu_data(int cpu, struct rcu_ctrlblk *rcp, - struct rcu_data *rdp) +static void rcu_init_percpu_data(int cpu, struct rcu_data *rdp) { memset(rdp, 0, sizeof(*rdp)); - rdp->curtail =3D &rdp->curlist; - rdp->nxttail =3D &rdp->nxtlist; - rdp->donetail =3D &rdp->donelist; - rdp->quiescbatch =3D rcp->completed; - rdp->qs_pending =3D 0; - rdp->cpu =3D cpu; - rdp->blimit =3D blimit; + INIT_LIST_HEAD(&rdp->pending); init_timer(&rdp->idle_timer, rcu_idle_timer_handler, rdp, cpu); } =20 @@ -658,11 +420,11 @@ static int cf_check cpu_callback( switch ( action ) { case CPU_UP_PREPARE: - rcu_init_percpu_data(cpu, &rcu_ctrlblk, rdp); + rcu_init_percpu_data(cpu, rdp); break; case CPU_UP_CANCELED: case CPU_DEAD: - rcu_offline_cpu(&this_cpu(rcu_data), &rcu_ctrlblk, rdp); + rcu_offline_cpu(&this_cpu(rcu_data), rdp); break; default: break; @@ -693,36 +455,18 @@ void __init rcu_init(void) } idle_timer_period =3D MILLISECS(idle_timer_period_ms); =20 - cpumask_clear(&rcu_ctrlblk.idle_cpumask); cpu_callback(&cpu_nfb, CPU_UP_PREPARE, cpu); register_cpu_notifier(&cpu_nfb); open_softirq(RCU_SOFTIRQ, rcu_process_callbacks); } =20 -/* - * The CPU is becoming idle, so no more read side critical - * sections, and one more step toward grace period. - */ +/* The CPU is becoming idle, ensure pending RCU work will get processed. */ void rcu_idle_enter(unsigned int cpu) { - ASSERT(!cpumask_test_cpu(cpu, &rcu_ctrlblk.idle_cpumask)); - cpumask_set_cpu(cpu, &rcu_ctrlblk.idle_cpumask); - /* - * If some other CPU is starting a new grace period, we'll notice that - * by seeing a new value in rcp->cur (different than our quiescbatch). - * That will force us all the way until cpu_quiet(), clearing our bit - * in rcp->cpumask, even in case we managed to get in there. - * - * Se the comment before cpumask_andnot() in rcu_start_batch(). - */ - smp_mb(); - rcu_idle_timer_start(); } =20 void rcu_idle_exit(unsigned int cpu) { rcu_idle_timer_stop(); - ASSERT(cpumask_test_cpu(cpu, &rcu_ctrlblk.idle_cpumask)); - cpumask_clear_cpu(cpu, &rcu_ctrlblk.idle_cpumask); } diff --git a/xen/include/xen/rcupdate.h b/xen/include/xen/rcupdate.h index 6c265c672c14..9c3e06bbe6e8 100644 --- a/xen/include/xen/rcupdate.h +++ b/xen/include/xen/rcupdate.h @@ -35,6 +35,7 @@ #include #include #include +#include #include #include =20 @@ -72,19 +73,21 @@ static inline bool rcu_quiesce_allowed(void) =20 /** * struct rcu_head - callback structure for use with RCU - * @next: next update requests in a list + * @list: list anchor. * @func: actual update function to call after the grace period. + * @added: epoch when the callback was added. */ struct rcu_head { - struct rcu_head *next; + struct list_head list; void (*func)(struct rcu_head *head); unsigned int added; }; =20 -#define RCU_HEAD_INIT { .next =3D NULL, .func =3D NULL } -#define RCU_HEAD(head) struct rcu_head head =3D RCU_HEAD_INIT +#define RCU_HEAD_INIT(head) { .list =3D LIST_HEAD_INIT((head).list), \ + .func =3D NULL } +#define RCU_HEAD(head) struct rcu_head head =3D RCU_HEAD_INIT(head) #define INIT_RCU_HEAD(ptr) do { \ - (ptr)->next =3D NULL; (ptr)->func =3D NULL; \ + INIT_LIST_HEAD(&(ptr)->list); (ptr)->func =3D NULL; \ } while (0) =20 =20 --=20 2.53.0 From nobody Thu Sep 24 20:20:59 2026 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) client-ip=192.237.175.120; envelope-from=xen-devel-bounces@lists.xenproject.org; helo=lists.xenproject.org; Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=pass(p=none dis=none) header.from=xenproject.org ARC-Seal: i=1; a=rsa-sha256; t=1788542099; cv=none; d=zohomail.com; s=zohoarc; b=nSsJFl8758py8/x+gTb3x1x7xigsdti5RRy+PeKjgUaYO0jDX36RLUZ+BwXd02iE1JfdEuigMBiFIzolU4hOnh+VDD6k8qxlUymHuEfc8lGph4PiDBIcNczOGKDbZOKjUFEymzbBQOMQcXFKSDfANECHeI/BQhQ6UlJWvjya5GM= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1788542099; h=Content-Type:Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=i3K9AqM6qwFFz7KWpD8EsWRfNCM+xPNYSDsWiwSOLiU=; b=nkaU+45D0j+ffLb6xWSC8mDGXLiG/QnOq5qr3NfE2hc5TuwCDwVQdnKazof+QAh4WEskELIDxoQDV3LvPaWyJLu+I81pYkbw5DPPhjsKf64SN96o8ksNMK3BDWChJ7ZTRAInNIA2HxN9cgTeYiZhsTPNwCTyA36kkJJzVNILkeU= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) by mx.zohomail.com with SMTPS id 1788542098346880.5323564656507; Fri, 4 Sep 2026 10:14:58 -0700 (PDT) Received: from list by lists.xenproject.org with outflank-mailman.1408906.1641208 (Exim 4.92) (envelope-from ) id 1x2XUn-0004BB-0U; Fri, 04 Sep 2026 17:14:33 +0000 Received: by outflank-mailman (output) from mailman id 1408906.1641208; Fri, 04 Sep 2026 17:14:32 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1x2XUm-0004B4-TE; Fri, 04 Sep 2026 17:14:32 +0000 Received: by outflank-mailman (input) for mailman id 1408906; Fri, 04 Sep 2026 17:14:31 +0000 Received: from mail.xenproject.org ([104.130.215.37]) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1x2XUl-000469-L5 for xen-devel@lists.xenproject.org; Fri, 04 Sep 2026 17:14:31 +0000 Received: from xenbits.xenproject.org ([104.239.192.120]) by mail.xenproject.org with esmtp (Exim 4.96) (envelope-from ) id 1x2XUk-005Kkr-2N; Fri, 04 Sep 2026 17:14:30 +0000 Received: from 224.pool85-54-217.dynamic.orange.es ([85.54.217.224] helo=localhost) by xenbits.xenproject.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.96) (envelope-from ) id 1x2XUk-000Q84-0Z; Fri, 04 Sep 2026 17:14:30 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=xenproject.org; s=20200302mail; h=Content-Transfer-Encoding:Content-Type: MIME-Version:References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From; bh=i3K9AqM6qwFFz7KWpD8EsWRfNCM+xPNYSDsWiwSOLiU=; b=loTAIcNBrVQJjRtRs4qxjFYDS1 FKHo8hmBkLhMqoDzF8TXlo7qmhgLMCU94UiJqMpTQ8qubpeckKzjXYOZoFiTwqAuYWKB6HK6/NPRn rDcFvoeW2qgK7x2eRQQiMZ6lUd2HVjdvIGZAuwLEFzF3gmytfvVmwPe2PU/nz/2UB080=; From: Roger Pau Monne To: xen-devel@lists.xenproject.org Cc: =?UTF-8?q?J=C3=BCrgen=20Gro=C3=9F?= , Roger Pau Monne , Andrew Cooper , Anthony PERARD , Michal Orzel , Jan Beulich , Julien Grall , Stefano Stabellini Subject: [PATCH 5/5] xen/rcu: remove rcu_needs_cpu() Date: Fri, 4 Sep 2026 19:11:21 +0200 Message-ID: <20260904171121.65300-6-roger@xenproject.org> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260904171121.65300-1-roger@xenproject.org> References: <20260904171121.65300-1-roger@xenproject.org> MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-ZohoMail-DKIM: pass (identity @xenproject.org) X-ZM-MESSAGEID: 1788542100207158500 After the changes to the RCU logic, there's no longer a difference between rcu_pending() and rcu_needs_cpu(). With the previous implementation rcu_pending() signaled whether there was RCU work ready to handle, while rcu_needs_cpu() signaled whether the CPU had queued RCU callback that could not yet execute. With the new logic figuring out whether callbacks can be executed requires more work, and hence is deferred to the processing logic in rcu_process_callbacks(). Both rcu_pending() and rcu_needs_cpu() return whether there's any pending work, without making guarantees any callbacks are ready to be executed. Given this lack of difference, remove rcu_needs_cpu() and use rcu_pending() in cpu_is_haltable(). Signed-off-by: Roger Pau Monn=C3=A9 --- xen/common/rcupdate.c | 11 ----------- xen/include/xen/rcupdate.h | 7 +++++-- xen/include/xen/sched.h | 2 +- 3 files changed, 6 insertions(+), 14 deletions(-) diff --git a/xen/common/rcupdate.c b/xen/common/rcupdate.c index d3c11f45bfa0..c8164b0ad7e0 100644 --- a/xen/common/rcupdate.c +++ b/xen/common/rcupdate.c @@ -298,17 +298,6 @@ bool rcu_pending(unsigned int cpu) return !!per_cpu(rcu_data, cpu).nr; } =20 -/* - * Check to see if any future RCU-related work will need to be done - * by the current CPU, even if none need be done immediately, returning - * true if so. This function is part of the RCU implementation; it is -no= t- - * an exported member of the RCU API. - */ -bool rcu_needs_cpu(unsigned int cpu) -{ - return rcu_pending(cpu); -} - /* * Timer for making sure the CPU where a callback is queued does * periodically poke rcu_pending(), so that it will invoke the callback diff --git a/xen/include/xen/rcupdate.h b/xen/include/xen/rcupdate.h index 9c3e06bbe6e8..1700b73a6c3b 100644 --- a/xen/include/xen/rcupdate.h +++ b/xen/include/xen/rcupdate.h @@ -90,9 +90,12 @@ struct rcu_head { INIT_LIST_HEAD(&(ptr)->list); (ptr)->func =3D NULL; \ } while (0) =20 - +/* + * Check whether there's pending RCU work queued on this CPU. This merely + * signals whether there are callbacks pending, there's no guarantee that = any + * callbacks are ready to be executed. + */ bool rcu_pending(unsigned int cpu); -bool rcu_needs_cpu(unsigned int cpu); =20 /* * Dummy lock type for passing to rcu_read_{lock,unlock}. Currently exists diff --git a/xen/include/xen/sched.h b/xen/include/xen/sched.h index e352e2b38e7d..5bccf9b748a6 100644 --- a/xen/include/xen/sched.h +++ b/xen/include/xen/sched.h @@ -1155,7 +1155,7 @@ void scheduler_disable(void); * the tasklet_work_to_do() helper). */ #define cpu_is_haltable(cpu) \ - (!rcu_needs_cpu(cpu) && \ + (!rcu_pending(cpu) && \ !softirq_pending(cpu) && \ cpu_online(cpu) && \ !per_cpu(tasklet_work_to_do, cpu)) --=20 2.53.0