From nobody Fri Dec 26 21:24:00 2025 Received: from eu-smtp-delivery-151.mimecast.com (eu-smtp-delivery-151.mimecast.com [185.58.86.151]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E0F4914267 for ; Fri, 29 Dec 2023 20:54:09 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=ACULAB.COM Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=aculab.com Received: from AcuMS.aculab.com (156.67.243.121 [156.67.243.121]) by relay.mimecast.com with ESMTP with both STARTTLS and AUTH (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id uk-mta-79-fRwTi2a-NdGsmCtHse5L_g-1; Fri, 29 Dec 2023 20:54:06 +0000 X-MC-Unique: fRwTi2a-NdGsmCtHse5L_g-1 Received: from AcuMS.Aculab.com (10.202.163.4) by AcuMS.aculab.com (10.202.163.4) with Microsoft SMTP Server (TLS) id 15.0.1497.48; Fri, 29 Dec 2023 20:53:49 +0000 Received: from AcuMS.Aculab.com ([::1]) by AcuMS.aculab.com ([::1]) with mapi id 15.00.1497.048; Fri, 29 Dec 2023 20:53:49 +0000 From: David Laight To: "'linux-kernel@vger.kernel.org'" , "'peterz@infradead.org'" , "'longman@redhat.com'" CC: "'mingo@redhat.com'" , "'will@kernel.org'" , "'boqun.feng@gmail.com'" , "'Linus Torvalds'" , "'xinhui.pan@linux.vnet.ibm.com'" , "'virtualization@lists.linux-foundation.org'" , 'Zeng Heng' Subject: [PATCH next 1/5] locking/osq_lock: Move the definition of optimistic_spin_node into osf_lock.c Thread-Topic: [PATCH next 1/5] locking/osq_lock: Move the definition of optimistic_spin_node into osf_lock.c Thread-Index: Ado6mSDjgxSGtr3xT6CXt/ONPypOxA== Date: Fri, 29 Dec 2023 20:53:49 +0000 Message-ID: <6eb23a47953b445281e04f5573aead65@AcuMS.aculab.com> References: <73a4b31c9c874081baabad9e5f2e5204@AcuMS.aculab.com> In-Reply-To: <73a4b31c9c874081baabad9e5f2e5204@AcuMS.aculab.com> Accept-Language: en-GB, en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ms-exchange-transport-fromentityheader: Hosted Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: aculab.com Content-Language: en-US Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" struct optimistic_spin_node is private to the implementation. Move it into the C file to ensure nothing is accessing it. Signed-off-by: David Laight Acked-by: Waiman Long --- include/linux/osq_lock.h | 5 ----- kernel/locking/osq_lock.c | 7 +++++++ 2 files changed, 7 insertions(+), 5 deletions(-) diff --git a/include/linux/osq_lock.h b/include/linux/osq_lock.h index 5581dbd3bd34..ea8fb31379e3 100644 --- a/include/linux/osq_lock.h +++ b/include/linux/osq_lock.h @@ -6,11 +6,6 @@ * An MCS like lock especially tailored for optimistic spinning for sleepi= ng * lock implementations (mutex, rwsem, etc). */ -struct optimistic_spin_node { - struct optimistic_spin_node *next, *prev; - int locked; /* 1 if lock acquired */ - int cpu; /* encoded CPU # + 1 value */ -}; =20 struct optimistic_spin_queue { /* diff --git a/kernel/locking/osq_lock.c b/kernel/locking/osq_lock.c index d5610ad52b92..d414eef4bec6 100644 --- a/kernel/locking/osq_lock.c +++ b/kernel/locking/osq_lock.c @@ -11,6 +11,13 @@ * called from interrupt context and we have preemption disabled while * spinning. */ + +struct optimistic_spin_node { + struct optimistic_spin_node *next, *prev; + int locked; /* 1 if lock acquired */ + int cpu; /* encoded CPU # + 1 value */ +}; + static DEFINE_PER_CPU_SHARED_ALIGNED(struct optimistic_spin_node, osq_node= ); =20 /* --=20 2.17.1 - Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1= PT, UK Registration No: 1397386 (Wales) From nobody Fri Dec 26 21:24:00 2025 Received: from eu-smtp-delivery-151.mimecast.com (eu-smtp-delivery-151.mimecast.com [185.58.86.151]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8CE6E14A97 for ; Fri, 29 Dec 2023 22:11:57 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=ACULAB.COM Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=aculab.com Received: from AcuMS.aculab.com (156.67.243.121 [156.67.243.121]) by relay.mimecast.com with ESMTP with both STARTTLS and AUTH (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id uk-mta-227-QXrAE9vJN0uhbMdRrMv0sA-1; Fri, 29 Dec 2023 22:11:54 +0000 X-MC-Unique: QXrAE9vJN0uhbMdRrMv0sA-1 Received: from AcuMS.Aculab.com (10.202.163.6) by AcuMS.aculab.com (10.202.163.6) with Microsoft SMTP Server (TLS) id 15.0.1497.48; Fri, 29 Dec 2023 22:11:40 +0000 Received: from AcuMS.Aculab.com ([::1]) by AcuMS.aculab.com ([::1]) with mapi id 15.00.1497.048; Fri, 29 Dec 2023 22:11:40 +0000 From: David Laight To: "'linux-kernel@vger.kernel.org'" , "'peterz@infradead.org'" , "'longman@redhat.com'" CC: "'mingo@redhat.com'" , "'will@kernel.org'" , "'boqun.feng@gmail.com'" , "'Linus Torvalds'" , "'xinhui.pan@linux.vnet.ibm.com'" , "'virtualization@lists.linux-foundation.org'" , 'Zeng Heng' Subject: [PATCH next 2/5] locking/osq_lock: Avoid dirtying the local cpu's 'node' in the osq_lock() fast path. Thread-Topic: [PATCH next 2/5] locking/osq_lock: Avoid dirtying the local cpu's 'node' in the osq_lock() fast path. Thread-Index: Ado6o/zBb4T2uRNuSTy7E9ZX7SDa6w== Date: Fri, 29 Dec 2023 22:11:40 +0000 Message-ID: References: <73a4b31c9c874081baabad9e5f2e5204@AcuMS.aculab.com> In-Reply-To: <73a4b31c9c874081baabad9e5f2e5204@AcuMS.aculab.com> Accept-Language: en-GB, en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ms-exchange-transport-fromentityheader: Hosted Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: aculab.com Content-Language: en-US Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" osq_lock() starts by setting node->next to NULL and node->locked to 0. Careful analysis shows that node->next is always NULL on entry. node->locked is set non-zero by another cpu to force a wakeup. This can only happen after the 'prev->next =3D node' assignment, so locked can be set to zero just before that (along with the assignment to node->prev). Only initialise node->cpu once, after that use its value instead of smp_processor_id() - which is probably a real function call. Should reduce cache-line bouncing a little. Signed-off-by: David Laight --- Re-send without the 'RE:' on the subject line. kernel/locking/osq_lock.c | 13 ++++++------- 1 file changed, 6 insertions(+), 7 deletions(-) diff --git a/kernel/locking/osq_lock.c b/kernel/locking/osq_lock.c index d414eef4bec6..55f5db896c02 100644 --- a/kernel/locking/osq_lock.c +++ b/kernel/locking/osq_lock.c @@ -51,7 +51,7 @@ osq_wait_next(struct optimistic_spin_queue *lock, struct optimistic_spin_node *prev) { struct optimistic_spin_node *next =3D NULL; - int curr =3D encode_cpu(smp_processor_id()); + int curr =3D node->cpu; int old; =20 /* @@ -98,12 +98,10 @@ bool osq_lock(struct optimistic_spin_queue *lock) { struct optimistic_spin_node *node =3D this_cpu_ptr(&osq_node); struct optimistic_spin_node *prev, *next; - int curr =3D encode_cpu(smp_processor_id()); int old; =20 - node->locked =3D 0; - node->next =3D NULL; - node->cpu =3D curr; + if (unlikely(node->cpu =3D=3D OSQ_UNLOCKED_VAL)) + node->cpu =3D encode_cpu(smp_processor_id()); =20 /* * We need both ACQUIRE (pairs with corresponding RELEASE in @@ -111,12 +109,13 @@ bool osq_lock(struct optimistic_spin_queue *lock) * the node fields we just initialised) semantics when updating * the lock tail. */ - old =3D atomic_xchg(&lock->tail, curr); + old =3D atomic_xchg(&lock->tail, node->cpu); if (old =3D=3D OSQ_UNLOCKED_VAL) return true; =20 prev =3D decode_cpu(old); node->prev =3D prev; + node->locked =3D 0; =20 /* * osq_lock() unqueue @@ -214,7 +213,7 @@ bool osq_lock(struct optimistic_spin_queue *lock) void osq_unlock(struct optimistic_spin_queue *lock) { struct optimistic_spin_node *node, *next; - int curr =3D encode_cpu(smp_processor_id()); + int curr =3D raw_cpu_read(osq_node.cpu); =20 /* * Fast path for the uncontended case. --=20 2.17.1 - Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1= PT, UK Registration No: 1397386 (Wales) From nobody Fri Dec 26 21:24:00 2025 Received: from eu-smtp-delivery-151.mimecast.com (eu-smtp-delivery-151.mimecast.com [185.58.86.151]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1801714281 for ; Fri, 29 Dec 2023 20:55:08 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=ACULAB.COM Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=aculab.com Received: from AcuMS.aculab.com (156.67.243.121 [156.67.243.121]) by relay.mimecast.com with ESMTP with both STARTTLS and AUTH (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id uk-mta-29-Sipm66Z9Nfy_PnSYO0hsLg-1; Fri, 29 Dec 2023 20:55:05 +0000 X-MC-Unique: Sipm66Z9Nfy_PnSYO0hsLg-1 Received: from AcuMS.Aculab.com (10.202.163.4) by AcuMS.aculab.com (10.202.163.4) with Microsoft SMTP Server (TLS) id 15.0.1497.48; Fri, 29 Dec 2023 20:54:47 +0000 Received: from AcuMS.Aculab.com ([::1]) by AcuMS.aculab.com ([::1]) with mapi id 15.00.1497.048; Fri, 29 Dec 2023 20:54:47 +0000 From: David Laight To: "'linux-kernel@vger.kernel.org'" , "'peterz@infradead.org'" , "'longman@redhat.com'" CC: "'mingo@redhat.com'" , "'will@kernel.org'" , "'boqun.feng@gmail.com'" , "'Linus Torvalds'" , "'xinhui.pan@linux.vnet.ibm.com'" , "'virtualization@lists.linux-foundation.org'" , 'Zeng Heng' Subject: RE: [PATCH next 2/5] locking/osq_lock: Avoid dirtying the local cpu's 'node' in the osq_lock() fast path. Thread-Topic: [PATCH next 2/5] locking/osq_lock: Avoid dirtying the local cpu's 'node' in the osq_lock() fast path. Thread-Index: Ado6mUd9raebt4vmQzObd4YJJW0D/g== Date: Fri, 29 Dec 2023 20:54:47 +0000 Message-ID: <6eacbdbd1e0f489783439af512dc8325@AcuMS.aculab.com> References: <73a4b31c9c874081baabad9e5f2e5204@AcuMS.aculab.com> In-Reply-To: <73a4b31c9c874081baabad9e5f2e5204@AcuMS.aculab.com> Accept-Language: en-GB, en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ms-exchange-transport-fromentityheader: Hosted Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: aculab.com Content-Language: en-US Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" osq_lock() starts by setting node->next to NULL and node->locked to 0. Careful analysis shows that node->next is always NULL on entry. node->locked is set non-zero by another cpu to force a wakeup. This can only happen after the 'prev->next =3D node' assignment, so locked can be set to zero just before that (along with the assignment to node->prev). Only initialise node->cpu once, after that use its value instead of smp_processor_id() - which is probably a real function call. Should reduce cache-line bouncing a little. Signed-off-by: David Laight --- kernel/locking/osq_lock.c | 13 ++++++------- 1 file changed, 6 insertions(+), 7 deletions(-) diff --git a/kernel/locking/osq_lock.c b/kernel/locking/osq_lock.c index d414eef4bec6..55f5db896c02 100644 --- a/kernel/locking/osq_lock.c +++ b/kernel/locking/osq_lock.c @@ -51,7 +51,7 @@ osq_wait_next(struct optimistic_spin_queue *lock, struct optimistic_spin_node *prev) { struct optimistic_spin_node *next =3D NULL; - int curr =3D encode_cpu(smp_processor_id()); + int curr =3D node->cpu; int old; =20 /* @@ -98,12 +98,10 @@ bool osq_lock(struct optimistic_spin_queue *lock) { struct optimistic_spin_node *node =3D this_cpu_ptr(&osq_node); struct optimistic_spin_node *prev, *next; - int curr =3D encode_cpu(smp_processor_id()); int old; =20 - node->locked =3D 0; - node->next =3D NULL; - node->cpu =3D curr; + if (unlikely(node->cpu =3D=3D OSQ_UNLOCKED_VAL)) + node->cpu =3D encode_cpu(smp_processor_id()); =20 /* * We need both ACQUIRE (pairs with corresponding RELEASE in @@ -111,12 +109,13 @@ bool osq_lock(struct optimistic_spin_queue *lock) * the node fields we just initialised) semantics when updating * the lock tail. */ - old =3D atomic_xchg(&lock->tail, curr); + old =3D atomic_xchg(&lock->tail, node->cpu); if (old =3D=3D OSQ_UNLOCKED_VAL) return true; =20 prev =3D decode_cpu(old); node->prev =3D prev; + node->locked =3D 0; =20 /* * osq_lock() unqueue @@ -214,7 +213,7 @@ bool osq_lock(struct optimistic_spin_queue *lock) void osq_unlock(struct optimistic_spin_queue *lock) { struct optimistic_spin_node *node, *next; - int curr =3D encode_cpu(smp_processor_id()); + int curr =3D raw_cpu_read(osq_node.cpu); =20 /* * Fast path for the uncontended case. --=20 2.17.1 - Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1= PT, UK Registration No: 1397386 (Wales) From nobody Fri Dec 26 21:24:00 2025 Received: from eu-smtp-delivery-151.mimecast.com (eu-smtp-delivery-151.mimecast.com [185.58.86.151]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B480914A84 for ; Fri, 29 Dec 2023 20:56:24 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=ACULAB.COM Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=aculab.com Received: from AcuMS.aculab.com (156.67.243.121 [156.67.243.121]) by relay.mimecast.com with ESMTP with both STARTTLS and AUTH (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id uk-mta-129-6k2WdJMjPBWp2oR0msDm8g-1; Fri, 29 Dec 2023 20:56:21 +0000 X-MC-Unique: 6k2WdJMjPBWp2oR0msDm8g-1 Received: from AcuMS.Aculab.com (10.202.163.4) by AcuMS.aculab.com (10.202.163.4) with Microsoft SMTP Server (TLS) id 15.0.1497.48; Fri, 29 Dec 2023 20:56:03 +0000 Received: from AcuMS.Aculab.com ([::1]) by AcuMS.aculab.com ([::1]) with mapi id 15.00.1497.048; Fri, 29 Dec 2023 20:56:03 +0000 From: David Laight To: "'linux-kernel@vger.kernel.org'" , "'peterz@infradead.org'" , "'longman@redhat.com'" CC: "'mingo@redhat.com'" , "'will@kernel.org'" , "'boqun.feng@gmail.com'" , "'Linus Torvalds'" , "'xinhui.pan@linux.vnet.ibm.com'" , "'virtualization@lists.linux-foundation.org'" , 'Zeng Heng' Subject: [PATCH next 3/5] locking/osq_lock: Clarify osq_wait_next() Thread-Topic: [PATCH next 3/5] locking/osq_lock: Clarify osq_wait_next() Thread-Index: Ado6mW78N1Y4ctAHQ36yzrF5D7u/HQ== Date: Fri, 29 Dec 2023 20:56:03 +0000 Message-ID: <7c8828aec72e42eeb841ca0ee3397e9a@AcuMS.aculab.com> References: <73a4b31c9c874081baabad9e5f2e5204@AcuMS.aculab.com> In-Reply-To: <73a4b31c9c874081baabad9e5f2e5204@AcuMS.aculab.com> Accept-Language: en-GB, en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ms-exchange-transport-fromentityheader: Hosted Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: aculab.com Content-Language: en-US Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" osq_wait_next() is passed 'prev' from osq_lock() and NULL from osq_unlock() but only needs the 'cpu' value to write to lock->tail. Just pass prev->cpu or OSQ_UNLOCKED_VAL instead. Also directly return NULL or 'next' instead of breaking the loop. Should have no effect on the generated code since gcc manages to assume that 'prev !=3D NULL' due to an earlier dereference. Signed-off-by: David Laight --- kernel/locking/osq_lock.c | 23 ++++++++++------------- 1 file changed, 10 insertions(+), 13 deletions(-) diff --git a/kernel/locking/osq_lock.c b/kernel/locking/osq_lock.c index 55f5db896c02..9bb3a077ba92 100644 --- a/kernel/locking/osq_lock.c +++ b/kernel/locking/osq_lock.c @@ -48,18 +48,17 @@ static inline struct optimistic_spin_node *decode_cpu(i= nt encoded_cpu_val) static inline struct optimistic_spin_node * osq_wait_next(struct optimistic_spin_queue *lock, struct optimistic_spin_node *node, - struct optimistic_spin_node *prev) + int old) { - struct optimistic_spin_node *next =3D NULL; + struct optimistic_spin_node *next; int curr =3D node->cpu; - int old; =20 /* - * If there is a prev node in queue, then the 'old' value will be - * the prev node's CPU #, else it's set to OSQ_UNLOCKED_VAL since if - * we're currently last in queue, then the queue will then become empty. + * If osq_lock() is being cancelled there must be a previous node + * and 'old' is its CPU #. + * For osq_unlock() there is never a previous node and old is set + * to OSQ_UNLOCKED_VAL. */ - old =3D prev ? prev->cpu : OSQ_UNLOCKED_VAL; =20 for (;;) { if (atomic_read(&lock->tail) =3D=3D curr && @@ -69,7 +68,7 @@ osq_wait_next(struct optimistic_spin_queue *lock, * will now observe @lock and will complete its * unlock()/unqueue(). */ - break; + return NULL; } =20 /* @@ -85,13 +84,11 @@ osq_wait_next(struct optimistic_spin_queue *lock, if (node->next) { next =3D xchg(&node->next, NULL); if (next) - break; + return next; } =20 cpu_relax(); } - - return next; } =20 bool osq_lock(struct optimistic_spin_queue *lock) @@ -192,7 +189,7 @@ bool osq_lock(struct optimistic_spin_queue *lock) * back to @prev. */ =20 - next =3D osq_wait_next(lock, node, prev); + next =3D osq_wait_next(lock, node, prev->cpu); if (!next) return false; =20 @@ -232,7 +229,7 @@ void osq_unlock(struct optimistic_spin_queue *lock) return; } =20 - next =3D osq_wait_next(lock, node, NULL); + next =3D osq_wait_next(lock, node, OSQ_UNLOCKED_VAL); if (next) WRITE_ONCE(next->locked, 1); } --=20 2.17.1 - Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1= PT, UK Registration No: 1397386 (Wales) From nobody Fri Dec 26 21:24:00 2025 Received: from eu-smtp-delivery-151.mimecast.com (eu-smtp-delivery-151.mimecast.com [185.58.86.151]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C13B914A83 for ; Fri, 29 Dec 2023 20:57:34 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=ACULAB.COM Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=aculab.com Received: from AcuMS.aculab.com (156.67.243.121 [156.67.243.121]) by relay.mimecast.com with ESMTP with both STARTTLS and AUTH (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id uk-mta-311-yCoQCupGOBSlWEcLQ3ukbA-1; Fri, 29 Dec 2023 20:57:31 +0000 X-MC-Unique: yCoQCupGOBSlWEcLQ3ukbA-1 Received: from AcuMS.Aculab.com (10.202.163.4) by AcuMS.aculab.com (10.202.163.4) with Microsoft SMTP Server (TLS) id 15.0.1497.48; Fri, 29 Dec 2023 20:57:14 +0000 Received: from AcuMS.Aculab.com ([::1]) by AcuMS.aculab.com ([::1]) with mapi id 15.00.1497.048; Fri, 29 Dec 2023 20:57:14 +0000 From: David Laight To: "'linux-kernel@vger.kernel.org'" , "'peterz@infradead.org'" , "'longman@redhat.com'" CC: "'mingo@redhat.com'" , "'will@kernel.org'" , "'boqun.feng@gmail.com'" , "'Linus Torvalds'" , "'xinhui.pan@linux.vnet.ibm.com'" , "'virtualization@lists.linux-foundation.org'" , 'Zeng Heng' Subject: [PATCH next 4/5] locking/osq_lock: Optimise per-cpu data accesses. Thread-Topic: [PATCH next 4/5] locking/osq_lock: Optimise per-cpu data accesses. Thread-Index: Ado6mZUJWFdx4PkETd+mn/PWVjPd0A== Date: Fri, 29 Dec 2023 20:57:13 +0000 Message-ID: References: <73a4b31c9c874081baabad9e5f2e5204@AcuMS.aculab.com> In-Reply-To: <73a4b31c9c874081baabad9e5f2e5204@AcuMS.aculab.com> Accept-Language: en-GB, en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ms-exchange-transport-fromentityheader: Hosted Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: aculab.com Content-Language: en-US Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" this_cpu_ptr() is rather more expensive than raw_cpu_read() since the latter can use an 'offset from register' (%gs for x86-84). Add a 'self' field to 'struct optimistic_spin_node' that can be read with raw_cpu_read(), initialise on first call. Signed-off-by: David Laight --- kernel/locking/osq_lock.c | 14 +++++++++----- 1 file changed, 9 insertions(+), 5 deletions(-) diff --git a/kernel/locking/osq_lock.c b/kernel/locking/osq_lock.c index 9bb3a077ba92..b60b0add0161 100644 --- a/kernel/locking/osq_lock.c +++ b/kernel/locking/osq_lock.c @@ -13,7 +13,7 @@ */ =20 struct optimistic_spin_node { - struct optimistic_spin_node *next, *prev; + struct optimistic_spin_node *self, *next, *prev; int locked; /* 1 if lock acquired */ int cpu; /* encoded CPU # + 1 value */ }; @@ -93,12 +93,16 @@ osq_wait_next(struct optimistic_spin_queue *lock, =20 bool osq_lock(struct optimistic_spin_queue *lock) { - struct optimistic_spin_node *node =3D this_cpu_ptr(&osq_node); + struct optimistic_spin_node *node =3D raw_cpu_read(osq_node.self); struct optimistic_spin_node *prev, *next; int old; =20 - if (unlikely(node->cpu =3D=3D OSQ_UNLOCKED_VAL)) - node->cpu =3D encode_cpu(smp_processor_id()); + if (unlikely(!node)) { + int cpu =3D encode_cpu(smp_processor_id()); + node =3D decode_cpu(cpu); + node->self =3D node; + node->cpu =3D cpu; + } =20 /* * We need both ACQUIRE (pairs with corresponding RELEASE in @@ -222,7 +226,7 @@ void osq_unlock(struct optimistic_spin_queue *lock) /* * Second most likely case. */ - node =3D this_cpu_ptr(&osq_node); + node =3D raw_cpu_read(osq_node.self); next =3D xchg(&node->next, NULL); if (next) { WRITE_ONCE(next->locked, 1); --=20 2.17.1 - Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1= PT, UK Registration No: 1397386 (Wales) From nobody Fri Dec 26 21:24:00 2025 Received: from eu-smtp-delivery-151.mimecast.com (eu-smtp-delivery-151.mimecast.com [185.58.85.151]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id ACEA91426F for ; Fri, 29 Dec 2023 20:58:41 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=ACULAB.COM Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=aculab.com Received: from AcuMS.aculab.com (156.67.243.121 [156.67.243.121]) by relay.mimecast.com with ESMTP with both STARTTLS and AUTH (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id uk-mta-202-bl5_I54CNsuf5Ujw93Ax2g-1; Fri, 29 Dec 2023 20:58:32 +0000 X-MC-Unique: bl5_I54CNsuf5Ujw93Ax2g-1 Received: from AcuMS.Aculab.com (10.202.163.4) by AcuMS.aculab.com (10.202.163.4) with Microsoft SMTP Server (TLS) id 15.0.1497.48; Fri, 29 Dec 2023 20:58:15 +0000 Received: from AcuMS.Aculab.com ([::1]) by AcuMS.aculab.com ([::1]) with mapi id 15.00.1497.048; Fri, 29 Dec 2023 20:58:15 +0000 From: David Laight To: "'linux-kernel@vger.kernel.org'" , "'peterz@infradead.org'" , "'longman@redhat.com'" CC: "'mingo@redhat.com'" , "'will@kernel.org'" , "'boqun.feng@gmail.com'" , "'Linus Torvalds'" , "'xinhui.pan@linux.vnet.ibm.com'" , "'virtualization@lists.linux-foundation.org'" , 'Zeng Heng' Subject: [PATCH next 5/5] locking/osq_lock: Optimise vcpu_is_preempted() check. Thread-Topic: [PATCH next 5/5] locking/osq_lock: Optimise vcpu_is_preempted() check. Thread-Index: Ado6mcFsTi5k8LaETrKavOOIB4in0Q== Date: Fri, 29 Dec 2023 20:58:15 +0000 Message-ID: <23cef5ac49494b9087953f529ae5df16@AcuMS.aculab.com> References: <73a4b31c9c874081baabad9e5f2e5204@AcuMS.aculab.com> In-Reply-To: <73a4b31c9c874081baabad9e5f2e5204@AcuMS.aculab.com> Accept-Language: en-GB, en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ms-exchange-transport-fromentityheader: Hosted Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: aculab.com Content-Language: en-US Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" The vcpu_is_preempted() test stops osq_lock() spinning if a virtual cpu is no longer running. Although patched out for bare-metal the code still needs the cpu number. Reading this from 'prev->cpu' is a pretty much guaranteed have a cache miss when osq_unlock() is waking up the next cpu. Instead save 'prev->cpu' in 'node->prev_cpu' and use that value instead. Update in the osq_lock() 'unqueue' path when 'node->prev' is changed. This is simpler than checking for 'node->prev' changing and caching 'prev->cpu'. Signed-off-by: David Laight Reviewed-by: Waiman Long --- kernel/locking/osq_lock.c | 14 ++++++-------- 1 file changed, 6 insertions(+), 8 deletions(-) diff --git a/kernel/locking/osq_lock.c b/kernel/locking/osq_lock.c index b60b0add0161..89be63627434 100644 --- a/kernel/locking/osq_lock.c +++ b/kernel/locking/osq_lock.c @@ -14,8 +14,9 @@ =20 struct optimistic_spin_node { struct optimistic_spin_node *self, *next, *prev; - int locked; /* 1 if lock acquired */ - int cpu; /* encoded CPU # + 1 value */ + int locked; /* 1 if lock acquired */ + int cpu; /* encoded CPU # + 1 value */ + int prev_cpu; /* actual CPU # for vpcu_is_preempted() */ }; =20 static DEFINE_PER_CPU_SHARED_ALIGNED(struct optimistic_spin_node, osq_node= ); @@ -29,11 +30,6 @@ static inline int encode_cpu(int cpu_nr) return cpu_nr + 1; } =20 -static inline int node_cpu(struct optimistic_spin_node *node) -{ - return node->cpu - 1; -} - static inline struct optimistic_spin_node *decode_cpu(int encoded_cpu_val) { int cpu_nr =3D encoded_cpu_val - 1; @@ -114,6 +110,7 @@ bool osq_lock(struct optimistic_spin_queue *lock) if (old =3D=3D OSQ_UNLOCKED_VAL) return true; =20 + node->prev_cpu =3D old - 1; prev =3D decode_cpu(old); node->prev =3D prev; node->locked =3D 0; @@ -148,7 +145,7 @@ bool osq_lock(struct optimistic_spin_queue *lock) * polling, be careful. */ if (smp_cond_load_relaxed(&node->locked, VAL || need_resched() || - vcpu_is_preempted(node_cpu(node->prev)))) + vcpu_is_preempted(node->prev_cpu))) return true; =20 /* unqueue */ @@ -205,6 +202,7 @@ bool osq_lock(struct optimistic_spin_queue *lock) * it will wait in Step-A. */ =20 + WRITE_ONCE(next->prev_cpu, prev->cpu - 1); WRITE_ONCE(next->prev, prev); WRITE_ONCE(prev->next, next); =20 --=20 2.17.1 - Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1= PT, UK Registration No: 1397386 (Wales)