From nobody Fri Dec 26 23:25:35 2025 Received: from eu-smtp-delivery-151.mimecast.com (eu-smtp-delivery-151.mimecast.com [185.58.86.151]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C13B914A83 for ; Fri, 29 Dec 2023 20:57:34 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=ACULAB.COM Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=aculab.com Received: from AcuMS.aculab.com (156.67.243.121 [156.67.243.121]) by relay.mimecast.com with ESMTP with both STARTTLS and AUTH (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id uk-mta-311-yCoQCupGOBSlWEcLQ3ukbA-1; Fri, 29 Dec 2023 20:57:31 +0000 X-MC-Unique: yCoQCupGOBSlWEcLQ3ukbA-1 Received: from AcuMS.Aculab.com (10.202.163.4) by AcuMS.aculab.com (10.202.163.4) with Microsoft SMTP Server (TLS) id 15.0.1497.48; Fri, 29 Dec 2023 20:57:14 +0000 Received: from AcuMS.Aculab.com ([::1]) by AcuMS.aculab.com ([::1]) with mapi id 15.00.1497.048; Fri, 29 Dec 2023 20:57:14 +0000 From: David Laight To: "'linux-kernel@vger.kernel.org'" , "'peterz@infradead.org'" , "'longman@redhat.com'" CC: "'mingo@redhat.com'" , "'will@kernel.org'" , "'boqun.feng@gmail.com'" , "'Linus Torvalds'" , "'xinhui.pan@linux.vnet.ibm.com'" , "'virtualization@lists.linux-foundation.org'" , 'Zeng Heng' Subject: [PATCH next 4/5] locking/osq_lock: Optimise per-cpu data accesses. Thread-Topic: [PATCH next 4/5] locking/osq_lock: Optimise per-cpu data accesses. Thread-Index: Ado6mZUJWFdx4PkETd+mn/PWVjPd0A== Date: Fri, 29 Dec 2023 20:57:13 +0000 Message-ID: References: <73a4b31c9c874081baabad9e5f2e5204@AcuMS.aculab.com> In-Reply-To: <73a4b31c9c874081baabad9e5f2e5204@AcuMS.aculab.com> Accept-Language: en-GB, en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ms-exchange-transport-fromentityheader: Hosted Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: aculab.com Content-Language: en-US Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" this_cpu_ptr() is rather more expensive than raw_cpu_read() since the latter can use an 'offset from register' (%gs for x86-84). Add a 'self' field to 'struct optimistic_spin_node' that can be read with raw_cpu_read(), initialise on first call. Signed-off-by: David Laight --- kernel/locking/osq_lock.c | 14 +++++++++----- 1 file changed, 9 insertions(+), 5 deletions(-) diff --git a/kernel/locking/osq_lock.c b/kernel/locking/osq_lock.c index 9bb3a077ba92..b60b0add0161 100644 --- a/kernel/locking/osq_lock.c +++ b/kernel/locking/osq_lock.c @@ -13,7 +13,7 @@ */ =20 struct optimistic_spin_node { - struct optimistic_spin_node *next, *prev; + struct optimistic_spin_node *self, *next, *prev; int locked; /* 1 if lock acquired */ int cpu; /* encoded CPU # + 1 value */ }; @@ -93,12 +93,16 @@ osq_wait_next(struct optimistic_spin_queue *lock, =20 bool osq_lock(struct optimistic_spin_queue *lock) { - struct optimistic_spin_node *node =3D this_cpu_ptr(&osq_node); + struct optimistic_spin_node *node =3D raw_cpu_read(osq_node.self); struct optimistic_spin_node *prev, *next; int old; =20 - if (unlikely(node->cpu =3D=3D OSQ_UNLOCKED_VAL)) - node->cpu =3D encode_cpu(smp_processor_id()); + if (unlikely(!node)) { + int cpu =3D encode_cpu(smp_processor_id()); + node =3D decode_cpu(cpu); + node->self =3D node; + node->cpu =3D cpu; + } =20 /* * We need both ACQUIRE (pairs with corresponding RELEASE in @@ -222,7 +226,7 @@ void osq_unlock(struct optimistic_spin_queue *lock) /* * Second most likely case. */ - node =3D this_cpu_ptr(&osq_node); + node =3D raw_cpu_read(osq_node.self); next =3D xchg(&node->next, NULL); if (next) { WRITE_ONCE(next->locked, 1); --=20 2.17.1 - Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1= PT, UK Registration No: 1397386 (Wales)