From nobody Fri Sep 25 23:53:11 2026 Received: from mail-wr1-f48.google.com (mail-wr1-f48.google.com [209.85.221.48]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B0E72446BF5 for ; Mon, 7 Sep 2026 08:41:52 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.221.48 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788770514; cv=none; b=tCY/MjhR/17Z5gKCABbrEIRnb2uhF6JNd4ZKQJx/lFIBNTCHvuuLjVXYJOjezQpFefe5HUd8ZYmW+2uQ3S30nOgvxhopPDbHXCZtI2qTu6UkMX7GIFAoLxgVz1eS7wYNxEYaOT5FHuf0l628m/R4lGYPPaWAuusrN4Mw0Hg9Ckg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788770514; c=relaxed/simple; bh=wfXE8uypbKYJA7BC8MjKFDfmD5UiS7JineuAHlHdzVU=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=NEMtle41B7M89EOxd5wiW/a7pgTvWX/+za3cwcutF2tO/Q5gZYVZwOYo5qA4ApKBcHj7bxrcenZodcmanRle5R4EcEL8yNrX9NTniztTQ7wTkWLsvYDEs53iNT/6kN8+70ealcm5BKT066CFfCqPue4Z6iPXejRxkiHp7B7T+3g= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=rHslAWBB; arc=none smtp.client-ip=209.85.221.48 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="rHslAWBB" Received: by mail-wr1-f48.google.com with SMTP id ffacd0b85a97d-4858303de5dso4042278f8f.2 for ; Mon, 07 Sep 2026 01:41:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1788770511; x=1789375311; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=AKJGGS8rHZVSsitHp/iDOAiBUo3rGEd7SOXEHhtBYe4=; b=rHslAWBBqhOLnkn9peWNZwbKTAyASpoplN0RUcEW78yNjECpoHlfaKp7Dm4WaskYAL YcgCX3pDrC+evX4GkjUO8buBr+XVpByc5Hu8l9Ypgz10ddFxUa9T4Gck5vYG1i1V9RaN bhDKsN5hL3B7nTikbn0OPRn4ZPZtc/nqgO3Qsi40xrwb/7g27j7UWi78jLLhYXy3rcY0 wxDNbfU2tsnPbDhP5DiN203dYih7T46RVnhRUDIjj20ZsB29qDxdUjUimgb76klCGfet mDSaK55puuOCcpiVJgSRoGv4NpbsCahVkxLkbPt1sK0UP5lqHpJu7oMfaHjYh4yOnb/2 yTQQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1788770511; x=1789375311; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=AKJGGS8rHZVSsitHp/iDOAiBUo3rGEd7SOXEHhtBYe4=; b=TVauXHp/zEplOzb80Jv3Y8f7FYzS8BwjQueLTb9SiSruI6XKXsW7cYFX74HZDzJtHg aOabhEZkDw+VvmvI5f7k5ZOaDHcCSIdxfGnxgH6G3WUxMK2cByGEhjXRTpYQhmSNg/sH rBF5oicHjr8E/MW4nnVwkG6jABcfwpKeLsgDkgrIaKcmWylOP4I1zpOZlIE5HRFkZlON gCx7NdGShuXEip7DPIMbBN5Cd8GyfE+S+OrtZ9tTaBfncIaS6sfrq0L05BFmVH5k7Qzj VL1PLU+l0IV/KVoRShQozKZ0VH02prtd3+Vd5VOUdLzMILlmY1RwWjluh130x0MlnlKh iTBQ== X-Forwarded-Encrypted: i=1; AKwUvBw95+hFnawbDEN9kOMVLXPkjhYEBb/WpGk9N+8IpBelkq1e8Ii4eBl/imqgP2TfTc9S0LOHxisJIIrnZn0=@vger.kernel.org X-Gm-Message-State: AFuF++nkOqon6xu+lC/N3aKTWmkDIfUptgGG0pZdTLOjEFUiPL0BD2mc J4W788qZJZRZa36CihGptEkgUufqOQ0cGD7s8f4yRIbo12xtuvdusSV2 X-Gm-Gg: AYBFou3+tAhJ/5nVtrxpw7bmsyiY2KumcpiQSGWM5KUGSdEvzJNU8so1tRk1jSQ3Ub7 4FHIPXhTxRdUcqBXNuZeqiX43kkAXkGsun8sJl24vy3p8EbnLoAPIl0V+OomAPiC6nIndKBfzO8 UDBfZXxOVUFuChBoB3GJb/C2JmgNd1WsobPqfsurqJYjkShv5T+5G0l86vE3VknQUzBh5FlZQBP 71i9JA4LwpISi8cxGYiSq9H6Al8wlcn+wqffvjQMEkXla/inhIT25hx5GRy9UwZGU+N9s7yBOWm cIoSYcTh6qeA4Hyzj0/gCNl8n+pZH0rEopipKNRC+oa3986MRpcBlryqDVretaJ8K3Sdk0EhfgE aYN1ybNow8Dw9e1sOTgHWuoYHssZyVom29p31G/ObFOdF2aA6IRKOhl/AS8sazAlmQi/KVT770X yq/KwjGa3Y3dUlNVfHPvtKzUWbmdf4TnKqF8IcdIBT8plTo2B7stJU67b0iGg7C8l5gAOG5ftWu XLOTxV+L7BiX2vXSol9FQlIoQeA+5PZw2A8XvzyYrAtcA== X-Received: by 2002:a05:6000:4555:b0:484:68ed:2afc with SMTP id ffacd0b85a97d-48587059c26mr19692495f8f.11.1788770510854; Mon, 07 Sep 2026 01:41:50 -0700 (PDT) Received: from snowdrop.snailnet.com (82-69-66-36.dsl.in-addr.zen.co.uk. [82.69.66.36]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-485885bbb51sm27883762f8f.30.2026.09.07.01.41.50 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 07 Sep 2026 01:41:50 -0700 (PDT) From: David Laight To: Waiman Long , Peter Zijlstra , Ingo Molnar , Will Deacon , Boqun Feng , linux-kernel@vger.kernel.org, Linus Torvalds , Yafang Shao , Steven Rostedt Cc: David Laight Subject: [PATCH v4 next 1/9] locking/osq_lock: Add some comments about how it works Date: Mon, 7 Sep 2026 09:41:25 +0100 Message-Id: <20260907084133.3696-2-david.laight.linux@gmail.com> X-Mailer: git-send-email 2.39.5 In-Reply-To: <20260907084133.3696-1-david.laight.linux@gmail.com> References: <20260907084133.3696-1-david.laight.linux@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" No code changes, just some extra explanations. Signed-off-by: David Laight Tested-by: H=C3=A5kon Bugge --- kernel/locking/osq_lock.c | 27 ++++++++++++++++++++++++--- 1 file changed, 24 insertions(+), 3 deletions(-) diff --git a/kernel/locking/osq_lock.c b/kernel/locking/osq_lock.c index b4233dc2c2b0..b17aa704c449 100644 --- a/kernel/locking/osq_lock.c +++ b/kernel/locking/osq_lock.c @@ -4,12 +4,33 @@ #include =20 /* - * An MCS like lock especially tailored for optimistic spinning for sleepi= ng - * lock implementations (mutex, rwsem, etc). + * An MCS like spin lock especially tailored for optimistic spinning for + * sleeping lock implementations (mutex, rwsem, etc). + * Each CPU spins on a local variable to avoid cache-line bounces. * - * Using a single mcs node per CPU is safe because sleeping locks should n= ot be + * The CPU that holds the osq_lock checks the mutex/rwsem, the other CPU s= pin + * in osq_lock() until either the osq_lock is obtained or the scheduler + * requests the process be preempted. + * + * Using a single osq node per CPU is safe because sleeping locks should n= ot be * called from interrupt context and we have preemption disabled while * spinning. + * + * The osq_nodes for the spinning CPU are put on a double-linked (non circ= ular) + * list. The list 'pointers' can either be the address of the osq_node or = the + * associated CPU number, the CPU numbers are offset by one so that zero c= an + * be used like a NULL ponter. + * The mutex/rwsem contains a pointer (CPU number) to the tail of the list. + * There is no equivalent pointer to the list head - the 'head' is the + * osq_node of the CPU that acquired the osq lock. + * + * The 'next' pointer of the tail must be NULL, all the other 'next' point= ers + * must either be valid or transiently NULL. + * The 'prev' pointers only need to be valid when node->prev makes sense a= nd, + * even then, can be transiently invalid (ie refer to the wrong node). + * They are only used for the node->prev->next =3D node->next update when + * 'node' is being removed. Atomically checking node->prev->next =3D=3D no= de + * ensures the list doesn't get corrupted. */ =20 struct optimistic_spin_node { --=20 2.39.5 From nobody Fri Sep 25 23:53:11 2026 Received: from mail-wr1-f52.google.com (mail-wr1-f52.google.com [209.85.221.52]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9618B3BF684 for ; Mon, 7 Sep 2026 08:41:53 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.221.52 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788770515; cv=none; b=ovAKQshHgo5TW73QnbBGRWBNrMzbbPik7+uxD3G+AVLa8joFaENknMzAIBzAp1oCmcDdK0O6gdPmkZJlgzATp1IdooMcVf7P/8nbxgw54WBEpwO2kILsTDokBnXiomr5FTETgN6dGoPRtJ+lReqOt4ZMOMnu6lFCynVDGKzHCxA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788770515; c=relaxed/simple; bh=lfF6QCpuhc2Dos0ZBBGQ0V6zVcntqow7aVKgsPvrhvg=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=Gp/qB6avDu84z7XZaHZSHpT41ZguSYenCZ//SzG8WTvGT3IO/aY6wXQhachjlzQ4dqCveV4K4dv16a4cczkiTiZjNuQz8/AdensMVGHxMM1DhiIIop5P0o5fybhoh8MZ4+kQP4Iv1qyNChYezMyGAWrv6XasYgh49Hu6zniSmAc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=tJrDSHuK; arc=none smtp.client-ip=209.85.221.52 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="tJrDSHuK" Received: by mail-wr1-f52.google.com with SMTP id ffacd0b85a97d-485843aeab8so3631237f8f.1 for ; Mon, 07 Sep 2026 01:41:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1788770512; x=1789375312; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=AK0HAdhUgdaBbUhHuXj+SBu9juYJIyCoUjyrh+fGc6A=; b=tJrDSHuKTQ83vbmSS/+AaBHFLta+0E0adnEVR2kf208nL0mMo4RDxyPJEIBjZP+xlm Jwh1grW3B2nOzGdWTGEnXzUzoV90oxHsau+ltcrS1UUEcucCtey70QsVyek5ED+lslV7 4D1+W4RM4ho2uJyJr5dt00SXsM7ItbAjem91juDS0IK3q/SVw1gYLGXhVgaLVaF1SLg8 gZnlUCMQAB6AApmFvzJbsoEYqXVjAyik1PgtljpCpUBtqdnEbrOduAQCyJGdMbovwd2X y/KDnOI1QTNvhT+2wSxl8r9caiZAeaR+QjKWKrza86KDATNzMnzRu/tco+eyp2eRe/kK zulQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1788770512; x=1789375312; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=AK0HAdhUgdaBbUhHuXj+SBu9juYJIyCoUjyrh+fGc6A=; b=eq7YOv8OGODfMJkRoIjYohsrsdyx3JHlHHTRxE0oB4tbzOeR1iSsLkuRtWZ/51HMOn tP34NBodtBofDHYhCCi/g+UbTw+vZHsl0wB0X30hLKqhiGQvppbk8yKEyIUFOsQHpP/i jmLGsdELMKxevbvW6HsgnMDFb+9+IQ07wZk1dy6OMOYIX4mUaJEn3Oid/nMFxK59JTiV mIJuJjKTEsMBhlF54WUXqOL1MEZuRWz92ZwrpUmNtVuvMctd4bg3PH3zkHuaVKUToAPs 5+x+0uQY+3xrNIE+cJ22vx32ZhlBcMQahYPO9ZwlGcGbn1AO/0LHoj2gN93vCNXbICDi p0eg== X-Forwarded-Encrypted: i=1; AKwUvBw6G+CS/3PAqDtOhRxf/PrmHQMg619Bgka85J3KglplxbXXhW2SPhOjiItHZ7jCPYh3RswJvY3WwVaxBOQ=@vger.kernel.org X-Gm-Message-State: AFuF++kN6GtKmwt0eVrhxk1u5R8H6PwrhW5C6Egfot5BNASWc4fHcOHR 1cH4jhAbd/y1FwAFS7rnml7SAlyTkxU9rcJgcM2Jt6cjHUT/adAgkNj6 X-Gm-Gg: AYBFou0ZRPkXHZxCwnM3p6h8Pt7qf1rTyX0yZHBCLiJ5RBaRcsG8bcwwhyMUkASAdPr AjNGybZndxygNiNvf9NIsgo5Aw8WqoeGz/eFJkC1RDBaw8V7ohfgLjRwjIs0u5PWiuWeKk3xUV6 DUCljy0vKIK1/2jdY0hG7cVIVVNJgYoybutS+dxJSxqFezJo2TO/gLdn1Hddh3plAoR1aOFNhf3 cEcFpQqH0BPNdeQWCU2jx17pQ9v/fSM2ogmBuO7e6wV2tP8XePN9Lndcw1DU9N3YCWcehdfPaHm nLzpV2DX/MAJDhY+8XoQIO1KLdeZKPP6IYRSMo2s2C7oqv7/7mEK2vxPWJ+r6liG19yRSFCmYt5 kuGXoPdxA8N4MMTopgsB68CHDt+4nBzixJwK3hUvMK7/ZZknluQHWvVBiy7qHkZ21DmZGNozOnh /awZMIiU4RPb/uEbzhxmwT9XgytK6xbNxvhP3hmDs8o7+/JZm4OnBb6vaogwo5YkMwrbt1eVbwu 2/7bps+J6SqOYJc1qgF+7ql7/46OwRT+CkSrShAL+UM1w== X-Received: by 2002:a05:6000:41d4:b0:484:3310:c4fc with SMTP id ffacd0b85a97d-48587090793mr44242185f8f.23.1788770511730; Mon, 07 Sep 2026 01:41:51 -0700 (PDT) Received: from snowdrop.snailnet.com (82-69-66-36.dsl.in-addr.zen.co.uk. [82.69.66.36]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-485885bbb51sm27883762f8f.30.2026.09.07.01.41.50 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 07 Sep 2026 01:41:51 -0700 (PDT) From: David Laight To: Waiman Long , Peter Zijlstra , Ingo Molnar , Will Deacon , Boqun Feng , linux-kernel@vger.kernel.org, Linus Torvalds , Yafang Shao , Steven Rostedt Cc: David Laight Subject: [PATCH v4 next 2/9] locking/osq_lock: Save the cpu number for 'prev' not the node address Date: Mon, 7 Sep 2026 09:41:26 +0100 Message-Id: <20260907084133.3696-3-david.laight.linux@gmail.com> X-Mailer: git-send-email 2.39.5 In-Reply-To: <20260907084133.3696-1-david.laight.linux@gmail.com> References: <20260907084133.3696-1-david.laight.linux@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" The cpu number of node->prev is needed for both the vcpu_is_preempted() test and to update lock->tail. This saves reading the cache line for the other cpu's per-cpu data. The cpu member of optimistic_spin_node is no longer needed. Merges patches 2 and 3 from v3. Signed-off-by: David Laight Reviewed-by: Waiman Long Tested-by: H=C3=A5kon Bugge --- kernel/locking/osq_lock.c | 33 ++++++++++++++------------------- 1 file changed, 14 insertions(+), 19 deletions(-) diff --git a/kernel/locking/osq_lock.c b/kernel/locking/osq_lock.c index b17aa704c449..01988d00c480 100644 --- a/kernel/locking/osq_lock.c +++ b/kernel/locking/osq_lock.c @@ -34,9 +34,9 @@ */ =20 struct optimistic_spin_node { - struct optimistic_spin_node *next, *prev; + struct optimistic_spin_node *next; int locked; /* 1 if lock acquired */ - int cpu; /* encoded CPU # + 1 value */ + int prev; /* CPU number offset by 1 */ }; =20 static DEFINE_PER_CPU_SHARED_ALIGNED(struct optimistic_spin_node, osq_node= ); @@ -50,11 +50,6 @@ static inline int encode_cpu(int cpu_nr) return cpu_nr + 1; } =20 -static inline int node_cpu(struct optimistic_spin_node *node) -{ - return node->cpu - 1; -} - static inline struct optimistic_spin_node *decode_cpu(int encoded_cpu_val) { int cpu_nr =3D encoded_cpu_val - 1; @@ -114,13 +109,12 @@ osq_wait_next(struct optimistic_spin_queue *lock, bool osq_lock(struct optimistic_spin_queue *lock) { struct optimistic_spin_node *node =3D this_cpu_ptr(&osq_node); - struct optimistic_spin_node *prev, *next; + struct optimistic_spin_node *prev_ptr, *next; int curr =3D encode_cpu(smp_processor_id()); - int old; + int prev; =20 node->locked =3D 0; node->next =3D NULL; - node->cpu =3D curr; =20 /* * We need both ACQUIRE (pairs with corresponding RELEASE in @@ -128,11 +122,11 @@ bool osq_lock(struct optimistic_spin_queue *lock) * the node fields we just initialised) semantics when updating * the lock tail. */ - old =3D atomic_xchg(&lock->tail, curr); - if (old =3D=3D OSQ_UNLOCKED_VAL) + prev =3D atomic_xchg(&lock->tail, curr); + if (prev =3D=3D OSQ_UNLOCKED_VAL) return true; =20 - prev =3D decode_cpu(old); + prev_ptr =3D decode_cpu(prev); node->prev =3D prev; =20 /* @@ -147,7 +141,7 @@ bool osq_lock(struct optimistic_spin_queue *lock) */ smp_wmb(); =20 - WRITE_ONCE(prev->next, node); + WRITE_ONCE(prev_ptr->next, node); =20 /* * Normally @prev is untouchable after the above store; because at that @@ -165,7 +159,7 @@ bool osq_lock(struct optimistic_spin_queue *lock) * polling, be careful. */ if (smp_cond_load_relaxed(&node->locked, VAL || need_resched() || - vcpu_is_preempted(node_cpu(node->prev)))) + vcpu_is_preempted(node->prev - 1))) return true; =20 /* unqueue */ @@ -182,8 +176,8 @@ bool osq_lock(struct optimistic_spin_queue *lock) * cpu_relax() below implies a compiler barrier which would * prevent this comparison being optimized away. */ - if (data_race(prev->next) =3D=3D node && - cmpxchg(&prev->next, node, NULL) =3D=3D node) + if (data_race(prev_ptr->next) =3D=3D node && + cmpxchg(&prev_ptr->next, node, NULL) =3D=3D node) break; =20 /* @@ -201,6 +195,7 @@ bool osq_lock(struct optimistic_spin_queue *lock) * case its step-C will write us a new @node->prev pointer. */ prev =3D READ_ONCE(node->prev); + prev_ptr =3D decode_cpu(prev); } =20 /* @@ -210,7 +205,7 @@ bool osq_lock(struct optimistic_spin_queue *lock) * back to @prev. */ =20 - next =3D osq_wait_next(lock, node, prev->cpu); + next =3D osq_wait_next(lock, node, prev); if (!next) return false; =20 @@ -223,7 +218,7 @@ bool osq_lock(struct optimistic_spin_queue *lock) */ =20 WRITE_ONCE(next->prev, prev); - WRITE_ONCE(prev->next, next); + WRITE_ONCE(prev_ptr->next, next); =20 return false; } --=20 2.39.5 From nobody Fri Sep 25 23:53:11 2026 Received: from mail-wr1-f45.google.com (mail-wr1-f45.google.com [209.85.221.45]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id CEFA34457C1 for ; Mon, 7 Sep 2026 08:41:54 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.221.45 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788770516; cv=none; b=ZHK8cRDf25qoPM9tPs08DqEcpi/j1sE61C+0kNk60jREkm0sVM+hy8oLJEYHoOPPJL2zttut4dGOgSBB3rLXhZShLxzmE6Vahrv3BB5CQsfP7EhKeD9XMgnQZ6A8GGOZr6LSC9lAYzHuHWAKYC40vr160BooF//+gX191tOvBqg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788770516; c=relaxed/simple; bh=hv8cXECJvwqDma7vH6hzcBZWkB16/ALMDaQN28aqyGQ=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=ij7ZlqEzIw57t3J732CivFIrHHF2bAzU2CJiXMZMsifrHYDLxjgEc2kgu94v1ZV2vdvDVn/LoYCwqHdi4MxbYQT/UFkQDANBLkmjBz3krAe/Xfk7MxinBib2m3pA+WaP/ZdEJT+h4hPuRKOmlrU/bEwZXpQSwsddLKZAIj4SmlU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=hlun8uXA; arc=none smtp.client-ip=209.85.221.45 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="hlun8uXA" Received: by mail-wr1-f45.google.com with SMTP id ffacd0b85a97d-482e067e908so2707700f8f.2 for ; Mon, 07 Sep 2026 01:41:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1788770513; x=1789375313; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=fGPUbzxQT9779lG2OcjBStlPVIZzNZS8ZPWK5ZjiNb0=; b=hlun8uXA6etLOBxaEBpNXe1CINSFpxbOGUOYlOXyTN4h8n8CeVSSzBcwmWqKeInKe/ rLK2n3pzKV7le4DXPstsQa6qDlXbNQ+07f4YVUWl1ELffq7fNOrQ4WZczxvFbJ5sZqlW UO4pcHWSdYN/zDkJvcuZ9suKMmaNPyika6YZe/mchFX7c3m4gwZtsIz5HAV92mJm8G2P rStXxL4q0F9zDsFjtazrAnttMz6ziO7jQ/e18VcRnLneSmHp38r315vqizE1ca8AToe8 qRjQZZqYIRCGo913Fsxrj4Au8xRyNINeeMtupFT7LPK8hskDjZC3kQ5Nsg8wdswmQlqY b64A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1788770513; x=1789375313; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=fGPUbzxQT9779lG2OcjBStlPVIZzNZS8ZPWK5ZjiNb0=; b=HoJEPUACzKPf2Uv7SZsY+UPh4enztVwNR2eE+XRXtUU5qh9QNBHqWjOPgvT7WOOaXH fP3229seGt6Gm3YZBo3SnqzOgAvJn2Ij6uXlJmb8xjcWLsR6HyRv/QWrHvdhAl+Kikyd mcqT0btjR303tQ95WdSZ8khLY3uMlL5y+DFQVsbiRtq8s+7DGdXNinLq5SHQLpPRSG18 tKZJvRFLdhqg7eDGTFKfmeaLLHf3eBLF+3kNrFqAjuruZy+Jyr2nsL/e0Q2FPGDIltII RxXKvyaoopirot/NCW0lPY8iX4UopK2alVH4SfzSpn9EPUsplUrAYRRfrtw2e8PsYa1G KulQ== X-Forwarded-Encrypted: i=1; AKwUvBxVU91nxXPDn681YOYwidOcpIIsf1a7Qt5loLmQeBDasFX6a52NKOJN13E0XOWRyz+mqnnnPRwi43m0Rqs=@vger.kernel.org X-Gm-Message-State: AFuF++k0sNlJA/1MJEYkKLVG5PLHKPujfeWnkI+au4SbdJz37EdOsibL 79aFndNRC7QOUYkwlVVl8UidAqNwbkrWYhI59l8IFpyejN1P0OSDbeUs X-Gm-Gg: AYBFou15nvC+IFy0WJHuD15FNHCwurqf7BRULgZrV6v+yFBT+dwLANOeAEOTU41Mr2B 768pxTnDYN3wbOxax24I/CDGXBmpQZ6JyjEZtbZWSXsxm2nNeKuXrlYB8Pozg/PF1XDG/pYXkDa 8RllbHOu3xezJGXooSB+6lkyN5Q5fma0lbVIj3wLTX/rxeIUwYRIJDgNsUfi8LT0tnFBJfS+VQZ G7p3x8M+IxWlyP0sn3JWyg3wvuKmmwmQIe1qDVqjxuodRZgHJxKZ5khg1+SkzHlsyEOxmWIaG01 YD7UAj1xIQepsFj9eATgHVDZXupeJWPfqcxBoGxobP+auwgIlYOX4ckEVy+qUbxH2YSn68IQ1y5 Uc09HCHrLYB5cBfG8Od4PflqtdTGFzRYLaWTDKkL6RqeOSuAGsIFaOx6xnDxW3gUpL0WZxDVSfn fJ2btOWbA4TxwWWPInq50M1TAs4Fl8UQbICOgFcdfVeLvj/IRGqXXmrsRfBcPsg/+lzQcWbm8hm allTMCS/4aOyPzskXCf2DbCDrDF/E4At4q0icF3M0OiLfrnE3MolnJL X-Received: by 2002:a05:6000:4555:b0:482:e1ce:d45c with SMTP id ffacd0b85a97d-485872c2589mr16242555f8f.21.1788770512841; Mon, 07 Sep 2026 01:41:52 -0700 (PDT) Received: from snowdrop.snailnet.com (82-69-66-36.dsl.in-addr.zen.co.uk. [82.69.66.36]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-485885bbb51sm27883762f8f.30.2026.09.07.01.41.51 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 07 Sep 2026 01:41:52 -0700 (PDT) From: David Laight To: Waiman Long , Peter Zijlstra , Ingo Molnar , Will Deacon , Boqun Feng , linux-kernel@vger.kernel.org, Linus Torvalds , Yafang Shao , Steven Rostedt Cc: David Laight Subject: [PATCH v4 next 3/9] locking/osq_lock: Set prev_cpu=0 instead of locked=1 Date: Mon, 7 Sep 2026 09:41:27 +0100 Message-Id: <20260907084133.3696-4-david.laight.linux@gmail.com> X-Mailer: git-send-email 2.39.5 In-Reply-To: <20260907084133.3696-1-david.laight.linux@gmail.com> References: <20260907084133.3696-1-david.laight.linux@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" There is no need for separate prev_cpu and locked members of struct optimistic_spin_node. Using a single field simplifies the code slightly. It also removes any possibility of the two values being out of sync. When cancelling a lock request explicitly set prev_cpu to zero. Nothing actually looks at the field, but it means that it will be zero after a subsequent 'fast path' osq_lock() call making things consistent. The cache line is likely to be dirty (or be dirtied) so there shouldn't be a performance hit. Signed-off-by: David Laight Tested-by: H=C3=A5kon Bugge --- kernel/locking/osq_lock.c | 57 +++++++++++++++++++-------------------- 1 file changed, 28 insertions(+), 29 deletions(-) diff --git a/kernel/locking/osq_lock.c b/kernel/locking/osq_lock.c index 01988d00c480..23f00c670507 100644 --- a/kernel/locking/osq_lock.c +++ b/kernel/locking/osq_lock.c @@ -35,7 +35,6 @@ =20 struct optimistic_spin_node { struct optimistic_spin_node *next; - int locked; /* 1 if lock acquired */ int prev; /* CPU number offset by 1 */ }; =20 @@ -113,7 +112,6 @@ bool osq_lock(struct optimistic_spin_queue *lock) int curr =3D encode_cpu(smp_processor_id()); int prev; =20 - node->locked =3D 0; node->next =3D NULL; =20 /* @@ -158,46 +156,47 @@ bool osq_lock(struct optimistic_spin_queue *lock) * is implemented with a monitor-wait. vcpu_is_preempted() relies on * polling, be careful. */ - if (smp_cond_load_relaxed(&node->locked, VAL || need_resched() || - vcpu_is_preempted(node->prev - 1))) - return true; + prev =3D smp_cond_load_relaxed(&node->prev, !VAL || need_resched() || + vcpu_is_preempted(VAL - 1)); =20 - /* unqueue */ /* - * Step - A -- stabilize @prev + * Step - A * - * Undo our @prev->next assignment; this will make @prev's - * unlock()/unqueue() wait for a next pointer since @lock points to us - * (or later). + * Loop until either node->prev is zero (lock acquired) or we + * atomically change prev->next from node to NULL (stopping prev + * handing on the lock). + * Note that 'prev' can unlink itself concurrently with this + * test so that prev/prev_ptr can be stale, but since it + * is per-cpu data the memory can always be read. */ =20 - for (;;) { - /* - * cpu_relax() below implies a compiler barrier which would - * prevent this comparison being optimized away. - */ + for (;; prev =3D READ_ONCE(node->prev)) { + if (!prev) + /* Lock acquired */ + return true; + + prev_ptr =3D decode_cpu(prev); + if (data_race(prev_ptr->next) =3D=3D node && cmpxchg(&prev_ptr->next, node, NULL) =3D=3D node) break; =20 /* - * We can only fail the cmpxchg() racing against an unlock(), - * in which case we should observe @node->locked becoming - * true. + * 'prev' must have unlinked (or be in the process of unlinking) + * itself from the list. */ - if (smp_load_acquire(&node->locked)) - return true; =20 cpu_relax(); - - /* - * Or we race against a concurrent unqueue()'s step-B, in which - * case its step-C will write us a new @node->prev pointer. - */ - prev =3D READ_ONCE(node->prev); - prev_ptr =3D decode_cpu(prev); } =20 + /* + * If 'prev' tries to remove itself from the list before we write + * a new value to prev->next it will spin in osq_wait_next(). + */ + + /* Invalidate prev_cpu matching osq_unlock() */ + node->prev =3D 0; + /* * Step - B -- stabilize @next * @@ -240,11 +239,11 @@ void osq_unlock(struct optimistic_spin_queue *lock) node =3D this_cpu_ptr(&osq_node); next =3D xchg(&node->next, NULL); if (next) { - WRITE_ONCE(next->locked, 1); + WRITE_ONCE(next->prev, 0); return; } =20 next =3D osq_wait_next(lock, node, OSQ_UNLOCKED_VAL); if (next) - WRITE_ONCE(next->locked, 1); + WRITE_ONCE(next->prev, 0); } --=20 2.39.5 From nobody Fri Sep 25 23:53:11 2026 Received: from mail-wr1-f54.google.com (mail-wr1-f54.google.com [209.85.221.54]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6F6F4448393 for ; Mon, 7 Sep 2026 08:41:56 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.221.54 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788770518; cv=none; b=kga6qV9thUMvZnwhFbUlf86HOysTCPjpA2amut/iaQAVdPL7bqmp6Mkq5dyb4qsrNGZbfKlmvLmxHijGLz7XP+Q7Ew6qfHEm4tsXxrcPw4iCK7Opf20Rb/4uul3hg57FSNOV5L0Nei2tQ0/ayZ43fwra533sYam2gr9IDlW1KdI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788770518; c=relaxed/simple; bh=1Q8spIejnM4aGTTfmyL4NK5XSVAAY4GV1tixC+PV06A=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=tyq8K9VIJJZdQFCUrchbNFrw8Abb2eolbIQdArJsDuuaLIkjdfswcZUd/HTR9ApFsAOlaaOO3WpASNCAF//KONLkPbwl4lXdnSPx6IfKM8pkiBCik9SdBdxEBoJnj/BJGgsUa1m/ucGBnuCWrUqtsifQthHVBN51yyv/AemTkH8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=Cx09ho72; arc=none smtp.client-ip=209.85.221.54 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="Cx09ho72" Received: by mail-wr1-f54.google.com with SMTP id ffacd0b85a97d-485850cf499so2023551f8f.3 for ; Mon, 07 Sep 2026 01:41:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1788770514; x=1789375314; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=6oF/RHZsuGRU+WdOikxDPvBXvsmcgSMqlQfNLaHuEiM=; b=Cx09ho72iiMvtYIRR/4OekbHd0aUxrBzw5ueuJbuzFazBLC8CCOozHdgbynR9Cm1bB rOlCidBoYMLw9fFSFNdkQphFoLdqjJNrINwlAohwohPWZpa4Kv6rGgaeD6TjXUFWPGFe XfdIdG6n5QyMXqH9GFT+7gqAqr2Kbv7WOX0osE9LZzBP/ATFwZBk4/qUE0pNXcEmOnZ0 07SD+U3ouJ8UrqYLxr96qHMlPuHF8AF8nSccE+UnY7LklB6x6s2LxRYIdXSpuwh1q7/5 FpCrp1HdfS7ZbjUYIXa+lgIu+yli9qVJHfFeo6cbp2depvBMVOk7tNBla8gF9qMhHYn9 ytcw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1788770514; x=1789375314; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=6oF/RHZsuGRU+WdOikxDPvBXvsmcgSMqlQfNLaHuEiM=; b=QehsPgeoF9IFlYOxrE14ss3FLLf6PcJpldT+Pj9IcbBvNpc6MkxWVsuKZYipNzuRof EeqsnNYErV2Wekne9cXV4YEqdPcTpVX60vLzSIvhYNSsahZSZfiAR1IMxNOkNKNkMcGh XEaQZrAjEuZEadjQO/ippV3YeWlYMaPKjlWcpPHPWdZkXVehJnnrbfA3yRfZIOCq/6Q4 WFre+zAfNYgQ6lPDCD6MSJkX4fK1F1hJegMiemYEuOxChdEQxhA3ybFL9VcQXLHHm+Fc 3JjYkajD0CV5ox5aEPeOFUYPBKPeUE9+fqYsDvMQe6IZ/7PaRZKglu6+/KYe32e5pLfG 6mcQ== X-Forwarded-Encrypted: i=1; AKwUvBwdNLKfuiDN3qcJy7xxFCXirGhp3htc752uvBp524Bn5AKUZtNxjsg2GVEV7wOZqJM/jMD8OTf0vzB9lvM=@vger.kernel.org X-Gm-Message-State: AFuF++mvkmvX66gW4Vq2iVMn1mgpKVTz9X51pEJS1oueJLlKHm1JUmnK Nl7gX7rX7o/FhZ91eU5EuCG8cOwqOM7O82rduFCVA1S3Qamy9Gycfz3aSiU+GEmC X-Gm-Gg: AYBFou1cp32cZLw1Vb3Eiw8ZOCi20JlzmJxcjkdEAGbpVnf7BShv8jNRkQ3yuYPq81B fMGhQE+Aj8Rx2GjoxUhRZSpP41eHrJcUFXL9Tltt08Ey1SN5ycEtbOkMmWcKCYtjDhk70/S5gZX Rt6WtEhF+Yd+0Kd/It2jNcJft1xs35haqdQzZyG5NCSwdbWLYN76hBFjCBUKVi2nSwZS7M5v2kQ yE0+NrwassmCWwulQQpuwTbKr0pzXqbtsBuvfh/im8FJhKB9hRVPuo8Oet1/k24wmB+Jgvq9F64 9Dtb+SNzTkNFZv5u62NYoB+gnCzJGEUkmqC1Eo79VT2kgeGS5LD97NSpR5P6uZ0brqFB23AOzpC ndV4YU9+axkF8JEKjKy63lVpdBmMTHs51mnkASPejb0QMmI66EJy84HQqedtlF8z77m4uqJV/PC 70549Mjmu4769pMpE7QZMEplRJcpIZJjNXKZG/ypn7AuMCMj7JYBJnBbpVv2ZEZq1NYw2CS8gIf Eom7e+Glc2nxwC/ZaS9AXxDPd64AAeGetZ/Pcnz0LoGpoxpmtd0IBuJ X-Received: by 2002:a05:6000:2311:b0:484:42d5:e6d5 with SMTP id ffacd0b85a97d-48587094b12mr22290563f8f.11.1788770514133; Mon, 07 Sep 2026 01:41:54 -0700 (PDT) Received: from snowdrop.snailnet.com (82-69-66-36.dsl.in-addr.zen.co.uk. [82.69.66.36]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-485885bbb51sm27883762f8f.30.2026.09.07.01.41.53 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 07 Sep 2026 01:41:53 -0700 (PDT) From: David Laight To: Waiman Long , Peter Zijlstra , Ingo Molnar , Will Deacon , Boqun Feng , linux-kernel@vger.kernel.org, Linus Torvalds , Yafang Shao , Steven Rostedt Cc: David Laight Subject: [PATCH v4 next 4/9] locking/osq_lock: Delete 'fast path' code from osq_unlock() Date: Mon, 7 Sep 2026 09:41:28 +0100 Message-Id: <20260907084133.3696-5-david.laight.linux@gmail.com> X-Mailer: git-send-email 2.39.5 In-Reply-To: <20260907084133.3696-1-david.laight.linux@gmail.com> References: <20260907084133.3696-1-david.laight.linux@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" The 'fast path' code in osq_unlock() is pretty much exactly the same as the first pass of the loop in osq_wait_next() except that it doesn't have the optimisation to avoid the locked RMW when not the tail of the list. So just call osq_wait_next(). Move the assignment next->prev_cpu =3D old_cpu into osq_wait_next() as it is always the next line. Rename osq_wait_next() to osq_unlink_from_next() since that is what is does. Change osq_wait_next() to use atomic_cmpxchg_release() (not _acquire) on lock->tail. This is what osq_unlock() did and seems right to me. Add an smp_wmb() before the 'prev->next =3D next' assignment when cancelling a lock. The previous 'next->prev =3D prev' assignment lets the 'prev' cpu complete an unlocking sequence and do its 'next->prev =3D prev' assignment first - corrupting the list. Signed-off-by: David Laight Tested-by: H=C3=A5kon Bugge --- kernel/locking/osq_lock.c | 140 ++++++++++++++++++++------------------ 1 file changed, 73 insertions(+), 67 deletions(-) diff --git a/kernel/locking/osq_lock.c b/kernel/locking/osq_lock.c index 23f00c670507..f39f77c3a07d 100644 --- a/kernel/locking/osq_lock.c +++ b/kernel/locking/osq_lock.c @@ -57,52 +57,67 @@ static inline struct optimistic_spin_node *decode_cpu(i= nt encoded_cpu_val) } =20 /* - * Get a stable @node->next pointer, either for unlock() or unqueue() purp= oses. - * Can return NULL in case we were the last queued and we updated @lock in= stead. + * Unlink the current cpu's node from the lock's node->prev list. * - * If osq_lock() is being cancelled there must be a previous node - * and 'old_cpu' is its CPU #. - * For osq_unlock() there is never a previous node and old_cpu is - * set to OSQ_UNLOCKED_VAL. + * More specifically atomically write its node->prev over the link that + * currently points to node. + * This is either: + * lock->tail =3D node->prev + * or: + * node->next->prev =3D node->prev + * The first is a simple cmpxchg(), the second is protected against + * node->next trying to unlink itself (after need_resched() is set) by usi= ng + * an xchg() on node->next that sets it to NULL. + * + * When a lock request is being cancelled the caller needs 'next' to + * set node->prev->next =3D next. */ static inline struct optimistic_spin_node * -osq_wait_next(struct optimistic_spin_queue *lock, - struct optimistic_spin_node *node, - int old_cpu) +osq_unlink_from_next(struct optimistic_spin_queue *lock, int prev) { int curr =3D encode_cpu(smp_processor_id()); + struct optimistic_spin_node *node, *next; =20 for (;;) { - if (atomic_read(&lock->tail) =3D=3D curr && - atomic_cmpxchg_acquire(&lock->tail, curr, old_cpu) =3D=3D curr) { + int tail =3D atomic_read(&lock->tail); + if (curr =3D=3D tail && + atomic_try_cmpxchg_release(&lock->tail, &tail, prev)) { /* - * We were the last queued, we moved @lock back. @prev - * will now observe @lock and will complete its - * unlock()/unqueue(). + * We were the last queued, lock->tail now references + * prev (or is 0 if the list is now empty). + * If prev was spinning in this loop it can continue. */ return NULL; } =20 + node =3D this_cpu_ptr(&osq_node); + /* - * We must xchg() the @node->next value, because if we were to - * leave it in, a concurrent unlock()/unqueue() from - * @node->next might complete Step-A and think its @prev is - * still valid. + * We must xchg() the @node->next value to ensure that a + * concurrent unqueue() from @node->next will find an invalid + * @prev value (node_next->prev->next !=3D node_next). * - * If the concurrent unlock()/unqueue() wins the race, we'll - * wait for either @lock to point to us, through its Step-B, or - * wait for a new @node->next from its Step-C. + * If @node->next is already NULL then we need to wait until + * the concurrent unqueue completes. */ if (node->next) { - struct optimistic_spin_node *next; - next =3D xchg(&node->next, NULL); if (next) - return next; + break; } =20 cpu_relax(); } + + /* + * When called from osq_unlock() prev is zero and this hands + * over the lock ownership. + * When called while unqueueing in osq_lock() this completes the + * backwards link, the forwards link is done by the caller. + */ + WRITE_ONCE(next->prev, prev); + + return next; } =20 bool osq_lock(struct optimistic_spin_queue *lock) @@ -130,7 +145,7 @@ bool osq_lock(struct optimistic_spin_queue *lock) /* * osq_lock() unqueue * - * node->prev =3D prev osq_wait_next() + * node->prev =3D prev osq_unlink_from_next() * WMB MB * prev->next =3D node next->prev =3D prev // unqueue-C * @@ -160,8 +175,6 @@ bool osq_lock(struct optimistic_spin_queue *lock) vcpu_is_preempted(VAL - 1)); =20 /* - * Step - A - * * Loop until either node->prev is zero (lock acquired) or we * atomically change prev->next from node to NULL (stopping prev * handing on the lock). @@ -191,59 +204,52 @@ bool osq_lock(struct optimistic_spin_queue *lock) =20 /* * If 'prev' tries to remove itself from the list before we write - * a new value to prev->next it will spin in osq_wait_next(). + * a new value to prev->next it will spin in osq_unlink_from_next(). + * This means we can no longer be given the lock and always + * return false. */ =20 - /* Invalidate prev_cpu matching osq_unlock() */ + /* + * Invalidate prev matching osq_unlock(). + * This isn't necessary but ensures that both unlocked and fast-path + * locked nodes (where the initial xchg() returned 0) have prev set + * to zero. + * If nothing else it lets the lock chain be followed from lock->tail + * whch may help diagnostics. + */ node->prev =3D 0; =20 /* - * Step - B -- stabilize @next - * - * Similar to unlock(), wait for @node->next or move @lock from @node - * back to @prev. + * Now that the linkage to prev cannot change underneath us + * remove ourselves from the node->prev list. + * This does: + * (node->next ? node->next->prev : lock->tail) =3D node->prev */ - - next =3D osq_wait_next(lock, node, prev); - if (!next) - return false; + next =3D osq_unlink_from_next(lock, prev); =20 /* - * Step - C -- unlink - * - * @prev is stable because its still waiting for a new @prev->next - * pointer, @next is stable because our @node->next pointer is NULL and - * it will wait in Step-A. + * Finally mend the node->next list that was 'broken' to + * stop node->prev trying to unlink from us. + * If next is NULL then lock->tail is prev_ptr and another node + * can be added - so we must not re-write the NULL. */ - - WRITE_ONCE(next->prev, prev); - WRITE_ONCE(prev_ptr->next, next); + if (next) { + /* + * This must happen after the write to node->next->prev. + * If swapped then prev could unlink itself before our + * write to node->next->prev and the the wrong value would + * end up in node->next->prev. + * Probably can't actually happen due to re-ordering of writes, + * but could happen without a compiler barrier. + */ + smp_wmb(); + WRITE_ONCE(prev_ptr->next, next); + } =20 return false; } =20 void osq_unlock(struct optimistic_spin_queue *lock) { - struct optimistic_spin_node *node, *next; - int curr =3D encode_cpu(smp_processor_id()); - - /* - * Fast path for the uncontended case. - */ - if (atomic_try_cmpxchg_release(&lock->tail, &curr, OSQ_UNLOCKED_VAL)) - return; - - /* - * Second most likely case. - */ - node =3D this_cpu_ptr(&osq_node); - next =3D xchg(&node->next, NULL); - if (next) { - WRITE_ONCE(next->prev, 0); - return; - } - - next =3D osq_wait_next(lock, node, OSQ_UNLOCKED_VAL); - if (next) - WRITE_ONCE(next->prev, 0); + osq_unlink_from_next(lock, OSQ_UNLOCKED_VAL); } --=20 2.39.5 From nobody Fri Sep 25 23:53:11 2026 Received: from mail-wm1-f44.google.com (mail-wm1-f44.google.com [209.85.128.44]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 35955448B8A for ; Mon, 7 Sep 2026 08:41:57 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.44 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788770518; cv=none; b=Ortu6LydJ0TWbvO3XxGeF4WZbWzTthWahAmwUPH2BrO9no+rXZU8C/G0/vJvf9+L/nflHuJn1Y+TfJJpYdaNh309FTIljxz2/kqQ3yu8EiBIxvxhVLFmgBQh2m3Uxiha1tOwXHkC2v6QEpRgtYwqxBV3ieldG/MEzXeFSitos8s= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788770518; c=relaxed/simple; bh=b66/JTp0PGZftKfxgh0vOj/JZB8PQCgl2YAq0patHng=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=ec06po3xcUZm82MyNzzXP1T0xM4nptBsPRdz9y1ueVvIlVAMZk+oEIBKkU7HRLKY6MBHWGN42NejQo5t7Fpr3Ppoy3Ru1z4P1I0GcF738+IKTQe7f/qSeeekHqYNONc8QEqB/B3eFDsygnziQ40Ck8/lTpzmOLUzU7X529KV09M= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=V0ZefUjc; arc=none smtp.client-ip=209.85.128.44 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="V0ZefUjc" Received: by mail-wm1-f44.google.com with SMTP id 5b1f17b1804b1-495590dde14so46795485e9.0 for ; Mon, 07 Sep 2026 01:41:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1788770515; x=1789375315; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=KUNx51fF4zO00Zyg4SYhi1EUTdrEwCLha5IxI5s1LAM=; b=V0ZefUjclb0S04ydB87Qrv5APX/+dWSkvgBsoS5GGUoYvT0SEegZppbkamNGm/M+1O VnwaGBtSWh1iNLBEY7cyjtkfw7r1yh8ZIB+viPzmDHTNiSaVLrMyh04ybi3fNFrccSIA c696T7g2huvCgohTt07YB1+EpRw9gKBq+M5euFiSIqlnn3fSpSS5Tjxl2dadwbKGcGVr b+BdsZxvd4dxhB7aiZ6zE2FzwVwGumfo5trFHk6Mq5LeDA1/BlXqGWgLmz+14FZAsYs3 raEJwiwYB+eJukYj13ntZ8FlJyeaJ7elz28QhFuyxLuTncha+j3LFw+fT2pPFekt1eV2 DAbw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1788770515; x=1789375315; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=KUNx51fF4zO00Zyg4SYhi1EUTdrEwCLha5IxI5s1LAM=; b=VZrXIQYVd5xB5BxIQl64rHQ0QqASOkpoPa0WTM0JfkvbjQjmC68Erj1NV6JaLUsUw0 J+mbwzGX5kPfb+IeYVn5lmLvRDiXeZQM41ZXLJEHghB3kh533ZNIVdVX1++zMBhzx4sg 4ywT8P7CmipFopCWAC26mWJy30Oj3KwHGZJtL4pu2Pmdp3u/9tya/9MElAw33+UJIDMF It+wBBZlh08rw2F7C69eqbXR30QeQJyr86a829ib8lxbJhJAs+XtwlzTg08wvJJ1s2pg 7Nq3lVDl/zjmLtmylK+9X7hWKdVZqgfaFvsdvFychzlZWK7KX66LJWkQihRhQq1Z347N ZCpw== X-Forwarded-Encrypted: i=1; AKwUvBxthg+oRnK5W4TTgsI20KpKpYgf1n7ANol+IOhTWHNT3rViihTNjYlMfgoJnKwQvJYI0ii/9ennFrik6y8=@vger.kernel.org X-Gm-Message-State: AFuF++n/czZ9rkWyuf+6+FD9SH8GosoNmfVrHFneTcp2jDfp6JqDVJdm rFQDtCccRM1oZjaSeOULnaA+ltpNXKH+vCRIKyQXPaCf/ZxIGphF0St5 X-Gm-Gg: AYBFou3O51SnSyUDAq8rLW6mD3+pqP3b4XpS9zWOlHtoAgeqb6yIIbZFGcEsxHykB11 WdoMBtfe/lHTg+zX+ywAkioY1d+JjC12LAnwoJnjSkTSko7JuAzBN4eprkNUL+HGPAoG3Kumd+G noSe0viUStwqgyxuSilYdzIlbpcx46mqHquHgA/MMl+AfVxNsHaWJnO8WC5th1ey5PHjtX/30R3 Fu/7eaafdGvRyA0ewW4le7odUoi6rOJHFwmAu8f2aWJEkgfVVAfb9uB70Ke7PPAYPyqUsRulNg2 0wm99gK1F+MQJWdgZAWLuUihdult9b1TsNXkJMkxDeXUmESzGtlt5oDegzohffG42vzzlB/+fEO JvLxurHRTYhcw0MgoefyV4pQnzv2yN4owGR9NgH8Hiples9F6cW/9ZyX9jAb32iOZmrAqWXNEpz OoOPOnxNDbMlbk5gcqcbqfWnXQT3UakwV/GxQFolyNVDMPBZ7tsfsbpXk24A5MhNo5Sphzm3y8D gT3nJL5GIH2aXNnWyploE7kw14NkYHIKyz9S2cWnwgpXQ== X-Received: by 2002:a05:600c:860b:b0:49c:ee20:e787 with SMTP id 5b1f17b1804b1-49cf7fe62e9mr440579205e9.1.1788770515073; Mon, 07 Sep 2026 01:41:55 -0700 (PDT) Received: from snowdrop.snailnet.com (82-69-66-36.dsl.in-addr.zen.co.uk. [82.69.66.36]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-485885bbb51sm27883762f8f.30.2026.09.07.01.41.54 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 07 Sep 2026 01:41:54 -0700 (PDT) From: David Laight To: Waiman Long , Peter Zijlstra , Ingo Molnar , Will Deacon , Boqun Feng , linux-kernel@vger.kernel.org, Linus Torvalds , Yafang Shao , Steven Rostedt Cc: David Laight Subject: [PATCH v4 next 5/9] locking/osq_lock: Avoid writing to node->next in the osq_lock() fast path Date: Mon, 7 Sep 2026 09:41:29 +0100 Message-Id: <20260907084133.3696-6-david.laight.linux@gmail.com> X-Mailer: git-send-email 2.39.5 In-Reply-To: <20260907084133.3696-1-david.laight.linux@gmail.com> References: <20260907084133.3696-1-david.laight.linux@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" osq_unlink_from_next() is called by osq_unlock() and when osq_lock() returns false (lock not acquired). osq_unlink_from_next() will either have done an explicit xchg(&node->next, = NULL) or a cmpxchg() that checked that node was lock->tail. In both cases node->next will be NULL in exit. Since it can't be changed when not referenced by an osq_lock there is no need to initialise it at the top of osq_lock(). The atomic_xchg(&lock->tail, curr) could probably changed back to the '_acquire' version or even the _relaxed version. The important barrier is after the write to node->prev. Defer determining the address of the CPU's 'node' until after the atomic_exchange() so that it isn't done in the uncontented path. Signed-off-by: David Laight Tested-by: H=C3=A5kon Bugge --- kernel/locking/osq_lock.c | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/kernel/locking/osq_lock.c b/kernel/locking/osq_lock.c index f39f77c3a07d..2f92d3d63da9 100644 --- a/kernel/locking/osq_lock.c +++ b/kernel/locking/osq_lock.c @@ -86,6 +86,9 @@ osq_unlink_from_next(struct optimistic_spin_queue *lock, = int prev) * We were the last queued, lock->tail now references * prev (or is 0 if the list is now empty). * If prev was spinning in this loop it can continue. + * + * Since we are the tail of the list, node->next + * must be NULL. */ return NULL; } @@ -122,13 +125,10 @@ osq_unlink_from_next(struct optimistic_spin_queue *lo= ck, int prev) =20 bool osq_lock(struct optimistic_spin_queue *lock) { - struct optimistic_spin_node *node =3D this_cpu_ptr(&osq_node); - struct optimistic_spin_node *prev_ptr, *next; + struct optimistic_spin_node *node, *prev_ptr, *next; int curr =3D encode_cpu(smp_processor_id()); int prev; =20 - node->next =3D NULL; - /* * We need both ACQUIRE (pairs with corresponding RELEASE in * unlock() uncontended, or fastpath) and RELEASE (to publish @@ -139,6 +139,7 @@ bool osq_lock(struct optimistic_spin_queue *lock) if (prev =3D=3D OSQ_UNLOCKED_VAL) return true; =20 + node =3D this_cpu_ptr(&osq_node); prev_ptr =3D decode_cpu(prev); node->prev =3D prev; =20 --=20 2.39.5 From nobody Fri Sep 25 23:53:11 2026 Received: from mail-wr1-f50.google.com (mail-wr1-f50.google.com [209.85.221.50]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 97826448BA0 for ; Mon, 7 Sep 2026 08:41:57 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.221.50 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788770519; cv=none; b=ere+i6W7mjKonkNiyrx4Hgyl+K+0fc1ZdAx2j6Y3c4Szw0dZ3I4rZJ779l56x/whYyDP4kBx6QH76fuZ1m1j/jS7lAItfJUw81FLlfuZL3oWWqetjOywoRcdMsImKHa1UmZaxvzFdaq0jROxa0Y77dGME1l8FqG8ET31j9CN0Po= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788770519; c=relaxed/simple; bh=AeLsjWU05tP6qUzpR227GlkkcY3aVa0dNsWWq6l86OY=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=d57NEjD7L+5wc/YNtUe3vMvMqe6RNXPreNr0lHdJ2fiCyTHJyR0TdiN9hkVylFrzCAy0W3BJyNfeYId072IuwP0U7mxff2pGVixCbZIBA9Z6CYTahW0p/ZQLi/M+5369YxvU+8XG03akdvpHaPbn2vYwgYZNWyjHHi8bnR18U5k= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=ebLnBjNI; arc=none smtp.client-ip=209.85.221.50 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="ebLnBjNI" Received: by mail-wr1-f50.google.com with SMTP id ffacd0b85a97d-482e4998d28so2547405f8f.2 for ; Mon, 07 Sep 2026 01:41:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1788770516; x=1789375316; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=izxSa6U3ciFYO1lIJ9QQ+OE/FD1/ZCd/7scZJPcwER4=; b=ebLnBjNIvUfpa4JDn0/FfnuyBSOwgNRQp0NxmfO8oOSdvOwkNKKC1mjEEYC49vaYKM QaM7gRfxE6Jv6mP/nj9d0+ZNy4GBq+AQMnvIO7AhVpwM+e8sdrpush3ExjsdsGN4uU34 eDWIrfeuIFQTFTmNV0OjtWAq2/kHP0cJP2mloR12Mp2IJhhwcn/2bT3aMDTj0vcbg2or ISk05C/znfZTbAs4RVPgPOxiJWAPfFib1wDEzbfUJk3poE0kSQEl2gsX0cq3TAXWU8HP TNSQu79xhoSlb4nr8EY8m2FGtptn+gryq7Q/T5CbeFCUqkLLbVw+npFGOfD0f5QdbT2l KYeA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1788770516; x=1789375316; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=izxSa6U3ciFYO1lIJ9QQ+OE/FD1/ZCd/7scZJPcwER4=; b=UXJ8Onkz6AHJl9yW9qnHOzoQliZBs3hfqksHj+BIj9yS3jT3sdzdhEO34EKLZ/5eOA 2LF+UQeGeD9x3qDqDid8YOhC+0r1K7caJjqIZtC73VL4FQJfF2HHVWe9JwPYuTQYLoqD wmd+PEiDvwdolBpNGOLeUX0WWaGwFW34j6OcPxbF8oGq225liycKrzVY6OAmvja8JSXV aiSlhgwbUelmqiEYnRu3KYfQrs6E2LMjBQMM2U1Rn8NjGLUkLbMxqIdnMK5UcPcOALCQ MJqjjWzUkz4l1gR3FgeQC9QjTujEsV8vmcLPFQYqfVIywa5tB3r4qZfiI6iGqAtmU/Bk zrQQ== X-Forwarded-Encrypted: i=1; AKwUvBxx8J8q33jM7MP01X/fCfchApHJ12Ickycbk44N0zuLc5QeBJcLTHfQZLtCHtn3uxfP3Bgv7RZO5s+qfZE=@vger.kernel.org X-Gm-Message-State: AFuF++mC8tbfmyaY7EL8jemD0UqxznPsx/Vopat780EWcymqMNS/J/nb DNjKTJCRqHSV5JLzxpYMYr8b2XSLPIlxZW2Z8TDvqqCtRLYmnx/rUB4b X-Gm-Gg: AYBFou0/pWCYnf1bvXDBsICh4qXjCb0AvSvia7My1I4xvlV6KisuXLzRUK83Lriy1rw cD2DG0DLo0ArYhONke4ISwjNDSzM5b6lRsL7RXBdctR2hEwNrdET626vvZOwmDZNZ2LBLLbzqTb rU9filRsYQm7Ezd3/11Lxv6Ko4s53hptMJQCl9iMdJeX0PQ++uEyihiBwQlWoraM/Oo8Mhdw0kX tWsa/iCJdj4VTvQZszIBJrJxHWbr6NRIwyeIQOUpc4X1gccAK6hAHpO++lDqSEmHtjamXYALMLb 8c956Y5/wK9F4RGdSqjuaTD+uWtj95mWutgQ8XXkt9kVQ7SKqimAjzge9H7JfogzP+b+q8HwiUC ZKBq/BgOaMl6vZg5wSoNDEG77/pBhugSZFvbmhCW/3m/0vMcudGVjCsLFo+8hld0uVdy1OiVMuK J2NxDcNPFK/gD6sIsAJm+HK4Gw0TbFBIa0TBGIAtTJaVnHYE7X7nvOF7JhhtSfC/bsm05k3OHih thuwGZghH0A6pVrWUcOAGlFfDM+g5IxPqVtVXGsaFzhN1OivwwHOEaf X-Received: by 2002:a05:600c:8b88:b0:49c:fc6e:a3d8 with SMTP id 5b1f17b1804b1-49cffde8d95mr160935545e9.23.1788770515656; Mon, 07 Sep 2026 01:41:55 -0700 (PDT) Received: from snowdrop.snailnet.com (82-69-66-36.dsl.in-addr.zen.co.uk. [82.69.66.36]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-485885bbb51sm27883762f8f.30.2026.09.07.01.41.55 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 07 Sep 2026 01:41:55 -0700 (PDT) From: David Laight To: Waiman Long , Peter Zijlstra , Ingo Molnar , Will Deacon , Boqun Feng , linux-kernel@vger.kernel.org, Linus Torvalds , Yafang Shao , Steven Rostedt Cc: David Laight Subject: [PATCH v4 next 6/9] locking/osq: Use cpu number for 'next' pointer Date: Mon, 7 Sep 2026 09:41:30 +0100 Message-Id: <20260907084133.3696-7-david.laight.linux@gmail.com> X-Mailer: git-send-email 2.39.5 In-Reply-To: <20260907084133.3696-1-david.laight.linux@gmail.com> References: <20260907084133.3696-1-david.laight.linux@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" There is only one write done through node->next (setting prev) when 'node' is being removed. So the code can consistently use the cpu numbers pretty much throughout (this was suggested by Linux a while back). This reduces struct optimistic_spin_node to 8 bytes. This is currently padded out to a cache line which is silly. Change to be __aligned(8) so that it isn't split between cache lines. Accesses to 'other cpu' data are very limited and only happen during the enque and deque operation. Signed-off-by: David Laight Tested-by: H=C3=A5kon Bugge --- kernel/locking/osq_lock.c | 43 ++++++++++++++++++++------------------- 1 file changed, 22 insertions(+), 21 deletions(-) diff --git a/kernel/locking/osq_lock.c b/kernel/locking/osq_lock.c index 2f92d3d63da9..0f68ee017b54 100644 --- a/kernel/locking/osq_lock.c +++ b/kernel/locking/osq_lock.c @@ -24,21 +24,21 @@ * There is no equivalent pointer to the list head - the 'head' is the * osq_node of the CPU that acquired the osq lock. * - * The 'next' pointer of the tail must be NULL, all the other 'next' point= ers - * must either be valid or transiently NULL. - * The 'prev' pointers only need to be valid when node->prev makes sense a= nd, - * even then, can be transiently invalid (ie refer to the wrong node). - * They are only used for the node->prev->next =3D node->next update when - * 'node' is being removed. Atomically checking node->prev->next =3D=3D no= de + * The 'next' pointer of the tail must be zero, all the other 'next' point= ers + * must either be valid or transiently zero. + * The 'prev' pointer is zero unless the node is waiting for the lock, when + * waiting it may refer to the wrong node (node->prev->next !=3D node). + * The 'prev' value is only needed for the node->prev->next =3D node->next= update + * when 'node' is being removed. Atomically checking node->prev->next =3D= =3D node * ensures the list doesn't get corrupted. */ =20 struct optimistic_spin_node { - struct optimistic_spin_node *next; - int prev; /* CPU number offset by 1 */ -}; + int next; /* CPU number offset by 1, 0 if no next */ + int prev; /* CPU number offset by 1, 0 if lock held */ +} __aligned(8); =20 -static DEFINE_PER_CPU_SHARED_ALIGNED(struct optimistic_spin_node, osq_node= ); +static DEFINE_PER_CPU(struct optimistic_spin_node, osq_node); =20 /* * We use the value 0 to represent "no CPU", thus the encoded value @@ -72,11 +72,12 @@ static inline struct optimistic_spin_node *decode_cpu(i= nt encoded_cpu_val) * When a lock request is being cancelled the caller needs 'next' to * set node->prev->next =3D next. */ -static inline struct optimistic_spin_node * +static inline int osq_unlink_from_next(struct optimistic_spin_queue *lock, int prev) { int curr =3D encode_cpu(smp_processor_id()); - struct optimistic_spin_node *node, *next; + struct optimistic_spin_node *node; + int next; =20 for (;;) { int tail =3D atomic_read(&lock->tail); @@ -88,9 +89,9 @@ osq_unlink_from_next(struct optimistic_spin_queue *lock, = int prev) * If prev was spinning in this loop it can continue. * * Since we are the tail of the list, node->next - * must be NULL. + * must be zero. */ - return NULL; + return 0; } =20 node =3D this_cpu_ptr(&osq_node); @@ -104,7 +105,7 @@ osq_unlink_from_next(struct optimistic_spin_queue *lock= , int prev) * the concurrent unqueue completes. */ if (node->next) { - next =3D xchg(&node->next, NULL); + next =3D xchg(&node->next, 0); if (next) break; } @@ -118,16 +119,16 @@ osq_unlink_from_next(struct optimistic_spin_queue *lo= ck, int prev) * When called while unqueueing in osq_lock() this completes the * backwards link, the forwards link is done by the caller. */ - WRITE_ONCE(next->prev, prev); + WRITE_ONCE(decode_cpu(next)->prev, prev); =20 return next; } =20 bool osq_lock(struct optimistic_spin_queue *lock) { - struct optimistic_spin_node *node, *prev_ptr, *next; + struct optimistic_spin_node *node, *prev_ptr; int curr =3D encode_cpu(smp_processor_id()); - int prev; + int next, prev; =20 /* * We need both ACQUIRE (pairs with corresponding RELEASE in @@ -155,7 +156,7 @@ bool osq_lock(struct optimistic_spin_queue *lock) */ smp_wmb(); =20 - WRITE_ONCE(prev_ptr->next, node); + WRITE_ONCE(prev_ptr->next, curr); =20 /* * Normally @prev is untouchable after the above store; because at that @@ -191,8 +192,8 @@ bool osq_lock(struct optimistic_spin_queue *lock) =20 prev_ptr =3D decode_cpu(prev); =20 - if (data_race(prev_ptr->next) =3D=3D node && - cmpxchg(&prev_ptr->next, node, NULL) =3D=3D node) + if (data_race(prev_ptr->next) =3D=3D curr && + cmpxchg(&prev_ptr->next, curr, 0) =3D=3D curr) break; =20 /* --=20 2.39.5 From nobody Fri Sep 25 23:53:11 2026 Received: from mail-wr1-f46.google.com (mail-wr1-f46.google.com [209.85.221.46]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 41E67448BA3 for ; Mon, 7 Sep 2026 08:41:58 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.221.46 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788770520; cv=none; b=WGJdE3ww69zmCYVUYEcCnfGEL2gd/Rlzi31x+KYDa7Yd9bP2YsR7qmPumuDowyG7GSqWH26niT0W3DVP0YCKgu1OJYyiVoNBhzCB86vDA2uYHb5yQmoZRka7lgJL5Ev0R+v/kPnt5s9Hag59PUkeZ3XGKAkMOij4ncM9uasjb2A= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788770520; c=relaxed/simple; bh=ZTmmndy7/AcjbfShhrBZFEsnXBPekn0HGXjSLaxXqJQ=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=Ph4savWCErgAE2wmrznXm0bAOxtqmgALZETbHAYyfRb7bl2SEkw6v6PHTjl7oCnaPK/tkWJzbM2yPnixooeMh6OiRCGYQs+BqGssB62cOtd9fVbMLUnOLXpa/A3cNOOItaum9++0q9dQVhbIspmure9EBMX31YFcZLsRmYiFlIY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=TQDJs4WU; arc=none smtp.client-ip=209.85.221.46 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="TQDJs4WU" Received: by mail-wr1-f46.google.com with SMTP id ffacd0b85a97d-48586861639so2403725f8f.0 for ; Mon, 07 Sep 2026 01:41:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1788770516; x=1789375316; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=tH/kRPPDD2gV1vMx/S94hLYP+9/dPU1ymPrzNYM8wtg=; b=TQDJs4WUSr5+TPt3zpY5rOWZpcV5bM935d68lM1XOHz5qNX3W3eXESHHiwUemH/KSB TcmTeNYMCSty8pzGKl2Txbt/Wb/sY5r8hCUzLcepg94OEW4MJtO3Ln229n1BZMEFT6y4 DGwZ6F4gcxIiJelOgb/Pbx5aHZGgzbHjkyqCcWC7ZUw/A4c+wVO+ae7ouesJrj+fDjTX dm5F3uRzcEvf9ENTvj9u4qcicI0L6W9eKjBK/Qiv5//ECUFrMgrQl488qGhcdPY97TJF 8va9td9AyjIBMQchuV2T7eyqMCu4K+kn+aISMzUznexW5b4P3dN7IMn/+JFpufachwuC RGCw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1788770516; x=1789375316; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=tH/kRPPDD2gV1vMx/S94hLYP+9/dPU1ymPrzNYM8wtg=; b=lKApyoKUYMzTwSCFMAAHsiXTjtWwEOTeEdlTdsSE8v+AmLOBalmm3obrkyrmhVFclg 3zRnK3zDwsgGzR8LVqoEz/0BElt9APcXRRqTCrHuYvNgAL0Gok+1DkoetTQAtqNizJ4o eiWM4+HI7YLf+1I5unBPKIC/bGZPhu7kMjVMXE8Cx4ywNt1yIUmOPiz4Q1CA1usPP0wo sK3h70HUaS8va6XbAeKa4hXWQXLp/GEe6/N58hxSLTg9Jdip05OvhJ8XJMar0W7q7wGf DNuDOhEUCrhj/cTjjzMSedbQOCUQ14apbhkCARQZ9VaD4y5EOdhlWRDCwfvwbIwpL091 xQPA== X-Forwarded-Encrypted: i=1; AKwUvBwyjvyl0wfl0Mx0YnBYYghHgas//rHJg25KjsiAf8Lq5Kqx9i7Oq5Bp/VVwIVkJ7SkUb1N4xE/d6VdiZ0E=@vger.kernel.org X-Gm-Message-State: AFuF++mJUXwPvh8bswJPtoSVNpXJ8f0yJi9quRNF4TDz3Ir+4Z63CK36 hd86GI/OJK+IaPvF1NqK81QbutoM7U3ITLnZAqEmRfzigDLNI4nZm8Bl X-Gm-Gg: AYBFou0q9Oj3AkFnREiwuXFOERzcKpMmWj9Uy5c/+NDR+LEQVfjBZIBuFM6Xic1cvDh DE3tqUKV0Tgff5awQ7hWN9MhRPOPrYr3tk7NNDi2NAg0QGWOuCbThp0AFJr16FDruWP15Mjy0jl grWcBHftWNpB6eY/brvWablZv5SRdJWmEDRmtMnGhZOCMT60XkkzcLgnn22FQlEEKSTl+5lh3Rf 7G44vH026IP0A3tGLinedCTniIAt0b+5b+a4hocLtSiMTgVn4ZdsDACC6qvFJ7PgsJpilLGVxtq d9IWT3iRkNRue9GDrYr2VOdfomRRRRgQJ7rBI4xCoIE6AuKaVOEOJLMG+ljlB/Cn7y8oY2KJQTY gYzKQ6wD/3QrgBlgLfvAUg+AkXQ5cCy8kZStXuviiAiVanhDLsTLHcJvIWaYNHjXiW9BY9F+qkx l8EPwEAWfZFQwdZksyvqyahW5qxC9LbGMQRFMshksfA+pipZ/If0B0SDKHXaK8DhJ8a2eEVAFVq AqZm4l4ujCWiec0GUeB7Kt58vhre0p7opDjrzgnWslG0w== X-Received: by 2002:adf:edc8:0:b0:482:e658:7a39 with SMTP id ffacd0b85a97d-4857e5171femr23993264f8f.11.1788770516418; Mon, 07 Sep 2026 01:41:56 -0700 (PDT) Received: from snowdrop.snailnet.com (82-69-66-36.dsl.in-addr.zen.co.uk. [82.69.66.36]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-485885bbb51sm27883762f8f.30.2026.09.07.01.41.55 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 07 Sep 2026 01:41:56 -0700 (PDT) From: David Laight To: Waiman Long , Peter Zijlstra , Ingo Molnar , Will Deacon , Boqun Feng , linux-kernel@vger.kernel.org, Linus Torvalds , Yafang Shao , Steven Rostedt Cc: David Laight Subject: [PATCH v4 next 7/9] locking/osq: Use 'unsigned int' for next/prev/tail Date: Mon, 7 Sep 2026 09:41:31 +0100 Message-Id: <20260907084133.3696-8-david.laight.linux@gmail.com> X-Mailer: git-send-email 2.39.5 In-Reply-To: <20260907084133.3696-1-david.laight.linux@gmail.com> References: <20260907084133.3696-1-david.laight.linux@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Consistently use 'unsigned int' for all the 'offset by 1' cpu numbers. This makes the code only use one set of xchg primitives. The unsigned type gives marginally better code inside per_cpu_ptr(). Signed-off-by: David Laight Tested-by: H=C3=A5kon Bugge --- include/linux/osq_lock.h | 8 ++++---- kernel/locking/osq_lock.c | 31 +++++++++++++++---------------- 2 files changed, 19 insertions(+), 20 deletions(-) diff --git a/include/linux/osq_lock.h b/include/linux/osq_lock.h index ea8fb31379e3..9e637e265189 100644 --- a/include/linux/osq_lock.h +++ b/include/linux/osq_lock.h @@ -12,17 +12,17 @@ struct optimistic_spin_queue { * Stores an encoded value of the CPU # of the tail node in the queue. * If the queue is empty, then it's set to OSQ_UNLOCKED_VAL. */ - atomic_t tail; + unsigned int tail; }; =20 #define OSQ_UNLOCKED_VAL (0) =20 /* Init macro and function. */ -#define OSQ_LOCK_UNLOCKED { ATOMIC_INIT(OSQ_UNLOCKED_VAL) } +#define OSQ_LOCK_UNLOCKED { OSQ_UNLOCKED_VAL } =20 static inline void osq_lock_init(struct optimistic_spin_queue *lock) { - atomic_set(&lock->tail, OSQ_UNLOCKED_VAL); + WRITE_ONCE(lock->tail, OSQ_UNLOCKED_VAL); } =20 extern bool osq_lock(struct optimistic_spin_queue *lock); @@ -30,7 +30,7 @@ extern void osq_unlock(struct optimistic_spin_queue *lock= ); =20 static inline bool osq_is_locked(struct optimistic_spin_queue *lock) { - return atomic_read(&lock->tail) !=3D OSQ_UNLOCKED_VAL; + return READ_ONCE(lock->tail) !=3D OSQ_UNLOCKED_VAL; } =20 #endif diff --git a/kernel/locking/osq_lock.c b/kernel/locking/osq_lock.c index 0f68ee017b54..144eb446c867 100644 --- a/kernel/locking/osq_lock.c +++ b/kernel/locking/osq_lock.c @@ -34,8 +34,8 @@ */ =20 struct optimistic_spin_node { - int next; /* CPU number offset by 1, 0 if no next */ - int prev; /* CPU number offset by 1, 0 if lock held */ + unsigned int next; /* CPU number offset by 1, 0 if no next */ + unsigned int prev; /* CPU number offset by 1, 0 if lock held */ } __aligned(8); =20 static DEFINE_PER_CPU(struct optimistic_spin_node, osq_node); @@ -44,16 +44,15 @@ static DEFINE_PER_CPU(struct optimistic_spin_node, osq_= node); * We use the value 0 to represent "no CPU", thus the encoded value * will be the CPU number incremented by 1. */ -static inline int encode_cpu(int cpu_nr) +static inline unsigned int encode_cpu(unsigned int cpu_nr) { return cpu_nr + 1; } =20 -static inline struct optimistic_spin_node *decode_cpu(int encoded_cpu_val) +static inline struct optimistic_spin_node * +decode_cpu(unsigned int encoded_cpu_val) { - int cpu_nr =3D encoded_cpu_val - 1; - - return per_cpu_ptr(&osq_node, cpu_nr); + return per_cpu_ptr(&osq_node, encoded_cpu_val - 1); } =20 /* @@ -72,17 +71,17 @@ static inline struct optimistic_spin_node *decode_cpu(i= nt encoded_cpu_val) * When a lock request is being cancelled the caller needs 'next' to * set node->prev->next =3D next. */ -static inline int -osq_unlink_from_next(struct optimistic_spin_queue *lock, int prev) +static inline unsigned int +osq_unlink_from_next(struct optimistic_spin_queue *lock, unsigned int prev) { - int curr =3D encode_cpu(smp_processor_id()); + unsigned int curr =3D encode_cpu(smp_processor_id()); struct optimistic_spin_node *node; - int next; + unsigned int next; =20 for (;;) { - int tail =3D atomic_read(&lock->tail); + unsigned int tail =3D READ_ONCE(lock->tail); if (curr =3D=3D tail && - atomic_try_cmpxchg_release(&lock->tail, &tail, prev)) { + try_cmpxchg_release(&lock->tail, &tail, prev)) { /* * We were the last queued, lock->tail now references * prev (or is 0 if the list is now empty). @@ -127,8 +126,8 @@ osq_unlink_from_next(struct optimistic_spin_queue *lock= , int prev) bool osq_lock(struct optimistic_spin_queue *lock) { struct optimistic_spin_node *node, *prev_ptr; - int curr =3D encode_cpu(smp_processor_id()); - int next, prev; + unsigned int curr =3D encode_cpu(smp_processor_id()); + unsigned int next, prev; =20 /* * We need both ACQUIRE (pairs with corresponding RELEASE in @@ -136,7 +135,7 @@ bool osq_lock(struct optimistic_spin_queue *lock) * the node fields we just initialised) semantics when updating * the lock tail. */ - prev =3D atomic_xchg(&lock->tail, curr); + prev =3D xchg(&lock->tail, curr); if (prev =3D=3D OSQ_UNLOCKED_VAL) return true; =20 --=20 2.39.5 From nobody Fri Sep 25 23:53:11 2026 Received: from mail-wr1-f48.google.com (mail-wr1-f48.google.com [209.85.221.48]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E35DB448D0C for ; Mon, 7 Sep 2026 08:41:58 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.221.48 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788770521; cv=none; b=PEZx8R0wykR/SzFmY2ludVIsYnk0JQop9+4Qbf2NAZIx5TSvbkjYxawnrVYchKjN7KdT2Sz68bTX98ggjoXHprwRL5/N4eVs7FquaWtl28XImT0FFYk//KvcJQJQwWx6Sl13URjyrfVYXXxbnP1n+PyDkLxLSmA0W4usicuj//Q= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788770521; c=relaxed/simple; bh=i1nvPkZL8AYjVqF81FIyYnrrcNRmKU4Rb1W27F5h+8g=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=NjUNpK+T4AinYnuIVbnCKzvqbMMGFI3u3RJGiwgevlP+fCyXZwTkVvFkxM1M3uHMloe/1k7POb2XM7xG/141K/gNFRR2RfcIPVaLpwUISMl0rSNXyhNojta/YBEEabaAaJoZ5+347x53BFyT57oXiJYlbIP3FA+QcjvXEP44uFQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=Q4vrBGJm; arc=none smtp.client-ip=209.85.221.48 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="Q4vrBGJm" Received: by mail-wr1-f48.google.com with SMTP id ffacd0b85a97d-4843f205a5bso1832036f8f.1 for ; Mon, 07 Sep 2026 01:41:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1788770517; x=1789375317; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=aebnt8Kz95RzOuAHiVC9YDlZSkH27kAJite7oJqhniU=; b=Q4vrBGJmWjwtww+BUWpv/8w6JiSYu+NbnnOW+fX+NcvbKWVGz82harEIoNP3zvgznj OZ0ohv3xFp92bIur1ULntY7YMkupH/SfdcThKw4Km7ZR4kvuHEoAqhF3hnyfWX9OeIOt swSATzVf+JXUcz3qHlWTmSKst7SI7VZm9bMOdkXjZRnoSN+5IJnLG+eU4etKp1gnJHJF qfO3YkfewAkUTMWPQUx2cMXcTlXJVe4uvdwJF2lcro83m5k6mqP42eYY7TSBSiTK5E2d fJXYScq8Ebe9HS/oBABxaDJp8E63481qRG7QYyj759hdJze1/wNf9B9Ed6Xt/g1KKClP d9Mg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1788770517; x=1789375317; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=aebnt8Kz95RzOuAHiVC9YDlZSkH27kAJite7oJqhniU=; b=D1VWO8pmTtJUkG+4MCJyyrjHo8jUb27uRsFhDWN/hsC5S+95iN1yqj5IkWdXLFoaFn QmmoQb75kdFtRcHfyR6JKRz1JLdPUJebR6mW00CiIT1AvVDlzSOjwJOtcZMUB3fSEOZg d66gkZ8oJRJhw8kZgH7weU+qRQyNUFYcwM1hk91hza1QqIuTx/KNwXL9PTv1HTVzFN7l DMyKB5SuGadilgoNYPgU3sv6YDh3fWDTuK2FRbCLSyf+qTQiEjqlIpfhM9lpvf8+ubH6 FPulHkwkUiyeeCdMMuQ5i5oB5vlOwhaOa0fQ5vSinWNyzKR9gfh/8yB1sp/G3hNb6QXM y9PA== X-Forwarded-Encrypted: i=1; AKwUvBzMmTbyg2Y/lGer2/ydtbdNsfr6Sj5iHSEjNIGQga2pfyHkOV015attVwps0KMtNuA92ejB/gLRcYGi6hE=@vger.kernel.org X-Gm-Message-State: AFuF++l1FBurq5yjXS9sBdUwFvz8Cpo1lNB++AnHH63xLE6XbUe8UVmC kYZ9DkUV5ZAEI8eBBLyWJ9Rfzdu5szv4o64Rbr77e9qjuLD2smfNfN0s X-Gm-Gg: AYBFou10A9ix7AU0KSCBvuKImjKaedlp6YB2ZnDLFxX3GNVHF2ccF/5ca9eDiLYxsc4 dvs9WZvHvngHtsGGDGQrnCPymbH9CONyq46ECIOJ2afyoOSrcBCI4R9W1feZ/t+a0n/2i3Bq+n/ 8iVLHkJtnmOenlFFf26WdKtTeUH1rsFkwT//ZcQEJNYTfFL/A+BpYLePHnXS6RrHJMjMxpXzdz4 3uJ1gp4dXLOpp0QdKsNpBPSwbyUQSv1kVpaA/wxKPVNGHgvJlfti93OGAsaeMv81X/zVSVDgZjA zG9kLb81bl2kD85mYYFY+8Rup07hsvqhXPuTXQFTpAHIqO/8ogq5/seawvkrpRclmxFnU2X36jp HjLBq1meGEMJdJ0+7n3S5b4/nTYZcpinzdTyNzPAZdVSYt70k6zKr/S1qierSgkGDrcV8FL2VtJ bY0YEP0MQFbzizRM1qXgqAK+0fzT1XA97hR4dSXc9e6jD/oI5D5sBbsjyAN8nCeUFw0iRdpiNM7 cvfA6/bObZMtrx3Y9S4fCFndYMLDQnpb8m6KBf8FjrfaSdcjAwN92Cr X-Received: by 2002:a05:6000:2508:b0:485:8e8b:7bf4 with SMTP id ffacd0b85a97d-4858e8b7c78mr18050026f8f.33.1788770516952; Mon, 07 Sep 2026 01:41:56 -0700 (PDT) Received: from snowdrop.snailnet.com (82-69-66-36.dsl.in-addr.zen.co.uk. [82.69.66.36]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-485885bbb51sm27883762f8f.30.2026.09.07.01.41.56 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 07 Sep 2026 01:41:56 -0700 (PDT) From: David Laight To: Waiman Long , Peter Zijlstra , Ingo Molnar , Will Deacon , Boqun Feng , linux-kernel@vger.kernel.org, Linus Torvalds , Yafang Shao , Steven Rostedt Cc: David Laight Subject: [PATCH v4 next 8/9] locking/osq: inline encode_cpu() and rename decode_cpu() Date: Mon, 7 Sep 2026 09:41:32 +0100 Message-Id: <20260907084133.3696-9-david.laight.linux@gmail.com> X-Mailer: git-send-email 2.39.5 In-Reply-To: <20260907084133.3696-1-david.laight.linux@gmail.com> References: <20260907084133.3696-1-david.laight.linux@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Having a function to add the 1 offset to the cpu number doesn't make the code any easier to read. Just use smp_processor_id() + 1. Rename decode_cpu() to cpu_spin_node() to match what it does. No functional change. Signed-off-by: David Laight Tested-by: H=C3=A5kon Bugge --- kernel/locking/osq_lock.c | 23 +++++++---------------- 1 file changed, 7 insertions(+), 16 deletions(-) diff --git a/kernel/locking/osq_lock.c b/kernel/locking/osq_lock.c index 144eb446c867..8dfd729d1a81 100644 --- a/kernel/locking/osq_lock.c +++ b/kernel/locking/osq_lock.c @@ -40,19 +40,10 @@ struct optimistic_spin_node { =20 static DEFINE_PER_CPU(struct optimistic_spin_node, osq_node); =20 -/* - * We use the value 0 to represent "no CPU", thus the encoded value - * will be the CPU number incremented by 1. - */ -static inline unsigned int encode_cpu(unsigned int cpu_nr) -{ - return cpu_nr + 1; -} - static inline struct optimistic_spin_node * -decode_cpu(unsigned int encoded_cpu_val) +cpu_spin_node(unsigned int offset_cpu_num) { - return per_cpu_ptr(&osq_node, encoded_cpu_val - 1); + return per_cpu_ptr(&osq_node, offset_cpu_num - 1); } =20 /* @@ -74,7 +65,7 @@ decode_cpu(unsigned int encoded_cpu_val) static inline unsigned int osq_unlink_from_next(struct optimistic_spin_queue *lock, unsigned int prev) { - unsigned int curr =3D encode_cpu(smp_processor_id()); + unsigned int curr =3D smp_processor_id() + 1; struct optimistic_spin_node *node; unsigned int next; =20 @@ -118,7 +109,7 @@ osq_unlink_from_next(struct optimistic_spin_queue *lock= , unsigned int prev) * When called while unqueueing in osq_lock() this completes the * backwards link, the forwards link is done by the caller. */ - WRITE_ONCE(decode_cpu(next)->prev, prev); + WRITE_ONCE(cpu_spin_node(next)->prev, prev); =20 return next; } @@ -126,7 +117,7 @@ osq_unlink_from_next(struct optimistic_spin_queue *lock= , unsigned int prev) bool osq_lock(struct optimistic_spin_queue *lock) { struct optimistic_spin_node *node, *prev_ptr; - unsigned int curr =3D encode_cpu(smp_processor_id()); + unsigned int curr =3D smp_processor_id() + 1; unsigned int next, prev; =20 /* @@ -140,7 +131,7 @@ bool osq_lock(struct optimistic_spin_queue *lock) return true; =20 node =3D this_cpu_ptr(&osq_node); - prev_ptr =3D decode_cpu(prev); + prev_ptr =3D cpu_spin_node(prev); node->prev =3D prev; =20 /* @@ -189,7 +180,7 @@ bool osq_lock(struct optimistic_spin_queue *lock) /* Lock acquired */ return true; =20 - prev_ptr =3D decode_cpu(prev); + prev_ptr =3D cpu_spin_node(prev); =20 if (data_race(prev_ptr->next) =3D=3D curr && cmpxchg(&prev_ptr->next, curr, 0) =3D=3D curr) --=20 2.39.5 From nobody Fri Sep 25 23:53:11 2026 Received: from mail-wr1-f53.google.com (mail-wr1-f53.google.com [209.85.221.53]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id AFEFC449B21 for ; Mon, 7 Sep 2026 08:41:59 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.221.53 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788770522; cv=none; b=i5vi4HOVkhNosF3tHK6qLIxqt9FJ9daqW7X6fB9yFH9jzNsxy0Kkf6MwT95jW7X7q2bMImf6wjkV/qIVTgAmv38RqM9FJdMQuLQjM8gN79dAAmYfJBps2jcYxbp7V/MDHX5ciFp8/SpsOuYSnhUW3jftloSUbYcBB0muooVQjP4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788770522; c=relaxed/simple; bh=dYg9aLSnApFgnIWYTGR8etCGK0yBPIva5TopG4FWNyk=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=bKjjd2gGfcdZI7he6Pw5+TJ7HE83IiYfLIbBDi0iUekvunwq9x/x44s65VDbZL8gz1sX3h85IRMFeR0eUE+ST9q8CVZax2xuaD1j+feoUcQDtZJ2GOolwfxr+KKMzNweBxpJi+W0G5vP2YYYwzOr0RMJxKYwTGzZaB7MuUTV8HM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=IDB4IGO0; arc=none smtp.client-ip=209.85.221.53 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="IDB4IGO0" Received: by mail-wr1-f53.google.com with SMTP id ffacd0b85a97d-482e257a23aso2240119f8f.0 for ; Mon, 07 Sep 2026 01:41:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1788770518; x=1789375318; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=fS4AFVoVfQm6rlehNhZyNvIrVZ1gzCjIRTDz5TvulTE=; b=IDB4IGO09610dIGOmpw//zotOFcswjyEMolSPP+vYKyt8os+hKo13nkUM4/QXoCd3A 55PqxG2LdD9Vm+wN6n2ascW1/7Alv8qOMjA3JuK5W6NvF00mVvvotMRfjixVrCWh/fcY xANPPEhFndWkCGXB4Wj26jhACLL5JN+Kht0q+N3JUv461bf2bu2aMxJ9tL++ikua5vef Pq7Regz/VOHEshovxz4B3Dv9ndx4fcbYwkC4RwRYCQGJ34glqFhb8jDvlgPU8jajvguJ SsAdU6wE9I4JPt18V6mJbJI5+DS4ZUFXGFUvkiL+l71F2gTxfGzAaApwtsBCTnNpVlcu ShxA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1788770518; x=1789375318; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=fS4AFVoVfQm6rlehNhZyNvIrVZ1gzCjIRTDz5TvulTE=; b=h3baLOFYOl0TvjyM1yn/D8p9xqEiv/MIwvdplY+HGYeiL2YEAJWnEbw5PmZqq7rg++ /mCnVE5UZ9jBGl5NR3NByU5dUp/JOALWdgnHxC8gbWJFJZcK6/8/GF1UIwhsPazyLtsl w5iGaUJiuO3kQHw4qy7hpGkmUahSFuLmhfeq25WjsesS049Iq6y5ak4JQZZcfzAwoCxN SBcSFbIJbqF2blVBdR5uuZC/NCPDIaZOvcYPRKthLPlDseYqkPKf3r2FJd6v1uhgAqjO Q2JeIl/zav9/0LPn6PH//VMdkK5eEQ2Qg+Th3aQrvOc/doDHYoqihDCmAAnydUfqJaX2 SfgQ== X-Forwarded-Encrypted: i=1; AKwUvBznBdbfDEhwM0+jyb2/hmmhkIFShdq5sj4pBNqktVR33qMpIBwm9n10+7shPXW8RiBh09ZPsQmcGbgDc1M=@vger.kernel.org X-Gm-Message-State: AFuF++kZG0ydqWSaDCCAME7rhfIL45zJ4EFREfKc9xpPIcAMRj89l7ot 1l50RlIAX+bA2A3e+TKqQLjmovypG/TmetH4fLRfscrKQMnuiTaz8rFZVK4Ej51a X-Gm-Gg: AYBFou0DEh+pxLZv/wZoa3J02szQHg2bkphgNxh2z2AA4GjRRqNf1ycr6i2nzeRjtQA GYXi9CRgRyfsT32uscONaKej8Jo/PvN9CCynArL6u47WrS5APfEP4p2gNUlKNc8LO/Gy3MjCSDI LNvOBQf9lX/0Ck6ykDVcmGVqIFLuVQOTnYsvkljO3MlcjQuHNhgFNUmuAwVW5JcPDYab3DFZrsy /+sCswBmep4yu8N6KP41h3b7UwrIipNvUDrzxmAQwNJ1jF8Uu/PyzEto1Izlt2KT80iMz7xPaxA Nf3axd+o5L3dL8if/WzzbU+/ZmmnZ4K4poHmljNLOETdyoZZ2UYPJEkE9EFXEguT00o1EPGIys5 Hp9RirbJgbjdYRQSvm9XhMk4AknjqR03WmZfMOH2N73T2s2PVP3n47IochBuLTWM3XjtlMkfz8M BsbHnSL1CaRLPerfUn9TcguzLh4t7RO/vcUHoJJv+CJq7vVsEwxWjrMw610FT417dB7X+/OSlP4 VZ1HMHL1dewsnyIlrVQ7/PqegYJcR1sqnDFbM9QUWMzSQ== X-Received: by 2002:a05:6000:240e:b0:485:8de9:4b85 with SMTP id ffacd0b85a97d-4858de94e2emr16063005f8f.0.1788770517514; Mon, 07 Sep 2026 01:41:57 -0700 (PDT) Received: from snowdrop.snailnet.com (82-69-66-36.dsl.in-addr.zen.co.uk. [82.69.66.36]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-485885bbb51sm27883762f8f.30.2026.09.07.01.41.57 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 07 Sep 2026 01:41:57 -0700 (PDT) From: David Laight To: Waiman Long , Peter Zijlstra , Ingo Molnar , Will Deacon , Boqun Feng , linux-kernel@vger.kernel.org, Linus Torvalds , Yafang Shao , Steven Rostedt Cc: David Laight Subject: [PATCH v4 next 9/9] locking/osq_lock: Swap next<->prev and tail<->head Date: Mon, 7 Sep 2026 09:41:33 +0100 Message-Id: <20260907084133.3696-10-david.laight.linux@gmail.com> X-Mailer: git-send-email 2.39.5 In-Reply-To: <20260907084133.3696-1-david.laight.linux@gmail.com> References: <20260907084133.3696-1-david.laight.linux@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" I think it is easier to understand this code if you think that cpu add themselves to the head of the list and the cpu at the tail owns the lock. Remember nodes are never removed from the list - they only remove themselves. So swap the field names over. Pretty much all the code has already been changed in this series so it doesn't make that much difference to the overall lines changed. Signed-off-by: David Laight Tested-by: H=C3=A5kon Bugge --- include/linux/osq_lock.h | 8 +- kernel/locking/osq_lock.c | 165 +++++++++++++++++++------------------- 2 files changed, 86 insertions(+), 87 deletions(-) diff --git a/include/linux/osq_lock.h b/include/linux/osq_lock.h index 9e637e265189..d471f3ab4238 100644 --- a/include/linux/osq_lock.h +++ b/include/linux/osq_lock.h @@ -9,10 +9,10 @@ =20 struct optimistic_spin_queue { /* - * Stores an encoded value of the CPU # of the tail node in the queue. + * Stores an encoded value of the CPU # of the head node in the queue. * If the queue is empty, then it's set to OSQ_UNLOCKED_VAL. */ - unsigned int tail; + unsigned int head; }; =20 #define OSQ_UNLOCKED_VAL (0) @@ -22,7 +22,7 @@ struct optimistic_spin_queue { =20 static inline void osq_lock_init(struct optimistic_spin_queue *lock) { - WRITE_ONCE(lock->tail, OSQ_UNLOCKED_VAL); + WRITE_ONCE(lock->head, OSQ_UNLOCKED_VAL); } =20 extern bool osq_lock(struct optimistic_spin_queue *lock); @@ -30,7 +30,7 @@ extern void osq_unlock(struct optimistic_spin_queue *lock= ); =20 static inline bool osq_is_locked(struct optimistic_spin_queue *lock) { - return READ_ONCE(lock->tail) !=3D OSQ_UNLOCKED_VAL; + return READ_ONCE(lock->head) !=3D OSQ_UNLOCKED_VAL; } =20 #endif diff --git a/kernel/locking/osq_lock.c b/kernel/locking/osq_lock.c index 8dfd729d1a81..8dd01fee80bf 100644 --- a/kernel/locking/osq_lock.c +++ b/kernel/locking/osq_lock.c @@ -17,25 +17,24 @@ * spinning. * * The osq_nodes for the spinning CPU are put on a double-linked (non circ= ular) - * list. The list 'pointers' can either be the address of the osq_node or = the - * associated CPU number, the CPU numbers are offset by one so that zero c= an - * be used like a NULL ponter. - * The mutex/rwsem contains a pointer (CPU number) to the tail of the list. - * There is no equivalent pointer to the list head - the 'head' is the - * osq_node of the CPU that acquired the osq lock. + * list similar to an hlist. + * The list 'pointers' are the CPU numbers (offset by one so that zero can + * be used like a NULL ponter). + * Waiting cpu are added to the head of the list, the tail of the list is + * the osq_node of the CPU that acquired the osq lock. * - * The 'next' pointer of the tail must be zero, all the other 'next' point= ers + * The 'prev' pointer of the head must be zero, all the other 'prev' point= ers * must either be valid or transiently zero. - * The 'prev' pointer is zero unless the node is waiting for the lock, when - * waiting it may refer to the wrong node (node->prev->next !=3D node). - * The 'prev' value is only needed for the node->prev->next =3D node->next= update - * when 'node' is being removed. Atomically checking node->prev->next =3D= =3D node + * The 'next' pointer is zero unless the node is waiting for the lock, when + * waiting it may refer to the wrong node (node->next->prev !=3D node). + * The 'next' value is only needed for the node->next->prev =3D node->prev= update + * when 'node' is being removed. Atomically checking node->next->prev =3D= =3D node * ensures the list doesn't get corrupted. */ =20 struct optimistic_spin_node { - unsigned int next; /* CPU number offset by 1, 0 if no next */ - unsigned int prev; /* CPU number offset by 1, 0 if lock held */ + unsigned int next; /* CPU number offset by 1, 0 if lock held */ + unsigned int prev; /* CPU number offset by 1, 0 if no prev */ } __aligned(8); =20 static DEFINE_PER_CPU(struct optimistic_spin_node, osq_node); @@ -47,38 +46,38 @@ cpu_spin_node(unsigned int offset_cpu_num) } =20 /* - * Unlink the current cpu's node from the lock's node->prev list. + * Unlink the current cpu's node from the lock's node->next list. * - * More specifically atomically write its node->prev over the link that + * More specifically atomically write its node->next over the link that * currently points to node. * This is either: - * lock->tail =3D node->prev + * lock->head =3D node->next * or: - * node->next->prev =3D node->prev + * node->prev->next =3D node->next * The first is a simple cmpxchg(), the second is protected against - * node->next trying to unlink itself (after need_resched() is set) by usi= ng - * an xchg() on node->next that sets it to NULL. + * node->prev trying to unlink itself (after need_resched() is set) by usi= ng + * an xchg() on node->prev that sets it to NULL. * - * When a lock request is being cancelled the caller needs 'next' to - * set node->prev->next =3D next. + * When a lock request is being cancelled the caller needs 'prev' to + * set node->next->prev =3D prev. */ static inline unsigned int -osq_unlink_from_next(struct optimistic_spin_queue *lock, unsigned int prev) +osq_unlink_from_prev(struct optimistic_spin_queue *lock, unsigned int next) { unsigned int curr =3D smp_processor_id() + 1; struct optimistic_spin_node *node; - unsigned int next; + unsigned int prev; =20 for (;;) { - unsigned int tail =3D READ_ONCE(lock->tail); - if (curr =3D=3D tail && - try_cmpxchg_release(&lock->tail, &tail, prev)) { + unsigned int head =3D READ_ONCE(lock->head); + if (curr =3D=3D head && + try_cmpxchg_release(&lock->head, &head, next)) { /* - * We were the last queued, lock->tail now references - * prev (or is 0 if the list is now empty). - * If prev was spinning in this loop it can continue. + * We were the last queued, lock->head now references + * next (or is 0 if the list is now empty). + * If next was spinning in this loop it can continue. * - * Since we are the tail of the list, node->next + * Since we are the head of the list, node->prev * must be zero. */ return 0; @@ -87,16 +86,16 @@ osq_unlink_from_next(struct optimistic_spin_queue *lock= , unsigned int prev) node =3D this_cpu_ptr(&osq_node); =20 /* - * We must xchg() the @node->next value to ensure that a - * concurrent unqueue() from @node->next will find an invalid - * @prev value (node_next->prev->next !=3D node_next). + * We must xchg() the @node->prev value to ensure that a + * concurrent unqueue() from @node->prev will find an invalid + * @next value (node_prev->next->prev !=3D node_prev). * - * If @node->next is already NULL then we need to wait until + * If @node->prev is already NULL then we need to wait until * the concurrent unqueue completes. */ - if (node->next) { - next =3D xchg(&node->next, 0); - if (next) + if (node->prev) { + prev =3D xchg(&node->prev, 0); + if (prev) break; } =20 @@ -104,52 +103,52 @@ osq_unlink_from_next(struct optimistic_spin_queue *lo= ck, unsigned int prev) } =20 /* - * When called from osq_unlock() prev is zero and this hands + * When called from osq_unlock() next is zero and this hands * over the lock ownership. * When called while unqueueing in osq_lock() this completes the * backwards link, the forwards link is done by the caller. */ - WRITE_ONCE(cpu_spin_node(next)->prev, prev); + WRITE_ONCE(cpu_spin_node(prev)->next, next); =20 - return next; + return prev; } =20 bool osq_lock(struct optimistic_spin_queue *lock) { - struct optimistic_spin_node *node, *prev_ptr; + struct optimistic_spin_node *node, *next_ptr; unsigned int curr =3D smp_processor_id() + 1; - unsigned int next, prev; + unsigned int prev, next; =20 /* * We need both ACQUIRE (pairs with corresponding RELEASE in * unlock() uncontended, or fastpath) and RELEASE (to publish * the node fields we just initialised) semantics when updating - * the lock tail. + * the lock head. */ - prev =3D xchg(&lock->tail, curr); - if (prev =3D=3D OSQ_UNLOCKED_VAL) + next =3D xchg(&lock->head, curr); + if (next =3D=3D OSQ_UNLOCKED_VAL) return true; =20 node =3D this_cpu_ptr(&osq_node); - prev_ptr =3D cpu_spin_node(prev); - node->prev =3D prev; + next_ptr =3D cpu_spin_node(next); + node->next =3D next; =20 /* * osq_lock() unqueue * - * node->prev =3D prev osq_unlink_from_next() + * node->next =3D next osq_unlink_from_prev() * WMB MB - * prev->next =3D node next->prev =3D prev // unqueue-C + * next->prev =3D node prev->next =3D next // unqueue-C * - * Here 'node->prev' and 'next->prev' are the same variable and we need + * Here 'node->next' and 'prev->next' are the same variable and we need * to ensure these stores happen in-order to avoid corrupting the list. */ smp_wmb(); =20 - WRITE_ONCE(prev_ptr->next, curr); + WRITE_ONCE(next_ptr->prev, curr); =20 /* - * Normally @prev is untouchable after the above store; because at that + * Normally @next is untouchable after the above store; because at that * moment unlock can proceed and wipe the node element from stack. * * However, since our nodes are static per-cpu storage, we're @@ -163,31 +162,31 @@ bool osq_lock(struct optimistic_spin_queue *lock) * is implemented with a monitor-wait. vcpu_is_preempted() relies on * polling, be careful. */ - prev =3D smp_cond_load_relaxed(&node->prev, !VAL || need_resched() || + next =3D smp_cond_load_relaxed(&node->next, !VAL || need_resched() || vcpu_is_preempted(VAL - 1)); =20 /* - * Loop until either node->prev is zero (lock acquired) or we - * atomically change prev->next from node to NULL (stopping prev + * Loop until either node->next is zero (lock acquired) or we + * atomically change next->prev from node to NULL (stopping next * handing on the lock). - * Note that 'prev' can unlink itself concurrently with this - * test so that prev/prev_ptr can be stale, but since it + * Note that 'next' can unlink itself concurrently with this + * test so that next/next_ptr can be stale, but since it * is per-cpu data the memory can always be read. */ =20 - for (;; prev =3D READ_ONCE(node->prev)) { - if (!prev) + for (;; next =3D READ_ONCE(node->next)) { + if (!next) /* Lock acquired */ return true; =20 - prev_ptr =3D cpu_spin_node(prev); + next_ptr =3D cpu_spin_node(next); =20 - if (data_race(prev_ptr->next) =3D=3D curr && - cmpxchg(&prev_ptr->next, curr, 0) =3D=3D curr) + if (data_race(next_ptr->prev) =3D=3D curr && + cmpxchg(&next_ptr->prev, curr, 0) =3D=3D curr) break; =20 /* - * 'prev' must have unlinked (or be in the process of unlinking) + * 'next' must have unlinked (or be in the process of unlinking) * itself from the list. */ =20 @@ -195,47 +194,47 @@ bool osq_lock(struct optimistic_spin_queue *lock) } =20 /* - * If 'prev' tries to remove itself from the list before we write - * a new value to prev->next it will spin in osq_unlink_from_next(). + * If 'next' tries to remove itself from the list before we write + * a new value to next->prev it will spin in osq_unlink_from_prev(). * This means we can no longer be given the lock and always * return false. */ =20 /* - * Invalidate prev matching osq_unlock(). + * Invalidate next matching osq_unlock(). * This isn't necessary but ensures that both unlocked and fast-path - * locked nodes (where the initial xchg() returned 0) have prev set + * locked nodes (where the initial xchg() returned 0) have next set * to zero. - * If nothing else it lets the lock chain be followed from lock->tail + * If nothing else it lets the lock chain be followed from lock->head * whch may help diagnostics. */ - node->prev =3D 0; + node->next =3D 0; =20 /* - * Now that the linkage to prev cannot change underneath us - * remove ourselves from the node->prev list. + * Now that the linkage to next cannot change underneath us + * remove ourselves from the node->next list. * This does: - * (node->next ? node->next->prev : lock->tail) =3D node->prev + * (node->prev ? node->prev->next : lock->head) =3D node->next */ - next =3D osq_unlink_from_next(lock, prev); + prev =3D osq_unlink_from_prev(lock, next); =20 /* - * Finally mend the node->next list that was 'broken' to - * stop node->prev trying to unlink from us. - * If next is NULL then lock->tail is prev_ptr and another node + * Finally mend the node->prev list that was 'broken' to + * stop node->next trying to unlink from us. + * If prev is NULL then lock->head is next_ptr and another node * can be added - so we must not re-write the NULL. */ - if (next) { + if (prev) { /* - * This must happen after the write to node->next->prev. - * If swapped then prev could unlink itself before our - * write to node->next->prev and the the wrong value would - * end up in node->next->prev. + * This must happen after the write to node->prev->next. + * If swapped then next could unlink itself before our + * write to node->prev->next and the the wrong value would + * end up in node->prev->next. * Probably can't actually happen due to re-ordering of writes, * but could happen without a compiler barrier. */ smp_wmb(); - WRITE_ONCE(prev_ptr->next, next); + WRITE_ONCE(next_ptr->prev, prev); } =20 return false; @@ -243,5 +242,5 @@ bool osq_lock(struct optimistic_spin_queue *lock) =20 void osq_unlock(struct optimistic_spin_queue *lock) { - osq_unlink_from_next(lock, OSQ_UNLOCKED_VAL); + osq_unlink_from_prev(lock, OSQ_UNLOCKED_VAL); } --=20 2.39.5