From nobody Tue Apr 7 07:29:15 2026 Received: from mail-wm1-f45.google.com (mail-wm1-f45.google.com [209.85.128.45]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E753C32ABCA for ; Fri, 6 Mar 2026 22:51:56 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.45 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1772837518; cv=none; b=u+NljCTwMek3YRC8gzXo52A7h5sY9xgaL79IDhyhm9mCpvCKYsDS9pzTVA0AWj/k+qjNY+bZhHqm2YnW7/Ut2UM3Zkl1yyy9H4Th7qCPFIhtY9LIdKiYqFnS3HZ5to65y/7cLW/R7Y1sDQCfkVki+cYzPzknoZN0ykrsqVMv7cY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1772837518; c=relaxed/simple; bh=HQx4FZP0hwoFRaYP+yZVzvupoNEj7XezlDV451i1JR4=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=ip+E62X+hrVeC/srPWhvuFhvwCm4XYb0yQd0Og/YWAAU2ka2q1nPvRqbG/9OyaZn4emiyeAumwUkwTDRcmJZ3mCNBn5YOzxmGLGeHBYcbHI65DTSHeE+EQtfoxPqvtSpeBvuj7AVZgyY7N2bQqpgCylx7fafi4nuk+NziE5pNo4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=SM5o5VZK; arc=none smtp.client-ip=209.85.128.45 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="SM5o5VZK" Received: by mail-wm1-f45.google.com with SMTP id 5b1f17b1804b1-4852b81c73aso4033315e9.3 for ; Fri, 06 Mar 2026 14:51:56 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1772837515; x=1773442315; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=UItt9/FwrhN7roAlUPWfDla/4L+V2++SjZUd0w1EyJ0=; b=SM5o5VZKVwF2AelxIaFScaioC398hdWs04KQMexdHUEtCKyQ1VOq3MYVakVuVx6n/X N7gPuyklyVdCZ+qO9ddGZQJmln7RkhYLBXkUy3N/6uhnXUC1l6DRf1Z4FGU0o0s8B/pw GuLvsfHgP+tVmh/s9sLbaH6qe0jLHae7pd7o7HkNgnveFhwodM1ho8VYfQsgY0tiBALC JE+ivRXHwmsjY+fAsVw+yCtUNpI7YlAg9jer40ukfCAYdQtmzMeKB9uU4ZhqhIAERm7N b3dKpwddvQDPjtFhh6ifl38Xr2cexaMcwLTZX0RasqFi86T6eMuO4FMoYqWuWP0p3gJr HRvg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1772837515; x=1773442315; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=UItt9/FwrhN7roAlUPWfDla/4L+V2++SjZUd0w1EyJ0=; b=NTIDK1lUgpRQuIHHDCPiz9MDL7gunFGDKEXmEZpngDk059OX6j+DdKhRY+bAQn5CGY t9/Wpt1BuqZutGlzr04/+wkUDbfKFLt9Rr5Mb1d8+VJC56cN5kPN+UzugOnEGzeW51Pe rGFHUXexaHRD7V4GafNXIOd29iIW7ofJuU0uQVUq8tYf/O+1gIZZe/SRMk1U9JFn8IBL ELS7AsSf4rcWk18p+TK5682vAe5SXYw+kaH0YurD6AVKQwxbMtQmXDa+M4cE4Zd25ggd anf0vXRm2tefgPE7Ezs+oBX9vA7ZlczwPfsmIIIVjyBoB1dhbhg+8vPxko4XIbRrikm8 ezhA== X-Forwarded-Encrypted: i=1; AJvYcCW+AYVWyRcM+48FUnWYRWcSK9gx8Hjb6vw0b6P6QR94BYOYE1nK1I8cDqbBHRNNudNR+cfWMcNfaFe6KYc=@vger.kernel.org X-Gm-Message-State: AOJu0YyBal9reCJ22AnIPT78LP+YwPbZndHk7tNTM4GcPUhpfmFSk1pP eCQUWCAFUWfcq2QLLcAaRQNiUDxladKayp2z5LQgO3GRRDvOxUQWi40+ X-Gm-Gg: ATEYQzyY+HHpeAI4Hzjww37eAFBDObB1ylAOl5/LaSeg4GtOkZpfZf1brq2RUI1JDij d2v6tjFy0Si+P5A2J3qfWF6OnkI1wfD5pc9UM9FShRzoXY2q0bvgw+ECSM4/RAiHPE8mkZfZ8Qw gv07nIP7u/X0U+CjHKZJM2/JtORESGI8krghu2hd44eppBWwipe4VZCsdbDAY8K9IGxdZTbK1cF 57O14QFFityWGdEgeMAvK4PNDx29bumL6ugS0+ra4jOEzQB7qFtUYKOf1aQJtwnIUZvRb1naoco sQx5i7x+GL4cHX84a8VRtRWlN44qGQC0yRKk3wcNGLw7lFb6qX2kBFMjnhP8u28o8Lv4cR6mH+U MfS6D977OzuKhLqt44eRwsKJuP5sgYdVQR+wtwyvz+QTB5JA5OiRnCflqGSCZs/a651QvssTU1e Tc6xSk7yRKWdVG/Q/cWgaESMaTtC71lLn0wzLq8rxjCKNyLCZ2zZA8BLmqVUFZPl8U5ZB9uqH7y SjSRsp04Fxr X-Received: by 2002:a05:600c:3acf:b0:480:1c10:5633 with SMTP id 5b1f17b1804b1-48526979e65mr61521555e9.26.1772837515220; Fri, 06 Mar 2026 14:51:55 -0800 (PST) Received: from snowdrop.snailnet.com (82-69-66-36.dsl.in-addr.zen.co.uk. [82.69.66.36]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-439dae57c05sm6160845f8f.39.2026.03.06.14.51.54 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 06 Mar 2026 14:51:54 -0800 (PST) From: david.laight.linux@gmail.com To: Waiman Long , Peter Zijlstra , Ingo Molnar , Will Deacon , Boqun Feng , linux-kernel@vger.kernel.org, Linus Torvalds , Yafang Shao , Steven Rostedt Cc: David Laight , David Laight Subject: [PATCH v3 next 1/5] Only clear node->locked in the slow osq_lock() path Date: Fri, 6 Mar 2026 22:51:46 +0000 Message-Id: <20260306225150.93178-2-david.laight.linux@gmail.com> X-Mailer: git-send-email 2.39.5 In-Reply-To: <20260306225150.93178-1-david.laight.linux@gmail.com> References: <20260306225150.93178-1-david.laight.linux@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: David Laight node->locked is used to indicate that the owner of the lock has handed it off to the waiting CPU. As such it's value is only relevant in the slow path so it need not be initialised in the fast path. Signed-off-by: David Laight --- kernel/locking/osq_lock.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/kernel/locking/osq_lock.c b/kernel/locking/osq_lock.c index b4233dc2c2b0..96c6094157b5 100644 --- a/kernel/locking/osq_lock.c +++ b/kernel/locking/osq_lock.c @@ -97,7 +97,6 @@ bool osq_lock(struct optimistic_spin_queue *lock) int curr =3D encode_cpu(smp_processor_id()); int old; =20 - node->locked =3D 0; node->next =3D NULL; node->cpu =3D curr; =20 @@ -113,6 +112,7 @@ bool osq_lock(struct optimistic_spin_queue *lock) =20 prev =3D decode_cpu(old); node->prev =3D prev; + node->locked =3D 0; =20 /* * osq_lock() unqueue --=20 2.39.5 From nobody Tue Apr 7 07:29:15 2026 Received: from mail-wm1-f51.google.com (mail-wm1-f51.google.com [209.85.128.51]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7269034F279 for ; Fri, 6 Mar 2026 22:51:57 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.51 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1772837518; cv=none; b=nCAauxHpswk0IVZ2l2e9GcLtBpmhxvaFtiQSIDMkijo0qqyKGWYNn5Jn0Y1suInmOvQQ3bF14FDuC6Ui0mzEeV26XzJQzCMdI0MgSO0GRsgyzhoOiI6+3gPEyKXxzeisIF/RRNQIUHiF6i0LfRki9hsXN9Q+0NxE0+dmYdz1TdE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1772837518; c=relaxed/simple; bh=tx+j7JVVMzme9A8nZVX+nxHi38OaZRMPAgTMYNrU8ZE=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=FvWYWW1spzujiRpMEKViYHK3NRWYPTJGwRpuwMKlaypvsOjaURmeZh/T2jEyfRG1NSsQaCmIGjkn/c+JANR23oe3WAunj5BYjWtX7rZdbJ9cNWophQ5rQUp9I+J/W2gznvwhozo4ZP+C5LQBkrXm03dzbzPLdE4Pegx9XcP9iUU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=M8K+PrkD; arc=none smtp.client-ip=209.85.128.51 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="M8K+PrkD" Received: by mail-wm1-f51.google.com with SMTP id 5b1f17b1804b1-4852a9c6309so5234045e9.0 for ; Fri, 06 Mar 2026 14:51:57 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1772837516; x=1773442316; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=dYRb/0buUb3p9AxqkSCYQnLtLdpN/kPE8ow1mG/Cv8E=; b=M8K+PrkDZ9rRmaC5Qj5DM0EmW/KmomrYpY+S6QqIcS0/FMA/jAyey+K5ruulaz6vng A8Xs789IdxZqgl5m/ggTKJHAqp/DisbPWtXqqwoXjUiypOetmoUeZ4sMWjl48UvcS+ro 8hAnGuFkBqDfkEvNWdf74GwJq9o/Jm0QtA/Btd0Isx0WZslghLxXVP/YIZx6ra5kZUIn jhViKs+oSOwDJqDlGj4zwsYV1ii0phGm5EfjG3jM3TNJCEY2N5VANZJ+FDgcx8Wmz+0m tQQBDkKKOniTUKqxdLMwwT9p7kCCVdDS4qaD3DR//QHcclSTY3ban64y0R7Odsfk6Lxv /ueA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1772837516; x=1773442316; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=dYRb/0buUb3p9AxqkSCYQnLtLdpN/kPE8ow1mG/Cv8E=; b=lENTVKBV27wi12kdOkPSBdRl8ONCSPaqkr5gQIghSR5+3VLPeMwLXjwfMqdyfc6Cst xGNtAVxlnaJK1FH16SamyB8UPAJdlSQ0T6ORF8GgloD43MtZSlgl8ToQBC1ckpqzNxi4 maDZ+SgDmQYeBw+XKuKdYDd8pyGfcv6EwPkVeUKLkQggW/AzGJjzTt6lXGT04KLTxmYt PYMQwVUvstBFfa592nfjN60uFPmldxaR8kwvq1quh86viFAvMp+0k9T9V5QdoqCZfwqy wc9Az96Nke1XM3ggdeiMowuDlyjUm6/ytd1Xfsf57pnXQ6S7Fkn3l9DCiIm3iGFAKFrd 0EVA== X-Forwarded-Encrypted: i=1; AJvYcCV7Rv8TChh3X+lUqwNmZMeC4YXyxJk/WrBrzEaMGUw7hjkjXRH3q/hI03MEOKScAl5z4WYeUSCyaZP3ppE=@vger.kernel.org X-Gm-Message-State: AOJu0Yzn/MDZBBzyhemB7gZFOgP9b87pvTy6rkcyjVlTZKfVI1gbuWXY JE8uiCfBthGBEXGKGqkhebRfN2DhZGq2MUoe93XITQTBsx+Iq9PSi/Jp X-Gm-Gg: ATEYQzzs9gmezKkZgj+9NtfMgIKJ3jbdFQ1VN5IxW3kvaatK6D0PgbDsGFG0Q/EJngV lrV5YD/tdFMJ7koC/ltX3XoU7yrHfDSYZf9CRf6/jwOgh5x5V7PTTy7xl1xvaE7mSvn+RBlPW8O 3WDXPsyuRZY17zLoCcPRJnQ5a8r1hSPW94Mv2sXR0Wj6aXBOLY5I3OyLl8vE++C/+xdkC71ioYm jdRAjHLqntfOTptmXvwdyHnzp/V12bUoVwLoiNHhKdWfdcRgFMTeTIyYWQIHtV7uI8QICRSBxDl RZOhjpz7koHxg5zb0l4C8t2hpnH82WD0NtW1QuhTDykylEcSYXPPHyZuTa5Evfc/OVgR0OUC46k 7FJijsTZ3drjUx47nAjo2/i5ttvd0CbeajeZFMMwv3P2C7oqGEpYPZA9YmUMjzK2/sHfzvO0XFU 6rovZoPMuEVFkps8R8L7HDSkUzqm44qxHV+DnbOo9SwXARWW+NtuB/WKzkPuv1Ys5Fr//ZclvSu N41zlZqpjIt X-Received: by 2002:a05:600c:c05a:b0:483:129e:b573 with SMTP id 5b1f17b1804b1-4852696b8d7mr49188385e9.18.1772837515834; Fri, 06 Mar 2026 14:51:55 -0800 (PST) Received: from snowdrop.snailnet.com (82-69-66-36.dsl.in-addr.zen.co.uk. [82.69.66.36]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-439dae57c05sm6160845f8f.39.2026.03.06.14.51.55 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 06 Mar 2026 14:51:55 -0800 (PST) From: david.laight.linux@gmail.com To: Waiman Long , Peter Zijlstra , Ingo Molnar , Will Deacon , Boqun Feng , linux-kernel@vger.kernel.org, Linus Torvalds , Yafang Shao , Steven Rostedt Cc: David Laight Subject: [PATCH v3 next 2/5] Optimise vcpu_is_preempted() check Date: Fri, 6 Mar 2026 22:51:47 +0000 Message-Id: <20260306225150.93178-3-david.laight.linux@gmail.com> X-Mailer: git-send-email 2.39.5 In-Reply-To: <20260306225150.93178-1-david.laight.linux@gmail.com> References: <20260306225150.93178-1-david.laight.linux@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: David Laight The vcpu_is_preempted() test stops osq_lock() spinning if a virtual CPU is no longer running. Although patched out for bare-metal, the code still needs the CPU number. Reading this from 'prev->cpu' is a pretty much guaranteed have a cache miss when osq_unlock() is waking up the next cpu. Instead save 'prev->cpu' in 'node->prev_cpu' and use that value instead. Update in the osq_lock() 'unqueue' path when 'node->prev' is changed. Signed-off-by: David Laight --- kernel/locking/osq_lock.c | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/kernel/locking/osq_lock.c b/kernel/locking/osq_lock.c index 96c6094157b5..0e1c7d11b6c0 100644 --- a/kernel/locking/osq_lock.c +++ b/kernel/locking/osq_lock.c @@ -16,6 +16,7 @@ struct optimistic_spin_node { struct optimistic_spin_node *next, *prev; int locked; /* 1 if lock acquired */ int cpu; /* encoded CPU # + 1 value */ + int prev_cpu; /* encoded CPU # + 1 value */ }; =20 static DEFINE_PER_CPU_SHARED_ALIGNED(struct optimistic_spin_node, osq_node= ); @@ -29,9 +30,9 @@ static inline int encode_cpu(int cpu_nr) return cpu_nr + 1; } =20 -static inline int node_cpu(struct optimistic_spin_node *node) +static inline int prev_cpu_nr(struct optimistic_spin_node *node) { - return node->cpu - 1; + return READ_ONCE(node->prev_cpu) - 1; } =20 static inline struct optimistic_spin_node *decode_cpu(int encoded_cpu_val) @@ -110,6 +111,7 @@ bool osq_lock(struct optimistic_spin_queue *lock) if (old =3D=3D OSQ_UNLOCKED_VAL) return true; =20 + WRITE_ONCE(node->prev_cpu, old); prev =3D decode_cpu(old); node->prev =3D prev; node->locked =3D 0; @@ -144,7 +146,7 @@ bool osq_lock(struct optimistic_spin_queue *lock) * polling, be careful. */ if (smp_cond_load_relaxed(&node->locked, VAL || need_resched() || - vcpu_is_preempted(node_cpu(node->prev)))) + vcpu_is_preempted(prev_cpu_nr(node)))) return true; =20 /* unqueue */ @@ -201,6 +203,7 @@ bool osq_lock(struct optimistic_spin_queue *lock) * it will wait in Step-A. */ =20 + WRITE_ONCE(next->prev_cpu, prev->cpu); WRITE_ONCE(next->prev, prev); WRITE_ONCE(prev->next, next); =20 --=20 2.39.5 From nobody Tue Apr 7 07:29:15 2026 Received: from mail-wr1-f43.google.com (mail-wr1-f43.google.com [209.85.221.43]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id DEDD135AC2D for ; Fri, 6 Mar 2026 22:51:57 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.221.43 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1772837519; cv=none; b=kih6dgipIHYnKuuHKrfulkVUpICn/vYYMR7C+uVfw7R4y0ZREEc4yQCjYJisnlJd5twhcOqNqcrvkLr46OIT/kzzLbheejAC6Mpz70SlW+0OruStNqkfeZ3sjIS40sRTPyys67a0+mRcYEtrEg1TW2ONV7+3gdBB1vSpGZr7TqY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1772837519; c=relaxed/simple; bh=kBmrmMgLNdyt4E9Zugo/Q64IQu7gAlgFQJUnp65MRdI=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=XtKc9AGKlTOTxEXHWYJ4ejsC3iowquhuYc6SbHtcBZFr70jFe4Koj3HZhgGlLkYLETMq78qgrFXog+FN5SOC5wU03QCy3BCoOyV56UGPQeCuAYY8KFcDpXjVGXAVpuK3DehIfm87/CXgrnIvw3Yxf4d16IlPoTDac0VTjBaMEng= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=bdN2szU3; arc=none smtp.client-ip=209.85.221.43 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="bdN2szU3" Received: by mail-wr1-f43.google.com with SMTP id ffacd0b85a97d-439b790af67so3759071f8f.0 for ; Fri, 06 Mar 2026 14:51:57 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1772837516; x=1773442316; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=XC9jtm4WiIgFtvh7yHCK73pPr3f6g+wj+j5SOjMROmk=; b=bdN2szU3PpsLfs4rXfy8WRIeN+xFNwcEu/x/OdAEUG6GDRA2loVVBo3KKC5SmK8P+e Z4HfRilH60zECKfJo7VB/KLqjcay9+bL+3kS8Jvi411905SJMst2Z74E6mjMDhGZhD3J knw854LFsQhqLMiI5cYEcP9NQe5fR43PRvSpGP06aFma0zv8fY4/IY5jAU2gkXXAQgzY nJw77VSt+TdvNU8z0E/A7kXqlMFUvlZkUpR+pdroHmF2AgbjY4JlCu39uA5zRcxszUG2 UzADUN/sz9Q7jjeFtJ2LNBl4AKMDzkWpM8CFtEDvVJsh2M1QzrGsXCqbXpLXxuiknfRe LCUg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1772837516; x=1773442316; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=XC9jtm4WiIgFtvh7yHCK73pPr3f6g+wj+j5SOjMROmk=; b=VvZSkZ9bnSV0U3ggh+wMiDcFfPBB7AimdW4afc5WMBFqkwTEOxHiQ16DIuYGWEnc2x 4syjlbKQIirFDA+OfLIF48d8erDhLsSw8YkLRwQXdcOQVXnjugcO9uazi8NaJeTRFz3Q 0YrpPaMVKYJaAlbvm60Cw65ZItIFD4TtO7H2+pdCN3lQkG4MUFdVQkYvr3e1OYMqkvsZ HzZSpmT+ThDbVk3d9H1hP6G4iyhSriFe5kK0DbNUE1ByTDGBI/uRF/vogU1EvHAqRgMV B13+eBGJqNh1a9CGyDjRL1IIjvRdr4n12PBwLC3vnClPe9RTVf59Ua3xpt5wgvE5zP73 2/Hw== X-Forwarded-Encrypted: i=1; AJvYcCVjhAjhZZ1W3rjhBzn+8wOmY85MXS4LnWc0hYzD4USuITIv9BSyxguDG8EkO1o3KklVGZROsLlkixuaJSo=@vger.kernel.org X-Gm-Message-State: AOJu0YyHeIHabgzaNCRx2QgaNvQZGatcbQg2No2Xt/YCdtXnHrN84dFo 3l22KgsBZe91rmZdMORlwJW2GvwNIyQMz26lSz5wZF6ssSYWmUtMOGAB X-Gm-Gg: ATEYQzxxaJwUhx0jafgfM9XfmFStGLH5VOYfAVPsvYW1Q+QI4vqGncolaKARdUytWwG f4npUAYkbd2w65opoFWWO8ycoGUxxrwZ7fRilbuHxliN6PqJ9uRLWpxk4dvqfMfI5RyBJHGX5wS xn2M+lRKGeVDw55tz1WemSGTz+QZBDScJomFR2Ryght5GXCbvY0Z4Za/HetojVwLN4wrtdJOhlT sLMonVh837Jm7eOrB0vqGZSt/ifBLb2OMApfQ+mEv17WqSBtSg1gvdQzYbVER7SdH+Jh+LgJxuW N5VGqBkN6/Anq4MpLoG34l5LHC8RZs8Q7l+/Am51yvpo3BzanvTu7aNHa0LmpZd50X2vcXRwSa5 AEZEuWGUebRUWoD5lHVL7E+6ik15D5Wixx2+DZFy5w2vD9yIxxOVDNFsdAN/wfaLP97bjBPONSb rYMsRfniP9FSjyTqC55nX3BYEJAe9OqNPPCenE7lkPcpUIp+m27DQWQj1wzqT1UDbJ1aDqXoMqy 5TpFLBtliOh X-Received: by 2002:a05:600c:1e88:b0:479:13e9:3d64 with SMTP id 5b1f17b1804b1-48526759d12mr71041765e9.15.1772837516297; Fri, 06 Mar 2026 14:51:56 -0800 (PST) Received: from snowdrop.snailnet.com (82-69-66-36.dsl.in-addr.zen.co.uk. [82.69.66.36]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-439dae57c05sm6160845f8f.39.2026.03.06.14.51.55 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 06 Mar 2026 14:51:56 -0800 (PST) From: david.laight.linux@gmail.com To: Waiman Long , Peter Zijlstra , Ingo Molnar , Will Deacon , Boqun Feng , linux-kernel@vger.kernel.org, Linus Torvalds , Yafang Shao , Steven Rostedt Cc: David Laight Subject: [PATCH v3 next 3/5] Use node->prev_cpu instead of saving node->prev Date: Fri, 6 Mar 2026 22:51:48 +0000 Message-Id: <20260306225150.93178-4-david.laight.linux@gmail.com> X-Mailer: git-send-email 2.39.5 In-Reply-To: <20260306225150.93178-1-david.laight.linux@gmail.com> References: <20260306225150.93178-1-david.laight.linux@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: David Laight node->prev is only used to update 'prev' in the unlikely case of concurrent unqueues. The new 'prev' pointer can be obtained from prev_cpu. node->cpu (or more particularly) prev->cpu is only used for the osq_wait_next() call in the unqueue path. Normally this is exactly the value that the initial xchg() read from lock->tail (used to obtain 'prev'), but can get updated by concurrent unqueues. Both the 'prev' and 'cpu' members of optimistic_spin_node are now unused and can be deleted. Signed-off-by: David Laight --- kernel/locking/osq_lock.c | 31 ++++++++++++++----------------- 1 file changed, 14 insertions(+), 17 deletions(-) diff --git a/kernel/locking/osq_lock.c b/kernel/locking/osq_lock.c index 0e1c7d11b6c0..5dd7e08d4fda 100644 --- a/kernel/locking/osq_lock.c +++ b/kernel/locking/osq_lock.c @@ -13,9 +13,8 @@ */ =20 struct optimistic_spin_node { - struct optimistic_spin_node *next, *prev; + struct optimistic_spin_node *next; int locked; /* 1 if lock acquired */ - int cpu; /* encoded CPU # + 1 value */ int prev_cpu; /* encoded CPU # + 1 value */ }; =20 @@ -96,10 +95,9 @@ bool osq_lock(struct optimistic_spin_queue *lock) struct optimistic_spin_node *node =3D this_cpu_ptr(&osq_node); struct optimistic_spin_node *prev, *next; int curr =3D encode_cpu(smp_processor_id()); - int old; + int prev_cpu; =20 node->next =3D NULL; - node->cpu =3D curr; =20 /* * We need both ACQUIRE (pairs with corresponding RELEASE in @@ -107,23 +105,22 @@ bool osq_lock(struct optimistic_spin_queue *lock) * the node fields we just initialised) semantics when updating * the lock tail. */ - old =3D atomic_xchg(&lock->tail, curr); - if (old =3D=3D OSQ_UNLOCKED_VAL) + prev_cpu =3D atomic_xchg(&lock->tail, curr); + if (prev_cpu =3D=3D OSQ_UNLOCKED_VAL) return true; =20 - WRITE_ONCE(node->prev_cpu, old); - prev =3D decode_cpu(old); - node->prev =3D prev; + WRITE_ONCE(node->prev_cpu, prev_cpu); + prev =3D decode_cpu(prev_cpu); node->locked =3D 0; =20 /* * osq_lock() unqueue * - * node->prev =3D prev osq_wait_next() + * node->prev_cpu =3D prev_cpu osq_wait_next() * WMB MB - * prev->next =3D node next->prev =3D prev // unqueue-C + * prev->next =3D node next->prev_cpu =3D prev_cpu // unqueue-C * - * Here 'node->prev' and 'next->prev' are the same variable and we need + * Here 'node->prev_cpu' and 'next->prev_cpu' are the same variable and w= e need * to ensure these stores happen in-order to avoid corrupting the list. */ smp_wmb(); @@ -179,9 +176,10 @@ bool osq_lock(struct optimistic_spin_queue *lock) =20 /* * Or we race against a concurrent unqueue()'s step-B, in which - * case its step-C will write us a new @node->prev pointer. + * case its step-C will write us a new @node->prev_cpu value. */ - prev =3D READ_ONCE(node->prev); + prev_cpu =3D READ_ONCE(node->prev_cpu); + prev =3D decode_cpu(prev_cpu); } =20 /* @@ -191,7 +189,7 @@ bool osq_lock(struct optimistic_spin_queue *lock) * back to @prev. */ =20 - next =3D osq_wait_next(lock, node, prev->cpu); + next =3D osq_wait_next(lock, node, prev_cpu); if (!next) return false; =20 @@ -203,8 +201,7 @@ bool osq_lock(struct optimistic_spin_queue *lock) * it will wait in Step-A. */ =20 - WRITE_ONCE(next->prev_cpu, prev->cpu); - WRITE_ONCE(next->prev, prev); + WRITE_ONCE(next->prev_cpu, prev_cpu); WRITE_ONCE(prev->next, next); =20 return false; --=20 2.39.5 From nobody Tue Apr 7 07:29:15 2026 Received: from mail-wr1-f44.google.com (mail-wr1-f44.google.com [209.85.221.44]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9833435F17D for ; Fri, 6 Mar 2026 22:51:58 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.221.44 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1772837520; cv=none; b=CW5JuD73wumxQI3CJ13vH/7SF5hVILhCWf7X1D8zTMDMB4f4v7NUMLpy1NUVCvNksZVBskLbofPS7WyNe44jS3e9eHFafN03Dqgej3jkiNAOtGw9fU/5qvFWnlj3ZiLrhNsAhqGa1WYAtN+mYFMHjdgqEsHAqQUGhOjus6rWNxw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1772837520; c=relaxed/simple; bh=2wHXZAdPDeWXiT3dF/iJP5pNHB+KvzI+EJ5gqZu10Ik=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=eLwcDC5MXgnpkrrEJ9nXF+PWc0RPbPXlxFcj5P1TQTl3CIkLt+jH+MAd6taQ+SkgQnOreD+tv4K52NKLvFcg50pEfkpOWXNS+965UmsqQ4UFb34AvknNzep4dE/yFYyj3rk4mVA4NxdH94/ekg+A9Vz5oMow8HUdKK3mS6cxzXw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=OZVOSZ6Z; arc=none smtp.client-ip=209.85.221.44 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="OZVOSZ6Z" Received: by mail-wr1-f44.google.com with SMTP id ffacd0b85a97d-439b7c2788dso4405547f8f.1 for ; Fri, 06 Mar 2026 14:51:58 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1772837517; x=1773442317; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=FFLGRcb+7+DWSwwR0Q53uNLMV1PZ5s9OjseZXq2vX6g=; b=OZVOSZ6Z6IXQ6ytdwnP8vQoSGYRB5P27wyY7MBO1Lm2ttwSqIfYrwOaczxb1X7GjsW t2RzL54/GC8KMMR49DPRoX1fwbSJfkdLfq4aihlOeZcm1iVLsV2fwexRTiJ3pkVqWUWT 4RUDwUYOFxSBYEix/fkH9ZZNgt1lL9DMcC498mDsBfIjlZ8DK43gUtKtuKHET8H3rUA6 hmOsmHX5I56+a8U2SHwc3uTQ3kY9fTkk7hvbl4GXj2hXalaLaJQziOVEd8gFwOS3GgxV 5LG/CFlQOck3TXqzTho2Iz5YmNqEfR+V30sBazCEeRc8TfqdlQpq/UY6qj3dMhTqi5BK g7HQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1772837517; x=1773442317; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=FFLGRcb+7+DWSwwR0Q53uNLMV1PZ5s9OjseZXq2vX6g=; b=LutnCftfoujWLb1MrsdUJ2DV4DFKrzW1iJH9WslxM6UKi/Ka0BMzfzxsN0l1qRn2bd glXXpR7bY4LKQcm/rCGIvO1ilr7wzAhh1qvQtd/WuIzLZDc+WgH8wj4lA5ILrpP3IY2j FP9chvVlhDBo15A2w3BNH2XHa+c6Ifr1NXt7vJEilGYDk5yxhTKrPnuIRyFP1ONmzXyh 1uijl55Gr3zZosAwDESuJE/2nzMczIHQBMbmtnMbhQc92ogd6OkjNM/Wrt58Lg5GETvx tRWbqJfDlQj7QpwFz21gV+Gk1pbCGSAvJhhZ1UVjJ4k1ImDvKiv9ZY4EixNKayvEivEb z/1Q== X-Forwarded-Encrypted: i=1; AJvYcCV5CBj+8LhbHWTAyUPP7tjg52HAOSKU42QL58UjAkH6wgSnfkocY0OIK8KgbC/VpY13HTCMLF087GyuoE4=@vger.kernel.org X-Gm-Message-State: AOJu0Yzp0kp2ghOpp4CuPbaQt7RbNmt/L/N7PuZ8t43OqZOZ3ue7VUWU Se7tkVHMK/6svEYUSLXObSXFW5sygS3LtOz4sMOnI0NFrbYHV1GrQ2U7 X-Gm-Gg: ATEYQzwnV9N27Khi1Z3KmXAfXJEkQ61ClTet0URJJVD12n3i+RP2mhVaiLYyKnYh0c4 kj0WrKqLXEpIlJO7yCKoARUv+xaIJL30ksd8C/lt5aBfBtSbY0MT0cwoFbcdvBNvjK2ujZ8X6zS H/EhyiNe8bmLrl+WyZZQZI8ooaJlh3r3ICDXbSUWniZTR55Oi5KBJnc0kI+QqQDtUCEFt8f8IKa uh5tBT7wgDEXEehrcVDPtNzw3H8bdywWZAJhyCFU6UpRuzP7FnfjxXds7bTBvkSj3ZG4nyA8t0g i29Llq+FBC7m3KkPEnqxQeaLrETMys7+8J+tfxUX3h1S0Okye02ixX+NCtYjsRuRGtyt35il1Jb tfQBsffKuFdT6fTbt/b4WAKeIFtiFDsTmWIcGPMjxzDrYMYz0nuEOoMoWqowSt5YhiN4Nr+1H3C bjfYftAQBLb5/4+3dP9W6WZVIqGZVIqy2giMsEGqva1niXsrB/ag9aml/Sc1Qm+gYpqeeDWTpAv ovzLf/UIz5J X-Received: by 2002:a05:6000:230f:b0:439:d755:a894 with SMTP id ffacd0b85a97d-439da348b29mr6708277f8f.3.1772837516703; Fri, 06 Mar 2026 14:51:56 -0800 (PST) Received: from snowdrop.snailnet.com (82-69-66-36.dsl.in-addr.zen.co.uk. [82.69.66.36]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-439dae57c05sm6160845f8f.39.2026.03.06.14.51.56 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 06 Mar 2026 14:51:56 -0800 (PST) From: david.laight.linux@gmail.com To: Waiman Long , Peter Zijlstra , Ingo Molnar , Will Deacon , Boqun Feng , linux-kernel@vger.kernel.org, Linus Torvalds , Yafang Shao , Steven Rostedt Cc: David Laight Subject: [PATCH v3 next 4/5] Optimise decode_cpu() and per_cpu_ptr() Date: Fri, 6 Mar 2026 22:51:49 +0000 Message-Id: <20260306225150.93178-5-david.laight.linux@gmail.com> X-Mailer: git-send-email 2.39.5 In-Reply-To: <20260306225150.93178-1-david.laight.linux@gmail.com> References: <20260306225150.93178-1-david.laight.linux@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: David Laight Changing the 'cpu number' variables to 'unsigned int' generates slightly better code (and the values can never be negative). More specifically gcc knows that decrementing the 'encoded' value zeros the high 32bits (on sane 64bit architectures) so that it doesn't need to zero/sign extend the value to index __per_cpu_offset[]. Not massive but saves two instructions. Signed-off-by: David Laight --- Proposed by Linus. Part of a discussion from v1 about whether removing the offset would help. kernel/locking/osq_lock.c | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/kernel/locking/osq_lock.c b/kernel/locking/osq_lock.c index 5dd7e08d4fda..0619691e2756 100644 --- a/kernel/locking/osq_lock.c +++ b/kernel/locking/osq_lock.c @@ -15,7 +15,7 @@ struct optimistic_spin_node { struct optimistic_spin_node *next; int locked; /* 1 if lock acquired */ - int prev_cpu; /* encoded CPU # + 1 value */ + unsigned int prev_cpu; /* encoded CPU # + 1 value */ }; =20 static DEFINE_PER_CPU_SHARED_ALIGNED(struct optimistic_spin_node, osq_node= ); @@ -24,19 +24,19 @@ static DEFINE_PER_CPU_SHARED_ALIGNED(struct optimistic_= spin_node, osq_node); * We use the value 0 to represent "no CPU", thus the encoded value * will be the CPU number incremented by 1. */ -static inline int encode_cpu(int cpu_nr) +static inline unsigned int encode_cpu(unsigned int cpu_nr) { return cpu_nr + 1; } =20 -static inline int prev_cpu_nr(struct optimistic_spin_node *node) +static inline unsigned int prev_cpu_nr(struct optimistic_spin_node *node) { return READ_ONCE(node->prev_cpu) - 1; } =20 -static inline struct optimistic_spin_node *decode_cpu(int encoded_cpu_val) +static inline struct optimistic_spin_node *decode_cpu(unsigned int encoded= _cpu_val) { - int cpu_nr =3D encoded_cpu_val - 1; + unsigned int cpu_nr =3D encoded_cpu_val - 1; =20 return per_cpu_ptr(&osq_node, cpu_nr); } @@ -53,9 +53,9 @@ static inline struct optimistic_spin_node *decode_cpu(int= encoded_cpu_val) static inline struct optimistic_spin_node * osq_wait_next(struct optimistic_spin_queue *lock, struct optimistic_spin_node *node, - int old_cpu) + unsigned int old_cpu) { - int curr =3D encode_cpu(smp_processor_id()); + unsigned int curr =3D encode_cpu(smp_processor_id()); =20 for (;;) { if (atomic_read(&lock->tail) =3D=3D curr && @@ -94,8 +94,8 @@ bool osq_lock(struct optimistic_spin_queue *lock) { struct optimistic_spin_node *node =3D this_cpu_ptr(&osq_node); struct optimistic_spin_node *prev, *next; - int curr =3D encode_cpu(smp_processor_id()); - int prev_cpu; + unsigned int curr =3D encode_cpu(smp_processor_id()); + unsigned int prev_cpu; =20 node->next =3D NULL; =20 --=20 2.39.5 From nobody Tue Apr 7 07:29:15 2026 Received: from mail-wr1-f46.google.com (mail-wr1-f46.google.com [209.85.221.46]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id CAEE535F605 for ; Fri, 6 Mar 2026 22:51:58 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.221.46 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1772837520; cv=none; b=B2+Ph9xZumW1fFPfKGT0zRUqnR6A4/nJWrgY1rBH+jGuP0LILKno+cnc8K1CGf5kKJDJRToIRe61aXG3JE6kcXFyaqbIfCRxr7Y7+OXMnxIVpBxMQYQjna0rC4787FGn3Qo3tN0DLNSt+vK9/Crw02d5YP++gqnUTZRrODXIHVk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1772837520; c=relaxed/simple; bh=DJS65YAxdEKgmmrRmE62iYKU6tqUYfBx7qz154siKGk=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=qSLnzHEsCmFhP356S+RcS/Tt70uEp8V/S333zcra7gZfnaApf8MSlCs4nuQ0xMdhjVK7gtUpmWdGcyP0VloPaOGiNaJCZRJID4WUvpb3HI6gQ6Ci9iW2Ity2XaW1QbQcuvoPHU0castji8q75UczNyc1Wxp9f2nGVznvH1ZTBps= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=KLhn57eN; arc=none smtp.client-ip=209.85.221.46 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="KLhn57eN" Received: by mail-wr1-f46.google.com with SMTP id ffacd0b85a97d-439aeed8a5bso6843762f8f.3 for ; Fri, 06 Mar 2026 14:51:58 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1772837517; x=1773442317; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Vz6EDb2VJ3SyH7oyHxjL4QyEQVmIWY9QfoNC0OG/wKw=; b=KLhn57eNn6j3z/IX9BaNLB3LfBxR/kyac88PAoZ57Pq9ehKJmxXpQewtdtSe55Q5WC mJnoziIhNV+LBb/ZYsdlsCkPI/MIW51YS6M/73LjwVXBx66GeiSufBpH/FcweQWMtlIx sctQynb560UjlYz4SU6+LUijmPMi+bu+WpZ5cGh6JVjVAtWtlXPmWYK6iCFovp4jZl4G KIVEavtnz5pbPVT5o9lozV95Bi2j27PleEJf9f2OmVR2ZzLo5I6gwTAjbZHislkaSHv8 CUyf/uMVAd3S+/dIPkpif0huTQkpGCsBH6FnDCZtAtFLyvR6yA6+lM4x/9X65uJjHVwU TazA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1772837517; x=1773442317; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=Vz6EDb2VJ3SyH7oyHxjL4QyEQVmIWY9QfoNC0OG/wKw=; b=WqZJgVWFolGisDUeBToE5NfKBtYR0UvNK5K0UHif0QUnGOGgqZ4uuLfmEY6lk1ANFf mhMRmOQ1ov3VCzF8IdFw0355bU5Zm1LIK2rCnKbNJITq3SIDHbHFTNPqcvRmJjws2dqa +WZhP/ie91Eu86KF7BvMBfLZoiRNvJktxOM5+UawtYCSNNnwIpsplISkEPCBiBbxPvm1 byJsB1L6Rk/onb5w8NHokjIF9olYDYpqARgktBOaajlGBPpz6pPD+bT20nLu3vRBnlBH CnNNXXNgsmdUPxHGcNCQTh0MgU9a8kvwMrWl6Y3JFxH8znVgFDKarE8e2DZMySGzAOpK dMSQ== X-Forwarded-Encrypted: i=1; AJvYcCW3i/vXbjeu+q+MmIcKTHRpRAdB2TfCpJowPA5vJLuOBNjI5syug4YFLujZX+CI/FvJMrRbb2lPh6UPlY0=@vger.kernel.org X-Gm-Message-State: AOJu0YxZyet/rDhxBvAUIYyKWZVwMwZKlyhg4dKwMJg+6GFT5RBZ0Ut1 kIl+4VL6qwCOx0nsBuRbW6XXcEJkcb79IRfTLLXpwGD2BslGry2sg1G1yDEYJzBN X-Gm-Gg: ATEYQzxzwi30wRbfXxq0yDzkuV1nZJ0sePd28aP7kIfpTW7SdhIHMfAcvO9WJM3hDc+ WnLrxMCIxpQoPFty4/TBRSklK5eReW0gzmtWhxu2SDbt3zeF85Q5iYaeBK+YC6GMJR/9PWPwZpG lQL8d2zv870ySwESUcOfYj5wEBn2/Fm81CDAMspxit983X6M+e5+YToaHnj+r4vKS95M0iIFXoW cWhqkfDqfu8hd3o0ZKhDV0HPmeQjjwqxHtzUOZC4WeVT750x+RaBJaDcTI8vf4QIleR3VZzEBn8 xgWWHBkRQTo4767TKx5mycvmL2Amrb7YnxU+1b7XxokTKnx5nbeLU/or+Os7b3vh9poSt/QKN9C DrKpfIzsWOc8Y6TxzJxvQJ+HpH9ED0SfUmwbBPTRA+QsSfFCAVE71I13WeEvlXiZh1qlyHCgZD6 25vw0ijkhBZFOHs6IEQy0elSPgyiNKkIdArNhmLp5dMHPAp599aK6CRRd2pVCgSZdVhaOKFMmbb wGy+hVzsVDX X-Received: by 2002:a05:6000:290d:b0:439:b6ae:5d5f with SMTP id ffacd0b85a97d-439da882118mr6888653f8f.36.1772837517173; Fri, 06 Mar 2026 14:51:57 -0800 (PST) Received: from snowdrop.snailnet.com (82-69-66-36.dsl.in-addr.zen.co.uk. [82.69.66.36]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-439dae57c05sm6160845f8f.39.2026.03.06.14.51.56 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 06 Mar 2026 14:51:56 -0800 (PST) From: david.laight.linux@gmail.com To: Waiman Long , Peter Zijlstra , Ingo Molnar , Will Deacon , Boqun Feng , linux-kernel@vger.kernel.org, Linus Torvalds , Yafang Shao , Steven Rostedt Cc: David Laight Subject: [PATCH v3 next 5/5] Avoid writing to node->next in the osq_lock() fast path Date: Fri, 6 Mar 2026 22:51:50 +0000 Message-Id: <20260306225150.93178-6-david.laight.linux@gmail.com> X-Mailer: git-send-email 2.39.5 In-Reply-To: <20260306225150.93178-1-david.laight.linux@gmail.com> References: <20260306225150.93178-1-david.laight.linux@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: David Laight When osq_lock() returns false or osq_unlock() returns static analysis shows that node->next should always be NULL. This means that it isn't necessary to explicitly set it to NULL prior to atomic_xchg(&lock->tail, curr) on entry to osq_lock(). Defer determining the address of the CPU's 'node' until after the atomic_exchange() so that it isn't done in the uncontented path. Signed-off-by: David Laight --- kernel/locking/osq_lock.c | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/kernel/locking/osq_lock.c b/kernel/locking/osq_lock.c index 0619691e2756..3f0cfdf1cd0f 100644 --- a/kernel/locking/osq_lock.c +++ b/kernel/locking/osq_lock.c @@ -92,13 +92,10 @@ osq_wait_next(struct optimistic_spin_queue *lock, =20 bool osq_lock(struct optimistic_spin_queue *lock) { - struct optimistic_spin_node *node =3D this_cpu_ptr(&osq_node); - struct optimistic_spin_node *prev, *next; + struct optimistic_spin_node *node, *prev, *next; unsigned int curr =3D encode_cpu(smp_processor_id()); unsigned int prev_cpu; =20 - node->next =3D NULL; - /* * We need both ACQUIRE (pairs with corresponding RELEASE in * unlock() uncontended, or fastpath) and RELEASE (to publish @@ -109,6 +106,7 @@ bool osq_lock(struct optimistic_spin_queue *lock) if (prev_cpu =3D=3D OSQ_UNLOCKED_VAL) return true; =20 + node =3D this_cpu_ptr(&osq_node); WRITE_ONCE(node->prev_cpu, prev_cpu); prev =3D decode_cpu(prev_cpu); node->locked =3D 0; --=20 2.39.5