From nobody Fri Dec 19 07:32:33 2025 Received: from mail-pl1-f181.google.com (mail-pl1-f181.google.com [209.85.214.181]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 63FC2217654 for ; Fri, 6 Jun 2025 20:27:39 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.181 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1749241660; cv=none; b=afZaoTV/r9xZWQN7ITi6E5PP9OWYCUPPE8SICfp+7fBOb3musya8OThbyY4uUZ/G151OuvvGQZE47lp8fpG4JSfq96Ny5Fr5uNXd8xF5vtJpIehrsJHzwga3S5eFGe5RvmKxZGheFjznf3CkdSfc57b4CVZU14zu3m3CyEtj5PQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1749241660; c=relaxed/simple; bh=c9XzlPzNbHPZv5CZbjDoLXJvrIRlDE9rLXfwLl+VDZo=; h=From:To:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=R/OPGjodoQ965fjbtPuj7zx+KPalvxcca0kW80nmVlepGzktCh4pJsMiFwgRe3mVr2IMJwPu591w2oPWnm2Pzc0L9uxC65KINOdNLW9SUaRAUW8PHIZINDBj96csMxH4rzyto+ARvakpGm84jh/OTJffrr1Glzhnc9WM8faieEc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=G/Kw52v6; arc=none smtp.client-ip=209.85.214.181 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="G/Kw52v6" Received: by mail-pl1-f181.google.com with SMTP id d9443c01a7336-22d95f0dda4so32468745ad.2 for ; Fri, 06 Jun 2025 13:27:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1749241659; x=1749846459; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=nytlBN0i+cuzwSh1IkdjpnTfFhTj3b2gPCYDP83uMJI=; b=G/Kw52v6iXOk3z3TgwNlKd3S5YBlH4A2mHrNu6SrtUFG/exjOEmkCaKoZAGh7F9CM2 SZ0heKK0EzhCjwjltVxRt8jQBD3xP1O0ghcySQJptFSLNGu3ne07WBvY/chHekm5nwe5 diO2VYsO1GfBMidMnQpUL/iFeyhhj5N1QqA7rcNSmkluyMXx5yTwo78vhLQdJWQUdpfi 8ef4Y2shuYfSu68n+wBy0dCWUuZr9OSNZKndsZKwG0FKyjTA2XFVFSV4XAFR6QYbyvOk M1b/uKQmtPOw1QlGTztoF5sl7DL6IXoo8PP9H/O+p3Km23mPgjXgO3I5X4oTvDcCWcgc 79Lw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1749241659; x=1749846459; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=nytlBN0i+cuzwSh1IkdjpnTfFhTj3b2gPCYDP83uMJI=; b=u03gdRLDE4qIwZTjtIuwsi6LEposBZhJo7qoWrg5MQ77g2GUC0K+acC1NRgzw4bLar r5XlRS4LccCOtKqKcXR0VWYgaXh2NFTDn73PLAD/DvfWwggueHo5Mduna21Z+8FqxCPa D3QBmmlAZh8Zc46VOI3+nzc5kB7M9hF6DYSSmk2EGnKHeukv0KHdjPycOMzwmSnacMvJ SaedlT3PUj5tJ1xu0F3mYJ5ngVqa4Xz2G4t9wat+8dllb8rxES4UHkcLI0gGcjsNFtyV ubzVxPDuZgqqkxlqeAmMHSsY+Fb545LzW7SWQATlok2jrpTDjGuNng1SymvahtTObE2v DNCQ== X-Forwarded-Encrypted: i=1; AJvYcCV4hql1sguXzTzQz9P45aGZ7Y2isQl+RxEdY/MGY0dOpxG95tT+ZV8lydkdawvSUljBFJ6F1qyzL6u8X2Y=@vger.kernel.org X-Gm-Message-State: AOJu0YxBoTyu82FhHif127LppW7RJXqGpR5z/WgtZoxkvPNhNJCvMyOQ bCqIabewixyxQxxnkzbvDC5cEirsddjafc1YBetn2TFJ9XVuviUvuM2feDJ5hg== X-Gm-Gg: ASbGncswomZY/fpm7Wt77H0d9CLXSHRuarkZhdlHrqz/L3Ht+KDQasNjYoE//WvBoky 9UmvkGq5THyWpVZ0mPxkYurHBMTXwvQFyALlvmUgDvCdXu1iztT2C4xMWLBgRDESRyaepgqGxfU XvG7HDdvGqqkSSN99iEj/BXreE/Fl/MGupSvoaK3o6AsHi8/J1L4iODOUDtQdG0Xy8iGyzvsolJ lkbG9NxCvxrMIGIY9QeTF6iCrlVLkxqbZU3qB+j68dklsOz6cA1/zgyBCAMZWnf0FsH4oBPBnZE fJAbVo3/sUZSb176Tz5qVo73WN4f2vzHnfwWrbuaClzPF0G5FovEJR3+LnXXgg== X-Google-Smtp-Source: AGHT+IFvSK04vs3WhWQHLT5eN87+FQg0p3ufEnJAhQcEDMjpkFCRH5jzoYPAxHi1IGln1wO8fWOUJg== X-Received: by 2002:a17:902:f54b:b0:235:5a9:976f with SMTP id d9443c01a7336-23601d14fe8mr71371145ad.24.1749241658698; Fri, 06 Jun 2025 13:27:38 -0700 (PDT) Received: from localhost ([216.228.127.129]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-236034109c4sm16331475ad.210.2025.06.06.13.27.37 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 06 Jun 2025 13:27:38 -0700 (PDT) From: Yury Norov To: "Paul E. McKenney" , Neeraj Upadhyay , "Yury Norov [NVIDIA]" , Thomas Gleixner , Thorsten Blum , Zqiang , Mathieu Desnoyers , linux-kernel@vger.kernel.org Subject: [PATCH 1/4] smp: Improve locality in smp_call_function_any() Date: Fri, 6 Jun 2025 16:27:28 -0400 Message-ID: <20250606202732.426551-2-yury.norov@gmail.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20250606202732.426551-1-yury.norov@gmail.com> References: <20250606202732.426551-1-yury.norov@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: "Yury Norov [NVIDIA]" smp_call_function_any() tries to make a local call as it's the cheapest option, or switches to a CPU in the same node. If it's not possible, the algorithm gives up and searches for any CPU, in a numerical order. Instead, we can search for the best CPU based on NUMA locality, including 2nd nearest hop (a set of equidistant nodes), and higher. sched_numa_find_nth_cpu() does exactly that, and also helps to drop most of housekeeping code. Signed-off-by: Yury Norov [NVIDIA] --- kernel/smp.c | 19 +++---------------- 1 file changed, 3 insertions(+), 16 deletions(-) diff --git a/kernel/smp.c b/kernel/smp.c index 974f3a3962e8..7c8cfab0ce55 100644 --- a/kernel/smp.c +++ b/kernel/smp.c @@ -741,32 +741,19 @@ EXPORT_SYMBOL_GPL(smp_call_function_single_async); * * Selection preference: * 1) current cpu if in @mask - * 2) any cpu of current node if in @mask - * 3) any other online cpu in @mask + * 2) nearest cpu in @mask, based on NUMA topology */ int smp_call_function_any(const struct cpumask *mask, smp_call_func_t func, void *info, int wait) { unsigned int cpu; - const struct cpumask *nodemask; int ret; =20 /* Try for same CPU (cheapest) */ cpu =3D get_cpu(); - if (cpumask_test_cpu(cpu, mask)) - goto call; - - /* Try for same node. */ - nodemask =3D cpumask_of_node(cpu_to_node(cpu)); - for (cpu =3D cpumask_first_and(nodemask, mask); cpu < nr_cpu_ids; - cpu =3D cpumask_next_and(cpu, nodemask, mask)) { - if (cpu_online(cpu)) - goto call; - } + if (!cpumask_test_cpu(cpu, mask)) + cpu =3D sched_numa_find_nth_cpu(mask, 0, cpu_to_node(cpu)); =20 - /* Any online will do: smp_call_function_single handles nr_cpu_ids. */ - cpu =3D cpumask_any_and(mask, cpu_online_mask); -call: ret =3D smp_call_function_single(cpu, func, info, wait); put_cpu(); return ret; --=20 2.43.0 From nobody Fri Dec 19 07:32:33 2025 Received: from mail-pl1-f171.google.com (mail-pl1-f171.google.com [209.85.214.171]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 579E221B91D for ; Fri, 6 Jun 2025 20:27:42 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.171 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1749241663; cv=none; b=HscPcgtidyVOdM8iV/v57caIhtKnuovsfmOOBNk/6p0+pJ+/i0bDnrcSBOUNA5GkE+kweuTG2gN8NnqRySWP0X3rmPavZ84ogn/HT/bKCIv7uMZ4R7gNzO0smtC4pJHnTC1OWyd56lUfK5AOxGac9QmQEynQ/UiTtAGgUruJnuI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1749241663; c=relaxed/simple; bh=2ExBq4btEL9f6gFJ611GKCHP0/rYKFM+FGG9C8i9iXE=; h=From:To:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=k4lyK/CgdtnWL8YojuLo1H8Zw/Abk94hTtztz0JR4TuLsTlp6Ey5aTm3Pmh28jMSDBzqGAyJ4sZHoCmdDjbLoiwaqLRUy6uN8XPjdUUO/ebz30Xen1saS6pyQcLJ7jjxQ7vQELo5gEHm3XV3EknxnpibU9uKuFyrbruDfFwvkZw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=hYSWzxJ4; arc=none smtp.client-ip=209.85.214.171 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="hYSWzxJ4" Received: by mail-pl1-f171.google.com with SMTP id d9443c01a7336-234fcadde3eso30481875ad.0 for ; Fri, 06 Jun 2025 13:27:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1749241661; x=1749846461; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=eDiZyKcaQebxUPimMBVk6ki6bnrdNDkZ3MzJQYGddWQ=; b=hYSWzxJ4ETB8eiCV+o/NrDsPcraajE+qo9TIAHrEfcT0k+FVsBHUw92I0HbFdnzSUu lI3z0mUEPaSc3krktnXjlJchWlKJzuXnT3jfOxNlXEKxa2damz0oQJQTR1HVtngCABs5 JOvp6PpVyEWWPHafS9MWgyU9knvpPTvjHdk54Db7nf8A5f+8qPjpd1zcjjXtohvcJjYo vsQJVl3B+ygDHOSQvFQD2iQ59qpnQNI0GJwU6uF8G75mv+j0HBkUKsPIxLySzwCSKd67 0ydlZuwETi39t8aC0/stF4OxGqvjVK1TGnq9IteoEdg7GaiHFw1baQsvJqJSLONv+afV LRfg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1749241661; x=1749846461; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=eDiZyKcaQebxUPimMBVk6ki6bnrdNDkZ3MzJQYGddWQ=; b=AkG5+q7S5svK512AMjyr1bj9FmuFjw//q0rSHQMgqkX7jkPgq2AYil3F7DLUuvvvuG Pj+mDD6vPtm3IPCK4xnPCwb8MD3+dgUNUg9pIiy7oE64mVkVZ2GP9e8QhtlW9VCMn8Ao FlYiQGbiTx+ZuV65doOxbPonj7urW79PJ/OJq+w38rEVzuEVcNKFe/szNIriis3a01K8 E6XK0ZKli8FryDkejP21z/mnPoyk0xKY2/YsCEh0tv1QxhPJw9XS42Y4opawFUVIsev3 pQDLBPWYoIvE0ScL2fqhB/gM/X7rxHPGkfKEgV0LjStlihfm/wZ3ZYXxe2Fpq5QSTYEp g6Bg== X-Forwarded-Encrypted: i=1; AJvYcCWzmFmjUBkASxmbPJYIk6GI6Z8i8ksgHsqd+9Bb24cjyxTXOCg1aKIR0Xd8gU6AP+7AaZP4rV7kY2woNtE=@vger.kernel.org X-Gm-Message-State: AOJu0Yx1QdOuM1FdaHfI9ekt6oLNBbdMLOZv9qfOIbJ7hBCCexH1JJw9 szijYr6R5aQbyfBn6iV0ZuU7r15Xj2QF1Z125r6wJBe2HUz+BzWKFVgw X-Gm-Gg: ASbGncvOxIejO7oQa3t8aIFRHrRsfoKv660xOYmn2TxybrB8iP9j/SfC4cI6ssuCz/N RHsDeWUjurHH+EmttfAZ+tGjE4lbXsx2eSSeVQSP80gE9E5pSWR1/OBfXlbdjDPtqZYJ/Gwib7C 9rLmMrUfK43uFJqSoBphr6n+5GC38677K8VRQch8rYwNM5Ux7dACAZ5ABPYf880k5goZBDFquQW Hm4zFNMm8+Sx5unKiD6QBfZUZ+pDRnfCoiFgRC3VQGGT8h9M+ZffFwgTt6ROLxgMbG1fMIpETa1 6F4HoyMPct6E1f8DLcY8mst1o5SfcsVujFl5QcSJOTiQ1CwQRUao6+LlXq7z4w== X-Google-Smtp-Source: AGHT+IFGsBBaUonibtpg5yxv+69bvm1J05wESm/g94ZM3rmsvrj1DBqeqU2SsZoo82VG/F9DsXx7vg== X-Received: by 2002:a17:903:22d0:b0:234:ed31:fca8 with SMTP id d9443c01a7336-23601d25bbfmr66393505ad.27.1749241661570; Fri, 06 Jun 2025 13:27:41 -0700 (PDT) Received: from localhost ([216.228.127.129]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-236032fcd08sm16411785ad.131.2025.06.06.13.27.40 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 06 Jun 2025 13:27:40 -0700 (PDT) From: Yury Norov To: "Paul E. McKenney" , Neeraj Upadhyay , "Yury Norov [NVIDIA]" , Thomas Gleixner , Thorsten Blum , Zqiang , Mathieu Desnoyers , linux-kernel@vger.kernel.org Subject: [PATCH 2/4] smp: Use cpumask_any_but() in smp_call_function_many_cond() Date: Fri, 6 Jun 2025 16:27:29 -0400 Message-ID: <20250606202732.426551-3-yury.norov@gmail.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20250606202732.426551-1-yury.norov@gmail.com> References: <20250606202732.426551-1-yury.norov@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: "Yury Norov [NVIDIA]" smp_call_function_many_cond() opencodes cpumask_any_but(). Fix it. Signed-off-by: Yury Norov [NVIDIA] --- kernel/smp.c | 7 +------ 1 file changed, 1 insertion(+), 6 deletions(-) diff --git a/kernel/smp.c b/kernel/smp.c index 7c8cfab0ce55..5871acf3cd45 100644 --- a/kernel/smp.c +++ b/kernel/smp.c @@ -807,13 +807,8 @@ static void smp_call_function_many_cond(const struct c= pumask *mask, run_local =3D true; =20 /* Check if we need remote execution, i.e., any CPU excluding this one. */ - cpu =3D cpumask_first_and(mask, cpu_online_mask); - if (cpu =3D=3D this_cpu) - cpu =3D cpumask_next_and(cpu, mask, cpu_online_mask); - if (cpu < nr_cpu_ids) + if (cpumask_any_and_but(mask, cpu_online_mask, this_cpu) < nr_cpu_ids) { run_remote =3D true; - - if (run_remote) { cfd =3D this_cpu_ptr(&cfd_data); cpumask_and(cfd->cpumask, mask, cpu_online_mask); __cpumask_clear_cpu(this_cpu, cfd->cpumask); --=20 2.43.0 From nobody Fri Dec 19 07:32:33 2025 Received: from mail-pf1-f174.google.com (mail-pf1-f174.google.com [209.85.210.174]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id F1B8021CC7D for ; Fri, 6 Jun 2025 20:27:44 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.174 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1749241666; cv=none; b=TxZ6zaGmJgULR6Xj6p1wVY0zvARuHnCN2D3MaO9rW9pQ4F7Pkc21VEuYBJKG8NlO2z63cbS8h8EOyqBdpMtr3PTQ6ABqNpMFmGXdKdIEswLqF5xb9Ce/ZJAPXn3M50cDSzG/H2wlgxYPl6cul6NbnbYm6dVLty0ecvAbN8xYOf0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1749241666; c=relaxed/simple; bh=imuK2T8drX+ZLlrEtBHjX3bcu+iaFBiL6jam/tfVMAA=; h=From:To:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=nmnhgm3z+InUkIJg5FxrG43WK3Saiyr0khVAdfW/P++AhdtjuXI9fuCvvZtWR7LtY9UDAaqRLYnteAuRQB9JxIzv29EboS3E0liJfiP7c4B+6Tkl/sdCD4pLVwAMpEpCsxuXGjjcjvcolvFNi7ls/1XYQNyGBj67d1dyUH1LNHs= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=hynTcdAT; arc=none smtp.client-ip=209.85.210.174 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="hynTcdAT" Received: by mail-pf1-f174.google.com with SMTP id d2e1a72fcca58-7426c44e014so2238361b3a.3 for ; Fri, 06 Jun 2025 13:27:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1749241664; x=1749846464; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=N7UVaCS12Xf6Wz8HX/a2PTgWqADCA4GJZ/2LGMJLYoE=; b=hynTcdATjDTZQznW5g+ehO2HQS0Ky5y5DC+CidUDvz71YHkR1a6I7HswOtku34wfss bAA7k6biIdoON2xtp/EAqb3WvZWVP2YucAwpo0pUjVr8I9r5VdkhgeqqX0dE1oHEvm19 bsqPP3nekJvi6ZP9joCUPZ0UJxODXIdKgM47Rc3KSALeDZAqz8ICzCNGI7gb8H3hNGDg AixiDkNhZgPwi4mgARPXJxvRnWBhaWEg3LGqUSs61q4Wl993XYa5a4YL/CI6Q+BMZKSE 2VPSEwhMDIBfggdztO//sAcz0j3R9n2nv+F0dCoH/Fg2bkBWHJvrw3oAUMN8QO8wN+xE WlEg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1749241664; x=1749846464; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=N7UVaCS12Xf6Wz8HX/a2PTgWqADCA4GJZ/2LGMJLYoE=; b=B2BTtQWqyYhJPDgKxQwu4qsgB0FYGEXk+RXM8gxhslkMeb3RhmsxaKwTu454yKpAhz vvbhUqeQ+ri4VL7cpJJqHKkImzG+EuLlZNCNsmBMQpwhRUiW12qG18ur9Zd6dOl2fpPP VKwBkLNzkeAc4wkbsAL6llpanCLvrfidsYOU3FiknXEiF39qW2NhOnOl+UBG0tEL9BkI LUcBi0IYOHFxnIkVZ+9IDqWePzxYXJA8941HEIyurG6UHgw4UqnLGyip5VAIv3OGAb0K juWOhtaWsE1dTsWK9HK1Y6dOirUpzeljMtr+RY2r2+Hl7RmVVRjvJqhOodoL99hVcK1V 4vXw== X-Forwarded-Encrypted: i=1; AJvYcCVA1D9n70nMm51LdKGfc1Bd8/lhB9fzvvrYWYmE/ml1I8oTPXenDn3wgd4K8zLVMd3x6wSS3aC3i1eEXNU=@vger.kernel.org X-Gm-Message-State: AOJu0YysaGSz6qaY7Ox71KRAVyHJCTLDqjGK8WmpSfLeu5/blJyaxooc rFgDwtPYUr201x65cvnD4CM0ZvLPrBe1Icoq7HtgH12DHetgJUwiamsI X-Gm-Gg: ASbGncuilrqy2x25K53IJIlm+rAxJcs48P2mnHpiOW7dsqin6D7MY/O+j15DlC7od3x KYNXCG0wqBl0YPPCKb8sWnligoDq+IoOKH5Ig27iMlXiAqusrPLKs8BmtoBsmnyuAL1Kd+0feRW w3Pm0QA9qwvcYr+/NTH7axx19qzlDzZ1c5k6XbzIkEI5a2vE8rNl1fIfjgE6/r67Nb7xWGZ2jaq crPrC4aBmzkqGjYij+8kcgGrrypR7RMjWsNjg1HOZHIxwsrveaVcEH3hNRROhyT9X+LmhG6xrZV jQFP0jociTXvum1HTPxVfymcWVfEH08GynvuW7xxrvL8bdLutYkBFsc/QHG2nA== X-Google-Smtp-Source: AGHT+IF5fgvUogrWQAOgI5ismWWURFnFZHwy7H4dYKsH+jMRP7wpoiSIgYyWxU6LgGdYb21++M09OQ== X-Received: by 2002:a05:6a00:194d:b0:740:b394:3ebd with SMTP id d2e1a72fcca58-74827e7faa7mr6654698b3a.7.1749241664288; Fri, 06 Jun 2025 13:27:44 -0700 (PDT) Received: from localhost ([216.228.127.129]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-7482b0847e9sm1673002b3a.87.2025.06.06.13.27.43 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 06 Jun 2025 13:27:43 -0700 (PDT) From: Yury Norov To: "Paul E. McKenney" , Neeraj Upadhyay , "Yury Norov [NVIDIA]" , Thomas Gleixner , Thorsten Blum , Zqiang , Mathieu Desnoyers , linux-kernel@vger.kernel.org Subject: [PATCH 3/4] smp: Don't wait until remote work done if not needed in smp_call_function_many_cond() Date: Fri, 6 Jun 2025 16:27:30 -0400 Message-ID: <20250606202732.426551-4-yury.norov@gmail.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20250606202732.426551-1-yury.norov@gmail.com> References: <20250606202732.426551-1-yury.norov@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: "Yury Norov [NVIDIA]" If we don't actually send any IPIs, there's no need to wait for a job completion. Signed-off-by: Yury Norov [NVIDIA] --- kernel/smp.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/kernel/smp.c b/kernel/smp.c index 5871acf3cd45..715190669e94 100644 --- a/kernel/smp.c +++ b/kernel/smp.c @@ -849,6 +849,8 @@ static void smp_call_function_many_cond(const struct cp= umask *mask, send_call_function_single_ipi(last_cpu); else if (likely(nr_cpus > 1)) send_call_function_ipi_mask(cfd->cpumask_ipi); + else + run_remote =3D false; } =20 if (run_local) { --=20 2.43.0 From nobody Fri Dec 19 07:32:33 2025 Received: from mail-pg1-f180.google.com (mail-pg1-f180.google.com [209.85.215.180]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 24301218827 for ; Fri, 6 Jun 2025 20:27:48 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.215.180 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1749241669; cv=none; b=vFrG9RddograYAVdTBwh4jprwAem1paP8vYxp6iyfnD3T3y8o3fFT4GvOsSL/m5TkrhATmrdVMmiSUIR4ueItujkgHZgqyPh5uvQXXq3IwztYY2KeT9Opz1vin8c4rHPnIesHqT6sfzsWxoDEyVyHTC+hHTY+uOf+Be19udMZ3Y= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1749241669; c=relaxed/simple; bh=DnmuKb4te8Dpg4MpVG4LfMdpxQAkeF48rj6DbMQ/hwU=; h=From:To:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=T2AewdOKevwThZMHVsYmdGujVsC/rPTty74lzhmID6gtX9FWfLorFpRtSEbCHog5o+LTHGXxQUmr07mLT/HECmk5Wae/U9YP24tq+0nwzveGCptLtC3j+x7qzelnkONEKMUx9p66DCUjZxgMQPhAtzr3OP1l1VKB+9wZBh3ZOu8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=F3qpBYls; arc=none smtp.client-ip=209.85.215.180 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="F3qpBYls" Received: by mail-pg1-f180.google.com with SMTP id 41be03b00d2f7-b2f645eba5dso594939a12.1 for ; Fri, 06 Jun 2025 13:27:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1749241667; x=1749846467; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=jc+5CHD3tDL3SNSorIV87nIdNFJZNqq5KxOMmRZzmKU=; b=F3qpBYls+ard4gndsWXDYnFZ1DrkE1ObHvIVRyw2954hA6XppJ7/QV47JC50YkQEyx BPtkZo+Gv+zRDJBy1UO454vxORVY7UbKAuym6ItJ197XpLcsyt9p3nE3bplPua4hqYfN MOAWi0Zaz+i6MHbWSy8FDYHV15jq7fT7htM66SJzVkWYY6xLi6wVs/D6mVR97p8oCYnv aryrIhRd0fsO50sEfU1ZtfxuL8tQQxh0jZZPrUEZLy4impssESIG/63OxPMtWiz8sCDK p5UFn4qzsy5Olk5s3Aw4MnoYLPgb00/MfgQEl2wLa0wrtFidnUXVoPwr7Pd0jpDF749K EAmg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1749241667; x=1749846467; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=jc+5CHD3tDL3SNSorIV87nIdNFJZNqq5KxOMmRZzmKU=; b=moyiVcwI2U44z3FR8NDa8Ux+paJcrN5fjwbJoxCukuxPwvdeFPbanSNl59rTNRUTe2 YCl8sqHKoQRXjqN0t4RiaS8iL+tFvX+GbPqaLfzh9NgkpTjqbvQIfij/aRMb34jyANMu nKFQ/xn1amwkS82SVmUF8KJiTKZ23Ju2FH+t6OL9HhbLO8Yu6ZOTd7qadFr7wB562Qx2 dIRShSQgeTvw9jkB2llQlz1tsO79MGPGyN+O8qMMrwtU5cwAGPW0GHWdAUZmVa6/+nOp fPx4cylZ6H/6Vz5LocqfAvdXnwMYb1Ke4lIngcaPBYm0f00zhq7k3qLtWKyYpfcay3wQ aEqA== X-Forwarded-Encrypted: i=1; AJvYcCUO7F3HOrfNP1x4wkVm4vhwz8EyCMjyhLp/TJ3vx5brIi2JbUIN3VQAlx5LjLx1q3/hGNqYsVNoWxZ+9Yg=@vger.kernel.org X-Gm-Message-State: AOJu0Yy2hn4Li40LzxuE+X6YLIi6u8MgsNCuMZzpi+AERVmOw04efNZB 5ef6zdbg8UGxozdOF4mxHtwJIfmoBT3pX9zuSxn2evL7cUN8HPPZsVyy X-Gm-Gg: ASbGncu0Xz8Q0xj6vNnEZLMZyQ5yv14TqDW6vPS0jFlGjH2yr6AOapZcOUYIU4SxoiN fX4bcf9K2tyyx7nIEGGtfaWA/LWiGNWtBk39nKY9ZQNT+88yLneB08YCRFV1lAjFdHevqJn75lJ gt7gyXbWXFTZvIy8y4e5ABlFRjiB1stqO42oB0/VQlG8xsKqEKUmXyg+JUu51G+eBK30jYDNOs3 J3o4l0VpvD/lMrqpj+1cBCEL4g9TIwTYxup1Bt5fUVzMtGl2eiFtTr8stIH+fVRbY80wTGg60Xo 3e2y+CjVmResRTQjFr41BilIm/7GRXbaG1tyAdhbb/ntKJYImTM= X-Google-Smtp-Source: AGHT+IHIl/2sIz0QzhUU71ZO2gTtcWGbLD5DKJni6Zi4pQpGhtBHoXECcyo381A/ocFKz4V9mKzNlA== X-Received: by 2002:a17:90b:520d:b0:311:b5ac:6f7d with SMTP id 98e67ed59e1d1-31349f4311emr6162152a91.6.1749241667286; Fri, 06 Jun 2025 13:27:47 -0700 (PDT) Received: from localhost ([216.228.127.129]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-31349fc374asm1702099a91.29.2025.06.06.13.27.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 06 Jun 2025 13:27:46 -0700 (PDT) From: Yury Norov To: "Paul E. McKenney" , Neeraj Upadhyay , "Yury Norov [NVIDIA]" , Thomas Gleixner , Thorsten Blum , Zqiang , Mathieu Desnoyers , linux-kernel@vger.kernel.org Subject: [PATCH 4/4] smp: Defer check for local execution in smp_call_function_many_cond() Date: Fri, 6 Jun 2025 16:27:31 -0400 Message-ID: <20250606202732.426551-5-yury.norov@gmail.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20250606202732.426551-1-yury.norov@gmail.com> References: <20250606202732.426551-1-yury.norov@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: "Yury Norov [NVIDIA]" Defer check for local execution to the actual place where it is needed, and save some stack on a useless local variable. Signed-off-by: Yury Norov [NVIDIA] --- kernel/smp.c | 10 +++------- 1 file changed, 3 insertions(+), 7 deletions(-) diff --git a/kernel/smp.c b/kernel/smp.c index 715190669e94..867f79689684 100644 --- a/kernel/smp.c +++ b/kernel/smp.c @@ -779,7 +779,6 @@ static void smp_call_function_many_cond(const struct cp= umask *mask, bool wait =3D scf_flags & SCF_WAIT; int nr_cpus =3D 0; bool run_remote =3D false; - bool run_local =3D false; =20 lockdep_assert_preemption_disabled(); =20 @@ -801,11 +800,6 @@ static void smp_call_function_many_cond(const struct c= pumask *mask, */ WARN_ON_ONCE(!in_task()); =20 - /* Check if we need local execution. */ - if ((scf_flags & SCF_RUN_LOCAL) && cpumask_test_cpu(this_cpu, mask) && - (!cond_func || cond_func(this_cpu, info))) - run_local =3D true; - /* Check if we need remote execution, i.e., any CPU excluding this one. */ if (cpumask_any_and_but(mask, cpu_online_mask, this_cpu) < nr_cpu_ids) { run_remote =3D true; @@ -853,7 +847,9 @@ static void smp_call_function_many_cond(const struct cp= umask *mask, run_remote =3D false; } =20 - if (run_local) { + /* Check if we need local execution. */ + if ((scf_flags & SCF_RUN_LOCAL) & cpumask_test_cpu(this_cpu, mask) && + (!cond_func || cond_func(this_cpu, info))) { unsigned long flags; =20 local_irq_save(flags); --=20 2.43.0