From nobody Thu Oct 9 00:37:50 2025 Received: from mail-yw1-f176.google.com (mail-yw1-f176.google.com [209.85.128.176]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5269A146A66 for ; Mon, 23 Jun 2025 00:00:19 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.176 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1750636821; cv=none; b=d1togWsEk8jrq9v1Ut8bj2KGXT9sQWkh0U5znRp+vBUk5TBno/cqa5D41Pxn6Uf3jLLKgJIPmkdc5jRlAfBe08e1TO6P6K6NeDKJr29L6nV2ZuXbn45UnKiT2fEGWuZX7daiJw6g7iE3xxrxLahJFlqIEzRVyXaqy+OkVePXArk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1750636821; c=relaxed/simple; bh=c9XzlPzNbHPZv5CZbjDoLXJvrIRlDE9rLXfwLl+VDZo=; h=From:To:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=ra383xnz+YfT8eF6LSlSHvfPAt1J8/Bmz9IzwUN321pZJGHWTIwr9hK7x2pSLjYg/jBaY8pia1Emo5zfANjE8mvQD6fA32pnce4ZE/+mvy4dMqQT+df82lRp5dZcDgqLt5I3vq8F/EnP8WgNSZlMdeVp4tflUxxM7U/3TflIqKY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=Pl8ewu27; arc=none smtp.client-ip=209.85.128.176 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="Pl8ewu27" Received: by mail-yw1-f176.google.com with SMTP id 00721157ae682-7086dcab64bso29336367b3.1 for ; Sun, 22 Jun 2025 17:00:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1750636818; x=1751241618; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=nytlBN0i+cuzwSh1IkdjpnTfFhTj3b2gPCYDP83uMJI=; b=Pl8ewu27zh3m4wfl18IR3cMfD7x0I7ZxY6No61tZNlWWNoF4XsTk1zOwnxCZ6YEsgk ZNlj3e+DZLpNyyVusLHzfvzITsuPWTKcWXwp6w3RyHk/3NmbpkDL79kB6bSMiyQDgkoG uGWg8ohpLBUImGufvfOyQwHUaSMn8JKWmL68iG3SajNxvBxca72ITijG9YfIS2i+gml+ Gy9ixIGymw8mdGmAWntvdRTF8rbPH/3LRSxABmoliLBiSwg1aiIfRhQmZFoEnRnAnsQA ENpwUP4HiI7xK33Afg5dCLRONKrNbhvqmC3bN2PzVRm1iKoAgB35Hi+Z3DVaHkk2QgTz as6A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1750636818; x=1751241618; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=nytlBN0i+cuzwSh1IkdjpnTfFhTj3b2gPCYDP83uMJI=; b=WVLTVSt87prwvcztLBqWsYx8M1gunNuJ1XdQqn7xTKImFQzv9qQ9l/+3qn6Y8c880g hdIrRHS/ex5Z8tWm9vQPV5v3gnHsvpLhjdMtkd2R/jEg5BrlZLo0ysKi3E7wdC0YULf7 klgcs0JXy5OrmEgOoCjGflzfJ0A1yvjUcaaLBYtjXkXcRG8xZfFjU93RS4FgXLVqzYca Y4Jza2JWLOLB3IEGYFTFspNV83m3YSfLWUSJJgq5SsmptQp8exYGioYUYEe0uWM3e9eD S3NfIUOr3r8jP7dCsXGEG8u00FOQyux7mKZHFEcKwQpjhsZCAdRGvGP+2g5tSxqSbUqQ 7LUw== X-Forwarded-Encrypted: i=1; AJvYcCUWzyGFmt5NhP0TYOXyYDOZ47hAemNGZxD9zupeM0D2OPIDWoyHWZE9uLiKyKNyLGiSzbtv7+kKjiKycOk=@vger.kernel.org X-Gm-Message-State: AOJu0YziULnnAZSEMPAYVtMjEkDcB6F4iLrZhHjwesvcHnmgWY8izMLv pcte3EgTdIgCRsUTGH298b/Ob1r/sbbx9KfYDA5hQLWnuDblQKPTkDgg X-Gm-Gg: ASbGnctAgdhm5ChawpakxxYhC6EpYGH7F+7S8SX8zpl24izq/heb133ZZejxh9FtRUg 8WIl5ifl6znGv2wc8kDOqtB68ot9x9oz0PbgpJpSe7kXoU1lbWNErQSSZC0nIGTayBQfd7+SWKi 2rWfX04oiKAr4bL9q10UuQR1tuVoeHpc+YpGrxvDrRDNm6jyopszcLIjF9QZmjiX9jQmeYsooZE 1h/SGAHSSReya4I3rgX5J+Lpo4MKAYkQ7C3IIj7e+caOOV0LKSeuDoWBfJ6TrCFeWn2OyWAg/xj hEr+dsZRsoIeM/DOjcS7Pb2Qdtmkq/T+rppIDEEnfAvX8NIn5F3PjvZVMGBWb2l+wH6kX+7d2hB vM1hANc2DXSMVi6L6KuOlLA== X-Google-Smtp-Source: AGHT+IEjndLSyL1o1cfH1lCo7hIJwKaxxNNUvv/OmcR02y4Gcge7cAXdLe9SYVgvp2ort+THaj+9/w== X-Received: by 2002:a05:690c:6ac2:b0:70f:8884:17af with SMTP id 00721157ae682-712c6386e26mr159251457b3.6.1750636817719; Sun, 22 Jun 2025 17:00:17 -0700 (PDT) Received: from localhost (c-73-224-175-84.hsd1.fl.comcast.net. [73.224.175.84]) by smtp.gmail.com with ESMTPSA id 00721157ae682-712c4c1d4basm13341037b3.110.2025.06.22.17.00.16 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 22 Jun 2025 17:00:17 -0700 (PDT) From: Yury Norov To: "Paul E. McKenney" , Neeraj Upadhyay , "Yury Norov [NVIDIA]" , Thomas Gleixner , Thorsten Blum , Zqiang , Mathieu Desnoyers , linux-kernel@vger.kernel.org Subject: [PATCH v2 1/4] smp: Improve locality in smp_call_function_any() Date: Sun, 22 Jun 2025 20:00:06 -0400 Message-ID: <20250623000010.10124-2-yury.norov@gmail.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20250623000010.10124-1-yury.norov@gmail.com> References: <20250623000010.10124-1-yury.norov@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: "Yury Norov [NVIDIA]" smp_call_function_any() tries to make a local call as it's the cheapest option, or switches to a CPU in the same node. If it's not possible, the algorithm gives up and searches for any CPU, in a numerical order. Instead, we can search for the best CPU based on NUMA locality, including 2nd nearest hop (a set of equidistant nodes), and higher. sched_numa_find_nth_cpu() does exactly that, and also helps to drop most of housekeeping code. Signed-off-by: Yury Norov [NVIDIA] --- kernel/smp.c | 19 +++---------------- 1 file changed, 3 insertions(+), 16 deletions(-) diff --git a/kernel/smp.c b/kernel/smp.c index 974f3a3962e8..7c8cfab0ce55 100644 --- a/kernel/smp.c +++ b/kernel/smp.c @@ -741,32 +741,19 @@ EXPORT_SYMBOL_GPL(smp_call_function_single_async); * * Selection preference: * 1) current cpu if in @mask - * 2) any cpu of current node if in @mask - * 3) any other online cpu in @mask + * 2) nearest cpu in @mask, based on NUMA topology */ int smp_call_function_any(const struct cpumask *mask, smp_call_func_t func, void *info, int wait) { unsigned int cpu; - const struct cpumask *nodemask; int ret; =20 /* Try for same CPU (cheapest) */ cpu =3D get_cpu(); - if (cpumask_test_cpu(cpu, mask)) - goto call; - - /* Try for same node. */ - nodemask =3D cpumask_of_node(cpu_to_node(cpu)); - for (cpu =3D cpumask_first_and(nodemask, mask); cpu < nr_cpu_ids; - cpu =3D cpumask_next_and(cpu, nodemask, mask)) { - if (cpu_online(cpu)) - goto call; - } + if (!cpumask_test_cpu(cpu, mask)) + cpu =3D sched_numa_find_nth_cpu(mask, 0, cpu_to_node(cpu)); =20 - /* Any online will do: smp_call_function_single handles nr_cpu_ids. */ - cpu =3D cpumask_any_and(mask, cpu_online_mask); -call: ret =3D smp_call_function_single(cpu, func, info, wait); put_cpu(); return ret; --=20 2.43.0 From nobody Thu Oct 9 00:37:50 2025 Received: from mail-yw1-f179.google.com (mail-yw1-f179.google.com [209.85.128.179]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 511321ACEDF for ; Mon, 23 Jun 2025 00:00:20 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.179 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1750636822; cv=none; b=BDVrBjoolsLifSl9nVGpH/cggfBaDjL4bVSu7PlxNZsJ9qIszVuPlsvlZ7uzF0jrrFomG+zZMNqoZzqe0emG7qeUSykTAycMBfplDgkNdKjLRchdt0zRJq6cM1Qa3qwfaqtKd3oXLexXbDgvuuRwbZ6HTLBmPxyfIt/MsaD0iYI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1750636822; c=relaxed/simple; bh=TMsZXcUywPujCsSRw4HH9su9Z9AEjDr4uzM4LVhCzAI=; h=From:To:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=OCK7QYYdKCeUbR2mR3JnoammlS5U1AJzWHUmzRzI6HloTK+vILldIaaTSRPeEHcJBR+5C2wXcms+VkmaOS/Lmh77nvmEpkvtyr5w6Rb1hpWvXdM/7MgJGB7WJapWlJoSi+cwDt9GKHOl/GcGsCOIUyJ/v/dglzq7BbzJOO7BVuY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=mz6o9ucT; arc=none smtp.client-ip=209.85.128.179 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="mz6o9ucT" Received: by mail-yw1-f179.google.com with SMTP id 00721157ae682-70e75f30452so25160237b3.2 for ; Sun, 22 Jun 2025 17:00:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1750636819; x=1751241619; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=oVQ8EMAdEccHxXCbbgyW+fp2hkeAhkZriGFFCYXHijo=; b=mz6o9ucTWiJzfddwgmqk61WlF2euiFnQEnICv4wr4208DcqsdzhbSLOGwiMcqLBSWu eguhcA/pEdFaV6NblqgvAjGbRnMF1JqpVrM092Czt/sSRcT/olK4v+dwqUXaWZ+H7vnM H8DNdc005w3RJKPkQotTeuY0yotAwXtkEXdrKiF9h+DKI8FamzTGCgps6qm0/pw1vzx9 Geeol4g3rEeziWmjbTFlRSkKVwR98hBQ/AyWNAdGRbCC4pWw+sYF1RzZi6YfUztcdBvs 2rVg457A2uuwG5iWuZ6eh79a4iWzi/xPRR5UJnnBYVla0DMW8DQVJbaUOqVbiTfHrVuT 587A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1750636819; x=1751241619; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=oVQ8EMAdEccHxXCbbgyW+fp2hkeAhkZriGFFCYXHijo=; b=e0A03rJGbvM7DJi4fd05Xa9+iCSzr2+EV/vcGnJGalerUqroB/XkLHhtLghBf5lTc4 GviQPcSA5wEn6puOvBrYRmuV9BkDqx3+RbySTNiyMS+XV1E4C/X3zERrtLETLEbtPRzr +j7bPuAeTgFsCV2EIYS8WnKvVZTGwIJLT1LSMHbHxksA71AuDM/eKGR1PK6gxDGQfxHY csYxmIjAYCZDMrN29iuYALSb8nidjRBecadJszwmQsrIqXy+5C9MvJ7VZNf/SySEeJFM 79CA05W5rhCl7vjK95bXTORuMJnEZ38Ly5ogxU2sg1u1zHMd5dmJIXwO/kcVpv1O16GH WXHg== X-Forwarded-Encrypted: i=1; AJvYcCU+0A2KnL0TA0YKQbJhcJ3cE5sUrvjWQrj9s6/n58rJjq55QsH5AHrlPGl1S3kAwn0Valpxto496gR9654=@vger.kernel.org X-Gm-Message-State: AOJu0YwT18dxkoIFjuXQulyPKcTd6nJk5L8nuNBLSlsb3coPOK0yzzaI hjZ5RdoIm5Q1hDRT2SN+iYCgdBS+0lchhHzqLQpcsPxGEoaTyuggN1cI X-Gm-Gg: ASbGncsQNA3+Ubhfl2XHdnk8Z/rULALH7RydQMlN+6kMkMjGkhE+hWOAQCRq+y1zvd6 rzR/G2gmSX/vmSuzRe/hG0Ju3CAjFNAlVBSuNZqKfJ1H+aWazpTPLqfudVYdm1QHi5BsjgyC/4e DKzTnlnRFT0cCmxoqoeslpJXHiYRALff7BYIbRHK4tMJEVC4mi13FOBnA1rdx4NkSSwLH5x/xo1 P9DPEhxo5PCV/p4mq/p5UbCafjZ76aImcV8ZXUa1SK6fNP7tztd4uN5nLZkVpeePazYu15ENEUX fj3DUCecN8mim/GQuUzs72UgzpoeKYCXH1b/Vpzsf91xJv+3lxb7PZ2zsnT4BCYgKZ/O5ZBmzGI OEDnogxKhm0hFNYOxrCra+A== X-Google-Smtp-Source: AGHT+IElvB3NcUAmY80qRbL6KgaIcTyyrp55/JzMAonUYocimfC2D1VCp3iAZRl9pWEIU2WnmZm8Qg== X-Received: by 2002:a05:690c:3502:b0:70c:bd93:4519 with SMTP id 00721157ae682-712c65175f2mr145911947b3.21.1750636819131; Sun, 22 Jun 2025 17:00:19 -0700 (PDT) Received: from localhost (c-73-224-175-84.hsd1.fl.comcast.net. [73.224.175.84]) by smtp.gmail.com with ESMTPSA id 00721157ae682-712c4b9725esm13591637b3.70.2025.06.22.17.00.18 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 22 Jun 2025 17:00:18 -0700 (PDT) From: Yury Norov To: "Paul E. McKenney" , Neeraj Upadhyay , "Yury Norov [NVIDIA]" , Thomas Gleixner , Thorsten Blum , Zqiang , Mathieu Desnoyers , linux-kernel@vger.kernel.org Subject: [PATCH v2 2/4] smp: Use cpumask_any_but() in smp_call_function_many_cond() Date: Sun, 22 Jun 2025 20:00:07 -0400 Message-ID: <20250623000010.10124-3-yury.norov@gmail.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20250623000010.10124-1-yury.norov@gmail.com> References: <20250623000010.10124-1-yury.norov@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: "Yury Norov [NVIDIA]" smp_call_function_many_cond() opencodes cpumask_any_but(). Signed-off-by: Yury Norov [NVIDIA] --- kernel/smp.c | 7 +------ 1 file changed, 1 insertion(+), 6 deletions(-) diff --git a/kernel/smp.c b/kernel/smp.c index 7c8cfab0ce55..5871acf3cd45 100644 --- a/kernel/smp.c +++ b/kernel/smp.c @@ -807,13 +807,8 @@ static void smp_call_function_many_cond(const struct c= pumask *mask, run_local =3D true; =20 /* Check if we need remote execution, i.e., any CPU excluding this one. */ - cpu =3D cpumask_first_and(mask, cpu_online_mask); - if (cpu =3D=3D this_cpu) - cpu =3D cpumask_next_and(cpu, mask, cpu_online_mask); - if (cpu < nr_cpu_ids) + if (cpumask_any_and_but(mask, cpu_online_mask, this_cpu) < nr_cpu_ids) { run_remote =3D true; - - if (run_remote) { cfd =3D this_cpu_ptr(&cfd_data); cpumask_and(cfd->cpumask, mask, cpu_online_mask); __cpumask_clear_cpu(this_cpu, cfd->cpumask); --=20 2.43.0 From nobody Thu Oct 9 00:37:50 2025 Received: from mail-yw1-f174.google.com (mail-yw1-f174.google.com [209.85.128.174]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8D78621C17D for ; Mon, 23 Jun 2025 00:00:21 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.174 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1750636824; cv=none; b=KPIdxAw6CcjsUfLZdOGG3m9MaxMjgm9BRFyCGki6WfxD4PlUIegG4mCppyVMbWqYH5Cf4R8hHyO860nBuWtyD5izhc5S5L8RY4gREYYmmrUdH8k7F+5tdnLZ4bqPzt4jl7ETxhZz6wHHv8QLczXGCSQMM8YGC/3CILB+cKjCTTI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1750636824; c=relaxed/simple; bh=imuK2T8drX+ZLlrEtBHjX3bcu+iaFBiL6jam/tfVMAA=; h=From:To:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=fvI5IDLbfQGxhMO1kZPwtaajZxGp4Sf+UtfnQewmBQlzRxQMRHVuf/NXrElVrhDyGlNzT8BZXwhzt6WTFRFIZ7E4Q8VkQuFK6eCn53sBmKAu7iFLwlGHxk6gUBR85JUoR8BB25uir0b5JsM0Jr8kDTD69GT0KUnQ1Yrlnym/x0Q= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=XO55MCT0; arc=none smtp.client-ip=209.85.128.174 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="XO55MCT0" Received: by mail-yw1-f174.google.com with SMTP id 00721157ae682-70e77831d68so33346667b3.2 for ; Sun, 22 Jun 2025 17:00:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1750636820; x=1751241620; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=N7UVaCS12Xf6Wz8HX/a2PTgWqADCA4GJZ/2LGMJLYoE=; b=XO55MCT0AK9b7DWSkC5YuKcAu92HoLxxtEzGmZRMBOthz4Se7NARK298LgN+dIcX8Q 4BssDGkdbSJdNw7eXq1MtxTCW6EiLUwQuZw1pqwGyvGkj4rtBjMv+DHsEEfmwrYRi9GM 027jGIZul/Nrtl62VtIcd9QByircYHBb+xwRKttjPUg0322OUwU03/sE/pHc0MZYV/J9 qH4pLuqyMRd+7BuR8ZLagx+3xKxGvE/YBxVzglJAGfA5xWxcgrp3uJ+7yh8VvwF0HJNu fVF4RI3J8HmpAYCnFF4s2Ia4XWukbHshuI49HaCZuohbPtnxFKUYKc8B9zpp25YPPZ97 Sckw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1750636820; x=1751241620; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=N7UVaCS12Xf6Wz8HX/a2PTgWqADCA4GJZ/2LGMJLYoE=; b=YEVgl9tZXwPXu7y+uqA96V0KudNed7tCTHN7nLQnI4wz367C1r5u8Q2oDwNX8RAGnj QYrAWeZG/lmhXJoa3H3q3IWxCULlof8ZaVvOit/8DSvLuTT6MIOSOyNCUEohUD57sjFw +GcFn4AlHpZw9f/VX99SLyWZfihZNcLGc+otedkiPQ3iA8llZQmRFOR0n5ejXDGV/Yh9 kbCrnTTo9tpOCLyUFcZAuyGC65xaMIfqAOS3fic+TRYCpedlAg3e0tY3kfti4H21954Q Q6zhQFiD5qbRURRPGLDhE9Su7poIK0OieOPBCrG9O80nkCd0Uc/Jzia0u0tcITpCgI4L or3Q== X-Forwarded-Encrypted: i=1; AJvYcCWhwoVj4WattKn3bmjbnK2oRX2UchAs4H3LOgVZu7E9YcI0B8BP6O2wqVsI75AGszlM3mTXCX/8e/oOxmk=@vger.kernel.org X-Gm-Message-State: AOJu0YwALwCDGd6EPmoRXxSFaMEJArJBYZK/TTN9zZ2W93h2Fs16Fv/x rBnL25w6meMLMWKmNxR7U1fOB3hTIS6gzjeAM2riAlbT4mMinjs98xjj X-Gm-Gg: ASbGnct8isgDqSocQzjyQZ0E/uo8XtumAGwBrw0tl6tAjjI9G4KQewxZVYrlUBfKhtr OLRqtGO8Kp7exIo6epUHZN8fVjIuElUuXDgRa1/aTPFJdbdCBKzc4bznEgp1KAzJPjcVSNkbazP kGQBC5uPDM+AryKymHrH7N82nkcVtu5cfTGPeKPYdK+Af2YdU/n3LJbmEQBJUQqSur4qBry2riY aLeEI27g1gsq0a96tiJSfcpXAQkprGe7ODlPjzLLQfmNKXmRMMEquSG8Pk9RQ5TIbXAoH9172i0 GJb9Mk9ecS5YnCGn1NP983RWCFpVHUylAFGPWmGllglo4TiBCrazWsJEEVxCAKR3Vaac55EYd3Y lp3pzImkR85TpoMZ7vrMEww== X-Google-Smtp-Source: AGHT+IHM7doYTAmYmivYFzoUuG7mrbbh5bv+OzjJWZsq+Z2PPMsk75/7913K9T2MqMjKzfHBIN6qHw== X-Received: by 2002:a05:690c:6e0d:b0:70e:70de:64f6 with SMTP id 00721157ae682-712c601b6c9mr154977777b3.0.1750636820510; Sun, 22 Jun 2025 17:00:20 -0700 (PDT) Received: from localhost (c-73-224-175-84.hsd1.fl.comcast.net. [73.224.175.84]) by smtp.gmail.com with ESMTPSA id 00721157ae682-712c4a2359csm13537517b3.38.2025.06.22.17.00.19 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 22 Jun 2025 17:00:19 -0700 (PDT) From: Yury Norov To: "Paul E. McKenney" , Neeraj Upadhyay , "Yury Norov [NVIDIA]" , Thomas Gleixner , Thorsten Blum , Zqiang , Mathieu Desnoyers , linux-kernel@vger.kernel.org Subject: [PATCH v2 3/4] smp: Don't wait for remote work done if not needed in smp_call_function_many_cond() Date: Sun, 22 Jun 2025 20:00:08 -0400 Message-ID: <20250623000010.10124-4-yury.norov@gmail.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20250623000010.10124-1-yury.norov@gmail.com> References: <20250623000010.10124-1-yury.norov@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: "Yury Norov [NVIDIA]" If we don't actually send any IPIs, there's no need to wait for a job completion. Signed-off-by: Yury Norov [NVIDIA] --- kernel/smp.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/kernel/smp.c b/kernel/smp.c index 5871acf3cd45..715190669e94 100644 --- a/kernel/smp.c +++ b/kernel/smp.c @@ -849,6 +849,8 @@ static void smp_call_function_many_cond(const struct cp= umask *mask, send_call_function_single_ipi(last_cpu); else if (likely(nr_cpus > 1)) send_call_function_ipi_mask(cfd->cpumask_ipi); + else + run_remote =3D false; } =20 if (run_local) { --=20 2.43.0 From nobody Thu Oct 9 00:37:50 2025 Received: from mail-yw1-f170.google.com (mail-yw1-f170.google.com [209.85.128.170]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1077C21D3EC for ; Mon, 23 Jun 2025 00:00:22 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.170 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1750636824; cv=none; b=lJIQwzNF2iSnKjnTY2428XSYs9159CtSLvUb/UInfZt3mDbKBhBfxwrsNVkAa3d8lDqZD+7HjYHUahpu7xrDJerqKK71snD+6ZQW5KyTx/WAWn0Y7GlJQ7rfN/x0e7zcs2h+kHCLM8DsjTKb1M0Wxq/7iioqq+5Bf3JzrzzoqkI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1750636824; c=relaxed/simple; bh=XNhTVp0zHBlF+0PsXbzMdH+Fm4DPs2k4Bhj5e2VKuYg=; h=From:To:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=KqtbaWo/zCoKt01h8hwRNMpaBIcX6/HQz7tVv9DAMUbOL3qVm5TPH3U7eW1hWXUv+smoZJGLcRAAP4Aa6aNRzJ60Wu/dHTE+cgi1VScCgeKvQRJQuX2a3p49DIq8qP8+W7mPdAYYai3Kdmia0XXdO7OuuBC7dl0u8msVmwid3uk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=Mze+4VuT; arc=none smtp.client-ip=209.85.128.170 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="Mze+4VuT" Received: by mail-yw1-f170.google.com with SMTP id 00721157ae682-70f862dbeaeso32656997b3.1 for ; Sun, 22 Jun 2025 17:00:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1750636822; x=1751241622; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=tTdIBCLwszMjDGDcQS0PiAhY3upxjYQq8cnaKLJOLs4=; b=Mze+4VuTozIEc3i8gMvzFVL9CvBFQcpQjEi4A839p8mFCDZJFuwveZi50aE8/1fEKf G5faF+6ZjcUebVOrftMDMzTITbDAtZBo2ZuYQCEJC+3slTxpi4xMNUd7BkF4RBbbstKF mHD2ciKx73PLw26IzXXNKAfa1kriXTodGw4JLHdKAqyQ2QekglMkU6W00Qz2rgbO3kt5 n6frnnxT5jkV1c/SkgpSdeonOu2KkY2dp1JIlZWBd063dGsCBUow70ZUXDiPjUpjB6PX YCo9bkXpcIXq5UJLqJZr42HFiD9SrRpCqAn46bEw9+zuh7aFcqRxgTw2zoxhAiKWegY/ Qh7g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1750636822; x=1751241622; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=tTdIBCLwszMjDGDcQS0PiAhY3upxjYQq8cnaKLJOLs4=; b=dERXb/Abc53SAEZwjiyp3xDB6+uDpGWLjfRaBsNqDL9cptTZKlA8uMO+6hL29e4Dui Y9KyncfwxoydD16/S7uNZkBtk0g+HdIAbuODLQySEHObHx3VzXOrWErsCv4EB5yEFvSZ S1GifJZ/nvOHTuG/YrXbqVORcoF5O/EFPI5kWuVOGAdw8DGr56ndLp7ulJ6lXBr8YSAu dKwXutaLp/rQjBM0Utn7BxYqRvaT6zc+lkXBpilQk1jvH7pPW8lOwdFiEx3mH5B3kLC0 IOtqkbf4XZDKMeBJ/+pYiY6V6jfLyRCtg/R4r0+x1SAoHXiCbKfURByeMbNNYnyC4ITQ 6E5g== X-Forwarded-Encrypted: i=1; AJvYcCXravfjvXzl/UUZ2PRmYGSkAxWqJRkeSYMqPfKPtwExJbcTLJW11HDaIu/BxRxHQgw0VAWUAnJPmcqDM7s=@vger.kernel.org X-Gm-Message-State: AOJu0YyJ1xeMGPzgaTzzndt6H5tchsOg8O43GSaPakXGNASfeMbVXs9A lh+ZDCr36/aQ91CXfYAyoC4SJz/gMbvghry6R8T9Cya7sqsVMTpqNFfp X-Gm-Gg: ASbGncumsahEVpsBZN2svwpFXzhirhCakcHbnsRckIKGPZ3E4rc5jI8xyFDm45USJCc rlALBrM3efV7WJY31f4Kxrk10vIo3SBwkYqscwFaVwdyPEZ9JbCUweEvCQthIn4Xo6hEOw5p2Vj u/9BtjX/0/bShP+P2w7PuQYXDCrGcv9f3oXWZGq8lnMXRyx0pxb2eKx5zxppHeyAYTmiTS8S+xO LWhDU1OpMwTbOQn5vYfUyCjwu0T3yWyG25SUZpAdr47s+c5zRhPxXaJ+s1quH5UJUs/caIL/o2D kmR/iGfEdxEmeS+qNBLSLhVPAKuJpE2AB9Y+G2Qk+6b+2oC4vHRHJVXkNNbzFVhpp90ui/NQtAA RRAYhD6kB3o+CkWwNMIqzmA== X-Google-Smtp-Source: AGHT+IE/FToaPD7e3stm/h5Nd5LdAGsbRpIXHVZILc+iD7zhtzS2H6pcFI5YM5xmu9Jobq1oSeuY7A== X-Received: by 2002:a05:690c:6213:b0:70d:ff2a:d686 with SMTP id 00721157ae682-712c6511072mr142592417b3.28.1750636821915; Sun, 22 Jun 2025 17:00:21 -0700 (PDT) Received: from localhost (c-73-224-175-84.hsd1.fl.comcast.net. [73.224.175.84]) by smtp.gmail.com with ESMTPSA id 00721157ae682-712c4a23909sm13531887b3.46.2025.06.22.17.00.21 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 22 Jun 2025 17:00:21 -0700 (PDT) From: Yury Norov To: "Paul E. McKenney" , Neeraj Upadhyay , "Yury Norov [NVIDIA]" , Thomas Gleixner , Thorsten Blum , Zqiang , Mathieu Desnoyers , linux-kernel@vger.kernel.org Subject: [PATCH v2 4/4] smp: Defer check for local execution in smp_call_function_many_cond() Date: Sun, 22 Jun 2025 20:00:09 -0400 Message-ID: <20250623000010.10124-5-yury.norov@gmail.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20250623000010.10124-1-yury.norov@gmail.com> References: <20250623000010.10124-1-yury.norov@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: "Yury Norov [NVIDIA]" Defer check for local execution to the actual place where it is needed, and save some stack on a useless local variable. Signed-off-by: Yury Norov [NVIDIA] --- kernel/smp.c | 10 +++------- 1 file changed, 3 insertions(+), 7 deletions(-) diff --git a/kernel/smp.c b/kernel/smp.c index 715190669e94..84561258cd22 100644 --- a/kernel/smp.c +++ b/kernel/smp.c @@ -779,7 +779,6 @@ static void smp_call_function_many_cond(const struct cp= umask *mask, bool wait =3D scf_flags & SCF_WAIT; int nr_cpus =3D 0; bool run_remote =3D false; - bool run_local =3D false; =20 lockdep_assert_preemption_disabled(); =20 @@ -801,11 +800,6 @@ static void smp_call_function_many_cond(const struct c= pumask *mask, */ WARN_ON_ONCE(!in_task()); =20 - /* Check if we need local execution. */ - if ((scf_flags & SCF_RUN_LOCAL) && cpumask_test_cpu(this_cpu, mask) && - (!cond_func || cond_func(this_cpu, info))) - run_local =3D true; - /* Check if we need remote execution, i.e., any CPU excluding this one. */ if (cpumask_any_and_but(mask, cpu_online_mask, this_cpu) < nr_cpu_ids) { run_remote =3D true; @@ -853,7 +847,9 @@ static void smp_call_function_many_cond(const struct cp= umask *mask, run_remote =3D false; } =20 - if (run_local) { + /* Check if we need local execution. */ + if ((scf_flags & SCF_RUN_LOCAL) && cpumask_test_cpu(this_cpu, mask) && + (!cond_func || cond_func(this_cpu, info))) { unsigned long flags; =20 local_irq_save(flags); --=20 2.43.0