From nobody Mon Feb 9 12:07:30 2026 Received: from mail-yw1-f201.google.com (mail-yw1-f201.google.com [209.85.128.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 51E6D1A4F1E for ; Tue, 13 Aug 2024 18:28:06 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723573687; cv=none; b=Qk8HZuuGK20pEOB7geMCyEKjGEy9ErK1+jwW/p2Bq3iV7XKUbSoENhEwsWXzxs+s7MvveMS7ZsrxzBIKAFalSZi+FGPAwWrcTd2Ook6Eby92L3W/2hy+wZtZyBBEEtOw4VAfp1GqUed/wjwGTprm8PEX0yvSoiruigqs4t+QPVw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723573687; c=relaxed/simple; bh=j69ckPh7E3QD6AiHqrqUQtHhaCzw1plGFQ8OeVu17UQ=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=qFufTm1JhGNDK2WlNKi3GDZOKppDAN++MUp1zUpWd4VyVfkykwZARyfxtfG+QjGXV+xXUKJf/ag44rwCg/c6k0aPplw5m0uZV3JJzyHZpxBknm8aai0dihGzB56HOioMEi8vtN1cN83sfD+X4HSpyv9lfy4d61ukcuMRI6Lk1AE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--manojvishy.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=nwazf42l; arc=none smtp.client-ip=209.85.128.201 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--manojvishy.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="nwazf42l" Received: by mail-yw1-f201.google.com with SMTP id 00721157ae682-672bea19c63so139343027b3.2 for ; Tue, 13 Aug 2024 11:28:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1723573685; x=1724178485; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=TP961J72JDd3W/A5AwIhTA95rpBeW3mVlPbF5keUAfI=; b=nwazf42lrD42GpXfM+jnF7888q8ncFWYTMYMigAohgEzXGv4S9yAa0SXQ0eOm3P4bR BzO5S2StBzMBoQ+hjldsU3GyApif5J8S1m9mayupmZZEWq70wwPjlzbiQdWeO8UZhiak 2EL9qtMAftz5I6UMFGiY4lA3ZBAl6ZzrsW7XI6AQqND8DEhuctR72AFniEq3oncViDuq 0q/I+1LC1ijdleIBL+0c9vY9VKuFOo2lnf/JjbiEcMhoK6Zx+tPVPqO4UG/NSNzuXPvq XqfFcud5tH/eyIl6Hpoe7W5sY8SwIugdkoRCrDXjecapJefBwqGKa5NyDbPeuNE0D+D9 z3aQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723573685; x=1724178485; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=TP961J72JDd3W/A5AwIhTA95rpBeW3mVlPbF5keUAfI=; b=JcQWOD1HKPACRhwHMj+NMbde3RTuMJBsJ15z7XeawFFfjBdP/eks3EvW2MTHz4S+4b YTrRGejY1mYJGxENglNZk2p3sgTzb7pmvTfL1O0C3YWBXEEDSrKDzIHADQjHH94r9oa3 7RFpjwL7r/CbDSN45zfu+nby4Sg8+6PTAKN2r0AaDHoZbFFQCxe+BE/PCaGbgHhu37Nx 2QpRGOE3f2xsaybaUUfYnJjCpbAY98d/SusqlUCdMe0v3E0wdSYY8jC1umn5qY0t3VbJ URwuM92+eXocMIBwh4hrWJkupR9v3R3vdcGHmp5IzNcc8keTM43I0YcoS1oJy8ZEzuIo 6ECg== X-Forwarded-Encrypted: i=1; AJvYcCV57L6iY1Fq6lD2yMcJ1zvMqIpk4q8C3qJk93x7yVCPMI25+CsxJYmmWzP9HhS17xJ+O3MSm5LVSNU6j4szs5ZvvDPd8gf4QchhUJtd X-Gm-Message-State: AOJu0YxKjaK78uU3r7dNY7dgs6MInp9RrnzHL7skEy/tIB7hPTV0vTZj cTQccQFqgo25WYZ4SPmCGZHCuBPEM+R6s2H+7cNLfMnh8KzHJMmJGpzlGKUg9iknfyoBMBGsHiB UcQdqEx5CY20pNIo5wA== X-Google-Smtp-Source: AGHT+IEcUJ4nno820COmCVkqEY2grkP8CkJXPuWKQlxjBtfvY5Cd6Kury80B6+oOUU3OFR28ZBOqgQRdHfLwWACY X-Received: from manojvishy.c.googlers.com ([fda3:e722:ac3:cc00:20:ed76:c0a8:413f]) (user=manojvishy job=sendgmr) by 2002:a25:c5c3:0:b0:e0b:f69b:da30 with SMTP id 3f1490d57ef6-e1155b7c6cbmr11272276.9.1723573685319; Tue, 13 Aug 2024 11:28:05 -0700 (PDT) Date: Tue, 13 Aug 2024 18:27:45 +0000 In-Reply-To: <20240813182747.1770032-1-manojvishy@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240813182747.1770032-1-manojvishy@google.com> X-Mailer: git-send-email 2.46.0.76.ge559c4bf1a-goog Message-ID: <20240813182747.1770032-4-manojvishy@google.com> Subject: [PATCH v1 3/5] idpf: convert workqueues to unbound From: Manoj Vishwanathan To: Tony Nguyen , Przemek Kitszel , "David S. Miller" , Eric Dumazet , intel-wired-lan@lists.osuosl.org Cc: netdev@vger.kernel.org, linux-kernel@vger.kernel.org, google-lan-reviews@googlegroups.com, Marco Leogrande , Manoj Vishwanathan Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Marco Leogrande When a workqueue is created with `WQ_UNBOUND`, its work items are served by special worker-pools, whose host workers are not bound to any specific CPU. In the default configuration (i.e. when `queue_delayed_work` and friends do not specify which CPU to run the work item on), `WQ_UNBOUND` allows the work item to be executed on any CPU in the same node of the CPU it was enqueued on. While this solution potentially sacrifices locality, it avoids contention with other processes that might dominate the CPU time of the processor the work item was scheduled on. This is not just a theoretical problem: in b/317234476, a misconfigured process was hogging most of the time from CPU0, leaving less than 0.5% of its CPU time to the kworker. The IDPF workqueues that were using the kworker on CPU0 suffered large completion delays as a result, causing performance degradation, timeouts and eventual system crash. Tested: * I have also run a manual test to gauge the performance improvement. The test consists of an antagonist process (`./stress --cpu 2`) consuming as much of CPU 0 as possible. This process is run under `taskset 01` to bind it to CPU0, and its priority is changed with `chrt -pQ 9900 10000 ${pid}` and `renice -n -20 ${pid}` after start. Then, the IDPF driver is forced to prefer CPU0 by editing all calls to `queue_delayed_work`, `mod_delayed_work`, etc... to use CPU 0. Finally, `ktraces` for the workqueue events are collected. Without the current patch, the antagonist process can force arbitrary delays between `workqueue_queue_work` and `workqueue_execute_start`, that in my tests were as high as `30ms`. With the current patch applied, the workqueue can be migrated to another unloaded CPU in the same node, and, keeping everything else equal, the maximum delay I could see was `6us`. Signed-off-by: Marco Leogrande Signed-off-by: Manoj Vishwanathan --- drivers/net/ethernet/intel/idpf/idpf_main.c | 15 ++++++++++----- 1 file changed, 10 insertions(+), 5 deletions(-) diff --git a/drivers/net/ethernet/intel/idpf/idpf_main.c b/drivers/net/ethe= rnet/intel/idpf/idpf_main.c index db476b3314c8..dfd56fc5ff65 100644 --- a/drivers/net/ethernet/intel/idpf/idpf_main.c +++ b/drivers/net/ethernet/intel/idpf/idpf_main.c @@ -174,7 +174,8 @@ static int idpf_probe(struct pci_dev *pdev, const struc= t pci_device_id *ent) pci_set_master(pdev); pci_set_drvdata(pdev, adapter); =20 - adapter->init_wq =3D alloc_workqueue("%s-%s-init", 0, 0, + adapter->init_wq =3D alloc_workqueue("%s-%s-init", + WQ_UNBOUND | WQ_MEM_RECLAIM, 0, dev_driver_string(dev), dev_name(dev)); if (!adapter->init_wq) { @@ -183,7 +184,8 @@ static int idpf_probe(struct pci_dev *pdev, const struc= t pci_device_id *ent) goto err_free; } =20 - adapter->serv_wq =3D alloc_workqueue("%s-%s-service", 0, 0, + adapter->serv_wq =3D alloc_workqueue("%s-%s-service", + WQ_UNBOUND | WQ_MEM_RECLAIM, 0, dev_driver_string(dev), dev_name(dev)); if (!adapter->serv_wq) { @@ -192,7 +194,8 @@ static int idpf_probe(struct pci_dev *pdev, const struc= t pci_device_id *ent) goto err_serv_wq_alloc; } =20 - adapter->mbx_wq =3D alloc_workqueue("%s-%s-mbx", 0, 0, + adapter->mbx_wq =3D alloc_workqueue("%s-%s-mbx", + WQ_UNBOUND | WQ_MEM_RECLAIM, 0, dev_driver_string(dev), dev_name(dev)); if (!adapter->mbx_wq) { @@ -201,7 +204,8 @@ static int idpf_probe(struct pci_dev *pdev, const struc= t pci_device_id *ent) goto err_mbx_wq_alloc; } =20 - adapter->stats_wq =3D alloc_workqueue("%s-%s-stats", 0, 0, + adapter->stats_wq =3D alloc_workqueue("%s-%s-stats", + WQ_UNBOUND | WQ_MEM_RECLAIM, 0, dev_driver_string(dev), dev_name(dev)); if (!adapter->stats_wq) { @@ -210,7 +214,8 @@ static int idpf_probe(struct pci_dev *pdev, const struc= t pci_device_id *ent) goto err_stats_wq_alloc; } =20 - adapter->vc_event_wq =3D alloc_workqueue("%s-%s-vc_event", 0, 0, + adapter->vc_event_wq =3D alloc_workqueue("%s-%s-vc_event", + WQ_UNBOUND | WQ_MEM_RECLAIM, 0, dev_driver_string(dev), dev_name(dev)); if (!adapter->vc_event_wq) { --=20 2.46.0.76.ge559c4bf1a-goog