From nobody Sat Feb 7 16:32:07 2026 Received: from mail-pj1-f74.google.com (mail-pj1-f74.google.com [209.85.216.74]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3779D4C61B for ; Fri, 2 Aug 2024 20:01:42 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.74 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1722628903; cv=none; b=M2hcaLF1AojgDV+OjzdSvEdw7YhE1rwYa/IVklxepBgfKlAkLrbUi+E1zj4xzwCQvskXdaVmHTJw2GHCJmGnws1+wLKFW2VHWDeha6MEl3X+6zGVfjXWeE9/F+Ifhpec7R+lhb3PhMS43OFrEbqmfmP0dq1cLjakeaqNqq013t8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1722628903; c=relaxed/simple; bh=eggCc2iIcNGUqxlTvkV0QVMXd9vYfqCFOgNyFcQOOTs=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=HNqzSR6yFmApH9Icfvhzba/WjEc0BvACgeaFOzzl2U2PruvzGrIDbIdPo/w+uBx+2gWJdS9dXqEReU1qveV4sUJodN1lCPXTfr+s/duUqLkjSCgU31T+AOhyQYSuzjU9ciRmW16xSqa3+oIFn3P20waORkgZPHX2Hnw/YbTzPbM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=B6tJGctS; arc=none smtp.client-ip=209.85.216.74 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="B6tJGctS" Received: by mail-pj1-f74.google.com with SMTP id 98e67ed59e1d1-2cb5ba80e77so10183904a91.1 for ; Fri, 02 Aug 2024 13:01:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1722628901; x=1723233701; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=xQLoSNAIlbI381p0LRgfP/Yiv8RuXN9KeorgYSd+JBM=; b=B6tJGctSiWJ7lpRmNAvOErsdBHwMzRlsInYceU9zUV9KlyUzkkB5gQHhqDh58y5Lhg cEWgko/bGFaj32siwTBXTZk+87Zi4wQmRx+EjsIh3MH1v4S8eSYE6i68nnKABkpXiEaX fOhK/DrINjZWWX1L+2CWrQ2jq0+GOLteH6bEG/73hGCBVujUJVINGglMbIubpphgo/ov GWSEceA6PewvJRWmp6kgxDSy6MXnH/+SMaKvxUJvmLW4xV54FugEkaQcd+T7E6E4VoC1 YzVDbLLaFPka8OCfZu/Ir0C3S3ixIwpdC9hCtpeyFasKBmNIGnQ5nsYBPF275BhbH3yD YE7w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1722628901; x=1723233701; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=xQLoSNAIlbI381p0LRgfP/Yiv8RuXN9KeorgYSd+JBM=; b=Yic2gl5U/HV0OD649KXAOfINJMJhQ5+b1dNQteZ/v3LN3zCBlcxromsDqqSxLKeoO8 udTPfWvp2QwjNYZEYH8YD8csz5w7cNpZPnzLglu+Z86wCE5tPcgV2uAtQW7UQjips8Yu LUTZab7brbT1//SfE+PWO1j+i2PPP8RbqqzN4Et4IgsboI4UDfHrtMkytRnG7xy6muRZ x0cymmJnpS6Y7eSpcscXrqUfYR+WdEArIp3jZdzwKIQEjXwL0927uVYVo5LuN0/d4RiG FvwssN/xz5OQsAyvcU16Ng41HKGrwr3Co0J/tRF49gLIVNE4COlzrzXE6Ze1YyemlGVk M8fQ== X-Forwarded-Encrypted: i=1; AJvYcCUJJ+DNfkCNmZtXBMpxJt26UiMD+xAWbN6CB/sFubg3VL6yE+mhNFlcj5SdIXw3lytByieCKmubAH3Wap6iOFl6Jrl+8n6gm7GLgxu7 X-Gm-Message-State: AOJu0Yyr17HLaGJ+QlxUPGpBjwJFu+vujguyqwJOuW0YjXI1dmR+9ELy kWWLnVLviV/Y1h4mbgWMsKQe6SqRC1SvnHWx91tG6Rh2jqYKmNDfOPJLXJuZ/14mdtO6GBwQPw+ RHg== X-Google-Smtp-Source: AGHT+IFId3HHn5L5fWCDKxPTuecTBLeAk2p3kpSjqAki6o3dAad+BGMQ+YjK6m4jW2ZqIb6OkUGH+CX4xls= X-Received: from zagreus.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:5c37]) (user=seanjc job=sendgmr) by 2002:a17:90a:ee94:b0:2c8:9052:58bb with SMTP id 98e67ed59e1d1-2cff93d2bfbmr119920a91.2.1722628901308; Fri, 02 Aug 2024 13:01:41 -0700 (PDT) Reply-To: Sean Christopherson Date: Fri, 2 Aug 2024 13:01:35 -0700 In-Reply-To: <20240802200136.329973-1-seanjc@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240802200136.329973-1-seanjc@google.com> X-Mailer: git-send-email 2.46.0.rc2.264.g509ed76dc8-goog Message-ID: <20240802200136.329973-2-seanjc@google.com> Subject: [PATCH 1/2] KVM: Return '0' directly when there's no task to yield to From: Sean Christopherson To: Marc Zyngier , Oliver Upton , Paolo Bonzini Cc: linux-arm-kernel@lists.infradead.org, kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Sean Christopherson , Steve Rutherford Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Do "return 0" instead of initializing and returning a local variable in kvm_vcpu_yield_to(), e.g. so that it's more obvious what the function returns if there is no task. No functional change intended. Signed-off-by: Sean Christopherson Acked-by: Oliver Upton --- virt/kvm/kvm_main.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c index d0788d0a72cc..91048a7ad3be 100644 --- a/virt/kvm/kvm_main.c +++ b/virt/kvm/kvm_main.c @@ -3932,7 +3932,7 @@ int kvm_vcpu_yield_to(struct kvm_vcpu *target) { struct pid *pid; struct task_struct *task =3D NULL; - int ret =3D 0; + int ret; =20 rcu_read_lock(); pid =3D rcu_dereference(target->pid); @@ -3940,7 +3940,7 @@ int kvm_vcpu_yield_to(struct kvm_vcpu *target) task =3D get_pid_task(pid, PIDTYPE_PID); rcu_read_unlock(); if (!task) - return ret; + return 0; ret =3D yield_to(task, 1); put_task_struct(task); =20 --=20 2.46.0.rc2.264.g509ed76dc8-goog From nobody Sat Feb 7 16:32:07 2026 Received: from mail-yw1-f201.google.com (mail-yw1-f201.google.com [209.85.128.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 15A067BB14 for ; Fri, 2 Aug 2024 20:01:44 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1722628906; cv=none; b=I9R/plL9y/49UQWCJ8VB5nwofKEdELDVcfl7apCNhLnuatt4pgchuqDnlAQwaFEXsEhN7uU08b4iqAvv7STbmlY4GKjZfaTlBE1D99xr4a3p/uhxIPWK9v4grYMTsUAxGmAhpnXEEdCDhtY7IiRyi8cvJ1A6AnzTHi+snDZD9P0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1722628906; c=relaxed/simple; bh=YXcCG8Han6oPFZMVR5EJq3rsCuUTs8798czh5TGgT6g=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=J+T8DPfsUbUOzAC6z64Su1H2Erb4qVWegzUl86U2rTYbhwZij/nSKN3OsZZ1CO5d2ZBs27UeUBmtYeigR8+AnBaJGy29vQNnrXZGN+XGK0gBSD9o1Z/vAJL/yzdrNIelewfe8NWZLEzQtVDHU8EkeXhqRVO/aca2yPuwv8vhW9U= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=P/RgV1bf; arc=none smtp.client-ip=209.85.128.201 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="P/RgV1bf" Received: by mail-yw1-f201.google.com with SMTP id 00721157ae682-666010fb35cso59121487b3.0 for ; Fri, 02 Aug 2024 13:01:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1722628904; x=1723233704; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=ykfKcxdUa9fLrTb96clu+XLU5WSDBheRMe8+PcsjFjM=; b=P/RgV1bfBlyAg+RXdvMXskfAtdEZMK4thSDE6UJaVFge1JaA/2LiXvTAKVfO88IiMX vjrv6sMRdjT6ewChUEDo/WZvIi62A1A+E5rUkn9/s01Qqh5qMCb6SBrUPW8yr7V+s3uJ NJj8phrDNR7hrCWy0oIZ1yEQKyuUycNF4E1dVj6sDkoZYlO3dujPv/Rsn2BHbkyA+Cay cvDJtSYN9VZdKnx7J3xAEXWyvhAQsFzu7dTxuWqxLv3nwazJNogvMw/kOztfXibbY+8S l1In1bsFf4+Ow7KJ0euZbGcV5NHl7zx5zQJxdoSr1rpsDpuEIPwwn3q5FKzR85VgIRpl DSCw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1722628904; x=1723233704; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=ykfKcxdUa9fLrTb96clu+XLU5WSDBheRMe8+PcsjFjM=; b=q3y/l1/qH0hwIVbBff/9gYNvdSp3gl50/sIkLPkfbZxF9vIF9NwSrZlsmLI9ngx+XX OYIkx5E88GsXAl9y7fqvb2IG5walHelARZ1fZuqBCnxUTxCmqOVzbwYmE2G3nLMLD0Ku hUrPIsASlR7vc77GEIiLoElmXKqyWunf/32+Kgo5s5w5OooElTi+mEUoGGa36u+U2+0C 3xJfVJyDl/KUIt3/VR574KlW4wJ56a5FAdDBtcIl+IeWfTdSMqRuZyTXcsZCIrV4J1Je bk39cvny+HYqirGxhvEjH/rVwVFNLo2byFQFbp1tQBjCXFAIQSf8RO15b9NBX5gMF3y6 Fm8g== X-Forwarded-Encrypted: i=1; AJvYcCWTbZ7moUFjqZSv9DpLwYJNbxaW8oluGE7Pk8dZVj6augpqo+TJpRhbo+kW52nkvh4Mf/Y8CTRLkN1X8droZsVc3PlNtkeFpoSc3Ha0 X-Gm-Message-State: AOJu0Yzj6i5Ut4MtYmWdV6M+dhwDmpFmDuLEIFgP7pyiJvZbiwg/CeCq npUrJt7SwOiH+GxvXtXsEA15QJYTM8jMwgYM815M8qH/w/p8RbOTttIPwbzDL+1b0FXtwGJfAXY gpQ== X-Google-Smtp-Source: AGHT+IGvtD2k+Z8XEDfN/3DgERyf+vS3FmJyCb7y6ktKFZouZ8vRLWbcTwmFsQsX8ntvC2Xbw1LZ/rv2S6Q= X-Received: from zagreus.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:5c37]) (user=seanjc job=sendgmr) by 2002:a05:690c:2901:b0:644:c4d6:add0 with SMTP id 00721157ae682-6884f7ffdfcmr402097b3.1.1722628903604; Fri, 02 Aug 2024 13:01:43 -0700 (PDT) Reply-To: Sean Christopherson Date: Fri, 2 Aug 2024 13:01:36 -0700 In-Reply-To: <20240802200136.329973-1-seanjc@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240802200136.329973-1-seanjc@google.com> X-Mailer: git-send-email 2.46.0.rc2.264.g509ed76dc8-goog Message-ID: <20240802200136.329973-3-seanjc@google.com> Subject: [PATCH 2/2] KVM: Protect vCPU's "last run PID" with rwlock, not RCU From: Sean Christopherson To: Marc Zyngier , Oliver Upton , Paolo Bonzini Cc: linux-arm-kernel@lists.infradead.org, kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Sean Christopherson , Steve Rutherford Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" To avoid jitter on KVM_RUN due to synchronize_rcu(), use a rwlock instead of RCU to protect vcpu->pid, a.k.a. the pid of the task last used to a vCPU. When userspace is doing M:N scheduling of tasks to vCPUs, e.g. to run SEV migration helper vCPUs during post-copy, the synchronize_rcu() needed to change the PID associated with the vCPU can stall for hundreds of milliseconds, which is problematic for latency sensitive post-copy operations. In the directed yield path, do not acquire the lock if it's contended, i.e. if the associated PID is changing, as that means the vCPU's task is already running. Reported-by: Steve Rutherford Signed-off-by: Sean Christopherson Acked-by: Oliver Upton Reviewed-by: Steve Rutherford --- arch/arm64/include/asm/kvm_host.h | 2 +- include/linux/kvm_host.h | 3 ++- virt/kvm/kvm_main.c | 32 +++++++++++++++++-------------- 3 files changed, 21 insertions(+), 16 deletions(-) diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm= _host.h index a33f5996ca9f..7199cb014806 100644 --- a/arch/arm64/include/asm/kvm_host.h +++ b/arch/arm64/include/asm/kvm_host.h @@ -1115,7 +1115,7 @@ int __kvm_arm_vcpu_set_events(struct kvm_vcpu *vcpu, void kvm_arm_halt_guest(struct kvm *kvm); void kvm_arm_resume_guest(struct kvm *kvm); =20 -#define vcpu_has_run_once(vcpu) !!rcu_access_pointer((vcpu)->pid) +#define vcpu_has_run_once(vcpu) (!!READ_ONCE((vcpu)->pid)) =20 #ifndef __KVM_NVHE_HYPERVISOR__ #define kvm_call_hyp_nvhe(f, ...) \ diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h index 689e8be873a7..d6f4e8b2b44c 100644 --- a/include/linux/kvm_host.h +++ b/include/linux/kvm_host.h @@ -342,7 +342,8 @@ struct kvm_vcpu { #ifndef __KVM_HAVE_ARCH_WQP struct rcuwait wait; #endif - struct pid __rcu *pid; + struct pid *pid; + rwlock_t pid_lock; int sigset_active; sigset_t sigset; unsigned int halt_poll_ns; diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c index 91048a7ad3be..fabffd85fa34 100644 --- a/virt/kvm/kvm_main.c +++ b/virt/kvm/kvm_main.c @@ -486,6 +486,7 @@ static void kvm_vcpu_init(struct kvm_vcpu *vcpu, struct= kvm *kvm, unsigned id) vcpu->kvm =3D kvm; vcpu->vcpu_id =3D id; vcpu->pid =3D NULL; + rwlock_init(&vcpu->pid_lock); #ifndef __KVM_HAVE_ARCH_WQP rcuwait_init(&vcpu->wait); #endif @@ -513,7 +514,7 @@ static void kvm_vcpu_destroy(struct kvm_vcpu *vcpu) * the vcpu->pid pointer, and at destruction time all file descriptors * are already gone. */ - put_pid(rcu_dereference_protected(vcpu->pid, 1)); + put_pid(vcpu->pid); =20 free_page((unsigned long)vcpu->run); kmem_cache_free(kvm_vcpu_cache, vcpu); @@ -3930,15 +3931,17 @@ EXPORT_SYMBOL_GPL(kvm_vcpu_kick); =20 int kvm_vcpu_yield_to(struct kvm_vcpu *target) { - struct pid *pid; struct task_struct *task =3D NULL; int ret; =20 - rcu_read_lock(); - pid =3D rcu_dereference(target->pid); - if (pid) - task =3D get_pid_task(pid, PIDTYPE_PID); - rcu_read_unlock(); + if (!read_trylock(&target->pid_lock)) + return 0; + + if (target->pid) + task =3D get_pid_task(target->pid, PIDTYPE_PID); + + read_unlock(&target->pid_lock); + if (!task) return 0; ret =3D yield_to(task, 1); @@ -4178,9 +4181,9 @@ static int vcpu_get_pid(void *data, u64 *val) { struct kvm_vcpu *vcpu =3D data; =20 - rcu_read_lock(); - *val =3D pid_nr(rcu_dereference(vcpu->pid)); - rcu_read_unlock(); + read_lock(&vcpu->pid_lock); + *val =3D pid_nr(vcpu->pid); + read_unlock(&vcpu->pid_lock); return 0; } =20 @@ -4466,7 +4469,7 @@ static long kvm_vcpu_ioctl(struct file *filp, r =3D -EINVAL; if (arg) goto out; - oldpid =3D rcu_access_pointer(vcpu->pid); + oldpid =3D vcpu->pid; if (unlikely(oldpid !=3D task_pid(current))) { /* The thread running this VCPU changed. */ struct pid *newpid; @@ -4476,9 +4479,10 @@ static long kvm_vcpu_ioctl(struct file *filp, break; =20 newpid =3D get_task_pid(current, PIDTYPE_PID); - rcu_assign_pointer(vcpu->pid, newpid); - if (oldpid) - synchronize_rcu(); + write_lock(&vcpu->pid_lock); + vcpu->pid =3D newpid; + write_unlock(&vcpu->pid_lock); + put_pid(oldpid); } vcpu->wants_to_run =3D !READ_ONCE(vcpu->run->immediate_exit__unsafe); --=20 2.46.0.rc2.264.g509ed76dc8-goog