From nobody Sun Sep 14 16:55:09 2025 Received: from mail-oo1-f74.google.com (mail-oo1-f74.google.com [209.85.161.74]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 53EDE1ABEC5 for ; Thu, 5 Jun 2025 15:02:58 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.161.74 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1749135781; cv=none; b=IQ6KH8bw4lln4cck7ZFRh943zmfEzaT54jfm1knqJppkFkQ61t+3SKNwwwCq4cyaCxvv3F/Wq4nb8pkSiuxe9gkdBoAXSU+AqvWbcNfBWOUUATad8cDqPDOMdGzYWYhZTAFzhu0qZhq7PAYX1AmR/Eh0khjQQ8KHXdARMCIPHvY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1749135781; c=relaxed/simple; bh=6ebvtx27JnbmpcCRxTaFDXwmEkzSZL+DSbCdff1DwQs=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=P+bSNBcW4zqRPziwncBLikQwmII/F7DFjkrK1Em2fIuJ2CWbHFYhi5ObNirIxx1ejuXG9Vlh/jE3yNnqJ1lQ1wZXzD7CNsmdsrs9GdzfGrrVlXW7KxN/m0gHScviQClLVMkH7WwNEWH1JgogwB6mMEPbBCCA4XMu75jnXlVs9Ik= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--dionnaglaze.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=inNINCqb; arc=none smtp.client-ip=209.85.161.74 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--dionnaglaze.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="inNINCqb" Received: by mail-oo1-f74.google.com with SMTP id 006d021491bc7-60436c85f85so914792eaf.3 for ; Thu, 05 Jun 2025 08:02:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1749135777; x=1749740577; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=F4QpCwuMDO3ieFJHgjbXdb25+Lz909bG6cS+4hVbc5I=; b=inNINCqbeUyMSgffnSG7LOznLR13o0rZGMxG1i3w1nTMx2n3hGzleCIPnRp53r+JEd 03HuyLrcVZHo1diGFfNb4jJRImE2hUtM+LDmnuG7XFyschD2psVQtnug/Gv9ih6mJhF4 JI6lATcu48XDfG/agCteHhg6EA4mcps/olDqMytMvnjw86XEVzfjvPIivSs3iFst4WW+ NnxMZMKOSkvqZRyKlVzWniI370KZARCX0tK9GKSrtP4sb0EV4EU0/T9dU/QAlhmSTGNW GdyZSWYpM7bAonMa5bXfCuUOVDeKAmiKyqZ375m0HtF48XSN3UIVh1NLYN8qJ6x8Yuvx 0XWg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1749135777; x=1749740577; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=F4QpCwuMDO3ieFJHgjbXdb25+Lz909bG6cS+4hVbc5I=; b=Vsct1nrood23vhNhoHHUqbeld3oWVGND6oWPuA56tbSbc2m+jgivSDPFZ0xL1lq9tv f+rB5qkmOeKSSj6dop/eVgMXJQGqfJT6Uq0qRM29tS9M2YAEud73hdEvtlkGrifnaUr5 lKfbQ5fzLqOO0HqcBbDROeBrF8/xD1XGxY2/9g2wz/2lsl1XOITVfaal8J9Da/QY5g0E V7EKip3VSZrJ9JlkFF7Pn5OdEgxEa1uhI+LVbGAND0yyQc3iKnnzxvDeXuHAvj0yYHmf oG2jUl6klRnAh18uvbr7ZnA4bhFocN33H83RCJN9CstZdp73zDVPpmX7jKwHXNAlERAb o1CQ== X-Forwarded-Encrypted: i=1; AJvYcCVB48Ednf8VC3LAydR2j1+RiI/3OIhMAAqztmXdW35Ft45TXUjnRRg06S8m6cEiHkJpl1zixRXZpkqZJas=@vger.kernel.org X-Gm-Message-State: AOJu0YzSGTcQ9wLCPAhPDE4XRyqzsQeQZsrGo6XOyqkwK9yCoNE8sRb8 HhBgGUrOQ+jC6fdNRzkQOV4Im9fhOFc04rg104VkTRMeFTucs5J2hgxK4tjOp37UZ7z4B2T8kpQ iBIRlfGldpRWZkcqn/MeA7bTLPw== X-Google-Smtp-Source: AGHT+IEGgg2iURpToiwcoNT/TtIwkShYVJHPggNKgfXkCJt3fQ3ClBkysBTk4Q3GO67DXJVVpKXITswreVrMxROnKw== X-Received: from oablb10.prod.google.com ([2002:a05:6871:414a:b0:2d5:5a26:d92]) (user=dionnaglaze job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6870:aa88:b0:296:9c08:51a3 with SMTP id 586e51a60fabf-2e9bf66b5f8mr4663231fac.39.1749135777310; Thu, 05 Jun 2025 08:02:57 -0700 (PDT) Date: Thu, 5 Jun 2025 15:02:35 +0000 In-Reply-To: <20250605150236.3775954-1-dionnaglaze@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20250605150236.3775954-1-dionnaglaze@google.com> X-Mailer: git-send-email 2.50.0.rc0.642.g800a2b2222-goog Message-ID: <20250605150236.3775954-2-dionnaglaze@google.com> Subject: [PATCH v6 1/2] kvm: sev: Add SEV-SNP guest request throttling From: Dionna Glaze To: kvm@vger.kernel.org, linux-kernel@vger.kernel.org Cc: linux-coco@lists.linux.dev, Dionna Glaze , Thomas Lendacky , Paolo Bonzini , Joerg Roedel , Peter Gonda , Borislav Petkov , Sean Christopherson Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" The AMD-SP is a precious resource that doesn't have a scheduler other than a mutex lock queue. To avoid customers from causing a DoS, a kernel module parameter for rate limiting guest requests is added. The default value does not impose any rate limiting. Throttling vs scheduling: Even though Linux kernel mutexes have fair scheduling, the SEV command mutex is not enough to balance the AMD-SP load in a manner that favors the host to run VM launches for low boot latency over traffic from the guest in the form of guests requests that it can't predict. Boot sequence commands and guest request commands all contend on the same mutex, so boot latency is affected by increased guest request contention. A VM launch may see dozens of SNP_LAUNCH_UPDATE commands before SNP_LAUNCH_FINISH, and boot times are a heavily protected metric in hyperscalars. To favor lower latency of VM launches over each VM's ability to request attestations at a high rate, the guest requests need a secondary scheduling mechanism. It's not good practice to hold a lock and return to user space, so using a secondary lock for VM launch sequences is not an appropriate solution. For simplicity, merely set a rate limit for every VM's guest requests and allow a system administrator to tune that rate limit to platform needs. Design decisions: The throttle rate for a VM cannot be changed once it has been started. The rate the VM gets is its level of service, so it should not be degradable by a mem_enc_ioctl for example. Empirical investigation: With a test methodology of turning up N-1 "antagonist" VMs with 2 vCPUs and 4GiB RAM that all request a SEV-SNP attestation a tight loop before measuring the boot latency of the Nth VM, an effective quality of service should keep the average boot latency at levels without any guest request contention. On a dedicated 256 core AMD Zen3 with 1TiB of RAM, continuous performance testing shows that a boot latency of 220ms +- 50ms is typical with N in {4, 16, 32, 64} when the request rate is set to 1/s. After N=3D64, the rate limit of 1 HZ is insufficient to hold back enough time for the final VM launch to succeed consistently in the contention. Cc: Thomas Lendacky Cc: Paolo Bonzini Cc: Joerg Roedel Cc: Peter Gonda Cc: Borislav Petkov Cc: Sean Christopherson Signed-off-by: Dionna Glaze --- arch/x86/kvm/svm/sev.c | 17 +++++++++++++++++ arch/x86/kvm/svm/svm.h | 3 +++ 2 files changed, 20 insertions(+) diff --git a/arch/x86/kvm/svm/sev.c b/arch/x86/kvm/svm/sev.c index 1aa0f07d3a63..e45f0cfae2bd 100644 --- a/arch/x86/kvm/svm/sev.c +++ b/arch/x86/kvm/svm/sev.c @@ -12,13 +12,16 @@ #include #include #include +#include #include #include #include #include #include #include +#include #include +#include #include =20 #include @@ -59,6 +62,10 @@ static bool sev_es_debug_swap_enabled =3D true; module_param_named(debug_swap, sev_es_debug_swap_enabled, bool, 0444); static u64 sev_supported_vmsa_features; =20 +/* set a per-VM rate limit for SEV-SNP guest requests on VM creation. 0 is= unlimited. */ +static int sev_snp_request_ratelimit_khz =3D 0; +module_param(sev_snp_request_ratelimit_khz, int, 0444); + #define AP_RESET_HOLD_NONE 0 #define AP_RESET_HOLD_NAE_EVENT 1 #define AP_RESET_HOLD_MSR_PROTO 2 @@ -367,6 +374,7 @@ static int snp_guest_req_init(struct kvm *kvm) { struct kvm_sev_info *sev =3D to_kvm_sev_info(kvm); struct page *req_page; + u64 throttle_interval; =20 req_page =3D alloc_page(GFP_KERNEL_ACCOUNT | __GFP_ZERO); if (!req_page) @@ -381,6 +389,9 @@ static int snp_guest_req_init(struct kvm *kvm) sev->guest_req_buf =3D page_address(req_page); mutex_init(&sev->guest_req_mutex); =20 + throttle_interval =3D ((u64)sev_snp_request_ratelimit_khz * HZ) / HZ_PER_= KHZ; + ratelimit_state_init(&sev->snp_guest_msg_rs, sev_snp_request_ratelimit_kh= z, 1); + return 0; } =20 @@ -4028,6 +4039,12 @@ static int snp_handle_guest_req(struct vcpu_svm *svm= , gpa_t req_gpa, gpa_t resp_ =20 mutex_lock(&sev->guest_req_mutex); =20 + if (!__ratelimit(&sev->snp_guest_msg_rs)) { + svm_vmgexit_no_action(svm, SNP_GUEST_ERR(SNP_GUEST_VMM_ERR_BUSY, 0)); + ret =3D 1; + goto out_unlock; + } + if (kvm_read_guest(kvm, req_gpa, sev->guest_req_buf, PAGE_SIZE)) { ret =3D -EIO; goto out_unlock; diff --git a/arch/x86/kvm/svm/svm.h b/arch/x86/kvm/svm/svm.h index f16b068c4228..2643c940d054 100644 --- a/arch/x86/kvm/svm/svm.h +++ b/arch/x86/kvm/svm/svm.h @@ -18,6 +18,7 @@ #include #include #include +#include =20 #include #include @@ -112,6 +113,8 @@ struct kvm_sev_info { void *guest_req_buf; /* Bounce buffer for SNP Guest Request input */ void *guest_resp_buf; /* Bounce buffer for SNP Guest Request output */ struct mutex guest_req_mutex; /* Must acquire before using bounce buffers= */ + + struct ratelimit_state snp_guest_msg_rs; /* Limit guest requests */ }; =20 struct kvm_svm { --=20 2.50.0.rc0.642.g800a2b2222-goog From nobody Sun Sep 14 16:55:09 2025 Received: from mail-pf1-f202.google.com (mail-pf1-f202.google.com [209.85.210.202]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 417971C5F14 for ; Thu, 5 Jun 2025 15:02:59 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.202 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1749135781; cv=none; b=HOsSLJOMEFiTWelUwING8nqmw8n8SGSxQYCq2FIpvHu5KXffhyOnCNwxGheV7OJgUYgG/YJdeL8FCQZnA8nxMsx5splxwTdlZJeAqxMfGDcRWZruTretmHuWfTvYqwB9yLe1K3QNtPsv3UzqGOa5Qau1uIDSWijqiNw2P7tyups= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1749135781; c=relaxed/simple; bh=3jeQmiKt06GXrIHfpjh0oQlnOaSY9Kr4rz+umFvQdDA=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=LRaQoZJ/CKkFq26VYx404jy/xn48Ww8BHio5ay3X7F3I+//jVQ2K8ditRtBWU/Hs808RBu5ftdEOKAW4ciT+gM8tJ4paRbQ6NHr6YT7YLKsjc9o7S2Rfo7VTM/2wp4YqL9SY+OaJwbLtSIzNJ5geLdm+Mzsgbd+gVI6HITiW37Q= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--dionnaglaze.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=xxcbR5lM; arc=none smtp.client-ip=209.85.210.202 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--dionnaglaze.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="xxcbR5lM" Received: by mail-pf1-f202.google.com with SMTP id d2e1a72fcca58-7377139d8b1so1071401b3a.0 for ; Thu, 05 Jun 2025 08:02:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1749135779; x=1749740579; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=Xzwm5+cjAdLoNMVYNUU/eCk5jF9CVdW14YhkjiBUlxs=; b=xxcbR5lMaILUjVpQrFjgd3/BOPL+dbjdx7N0iqLoFCCiWU48g512azL/kg8x5DHHsz UMEdP3DXImkfJea5LBpIp/R60bRrhqadUxRMBpJlMQiMQqQnExxL0dC/5kURg4OhZ+Rq 3I0v5cJxmk8BudW0COy79kuny57EQ+2LA2Nrwgrlwa23Pt7lPFpNtmfanm+AHmTJiXC2 sJzcTrz9/FusxTWSl3Ey8I2GC3Nr6I7D6UKoMkJEHJeoWX3jcG60CJCVNaJ7bZkANJZN nVf084+oupRHmDBN6HjaPMlEnz2qxAjMj9iv5y6DqwmqSCxPgDsJ/LBWz6PH/bt9WXp1 mqAw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1749135779; x=1749740579; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=Xzwm5+cjAdLoNMVYNUU/eCk5jF9CVdW14YhkjiBUlxs=; b=hmZyc5hcQugvMbTbn41HA6NMl3cfD8o0tjvKPm8LjL8TuF58KIU1COzuXqrHqkxZA0 z7zbdnojwtJu+F/k3w6ThBWzv2+NTzoP+jO/TgKt84zIcFbT593yeqR3JvuiYJUZH/Sd dw0Oz3+MwuOjN4q8QSi7eKrcVD97nnTyt1322LNQH1X2m6P91jw2X8xDWEF+mh5eyTH6 EVxVqj4HGEBWsx4Agui5s4Vr1ULNNY0nNQkpAurTA+VLhXMpeIlynWUGfJKFI6WJ5WIX 1/o9rP0S1WL0sE9q56a+rskNYwIC1MRFacgVQrTrwi1C9E+umvjB1OUB84+6Jmix0n6J 1quQ== X-Forwarded-Encrypted: i=1; AJvYcCVCv9E8JNOhNFJ/4zw7ICpvzm9EQQ9CXIqlc/5PmKQ7FsmxlNJC7TsrWd94vBsH81WLvlb9NMtoOshrO6k=@vger.kernel.org X-Gm-Message-State: AOJu0YymUv9nRY5VOrkwlZo14LXGK8Q8kemw3XpDLWvuHpEGWSi+0wC3 ZTzj45IKg7LXaK68rS1McUZA+RtBFcRnSAMPL51MjmLFoHOhJHNtuHJNV0JL83GB+9KLWN3pQ4Y t427Zzu9iJKzH3HYRjnsEjbyamw== X-Google-Smtp-Source: AGHT+IHb5hKDUYrgVOzSrfCbbqZtDuSQIRve0rxtHVkUEsiBbiESd9WvsZr9lyVAILAcpiMtqMiFe+S0tlpgh2kuCw== X-Received: from pfvx14.prod.google.com ([2002:a05:6a00:270e:b0:747:9faf:ed39]) (user=dionnaglaze job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6a00:b53:b0:747:ee09:1fb4 with SMTP id d2e1a72fcca58-74827f09679mr61292b3a.15.1749135779455; Thu, 05 Jun 2025 08:02:59 -0700 (PDT) Date: Thu, 5 Jun 2025 15:02:36 +0000 In-Reply-To: <20250605150236.3775954-1-dionnaglaze@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20250605150236.3775954-1-dionnaglaze@google.com> X-Mailer: git-send-email 2.50.0.rc0.642.g800a2b2222-goog Message-ID: <20250605150236.3775954-3-dionnaglaze@google.com> Subject: [PATCH v6 2/2] kvm: sev: If ccp is busy, report busy to guest From: Dionna Glaze To: kvm@vger.kernel.org, linux-kernel@vger.kernel.org Cc: linux-coco@lists.linux.dev, Dionna Glaze , Thomas Lendacky , Paolo Bonzini , Joerg Roedel , Peter Gonda , Borislav Petkov , Sean Christopherson Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" The ccp driver can be overloaded even with guest request rate limits. The return value of -EBUSY means that there is no firmware error to report back to user space, so the guest VM would see this as exitinfo2 =3D 0. The false success can trick the guest to update its message sequence number when it shouldn't have. Instead, when ccp returns -EBUSY, that is reported to userspace as the throttling return value. Cc: Thomas Lendacky Cc: Paolo Bonzini Cc: Joerg Roedel Cc: Peter Gonda Cc: Borislav Petkov Cc: Sean Christopherson Signed-off-by: Dionna Glaze --- arch/x86/kvm/svm/sev.c | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/arch/x86/kvm/svm/sev.c b/arch/x86/kvm/svm/sev.c index e45f0cfae2bd..0ceb7e83a98d 100644 --- a/arch/x86/kvm/svm/sev.c +++ b/arch/x86/kvm/svm/sev.c @@ -4060,6 +4060,11 @@ static int snp_handle_guest_req(struct vcpu_svm *svm= , gpa_t req_gpa, gpa_t resp_ * the PSP is dead and commands are timing out. */ ret =3D sev_issue_cmd(kvm, SEV_CMD_SNP_GUEST_REQUEST, &data, &fw_err); + if (ret =3D=3D -EBUSY) { + svm_vmgexit_no_action(svm, SNP_GUEST_ERR(SNP_GUEST_VMM_ERR_BUSY, fw_err)= ); + ret =3D 1; + goto out_unlock; + } if (ret && !fw_err) goto out_unlock; =20 --=20 2.50.0.rc0.642.g800a2b2222-goog