From nobody Wed Apr 15 19:25:11 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id C06E7C4332F for ; Thu, 17 Nov 2022 18:12:35 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S240341AbiKQSMd (ORCPT ); Thu, 17 Nov 2022 13:12:33 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50172 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S240200AbiKQSLv (ORCPT ); Thu, 17 Nov 2022 13:11:51 -0500 Received: from mail-pj1-x104a.google.com (mail-pj1-x104a.google.com [IPv6:2607:f8b0:4864:20::104a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9A0A37FF3B for ; Thu, 17 Nov 2022 10:11:48 -0800 (PST) Received: by mail-pj1-x104a.google.com with SMTP id m2-20020a17090a730200b0021020cce6adso4640033pjk.3 for ; Thu, 17 Nov 2022 10:11:48 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=+DW5grLQa/cplZRP/qrf8uis0AH4MqPXCwF/prNP4d8=; b=DLv4LF4LpZOvyIb/TYzDQiC/8sszsLJYJFK8iY4HzP7/psWC3YjVIJr266ElEqGZSf l6izPjXDNEAFyFR3bBWyRaw7t/07Xiw6m4Gss3tLkPzWBCF8Pqqxtx54czoF+7e83Egq 9FH5iFPJuF3CYdlREeniosEnCv7zRJrJPYiXVd1W1q5HpIyP18e9XwL7FQswYg4w5zO+ 6ojGsP/OIYqA/eBfMh0QYvkT6eEFqJOyir0B1t/3KT0ODFBvIMGsMZLsnW3JeKqfN7g+ hqKpztpqy/cDIOy82+z0qmEu12L1WRffpl8O0Uq2lr4JlzZKJUye57W+p58o9qMxX0Wb cbwQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=+DW5grLQa/cplZRP/qrf8uis0AH4MqPXCwF/prNP4d8=; b=FUW7YM7jXEKA0zHdfXeG4t6WmGBfKRARUqJk2JkEXC9YIiy5LDbFVYJYdIP/5JWTIh iFIoF1oP3FYTrXLnL/xBqy4V733GJ/oweuOoCUtdkMVj+a9dMrywMKZG+eg8t1rPZXo8 mc/E5OKyHacyhl+hF9rNRDkqAwMxVoti5+5cFJOxFp8PEik9Pv86zx5wbjg2aUL4UiVD 1May8RDlopK4cLjVAVqNg2D7RFPFkgk+cimDokqEc3vpx+malqFMy/dMaM1DZuMmmPKC Ni1k3sfgttg7Ps6p6B16yf/OLAK+44lchD3RgHpdhWg3qL3idjOKqdqI0N8AGhvBDpRD 0B/Q== X-Gm-Message-State: ANoB5pnBi9tqOK3fqSpDk17ZC9a72/X5eMLuZCj6s7aWjFXnenqx5wEH ETEyn9Wd+fKb/DlNgEParQ4RezY+XM8krbmYbsrxMFdwSEl0Ww4q4QP9oeqSnHJikQ431rNcMZJ 8FuizGnZCv2oY/litHMQvhv8V5V2lqbJuRfBZIKzDM3cQTnt4Sc+Z0oFhal97trdXlrGmSPRo26 6MhSJwq4M= X-Google-Smtp-Source: AA0mqf7UEP297Vu/sUmDgoVHzWF569uQiHTAkvOfWkYFTEsr/Ju7F2fAjrlnu0WoC+03Ri7+DQDjvO0kQrjF+WYMGQ== X-Received: from dionnaglaze.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:2ee6]) (user=dionnaglaze job=sendgmr) by 2002:a17:903:48d:b0:17c:620f:13ac with SMTP id jj13-20020a170903048d00b0017c620f13acmr3993630plb.9.1668708708046; Thu, 17 Nov 2022 10:11:48 -0800 (PST) Date: Thu, 17 Nov 2022 18:11:26 +0000 In-Reply-To: <20221117181127.1859634-1-dionnaglaze@google.com> Mime-Version: 1.0 References: <20221117181127.1859634-1-dionnaglaze@google.com> X-Mailer: git-send-email 2.38.1.584.g0f3c55d4c2-goog Message-ID: <20221117181127.1859634-2-dionnaglaze@google.com> Subject: [PATCH v2 1/2] kvm: sev: Add SEV-SNP guest request throttling From: Dionna Glaze To: linux-kernel@vger.kernel.org, x86@kernel.org Cc: Dionna Glaze , Thomas Lendacky , Paolo Bonzini , Joerg Roedel , Ingo Molnar , Andy Lutomirsky , John Allen , Herbert Xu , "David S. Miller" , Peter Gonda Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" The AMD-SP is a precious resource that doesn't have a scheduler other than a mutex lock queue. To avoid customers from causing a DoS, we implement a module_param-set rate limit with a default of 2 requests per 2 seconds. These defaults were chosen empirically with a the assumption that current server-grade SEV-SNP machines will rarely exceed 128 VMs under usual circumstance. The 2 burst per 2 seconds means on average 1 request every second. We allow 2 requests back to back to allow for the guest to query the certificate length in an extended guest request without a pause. The 1 second average is our target for quality of service since empirical tests show that 64 VMs can concurrently request an attestation report with a maximum latency of 1 second. We don't anticipate more concurrency than that for a seldom used request for a majority well- behaved set of VMs. The majority point is decided as >64 VMs given the assumed 128 VM count for "extreme load". The throttling code is 2 << 32 given that invalid length is 1 and 2 is the next available code. This was suggested by Tom Lendacky, and will be included in a new revision of the GHCB specification. Cc: Thomas Lendacky Cc: Paolo Bonzini Cc: Joerg Roedel Cc: Ingo Molnar Cc: Andy Lutomirsky Cc: John Allen Cc: Herbert Xu Cc: "David S. Miller" Cc: Peter Gonda Signed-off-by: Dionna Glaze --- arch/x86/include/asm/sev-common.h | 1 + arch/x86/kvm/svm/sev.c | 29 +++++++++++++++++++++++++++++ arch/x86/kvm/svm/svm.h | 3 +++ 3 files changed, 33 insertions(+) diff --git a/arch/x86/include/asm/sev-common.h b/arch/x86/include/asm/sev-c= ommon.h index 9573ee1573ed..225b5e88f9a4 100644 --- a/arch/x86/include/asm/sev-common.h +++ b/arch/x86/include/asm/sev-common.h @@ -158,6 +158,7 @@ struct snp_psc_desc { =20 /* Guest message request error code */ #define SNP_GUEST_REQ_INVALID_LEN BIT_ULL(32) +#define SNP_GUEST_REQ_THROTTLED (((u64)2) << 32) =20 #define GHCB_MSR_TERM_REQ 0x100 #define GHCB_MSR_TERM_REASON_SET_POS 12 diff --git a/arch/x86/kvm/svm/sev.c b/arch/x86/kvm/svm/sev.c index a35cd9f33f16..3ced06c6e07a 100644 --- a/arch/x86/kvm/svm/sev.c +++ b/arch/x86/kvm/svm/sev.c @@ -61,6 +61,14 @@ module_param_named(sev_es, sev_es_enabled, bool, 0444); /* enable/disable SEV-SNP support */ static bool sev_snp_enabled =3D true; module_param_named(sev_snp, sev_snp_enabled, bool, 0444); + +/* Throttle guest requests to a burst # per this many seconds */ +unsigned int guest_request_throttle_s =3D 2; +module_param(guest_request_throttle_s, int, 0444); + +/* Throttle guest requests to this many per the above many seconds */ +unsigned int guest_request_throttle_burst =3D 2; +module_param(guest_request_throttle_burst, int, 0444); #else #define sev_enabled false #define sev_es_enabled false @@ -338,6 +346,9 @@ static int sev_guest_init(struct kvm *kvm, struct kvm_s= ev_cmd *argp) init_srcu_struct(&sev->psc_srcu); ret =3D sev_snp_init(&argp->error); mutex_init(&sev->guest_req_lock); + ratelimit_state_init(&sev->snp_guest_msg_rs, + guest_request_throttle_s * HZ, + guest_request_throttle_burst); } else { ret =3D sev_platform_init(&argp->error); } @@ -3288,6 +3299,14 @@ static void snp_cleanup_guest_buf(struct sev_data_sn= p_guest_request *data, unsig *rc =3D SEV_RET_INVALID_ADDRESS; } =20 +static bool snp_throttle_guest_request(struct kvm_sev_info *sev) { + if (__ratelimit(&sev->snp_guest_msg_rs)) + return false; + + pr_info_ratelimited("svm: too many guest message requests\n"); + return true; +} + static void snp_handle_guest_request(struct vcpu_svm *svm, gpa_t req_gpa, = gpa_t resp_gpa) { struct sev_data_snp_guest_request data =3D {0}; @@ -3304,6 +3323,11 @@ static void snp_handle_guest_request(struct vcpu_svm= *svm, gpa_t req_gpa, gpa_t =20 sev =3D &to_kvm_svm(kvm)->sev_info; =20 + if (snp_throttle_guest_request(sev)) { + rc =3D SNP_GUEST_REQ_THROTTLED; + goto e_fail; + } + mutex_lock(&sev->guest_req_lock); =20 rc =3D snp_setup_guest_buf(svm, &data, req_gpa, resp_gpa); @@ -3341,6 +3365,11 @@ static void snp_handle_ext_guest_request(struct vcpu= _svm *svm, gpa_t req_gpa, gp =20 sev =3D &to_kvm_svm(kvm)->sev_info; =20 + if (snp_throttle_guest_request(sev)) { + rc =3D SNP_GUEST_REQ_THROTTLED; + goto e_fail; + } + data_gpa =3D vcpu->arch.regs[VCPU_REGS_RAX]; data_npages =3D vcpu->arch.regs[VCPU_REGS_RBX]; =20 diff --git a/arch/x86/kvm/svm/svm.h b/arch/x86/kvm/svm/svm.h index e68b3aab57d6..b2eaac449d60 100644 --- a/arch/x86/kvm/svm/svm.h +++ b/arch/x86/kvm/svm/svm.h @@ -18,6 +18,7 @@ #include #include #include +#include =20 #include #include @@ -95,6 +96,8 @@ struct kvm_sev_info { void *snp_certs_data; struct mutex guest_req_lock; =20 + struct ratelimit_state snp_guest_msg_rs; /* Limit guest requests */ +=09 u64 sev_features; /* Features set at VMSA creation */ }; =20 --=20 2.38.1.584.g0f3c55d4c2-goog From nobody Wed Apr 15 19:25:11 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id AE267C433FE for ; Thu, 17 Nov 2022 18:12:40 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S240182AbiKQSMi (ORCPT ); Thu, 17 Nov 2022 13:12:38 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50180 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S240329AbiKQSLw (ORCPT ); Thu, 17 Nov 2022 13:11:52 -0500 Received: from mail-yw1-x1149.google.com (mail-yw1-x1149.google.com [IPv6:2607:f8b0:4864:20::1149]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id EF5D682209 for ; Thu, 17 Nov 2022 10:11:50 -0800 (PST) Received: by mail-yw1-x1149.google.com with SMTP id 00721157ae682-370624ca2e8so25923677b3.16 for ; Thu, 17 Nov 2022 10:11:50 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=wteh+NFEi1GAsB9oF3PCiWMRRI+co3r9IDe7/2rAE34=; b=nMPxbK2ewjH+fsYNZqiLd//2LrKs65SwyM7mNXN787ZD4W6hIOaTT53xBtzBuwe83f dPp8qx9NrhlldMjEFYsXhWxhpArwLywx3EZ5yiRT9eCxLDCW2IyhYOwLtAnlL0wg8slI 8OMjRj6j07oCEOVTquGO1qFYXxy1XvyLgRXhOMv7B/OhbZbgtPsw9gFwd1D7YBQSRH4F EVkBwDdJV5Gzegu0E/MV/fztUe+LBFRAUgq58rdCy767/vUbt02dcLrd0ATI4ibZc3AM 8hluEOAvA6br6+3UyzVP7s6kTDbf78BNguWrUs1opz1P7+su76qBu1ToVn6P8e4UKWH3 /l1Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=wteh+NFEi1GAsB9oF3PCiWMRRI+co3r9IDe7/2rAE34=; b=v+tpHOz/IoKJR+jjW3s9DPogzlNab/GHIO4tvreTOJGe7pz8nkLszdiaX5pYpyeoOB mjDuGOTj/zlCU/sgM8VHthiV+H4xwWL00tF6kTRtxY7vBGvRMVqOxQsexM8n8z/xlakj VQHlUFEgYVsZYVG0jR/Fcu2VAZH7TVTqUPaHol84+DBwyjLqOVwRvrCsfQBgcK7I7L1n 7vPBe4cltE9/OnC7TdW+c9YrUBnm2N36fdjzKD2iOZPtVLrprD+Tx1v4/GE/O9XGduR7 44QYnXnQB1mBhK8vpni5JCPULH5hJg113Xn+L+HqC6PjkEmks98pFErHvlHWLsm2RvQZ Jemg== X-Gm-Message-State: ANoB5pnXKptbAo+DvbCVvEXVyjQxcs4vOkcIa4L8wTq+bENPg8IjvWjb pkxmedbLwWnTAHEzN+IM/vsz4XvbM0Xh9eVgOcCAF1XqkGDMJIB3mJs8dyaQA71WMX8+Bt/dW/u IftkB5IWv+NOpto5it0BR6SY6ONARVF3BwUsrAG8HKpt/1a9r2TjeryMDLp5GcKzptYVHCUCXl8 vznS6i12g= X-Google-Smtp-Source: AA0mqf6V5BCoS8nuaZkX6AKXJwgdSPrsb9yINktEj3ef0TIf06GgqLN7xzlorDuSHRHkCtTI+xWz5xk/yNS8Crmgvg== X-Received: from dionnaglaze.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:2ee6]) (user=dionnaglaze job=sendgmr) by 2002:a25:74c1:0:b0:6de:47e2:e344 with SMTP id p184-20020a2574c1000000b006de47e2e344mr3355252ybc.450.1668708709870; Thu, 17 Nov 2022 10:11:49 -0800 (PST) Date: Thu, 17 Nov 2022 18:11:27 +0000 In-Reply-To: <20221117181127.1859634-1-dionnaglaze@google.com> Mime-Version: 1.0 References: <20221117181127.1859634-1-dionnaglaze@google.com> X-Mailer: git-send-email 2.38.1.584.g0f3c55d4c2-goog Message-ID: <20221117181127.1859634-3-dionnaglaze@google.com> Subject: [PATCH v2 2/2] kvm: sev: If ccp is busy, report throttled to guest From: Dionna Glaze To: linux-kernel@vger.kernel.org, x86@kernel.org Cc: Dionna Glaze , Thomas Lendacky , Paolo Bonzini , Joerg Roedel , Ingo Molnar , Andy Lutomirsky , John Allen , Herbert Xu , "David S. Miller" , Peter Gonda Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" The ccp driver can be overloaded even with 1 HZ throttling. The return value of -EBUSY means that there is no firmware error to report back to user space, so the guest VM would see this as exitinfo2 =3D 0. The false success can trick the guest to update its the message sequence number when it shouldn't have. Instead, when ccp returns -EBUSY, we report that to userspace as the throttling return value. Cc: Thomas Lendacky Cc: Paolo Bonzini Cc: Joerg Roedel Cc: Ingo Molnar Cc: Andy Lutomirsky Cc: John Allen Cc: Herbert Xu Cc: "David S. Miller" Cc: Peter Gonda Signed-off-by: Dionna Glaze --- arch/x86/kvm/svm/sev.c | 19 ++++++++++++++++--- 1 file changed, 16 insertions(+), 3 deletions(-) diff --git a/arch/x86/kvm/svm/sev.c b/arch/x86/kvm/svm/sev.c index 3ced06c6e07a..81e4862126fb 100644 --- a/arch/x86/kvm/svm/sev.c +++ b/arch/x86/kvm/svm/sev.c @@ -3335,7 +3335,13 @@ static void snp_handle_guest_request(struct vcpu_svm= *svm, gpa_t req_gpa, gpa_t goto unlock; =20 rc =3D sev_issue_cmd(kvm, SEV_CMD_SNP_GUEST_REQUEST, &data, &err); - if (rc) + /* + * The ccp driver can return -EBUSY if the PSP is overloaded, so + * we offer that as a throttling signal too. + */ + if (rc =3D=3D -EBUSY) + rc =3D SNP_GUEST_REQ_THROTTLED; + else if (rc) /* use the firmware error code */ rc =3D err; =20 @@ -3368,7 +3374,7 @@ static void snp_handle_ext_guest_request(struct vcpu_= svm *svm, gpa_t req_gpa, gp if (snp_throttle_guest_request(sev)) { rc =3D SNP_GUEST_REQ_THROTTLED; goto e_fail; - } + } =20 data_gpa =3D vcpu->arch.regs[VCPU_REGS_RAX]; data_npages =3D vcpu->arch.regs[VCPU_REGS_RBX]; @@ -3392,7 +3398,14 @@ static void snp_handle_ext_guest_request(struct vcpu= _svm *svm, gpa_t req_gpa, gp =20 rc =3D snp_guest_ext_guest_request(&req, (unsigned long)sev->snp_certs_da= ta, &data_npages, &err); - if (rc) { + /* + * The ccp driver can return -EBUSY if the PSP is overloaded, so + * we offer that as a throttling signal too. + */ + if (rc =3D=3D -EBUSY) { + rc =3D SNP_GUEST_REQ_THROTTLED; + goto cleanup; + } else if (rc) { /* * If buffer length is small then return the expected * length in rbx. --=20 2.38.1.584.g0f3c55d4c2-goog