From nobody Thu Apr 9 10:43:51 2026 Received: from mail-qt1-f226.google.com (mail-qt1-f226.google.com [209.85.160.226]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7529E1F09A8 for ; Tue, 10 Mar 2026 00:17:13 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.160.226 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1773101835; cv=none; b=dDPSuUY8rVyBBvmLoKkr+Rkn2xE9KLpkXThVkanNWVOTLt0jzQMt1UuWsPVvGAI93r8ZbvOA7dHWSUyfP8JjGtCkTE6MNjP3h+9yfEzcplmdQUECFS7IPWZUvgWtdB6FvltsRkVKOFhj41J0eP/EH6IB/iws4TUK4qEOEB9WnCY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1773101835; c=relaxed/simple; bh=sYGIrnAJyjCt8iIoF30ta2b9Ec4Gre5mQsKFa6a6+tY=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=SzVMqHO8kSzxEM0OOksAHdnafNmN3s18BTnVu5eI/lMd1hp87YOZf2M6M4MgFuZJlsR02D/7LTbYk4QwOOkTnvWttFZSoVwJ0nrgqbd5SCIbXra4pcQCjGbcBjK6L3tzuoIs/FS+TaA124ja7XSDun5iZTJK9PY4eTQ6nhnPRrc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=broadcom.com; spf=fail smtp.mailfrom=broadcom.com; dkim=pass (1024-bit key) header.d=broadcom.com header.i=@broadcom.com header.b=f4X3Syj7; arc=none smtp.client-ip=209.85.160.226 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=broadcom.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=broadcom.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=broadcom.com header.i=@broadcom.com header.b="f4X3Syj7" Received: by mail-qt1-f226.google.com with SMTP id d75a77b69052e-506a747448dso108567211cf.0 for ; Mon, 09 Mar 2026 17:17:13 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1773101832; x=1773706632; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:dkim-signature:x-gm-gg :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=sDBpeRPGQkSyveVh0r0L9u3UrQSmg15QNxwWzbeb67Y=; b=NaamJrl6kuaRnOSPpf558SqJEfTvEUlQkHFo8LVOR3g2htRpseASYAaPeJYY3g3QHA QYpQGwad5oHX9vHlQ+Medxz+DqnedTQRCqSpfy0uF72zbkD13Dp7XRVODhfvK4BNcKMK lgWZbO6NIYjH1pf6FwTPpXcqAZdjrUiwGbyzfr2XGFlQ3iVkvjwMr4yKxoSl7IkxMxCv AKzsB8EAxyyFJ1A40gKH6tRtp4K3mrdoUtE/eqC90XHPkxcKii257uYWm3lt7TdP4mB3 2Jo/Da2XKH23SBpl8vX4OHaUDgNqc/sc/IarJq60z2NmW+I9qmqyc0+IIzJSYWFxaZjk O1EQ== X-Forwarded-Encrypted: i=1; AJvYcCUVICi35x3X9hbwxR/SP9fyPXKa5yQMGxnZVhTMgmZRBQueRNkHVKkknjIMJzQ9znLq0bHfHoAEXNtsX4I=@vger.kernel.org X-Gm-Message-State: AOJu0YwzJ5F6ItIS5vQlL7V92Ln9+OPtVgRwnfg3TaOe1Qk855kltgB5 eBVQh9XyD7SM6YZWE8qb91rv/bGXGsf3WgCxYFMThza55GKgmttzf/6SuAZX3IeKstCxs9IkHVM /u0XtMatjeo/mgUMwSvmjqZpY+8ZKm41PpclWusoz0YlInvZKX/Hl/nNKCEEuJeddriJg92HPtd NVqnQYRscHFJcDPZOzzF8xecW94fM1oFibkxHv7kx93rxYVN4HYjN/Qo/npdCLCwFNstH80lTXC B++fcZbiXhI0IFJVxfcaHU= X-Gm-Gg: ATEYQzxV3L0qdrkxseXnfajBKx+XqWdkl4l5bbsKlPlytc51i51RS8G9EZ33nS6fgAq NZYQwVV6KlUdlrsjRIyaj2hUoacS4JgDW/YVU7hFj1KEixT2lCtCEkkRLUkjfi5yhsU1AyVT47A SJsRuD7FITGAXN9rbI3XuwMcOlpigDgLuiSAsV62MwC1BBzSoh13XxXiMw6OiZW1KNeSTUsDS0v 36cPxVrqx+gouZYckNQiiJKx+uD/SEsZxROhUJ9oapbvlO4AcTvsKpWYJrTcu+fRZqVnuIyVGl9 jYKj0Ak6hpXrXFyTUqlB7gTqoLh7WsciryS58clgHuq/EXnVhfUNTyHD5okshQpi6rXWojN0uNj 7n40m2/1K+gF5txnpcb9g7S3ddd14JNokJpvSIGWYqlw3GGxYyXhsnZeAnOJRk65gOgsVCb2T0i NxV2hDrWqXn+nDPowspcHnTbTT1OFxqap6bfLfA100A11FgGeuwo7xJ+aGIA3J X-Received: by 2002:a05:622a:1814:b0:509:f18:84eb with SMTP id d75a77b69052e-5090f1896b0mr102868301cf.43.1773101832110; Mon, 09 Mar 2026 17:17:12 -0700 (PDT) Received: from smtp-us-east1-p01-i01-si01.dlp.protect.broadcom.com (address-144-49-247-29.dlp.protect.broadcom.com. [144.49.247.29]) by smtp-relay.gmail.com with ESMTPS id d75a77b69052e-5091702ef5dsm5200041cf.4.2026.03.09.17.17.11 for (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Mon, 09 Mar 2026 17:17:12 -0700 (PDT) X-Relaying-Domain: broadcom.com X-CFilter-Loop: Reflected Received: by mail-dy1-f198.google.com with SMTP id 5a478bee46e88-2be2f742c46so61826205eec.0 for ; Mon, 09 Mar 2026 17:17:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=broadcom.com; s=google; t=1773101831; x=1773706631; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=sDBpeRPGQkSyveVh0r0L9u3UrQSmg15QNxwWzbeb67Y=; b=f4X3Syj70+mcQhcPejd5U3SiSEt8rAUQHH17KUHO3peKSg/p37rlk0UTndji55vc4X QQf7GVwJDfinGEkZGd3lfEl68ZqrqPq9zFPahWgjB6jGW227Qyg7PxtEKmCTM9SSDj9t zW2okiap1kKYZt0J6OXAUFxes45BzYStpCYrc= X-Forwarded-Encrypted: i=1; AJvYcCXu6ds5ajTgrZ3ErwipeCOgi2yaGAzTZXL+E4AxN+G6FE8WyqTt7h8YY9dXpX866/J4t+CWGd2Jbk0BLh4=@vger.kernel.org X-Received: by 2002:a05:7300:8623:b0:2be:2409:6c6e with SMTP id 5a478bee46e88-2be4de767damr6029552eec.5.1773101830116; Mon, 09 Mar 2026 17:17:10 -0700 (PDT) X-Received: by 2002:a05:7300:8623:b0:2be:2409:6c6e with SMTP id 5a478bee46e88-2be4de767damr6029533eec.5.1773101829206; Mon, 09 Mar 2026 17:17:09 -0700 (PDT) Received: from amakhalov-build-vm.. ([192.19.161.250]) by smtp.gmail.com with ESMTPSA id 5a478bee46e88-2be4f948324sm12678991eec.17.2026.03.09.17.17.08 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 09 Mar 2026 17:17:08 -0700 (PDT) From: Alexey Makhalov To: x86@kernel.org, virtualization@lists.linux.dev, bp@alien8.de, hpa@zytor.com, dave.hansen@linux.intel.com, mingo@redhat.com, tglx@linutronix.de Cc: ajay.kaher@broadcom.com, brennan.lamoreaux@broadcom.com, bo.gan@broadcom.com, bcm-kernel-feedback-list@broadcom.com, linux-kernel@vger.kernel.org, kas@kernel.org, rick.p.edgecombe@intel.com, linux-coco@lists.linux.dev, Alexey Makhalov , Linus Torvalds Subject: [PATCH v2 1/4] x86/vmware: Introduce common vmware_hypercall() Date: Mon, 9 Mar 2026 23:52:47 +0000 Message-ID: <20260309235250.2611115-2-alexey.makhalov@broadcom.com> X-Mailer: git-send-email 2.43.7 In-Reply-To: <20260309235250.2611115-1-alexey.makhalov@broadcom.com> References: <20260307004238.1181299-1-alexey.makhalov@broadcom.com> <20260309235250.2611115-1-alexey.makhalov@broadcom.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-DetectorID-Processed: b00c1d49-9d2e-4205-b15f-d015386d3d5e Content-Type: text/plain; charset="utf-8" Introduce vmware_hypercall(), a unified low-bandwidth VMware hypercall API, and convert the static inlines vmware_hypercallX() family into thin wrappers on top of it. vmware_hypercall() is implemented as a static call with four backend implementations: backdoor, vmcall, vmmcall, and tdxcall. All share the same logical API but differ in their underlying register mappings. By updating the static call target early during boot, before the first hypercall is issued, the !alternatives_patched case no longer needs to be handled. This allows removal of vmware_hypercall_slow(). The new API implements the widest practical hypercall use case: up to six input and six output arguments. While this may be slightly less efficient due to clobbering all six registers and moving unused arguments - it avoids subtle ABI issues, including cases where other hypervisors implementing VMware hypercalls corrupt registers. See QEMU issue #3293 ("vmmouse driver corrupts upper 32 bits of registers on x86-64") for an example of such behavior. Additionally, enhance the VMware hypercall ABI documentation in . Link: https://gitlab.com/qemu-project/qemu/-/issues/3293 Suggested-by: Linus Torvalds Signed-off-by: Alexey Makhalov --- arch/x86/include/asm/vmware.h | 274 ++++++++++++++------------------- arch/x86/kernel/cpu/vmware.c | 276 +++++++++++++++++++--------------- 2 files changed, 267 insertions(+), 283 deletions(-) diff --git a/arch/x86/include/asm/vmware.h b/arch/x86/include/asm/vmware.h index 4220dae14a2d..6a084e088b30 100644 --- a/arch/x86/include/asm/vmware.h +++ b/arch/x86/include/asm/vmware.h @@ -3,48 +3,84 @@ #define _ASM_X86_VMWARE_H =20 #include -#include #include +#include =20 /* * VMware hypercall ABI. * - * - Low bandwidth (LB) hypercalls (I/O port based, vmcall and vmmcall) - * have up to 6 input and 6 output arguments passed and returned using - * registers: %eax (arg0), %ebx (arg1), %ecx (arg2), %edx (arg3), - * %esi (arg4), %edi (arg5). - * The following input arguments must be initialized by the caller: - * arg0 - VMWARE_HYPERVISOR_MAGIC - * arg2 - Hypercall command - * arg3 bits [15:0] - Port number, LB and direction flags + * - Low bandwidth (LB) hypercalls: I/O port based (aka backdoor), vmcall = and + * vmmcall have up to 6 input and 6 output on registers arguments, with the + * register mapping: + * +------+----------------------------------------+-----------------+ + * | Reg | Input argument | Output argument | + * +=3D=3D=3D=3D=3D=3D+=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D+= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D+ + * | %eax | VMWARE_HYPERVISOR_MAGIC | out0 | + * +------+----------------------------------------+-----------------+ + * | %ebx | (in1) | out1 | + * +------+----------------------------------------+-----------------+ + * | %ecx | (cmd) - Hypercall command | out2 | + * +------+----------------------------------------+-----------------+ + * | %edx | Bits [15:0] - Port number for backdoor | out3 | + * | | Zero for vmcall/vmmcall | | + * | | Bits [31:16] - (in3) | | + * +------+----------------------------------------+-----------------+ + * | %esi | (in4) | out4 | + * +------+----------------------------------------+-----------------+ + * | %edi | (in5) | out5 | + * +------+----------------------------------------+-----------------+ * - * - Low bandwidth TDX hypercalls (x86_64 only) are similar to LB - * hypercalls. They also have up to 6 input and 6 output on registers - * arguments, with different argument to register mapping: - * %r12 (arg0), %rbx (arg1), %r13 (arg2), %rdx (arg3), - * %rsi (arg4), %rdi (arg5). + * - Low bandwidth TDX hypercalls (x86_64 only) are similar to LB hypercal= ls. + * They also have up to 6 input and 6 output on registers arguments, with + * different argument to register mapping: + * +------+----------------------------------------+-----------------+ + * | Reg | Input argument | Output argument | + * +=3D=3D=3D=3D=3D=3D+=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D+= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D+ + * | %r12 | VMWARE_HYPERVISOR_MAGIC | out0 | + * +------+----------------------------------------+-----------------+ + * | %ebx | (in1) | out1 | + * +------+----------------------------------------+-----------------+ + * | %r13 | (cmd) - Hypercall command | out2 | + * +------+----------------------------------------+-----------------+ + * | %edx | Bits [15:0] - Must be zero | out3 | + * | | Bits [31:16] - (in3) | | + * +------+----------------------------------------+-----------------+ + * | %esi | (in4) | out4 | + * +------+----------------------------------------+-----------------+ + * | %edi | (in5) | out5 | + * +------+----------------------------------------+-----------------+ * - * - High bandwidth (HB) hypercalls are I/O port based only. They have - * up to 7 input and 7 output arguments passed and returned using - * registers: %eax (arg0), %ebx (arg1), %ecx (arg2), %edx (arg3), - * %esi (arg4), %edi (arg5), %ebp (arg6). - * The following input arguments must be initialized by the caller: - * arg0 - VMWARE_HYPERVISOR_MAGIC - * arg1 - Hypercall command - * arg3 bits [15:0] - Port number, HB and direction flags + * - High bandwidth (HB) hypercalls are I/O port based only. They have up = to 7 + * input and 7 output on reegister arguments with the following mapping: + * +------+----------------------------------------+-----------------+ + * | Reg | Input argument | Output argument | + * +=3D=3D=3D=3D=3D=3D+=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D+= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D+ + * | %eax | VMWARE_HYPERVISOR_MAGIC | out0 | + * +------+----------------------------------------+-----------------+ + * | %ebx | (cmd) - Hypercall command | out1 | + * +------+----------------------------------------+-----------------+ + * | %ebx | (in2) | out2 | + * +------+----------------------------------------+-----------------+ + * | %edx | Bits [15:0] - Port number and HB flag | out3 | + * | | Bits [31:16] - (in3) | | + * +------+----------------------------------------+-----------------+ + * | %esi | (in4) | out4 | + * +------+----------------------------------------+-----------------+ + * | %edi | (in5) | out5 | + * +------+----------------------------------------+-----------------+ + * | %ebp | (in6) | out6 | + * +------+----------------------------------------+-----------------+ * - * For compatibility purposes, x86_64 systems use only lower 32 bits - * for input and output arguments. + * For compatibility purposes, x86_64 systems use only lower 32 bits for i= nput + * and output arguments. * - * The hypercall definitions differ in the low word of the %edx (arg3) - * in the following way: the old I/O port based interface uses the port - * number to distinguish between high- and low bandwidth versions, and - * uses IN/OUT instructions to define transfer direction. + * The hypercall definitions differ in the low word of the %edx (arg3) in = the + * following way: the old I/O port based interface uses the port number, t= he + * bandwidth mode flag, and uses IN/OUT instructions to define transfer + * direction. * - * The new vmcall interface instead uses a set of flags to select - * bandwidth mode and transfer direction. The flags should be loaded - * into arg3 by any user and are automatically replaced by the port - * number if the I/O port method is used. + * The new vmcall interface instead uses a set of flags to select bandwidth + * mode and transfer direction. */ =20 #define VMWARE_HYPERVISOR_HB BIT(0) @@ -70,103 +106,64 @@ #define CPUID_VMWARE_FEATURES_ECX_VMMCALL BIT(0) #define CPUID_VMWARE_FEATURES_ECX_VMCALL BIT(1) =20 -extern unsigned long vmware_hypercall_slow(unsigned long cmd, - unsigned long in1, unsigned long in3, - unsigned long in4, unsigned long in5, - u32 *out1, u32 *out2, u32 *out3, - u32 *out4, u32 *out5); - #define VMWARE_TDX_VENDOR_LEAF 0x1af7e4909ULL #define VMWARE_TDX_HCALL_FUNC 1 =20 -extern unsigned long vmware_tdx_hypercall(unsigned long cmd, - unsigned long in1, unsigned long in3, - unsigned long in4, unsigned long in5, - u32 *out1, u32 *out2, u32 *out3, - u32 *out4, u32 *out5); +unsigned long dummy_vmware_hypercall(unsigned long cmd, + unsigned long in1, unsigned long in3, + unsigned long in4, unsigned long in5, + u32 *out1, u32 *out2, u32 *out3, + u32 *out4, u32 *out5); =20 /* - * The low bandwidth call. The low word of %edx is presumed to have OUT bit - * set. The high word of %edx may contain input data from the caller. + * Low bandwidth (LB) VMware hypercall. + * + * It is backed by the backdoor, vmcall, vmmcall or tdx call implementatio= n. + * + * Use inX/outX arguments naming as the register mappings vary between + * different implementations. See VMware hypercall ABI above. + * These 10 arguments could be nicely wrapped in in/out structures, but it + * will introduce unnecessary structs copy in vmware_tdx_hypercall(). + * + * NOTE: + * Do not merge vmware_{backdoor,vmcall,vmmcall}_hypercall implementations + * using alternative instructions. Such patching mechanism can not be used + * in vmware_hypercall path, as the first hypercall will be called much + * before the apply_alternatives(). See vmware_platform_setup(). */ -#define VMWARE_HYPERCALL \ - ALTERNATIVE_2("movw %[port], %%dx\n\t" \ - "inl (%%dx), %%eax", \ - "vmcall", X86_FEATURE_VMCALL, \ - "vmmcall", X86_FEATURE_VMW_VMMCALL) +DECLARE_STATIC_CALL(vmware_hypercall, dummy_vmware_hypercall); =20 +/* + * Set of commonly used vmware_hypercallX functions - wrappers on top of t= he + * vmware_hypercall. + */ static inline unsigned long vmware_hypercall1(unsigned long cmd, unsigned long in1) { - unsigned long out0; - - if (cpu_feature_enabled(X86_FEATURE_TDX_GUEST)) - return vmware_tdx_hypercall(cmd, in1, 0, 0, 0, - NULL, NULL, NULL, NULL, NULL); - - if (unlikely(!alternatives_patched) && !__is_defined(MODULE)) - return vmware_hypercall_slow(cmd, in1, 0, 0, 0, - NULL, NULL, NULL, NULL, NULL); + u32 out1, out2, out3, out4, out5; =20 - asm_inline volatile (VMWARE_HYPERCALL - : "=3Da" (out0) - : [port] "i" (VMWARE_HYPERVISOR_PORT), - "a" (VMWARE_HYPERVISOR_MAGIC), - "b" (in1), - "c" (cmd), - "d" (0) - : "cc", "memory"); - return out0; + return static_call_mod(vmware_hypercall)(cmd, in1, 0, 0, 0, + &out1, &out2, &out3, &out4, &out5); } =20 static inline unsigned long vmware_hypercall3(unsigned long cmd, unsigned long in1, u32 *out1, u32 *out2) { - unsigned long out0; - - if (cpu_feature_enabled(X86_FEATURE_TDX_GUEST)) - return vmware_tdx_hypercall(cmd, in1, 0, 0, 0, - out1, out2, NULL, NULL, NULL); - - if (unlikely(!alternatives_patched) && !__is_defined(MODULE)) - return vmware_hypercall_slow(cmd, in1, 0, 0, 0, - out1, out2, NULL, NULL, NULL); + u32 out3, out4, out5; =20 - asm_inline volatile (VMWARE_HYPERCALL - : "=3Da" (out0), "=3Db" (*out1), "=3Dc" (*out2) - : [port] "i" (VMWARE_HYPERVISOR_PORT), - "a" (VMWARE_HYPERVISOR_MAGIC), - "b" (in1), - "c" (cmd), - "d" (0) - : "di", "si", "cc", "memory"); - return out0; + return static_call_mod(vmware_hypercall)(cmd, in1, 0, 0, 0, + out1, out2, &out3, &out4, &out5); } =20 static inline unsigned long vmware_hypercall4(unsigned long cmd, unsigned long in1, u32 *out1, u32 *out2, u32 *out3) { - unsigned long out0; - - if (cpu_feature_enabled(X86_FEATURE_TDX_GUEST)) - return vmware_tdx_hypercall(cmd, in1, 0, 0, 0, - out1, out2, out3, NULL, NULL); - - if (unlikely(!alternatives_patched) && !__is_defined(MODULE)) - return vmware_hypercall_slow(cmd, in1, 0, 0, 0, - out1, out2, out3, NULL, NULL); + u32 out4, out5; =20 - asm_inline volatile (VMWARE_HYPERCALL - : "=3Da" (out0), "=3Db" (*out1), "=3Dc" (*out2), "=3Dd" (*out3) - : [port] "i" (VMWARE_HYPERVISOR_PORT), - "a" (VMWARE_HYPERVISOR_MAGIC), - "b" (in1), - "c" (cmd), - "d" (0) - : "di", "si", "cc", "memory"); - return out0; + return static_call_mod(vmware_hypercall)(cmd, in1, 0, 0, 0, + out1, out2, out3, &out4, &out5); } =20 static inline @@ -174,27 +171,10 @@ unsigned long vmware_hypercall5(unsigned long cmd, un= signed long in1, unsigned long in3, unsigned long in4, unsigned long in5, u32 *out2) { - unsigned long out0; - - if (cpu_feature_enabled(X86_FEATURE_TDX_GUEST)) - return vmware_tdx_hypercall(cmd, in1, in3, in4, in5, - NULL, out2, NULL, NULL, NULL); + u32 out1, out3, out4, out5; =20 - if (unlikely(!alternatives_patched) && !__is_defined(MODULE)) - return vmware_hypercall_slow(cmd, in1, in3, in4, in5, - NULL, out2, NULL, NULL, NULL); - - asm_inline volatile (VMWARE_HYPERCALL - : "=3Da" (out0), "=3Dc" (*out2) - : [port] "i" (VMWARE_HYPERVISOR_PORT), - "a" (VMWARE_HYPERVISOR_MAGIC), - "b" (in1), - "c" (cmd), - "d" (in3), - "S" (in4), - "D" (in5) - : "cc", "memory"); - return out0; + return static_call_mod(vmware_hypercall)(cmd, in1, in3, in4, in5, + &out1, out2, &out3, &out4, &out5); } =20 static inline @@ -202,26 +182,10 @@ unsigned long vmware_hypercall6(unsigned long cmd, un= signed long in1, unsigned long in3, u32 *out2, u32 *out3, u32 *out4, u32 *out5) { - unsigned long out0; - - if (cpu_feature_enabled(X86_FEATURE_TDX_GUEST)) - return vmware_tdx_hypercall(cmd, in1, in3, 0, 0, - NULL, out2, out3, out4, out5); + u32 out1; =20 - if (unlikely(!alternatives_patched) && !__is_defined(MODULE)) - return vmware_hypercall_slow(cmd, in1, in3, 0, 0, - NULL, out2, out3, out4, out5); - - asm_inline volatile (VMWARE_HYPERCALL - : "=3Da" (out0), "=3Dc" (*out2), "=3Dd" (*out3), "=3DS" (*out4), - "=3DD" (*out5) - : [port] "i" (VMWARE_HYPERVISOR_PORT), - "a" (VMWARE_HYPERVISOR_MAGIC), - "b" (in1), - "c" (cmd), - "d" (in3) - : "cc", "memory"); - return out0; + return static_call_mod(vmware_hypercall)(cmd, in1, in3, 0, 0, + &out1, out2, out3, out4, out5); } =20 static inline @@ -230,27 +194,10 @@ unsigned long vmware_hypercall7(unsigned long cmd, un= signed long in1, unsigned long in5, u32 *out1, u32 *out2, u32 *out3) { - unsigned long out0; - - if (cpu_feature_enabled(X86_FEATURE_TDX_GUEST)) - return vmware_tdx_hypercall(cmd, in1, in3, in4, in5, - out1, out2, out3, NULL, NULL); + u32 out4, out5; =20 - if (unlikely(!alternatives_patched) && !__is_defined(MODULE)) - return vmware_hypercall_slow(cmd, in1, in3, in4, in5, - out1, out2, out3, NULL, NULL); - - asm_inline volatile (VMWARE_HYPERCALL - : "=3Da" (out0), "=3Db" (*out1), "=3Dc" (*out2), "=3Dd" (*out3) - : [port] "i" (VMWARE_HYPERVISOR_PORT), - "a" (VMWARE_HYPERVISOR_MAGIC), - "b" (in1), - "c" (cmd), - "d" (in3), - "S" (in4), - "D" (in5) - : "cc", "memory"); - return out0; + return static_call_mod(vmware_hypercall)(cmd, in1, in3, in4, in5, + out1, out2, out3, &out4, &out5); } =20 #ifdef CONFIG_X86_64 @@ -322,6 +269,5 @@ unsigned long vmware_hypercall_hb_in(unsigned long cmd,= unsigned long in2, return out0; } #undef VMW_BP_CONSTRAINT -#undef VMWARE_HYPERCALL =20 #endif diff --git a/arch/x86/kernel/cpu/vmware.c b/arch/x86/kernel/cpu/vmware.c index a3e6936839b1..93acd3414e37 100644 --- a/arch/x86/kernel/cpu/vmware.c +++ b/arch/x86/kernel/cpu/vmware.c @@ -64,70 +64,140 @@ struct vmware_steal_time { }; =20 static unsigned long vmware_tsc_khz __ro_after_init; -static u8 vmware_hypercall_mode __ro_after_init; - -unsigned long vmware_hypercall_slow(unsigned long cmd, - unsigned long in1, unsigned long in3, - unsigned long in4, unsigned long in5, - u32 *out1, u32 *out2, u32 *out3, - u32 *out4, u32 *out5) -{ - unsigned long out0, rbx, rcx, rdx, rsi, rdi; - - switch (vmware_hypercall_mode) { - case CPUID_VMWARE_FEATURES_ECX_VMCALL: - asm_inline volatile ("vmcall" - : "=3Da" (out0), "=3Db" (rbx), "=3Dc" (rcx), - "=3Dd" (rdx), "=3DS" (rsi), "=3DD" (rdi) - : "a" (VMWARE_HYPERVISOR_MAGIC), - "b" (in1), - "c" (cmd), - "d" (in3), - "S" (in4), - "D" (in5) - : "cc", "memory"); - break; - case CPUID_VMWARE_FEATURES_ECX_VMMCALL: - asm_inline volatile ("vmmcall" - : "=3Da" (out0), "=3Db" (rbx), "=3Dc" (rcx), - "=3Dd" (rdx), "=3DS" (rsi), "=3DD" (rdi) - : "a" (VMWARE_HYPERVISOR_MAGIC), - "b" (in1), - "c" (cmd), - "d" (in3), - "S" (in4), - "D" (in5) - : "cc", "memory"); - break; - default: - asm_inline volatile ("movw %[port], %%dx; inl (%%dx), %%eax" - : "=3Da" (out0), "=3Db" (rbx), "=3Dc" (rcx), - "=3Dd" (rdx), "=3DS" (rsi), "=3DD" (rdi) - : [port] "i" (VMWARE_HYPERVISOR_PORT), - "a" (VMWARE_HYPERVISOR_MAGIC), - "b" (in1), - "c" (cmd), - "d" (in3), - "S" (in4), - "D" (in5) - : "cc", "memory"); - break; - } +static u8 vmware_hypercall_mode __initdata; + +static unsigned long vmware_backdoor_hypercall(unsigned long cmd, + unsigned long in1, unsigned long in3, + unsigned long in4, unsigned long in5, + u32 *out1, u32 *out2, u32 *out3, + u32 *out4, u32 *out5) +{ + unsigned long out0; + + /* The low word of in3(%edx) must have the backdoor port number */ + in3 =3D (in3 & ~0xffff) | VMWARE_HYPERVISOR_PORT; + + asm_inline volatile ("inl (%%dx), %%eax" + : "=3Da" (out0), "=3Db" (*out1), "=3Dc" (*out2), + "=3Dd" (*out3), "=3DS" (*out4), "=3DD" (*out5) + : "a" (VMWARE_HYPERVISOR_MAGIC), + "b" (in1), + "c" (cmd), + "d" (in3), + "S" (in4), + "D" (in5) + : "cc", "memory"); =20 - if (out1) - *out1 =3D rbx; - if (out2) - *out2 =3D rcx; - if (out3) - *out3 =3D rdx; - if (out4) - *out4 =3D rsi; - if (out5) - *out5 =3D rdi; + return out0; +} + +static unsigned long vmware_vmcall_hypercall(unsigned long cmd, + unsigned long in1, unsigned long in3, + unsigned long in4, unsigned long in5, + u32 *out1, u32 *out2, u32 *out3, + u32 *out4, u32 *out5) +{ + unsigned long out0; + + /* The low word of in3(%edx) must be zero: LB, IN */ + in3 &=3D ~0xffff; + + asm_inline volatile ("vmcall" + : "=3Da" (out0), "=3Db" (*out1), "=3Dc" (*out2), + "=3Dd" (*out3), "=3DS" (*out4), "=3DD" (*out5) + : "a" (VMWARE_HYPERVISOR_MAGIC), + "b" (in1), + "c" (cmd), + "d" (in3), + "S" (in4), + "D" (in5) + : "cc", "memory"); =20 return out0; } =20 +static unsigned long vmware_vmmcall_hypercall(unsigned long cmd, + unsigned long in1, unsigned long in3, + unsigned long in4, unsigned long in5, + u32 *out1, u32 *out2, u32 *out3, + u32 *out4, u32 *out5) +{ + unsigned long out0; + + /* The low word of in3(%edx) must be zero: LB, IN */ + in3 &=3D ~0xffff; + + asm_inline volatile ("vmmcall" + : "=3Da" (out0), "=3Db" (*out1), "=3Dc" (*out2), + "=3Dd" (*out3), "=3DS" (*out4), "=3DD" (*out5) + : "a" (VMWARE_HYPERVISOR_MAGIC), + "b" (in1), + "c" (cmd), + "d" (in3), + "S" (in4), + "D" (in5) + : "cc", "memory"); + + return out0; +} + +/* + * TDCALL[TDG.VP.VMCALL] uses %rax (arg0) and %rcx (arg2). Therefore, + * we remap those registers to %r12 and %r13, respectively. + */ +static unsigned long vmware_tdx_hypercall(unsigned long cmd, + unsigned long in1, unsigned long in3, + unsigned long in4, unsigned long in5, + u32 *out1, u32 *out2, u32 *out3, + u32 *out4, u32 *out5) +{ +#ifdef CONFIG_INTEL_TDX_GUEST + struct tdx_module_args args =3D {}; + + if (!hypervisor_is_type(X86_HYPER_VMWARE)) { + pr_warn_once("Incorrect usage\n"); + return ULONG_MAX; + } + + if (cmd & ~VMWARE_CMD_MASK) { + pr_warn_once("Out of range command %lx\n", cmd); + return ULONG_MAX; + } + + args.rbx =3D in1; + /* The low word of in3(%rdx) must be zero: LB, IN */ + args.rdx =3D in3 & ~0xffff; + args.rsi =3D in4; + args.rdi =3D in5; + args.r10 =3D VMWARE_TDX_VENDOR_LEAF; + args.r11 =3D VMWARE_TDX_HCALL_FUNC; + args.r12 =3D VMWARE_HYPERVISOR_MAGIC; + args.r13 =3D cmd; + /* CPL */ + args.r15 =3D 0; + + __tdx_hypercall(&args); + + *out1 =3D args.rbx; + *out2 =3D args.r13; + *out3 =3D args.rdx; + *out4 =3D args.rsi; + *out5 =3D args.rdi; + + return args.r12; +#else + return ULONG_MAX; +#endif +} + + +DEFINE_STATIC_CALL(vmware_hypercall, vmware_backdoor_hypercall); +EXPORT_STATIC_CALL_GPL(vmware_hypercall); + +/* + * Perform backdoor probbing of the hypervisor when + * X86_FEATURE_HYPERVISOR bit is not set. + */ static inline int __vmware_platform(void) { u32 eax, ebx, ecx; @@ -397,11 +467,35 @@ static void __init vmware_set_capabilities(void) setup_force_cpu_cap(X86_FEATURE_VMW_VMMCALL); } =20 +static void __init vmware_select_hypercall(void) +{ + char *mode; + + if (IS_ENABLED(CONFIG_INTEL_TDX_GUEST) && + cpu_feature_enabled(X86_FEATURE_TDX_GUEST)) { + static_call_update(vmware_hypercall, vmware_tdx_hypercall); + mode =3D "tdcall"; + } else if (vmware_hypercall_mode =3D=3D CPUID_VMWARE_FEATURES_ECX_VMCALL)= { + static_call_update(vmware_hypercall, vmware_vmcall_hypercall); + mode =3D "vmcall"; + } else if (vmware_hypercall_mode =3D=3D CPUID_VMWARE_FEATURES_ECX_VMMCALL= ) { + static_call_update(vmware_hypercall, vmware_vmmcall_hypercall); + mode =3D "vmmcall"; + } else { + mode =3D "backdoor"; + } + + pr_info("hypercall mode: %s\n", mode); +} + static void __init vmware_platform_setup(void) { u32 eax, ebx, ecx; u64 lpj, tsc_khz; =20 + /* Update vmware_hypercall() before the first use. */ + vmware_select_hypercall(); + eax =3D vmware_hypercall3(VMWARE_CMD_GETHZ, UINT_MAX, &ebx, &ecx); =20 if (ebx !=3D UINT_MAX) { @@ -443,7 +537,7 @@ static void __init vmware_platform_setup(void) vmware_set_capabilities(); } =20 -static u8 __init vmware_select_hypercall(void) +static u8 __init get_hypercall_mode(void) { int eax, ebx, ecx, edx; =20 @@ -456,8 +550,8 @@ static u8 __init vmware_select_hypercall(void) * While checking the dmi string information, just checking the product * serial key should be enough, as this will always have a VMware * specific string when running under VMware hypervisor. - * If !boot_cpu_has(X86_FEATURE_HYPERVISOR), vmware_hypercall_mode - * intentionally defaults to 0. + * If !boot_cpu_has(X86_FEATURE_HYPERVISOR), __vmware_platform() + * intentionally defaults to backdoor hypercall. */ static u32 __init vmware_platform(void) { @@ -470,11 +564,7 @@ static u32 __init vmware_platform(void) if (!memcmp(hyper_vendor_id, "VMwareVMware", 12)) { if (eax >=3D CPUID_VMWARE_FEATURES_LEAF) vmware_hypercall_mode =3D - vmware_select_hypercall(); - - pr_info("hypercall mode: 0x%02x\n", - (unsigned int) vmware_hypercall_mode); - + get_hypercall_mode(); return CPUID_VMWARE_INFO_LEAF; } } else if (dmi_available && dmi_name_in_serial("VMware") && @@ -494,58 +584,6 @@ static bool __init vmware_legacy_x2apic_available(void) (eax & GETVCPU_INFO_LEGACY_X2APIC); } =20 -#ifdef CONFIG_INTEL_TDX_GUEST -/* - * TDCALL[TDG.VP.VMCALL] uses %rax (arg0) and %rcx (arg2). Therefore, - * we remap those registers to %r12 and %r13, respectively. - */ -unsigned long vmware_tdx_hypercall(unsigned long cmd, - unsigned long in1, unsigned long in3, - unsigned long in4, unsigned long in5, - u32 *out1, u32 *out2, u32 *out3, - u32 *out4, u32 *out5) -{ - struct tdx_module_args args =3D {}; - - if (!hypervisor_is_type(X86_HYPER_VMWARE)) { - pr_warn_once("Incorrect usage\n"); - return ULONG_MAX; - } - - if (cmd & ~VMWARE_CMD_MASK) { - pr_warn_once("Out of range command %lx\n", cmd); - return ULONG_MAX; - } - - args.rbx =3D in1; - args.rdx =3D in3; - args.rsi =3D in4; - args.rdi =3D in5; - args.r10 =3D VMWARE_TDX_VENDOR_LEAF; - args.r11 =3D VMWARE_TDX_HCALL_FUNC; - args.r12 =3D VMWARE_HYPERVISOR_MAGIC; - args.r13 =3D cmd; - /* CPL */ - args.r15 =3D 0; - - __tdx_hypercall(&args); - - if (out1) - *out1 =3D args.rbx; - if (out2) - *out2 =3D args.r13; - if (out3) - *out3 =3D args.rdx; - if (out4) - *out4 =3D args.rsi; - if (out5) - *out5 =3D args.rdi; - - return args.r12; -} -EXPORT_SYMBOL_GPL(vmware_tdx_hypercall); -#endif - #ifdef CONFIG_AMD_MEM_ENCRYPT static void vmware_sev_es_hcall_prepare(struct ghcb *ghcb, struct pt_regs *regs) --=20 2.43.7