From nobody Fri Sep 25 06:02:54 2026 Received: from mail-vk1-f228.google.com (mail-vk1-f228.google.com [209.85.221.228]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 97BDE3B9956 for ; Wed, 16 Sep 2026 05:05:56 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.221.228 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789535158; cv=none; b=U9HvKcA5Y8URVh8v57QC6ZOmVrD9++OMUGYjEM0aoHKdDoVtXE8nJVOfU420gb8noE1+xciqGq4qgTGkYSsxWR2vJZjBsgZZukCXzbuoY86/OUU+vy+++vVuv1qFB5fYGQupWXqw/+fb8s148G5zFn2rd6TGKJY8GIvWYJSonzU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789535158; c=relaxed/simple; bh=LWc2s4qZvBb3TJVfltvRwFaklDcNXyLXGRlUr4C/OwI=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=jfQLGxRwqPhQ4pj7k4a5GwFkMjx6nBd4XVz1N3rGFxmlHeCs49TGHTncvhuuT4cdcSa+AoDVgpf9oOWz06l8nQrQjfh9KW8Xf1rBm1zZ56eqvL+xj+89x4q5cb8TUQh7OyIhsYRwbz9b0koomdtSTYXVeUfbmqN3Gh00BiLjvuA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=broadcom.com; spf=fail smtp.mailfrom=broadcom.com; dkim=pass (1024-bit key) header.d=broadcom.com header.i=@broadcom.com header.b=AgIxbS4l; arc=none smtp.client-ip=209.85.221.228 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=broadcom.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=broadcom.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=broadcom.com header.i=@broadcom.com header.b="AgIxbS4l" Received: by mail-vk1-f228.google.com with SMTP id 71dfb90a1353d-5c847eaa0aaso684715e0c.0 for ; Tue, 15 Sep 2026 22:05:56 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1789535155; x=1790139955; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:dkim-signature:x-gm-gg :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to :content-type; bh=6awU6L1NSAdZve1Go4DAFmiBVEoVbQdcfNtuUS9ma4s=; b=qCm6cJ9BE3o1PogY94WgdiZLVqh5sVw0cJbXyjm6s1I5yDtZ6rutuN+hAAge7sURI3 Qt3HqvjBkE80+6WRlECsZEnVZpw8YyXgOT2mEArbB/uF7CdXk1DJYC80YpVbztZN7lWY H7HZQQVtjjRTXeoQTcUIl6sIrT3QIsT7leqddbpRgAJ+QmNVp5IYVC33z1vjdsI/it0g PEtIg8nLSPhw2geQAwgpCYwWXmRFrgNloRtsVI7L+J8GXIaIu+xhUzFqIbsAFgFTlo6t aMROZm4X7ljHtSQKlR+jHIjUifpnNDnQ8AflJwyWDIl8BX8mExk19roywvHqurediSY4 lrLw== X-Forwarded-Encrypted: i=1; AKwUvByVb8J3mZqjHea6g76qOtplZ6VDUoOd0ZNOBA4fFScJeKabDyJWZ3fjmSooYtJLwddlZnAQ3kMWc2+p9To=@vger.kernel.org X-Gm-Message-State: AFuF++ku/bpNpLzfbwSJH5c3V7XLg3Rloiuysgru1OiVg1aUarsLeRwZ MK9asSsNdr+VroeirQRd6CE48DjdO28+kFFN25+gSV4O2ZkxIQwV9qZGtTkr99o0yVztXsmbZFf pOdC+bptkxIySRdQwC25rbknkBWDawF2CYFmGix3zJw/N0JzGMg2vJUqCgkVDz1l48InRkf4W0p tUOjZQDnbHr9whgAx2bU8Gg1imkqwjirjmACMnvDpQUNpdGXi8+wzCxs7bl2XXcaBcfOFXeLkRf AZVM+eiTJkTAhnp X-Gm-Gg: AYBFou2/StuMh6V6mr3kD3dyDqN7ubU7LrYwojjADA/vZGG7Aup00MXQ4fkidzEhvLQ dEoFxyv2Z9naCXuUbuo3GMOya4mTJAdPxQ0KBYZghqibfcJd4OihHd0VMckk8I9pOsAnT9cuhLa VhZdn2FtX1VRBcogadfq33ed+sWp3CNe0J6iRZd8ePkiRi8Fqohu4gIZ8CDQ/353syvvTN5sH/W eBflvLaq78A3EC2Om/rwHb62cJoxxNIY5lx8ywl0DtW5o2ab8ZTszaDv09xQ+vZcLTdi9dDhbLw 2gYG06xyjd2iPreOcqgXvWdvwqyyR98NV9zcbblt2iAJUkX5WzHWCj4Gn7bxFSauGx/J7oXwvgQ DUC780zTKspDECSvF8esTFRD8RgFJYj+LXfFnOtmA0+EycK9yCBccx3I8NAD4eCrGclOty0Zmui X6/3njqjT151Znao5pi0lRDfmDtWa3QDO0 X-Received: by 2002:a05:6122:786:b0:5bc:42be:f758 with SMTP id 71dfb90a1353d-5c99ac5f757mr933546e0c.3.1789535155282; Tue, 15 Sep 2026 22:05:55 -0700 (PDT) Received: from smtp-us-east1-p01-i01-si01.dlp.protect.broadcom.com (address-144-49-247-98.dlp.protect.broadcom.com. [144.49.247.98]) by smtp-relay.gmail.com with ESMTPS id 71dfb90a1353d-5c9988d02casm696570e0c.2.2026.09.15.22.05.54 for (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Tue, 15 Sep 2026 22:05:55 -0700 (PDT) X-Relaying-Domain: broadcom.com X-CFilter-Loop: Reflected Received: by mail-pg1-f199.google.com with SMTP id 41be03b00d2f7-c89704da8c7so7022242a12.0 for ; Tue, 15 Sep 2026 22:05:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=broadcom.com; s=google; t=1789535154; x=1790139954; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=6awU6L1NSAdZve1Go4DAFmiBVEoVbQdcfNtuUS9ma4s=; b=AgIxbS4l+sMGzOQKUymNIgMNamCZBEsk5FlD0p+PGzS5LljSWOGjmA8Qbd3KpX/tGK Ga9JsNwG9oSpFcbKpB7z/4M7GGHaU04iUlAhCCTyIFsJLRXbRZ7XHhtNcme5T5sNuCN0 +CiXxYnC2nhU2Cy2+vrjWZjf/EgUCPiN6Zb9I= X-Forwarded-Encrypted: i=1; AKwUvBzqwPbypMU+0LAIscez0lOPw4cAzn7jyzaOFPE5jOo74RvbAUUIBt+LcW8QFaOvpr19yxdc3ohs7zOWr54=@vger.kernel.org X-Received: by 2002:a05:6a20:4388:b0:3cc:8f53:26c0 with SMTP id adf61e73a8af0-3dd5f57bbfamr2773496637.17.1789535153993; Tue, 15 Sep 2026 22:05:53 -0700 (PDT) X-Received: by 2002:a05:6a20:4388:b0:3cc:8f53:26c0 with SMTP id adf61e73a8af0-3dd5f57bbfamr2773437637.17.1789535153567; Tue, 15 Sep 2026 22:05:53 -0700 (PDT) Received: from vertex.localdomain ([192.19.144.250]) by smtp.gmail.com with ESMTPSA id 5a478bee46e88-33bf5e40494sm3951386eec.15.2026.09.15.22.05.50 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 15 Sep 2026 22:05:52 -0700 (PDT) From: Zack Rusin To: Borislav Petkov , Ajay Kaher , Alexey Makhalov , x86@kernel.org, Dennis Zhou , Tejun Heo , Arnd Bergmann , Kiryl Shutsemau , Rick Edgecombe Cc: Thomas Gleixner , Ingo Molnar , Dave Hansen , "H . Peter Anvin" , virtualization@lists.linux.dev, bcm-kernel-feedback-list@broadcom.com, linux-kernel@vger.kernel.org, Christoph Lameter , Andrew Morton , Tom Lendacky , Bo Gan , linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-coco@lists.linux.dev, kvm@vger.kernel.org, Zack Rusin Subject: [PATCH v1 1/2] percpu: Use X86_MEM_ENCRYPT for decrypted per-CPU data Date: Wed, 16 Sep 2026 01:05:39 -0400 Message-ID: <07cdda86e2c60b27a18023e2e5fae339b78db0ab.1789488039.git.zack.rusin@broadcom.com> X-Mailer: git-send-email 2.53.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-DetectorID-Processed: b00c1d49-9d2e-4205-b15f-d015386d3d5e Content-Type: text/plain; charset="utf-8" Decrypted per-CPU data is gated by AMD_MEM_ENCRYPT. With INTEL_TDX_GUEST=3Dy and AMD_MEM_ENCRYPT=3Dn it falls back to ordinary per-CPU storage, so converting a variable to shared memory can expose unrelated data on the same page. Gate the declarations and linker section on X86_MEM_ENCRYPT, which both AMD_MEM_ENCRYPT and INTEL_TDX_GUEST select. Existing AMD configurations are unchanged. Keep the decrypted subsection page-aligned at both ends. The per-CPU template and each allocated per-CPU unit are also page-aligned, so page conversion cannot expose ordinary per-CPU data. TDX-only configurations may need additional space for this alignment and isolation. Signed-off-by: Zack Rusin Acked-by: Kiryl Shutsemau (Meta) --- include/asm-generic/vmlinux.lds.h | 2 +- include/linux/percpu-defs.h | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/include/asm-generic/vmlinux.lds.h b/include/asm-generic/vmlinu= x.lds.h index b2988aa12f66..1448e791773e 100644 --- a/include/asm-generic/vmlinux.lds.h +++ b/include/asm-generic/vmlinux.lds.h @@ -1013,7 +1013,7 @@ * Note: We use a separate section so that only this section gets * decrypted to avoid exposing more than we wish. */ -#ifdef CONFIG_AMD_MEM_ENCRYPT +#ifdef CONFIG_X86_MEM_ENCRYPT #define PERCPU_DECRYPTED_SECTION \ . =3D ALIGN(PAGE_SIZE); \ *(.data..percpu..decrypted) \ diff --git a/include/linux/percpu-defs.h b/include/linux/percpu-defs.h index dbe3267a0a13..54ce1442d918 100644 --- a/include/linux/percpu-defs.h +++ b/include/linux/percpu-defs.h @@ -172,7 +172,7 @@ * Declaration/definition used for per-CPU variables that should be access= ed * as decrypted when memory encryption is enabled in the guest. */ -#ifdef CONFIG_AMD_MEM_ENCRYPT +#ifdef CONFIG_X86_MEM_ENCRYPT #define DECLARE_PER_CPU_DECRYPTED(type, name) \ DECLARE_PER_CPU_SECTION(type, name, "..decrypted") =20 --=20 2.53.0 From nobody Fri Sep 25 06:02:54 2026 Received: from mail-pl1-f227.google.com (mail-pl1-f227.google.com [209.85.214.227]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 64CF93BB661 for ; Wed, 16 Sep 2026 05:06:01 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.227 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789535163; cv=none; b=Xjbr1oJQSC5dOfaC74SqC8gQZ/G1HeR/ajMH0pJXFnExao8gVZInxSEDbNgowxF5A+Gor6vNu1q52Vyc/fhWb3EXSqEsCsFqua3M3x3LroH3lHLgcd5qBFqytDQWS0UhGyNPBCeGCmxFSN6l/Mx/sleiLy0lFIrsRw34KhX6J20= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789535163; c=relaxed/simple; bh=ZCYsk3b8be7YzwPtZc9AlEHE3JpuvSauytIqzViEWP8=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=lNJUIMEkc6AhhuG+eUdQA0vUdKgRpjyKLLfil+g0GYePl8k/nY9qNzrwAh8sg5lrw5V3jGk4z6J6607ADWwz4EGxFgOoO/ye3z9wusY4cPubioc0/F1yQrGsVveGu/3BQ1F5ysmnFYkOGb+vhcm3RlMmMJMLnEyT3/a2i8andc8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=broadcom.com; spf=fail smtp.mailfrom=broadcom.com; dkim=pass (1024-bit key) header.d=broadcom.com header.i=@broadcom.com header.b=YW777VKk; arc=none smtp.client-ip=209.85.214.227 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=broadcom.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=broadcom.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=broadcom.com header.i=@broadcom.com header.b="YW777VKk" Received: by mail-pl1-f227.google.com with SMTP id d9443c01a7336-2d6d28aa26cso34054605ad.2 for ; Tue, 15 Sep 2026 22:06:01 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1789535161; x=1790139961; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:dkim-signature:x-gm-gg :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to :content-type; bh=qEqGUwhuGtd/IQ00br0ru4/E+DR8RR02AnrUv6Zwxpk=; b=q2CZ0jCABdyd+63njMzNbMsgtWcKuWTF0iw4gFn5NjqoDLrRsrdHkemJRjMV99zipe mSOmkj6kpZUYcT5QvsuldwAOSnG5Am+Na96SbDEEemFvBKU7yRKROyjr5k2S507CTzNr 02Jh9mH4n96PSXHI/SBxNSFUL6q5yu/XVZXQeygrfrQ/+xgxrRgu0u3t3dhsmwlFgbGv AuRZelzmgDTI+gHNyN/jsnPFTrfC2BsFDRvky6xdtDWGVgGIEXCYzWuk04wpaKAgO0k7 Unh9JgQpPZIEZPOhkRM6W6JGoWvS/R99Qu798mCFpX3SXyPR5uZLFbEAvZtBNxvGpCvp A8TA== X-Forwarded-Encrypted: i=1; AKwUvByiDucF00RuCSssQ5mnR+esfuZhK+nH6FwHDKkCVeW90BDmKOYFmUMNbkOrjVApPB9dSmBHRyUHfTqU5V8=@vger.kernel.org X-Gm-Message-State: AFuF++k+kMNx2ByjLDvwGebPmjZR7B7iRPj/C29Cu1g0w8fltiHMPrKZ c7AQPm1ORQEyfFzDv9SzE7I+O1wQUimS6LBl0lWKQe6E1It1F0Pf7T1nOEd+2ptnhjvUM0NNi0v e0U22LTQ4DEA+T6frXnrgJ1czKMib/QkVmFo9i7OvhD/IxAM1Q+RiGeHlZfKl+hwXSGDw+G8Daf 7kXp7OHZ2iL4LWOHMOpJ+FAbinzeXJmgfpjggAREOCT5ACdOXTf62EwxJTnAOl+wAnY+zgNFWWK zXzdcKBplqpGGsl X-Gm-Gg: AYBFou0m7C3eDekjCBQx4hGmY8OPm/Zs4h6fN9hmkArUU076BWjNHC8IRjiT+XLe/yB FtFjKIk0j8tHzqVjHkumT0JUUFC0dycC7UMVDpmd+kLkPcNrvE8a/bJKlROMiDncTmgTby+H4jc IsRTfUYMRQkGXH3tCBAVUBa2iX4VEU7JykfXmAI29Iw/GLWqTMHtHdAyssFKLonmxG3lOdIQcHa v9I+hbPsJxkMvPi5T6j/U1gHsYBtwEI7e2uimPWG9nDESgqOACPWsPCpUeB3q6098s7tMMLVw/v HpI0loeIvLyfXko4vqs05RPn0ddHOTsZzJcxb/gQv5+jujrYZ612NYKtkIPh0lm7QgUPhG8u0jd yhY5YodOucjPb4mpkfiE2VKvBPu0u7lV2PbQUR57hGy5qTSQ46L85lrXR/Lu0Hq3X/TZqlf9Wrh gRZX4DgeCyvArBnzeE0JwBiwawcaEmfCG7 X-Received: by 2002:a17:902:e74b:b0:2dd:3ddf:b895 with SMTP id d9443c01a7336-2dd8dd0b2fcmr22960145ad.3.1789535160712; Tue, 15 Sep 2026 22:06:00 -0700 (PDT) Received: from smtp-us-east1-p01-i01-si01.dlp.protect.broadcom.com (address-144-49-247-29.dlp.protect.broadcom.com. [144.49.247.29]) by smtp-relay.gmail.com with ESMTPS id d9443c01a7336-2dd89e94c81sm749675ad.12.2026.09.15.22.05.59 for (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Tue, 15 Sep 2026 22:06:00 -0700 (PDT) X-Relaying-Domain: broadcom.com X-CFilter-Loop: Reflected Received: by mail-pg1-f198.google.com with SMTP id 41be03b00d2f7-cc1a439db36so4351578a12.2 for ; Tue, 15 Sep 2026 22:05:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=broadcom.com; s=google; t=1789535158; x=1790139958; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=qEqGUwhuGtd/IQ00br0ru4/E+DR8RR02AnrUv6Zwxpk=; b=YW777VKkuthOhJll13Vfzz2TFuo7Dg4wmScpv2bjhDyzJ/f/p/MOQx8huOil4+PGWN WtMxrZuFv75DzHRdMmW8yrE7IHNh9G3HDENQQC02VhyktO6g++ZPQ8J/0inkvJ4CgURj R07zG3rTlx/SnJGV0+elbsCoqb3Xe0YjId85M= X-Forwarded-Encrypted: i=1; AKwUvBzquDHwQGCDhqsysPQ6fbpAyoPYtweQyjkTHZtvtN7Rj62ne7HPio4taAwQUc0gdCo4y1DjD7IeK65pnNk=@vger.kernel.org X-Received: by 2002:a05:6a21:1343:b0:3da:80a1:f2f with SMTP id adf61e73a8af0-3dd5f5dfd69mr2927550637.14.1789535158380; Tue, 15 Sep 2026 22:05:58 -0700 (PDT) X-Received: by 2002:a05:6a21:1343:b0:3da:80a1:f2f with SMTP id adf61e73a8af0-3dd5f5dfd69mr2927490637.14.1789535157892; Tue, 15 Sep 2026 22:05:57 -0700 (PDT) Received: from vertex.localdomain ([192.19.144.250]) by smtp.gmail.com with ESMTPSA id 5a478bee46e88-33bf5e40494sm3951386eec.15.2026.09.15.22.05.53 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 15 Sep 2026 22:05:56 -0700 (PDT) From: Zack Rusin To: Borislav Petkov , Ajay Kaher , Alexey Makhalov , x86@kernel.org, Dennis Zhou , Tejun Heo , Arnd Bergmann , Kiryl Shutsemau , Rick Edgecombe Cc: Thomas Gleixner , Ingo Molnar , Dave Hansen , "H . Peter Anvin" , virtualization@lists.linux.dev, bcm-kernel-feedback-list@broadcom.com, linux-kernel@vger.kernel.org, Christoph Lameter , Andrew Morton , Tom Lendacky , Bo Gan , linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-coco@lists.linux.dev, kvm@vger.kernel.org, Zack Rusin Subject: [PATCH v1 2/2] x86/vmware: Decrypt steal-time storage before sharing it Date: Wed, 16 Sep 2026 01:05:40 -0400 Message-ID: X-Mailer: git-send-email 2.53.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-DetectorID-Processed: b00c1d49-9d2e-4205-b15f-d015386d3d5e Content-Type: text/plain; charset="utf-8" VMware's steal-time counter must be in shared memory so the host can update it. Encrypted guests currently register its address without decrypting the storage. Move their setup to an early initcall, when the memory allocator is available for page-table splitting, but before secondary CPUs start. Leave ordinary guests' setup unchanged. Convert every possible CPU's storage before registering any address. Zero each object after conversion, since its contents may not survive conversion and the host initializes only the counter. Register the boot CPU with preemption disabled; the existing hotplug callbacks handle the others. If conversion or boot-CPU registration fails, attempt to re-encrypt all affected ranges, including a partially converted failing range, and disable steal time. TDX's conversion path requires directly mapped memory. Check all per-CPU objects before converting anything, and disable steal time if a TDX guest uses the page per-CPU allocator, which supplies vmalloc mappings. This also covers automatic fallback from the embedded allocator. AMD guests are unaffected by this restriction because their conversion path supports those mappings. Link: https://lore.kernel.org/r/20260309235250.2611115-5-alexey.makhalov@br= oadcom.com Co-developed-by: Bo Gan Signed-off-by: Bo Gan Co-developed-by: Alexey Makhalov Signed-off-by: Alexey Makhalov Signed-off-by: Zack Rusin --- arch/x86/kernel/cpu/vmware.c | 96 +++++++++++++++++++++++++++++++++++- 1 file changed, 95 insertions(+), 1 deletion(-) diff --git a/arch/x86/kernel/cpu/vmware.c b/arch/x86/kernel/cpu/vmware.c index 34b73573b108..f49898275d87 100644 --- a/arch/x86/kernel/cpu/vmware.c +++ b/arch/x86/kernel/cpu/vmware.c @@ -25,14 +25,19 @@ #include #include #include +#include #include #include +#include +#include #include +#include #include #include #include #include #include +#include #include #include #include @@ -147,6 +152,7 @@ static struct cyc2ns_data vmware_cyc2ns __ro_after_init; static bool vmw_sched_clock __initdata =3D true; static DEFINE_PER_CPU_DECRYPTED(struct vmware_steal_time, vmw_steal_time) = __aligned(64); static bool has_steal_clock; +static bool vmw_steal_time_ready; static bool steal_acc __initdata =3D true; /* steal time accounting */ =20 static __init int setup_vmw_sched_clock(char *s) @@ -280,7 +286,9 @@ static void vmware_disable_steal_time(void) =20 static void vmware_guest_cpu_init(void) { - if (has_steal_clock) + if (has_steal_clock && + (!cc_platform_has(CC_ATTR_GUEST_MEM_ENCRYPT) || + vmw_steal_time_ready)) vmware_register_steal_time(); } =20 @@ -325,6 +333,92 @@ static int vmware_cpu_down_prepare(unsigned int cpu) } #endif =20 +static void __init vmware_steal_time_range(int cpu, unsigned long *start, + int *numpages) +{ + unsigned long addr =3D (unsigned long)per_cpu_ptr(&vmw_steal_time, cpu); + + *start =3D addr & PAGE_MASK; + *numpages =3D DIV_ROUND_UP(offset_in_page(addr) + + sizeof(struct vmware_steal_time), PAGE_SIZE); +} + +static int __init vmware_decrypt_steal_time(void) +{ + int cpu, failed_cpu, numpages, ret; + unsigned long start; + + if (!has_steal_clock || + !cc_platform_has(CC_ATTR_GUEST_MEM_ENCRYPT)) + return 0; + + /* + * TDX's conversion callback derives the physical range with __pa(), + * so the storage must live in the direct map. The page per-CPU + * allocator, which is also the automatic fallback when the embedding + * allocator fails, hands out vmalloc addresses instead. Reject those + * before converting anything, so no page is left converted. + */ + if (cpu_feature_enabled(X86_FEATURE_TDX_GUEST)) { + for_each_possible_cpu(cpu) { + if (!is_vmalloc_addr(per_cpu_ptr(&vmw_steal_time, cpu))) + continue; + + pr_warn("steal time disabled: TDX requires a direct mapping\n"); + has_steal_clock =3D false; + return 0; + } + } + + for_each_possible_cpu(cpu) { + vmware_steal_time_range(cpu, &start, &numpages); + ret =3D set_memory_decrypted(start, numpages); + if (ret) { + failed_cpu =3D cpu; + goto rollback; + } + + /* + * Conversion need not preserve the zeroes. The host + * initializes only the counter on enable, so the guest + * must initialize the reserved words itself. + */ + memset(per_cpu_ptr(&vmw_steal_time, cpu), 0, + sizeof(struct vmware_steal_time)); + } + + vmw_steal_time_ready =3D true; + preempt_disable(); + vmware_guest_cpu_init(); + preempt_enable(); + if (!has_steal_clock) { + pr_warn("failed to register boot CPU steal-time memory\n"); + failed_cpu =3D nr_cpu_ids; + goto rollback_pages; + } + return 0; + +rollback: + pr_warn("failed to decrypt steal-time memory for CPU %d: %d\n", + failed_cpu, ret); + +rollback_pages: + for_each_possible_cpu(cpu) { + vmware_steal_time_range(cpu, &start, &numpages); + ret =3D set_memory_encrypted(start, numpages); + if (ret) + pr_warn("failed to re-encrypt steal-time memory for CPU %d: %d\n", + cpu, ret); + if (cpu =3D=3D failed_cpu) + break; + } + + vmw_steal_time_ready =3D false; + has_steal_clock =3D false; + return 0; +} +early_initcall(vmware_decrypt_steal_time); + static __init int activate_jump_labels(void) { if (has_steal_clock) { --=20 2.53.0