From nobody Wed Dec 24 01:33:42 2025 Received: from mail-wr1-f74.google.com (mail-wr1-f74.google.com [209.85.221.74]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E71883F9FB for ; Mon, 29 Jan 2024 18:05:32 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.221.74 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706551534; cv=none; b=FDq+RbB/Yxd8zZ00Pfu4gJVQQ7o+ifT++Ybkiaw2euVjYnA1pyQ8w1NcsQ4mhxunJ3RozWO6aLF4xg4VxMO0c9hUde/LiXp0O20ASJynfk5zEfcwWSseX9OKJ1ylKtB1YB2H4Nlac91ua5j2jI4QOjHL37pHe4IcwSDTULiP7+E= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706551534; c=relaxed/simple; bh=8EngcPiV4/gVfKgvYtooxy7CfkplaV+JIi8nleL+nv8=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=r5QMmG7waGinbozTBrIz9/mlIkBapWQ7BDylBiaY+gpe8lmPhKpF5tkskOyg6BImw3Bbeh/Y+xJocq4jk/aT1Ox3LDaAxelRioL+GFgoBJoX8UVy/S+xigFMXLyjlCDNecy3bBR6KcaXxW2bISA3hhQBoGxh+2fttcvXaxYoMAE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=HRNQRDmE; arc=none smtp.client-ip=209.85.221.74 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="HRNQRDmE" Received: by mail-wr1-f74.google.com with SMTP id ffacd0b85a97d-33af10efa37so439290f8f.1 for ; Mon, 29 Jan 2024 10:05:32 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1706551531; x=1707156331; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=ZKxz5I1lJ14YrwiTICZR28lgBpbE1gmuL2Ys4svPQiQ=; b=HRNQRDmEXLWEjS+DYUQxClPctOz2zbJJVbX+8mubQjDVea3RYXHQ2C+NaoFeQSce/9 YoqhBTr2lLrom/TScTQSSVl38LGRpu5CceDTEyNoeh324ql+BPdbCABtrtMBxz3qj7uA eAkmauGX8DH8qywnDiGSMO0DgQPt57fsPzsygNPjwwLQGr+BYLESvJ3B4svejMEqCezA kZ6zJgmTtPm+tEQiYhw8zh/JhVfWOesvuWMpQ8GKCQ4SVJ2MB3Fczrukd6rGby0HdmT2 /4yDA/X1aaxvCtqVHVkmSrj7Tu0496vKEJZWTTHROrAfHeC1beyjN7jv4wCQGZyrbXuK Mdaw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1706551531; x=1707156331; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=ZKxz5I1lJ14YrwiTICZR28lgBpbE1gmuL2Ys4svPQiQ=; b=iTWstKww44ree8ao/8f47JG0YsheEHMLaMomTl/pYcagNjUuG0x3AIdhLzuyFtuoUF IR5IPnqaejkmmtAflCGf3Z9Mkw4kpwoLU0zw2JbaegNHnmxFHFQ+uT9NrNqmcvobUYUA DFw2Qn/k6x7SywrBHcMmiteZx0m+WyDod14MJKwosS3xC6qgUHnRgzhC6lTKde0BD8sT fyWFbPzBPVpHn6wXoyaLEWG/TVpsHCJOtbZETHljf4EA9OMzGqdQfoazbie4JhV6xkGL APWVF2gkA9Yuyqe04B5gi8zpnODwQmYblirRJXgpor0vgS26twjVcY5HVU9BxN5+NAkB Ygvw== X-Gm-Message-State: AOJu0YxT/4GhZzW456E5I8u3YK944wDHO0kzzoKW4JTWCVbwIgWTCKrb vQA/KkaIP/K1/9VqNdk61T78+tyjgozDW9W2KSTKwbBAlAy7kyZ2LmExA3WRb8bWGEQb25cVtYo RdlvtjO1I1ngkUf/V1k0lZf379OLi/PA0hk+gShM7/nLjBxQo/iL+YLFUyExO5clvhIsUSVzAN7 bwR+ILqV0AwsT2vguf3U+OqjsVIacgjw== X-Google-Smtp-Source: AGHT+IHqXoqSxlnq+tXgVV8MITvhCNMHm/mI4bOyTcP8eXKfAYEXj2pInMqp+zkM42KE9U02vk3dc98V X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a5d:5a97:0:b0:33a:f355:7f2a with SMTP id bp23-20020a5d5a97000000b0033af3557f2amr3448wrb.8.1706551531045; Mon, 29 Jan 2024 10:05:31 -0800 (PST) Date: Mon, 29 Jan 2024 19:05:04 +0100 In-Reply-To: <20240129180502.4069817-21-ardb+git@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240129180502.4069817-21-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=2179; i=ardb@kernel.org; h=from:subject; bh=cYtM8PUyOXvYlw4FeyoL0t7P65MWzmFujl4gVlusCbs=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIXX7iwvJHgmKYd5qZlpNAZqZUfs+Xyudf+rqhncfljw00 JxmfF26o5SFQYyDQVZMkUVg9t93O09PlKp1niULM4eVCWQIAxenAExk9nNGhiUtf/QfJmu4l6R7 +lzQmanC+t38+7/fDQ2K2fFPtm50XcDIcNKRZbZu+RGrc37Gq9csORj73Tspw+TyQoeoVU5WT8O W8gMA X-Mailer: git-send-email 2.43.0.429.g432eaa2c6b-goog Message-ID: <20240129180502.4069817-22-ardb+git@google.com> Subject: [PATCH v3 01/19] efi/libstub: Add generic support for parsing mem_encrypt= From: Ard Biesheuvel To: linux-kernel@vger.kernel.org Cc: Ard Biesheuvel , Kevin Loughlin , Tom Lendacky , Dionna Glaze , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , Andy Lutomirski , Arnd Bergmann , Nathan Chancellor , Nick Desaulniers , Justin Stitt , Kees Cook , Brian Gerst , linux-arch@vger.kernel.org, llvm@lists.linux.dev Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Ard Biesheuvel Parse the mem_encrypt=3D command line parameter from the EFI stub if CONFIG_ARCH_HAS_MEM_ENCRYPT=3Dy, so that it can be passed to the early boot code by the arch code in the stub. This avoids the need for the core kernel to do any string parsing very early in the boot. Signed-off-by: Ard Biesheuvel --- drivers/firmware/efi/libstub/efi-stub-helper.c | 8 ++++++++ drivers/firmware/efi/libstub/efistub.h | 2 +- 2 files changed, 9 insertions(+), 1 deletion(-) diff --git a/drivers/firmware/efi/libstub/efi-stub-helper.c b/drivers/firmw= are/efi/libstub/efi-stub-helper.c index bfa30625f5d0..3dc2f9aaf08d 100644 --- a/drivers/firmware/efi/libstub/efi-stub-helper.c +++ b/drivers/firmware/efi/libstub/efi-stub-helper.c @@ -24,6 +24,8 @@ static bool efi_noinitrd; static bool efi_nosoftreserve; static bool efi_disable_pci_dma =3D IS_ENABLED(CONFIG_EFI_DISABLE_PCI_DMA); =20 +int efi_mem_encrypt; + bool __pure __efi_soft_reserve_enabled(void) { return !efi_nosoftreserve; @@ -75,6 +77,12 @@ efi_status_t efi_parse_options(char const *cmdline) efi_noinitrd =3D true; } else if (IS_ENABLED(CONFIG_X86_64) && !strcmp(param, "no5lvl")) { efi_no5lvl =3D true; + } else if (IS_ENABLED(CONFIG_ARCH_HAS_MEM_ENCRYPT) && + !strcmp(param, "mem_encrypt") && val) { + if (parse_option_str(val, "on")) + efi_mem_encrypt =3D 1; + else if (parse_option_str(val, "off")) + efi_mem_encrypt =3D -1; } else if (!strcmp(param, "efi") && val) { efi_nochunk =3D parse_option_str(val, "nochunk"); efi_novamap |=3D parse_option_str(val, "novamap"); diff --git a/drivers/firmware/efi/libstub/efistub.h b/drivers/firmware/efi/= libstub/efistub.h index 212687c30d79..a1c6ab24cd99 100644 --- a/drivers/firmware/efi/libstub/efistub.h +++ b/drivers/firmware/efi/libstub/efistub.h @@ -37,8 +37,8 @@ extern bool efi_no5lvl; extern bool efi_nochunk; extern bool efi_nokaslr; extern int efi_loglevel; +extern int efi_mem_encrypt; extern bool efi_novamap; - extern const efi_system_table_t *efi_system_table; =20 typedef union efi_dxe_services_table efi_dxe_services_table_t; --=20 2.43.0.429.g432eaa2c6b-goog From nobody Wed Dec 24 01:33:42 2025 Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 90BFA6F073 for ; Mon, 29 Jan 2024 18:05:34 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.202 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706551536; cv=none; b=EYXiJmo2f3qJueKibgCPCmoyntHkQOwy37ss9HUvgug28oPPaJHkcuxbubHc5CamoC7chfojs3UkdVBjgcOmiME7iC/jYE8jbhoRs0wTjyxuYZyTWvGUevltqzJ8lGeqmkTtczFoHO60sjp6Bw73X3s6D0T1CC8li+GOZ7nmwLY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706551536; c=relaxed/simple; bh=HFn1spOv4KPR4R1zHjTL4BDpzlPynxVt+ZjcueLDXCo=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=LvX+iqANvo0n7w6xXzGkd7LxTuRxRCEzVghiUiofv49OdpYucet7ckt0n9fyJFvP70mu6b0C0yfiMxD+e2kD4iKnRT8CJcuSX7qjqVRww1yiRAQjEU0wh/l6DTGkf6KMSm/W2ymF+MsXE+1HwzIzHHrlTCO5Rwh/EFOkEoB7z4o= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=g4paj4/2; arc=none smtp.client-ip=209.85.128.202 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="g4paj4/2" Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-5ff93902762so55570337b3.2 for ; Mon, 29 Jan 2024 10:05:34 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1706551533; x=1707156333; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=p3FWGTV10qie8rRiz9nelg+HW9vFOvPrRVG9vffZcTs=; b=g4paj4/287fNNvwYS7k+HS4qAaVswZNsu8T+4cLZSy+eXyl6tD/ouoK2Wslz4zIX/m KBU8+2MLvR5WGs25D+LZ6xTtUWnHlKt54Xp1ELBSJKyQiziiMoWi7Ybi3Y5hSGr7vh44 m2yTCoTwZ0LVLM9uEfgM45GyIoYkj0s6e4wURlotgBff5NfWo5b6RakFS1OV+RqViXFU uOmx1wLzQtZiuD5Q5zRr+JATpaq2jnXNnzxA9VQfCWsGaT6ejCZ6a0KQurR66r1skaRa OirpC0RIU3yegLWVK+yZT7tXuwsWWvrIPtVxcxtYmwCPB8/ZMfj9yv5p2eh8+3nEwK9P ecyg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1706551533; x=1707156333; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=p3FWGTV10qie8rRiz9nelg+HW9vFOvPrRVG9vffZcTs=; b=Bm+dvzP3R67hzbG5QoAIY188tJV5vkUNEmR2mDgRbl2BdGJSO2uX15MTXrymMwP3sd p2sz1Dd3Wkdl6XXSfPogx6lS8UbVBFHKGUHUQ34dJZiZRAT9Xq5v1TB2lPBXY5VQYiOL Y99QrwhPaS6mxezIgPGIzYRjMRgxBBbXwHGS9idtejQfZPUMhLYcSmQAj3DjcG4G+rT8 y4dO/4/nJ8PUC9qgJPrIIM5WtNRqA0bYHXIxL7wUSVRU1KsBoMvXN3hYPAUBMmVZzJVF yJUoFYzKVc+NX2UyIeUNo7dLPchGr9Hq88xzyeihbsRZ6b/Xb5RJykAwu0K3pb/7Hj9n 6Qwg== X-Gm-Message-State: AOJu0YzHvYGKiRTQ4WVTPpM/icBVFosM6jttEEt95Cn+Eir+2Tqe5G6c xOBQHhN18KsREDANLsBL9lX0HLR2WulFs/qBXt4qhUrf4rYKiExWT1oNrll9bKN2xpf3T27j+8k ULNzEGKCnB4gUt48S0NpLNxfp3t+hwhhmFbMd0vbZ1pau8PCtIilcguWoe+khfxduc1Oz3U81+8 jlPY1TjurXk27Cx/ehikE5pA2DiXb2CQ== X-Google-Smtp-Source: AGHT+IGVcq2pXpU6LfBnSbe3eFllbMwA7mlv7HhvFLN60TO62ZHJfEf419ADavLY79oFrfTS/Z1//fuD X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a05:690c:ed4:b0:5d8:4274:bae2 with SMTP id cs20-20020a05690c0ed400b005d84274bae2mr2074182ywb.6.1706551533566; Mon, 29 Jan 2024 10:05:33 -0800 (PST) Date: Mon, 29 Jan 2024 19:05:05 +0100 In-Reply-To: <20240129180502.4069817-21-ardb+git@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240129180502.4069817-21-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=7925; i=ardb@kernel.org; h=from:subject; bh=M0vxsf0gTqIKMZpFhFJ2BBWB91v5R63xIeoeRQ3uSVE=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIXX7i4vMAZebr19uPjtNTGWy4aWMzFNMrYUL3pafZBPbu /6AHf+KjlIWBjEOBlkxRRaB2X/f7Tw9UarWeZYszBxWJpAhDFycAjCRHYUM/0zmcif/jDiQ3vfl ukmjmedu7Q9+7mufCvEZl7/vklXvs2D4Z7mhl60oe8btsotvDwd9qtTNC7TYFrvMi1F2Z/gmS/V oNgA= X-Mailer: git-send-email 2.43.0.429.g432eaa2c6b-goog Message-ID: <20240129180502.4069817-23-ardb+git@google.com> Subject: [PATCH v3 02/19] x86/boot: Move mem_encrypt= parsing to the decompressor From: Ard Biesheuvel To: linux-kernel@vger.kernel.org Cc: Ard Biesheuvel , Kevin Loughlin , Tom Lendacky , Dionna Glaze , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , Andy Lutomirski , Arnd Bergmann , Nathan Chancellor , Nick Desaulniers , Justin Stitt , Kees Cook , Brian Gerst , linux-arch@vger.kernel.org, llvm@lists.linux.dev Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Ard Biesheuvel The early SME/SEV code parses the command line very early, in order to decide whether or not memory encryption should be enabled, which needs to occur even before the initial page tables are created. This is problematic for a number of reasons: - this early code runs from the 1:1 mapping provided by the decompressor or firmware, which uses a different translation than the one assumed by the linker, and so the code needs to be built in a special way; - parsing external input while the entire kernel image is still mapped writable is a bad idea in general, and really does not belong in security minded code; - the current code ignores the built-in command line entirely (although this appears to be the case for the entire decompressor) Given that the decompressor/EFI stub is an intrinsic part of the x86 bootable kernel image, move the command line parsing there and out of the core kernel. This removes the need to build lib/cmdline.o in a special way, or to use RIP-relative LEA instructions in inline asm blocks. This involves a pair of new xloadflags in the setup header to indicate that a) mem_encrypt=3D was provided, and b) whether it was set to on or off. What this actually means in terms of default behavior when the command line parameter is omitted is left up to the existing logic - this permits the same flags to be reused if the need arises. Signed-off-by: Ard Biesheuvel --- arch/x86/boot/compressed/misc.c | 22 ++++++++++ arch/x86/include/uapi/asm/bootparam.h | 2 + arch/x86/lib/Makefile | 13 ------ arch/x86/mm/mem_encrypt_identity.c | 45 +++----------------- drivers/firmware/efi/libstub/x86-stub.c | 6 +++ 5 files changed, 37 insertions(+), 51 deletions(-) diff --git a/arch/x86/boot/compressed/misc.c b/arch/x86/boot/compressed/mis= c.c index b99e08e6815b..d63a2dc7d0b1 100644 --- a/arch/x86/boot/compressed/misc.c +++ b/arch/x86/boot/compressed/misc.c @@ -357,6 +357,26 @@ unsigned long decompress_kernel(unsigned char *outbuf,= unsigned long virt_addr, return entry; } =20 +/* + * Set the memory encryption xloadflag based on the mem_encrypt=3D command= line + * parameter, if provided. If not, the consumer of the flag decides what t= he + * default behavior should be. + */ +static void set_mem_encrypt_flag(struct setup_header *hdr) +{ + hdr->xloadflags &=3D ~(XLF_MEM_ENCRYPTION | XLF_MEM_ENCRYPTION_ENABLED); + + if (IS_ENABLED(CONFIG_ARCH_HAS_MEM_ENCRYPT)) { + int on =3D cmdline_find_option_bool("mem_encrypt=3Don"); + int off =3D cmdline_find_option_bool("mem_encrypt=3Doff"); + + if (on || off) + hdr->xloadflags |=3D XLF_MEM_ENCRYPTION; + if (on > off) + hdr->xloadflags |=3D XLF_MEM_ENCRYPTION_ENABLED; + } +} + /* * The compressed kernel image (ZO), has been moved so that its position * is against the end of the buffer used to hold the uncompressed kernel @@ -387,6 +407,8 @@ asmlinkage __visible void *extract_kernel(void *rmode, = unsigned char *output) /* Clear flags intended for solely in-kernel use. */ boot_params_ptr->hdr.loadflags &=3D ~KASLR_FLAG; =20 + set_mem_encrypt_flag(&boot_params_ptr->hdr); + sanitize_boot_params(boot_params_ptr); =20 if (boot_params_ptr->screen_info.orig_video_mode =3D=3D 7) { diff --git a/arch/x86/include/uapi/asm/bootparam.h b/arch/x86/include/uapi/= asm/bootparam.h index 01d19fc22346..316784e17d38 100644 --- a/arch/x86/include/uapi/asm/bootparam.h +++ b/arch/x86/include/uapi/asm/bootparam.h @@ -38,6 +38,8 @@ #define XLF_EFI_KEXEC (1<<4) #define XLF_5LEVEL (1<<5) #define XLF_5LEVEL_ENABLED (1<<6) +#define XLF_MEM_ENCRYPTION (1<<7) +#define XLF_MEM_ENCRYPTION_ENABLED (1<<8) =20 #ifndef __ASSEMBLY__ =20 diff --git a/arch/x86/lib/Makefile b/arch/x86/lib/Makefile index ea3a28e7b613..f0dae4fb6d07 100644 --- a/arch/x86/lib/Makefile +++ b/arch/x86/lib/Makefile @@ -14,19 +14,6 @@ ifdef CONFIG_KCSAN CFLAGS_REMOVE_delay.o =3D $(CC_FLAGS_FTRACE) endif =20 -# Early boot use of cmdline; don't instrument it -ifdef CONFIG_AMD_MEM_ENCRYPT -KCOV_INSTRUMENT_cmdline.o :=3D n -KASAN_SANITIZE_cmdline.o :=3D n -KCSAN_SANITIZE_cmdline.o :=3D n - -ifdef CONFIG_FUNCTION_TRACER -CFLAGS_REMOVE_cmdline.o =3D -pg -endif - -CFLAGS_cmdline.o :=3D -fno-stack-protector -fno-jump-tables -endif - inat_tables_script =3D $(srctree)/arch/x86/tools/gen-insn-attr-x86.awk inat_tables_maps =3D $(srctree)/arch/x86/lib/x86-opcode-map.txt quiet_cmd_inat_tables =3D GEN $@ diff --git a/arch/x86/mm/mem_encrypt_identity.c b/arch/x86/mm/mem_encrypt_i= dentity.c index 7f72472a34d6..06466f6d5966 100644 --- a/arch/x86/mm/mem_encrypt_identity.c +++ b/arch/x86/mm/mem_encrypt_identity.c @@ -43,7 +43,6 @@ =20 #include #include -#include #include #include =20 @@ -95,10 +94,6 @@ struct sme_populate_pgd_data { */ static char sme_workarea[2 * PMD_SIZE] __section(".init.scratch"); =20 -static char sme_cmdline_arg[] __initdata =3D "mem_encrypt"; -static char sme_cmdline_on[] __initdata =3D "on"; -static char sme_cmdline_off[] __initdata =3D "off"; - static void __init sme_clear_pgd(struct sme_populate_pgd_data *ppd) { unsigned long pgd_start, pgd_end, pgd_size; @@ -504,11 +499,9 @@ void __init sme_encrypt_kernel(struct boot_params *bp) =20 void __init sme_enable(struct boot_params *bp) { - const char *cmdline_ptr, *cmdline_arg, *cmdline_on, *cmdline_off; unsigned int eax, ebx, ecx, edx; unsigned long feature_mask; unsigned long me_mask; - char buffer[16]; bool snp; u64 msr; =20 @@ -570,42 +563,18 @@ void __init sme_enable(struct boot_params *bp) msr =3D __rdmsr(MSR_AMD64_SYSCFG); if (!(msr & MSR_AMD64_SYSCFG_MEM_ENCRYPT)) return; + + if (bp->hdr.xloadflags & XLF_MEM_ENCRYPTION) { + if (bp->hdr.xloadflags & XLF_MEM_ENCRYPTION_ENABLED) + sme_me_mask =3D me_mask; + } else if (IS_ENABLED(CONFIG_AMD_MEM_ENCRYPT_ACTIVE_BY_DEFAULT)) { + sme_me_mask =3D me_mask; + } } else { /* SEV state cannot be controlled by a command line option */ sme_me_mask =3D me_mask; - goto out; } =20 - /* - * Fixups have not been applied to phys_base yet and we're running - * identity mapped, so we must obtain the address to the SME command - * line argument data using rip-relative addressing. - */ - asm ("lea sme_cmdline_arg(%%rip), %0" - : "=3Dr" (cmdline_arg) - : "p" (sme_cmdline_arg)); - asm ("lea sme_cmdline_on(%%rip), %0" - : "=3Dr" (cmdline_on) - : "p" (sme_cmdline_on)); - asm ("lea sme_cmdline_off(%%rip), %0" - : "=3Dr" (cmdline_off) - : "p" (sme_cmdline_off)); - - if (IS_ENABLED(CONFIG_AMD_MEM_ENCRYPT_ACTIVE_BY_DEFAULT)) - sme_me_mask =3D me_mask; - - cmdline_ptr =3D (const char *)((u64)bp->hdr.cmd_line_ptr | - ((u64)bp->ext_cmd_line_ptr << 32)); - - if (cmdline_find_option(cmdline_ptr, cmdline_arg, buffer, sizeof(buffer))= < 0) - goto out; - - if (!strncmp(buffer, cmdline_on, sizeof(buffer))) - sme_me_mask =3D me_mask; - else if (!strncmp(buffer, cmdline_off, sizeof(buffer))) - sme_me_mask =3D 0; - -out: if (sme_me_mask) { physical_mask &=3D ~sme_me_mask; cc_vendor =3D CC_VENDOR_AMD; diff --git a/drivers/firmware/efi/libstub/x86-stub.c b/drivers/firmware/efi= /libstub/x86-stub.c index 0d510c9a06a4..66e336cca0cc 100644 --- a/drivers/firmware/efi/libstub/x86-stub.c +++ b/drivers/firmware/efi/libstub/x86-stub.c @@ -879,6 +879,12 @@ void __noreturn efi_stub_entry(efi_handle_t handle, } } =20 + if (IS_ENABLED(CONFIG_ARCH_HAS_MEM_ENCRYPT) && efi_mem_encrypt) { + hdr->xloadflags |=3D XLF_MEM_ENCRYPTION; + if (efi_mem_encrypt > 0) + hdr->xloadflags |=3D XLF_MEM_ENCRYPTION_ENABLED; + } + status =3D efi_decompress_kernel(&kernel_entry); if (status !=3D EFI_SUCCESS) { efi_err("Failed to decompress kernel\n"); --=20 2.43.0.429.g432eaa2c6b-goog From nobody Wed Dec 24 01:33:42 2025 Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 085BB6F07C for ; Mon, 29 Jan 2024 18:05:37 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.202 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706551539; cv=none; b=J42fLjwjPMGVOan0Vnba4BtPOztaJgnd9uENmnlOXt66KiBTBnylmTnvJave+wJn+/cjgXpEnHYke5v/LNcQyqQK3301eOffaYB+KmW20w5SkwXykHGjfwADJQHlqgLjH31huN49AQE+mOVnxCjkbiAmrYWvCo5VH1gjUUQoSI4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706551539; c=relaxed/simple; bh=ksYtquI/zNJoo09z1kn9EIIXegn4h7RVzbkZ+iPv7gQ=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=mPCerUWbYB6ycF1/mm0lcb8wI+odz6RU/HC4eb2i3YrLGQ/fc4ek8pKzCTHqaZHtOQ8QHTXxpv7UAZDWu3iH2OnwoClh2jQIVAl5Vjx0X25dIzQ21h4UmLCUo2PHNVmF7g9iRCBuwsVbR+QKDo7AQ93KTGjFDF5AqctVWz7aCiY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=N5GNZ1+H; arc=none smtp.client-ip=209.85.128.202 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="N5GNZ1+H" Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-6029c85922dso59735997b3.3 for ; Mon, 29 Jan 2024 10:05:36 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1706551536; x=1707156336; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=VlzuORvF37cHhQgjoovr2a86HYjlNB8YTzXdR4Eq8S8=; b=N5GNZ1+HcdiMDaLoHriuiPIQ11RvN29Xx0ePSgvy2/TcZEUqNV476uWVlc5098jmV3 xp83LLEGPdgFnE8JTCrQrqjrOGwHRLfeJqtwmWg6lYstTdofsuyD/K22sfxFugdgwlWy ZP5uczstj19b0jCTfSyQINffGtO6p/7j/q2d1RNLMUkgmUNxqKYzM+pixGreb1NyhwiV QwFgxKa6LHA9JInQlmzSFjYtq0KdAzM2X9guUTUQ2EzdKy6IbOvJBBRJ1g5dbc3TsZSg bllZHKJrS+mIjlPL9JlSFz/Y+D45PA6aiCRZvfV+zWg5HX0vHs3Le6QcWfnLlu+dS6Al jLRQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1706551536; x=1707156336; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=VlzuORvF37cHhQgjoovr2a86HYjlNB8YTzXdR4Eq8S8=; b=pxiZbgYYwtsXyKRKVITm1pps+gbU58LYny+Nw5A/qxezaDYvPr7vqX43mPomHUluH/ /+mQkVg/1rFFXKPw+nBA6Gf/F7gmbEuWJ/lQ17Z9Zh7nsyJgYDJV1o/ou2Bd2Oauv7Pa wC9luQxKHBA4Rzi3sEXGPBwoGo8HOeJ33qP4RWj6JEpBWyyyHmFLEsp/AfPuEru+lfRl jBavOr4eU4GrBtc72IBSpxtJEeBbUzXB396mOtZXCvcgGnIuM98bYqmMZ08DRmt9WJ17 zFRp6aY87HRHWIgKPwGdkgtLUgHvt+LkmpdPdJN5XiRbPfK/CiT+foalOPgtXaaTDn9Q VAkw== X-Gm-Message-State: AOJu0Yydc5oeaDzPxPgpFUWXtKMxhEuEB2I9BRSjca1Baz04HZVwtY2V auDBn7ChSFb6ugD9iPKAlR8JH+PCctsJ3BTyy2KlDDfq7dtkuHtnRLOO/Avc7ua/+0vwiV2E/gg TE1rIm+bUgx+Z09DD0Yc0EQ1qvTgOariuJq9FSqL95rCDmZSDSshd3rxHtJbO12p+5fPGt4NU/S db+n+dxobKqUKHt0q+1qoaV28/aBhr/Q== X-Google-Smtp-Source: AGHT+IGM+P7GQyXug67b8mdUFbt5ptfNQBVn5gYHUdfmOmMI6DLMe8HFvy5Binsc1qtv+dC8mi8TsPNt X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a05:6902:1b8d:b0:dc2:2f33:bc28 with SMTP id ei13-20020a0569021b8d00b00dc22f33bc28mr2399097ybb.6.1706551536040; Mon, 29 Jan 2024 10:05:36 -0800 (PST) Date: Mon, 29 Jan 2024 19:05:06 +0100 In-Reply-To: <20240129180502.4069817-21-ardb+git@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240129180502.4069817-21-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=2268; i=ardb@kernel.org; h=from:subject; bh=TSUaLCMOyIRIkxI+e58H4dzOT8JQVh2xiOSo20k09Yg=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIXX7i0sznR4wlbbnZirKqb0/tlnidNu/zpjLc3ccECxra OiZErm+o5SFQYyDQVZMkUVg9t93O09PlKp1niULM4eVCWQIAxenAEzEVpXhD0+FxcHZC/v5WsNe XhTRq9Et+jDvZFyi/tRpcvrJhS3X1zAytLxUqjGVObPST0DFle299G5uPgeeD8eX7t+/rfqL0pE 8BgA= X-Mailer: git-send-email 2.43.0.429.g432eaa2c6b-goog Message-ID: <20240129180502.4069817-24-ardb+git@google.com> Subject: [PATCH v3 03/19] x86/startup_64: Drop long return to initial_code pointer From: Ard Biesheuvel To: linux-kernel@vger.kernel.org Cc: Ard Biesheuvel , Kevin Loughlin , Tom Lendacky , Dionna Glaze , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , Andy Lutomirski , Arnd Bergmann , Nathan Chancellor , Nick Desaulniers , Justin Stitt , Kees Cook , Brian Gerst , linux-arch@vger.kernel.org, llvm@lists.linux.dev Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Ard Biesheuvel Since commit 866b556efa12 ("x86/head/64: Install startup GDT"), the primary startup sequence sets the code segment register (CS) to __KERNEL_CS before calling into the startup code shared between primary and secondary boot. This means a simple indirect call is sufficient here. Signed-off-by: Ard Biesheuvel --- arch/x86/kernel/head_64.S | 35 ++------------------ 1 file changed, 3 insertions(+), 32 deletions(-) diff --git a/arch/x86/kernel/head_64.S b/arch/x86/kernel/head_64.S index d4918d03efb4..4017a49d7b76 100644 --- a/arch/x86/kernel/head_64.S +++ b/arch/x86/kernel/head_64.S @@ -428,39 +428,10 @@ SYM_INNER_LABEL(secondary_startup_64_no_verify, SYM_L= _GLOBAL) movq %r15, %rdi =20 .Ljump_to_C_code: - /* - * Jump to run C code and to be on a real kernel address. - * Since we are running on identity-mapped space we have to jump - * to the full 64bit address, this is only possible as indirect - * jump. In addition we need to ensure %cs is set so we make this - * a far return. - * - * Note: do not change to far jump indirect with 64bit offset. - * - * AMD does not support far jump indirect with 64bit offset. - * AMD64 Architecture Programmer's Manual, Volume 3: states only - * JMP FAR mem16:16 FF /5 Far jump indirect, - * with the target specified by a far pointer in memory. - * JMP FAR mem16:32 FF /5 Far jump indirect, - * with the target specified by a far pointer in memory. - * - * Intel64 does support 64bit offset. - * Software Developer Manual Vol 2: states: - * FF /5 JMP m16:16 Jump far, absolute indirect, - * address given in m16:16 - * FF /5 JMP m16:32 Jump far, absolute indirect, - * address given in m16:32. - * REX.W + FF /5 JMP m16:64 Jump far, absolute indirect, - * address given in m16:64. - */ - pushq $.Lafter_lret # put return address on stack for unwinder xorl %ebp, %ebp # clear frame pointer - movq initial_code(%rip), %rax - pushq $__KERNEL_CS # set correct cs - pushq %rax # target address in negative space - lretq -.Lafter_lret: - ANNOTATE_NOENDBR + ANNOTATE_RETPOLINE_SAFE + callq *initial_code(%rip) + int3 SYM_CODE_END(secondary_startup_64) =20 #include "verify_cpu.S" --=20 2.43.0.429.g432eaa2c6b-goog From nobody Wed Dec 24 01:33:42 2025 Received: from mail-yw1-f201.google.com (mail-yw1-f201.google.com [209.85.128.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id EBA3F76042 for ; Mon, 29 Jan 2024 18:05:39 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706551541; cv=none; b=f3g24PxsWqbQ66xxYKsojs9r+E+VxEgzdSaIBU0dqjkWoj52yZwvnHuCPq/nYsM+pefajZDR5adtezCM9HJo+OraO9dthv+ItwZvkAdOFwKUtjzSUKbt+2HAbI5R4GzeYvXYuSq8HZWupzya35+/qMFP0URRcS2BBV14/kfw+FM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706551541; c=relaxed/simple; bh=NfXCrvSrRHu5/o/a38jIuQQEOM5CDJx3GD3fZgnkl0s=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=S7elWN3q9JG90DNhaU04iQC46g9MC0O9bWFbX+K1vYNP9kGVK1JLv5pD2G7ScL3zgtvIy6S0hzBbVuZLWIvv6O1nx/1N36f59fMIuDZ1mXoehrigM7llZ3BlTj5EMJAMwuS5ML2Mr7RQDe9mFjoaSLx8JLd+O/UqRwTLd41z2mk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=MX9uMOEP; arc=none smtp.client-ip=209.85.128.201 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="MX9uMOEP" Received: by mail-yw1-f201.google.com with SMTP id 00721157ae682-5f874219ff9so39678257b3.0 for ; Mon, 29 Jan 2024 10:05:39 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1706551538; x=1707156338; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=iHVLq9Wa6m8PRjV68aukFNmIoFWxPV4eVUMjwp/oWNk=; b=MX9uMOEPvqCEzuFwVNq0sK6GsmH4zreaoZnLFIPuSq2JEuMNrYywE6eiti9DFDJQAd +ZDzDoAg54m6NW5y5DLrHlQZryjay+LdBDLUGUWP5VdV3C6ey4LELYpkUs9fZ2wqrGY9 pO7Gf/q2oKFFAd1ANNHlPDPXik++WcaTbMZbdQVb5fb7aqvH8a8eIiwDCzmCWKMbdSMg /Nz4gpIVvHvJUaGP9wCky3ed4DoR2qxK7Bbu6WrCkIcL2tfJdHwthY1WSP/13NPtSl/1 kZhTHVGYF15Xawdl8H0Q5a9b3mpYupt45KPnfQBDzu+zjcgIqNFWscDaawS6MLEPyBjE sdeg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1706551538; x=1707156338; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=iHVLq9Wa6m8PRjV68aukFNmIoFWxPV4eVUMjwp/oWNk=; b=EGsm2xPcUoWRKQj+yIFN3HaDQQbQJbBpNHPxefzlv+W0u2mnNypjSJo7GWpQDxSW78 lYBgjkJjhmfPVbNshCO2UFHQU3zJPMQkznXSc/nmLmevpXySEfldjqgOUW6zjgklPKeg 1J7Xmx3a0iEGjAfvAubEK2AipHez23QohMlDdVxzMYoZw3XKXuJdIdxwAOEXp7IOIdyX QI7vrW5PErI7wDycORu3pOWbbDUhfJd+LHtEAhRA78xaHI2EdTpoxOTxWphIC4Zt/mer 3RgItdLtPb3Xyn+6Gfu/SKWA1MoGnV86t4RHRqSVNo9JMx6/Q3p2K299OZlmxFiUwzhd Zqww== X-Gm-Message-State: AOJu0Yw53PNALJUHP2wrcnId0vE72RKULLeIf3DIYri4HaxCwI3IEO0h 7WZlFscnKnl1kzIMg4XwA5PK5zuG+9BbXuPbFOu287O5bHqUoAi7xglYvm4jyM4x5ZtMUYdxtRR aXOWvjs/k5rU/w/O7w+JhBh4eQXuBj67MfH4hv5OchXfjW+xHmEj6I7OP4L/OISr1BDav2dGVkj 99fZiBNIu1Gn6/fc6c1DC+93R3UoFU6g== X-Google-Smtp-Source: AGHT+IHxjP1IrVdmFk3AwDM0KDLSJNYxWIU38kLAEN4S3NCDdU+0TDUv7tiHgHfqnBU0X6ZiOaX62S1y X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a81:9991:0:b0:5ff:6ec3:b8da with SMTP id q139-20020a819991000000b005ff6ec3b8damr1836537ywg.1.1706551538650; Mon, 29 Jan 2024 10:05:38 -0800 (PST) Date: Mon, 29 Jan 2024 19:05:07 +0100 In-Reply-To: <20240129180502.4069817-21-ardb+git@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240129180502.4069817-21-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=2687; i=ardb@kernel.org; h=from:subject; bh=XmOltJN4+bmiY4MWHeNLYuJPJ6TOu7DKSx2rFcdng6s=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIXX7i8s8r3v1rr9Z/GV7jFFs1hRBhh+bbvyp+1p6tmPOX ONr6Zvfd5SyMIhxMMiKKbIIzP77bufpiVK1zrNkYeawMoEMYeDiFICJfDjF8Feyh+FfQIWc0h5F mWnGdoteLW5T478Teqbpf/okTp82pz5GhoaqDJOa37f3eH7suSHndEf1SL5VeaCY8gMVjT/qvzf rcQAA X-Mailer: git-send-email 2.43.0.429.g432eaa2c6b-goog Message-ID: <20240129180502.4069817-25-ardb+git@google.com> Subject: [PATCH v3 04/19] x86/startup_64: Simplify calculation of initial page table address From: Ard Biesheuvel To: linux-kernel@vger.kernel.org Cc: Ard Biesheuvel , Kevin Loughlin , Tom Lendacky , Dionna Glaze , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , Andy Lutomirski , Arnd Bergmann , Nathan Chancellor , Nick Desaulniers , Justin Stitt , Kees Cook , Brian Gerst , linux-arch@vger.kernel.org, llvm@lists.linux.dev Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Ard Biesheuvel Determining the address of the initial page table to program into CR3 involves: - taking the physical address - adding the SME encryption mask On the primary entry path, the code is mapped using a 1:1 virtual to physical translation, so the physical address can be taken directly using a RIP-relative LEA instruction. On the secondary entry path, the address can be obtained by taking the offset from the virtual kernel base (__START_kernel_map) and adding the physical kernel base. This is all very straight-forward, but the current code makes a mess of this. Clean this up. Signed-off-by: Ard Biesheuvel --- arch/x86/kernel/head_64.S | 25 ++++++-------------- 1 file changed, 7 insertions(+), 18 deletions(-) diff --git a/arch/x86/kernel/head_64.S b/arch/x86/kernel/head_64.S index 4017a49d7b76..6d24c2014759 100644 --- a/arch/x86/kernel/head_64.S +++ b/arch/x86/kernel/head_64.S @@ -113,13 +113,11 @@ SYM_CODE_START_NOALIGN(startup_64) call __startup_64 =20 /* Form the CR3 value being sure to include the CR3 modifier */ - addq $(early_top_pgt - __START_KERNEL_map), %rax + leaq early_top_pgt(%rip), %rcx + addq %rcx, %rax =20 #ifdef CONFIG_AMD_MEM_ENCRYPT mov %rax, %rdi - mov %rax, %r14 - - addq phys_base(%rip), %rdi =20 /* * For SEV guests: Verify that the C-bit is correct. A malicious @@ -128,12 +126,6 @@ SYM_CODE_START_NOALIGN(startup_64) * the next RET instruction. */ call sev_verify_cbit - - /* - * Restore CR3 value without the phys_base which will be added - * below, before writing %cr3. - */ - mov %r14, %rax #endif =20 jmp 1f @@ -173,18 +165,18 @@ SYM_INNER_LABEL(secondary_startup_64_no_verify, SYM_L= _GLOBAL) /* Clear %R15 which holds the boot_params pointer on the boot CPU */ xorq %r15, %r15 =20 + /* Derive the runtime physical address of init_top_pgt[] */ + movq phys_base(%rip), %rax + addq $(init_top_pgt - __START_KERNEL_map), %rax + /* * Retrieve the modifier (SME encryption mask if SME is active) to be * added to the initial pgdir entry that will be programmed into CR3. */ #ifdef CONFIG_AMD_MEM_ENCRYPT - movq sme_me_mask, %rax -#else - xorq %rax, %rax + addq sme_me_mask(%rip), %rax #endif =20 - /* Form the CR3 value being sure to include the CR3 modifier */ - addq $(init_top_pgt - __START_KERNEL_map), %rax 1: =20 #ifdef CONFIG_X86_MCE @@ -211,9 +203,6 @@ SYM_INNER_LABEL(secondary_startup_64_no_verify, SYM_L_G= LOBAL) #endif movq %rcx, %cr4 =20 - /* Setup early boot stage 4-/5-level pagetables. */ - addq phys_base(%rip), %rax - /* * Switch to new page-table * --=20 2.43.0.429.g432eaa2c6b-goog From nobody Wed Dec 24 01:33:42 2025 Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4C01176059 for ; Mon, 29 Jan 2024 18:05:42 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.219.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706551543; cv=none; b=mOqkAxXccgmfZ62uADrkr/gzxRkzTQHUYG9DOnRrc5tXMeVVNiEPMMAUDB8KLu/Py5XILwzanGh34uav5HUNQFa0JwvwLriG54F72Hgvae6fuRdNFr8CJScmcJSEhIeuJDSDab8JymbgetodqBqf/Y2yS3ff8lPMqPjdtnjH4VI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706551543; c=relaxed/simple; bh=eEloesx5WYIVvdIkhwITl3KfZCfyKY4osgEzAPlzJuE=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=pf1L5TqjXJyqI9qX+H5VDqAoZWYvHVa4zFEcNKEJivHwst5/Ng7c166Irie91/g/pAMWLvhpuO5bxEJxX9dOp3JciqEwKe6/SZK5+FF8Mwcp2cyS2Zz+fKYo9jQDHwPGWtbA+meoDj3Kzj0OlxyY5AJ9MUqporVY6JBQz3EquAY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=Z1WrCPS/; arc=none smtp.client-ip=209.85.219.201 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="Z1WrCPS/" Received: by mail-yb1-f201.google.com with SMTP id 3f1490d57ef6-dc604c99e95so3278475276.1 for ; Mon, 29 Jan 2024 10:05:42 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1706551541; x=1707156341; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=zW2Nb503VpTEmBC1IeOy5LYeXYJ6a1J8kf+z75WkqKU=; b=Z1WrCPS/4+R/EvI2fSSa4gSVK1FU7pbLA5GaL3tWhRQUfs1pwlf+lH5LrxACuRn79Y vE8L82E61vy8ogm3lIczvMhpBsEuEI6dPvh8f733kQXXiXjekXoi0RGyklkpVEGMa7MJ ONFh4VGsHRXDh4mKw9X9vTKOIpHIEAzyB/EvKbfQ4KgEhMajcVFUNjCry3n53Yk+fAUR pyQy8fLsXFj/794CCHnVzHJRx0SK3HR980ss6jpOTp6NKetiYrLjR4AHKWQI9aqB0EU0 klrn3dZ41aV3zSdYe0KreWabGbTakOo5VcoBUfTgWQbKC/klJw+7hq8A5TLjB+GMVpgG AcNg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1706551541; x=1707156341; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=zW2Nb503VpTEmBC1IeOy5LYeXYJ6a1J8kf+z75WkqKU=; b=BcmMRfTrHqFq4MD6naGZ8yy0MGqs2dh0mOoB9lIfScGHXUJWkoCIPZqMDWi6PqacYD C9XIIJHj+OhSj2WmX+klCQWl7cCPzv8dp6Sw3tT3B3qbui7Sv4CWWkwOWQ2ereZS2s2A BXlMqpdPGCAdWm0ktw6B/pXnIDH8mMs/IKNxHLmx3v43QTRSbOb9RasvRFVDK8BmsmGp ZpjfkS/X7IoPFJMdIC8npfkMSAKBQj4wPfmapNzUsWaovcMCzPs+U1iH47A/Sco4K9aV qcHgA42TO/vsqrFyavkiXLW7IVpY9rf59a2JmTCthFM+LvijvRhXBvz/ntMKwwsCQR/J JyIw== X-Gm-Message-State: AOJu0YzoONe2ERzQa1pW4V3ii50I4B73inCHBvI9tjwgwmucW969/Dq1 RK7x1/85Fb0OtuqNrsAz8H+9Pvqn2VDk22Z9oQVE7Vn/ExNsQX6WUwfBfZ9jyTNb3EMcmg5PotB a/mcW6yYAE+1HadoI1GT9METPW/SESiMaecmyhDrae12yAI58Ocu0KYmVhxeVlf6z2MUTCO0sQu IrKHMr/T1LwSs/dJDmzXA13ayNPqkoUQ== X-Google-Smtp-Source: AGHT+IHoG1nlT9xH7OrNUFwSwleSuhPvAk+iQSlpUq4EhaCK8UijteKSY+nXx9/XbYd43g38b4ws0dig X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a05:6902:2503:b0:dc2:661d:11fc with SMTP id dt3-20020a056902250300b00dc2661d11fcmr358172ybb.8.1706551541086; Mon, 29 Jan 2024 10:05:41 -0800 (PST) Date: Mon, 29 Jan 2024 19:05:08 +0100 In-Reply-To: <20240129180502.4069817-21-ardb+git@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240129180502.4069817-21-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=2468; i=ardb@kernel.org; h=from:subject; bh=N1gJDIfI4P+Gstfr/XNIaKJ5tEagJPCWb57TREbNgFg=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIXX7iytN/vLW339JiJ8PO9e3Z/3ET1Pub2Zt2uCvYNRvv M//oB1XRykLgxgHg6yYIovA7L/vdp6eKFXrPEsWZg4rE8gQBi5OAZhIbwgjw//8s5OSP57Rn9Hh f71rV/YhgY/1/IprLy4+HWhSn3qj6wHDP+1tetd/u38wCl61S0asJVPs5xlJU6n+D+77zl9u8m6 LYwUA X-Mailer: git-send-email 2.43.0.429.g432eaa2c6b-goog Message-ID: <20240129180502.4069817-26-ardb+git@google.com> Subject: [PATCH v3 05/19] x86/startup_64: Simplify CR4 handling in startup code From: Ard Biesheuvel To: linux-kernel@vger.kernel.org Cc: Ard Biesheuvel , Kevin Loughlin , Tom Lendacky , Dionna Glaze , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , Andy Lutomirski , Arnd Bergmann , Nathan Chancellor , Nick Desaulniers , Justin Stitt , Kees Cook , Brian Gerst , linux-arch@vger.kernel.org, llvm@lists.linux.dev Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Ard Biesheuvel When executing in long mode, the CR4.PAE and CR4.LA57 control bits cannot be updated, and so they can simply be preserved rather than reason about whether or not they need to be set. CR4.PSE has no effect in long mode so it can be omitted. CR4.PGE is used to flush the TLBs, by clearing it if it was set, and subsequently re-enabling it. So there is no need to set it just to disable and re-enable it later. CR4.MCE must be preserved unless the kernel was built without CONFIG_X86_MCE, in which case it must be cleared. Reimplement the above logic in a more straight-forward way, by defining a mask of CR4 bits to preserve, and applying that to CR4 at the point where it needs to be updated anyway. Signed-off-by: Ard Biesheuvel --- arch/x86/kernel/head_64.S | 27 ++++++++------------ 1 file changed, 10 insertions(+), 17 deletions(-) diff --git a/arch/x86/kernel/head_64.S b/arch/x86/kernel/head_64.S index 6d24c2014759..ca46995205d4 100644 --- a/arch/x86/kernel/head_64.S +++ b/arch/x86/kernel/head_64.S @@ -179,6 +179,12 @@ SYM_INNER_LABEL(secondary_startup_64_no_verify, SYM_L_= GLOBAL) =20 1: =20 + /* + * Define a mask of CR4 bits to preserve. PAE and LA57 cannot be + * modified while paging remains enabled. PGE will be toggled below if + * it is already set. + */ + movl $(X86_CR4_PAE | X86_CR4_PGE | X86_CR4_LA57), %edx #ifdef CONFIG_X86_MCE /* * Preserve CR4.MCE if the kernel will enable #MC support. @@ -187,22 +193,9 @@ SYM_INNER_LABEL(secondary_startup_64_no_verify, SYM_L_= GLOBAL) * configured will crash the system regardless of the CR4.MCE value set * here. */ - movq %cr4, %rcx - andl $X86_CR4_MCE, %ecx -#else - movl $0, %ecx + orl $X86_CR4_MCE, %edx #endif =20 - /* Enable PAE mode, PSE, PGE and LA57 */ - orl $(X86_CR4_PAE | X86_CR4_PSE | X86_CR4_PGE), %ecx -#ifdef CONFIG_X86_5LEVEL - testb $1, __pgtable_l5_enabled(%rip) - jz 1f - orl $X86_CR4_LA57, %ecx -1: -#endif - movq %rcx, %cr4 - /* * Switch to new page-table * @@ -218,10 +211,10 @@ SYM_INNER_LABEL(secondary_startup_64_no_verify, SYM_L= _GLOBAL) * entries from the identity mapping are flushed. */ movq %cr4, %rcx - movq %rcx, %rax - xorq $X86_CR4_PGE, %rcx + andl %edx, %ecx +0: btcl $X86_CR4_PGE_BIT, %ecx movq %rcx, %cr4 - movq %rax, %cr4 + jc 0b =20 /* Ensure I am executing from virtual addresses */ movq $1f, %rax --=20 2.43.0.429.g432eaa2c6b-goog From nobody Wed Dec 24 01:33:42 2025 Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id CB74E157050 for ; Mon, 29 Jan 2024 18:05:44 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.202 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706551546; cv=none; b=sD5h5ADKf2ZTCF4HHJL3KzCnw8NlvPQVkOqBgvwjFErh22MUNIUiN3Op24sFmhTezm83/eZcytMFrj/47AnI77YUisLFlhg/SSF/9IBBe+tSbJCRt39q47STFOXcRC1PyB8q5pg7y74hecyhd4xroI7imuvyanYQYU8MM1C9GCE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706551546; c=relaxed/simple; bh=H7fHoFCfZwfyk1oPodR7EeB+4BUH/Qm6LmEtPz6akBQ=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=NWFabsy4KVDqQ+OK8VmAHOT8yJtsVeKdED+f1aim64MY+yB7BSMeZXAZvuTHnM/evbkrUVQQJ1PxwJlXrTSJt9TcqmXkSow2brniGIdvcTDesigM+9mru7F7iC4sUh6dlwZ2hlPK1f2gzk8CD7uKYhBgK/NhxRu98dG1cWFjfRA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=gaTO+hyH; arc=none smtp.client-ip=209.85.128.202 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="gaTO+hyH" Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-5fffb2798bfso50675157b3.2 for ; Mon, 29 Jan 2024 10:05:44 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1706551544; x=1707156344; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=hn0YC42oF3TvJQQuUnzb1vQnmOgTTWKFEwunsc8Y+bs=; b=gaTO+hyHUZw1H7PvBpvPo742xc26H+bGoUW+q2WBJC51XvU0wfiZeunayq3zD8y7hs eyhS5+pbe/BqP1nEXqCajWCst/BJMou5j0Jx4CE0ZEQ8hgkkEYmtgnIV22BwX/jxZP3Z 53VtJwRvH1hX2yoOjMFbbCPwqPqUdielq5gXA4mvFww0D+9N+Das2NqaSlVZl00GhRjh oIsbUoZ1yg8mhx29Xp6hYMl2Ll+30Mh9z65B2LK9m6EKJ4gZnhbcfCxvaspsziG4dfKL aIeh2sqYWGRyhVYzSpwQp+LKJnn+WDnwY0POnKRirtmyHlJ61GW/P8tqfjewhvCF9jR3 Vrug== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1706551544; x=1707156344; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=hn0YC42oF3TvJQQuUnzb1vQnmOgTTWKFEwunsc8Y+bs=; b=Bj+//cfXEIyW+Yo6nHY3FK0GQTSc+KGX2iWO2OppxaggOe/sNLt+1qzTkyh1cdw8oo SaNzJiHXeUSTuzM1eODkRN0m2Hh4EjSrrcvNiZJ36TjnsSCnLNhmmX1R1hL4ADA/SRwU 8vA5V3af9exMreEHckN79/CYeDQharUY80esc+Ld1zQ0CUi99VFnVcoGoOCIHnKjzD0Q djlbPYEgPc2n+nVWIrM1OeKxKZVsZhCnq9lp6Tf1+b3aPVMBHBJY6AMjnH/LzR+TXSBh IBoiCggaeqnLkClJnikmuFMjc/kCsW0gUvyXX2tFzIVFIfmgZ9vE+HslZGTC758gYOeR 6amw== X-Gm-Message-State: AOJu0YwWOifJm9haQUw9ewoUFlPb/4bfVZ/YCUWdTdbuQUcGxNMFkJpd N0zJEK4KsVmZe9/yKqY6KBgLco8Uotf6nfEYMTCTBFn9GxohSkPq7a1OEi3TCB7L9rGJlqTymrh 83f+k/0XSuu1dKunrNd1Xri5KmTii10OEhLCI+wE+pIg1NNt2DUm7HBqfesrJownODlamUa9ovy aJ3rOy6QSxXiyDyKtRo2DD75Ny7RUHDA== X-Google-Smtp-Source: AGHT+IHc9J6Eygu67jBTVvQmRSn+tap7OMqKPbaRL96GW4Z6K516VElaEAOe7eg6d1emn1ED83GOQ4Dj X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a81:5743:0:b0:5ff:88f9:96f1 with SMTP id l64-20020a815743000000b005ff88f996f1mr1973294ywb.9.1706551543658; Mon, 29 Jan 2024 10:05:43 -0800 (PST) Date: Mon, 29 Jan 2024 19:05:09 +0100 In-Reply-To: <20240129180502.4069817-21-ardb+git@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240129180502.4069817-21-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=7926; i=ardb@kernel.org; h=from:subject; bh=8VKMuF0Y9EN2NaZalEvyHBoCi3s4P0N7R6CaJlurDdk=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIXX7i6tezr9nH/kcv/2l1P3V6mknuU5z5epaJe1z2nzpu eT271zKHaUsDGIcDLJiiiwCs/++23l6olSt8yxZmDmsTCBDGLg4BWAiMuWMDAu+dK/vvZPRIz7F PILH0+XDdhN2J1b3A7O7Qh8GhT59fIKRYXfYxQOhxt1fRWfMZK4WfqN/5mx286LrPi43qpeI57Z yswEA X-Mailer: git-send-email 2.43.0.429.g432eaa2c6b-goog Message-ID: <20240129180502.4069817-27-ardb+git@google.com> Subject: [PATCH v3 06/19] x86/startup_64: Drop global variables keeping track of LA57 state From: Ard Biesheuvel To: linux-kernel@vger.kernel.org Cc: Ard Biesheuvel , Kevin Loughlin , Tom Lendacky , Dionna Glaze , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , Andy Lutomirski , Arnd Bergmann , Nathan Chancellor , Nick Desaulniers , Justin Stitt , Kees Cook , Brian Gerst , linux-arch@vger.kernel.org, llvm@lists.linux.dev Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Ard Biesheuvel On x86_64, the core kernel is entered in long mode, which implies that paging is enabled. This means that the CR4.LA57 control bit is guaranteed to be in sync with the number of paging levels used by the kernel, and there is no need to store this in a variable. There is also no need to use variables for storing the calculations of pgdir_shift and ptrs_per_p4d, as they are easily determined on the fly. Other assignments of global variables related to the number of paging levels can be deferred to the primary C entrypoint that actually runs from the kernel virtual mapping. This removes the need for writing to __ro_after_init from the code that executes extremely early via the 1:1 mapping. Signed-off-by: Ard Biesheuvel --- arch/x86/boot/compressed/pgtable_64.c | 2 - arch/x86/include/asm/pgtable_64_types.h | 15 +++--- arch/x86/kernel/cpu/common.c | 2 - arch/x86/kernel/head64.c | 52 ++++---------------- arch/x86/mm/kasan_init_64.c | 3 -- arch/x86/mm/mem_encrypt_identity.c | 9 ---- 6 files changed, 15 insertions(+), 68 deletions(-) diff --git a/arch/x86/boot/compressed/pgtable_64.c b/arch/x86/boot/compress= ed/pgtable_64.c index 51f957b24ba7..0586cc216aa6 100644 --- a/arch/x86/boot/compressed/pgtable_64.c +++ b/arch/x86/boot/compressed/pgtable_64.c @@ -128,8 +128,6 @@ asmlinkage void configure_5level_paging(struct boot_par= ams *bp, void *pgtable) =20 /* Initialize variables for 5-level paging */ __pgtable_l5_enabled =3D 1; - pgdir_shift =3D 48; - ptrs_per_p4d =3D 512; } =20 /* diff --git a/arch/x86/include/asm/pgtable_64_types.h b/arch/x86/include/asm= /pgtable_64_types.h index 38b54b992f32..ecc010fbb377 100644 --- a/arch/x86/include/asm/pgtable_64_types.h +++ b/arch/x86/include/asm/pgtable_64_types.h @@ -22,28 +22,25 @@ typedef struct { pteval_t pte; } pte_t; typedef struct { pmdval_t pmd; } pmd_t; =20 #ifdef CONFIG_X86_5LEVEL +#ifdef USE_EARLY_PGTABLE_L5 extern unsigned int __pgtable_l5_enabled; =20 -#ifdef USE_EARLY_PGTABLE_L5 /* - * cpu_feature_enabled() is not available in early boot code. - * Use variable instead. + * CR4.LA57 may not be set to its final value yet in the early boot code. + * Use a variable instead. */ static inline bool pgtable_l5_enabled(void) { return __pgtable_l5_enabled; } #else -#define pgtable_l5_enabled() cpu_feature_enabled(X86_FEATURE_LA57) +#define pgtable_l5_enabled() !!(native_read_cr4() & X86_CR4_LA57) #endif /* USE_EARLY_PGTABLE_L5 */ =20 #else #define pgtable_l5_enabled() 0 #endif /* CONFIG_X86_5LEVEL */ =20 -extern unsigned int pgdir_shift; -extern unsigned int ptrs_per_p4d; - #endif /* !__ASSEMBLY__ */ =20 #define SHARED_KERNEL_PMD 0 @@ -53,7 +50,7 @@ extern unsigned int ptrs_per_p4d; /* * PGDIR_SHIFT determines what a top-level page table entry can map */ -#define PGDIR_SHIFT pgdir_shift +#define PGDIR_SHIFT (pgtable_l5_enabled() ? 48 : 39) #define PTRS_PER_PGD 512 =20 /* @@ -61,7 +58,7 @@ extern unsigned int ptrs_per_p4d; */ #define P4D_SHIFT 39 #define MAX_PTRS_PER_P4D 512 -#define PTRS_PER_P4D ptrs_per_p4d +#define PTRS_PER_P4D (pgtable_l5_enabled() ? 512 : 1) #define P4D_SIZE (_AC(1, UL) << P4D_SHIFT) #define P4D_MASK (~(P4D_SIZE - 1)) =20 diff --git a/arch/x86/kernel/cpu/common.c b/arch/x86/kernel/cpu/common.c index 0b97bcde70c6..20ac11a2c06b 100644 --- a/arch/x86/kernel/cpu/common.c +++ b/arch/x86/kernel/cpu/common.c @@ -1,6 +1,4 @@ // SPDX-License-Identifier: GPL-2.0-only -/* cpu_feature_enabled() cannot be used this early */ -#define USE_EARLY_PGTABLE_L5 =20 #include #include diff --git a/arch/x86/kernel/head64.c b/arch/x86/kernel/head64.c index dc0956067944..d636bb02213f 100644 --- a/arch/x86/kernel/head64.c +++ b/arch/x86/kernel/head64.c @@ -7,9 +7,6 @@ =20 #define DISABLE_BRANCH_PROFILING =20 -/* cpu_feature_enabled() cannot be used this early */ -#define USE_EARLY_PGTABLE_L5 - #include #include #include @@ -50,14 +47,6 @@ extern pmd_t early_dynamic_pgts[EARLY_DYNAMIC_PAGE_TABLE= S][PTRS_PER_PMD]; static unsigned int __initdata next_early_pgt; pmdval_t early_pmd_flags =3D __PAGE_KERNEL_LARGE & ~(_PAGE_GLOBAL | _PAGE_= NX); =20 -#ifdef CONFIG_X86_5LEVEL -unsigned int __pgtable_l5_enabled __ro_after_init; -unsigned int pgdir_shift __ro_after_init =3D 39; -EXPORT_SYMBOL(pgdir_shift); -unsigned int ptrs_per_p4d __ro_after_init =3D 1; -EXPORT_SYMBOL(ptrs_per_p4d); -#endif - #ifdef CONFIG_DYNAMIC_MEMORY_LAYOUT unsigned long page_offset_base __ro_after_init =3D __PAGE_OFFSET_BASE_L4; EXPORT_SYMBOL(page_offset_base); @@ -95,37 +84,6 @@ static unsigned long __head *fixup_long(void *ptr, unsig= ned long physaddr) return fixup_pointer(ptr, physaddr); } =20 -#ifdef CONFIG_X86_5LEVEL -static unsigned int __head *fixup_int(void *ptr, unsigned long physaddr) -{ - return fixup_pointer(ptr, physaddr); -} - -static bool __head check_la57_support(unsigned long physaddr) -{ - /* - * 5-level paging is detected and enabled at kernel decompression - * stage. Only check if it has been enabled there. - */ - if (!(native_read_cr4() & X86_CR4_LA57)) - return false; - - *fixup_int(&__pgtable_l5_enabled, physaddr) =3D 1; - *fixup_int(&pgdir_shift, physaddr) =3D 48; - *fixup_int(&ptrs_per_p4d, physaddr) =3D 512; - *fixup_long(&page_offset_base, physaddr) =3D __PAGE_OFFSET_BASE_L5; - *fixup_long(&vmalloc_base, physaddr) =3D __VMALLOC_BASE_L5; - *fixup_long(&vmemmap_base, physaddr) =3D __VMEMMAP_BASE_L5; - - return true; -} -#else -static bool __head check_la57_support(unsigned long physaddr) -{ - return false; -} -#endif - static unsigned long __head sme_postprocess_startup(struct boot_params *bp= , pmdval_t *pmd) { unsigned long vaddr, vaddr_end; @@ -189,7 +147,7 @@ unsigned long __head __startup_64(unsigned long physadd= r, int i; unsigned int *next_pgt_ptr; =20 - la57 =3D check_la57_support(physaddr); + la57 =3D pgtable_l5_enabled(); =20 /* Is the address too large? */ if (physaddr >> MAX_PHYSMEM_BITS) @@ -486,6 +444,14 @@ asmlinkage __visible void __init __noreturn x86_64_sta= rt_kernel(char * real_mode (__START_KERNEL & PGDIR_MASK))); BUILD_BUG_ON(__fix_to_virt(__end_of_fixed_addresses) <=3D MODULES_END); =20 +#ifdef CONFIG_DYNAMIC_MEMORY_LAYOUT + if (pgtable_l5_enabled()) { + page_offset_base =3D __PAGE_OFFSET_BASE_L5; + vmalloc_base =3D __VMALLOC_BASE_L5; + vmemmap_base =3D __VMEMMAP_BASE_L5; + } +#endif + cr4_init_shadow(); =20 /* Kill off the identity-map trampoline */ diff --git a/arch/x86/mm/kasan_init_64.c b/arch/x86/mm/kasan_init_64.c index 0302491d799d..85ae1ef840cc 100644 --- a/arch/x86/mm/kasan_init_64.c +++ b/arch/x86/mm/kasan_init_64.c @@ -2,9 +2,6 @@ #define DISABLE_BRANCH_PROFILING #define pr_fmt(fmt) "kasan: " fmt =20 -/* cpu_feature_enabled() cannot be used this early */ -#define USE_EARLY_PGTABLE_L5 - #include #include #include diff --git a/arch/x86/mm/mem_encrypt_identity.c b/arch/x86/mm/mem_encrypt_i= dentity.c index 06466f6d5966..2e195866a7fe 100644 --- a/arch/x86/mm/mem_encrypt_identity.c +++ b/arch/x86/mm/mem_encrypt_identity.c @@ -27,15 +27,6 @@ #undef CONFIG_PARAVIRT_XXL #undef CONFIG_PARAVIRT_SPINLOCKS =20 -/* - * This code runs before CPU feature bits are set. By default, the - * pgtable_l5_enabled() function uses bit X86_FEATURE_LA57 to determine if - * 5-level paging is active, so that won't work here. USE_EARLY_PGTABLE_L5 - * is provided to handle this situation and, instead, use a variable that - * has been set by the early boot code. - */ -#define USE_EARLY_PGTABLE_L5 - #include #include #include --=20 2.43.0.429.g432eaa2c6b-goog From nobody Wed Dec 24 01:33:42 2025 Received: from mail-wm1-f74.google.com (mail-wm1-f74.google.com [209.85.128.74]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C139F76059 for ; Mon, 29 Jan 2024 18:05:47 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.74 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706551549; cv=none; b=Odc2p8emXGqA/TkhNVip5mWTNXdSAxjUZ+0+PgZTPaEPcvMaCo6/zlhgeANgQk0gY2GD58JEsfSGhTbjciRPfUk3whnysUSO+FbUeTJqmMRCpZa6SfRDqxvnDFg6FsbjwEfPJ3bMFKckoMcDobDBhUJ2xUbn7a3HxDnjlPMzRGc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706551549; c=relaxed/simple; bh=WA3GaPDqZzEz1qs5YrhEzMU8VCRLnQnkxmbA8Q+jrL4=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=Wa3NhDkpgrYWcTY4NdMqKMLkYT2B8L8iNfTnmY5IA5Qbatcv67eqgiJ9cAIu97jDaTzKFzoFRbNxg6fbW43q9K0ebgST38KdfIezZOxozC4vU6+vwX3HFuwqj8bELsPXSjlKeo4xEDyEFvM/efKp2TNrlPlAJbF5JelyS4Yzy+Y= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=y3Ia4pFK; arc=none smtp.client-ip=209.85.128.74 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="y3Ia4pFK" Received: by mail-wm1-f74.google.com with SMTP id 5b1f17b1804b1-40efbba82c8so2990235e9.0 for ; Mon, 29 Jan 2024 10:05:47 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1706551546; x=1707156346; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=doBLlrV3uqsZT11TuQPgH059AZeb19QGBGbs9zJgK3U=; b=y3Ia4pFKd0uLGRgSsnlMraO2Q/3Y3NGRaXEXs8QsSUuoOTw1rAbM+mXZ82yV/omYhv XccppK2eiv3Jl03XxiZUgP8UF8pEyeiCVCfXHFXnhc7hVPtZkpdANwuoL0fD8H9Rmt1p e24HoU3f9CvglTesVnzSwcQvjIGhPyEHnfnf88ru/oRfiRpa2AOsd4su0KaZlhI8aWxx hWbh8PNf5EbVfaDyajni5DJTK7lDkVEs8MFoOND73cWeA/0hkhXJeJsYbK5ZVamR79qm TModA3gQriAqvhPhMRfXec2EpZhSOIRLfX9JBufqkREv2t9Lho36uCGrHIwFJDYAMAN+ 5+TQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1706551546; x=1707156346; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=doBLlrV3uqsZT11TuQPgH059AZeb19QGBGbs9zJgK3U=; b=pIhuB/bC3dYeXXevgANqMSGMyoHIisZVBwNjWMVW6he2s9/fDk2HRWTP6mo4ZWp4h+ oAwqf4JrfWwhHaZsp3ckE4zVbUQfvwdv3KfbO+dfMB0poWcV5zSx64uVWoBPxkOZ9R23 nHdy5/4I8fxDqoQBR6AFzeCvaTdqDSL08II06u3PgTJYnvXerp9JYp45R80sZ7g1/5Tn wGm2OB/hNrEZ9/8GL/mGkiQB6Kg5x9lFvasIPXvGYUpD6sM4aTEkrM4HpKI2Q6s+cXT/ SqUFt1+Px8GChj6IKHShzXrCsl0NfZa5XiCYHJA0d9g9p/nBqjF9h5UCcrxAbgLYvb73 Mphw== X-Gm-Message-State: AOJu0YwHd19rKFn1sQ4lYOCbPS/uS3I+uRALgDIQBGCkEgbs7nSUYTy/ ckaFqG4FlfKWeBuyZ8UqCrC6mc7G3lVDGNjm3yxFD75ZM0g6PuVzslTSFu+0QwerlDPQjScrzFm CWQDur3Mwaic9ONX/66AQToMBA+bnYwjpQg/cxA0xmL7eXcSD5t5t5H8GQBY9/Httg5FTZ9TrAx 1rAFf34+OyD3HKYC+6iFsX9PRmKP87OQ== X-Google-Smtp-Source: AGHT+IGgLvB19x2zAJjRldl0JEb/xb9QA3oXZZrv8DOC9t5a0RO4E+sxtTda+dI1q6WUKq+yPoiI36ws X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a05:600c:6003:b0:40e:fc7f:56d5 with SMTP id az3-20020a05600c600300b0040efc7f56d5mr2818wmb.2.1706551545993; Mon, 29 Jan 2024 10:05:45 -0800 (PST) Date: Mon, 29 Jan 2024 19:05:10 +0100 In-Reply-To: <20240129180502.4069817-21-ardb+git@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240129180502.4069817-21-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=3319; i=ardb@kernel.org; h=from:subject; bh=+6neEz5nbm/aVlO1e2fNUKJ40Al3PcnJxf6DJ7u3+Uc=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIXX7i2sfjtrc7f3APb8q8oWkdEncnIuz/V6+7d5ssnK9v rNa2+qNHaUsDGIcDLJiiiwCs/++23l6olSt8yxZmDmsTCBDGLg4BWAilawM/8NErkz3/zatq/+f L/fj020ntBoMtp0V9dVZ1bdptmTND2dGhnnC9kwqy0+rX35ZxMBYltTQvDvxIHejVca1+mRfDqY fPAA= X-Mailer: git-send-email 2.43.0.429.g432eaa2c6b-goog Message-ID: <20240129180502.4069817-28-ardb+git@google.com> Subject: [PATCH v3 07/19] x86/startup_64: Simplify virtual switch on primary boot From: Ard Biesheuvel To: linux-kernel@vger.kernel.org Cc: Ard Biesheuvel , Kevin Loughlin , Tom Lendacky , Dionna Glaze , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , Andy Lutomirski , Arnd Bergmann , Nathan Chancellor , Nick Desaulniers , Justin Stitt , Kees Cook , Brian Gerst , linux-arch@vger.kernel.org, llvm@lists.linux.dev Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Ard Biesheuvel The secondary startup code is used on the primary boot path as well, but in this case, the initial part runs from a 1:1 mapping, until an explicit cross-jump is made to the kernel virtual mapping of the same code. On the secondary boot path, this jump is pointless as the code already executes from the mapping targeted by the jump. So combine this cross-jump with the jump from startup_64() into the common boot path. This simplifies the execution flow, and clearly separates code that runs from a 1:1 mapping from code that runs from the kernel virtual mapping. Note that this requires a page table switch, so hoist the CR3 assignment into startup_64() as well. Given that the secondary startup code does not require a special placement inside the executable, move it to the .text section. Signed-off-by: Ard Biesheuvel --- arch/x86/kernel/head_64.S | 41 +++++++++----------- 1 file changed, 19 insertions(+), 22 deletions(-) diff --git a/arch/x86/kernel/head_64.S b/arch/x86/kernel/head_64.S index ca46995205d4..953b82be4cd4 100644 --- a/arch/x86/kernel/head_64.S +++ b/arch/x86/kernel/head_64.S @@ -39,7 +39,6 @@ L4_START_KERNEL =3D l4_index(__START_KERNEL_map) =20 L3_START_KERNEL =3D pud_index(__START_KERNEL_map) =20 - .text __HEAD .code64 SYM_CODE_START_NOALIGN(startup_64) @@ -128,9 +127,19 @@ SYM_CODE_START_NOALIGN(startup_64) call sev_verify_cbit #endif =20 - jmp 1f + /* + * Switch to early_top_pgt which still has the identity mappings + * present. + */ + movq %rax, %cr3 + + /* Branch to the common startup code at its kernel virtual address */ + movq $common_startup_64, %rax + ANNOTATE_RETPOLINE_SAFE + jmp *%rax SYM_CODE_END(startup_64) =20 + .text SYM_CODE_START(secondary_startup_64) UNWIND_HINT_END_OF_STACK ANNOTATE_NOENDBR @@ -176,8 +185,15 @@ SYM_INNER_LABEL(secondary_startup_64_no_verify, SYM_L_= GLOBAL) #ifdef CONFIG_AMD_MEM_ENCRYPT addq sme_me_mask(%rip), %rax #endif + /* + * Switch to the init_top_pgt here, away from the trampoline_pgd and + * unmap the identity mapped ranges. + */ + movq %rax, %cr3 =20 -1: +SYM_INNER_LABEL(common_startup_64, SYM_L_LOCAL) + UNWIND_HINT_END_OF_STACK + ANNOTATE_NOENDBR // above =20 /* * Define a mask of CR4 bits to preserve. PAE and LA57 cannot be @@ -195,17 +211,6 @@ SYM_INNER_LABEL(secondary_startup_64_no_verify, SYM_L_= GLOBAL) */ orl $X86_CR4_MCE, %edx #endif - - /* - * Switch to new page-table - * - * For the boot CPU this switches to early_top_pgt which still has the - * identity mappings present. The secondary CPUs will switch to the - * init_top_pgt here, away from the trampoline_pgd and unmap the - * identity mapped ranges. - */ - movq %rax, %cr3 - /* * Do a global TLB flush after the CR3 switch to make sure the TLB * entries from the identity mapping are flushed. @@ -216,14 +221,6 @@ SYM_INNER_LABEL(secondary_startup_64_no_verify, SYM_L_= GLOBAL) movq %rcx, %cr4 jc 0b =20 - /* Ensure I am executing from virtual addresses */ - movq $1f, %rax - ANNOTATE_RETPOLINE_SAFE - jmp *%rax -1: - UNWIND_HINT_END_OF_STACK - ANNOTATE_NOENDBR // above - #ifdef CONFIG_SMP /* * For parallel boot, the APIC ID is read from the APIC, and then --=20 2.43.0.429.g432eaa2c6b-goog From nobody Wed Dec 24 01:33:42 2025 Received: from mail-wr1-f74.google.com (mail-wr1-f74.google.com [209.85.221.74]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B137B1586DC for ; Mon, 29 Jan 2024 18:05:49 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.221.74 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706551552; cv=none; b=fDh8Bk83sng/8W2qb5nOK6GOAlURYI9tx3mBD3kl4cuMfxiAIjWF7sTvVHLTrLe+lLeYEXtJ4fPpKiwhXsF9M5hvQwG57TZ4dKUr4neOHuKAptMQIXkRePifonN1sRL8/zZb/vA8bXFuvlj7MKvZvNWVcJ0E/o0iAL9od3+QjSY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706551552; c=relaxed/simple; bh=v25fWsh3EuNTmaKxasEu19T5yVTc0toeSGH/1nXhFd8=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=T8Sxow4WMGrXtu/v9WWEB6dqvHq3XYyBC3jUH8aEkdNGfYu4WRltTMMhq9/5nTVlh/yw+iRFPS0WeJpM7tWIvGk6a2EfXtHn1pWWHoAMIgBpt365RmZJeh0QGWe6FL6q242ahe8ipEH1I+neQaC0auLwVjLUiHvTxFVogIEOVI4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=GBOb2r6E; arc=none smtp.client-ip=209.85.221.74 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="GBOb2r6E" Received: by mail-wr1-f74.google.com with SMTP id ffacd0b85a97d-33aeb0d78fbso548323f8f.1 for ; Mon, 29 Jan 2024 10:05:49 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1706551548; x=1707156348; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=9xEowFwD+5Wv00RDtSivSyL+BDRLO0ZehSpXivDed+A=; b=GBOb2r6EC9THXIBIHjRy+X+I9Q0V36WLEyemUlVjTQt+dKB5+VgD43sZHnnfEICZ/R REw+2moR/SYlxFmzxbCbtmwJXbzMe3GOZpmRvZGmTz+MI64LjB/Luaty7qlP7IK/IDAG RP4L9jep+V2Rz9kZ9mf67XIqSPIo0PtlAnt7xN/fhwUh0ySpYTAkutNQY4R7kpgZ2pgN fLZYw1PbVi/W6AEFjLsQnkqLiPOSSbnq2ZVf7W7HrxS0OuLdpBoMQgJCbljEo4DJvHOe ATgOVY2bd8AE5YK5jaHKzGpYeNBt5MtXI+aXFrJSFenRDzxwv1UiBzULFzfdSNC5BHJi ZtUg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1706551548; x=1707156348; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=9xEowFwD+5Wv00RDtSivSyL+BDRLO0ZehSpXivDed+A=; b=ARXk45DRLubKce7MQWbjs72dere1L5s39vQXmiUDGd5DZboDsMhfi63iW7C5zEi1FA wLGcG3vrzSLARlpASl2sDT8NXE+Qhpzvjw+2a4kTRAex1oXgOzF9A3KCHLOQtu3dQKZD KAQJctVb9TgAPsYweHoJFKQ2o0mwyGnVn0mpm9wYSYQSqHsT22NWdvL82b0RDMp9gfI/ srzoVhFDUo0pRD4GhoEebelodm31wbNQmbBmLrbmdN9I18uCla2DgUTN4GCv2z2QWTlc nHrrBb9O8arJNgIo4E22uoIsVio5PCZEJu4S3L44wzFTfCkEO4PJZ40siX3syAEs/lv/ R3GA== X-Gm-Message-State: AOJu0YwyEaiAnBglCV417Pik9XAcDWpGscYTPlVfprN40MxQxYlaKhsD Yq+xI0lJMjpp9a9CD2r02mEB2upBw87k0+Cs3P5bNHnwonwW+qwXklA2YWzbT2ilY03Y0rekZVw llXiSuHFLEiRXqJEf0JvBMRs4tln04desyv9Fc8N9iE3gMHvPdHMJwZe++FBNX8hulzomuyw6hl SvZXY9F+H/fxE/NtqMHH/6VwlKh847tA== X-Google-Smtp-Source: AGHT+IH/PFRjuZAiudnZlR/yChfYKO27SruDlwFgtIWg0D/3dPiM91UIpFpq9eDVXY4Br0fCstrntiUv X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a05:6000:809:b0:33a:e9d6:668a with SMTP id bt9-20020a056000080900b0033ae9d6668amr75756wrb.3.1706551548069; Mon, 29 Jan 2024 10:05:48 -0800 (PST) Date: Mon, 29 Jan 2024 19:05:11 +0100 In-Reply-To: <20240129180502.4069817-21-ardb+git@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240129180502.4069817-21-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=12667; i=ardb@kernel.org; h=from:subject; bh=xCp9y/zniv6u2e+W/u3rfNN518sZwfjch2udjGr9xyA=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIXX7i+szZYN3bri+0CFQTrw8lDWdWVexUljcw+/gh/15D 7i6Vlt1lLIwiHEwyIopsgjM/vtu5+mJUrXOs2Rh5rAygQxh4OIUgIkctWBk2KFXfH+q7bTSpUop G2pP6R1KWrW9fV3eqX/Xkxo2y+bohTD8z/se3tZV42x+fUa+myfraxZTRfaitVVmKjJ/41a5FuQ zAQA= X-Mailer: git-send-email 2.43.0.429.g432eaa2c6b-goog Message-ID: <20240129180502.4069817-29-ardb+git@google.com> Subject: [PATCH v3 08/19] x86/head64: Replace pointer fixups with PIE codegen From: Ard Biesheuvel To: linux-kernel@vger.kernel.org Cc: Ard Biesheuvel , Kevin Loughlin , Tom Lendacky , Dionna Glaze , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , Andy Lutomirski , Arnd Bergmann , Nathan Chancellor , Nick Desaulniers , Justin Stitt , Kees Cook , Brian Gerst , linux-arch@vger.kernel.org, llvm@lists.linux.dev Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Ard Biesheuvel Some of the C code in head64.c may be called from a different virtual address than it was linked at. Currently, we deal with this by using ordinary, position dependent codegen, and fixing up all symbol references on the fly. This is fragile and tricky to maintain. It is also unnecessary: we can use position independent codegen (with hidden visibility) to ensure that all compiler generated symbol references are RIP-relative, removing the need for fixups entirely. It does mean we need explicit references to kernel virtual addresses to be generated by hand, so generate those using a movabs instruction in inline asm in the handful places where we actually need this. Signed-off-by: Ard Biesheuvel --- arch/x86/Makefile | 8 ++ arch/x86/boot/compressed/Makefile | 2 +- arch/x86/include/asm/desc.h | 3 +- arch/x86/include/asm/setup.h | 4 +- arch/x86/kernel/Makefile | 5 ++ arch/x86/kernel/head64.c | 88 +++++++------------- arch/x86/kernel/head_64.S | 5 +- 7 files changed, 51 insertions(+), 64 deletions(-) diff --git a/arch/x86/Makefile b/arch/x86/Makefile index 1a068de12a56..2b5954e75318 100644 --- a/arch/x86/Makefile +++ b/arch/x86/Makefile @@ -168,6 +168,14 @@ else KBUILD_CFLAGS +=3D -mcmodel=3Dkernel KBUILD_RUSTFLAGS +=3D -Cno-redzone=3Dy KBUILD_RUSTFLAGS +=3D -Ccode-model=3Dkernel + + PIE_CFLAGS-$(CONFIG_STACKPROTECTOR) +=3D -fno-stack-protector + PIE_CFLAGS-$(CONFIG_LTO) +=3D -fno-lto + + PIE_CFLAGS :=3D -fpie -mcmodel=3Dsmall $(PIE_CFLAGS-y) \ + -include $(srctree)/include/linux/hidden.h + + export PIE_CFLAGS endif =20 # diff --git a/arch/x86/boot/compressed/Makefile b/arch/x86/boot/compressed/M= akefile index f19c038409aa..bccee07eae60 100644 --- a/arch/x86/boot/compressed/Makefile +++ b/arch/x86/boot/compressed/Makefile @@ -84,7 +84,7 @@ LDFLAGS_vmlinux +=3D -T hostprogs :=3D mkpiggy HOST_EXTRACFLAGS +=3D -I$(srctree)/tools/include =20 -sed-voffset :=3D -e 's/^\([0-9a-fA-F]*\) [ABCDGRSTVW] \(_text\|__bss_start= \|_end\)$$/\#define VO_\2 _AC(0x\1,UL)/p' +sed-voffset :=3D -e 's/^\([0-9a-fA-F]*\) [ABbCDGRSTtVW] \(_text\|__bss_sta= rt\|_end\)$$/\#define VO_\2 _AC(0x\1,UL)/p' =20 quiet_cmd_voffset =3D VOFFSET $@ cmd_voffset =3D $(NM) $< | sed -n $(sed-voffset) > $@ diff --git a/arch/x86/include/asm/desc.h b/arch/x86/include/asm/desc.h index ab97b22ac04a..2e9809feeacd 100644 --- a/arch/x86/include/asm/desc.h +++ b/arch/x86/include/asm/desc.h @@ -134,7 +134,8 @@ static inline void paravirt_free_ldt(struct desc_struct= *ldt, unsigned entries) =20 #define store_ldt(ldt) asm("sldt %0" : "=3Dm"(ldt)) =20 -static inline void native_write_idt_entry(gate_desc *idt, int entry, const= gate_desc *gate) +static __always_inline void +native_write_idt_entry(gate_desc *idt, int entry, const gate_desc *gate) { memcpy(&idt[entry], gate, sizeof(*gate)); } diff --git a/arch/x86/include/asm/setup.h b/arch/x86/include/asm/setup.h index 5c83729c8e71..b004f1b9a052 100644 --- a/arch/x86/include/asm/setup.h +++ b/arch/x86/include/asm/setup.h @@ -47,8 +47,8 @@ extern unsigned long saved_video_mode; =20 extern void reserve_standard_io_resources(void); extern void i386_reserve_resources(void); -extern unsigned long __startup_64(unsigned long physaddr, struct boot_para= ms *bp); -extern void startup_64_setup_env(unsigned long physbase); +extern unsigned long __startup_64(struct boot_params *bp); +extern void startup_64_setup_env(void); extern void early_setup_idt(void); extern void __init do_early_exception(struct pt_regs *regs, int trapnr); =20 diff --git a/arch/x86/kernel/Makefile b/arch/x86/kernel/Makefile index 0000325ab98f..42db41b04d8e 100644 --- a/arch/x86/kernel/Makefile +++ b/arch/x86/kernel/Makefile @@ -21,6 +21,11 @@ CFLAGS_REMOVE_sev.o =3D -pg CFLAGS_REMOVE_rethook.o =3D -pg endif =20 +# head64.c contains C code that may execute from a different virtual addre= ss +# than it was linked at, so we always build it using PIE codegen +CFLAGS_head64.o +=3D $(PIE_CFLAGS) +UBSAN_SANITIZE_head64.o :=3D n + KASAN_SANITIZE_head$(BITS).o :=3D n KASAN_SANITIZE_dumpstack.o :=3D n KASAN_SANITIZE_dumpstack_$(BITS).o :=3D n diff --git a/arch/x86/kernel/head64.c b/arch/x86/kernel/head64.c index d636bb02213f..a4a380494703 100644 --- a/arch/x86/kernel/head64.c +++ b/arch/x86/kernel/head64.c @@ -74,15 +74,10 @@ static struct desc_ptr startup_gdt_descr __initdata =3D= { .address =3D 0, }; =20 -static void __head *fixup_pointer(void *ptr, unsigned long physaddr) -{ - return ptr - (void *)_text + (void *)physaddr; -} - -static unsigned long __head *fixup_long(void *ptr, unsigned long physaddr) -{ - return fixup_pointer(ptr, physaddr); -} +#define __va_symbol(sym) ({ \ + unsigned long __v; \ + asm("movq $" __stringify(sym) ", %0":"=3Dr"(__v)); \ + __v; }) =20 static unsigned long __head sme_postprocess_startup(struct boot_params *bp= , pmdval_t *pmd) { @@ -99,8 +94,8 @@ static unsigned long __head sme_postprocess_startup(struc= t boot_params *bp, pmdv * attribute. */ if (sme_get_me_mask()) { - vaddr =3D (unsigned long)__start_bss_decrypted; - vaddr_end =3D (unsigned long)__end_bss_decrypted; + vaddr =3D __va_symbol(__start_bss_decrypted); + vaddr_end =3D __va_symbol(__end_bss_decrypted); =20 for (; vaddr < vaddr_end; vaddr +=3D PMD_SIZE) { /* @@ -127,25 +122,17 @@ static unsigned long __head sme_postprocess_startup(s= truct boot_params *bp, pmdv return sme_get_me_mask(); } =20 -/* Code in __startup_64() can be relocated during execution, but the compi= ler - * doesn't have to generate PC-relative relocations when accessing globals= from - * that function. Clang actually does not generate them, which leads to - * boot-time crashes. To work around this problem, every global pointer mu= st - * be adjusted using fixup_pointer(). - */ -unsigned long __head __startup_64(unsigned long physaddr, - struct boot_params *bp) +unsigned long __head __startup_64(struct boot_params *bp) { + unsigned long physaddr =3D (unsigned long)_text; unsigned long load_delta, *p; unsigned long pgtable_flags; pgdval_t *pgd; p4dval_t *p4d; pudval_t *pud; pmdval_t *pmd, pmd_entry; - pteval_t *mask_ptr; bool la57; int i; - unsigned int *next_pgt_ptr; =20 la57 =3D pgtable_l5_enabled(); =20 @@ -157,7 +144,7 @@ unsigned long __head __startup_64(unsigned long physadd= r, * Compute the delta between the address I am compiled to run at * and the address I am actually running at. */ - load_delta =3D physaddr - (unsigned long)(_text - __START_KERNEL_map); + load_delta =3D physaddr - (__va_symbol(_text) - __START_KERNEL_map); =20 /* Is the address not 2M aligned? */ if (load_delta & ~PMD_MASK) @@ -168,26 +155,24 @@ unsigned long __head __startup_64(unsigned long physa= ddr, =20 /* Fixup the physical addresses in the page table */ =20 - pgd =3D fixup_pointer(early_top_pgt, physaddr); + pgd =3D (pgdval_t *)early_top_pgt; p =3D pgd + pgd_index(__START_KERNEL_map); if (la57) *p =3D (unsigned long)level4_kernel_pgt; else *p =3D (unsigned long)level3_kernel_pgt; - *p +=3D _PAGE_TABLE_NOENC - __START_KERNEL_map + load_delta; + *p +=3D _PAGE_TABLE_NOENC + sme_get_me_mask(); =20 if (la57) { - p4d =3D fixup_pointer(level4_kernel_pgt, physaddr); + p4d =3D (p4dval_t *)level4_kernel_pgt; p4d[511] +=3D load_delta; } =20 - pud =3D fixup_pointer(level3_kernel_pgt, physaddr); - pud[510] +=3D load_delta; - pud[511] +=3D load_delta; + level3_kernel_pgt[510].pud +=3D load_delta; + level3_kernel_pgt[511].pud +=3D load_delta; =20 - pmd =3D fixup_pointer(level2_fixmap_pgt, physaddr); for (i =3D FIXMAP_PMD_TOP; i > FIXMAP_PMD_TOP - FIXMAP_PMD_NUM; i--) - pmd[i] +=3D load_delta; + level2_fixmap_pgt[i].pmd +=3D load_delta; =20 /* * Set up the identity mapping for the switchover. These @@ -196,15 +181,13 @@ unsigned long __head __startup_64(unsigned long physa= ddr, * it avoids problems around wraparound. */ =20 - next_pgt_ptr =3D fixup_pointer(&next_early_pgt, physaddr); - pud =3D fixup_pointer(early_dynamic_pgts[(*next_pgt_ptr)++], physaddr); - pmd =3D fixup_pointer(early_dynamic_pgts[(*next_pgt_ptr)++], physaddr); + pud =3D (pudval_t *)early_dynamic_pgts[next_early_pgt++]; + pmd =3D (pmdval_t *)early_dynamic_pgts[next_early_pgt++]; =20 pgtable_flags =3D _KERNPG_TABLE_NOENC + sme_get_me_mask(); =20 if (la57) { - p4d =3D fixup_pointer(early_dynamic_pgts[(*next_pgt_ptr)++], - physaddr); + p4d =3D (p4dval_t *)early_dynamic_pgts[next_early_pgt++]; =20 i =3D (physaddr >> PGDIR_SHIFT) % PTRS_PER_PGD; pgd[i + 0] =3D (pgdval_t)p4d + pgtable_flags; @@ -225,8 +208,7 @@ unsigned long __head __startup_64(unsigned long physadd= r, =20 pmd_entry =3D __PAGE_KERNEL_LARGE_EXEC & ~_PAGE_GLOBAL; /* Filter out unsupported __PAGE_KERNEL_* bits: */ - mask_ptr =3D fixup_pointer(&__supported_pte_mask, physaddr); - pmd_entry &=3D *mask_ptr; + pmd_entry &=3D __supported_pte_mask; pmd_entry +=3D sme_get_me_mask(); pmd_entry +=3D physaddr; =20 @@ -252,14 +234,14 @@ unsigned long __head __startup_64(unsigned long physa= ddr, * error, causing the BIOS to halt the system. */ =20 - pmd =3D fixup_pointer(level2_kernel_pgt, physaddr); + pmd =3D (pmdval_t *)level2_kernel_pgt; =20 /* invalidate pages before the kernel image */ - for (i =3D 0; i < pmd_index((unsigned long)_text); i++) + for (i =3D 0; i < pmd_index(__va_symbol(_text)); i++) pmd[i] &=3D ~_PAGE_PRESENT; =20 /* fixup pages that are part of the kernel image */ - for (; i <=3D pmd_index((unsigned long)_end); i++) + for (; i <=3D pmd_index(__va_symbol(_end)); i++) if (pmd[i] & _PAGE_PRESENT) pmd[i] +=3D load_delta; =20 @@ -271,7 +253,7 @@ unsigned long __head __startup_64(unsigned long physadd= r, * Fixup phys_base - remove the memory encryption mask to obtain * the true physical address. */ - *fixup_long(&phys_base, physaddr) +=3D load_delta - sme_get_me_mask(); + phys_base +=3D load_delta - sme_get_me_mask(); =20 return sme_postprocess_startup(bp, pmd); } @@ -553,22 +535,16 @@ static void set_bringup_idt_handler(gate_desc *idt, i= nt n, void *handler) } =20 /* This runs while still in the direct mapping */ -static void __head startup_64_load_idt(unsigned long physbase) +static void __head startup_64_load_idt(void) { - struct desc_ptr *desc =3D fixup_pointer(&bringup_idt_descr, physbase); - gate_desc *idt =3D fixup_pointer(bringup_idt_table, physbase); - - - if (IS_ENABLED(CONFIG_AMD_MEM_ENCRYPT)) { - void *handler; + gate_desc *idt =3D bringup_idt_table; =20 + if (IS_ENABLED(CONFIG_AMD_MEM_ENCRYPT)) /* VMM Communication Exception */ - handler =3D fixup_pointer(vc_no_ghcb, physbase); - set_bringup_idt_handler(idt, X86_TRAP_VC, handler); - } + set_bringup_idt_handler(idt, X86_TRAP_VC, vc_no_ghcb); =20 - desc->address =3D (unsigned long)idt; - native_load_idt(desc); + bringup_idt_descr.address =3D (unsigned long)idt; + native_load_idt(&bringup_idt_descr); } =20 /* This is used when running on kernel addresses */ @@ -587,10 +563,10 @@ void early_setup_idt(void) /* * Setup boot CPU state needed before kernel switches to virtual addresses. */ -void __head startup_64_setup_env(unsigned long physbase) +void __head startup_64_setup_env(void) { /* Load GDT */ - startup_gdt_descr.address =3D (unsigned long)fixup_pointer(startup_gdt, p= hysbase); + startup_gdt_descr.address =3D (unsigned long)startup_gdt; native_load_gdt(&startup_gdt_descr); =20 /* New GDT is live - reload data segment registers */ @@ -598,5 +574,5 @@ void __head startup_64_setup_env(unsigned long physbase) "movl %%eax, %%ss\n" "movl %%eax, %%es\n" : : "a"(__KERNEL_DS) : "memory"); =20 - startup_64_load_idt(physbase); + startup_64_load_idt(); } diff --git a/arch/x86/kernel/head_64.S b/arch/x86/kernel/head_64.S index 953b82be4cd4..b0508e84f756 100644 --- a/arch/x86/kernel/head_64.S +++ b/arch/x86/kernel/head_64.S @@ -67,8 +67,6 @@ SYM_CODE_START_NOALIGN(startup_64) /* Set up the stack for verify_cpu() */ leaq (__end_init_task - PTREGS_SIZE)(%rip), %rsp =20 - leaq _text(%rip), %rdi - /* Setup GSBASE to allow stack canary access for C code */ movl $MSR_GS_BASE, %ecx leaq INIT_PER_CPU_VAR(fixed_percpu_data)(%rip), %rdx @@ -107,8 +105,7 @@ SYM_CODE_START_NOALIGN(startup_64) * is active) to be added to the initial pgdir entry that will be * programmed into CR3. */ - leaq _text(%rip), %rdi - movq %r15, %rsi + movq %r15, %rdi call __startup_64 =20 /* Form the CR3 value being sure to include the CR3 modifier */ --=20 2.43.0.429.g432eaa2c6b-goog From nobody Wed Dec 24 01:33:42 2025 Received: from mail-yb1-f202.google.com (mail-yb1-f202.google.com [209.85.219.202]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7E72B159562 for ; Mon, 29 Jan 2024 18:05:51 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.219.202 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706551553; cv=none; b=ElIZjDiBU/L5xvz0ERghSfqGGrcvEsuZEMOTfjA6StJuxFl6bI9Ajo0EnXc4A4tG9baAVDxACKHWpXHxJeXmbZRj6jrUMWik3ARTrxzxjCxS2p7t5ikAFzEgEKI8WhfAUTHCgD87wzMnQflaWpY1YaO3Yma5qqGGqgnTB37dUzU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706551553; c=relaxed/simple; bh=vnJpTtkwXP8em8CZ9TF/csF2frUk4VGX3QPQ5IolY7Q=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=i+ijttvzxm0xVhDCVzKJiIcGBx4WOY5OD11rF6fcBaOYgDHyG26u99djq+8Mqa3AcZGsYayPscB5HizsqbpNmg8TUakOdWoE6I579mSs8pwmgt9aRDhhl4DBdcIG/uE63LSdJjQ1CLwUbDxj7k9gBOfUkWousl4+UvKx8hiH4rY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=S3uSkdJ1; arc=none smtp.client-ip=209.85.219.202 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="S3uSkdJ1" Received: by mail-yb1-f202.google.com with SMTP id 3f1490d57ef6-d9a541b720aso5390644276.0 for ; Mon, 29 Jan 2024 10:05:51 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1706551550; x=1707156350; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=f5x1F22voMMyYv1Zr6tqhVy0DTOzfkx0XvX+Zrp+ZGY=; b=S3uSkdJ16H3weJLQ/5O4Q3Epqky2y6rRWziWJTHoGi03HYXCwjBFdPoodUKn4RaYrn H5Cl/whowaymXw0dKReDC2Oq7XrvPmA+Xnw0i7FP+1eL8qHnImMk0wQmcL9o1K0dYfId P+xP9dtpb1ULTYodTcpK2L8er53Gvr+4gOHcDNeOG/BlPJ6kepZBEHsUnWWkQry7C9TX yUy/FIaSB+O0bCWKhlroKXmI5qx3NP7LfSapvt97FKICc0yuqEFRsA3c6RK85AuEKS/7 yUAXmISYC+1p+WCkF6GzTFzlRPxlRS2C8k15LQP6y8HmDx7JV4dOAV/xnpjvem6ld0o9 kV9w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1706551550; x=1707156350; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=f5x1F22voMMyYv1Zr6tqhVy0DTOzfkx0XvX+Zrp+ZGY=; b=FzRyfPg7JHQje9ottxG1U4J2yytHpzh2Z40sakQqfF5IAtkHXQb8cXasM2LpqavIHI yki8k0vmmFnhKSDnDnkANqw96Uo+y5g4uWdUv5DWUngS/wwb3H2CT9sCVkR47hS9HnQo Is7Ro6YRiAq+jLgNQDDGXbR994iUV+RQHZe8FWQEcK5oRPskuii6fr7UujHKR+9cgh4J +3DQtv23YaMrwxXZmhoxBV8EybL0pOM/+1Y0BgRKhTZ6pZjrsqrPiGQBFEWNIR7woFvl bIq4KiXILTO28OnGC/qav4lNoV3p0xaW+XGerOg51MqToxKKzvR+YpwyW4UYxtI5hL7z hIqQ== X-Gm-Message-State: AOJu0YyI8nKXOwhXzwBQFBSQEnyL7DR18DgLgNURwn2K5921kOvuFWOm N8CzDcqGjP5UuY1wgZ5QlsEf2saARUxcQZDlzNiUhIVIYQyd+7XiwsM/hDVme/NtxA5Uvxi0yID nL95mbYiRvGDUAnglnm00+IQnJIRIUdUymMTfjugKSp5eCE3k6XK8N1dplErQM48/y0fSfo0dNw 9pTHMGUmc7y9oRSPtXvSP7o9GDzbdlog== X-Google-Smtp-Source: AGHT+IGgNqs4arPCH5rAGfnn6yAEx3ibS3mxOByBiz/gXz1ea0MTq5J9nfV4BBbOTaOTO6X7O+L/ylT4 X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a05:6902:108f:b0:dc2:23d8:722d with SMTP id v15-20020a056902108f00b00dc223d8722dmr2386953ybu.13.1706551550415; Mon, 29 Jan 2024 10:05:50 -0800 (PST) Date: Mon, 29 Jan 2024 19:05:12 +0100 In-Reply-To: <20240129180502.4069817-21-ardb+git@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240129180502.4069817-21-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=4031; i=ardb@kernel.org; h=from:subject; bh=GRGmQn1xgXFPK1qedgfyRwvXHekb7bPD6uUy2cYh0y8=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIXX7i5vrWbMWbRB/pTXhQYTroe6r825LXbbdoZi54/syE faa5cFLOkpZGMQ4GGTFFFkEZv99t/P0RKla51myMHNYmUCGMHBxCsBEGP4y/OFJ1fCZvC+r871W QXDCwtItz6p5Qo1+Beg0rJB627tLQ5mR4Wub4nvV5lBLhY6ik+UvEn+ZrPQ889P7+dUNae+vLHl 0ixcA X-Mailer: git-send-email 2.43.0.429.g432eaa2c6b-goog Message-ID: <20240129180502.4069817-30-ardb+git@google.com> Subject: [PATCH v3 09/19] x86/head64: Simplify GDT/IDT initialization code From: Ard Biesheuvel To: linux-kernel@vger.kernel.org Cc: Ard Biesheuvel , Kevin Loughlin , Tom Lendacky , Dionna Glaze , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , Andy Lutomirski , Arnd Bergmann , Nathan Chancellor , Nick Desaulniers , Justin Stitt , Kees Cook , Brian Gerst , linux-arch@vger.kernel.org, llvm@lists.linux.dev Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Ard Biesheuvel There used to be two separate code paths for programming the IDT early: one that was called via the 1:1 mapping, and one via the kernel virtual mapping, where the former used explicit pointer fixups to obtain 1:1 mapped addresses. That distinction is now gone so the GDT/IDT init code can be unified and simplified accordingly. Signed-off-by: Ard Biesheuvel --- arch/x86/kernel/head64.c | 57 +++++++------------- 1 file changed, 18 insertions(+), 39 deletions(-) diff --git a/arch/x86/kernel/head64.c b/arch/x86/kernel/head64.c index a4a380494703..58c58c66dec9 100644 --- a/arch/x86/kernel/head64.c +++ b/arch/x86/kernel/head64.c @@ -59,21 +59,12 @@ EXPORT_SYMBOL(vmemmap_base); /* * GDT used on the boot CPU before switching to virtual addresses. */ -static struct desc_struct startup_gdt[GDT_ENTRIES] __initdata =3D { +static struct desc_struct startup_gdt[GDT_ENTRIES] __initconst =3D { [GDT_ENTRY_KERNEL32_CS] =3D GDT_ENTRY_INIT(DESC_CODE32, 0, 0xffff= f), [GDT_ENTRY_KERNEL_CS] =3D GDT_ENTRY_INIT(DESC_CODE64, 0, 0xffff= f), [GDT_ENTRY_KERNEL_DS] =3D GDT_ENTRY_INIT(DESC_DATA64, 0, 0xffff= f), }; =20 -/* - * Address needs to be set at runtime because it references the startup_gdt - * while the kernel still uses a direct mapping. - */ -static struct desc_ptr startup_gdt_descr __initdata =3D { - .size =3D sizeof(startup_gdt)-1, - .address =3D 0, -}; - #define __va_symbol(sym) ({ \ unsigned long __v; \ asm("movq $" __stringify(sym) ", %0":"=3Dr"(__v)); \ @@ -517,47 +508,32 @@ void __init __noreturn x86_64_start_reservations(char= *real_mode_data) */ static gate_desc bringup_idt_table[NUM_EXCEPTION_VECTORS] __page_aligned_d= ata; =20 -static struct desc_ptr bringup_idt_descr =3D { - .size =3D (NUM_EXCEPTION_VECTORS * sizeof(gate_desc)) - 1, - .address =3D 0, /* Set at runtime */ -}; - -static void set_bringup_idt_handler(gate_desc *idt, int n, void *handler) -{ -#ifdef CONFIG_AMD_MEM_ENCRYPT - struct idt_data data; - gate_desc desc; - - init_idt_data(&data, n, handler); - idt_init_desc(&desc, &data); - native_write_idt_entry(idt, n, &desc); -#endif -} - -/* This runs while still in the direct mapping */ -static void __head startup_64_load_idt(void) +static void early_load_idt(void (*handler)(void)) { gate_desc *idt =3D bringup_idt_table; + struct desc_ptr bringup_idt_descr; + + if (IS_ENABLED(CONFIG_AMD_MEM_ENCRYPT)) { + struct idt_data data; + gate_desc desc; =20 - if (IS_ENABLED(CONFIG_AMD_MEM_ENCRYPT)) /* VMM Communication Exception */ - set_bringup_idt_handler(idt, X86_TRAP_VC, vc_no_ghcb); + init_idt_data(&data, X86_TRAP_VC, handler); + idt_init_desc(&desc, &data); + native_write_idt_entry(idt, X86_TRAP_VC, &desc); + } =20 bringup_idt_descr.address =3D (unsigned long)idt; + bringup_idt_descr.size =3D sizeof(bringup_idt_table); native_load_idt(&bringup_idt_descr); } =20 -/* This is used when running on kernel addresses */ void early_setup_idt(void) { - /* VMM Communication Exception */ - if (IS_ENABLED(CONFIG_AMD_MEM_ENCRYPT)) { + if (IS_ENABLED(CONFIG_AMD_MEM_ENCRYPT)) setup_ghcb(); - set_bringup_idt_handler(bringup_idt_table, X86_TRAP_VC, vc_boot_ghcb); - } =20 - bringup_idt_descr.address =3D (unsigned long)bringup_idt_table; - native_load_idt(&bringup_idt_descr); + early_load_idt(vc_boot_ghcb); } =20 /* @@ -565,8 +541,11 @@ void early_setup_idt(void) */ void __head startup_64_setup_env(void) { + struct desc_ptr startup_gdt_descr; + /* Load GDT */ startup_gdt_descr.address =3D (unsigned long)startup_gdt; + startup_gdt_descr.size =3D sizeof(startup_gdt) - 1; native_load_gdt(&startup_gdt_descr); =20 /* New GDT is live - reload data segment registers */ @@ -574,5 +553,5 @@ void __head startup_64_setup_env(void) "movl %%eax, %%ss\n" "movl %%eax, %%es\n" : : "a"(__KERNEL_DS) : "memory"); =20 - startup_64_load_idt(); + early_load_idt(vc_no_ghcb); } --=20 2.43.0.429.g432eaa2c6b-goog From nobody Wed Dec 24 01:33:42 2025 Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B32EB159560 for ; Mon, 29 Jan 2024 18:05:53 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.219.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706551555; cv=none; b=lMgqJNSBGgRssKb99/4f5NSuuIMQhYH0cOBpmc1A29hU3D7OB5ZdtpaNxF1ER0RHBtNXCwaBp0LoXzT7GhTNqrizv1kWuVSOD95aR1rvqQF2uBv+ZiF7VY3f0UYuYZWAlgvH8/ejuTADj1vwIDXyVrn41dxA+87ma5crWcfIIVI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706551555; c=relaxed/simple; bh=UonHlMqjgEJH+56iHsla4J1Fp0ovIjuTCcCNqIOSjk0=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=Qltc/y02OhKu3H9nKGjvMZo/UPVrj4+r7xnuow8XyGq5mWTKl0jLQjd23Z017LhqZ5xGGw9Fosj+rMQbDAjV2Y0WwDQq3NC4Oqq/owZjPOrAflyTgSCSKaz7p4ZzsXvmBCFVddZM9InWe3ZsbRhSqWNfi+opSThyETfDJypb17Q= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=4GM0rg7A; arc=none smtp.client-ip=209.85.219.201 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="4GM0rg7A" Received: by mail-yb1-f201.google.com with SMTP id 3f1490d57ef6-dc64e0fc7c8so4071021276.2 for ; Mon, 29 Jan 2024 10:05:53 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1706551553; x=1707156353; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=Ap19tN/uXJaW7OWqMmvQ4dnQyY/I7jMfOfg7BtI2vyw=; b=4GM0rg7AYbDNE5b5L8LZUCm/+mATnFwYmb2Xkjvp2c9whD1wMHMzU+vmAr5Bzjofcr iT1wBEwdmky7FvDrDhkWQ2iPjijLD6RlNS41sbIZuREqAmuGX+oPCconluCWvxlEfbt/ Ti5Ughww7+m0YYmff6vRvOtlIKsUwXPtIaeqKASD0sP5OcQ5x63Mh6uNzu9vDltoD5DC 4myoQEx64XVmkdjN22Smx1YfZJndmb+qt+N6p6vKH+wfYKYZjb7g3gP9sMPy8uyjjDPN UdYc5rDHSroKTGu9WgviNBbpR3CTNWXI+apI9zQ0R/IKuqt5stMuPtLodGpi8S0KZ0Av Decw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1706551553; x=1707156353; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=Ap19tN/uXJaW7OWqMmvQ4dnQyY/I7jMfOfg7BtI2vyw=; b=Qff15sBM2m8OpA3IEL7JCEi72UNGzTXjl3jS3oI/s8B0Sbg6h5dMMqY+f30g+0EmPI SWprxY95wnWxAs1cPnOwxZ6WrgnGS/YME2S9ls6c00ueVsBgJabCIkTqG5ZPxiX+UCOs SoL5Ul9MHtnF2ovBTX7x+nxIXlC2V9YY4+KAn1pgC/MEiqH2fnSogW/9RqaoIY1sPeCE k5YJF/+0j8Y2zU9JrrWiLrGGYEoJfhcPVrP9JbUsEKhiRm5/5cB2k/U87RLm2el53Ur0 u83rYIrpZuYTy2j2A8VSoH90QqRPVwXBLClnsUHLQhp2dKEb0ZQi9LUO8/MJsNw29Dam wtMQ== X-Gm-Message-State: AOJu0YxfyCgg886uxsXGIOC0SpmEn44RjPiH3sC9otxLkf7t06AWvCuu RUb5c6+7rTq190p7f90nIAdX+wRg7rOBWErsYNlZbyK3LtHJD8wHIy3K8p4GosNYSPyFodoCO7v TcEsKJkVJ2VD26Qk2e29pVqP2h1h9hwPApmvttLvkQSX1MClhs/fgbLwGYwL1rDAZIT9d2c2HUz NabhZEP8DjF8khqR7LDHd3KAsLaEMcqA== X-Google-Smtp-Source: AGHT+IFdlkgRmOLuPYQ4HYGJOcfOyvmrF8pJsjH+7Ju2xNGTWiYFEILGYxiFeCWVd7IhoWiApQSqZNZJ X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a05:6902:1b01:b0:dc2:3619:e94e with SMTP id eh1-20020a0569021b0100b00dc23619e94emr414348ybb.6.1706551552670; Mon, 29 Jan 2024 10:05:52 -0800 (PST) Date: Mon, 29 Jan 2024 19:05:13 +0100 In-Reply-To: <20240129180502.4069817-21-ardb+git@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240129180502.4069817-21-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=4822; i=ardb@kernel.org; h=from:subject; bh=Kb1Zn5CHQVUPzxgeHo6uRlzKgwc2NWlyfd+RzWsJs24=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIXX7i1tSpVekMhqlzlrlfb/JMsl/O//tz/rfnvtxfTN9K xfxQaivo5SFQYyDQVZMkUVg9t93O09PlKp1niULM4eVCWQIAxenAEwkX5Lhv5OomU7pjraIyH/7 ztt9/Of40YFDqmqzV+W3Fp85b/RspRj+yorzxqrcmK60VSHRPl6b5eeETAbXibMsFrfufP7hcKI IBwA= X-Mailer: git-send-email 2.43.0.429.g432eaa2c6b-goog Message-ID: <20240129180502.4069817-31-ardb+git@google.com> Subject: [PATCH v3 10/19] asm-generic: Add special .pi.text section for position independent code From: Ard Biesheuvel To: linux-kernel@vger.kernel.org Cc: Ard Biesheuvel , Kevin Loughlin , Tom Lendacky , Dionna Glaze , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , Andy Lutomirski , Arnd Bergmann , Nathan Chancellor , Nick Desaulniers , Justin Stitt , Kees Cook , Brian Gerst , linux-arch@vger.kernel.org, llvm@lists.linux.dev Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Ard Biesheuvel Add a special .pi.text section that architectures will use to carry code that can be called while the kernel is executing from a different virtual address than its link time address. This is typically needed by very early boot code that executes from a 1:1 mapping, and may need to call into other code to perform preparatory tasks that must be completed before switching to the kernel's ordinary virtual mapping. Note that this implies that the code in question cannot generally be instrumented safely, and so the contents are combined with the existing .noinstr.text section, making .pi.text a proper subset of the former. Signed-off-by: Ard Biesheuvel --- include/asm-generic/vmlinux.lds.h | 3 +++ include/linux/init.h | 12 +++++++++ scripts/mod/modpost.c | 5 +++- tools/objtool/check.c | 26 ++++++++------------ 4 files changed, 29 insertions(+), 17 deletions(-) diff --git a/include/asm-generic/vmlinux.lds.h b/include/asm-generic/vmlinu= x.lds.h index 5dd3a61d673d..70c9767cac5a 100644 --- a/include/asm-generic/vmlinux.lds.h +++ b/include/asm-generic/vmlinux.lds.h @@ -553,6 +553,9 @@ __cpuidle_text_start =3D .; \ *(.cpuidle.text) \ __cpuidle_text_end =3D .; \ + __pi_text_start =3D .; \ + *(.pi.text) \ + __pi_text_end =3D .; \ __noinstr_text_end =3D .; =20 /* diff --git a/include/linux/init.h b/include/linux/init.h index 3fa3f6241350..85bb701b664c 100644 --- a/include/linux/init.h +++ b/include/linux/init.h @@ -55,6 +55,17 @@ #define __exitdata __section(".exit.data") #define __exit_call __used __section(".exitcall.exit") =20 +/* + * __pitext should be used to mark code that can execute correctly from a + * different virtual offset than the kernel was linked at. This is used for + * code that is called extremely early during boot. + * + * Note that this is incompatible with KAsan, which applies an affine + * translation to the virtual address to obtain the shadow address which is + * strictly tied to the kernel's virtual address space. + */ +#define __pitext __section(".pi.text") __no_sanitize_address notrace + /* * modpost check for section mismatches during the kernel build. * A section mismatch happens when there are references from a @@ -92,6 +103,7 @@ =20 /* For assembly routines */ #define __HEAD .section ".head.text","ax" +#define __PITEXT .section ".pi.text","ax" #define __INIT .section ".init.text","ax" #define __FINIT .previous =20 diff --git a/scripts/mod/modpost.c b/scripts/mod/modpost.c index 795b21154446..962d00df47ab 100644 --- a/scripts/mod/modpost.c +++ b/scripts/mod/modpost.c @@ -813,9 +813,12 @@ static void check_section(const char *modname, struct = elf_info *elf, =20 #define INIT_SECTIONS ".init.*" =20 -#define ALL_TEXT_SECTIONS ".init.text", ".meminit.text", ".exit.text", \ +#define ALL_PI_TEXT_SECTIONS ".pi.text", ".pi.text.*" +#define ALL_NON_PI_TEXT_SECTIONS ".init.text", ".meminit.text", ".exit.te= xt", \ TEXT_SECTIONS, OTHER_TEXT_SECTIONS =20 +#define ALL_TEXT_SECTIONS ALL_NON_PI_TEXT_SECTIONS, ALL_PI_TEXT_SECTIONS + enum mismatch { TEXTDATA_TO_ANY_INIT_EXIT, XXXINIT_TO_SOME_INIT, diff --git a/tools/objtool/check.c b/tools/objtool/check.c index 548ec3cd7c00..af8f23a96037 100644 --- a/tools/objtool/check.c +++ b/tools/objtool/check.c @@ -389,6 +389,7 @@ static int decode_instructions(struct objtool_file *fil= e) if (!strcmp(sec->name, ".noinstr.text") || !strcmp(sec->name, ".entry.text") || !strcmp(sec->name, ".cpuidle.text") || + !strncmp(sec->name, ".pi.text", 8) || !strncmp(sec->name, ".text..__x86.", 13)) sec->noinstr =3D true; =20 @@ -4234,23 +4235,16 @@ static int validate_noinstr_sections(struct objtool= _file *file) { struct section *sec; int warnings =3D 0; + static char const *noinstr_sections[] =3D { + ".noinstr.text", ".entry.text", ".cpuidle.text", ".pi.text", + }; =20 - sec =3D find_section_by_name(file->elf, ".noinstr.text"); - if (sec) { - warnings +=3D validate_section(file, sec); - warnings +=3D validate_unwind_hints(file, sec); - } - - sec =3D find_section_by_name(file->elf, ".entry.text"); - if (sec) { - warnings +=3D validate_section(file, sec); - warnings +=3D validate_unwind_hints(file, sec); - } - - sec =3D find_section_by_name(file->elf, ".cpuidle.text"); - if (sec) { - warnings +=3D validate_section(file, sec); - warnings +=3D validate_unwind_hints(file, sec); + for (int i =3D 0; i < ARRAY_SIZE(noinstr_sections); i++) { + sec =3D find_section_by_name(file->elf, noinstr_sections[i]); + if (sec) { + warnings +=3D validate_section(file, sec); + warnings +=3D validate_unwind_hints(file, sec); + } } =20 return warnings; --=20 2.43.0.429.g432eaa2c6b-goog From nobody Wed Dec 24 01:33:42 2025 Received: from mail-wm1-f73.google.com (mail-wm1-f73.google.com [209.85.128.73]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E489515A48D for ; Mon, 29 Jan 2024 18:05:56 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.73 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706551558; cv=none; b=VNcxnEJenie8z8Njx6ZJdAEON5xw5GAA91w49GwmjxvzsnrHbMuDr8AWQqXlKv0DyMXnBZ2DPXPC6YYzuF82ZLVwkLgIRYCAHxkyjTxeWxMoy7KalnvuEspHZpM5Gf6R1Cnj7UZ1p8e/KR+VLk1J9Hwr5DElpzprhTTwEhF/7GM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706551558; c=relaxed/simple; bh=BHUup4Ao15H5a0x4s+HXfGoWgWU/14gkkZKIK0ZXAdk=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=hjviLRNptwlypvTGWQrbVtwNlz+riX20FhqoApUHmBhyUp7h+Y4TQoKrTabsFLD6OwXg5U/hx+MPsmGMHY4pLHCQTwmbvZB4fWH1YyWI+/FT6vPkefTskE0PN7W1rZACYzbNGrVfa5hlnhfNxhETbztdIMuEFNP7Vf0QQe/iTao= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=I1V788Gb; arc=none smtp.client-ip=209.85.128.73 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="I1V788Gb" Received: by mail-wm1-f73.google.com with SMTP id 5b1f17b1804b1-40e8810b5f3so25160775e9.3 for ; Mon, 29 Jan 2024 10:05:56 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1706551555; x=1707156355; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=usiDcusxejCbluC36Vpk0on80rVNZ7Itzv8e5WwmSQI=; b=I1V788GbqvKtKxxYQnLQzkd87zUujtwsJrEsJtzWQtl7X1BEhozGG3JraJCpp6pEAT wjOnayYq44aWOdRcgjaYdefkwCD532Aw3Awogc8dGVhjbMDxL2/8WiEKC+7ksff3GRkE MYP6U+QivGClA9HEiE4ve+CxSty3BXEGTJZobqd1MIBTcJ9Ewi/mSSFtQ2x/04NjNAN7 xpzk3Xop3BJh3sDvfcM3d3HamVDH7wI4LeemoYRMI5udVyRPe+KUiZF5fX+X8Djjg8hJ qYgUK9VBReJpA0xc/5IbQl4iAijpH1zpSNzcaXHDyuT+OX20QIxmTQ2p1wSt5FED6gJ6 BNBw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1706551555; x=1707156355; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=usiDcusxejCbluC36Vpk0on80rVNZ7Itzv8e5WwmSQI=; b=N+Vv9qMVxMiv/0pHvKTg9hhiZlKIKAuowrtvCppM0a5xM2ek/MLl9Gq0uDphQkZfiN 0zc2Kv8j7KxtuJ0LYoqlpvpvCln/GpvrfCXLSF7pCRMmIc2uGYaYYE2bnyHwtO9KW1ML 6s2SUHosBKuCZrDKGKFV3k5MY3WGM4+ZsmueUXyyaydrIoiomT+MqDsRAF6AHUX8xA61 FgR4rT4f4Hkysk25QOsMeZfFwVVcgFO3Wr2YS+0pn64KI1lPI7pFtcLc88xBG3ryhtvX PD6NbiUdLasLXvXSQmpZJahV2Kxy/Cz/ck04joxMesDMLt4HcKCnpNYS3/hvJLwkuSPs PnWg== X-Gm-Message-State: AOJu0YwG3V+lMrGnquMtUhG1hHdg5ApEdOpY09nKSJYnMEvWyo7bwMVO az+2wchLTDw1m22qzC1H23WFkO56L/0aavsyeULxZuGh5RT1CZ4mW3Lf4Z8T4C4huHRXsxA34ay 9lq/SZ/D0joIty1qJPqHsan/SkcsCfaJK6JJxe3Lht89OMIJCtlfS73I6ZQDhzFwWA9DcZteb0y WuGJ2HE/wJYMOtiiW/J/lDvvtIf2a83g== X-Google-Smtp-Source: AGHT+IEH4d52wGH8vbNwfzvpV2wqsfdLfYYhb20TejcQqFRaHlU9GjFY3tNK9qHPkBnSWswi3vS1ehXT X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a05:600c:1d88:b0:40e:f9fa:14ea with SMTP id p8-20020a05600c1d8800b0040ef9fa14eamr11573wms.6.1706551554793; Mon, 29 Jan 2024 10:05:54 -0800 (PST) Date: Mon, 29 Jan 2024 19:05:14 +0100 In-Reply-To: <20240129180502.4069817-21-ardb+git@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240129180502.4069817-21-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=1335; i=ardb@kernel.org; h=from:subject; bh=+Hn8XwOONyFACfVl0+TV+ToDQrCQwdquUdW+H4YgWKM=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIXX7i9u7Fp4sj3V7udXtpNY9MXsR10NPBb0cuP92X1ry+ VW5ZJd4RykLgxgHg6yYIovA7L/vdp6eKFXrPEsWZg4rE8gQBi5OAZjIfTOG/9We01eGNZ24aN3g GLjUIHZG9EL9Tn/mrwzhRufLA4NNixl+s9z50czfGr3v0OlmYdt8rSt7pp3lK5l6zCp89pbDm8z uswAA X-Mailer: git-send-email 2.43.0.429.g432eaa2c6b-goog Message-ID: <20240129180502.4069817-32-ardb+git@google.com> Subject: [PATCH v3 11/19] x86: Move return_thunk to __pitext section From: Ard Biesheuvel To: linux-kernel@vger.kernel.org Cc: Ard Biesheuvel , Kevin Loughlin , Tom Lendacky , Dionna Glaze , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , Andy Lutomirski , Arnd Bergmann , Nathan Chancellor , Nick Desaulniers , Justin Stitt , Kees Cook , Brian Gerst , linux-arch@vger.kernel.org, llvm@lists.linux.dev Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Ard Biesheuvel The x86 return thunk will function correctly even when it is called via a different virtual mapping than the one it was linked at, so it can safely be moved to .pi.text. This allows other code in that section to call it. Signed-off-by: Ard Biesheuvel --- arch/x86/kernel/vmlinux.lds.S | 2 +- arch/x86/lib/retpoline.S | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/arch/x86/kernel/vmlinux.lds.S b/arch/x86/kernel/vmlinux.lds.S index a349dbfc6d5a..77262e804250 100644 --- a/arch/x86/kernel/vmlinux.lds.S +++ b/arch/x86/kernel/vmlinux.lds.S @@ -134,7 +134,7 @@ SECTIONS SOFTIRQENTRY_TEXT #ifdef CONFIG_RETPOLINE *(.text..__x86.indirect_thunk) - *(.text..__x86.return_thunk) + *(.pi.text..__x86.return_thunk) #endif STATIC_CALL_TEXT =20 diff --git a/arch/x86/lib/retpoline.S b/arch/x86/lib/retpoline.S index 7b2589877d06..003b35445bbb 100644 --- a/arch/x86/lib/retpoline.S +++ b/arch/x86/lib/retpoline.S @@ -136,7 +136,7 @@ SYM_CODE_END(__x86_indirect_jump_thunk_array) * relocations for same-section JMPs and that breaks the returns * detection logic in apply_returns() and in objtool. */ - .section .text..__x86.return_thunk + .section .pi.text..__x86.return_thunk, "ax" =20 #ifdef CONFIG_CPU_SRSO =20 --=20 2.43.0.429.g432eaa2c6b-goog From nobody Wed Dec 24 01:33:42 2025 Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 933736F08C for ; Mon, 29 Jan 2024 18:05:58 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.202 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706551560; cv=none; b=S2haB4SHFbW7lAtEvq3jX4ew2bB/VLEPhp3M5iS3pf1WUNNJGDLq/F1mG/liPldmtq7F6shznxA+cCUHEk+u59Lm4dshV5UYj+PMea/naOYWXWp9aB9jnInzKP2OEg2ms+HT05+G0W8+296tp/lvup3dmPyUUXVr5SFkdwlGMm4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706551560; c=relaxed/simple; bh=7IrvxjfmjsArA7ueoVb4ihX33flmNaxHeDwW0+SLbEM=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=V8xicTakzqJi1UaHzpFnuCERv+yfTEYX5/vYQgUtK3D6rstFXhZj/uiu4nzv4zg9qYlWFlwQyqO2+zxigPbeJQw8DNrMCfQ8RKop0BCBb5lENDphbesh+LSQ41sBfOwGZdB0iyLSqAwtFdUDVku1zkk1uGwHfUrmlJFfhNgiGao= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=ZD8UOIxr; arc=none smtp.client-ip=209.85.128.202 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="ZD8UOIxr" Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-5ee22efe5eeso47851397b3.3 for ; Mon, 29 Jan 2024 10:05:58 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1706551557; x=1707156357; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=U5uVNAFoC2AEC7S4OvYtDKlpwaA5zaGsPfiWByf41yo=; b=ZD8UOIxrEwMCy4JAInsIJ84bcGwJ13h74rpJAoc75IU+KOQoUi2vH1aJcYF/XSclWY zpBnpXjvKMdlVaVmG9XskEVa3xyjfW5N9yedLFWz2wROXKa+4csMDgoHmg0Lu3vmWHTd lQa0mXAoy2ED5R7myb0nNQCbJBWpk8jSO4boAJGvSS+ANQQuI3ZcVVVBkZPjLU3K+XR7 FrfB/+4VRAZF6dNdXPEq9bK2BHA5ydHXMYqxG/NUsclLsePKeDhsjM1mJZQsTFYT+12V Umya7gcN5QORstvgV//AoI72+WLjSl5TpnGB8VoEw9Gio/wvK5gWNlbbu8PDUUKH8qkH vK+g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1706551557; x=1707156357; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=U5uVNAFoC2AEC7S4OvYtDKlpwaA5zaGsPfiWByf41yo=; b=lM2DuUHflRRUn/A1NPuGjbCYlpRFB0LPF2ai8zqM8TAW1lW9pqY+6Km9VlxahxHRpG BBuF7oaXvOnhoYlF1+zZFuzi5DKCN8r3HPpIL2QAKw6ZcsRPDeJd0oQwXVU/GDPomcQq BwuYI1bbFTtJ4CAHbhXgazJTnaprUWA2oNTNE3oKQ8RWUA+TNc8QwCZEuYXCRtg3FqRQ f+3ur4OfZBygcM9YQO75rSnf7+ZAogSEXYBeevD4oq8xscnV/BUC2o5nHaqZuyXCnp+h 5dSd0/7EZ1eT5gZXv2dF3KCGkhNPrQaVsWcFXVePJyBe6vrcckCokQRUxKFgxdZj5C82 fKNQ== X-Gm-Message-State: AOJu0Yxe+AwHGDlSP3kfu2FD4ngaApzdLkYFI03L2uoXXUfUp59xNKMa Sl2UU+eEeM+MY0ZEFKsqHHxIJYEJ9vLniyUMxd+Q+9XcaW3NmgCmW+lV4GqIWYsiggB/tACmNie GfzNJKz/diAauPFLKb0hrs6D9AIu/zceb4BRcS6SWj7UZZ6uQSb6b1adPua84bvX+1Ka7ikqNnb b3r9ieqm2RWGPYPbRja1dmZeFukDL3CA== X-Google-Smtp-Source: AGHT+IEPX/1mTvU9tBV1ZCJamBy0TSW0W0te8Jq+ZIXXz1GsefoW/9OUoSqy2sCKC6Re0GCQKvyvAqLx X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a81:9a14:0:b0:5ff:b07b:fb83 with SMTP id r20-20020a819a14000000b005ffb07bfb83mr1279875ywg.4.1706551557485; Mon, 29 Jan 2024 10:05:57 -0800 (PST) Date: Mon, 29 Jan 2024 19:05:15 +0100 In-Reply-To: <20240129180502.4069817-21-ardb+git@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240129180502.4069817-21-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=4435; i=ardb@kernel.org; h=from:subject; bh=abTz563MnS4V6kIRqijux/WQS0ziAyixwK6axwUq9ik=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIXX7izvdP/zFc827r854em9X48c7So/Wl4U/sPv6fnqQ1 vE/rD0WHaUsDGIcDLJiiiwCs/++23l6olSt8yxZmDmsTCBDGLg4BWAiB9MYGU5+s3TRaOITdaiq bFmgo9uVJuPi6GWe92HL9PRPGv0TzzH8L8t5IjR7i2tZqmH80rqXhUs+b550MjtqX5HksRuOXfK vuQE= X-Mailer: git-send-email 2.43.0.429.g432eaa2c6b-goog Message-ID: <20240129180502.4069817-33-ardb+git@google.com> Subject: [PATCH v3 12/19] x86/head64: Move early startup code into __pitext From: Ard Biesheuvel To: linux-kernel@vger.kernel.org Cc: Ard Biesheuvel , Kevin Loughlin , Tom Lendacky , Dionna Glaze , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , Andy Lutomirski , Arnd Bergmann , Nathan Chancellor , Nick Desaulniers , Justin Stitt , Kees Cook , Brian Gerst , linux-arch@vger.kernel.org, llvm@lists.linux.dev Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Ard Biesheuvel The boot CPU runs some early startup C code using a 1:1 mapping of memory, which deviates from the normal kernel virtual mapping that is used for calculating statically initialized pointer variables. This makes it necessary to strictly limit which C code will actually be called from that early boot path. Implement this by moving the early startup code into __pitext. Signed-off-by: Ard Biesheuvel --- arch/x86/include/asm/init.h | 2 -- arch/x86/kernel/head64.c | 9 ++++---- arch/x86/kernel/head_64.S | 24 ++++++++++++-------- 3 files changed, 20 insertions(+), 15 deletions(-) diff --git a/arch/x86/include/asm/init.h b/arch/x86/include/asm/init.h index cc9ccf61b6bd..5f1d3c421f68 100644 --- a/arch/x86/include/asm/init.h +++ b/arch/x86/include/asm/init.h @@ -2,8 +2,6 @@ #ifndef _ASM_X86_INIT_H #define _ASM_X86_INIT_H =20 -#define __head __section(".head.text") - struct x86_mapping_info { void *(*alloc_pgt_page)(void *); /* allocate buf for page table */ void *context; /* context for alloc_pgt_page */ diff --git a/arch/x86/kernel/head64.c b/arch/x86/kernel/head64.c index 58c58c66dec9..0ecd36f5326a 100644 --- a/arch/x86/kernel/head64.c +++ b/arch/x86/kernel/head64.c @@ -70,7 +70,8 @@ static struct desc_struct startup_gdt[GDT_ENTRIES] __init= const =3D { asm("movq $" __stringify(sym) ", %0":"=3Dr"(__v)); \ __v; }) =20 -static unsigned long __head sme_postprocess_startup(struct boot_params *bp= , pmdval_t *pmd) +static unsigned long __pitext sme_postprocess_startup(struct boot_params *= bp, + pmdval_t *pmd) { unsigned long vaddr, vaddr_end; int i; @@ -113,7 +114,7 @@ static unsigned long __head sme_postprocess_startup(str= uct boot_params *bp, pmdv return sme_get_me_mask(); } =20 -unsigned long __head __startup_64(struct boot_params *bp) +unsigned long __pitext __startup_64(struct boot_params *bp) { unsigned long physaddr =3D (unsigned long)_text; unsigned long load_delta, *p; @@ -508,7 +509,7 @@ void __init __noreturn x86_64_start_reservations(char *= real_mode_data) */ static gate_desc bringup_idt_table[NUM_EXCEPTION_VECTORS] __page_aligned_d= ata; =20 -static void early_load_idt(void (*handler)(void)) +static void __pitext early_load_idt(void (*handler)(void)) { gate_desc *idt =3D bringup_idt_table; struct desc_ptr bringup_idt_descr; @@ -539,7 +540,7 @@ void early_setup_idt(void) /* * Setup boot CPU state needed before kernel switches to virtual addresses. */ -void __head startup_64_setup_env(void) +void __pitext startup_64_setup_env(void) { struct desc_ptr startup_gdt_descr; =20 diff --git a/arch/x86/kernel/head_64.S b/arch/x86/kernel/head_64.S index b0508e84f756..e671caafd932 100644 --- a/arch/x86/kernel/head_64.S +++ b/arch/x86/kernel/head_64.S @@ -42,6 +42,15 @@ L3_START_KERNEL =3D pud_index(__START_KERNEL_map) __HEAD .code64 SYM_CODE_START_NOALIGN(startup_64) + UNWIND_HINT_END_OF_STACK + jmp primary_startup_64 +SYM_CODE_END(startup_64) + + __PITEXT +#include "verify_cpu.S" +#include "sev_verify_cbit.S" + +SYM_CODE_START_LOCAL(primary_startup_64) UNWIND_HINT_END_OF_STACK /* * At this point the CPU runs in 64bit mode CS.L =3D 1 CS.D =3D 0, @@ -131,10 +140,12 @@ SYM_CODE_START_NOALIGN(startup_64) movq %rax, %cr3 =20 /* Branch to the common startup code at its kernel virtual address */ - movq $common_startup_64, %rax ANNOTATE_RETPOLINE_SAFE - jmp *%rax -SYM_CODE_END(startup_64) + jmp *.Lcommon_startup_64(%rip) +SYM_CODE_END(primary_startup_64) + + __INITRODATA +SYM_DATA_LOCAL(.Lcommon_startup_64, .quad common_startup_64) =20 .text SYM_CODE_START(secondary_startup_64) @@ -410,9 +421,6 @@ SYM_INNER_LABEL(common_startup_64, SYM_L_LOCAL) int3 SYM_CODE_END(secondary_startup_64) =20 -#include "verify_cpu.S" -#include "sev_verify_cbit.S" - #if defined(CONFIG_HOTPLUG_CPU) && defined(CONFIG_AMD_MEM_ENCRYPT) /* * Entry point for soft restart of a CPU. Invoked from xxx_play_dead() for @@ -539,10 +547,8 @@ SYM_CODE_END(early_idt_handler_common) * paravirtualized INTERRUPT_RETURN and pv-ops don't work that early. * * XXX it does, fix this. - * - * This handler will end up in the .init.text section and not be - * available to boot secondary CPUs. */ + __PITEXT SYM_CODE_START_NOALIGN(vc_no_ghcb) UNWIND_HINT_IRET_REGS offset=3D8 ENDBR --=20 2.43.0.429.g432eaa2c6b-goog From nobody Wed Dec 24 01:33:42 2025 Received: from mail-wm1-f73.google.com (mail-wm1-f73.google.com [209.85.128.73]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id DDA946F090 for ; Mon, 29 Jan 2024 18:06:01 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.73 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706551563; cv=none; b=JQYcdQJYzLVE9zu8+EjT6xAFgAerpRtrrbHNsKii6Bs1RO+XBusbvbBfBFI5LIdhIBkb0dfgpFIG0KA8OdhaQYSHIzcONGcC9q+yXt23lm59sfhF10xPQJcHKcUIAjmp4tuP46P8AORl7V9IQt7s8niVxJr8XwdAtlFtW2fXRpQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706551563; c=relaxed/simple; bh=ywQcTtrCDMHJcI+QkXWX2nFyJwoVE8xehI2i+Z3nGsk=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=d8G1Yg2a6LhI1C2hqO0uQxCP4f2uMxB8+X1U2DVD4dy8+uYUH6dSS8+lsBAwbREfVD3UrQjwcU3iIsMjVXMFBF51Ln9Bhfd1zKr9SJSV2gVd+u39mzs+34ge6qoXHtHWqhXEm36z4iGW69xa9XZjiEdbsfyI0XirdKQK3AbJ4Oc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=PePKihMI; arc=none smtp.client-ip=209.85.128.73 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="PePKihMI" Received: by mail-wm1-f73.google.com with SMTP id 5b1f17b1804b1-40e478693bcso20448755e9.1 for ; Mon, 29 Jan 2024 10:06:01 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1706551560; x=1707156360; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=fLLl1bcVbRYtnIzLBeH2xJGti8beoepv/z/uSZlJPHo=; b=PePKihMIIMoDRiafr8/B5H0JSooV6zBWN12mgzsMtuezB4BxjNtZqUuUwJ0AX7Jo1X LTg84Ia83obzKLG+p06g835IqPxjaUAKnuOAyRCW+dSe2tDftB5D4lpvIi9oUgVDMAfA upbEeHjGTvTxNgCeGTYe1kYJ1f0pB4gdN9AgmDBul4vNOGGv221vJmK6BeQkqs2wl155 pGy8ltGR0DTI01dlvkwIcEPlAwAC7pYA5ILDkBklmYCpkh1EG7xhbzDCkmadwkBWDg0n fzyEBI18fOPI89duJF+eQ+wnTJVV4XDHXp6/xAZm3Q2x13N2V1iYeMp86YnevNLWM/4a TfIw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1706551560; x=1707156360; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=fLLl1bcVbRYtnIzLBeH2xJGti8beoepv/z/uSZlJPHo=; b=NiUJRFcLdkwdILrmVRo8COPXeb+8HG+Kgz3G1BYYdgIkROXW8XbN40b7NDa3L3XZme YQU2iChlh0nD5mTVw3OaARVmig7mclxOiwg5XZiNt5p6fyNs27VwMJ8Uzg5/ONm+Jn65 J4pTdZxstYMrlSwrZkqfSpSlMqd/IbeiRQz2uNIRsB40rh903YTS+gNcvE9750xVhsGz pDxdW7jZZBkepmh0xEDloKygqq2YEcigIDn3QRbARbrtAhlnpqBOTvHHBKOgvPIxSk43 jYJR3pT3tvIPqGLn/TWaj2q8O5ysg1ywMajAW0B7ACBEayrT+1/esYkkj/Oiyk6G80vX 8H6g== X-Gm-Message-State: AOJu0YwAl1WEuxV0GsL1T5nIolDWPLek/UDfOfp2GSDN+fVAsmom6rdS oORUtxwnxmpzD9Iw0GAo3ek8+1aPJ6LTYWw2cC75/lXjbFVdUxs3zvSjDwlDGB7y4jNyo3qolse Y7PPFWJRK9UzFzhD1lXVmlMyQtzFmjEM7jec1ix91eWSnUC27StfQUHzqYWMPL3j+4+MC60Cduo M4Gy1iGKqkNv9PFcFwFcdKIo1lSQCv2g== X-Google-Smtp-Source: AGHT+IFRYaQgWgn7lS/fBsOiZY67lNa50V8yvvYuuoCz5vPDJOnfUaUEOPLRn1IO8EMDt34AydE5rmRl X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a05:6000:1e06:b0:33a:e517:57f1 with SMTP id bj6-20020a0560001e0600b0033ae51757f1mr89123wrb.0.1706551560216; Mon, 29 Jan 2024 10:06:00 -0800 (PST) Date: Mon, 29 Jan 2024 19:05:16 +0100 In-Reply-To: <20240129180502.4069817-21-ardb+git@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240129180502.4069817-21-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=979; i=ardb@kernel.org; h=from:subject; bh=DDaOUA3J+CFMHhm3e2XXAeDqtyhjkdr8A2J8hJRp8QM=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIXX7i7vMP9b7C0XfsczNYLSSucj1x3p6wJuiY837VhzfN +vbpewPHaUsDGIcDLJiiiwCs/++23l6olSt8yxZmDmsTCBDGLg4BWAiakcYGVby7QyTU6idwMOc J38qUmJ1p+F91buzM3Z8Xzphas7LJ+sZGZYqnkl8ycjtbnKv6+ovZ1E/e8uaWONjpw6xe+z87Cp rxgYA X-Mailer: git-send-email 2.43.0.429.g432eaa2c6b-goog Message-ID: <20240129180502.4069817-34-ardb+git@google.com> Subject: [PATCH v3 13/19] modpost: Warn about calls from __pitext into other text sections From: Ard Biesheuvel To: linux-kernel@vger.kernel.org Cc: Ard Biesheuvel , Kevin Loughlin , Tom Lendacky , Dionna Glaze , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , Andy Lutomirski , Arnd Bergmann , Nathan Chancellor , Nick Desaulniers , Justin Stitt , Kees Cook , Brian Gerst , linux-arch@vger.kernel.org, llvm@lists.linux.dev Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Ard Biesheuvel Ensure that code that is marked as being able to safely run from a 1:1 mapping does not call into other code which might lack that property. Signed-off-by: Ard Biesheuvel --- scripts/mod/modpost.c | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/scripts/mod/modpost.c b/scripts/mod/modpost.c index 962d00df47ab..33b56d6b4e7b 100644 --- a/scripts/mod/modpost.c +++ b/scripts/mod/modpost.c @@ -825,6 +825,7 @@ enum mismatch { ANY_INIT_TO_ANY_EXIT, ANY_EXIT_TO_ANY_INIT, EXTABLE_TO_NON_TEXT, + PI_TEXT_TO_NON_PI_TEXT, }; =20 /** @@ -887,6 +888,11 @@ static const struct sectioncheck sectioncheck[] =3D { .bad_tosec =3D { ".altinstr_replacement", NULL }, .good_tosec =3D {ALL_TEXT_SECTIONS , NULL}, .mismatch =3D EXTABLE_TO_NON_TEXT, +}, +{ + .fromsec =3D { ALL_PI_TEXT_SECTIONS, NULL }, + .bad_tosec =3D { ALL_NON_PI_TEXT_SECTIONS, NULL }, + .mismatch =3D PI_TEXT_TO_NON_PI_TEXT, } }; =20 --=20 2.43.0.429.g432eaa2c6b-goog From nobody Wed Dec 24 01:33:42 2025 Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6BF1F3F9DA for ; Mon, 29 Jan 2024 18:06:03 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.219.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706551564; cv=none; b=QmHsHayNX9rG0duGfoCLl3pf1zZwqsMiK0Ow9un75TY9x/lUd0gU2CIKpRe4qDWUnpQ6pGn0sEpjNAjqFz75KMiWZkGC7kZzUaL9RriopOREq3GFONo5MA7UHRMbv8bRlRjW+abWhmyozGAUJsCHi9l1n8NPCoBDN3UAfE/Ej3g= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706551564; c=relaxed/simple; bh=2tkXbI4kkP7Bg5PktzWdMGd8q8tWspLGt7tFV60fDKM=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=gxnYA98Qq9BGlb6TJ64/Q7ZJn42elTGMQCwo9MRl/lEfPyIaNPDdV33mPLfAym8jX9MM/xDa92RsWGf8Q14cVf3cfD2Mt/qimYPD7h7SJZcfEgKS7wvx0ZV8nQ5oxLREMCKr1G5pzDSU1yQje0P47fKcqUuGPdRO0JB4MPA/HSI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=HHxwKq+n; arc=none smtp.client-ip=209.85.219.201 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="HHxwKq+n" Received: by mail-yb1-f201.google.com with SMTP id 3f1490d57ef6-dc2629d180fso6441691276.0 for ; Mon, 29 Jan 2024 10:06:03 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1706551562; x=1707156362; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=6DdM1fA2/BUC7ScxyUb41JZesEtpKP4YlIE0jOqBF4A=; b=HHxwKq+nXkk2HJ3qoSkywtt5wIE0R0J/2KgfyyeyC9rWcHw7NdABIQGtgZpoLe3Uqj Uik/7h1LPurFgs4BIxuoXsUtCMgybzfcQI8QWBpt0FkTGvWJKebEUh6Os0xiMhENdQSq sLKfBDBJj34bxS7dj8ld1eanbODydLv5qIHml66Fl3PXVhZhc7DEiWfD55G9UgkbC5+Z ZmKXqE1kQ+9Z9BxW9q/NrYeDkogHX8vwGju/ZBDDZzUbVRmTZuAuv3HsbQuL0aWDr9u+ HEwdVPtLlrXMZmvt25bVK7JSqIuQlsNzNhXbUpS3UQlr3rBx3SkOKA5MjgOgdcntMe+K Hb/w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1706551562; x=1707156362; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=6DdM1fA2/BUC7ScxyUb41JZesEtpKP4YlIE0jOqBF4A=; b=dV4dmbJ5Ij0x+Sqa4BOeXaLpjF+GN68mvZyHf1WQheQ+AT2yANKT+XK694qpvQcgI9 7dixpm9Hg8Qk/jcHYSbJgq6O2KmqOOpmjTwT+zrAq40TsJ0Adto2xMX/gVgCDURgFSQ4 KMKcRYyd17KuTEzlnj+DAnNHdiyafd6DeB8B+ig2YpSSqh6/cgfXSU9Cx6BJhnhNa7cl VQKbxfwIepvmGMH8x9GtPjLeubqrtRJvezxlo9B1tAhyV17mcIaQDZVEODGrqfsub6QM lE6lDHaaXq3EvI0mdBoqAbYYpWldW5Hw+Hbw87C4r1QufLyLpVsXhGMZDTb3aE1mtuUl /QjQ== X-Gm-Message-State: AOJu0YzEdWJhVgYxbEtwJy4M+q9aOmEfgIm5YUPjZebGXKH2Ghbzjm4X KPZchSm3FjyUm2WZwetF/ffPTcy4KFOM35qXRODgS/uRD1bnE3bvtU8q3qpYYgwsjUkvlAdGW9T Aac51ibUNkZ2uMzaqN7q4qHNmvAaolbyitc/aRWG13jANWnXbZe6Zblw9xKGP6S+9PJPjVXvZ7s LFsoh+G3IrtG8TJAN1KOHTj7jTmLJ7+g== X-Google-Smtp-Source: AGHT+IEtY0wHG6Bv+Cfuwk98fDoP1mB4rkr70PowOvdhb/HTma9T9dk4Ph+McoAi3VylYPwwX4+//SgC X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a05:6902:e0d:b0:dc3:721f:7a4e with SMTP id df13-20020a0569020e0d00b00dc3721f7a4emr2254113ybb.12.1706551562591; Mon, 29 Jan 2024 10:06:02 -0800 (PST) Date: Mon, 29 Jan 2024 19:05:17 +0100 In-Reply-To: <20240129180502.4069817-21-ardb+git@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240129180502.4069817-21-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=1516; i=ardb@kernel.org; h=from:subject; bh=XIt7BesiIe5Ouf8eLT5Jk6gT3ML8ZNdNuPHWU+248+Y=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIXX7i3sXJ3YtNZs6V0Vn46sNwfPVvywvyvumoxTpaWPlJ Nz4QT6mo5SFQYyDQVZMkUVg9t93O09PlKp1niULM4eVCWQIAxenAExk7QxGhtn/Pbk3KEkWcd6X n8erVbj74eH4N1o8/57WlE5hkr1zU4XhN8vqb2tihGJOxm3c8mZjeq3tTemlor/sxaf2e0qmZ2w +wAoA X-Mailer: git-send-email 2.43.0.429.g432eaa2c6b-goog Message-ID: <20240129180502.4069817-35-ardb+git@google.com> Subject: [PATCH v3 14/19] x86/coco: Make cc_set_mask() static inline From: Ard Biesheuvel To: linux-kernel@vger.kernel.org Cc: Ard Biesheuvel , Kevin Loughlin , Tom Lendacky , Dionna Glaze , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , Andy Lutomirski , Arnd Bergmann , Nathan Chancellor , Nick Desaulniers , Justin Stitt , Kees Cook , Brian Gerst , linux-arch@vger.kernel.org, llvm@lists.linux.dev Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Ard Biesheuvel Setting the cc_mask global variable may be done early in the boot while running fromm a 1:1 translation. This code is built with -fPIC in order to support this. Make cc_set_mask() static inline so it can execute safely in this context as well. Signed-off-by: Ard Biesheuvel --- arch/x86/coco/core.c | 7 +------ arch/x86/include/asm/coco.h | 8 +++++++- 2 files changed, 8 insertions(+), 7 deletions(-) diff --git a/arch/x86/coco/core.c b/arch/x86/coco/core.c index eeec9986570e..d07be9d05cd0 100644 --- a/arch/x86/coco/core.c +++ b/arch/x86/coco/core.c @@ -14,7 +14,7 @@ #include =20 enum cc_vendor cc_vendor __ro_after_init =3D CC_VENDOR_NONE; -static u64 cc_mask __ro_after_init; +u64 cc_mask __ro_after_init; =20 static bool noinstr intel_cc_platform_has(enum cc_attr attr) { @@ -148,8 +148,3 @@ u64 cc_mkdec(u64 val) } } EXPORT_SYMBOL_GPL(cc_mkdec); - -__init void cc_set_mask(u64 mask) -{ - cc_mask =3D mask; -} diff --git a/arch/x86/include/asm/coco.h b/arch/x86/include/asm/coco.h index 6ae2d16a7613..ecc29d6136ad 100644 --- a/arch/x86/include/asm/coco.h +++ b/arch/x86/include/asm/coco.h @@ -13,7 +13,13 @@ enum cc_vendor { extern enum cc_vendor cc_vendor; =20 #ifdef CONFIG_ARCH_HAS_CC_PLATFORM -void cc_set_mask(u64 mask); +static inline void cc_set_mask(u64 mask) +{ + extern u64 cc_mask; + + cc_mask =3D mask; +} + u64 cc_mkenc(u64 val); u64 cc_mkdec(u64 val); #else --=20 2.43.0.429.g432eaa2c6b-goog From nobody Wed Dec 24 01:33:42 2025 Received: from mail-wm1-f73.google.com (mail-wm1-f73.google.com [209.85.128.73]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9695D15B0ED for ; Mon, 29 Jan 2024 18:06:06 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.73 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706551568; cv=none; b=TBBlKs7VuKjvFs7bfPSUhxA1Wj5EILwYnurggFzMNyx9etDCPahY6CfmA47vaBS4342IpJ0RBKsBXamedVMCdQn3tH6ANBVD1dU5j4ZigZGM0vIiBJ0DGlEFbUcvG88qvNs44VuoPH06WmC92XYnmjSjcLMp5btJWC+YcsUJHxg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706551568; c=relaxed/simple; bh=NZI+VSlxzyPFRmF15Jcr5EHokjSqQLfxpmgd8iObKfA=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=HGWLuJlvuc24O3z58iawe8DaobW1TOwJH1tL/9Fl3wD/EywiKWsdWsMTJzAUpCdtkPu7XpQ0o1zeHL9kFzS/L/p27jzcZYxVWiZpB9RfZcAc9p6V4EQRgslF7OJRVPoZOvLFCF15kveIXqpWlobO+qPaLCiVEaozUWtXjzvd+A0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=vuO9Qazu; arc=none smtp.client-ip=209.85.128.73 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="vuO9Qazu" Received: by mail-wm1-f73.google.com with SMTP id 5b1f17b1804b1-40efbe60d32so2967645e9.3 for ; Mon, 29 Jan 2024 10:06:06 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1706551565; x=1707156365; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=KJ+eFSML+GjQ9gUM/4lB5AYEA1a3EqVM82ZLULKpZgo=; b=vuO9QazuU/q2p74fAXMgjT9VjxXgBIFXO5IYsm27ohPMgFpYSD7LZdOVZZVHWd2jhb JDX1r6I/6rFmXgfkoFirVyaprv7lh4Q7aPnXez4JQyf74rIYKuS0DMUnf23Lxw/RWoHz gKnNQ+iIJBJXD4KFihFBpcJxRP5Us46Hzi3LJ+Tn7nitjfkf7GlubZTYbhIYbB11WHs6 +3KlQmAWbj+SAiejztOi+SUnjzKGu4bQ+JHz0l3mRsiuG04ynwPhebb4FH9A+6GJmOWA zJi3Sx0HQ3zCSGO9qQGBiaoEh1DYIqHT2O0Petb9lKpc/wvhpq77pE6w/zlrzO2cbnIG oWQw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1706551565; x=1707156365; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=KJ+eFSML+GjQ9gUM/4lB5AYEA1a3EqVM82ZLULKpZgo=; b=v3AjITTPGj2Ra/EsxblYBueD4PVgGq2RpnmJ2QRURyxqcgbUYxtKVuMKbaFk2uRQLM ZztyoC2K+hZbcEZdPPcii/tc3AbzSceQeSm4WO/KyUNBMP4digEnSs/DQ9fISiC6zBGT fIEuo/i4Hyuhv3MOXCoyT9+exp7Q5k5Sbci+WNCtuHhJgCe0ymybWeFnRacCHGb3ld6N KIh5YYRF9xyfIp62z/JjqWAeJ5uZdP187EvLBHTZoBXX3YN0ubd+ZO7tFf0cp6dmFDlu lgkBpR6BGYW5sBO1i1hhKytt7iN9hKU/0pF0JiSYBAxxa8rwyutBfMYIpH9jLQtPUBtD 8foQ== X-Gm-Message-State: AOJu0Yw53C0pEqsWEpWq2mL2Pp98XIrmHP8uGISju+y9ocgjDmXfL5Vc nIbHFZJn90HKagYtIzL0AnmiP5WtCMwrkctV6lAcJmsYP71wwnY0veIiR18JSzaqZ9pNHLiJjF/ /IT4IkvJwOSkZyequioAB5RCmpMbXam13jBA/1IE83GVakTfSiyuC5ubTwUGhIXBDnJNsQCC6Lp Z9qVt5QnU/5rOC4wxz8koNvJsT+Fqw1A== X-Google-Smtp-Source: AGHT+IHQBSeNys/uis3YJHXSl03WHh2SW+9GfrXVNKmkG6zseAajGFPK4L092X4fyF6kwH9jlP8AzgXo X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a05:600c:3d98:b0:40e:f742:c838 with SMTP id bi24-20020a05600c3d9800b0040ef742c838mr12658wmb.7.1706551564801; Mon, 29 Jan 2024 10:06:04 -0800 (PST) Date: Mon, 29 Jan 2024 19:05:18 +0100 In-Reply-To: <20240129180502.4069817-21-ardb+git@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240129180502.4069817-21-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=20645; i=ardb@kernel.org; h=from:subject; bh=55OGssAUsDuXeb2b/A15axTFWbEGM1IToSgXf7PCzwk=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIXX7i/vPU3qE5E97KE+L8V2yVGCB7axHB++8zXx0y1dfm 6mur1umo5SFQYyDQVZMkUVg9t93O09PlKp1niULM4eVCWQIAxenAEykdysjw5GXq9r/eMs+v/P9 5VK5ld+sva4Zfg3Qa7pcKf3M+/UFucMMf4XaYhZw3//8NqHo0eE+/+9xHYd3RemZy2kZ93U0ioc s5wUA X-Mailer: git-send-email 2.43.0.429.g432eaa2c6b-goog Message-ID: <20240129180502.4069817-36-ardb+git@google.com> Subject: [PATCH v3 15/19] x86/sev: Make all code reachable from 1:1 mapping __pitext From: Ard Biesheuvel To: linux-kernel@vger.kernel.org Cc: Ard Biesheuvel , Kevin Loughlin , Tom Lendacky , Dionna Glaze , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , Andy Lutomirski , Arnd Bergmann , Nathan Chancellor , Nick Desaulniers , Justin Stitt , Kees Cook , Brian Gerst , linux-arch@vger.kernel.org, llvm@lists.linux.dev Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Ard Biesheuvel We cannot safely call any code when still executing from the 1:1 mapping at early boot. The SEV init code in particular does a fair amount of work this early, and calls into ordinary APIs, which is not safe, as these may be instrumented by the sanitizers or by things link CONFIG_DEBUG_VM or CONFIG_DEBUG_VIRTUAL. So annotate all SEV code used early as __pitext and along with it, some of the shared code that it relies on. Also override some definition of the __pa/__va translation macros to avoid pulling in debug versions. Signed-off-by: Ard Biesheuvel --- arch/x86/boot/compressed/sev.c | 6 +++ arch/x86/include/asm/mem_encrypt.h | 8 ++-- arch/x86/include/asm/pgtable_64.h | 12 +++++- arch/x86/include/asm/sev.h | 6 +-- arch/x86/kernel/head64.c | 20 ++++++---- arch/x86/kernel/sev-shared.c | 40 +++++++++++--------- arch/x86/kernel/sev.c | 14 +++---- arch/x86/lib/memcpy_64.S | 3 +- arch/x86/lib/memset_64.S | 3 +- arch/x86/mm/mem_encrypt_boot.S | 3 +- arch/x86/mm/mem_encrypt_identity.c | 35 ++++++++--------- 11 files changed, 90 insertions(+), 60 deletions(-) diff --git a/arch/x86/boot/compressed/sev.c b/arch/x86/boot/compressed/sev.c index 073291832f44..ada6cd8d600b 100644 --- a/arch/x86/boot/compressed/sev.c +++ b/arch/x86/boot/compressed/sev.c @@ -25,6 +25,9 @@ #include "error.h" #include "../msr.h" =20 +#undef __pa_nodebug +#define __pa_nodebug __pa + static struct ghcb boot_ghcb_page __aligned(PAGE_SIZE); struct ghcb *boot_ghcb; =20 @@ -116,6 +119,9 @@ static bool fault_in_kernel_space(unsigned long address) #undef __init #define __init =20 +#undef __pitext +#define __pitext + #define __BOOT_COMPRESSED =20 /* Basic instruction decoding support needed */ diff --git a/arch/x86/include/asm/mem_encrypt.h b/arch/x86/include/asm/mem_= encrypt.h index 359ada486fa9..48469e22a75e 100644 --- a/arch/x86/include/asm/mem_encrypt.h +++ b/arch/x86/include/asm/mem_encrypt.h @@ -46,8 +46,8 @@ void __init sme_unmap_bootdata(char *real_mode_data); =20 void __init sme_early_init(void); =20 -void __init sme_encrypt_kernel(struct boot_params *bp); -void __init sme_enable(struct boot_params *bp); +void sme_encrypt_kernel(struct boot_params *bp); +void sme_enable(struct boot_params *bp); =20 int __init early_set_memory_decrypted(unsigned long vaddr, unsigned long s= ize); int __init early_set_memory_encrypted(unsigned long vaddr, unsigned long s= ize); @@ -75,8 +75,8 @@ static inline void __init sme_unmap_bootdata(char *real_m= ode_data) { } =20 static inline void __init sme_early_init(void) { } =20 -static inline void __init sme_encrypt_kernel(struct boot_params *bp) { } -static inline void __init sme_enable(struct boot_params *bp) { } +static inline void sme_encrypt_kernel(struct boot_params *bp) { } +static inline void sme_enable(struct boot_params *bp) { } =20 static inline void sev_es_init_vc_handling(void) { } =20 diff --git a/arch/x86/include/asm/pgtable_64.h b/arch/x86/include/asm/pgtab= le_64.h index 24af25b1551a..3a6d90f47f32 100644 --- a/arch/x86/include/asm/pgtable_64.h +++ b/arch/x86/include/asm/pgtable_64.h @@ -139,12 +139,17 @@ static inline pud_t native_pudp_get_and_clear(pud_t *= xp) #endif } =20 +static inline void set_p4d_kernel(p4d_t *p4dp, p4d_t p4d) +{ + WRITE_ONCE(*p4dp, p4d); +} + static inline void native_set_p4d(p4d_t *p4dp, p4d_t p4d) { pgd_t pgd; =20 if (pgtable_l5_enabled() || !IS_ENABLED(CONFIG_PAGE_TABLE_ISOLATION)) { - WRITE_ONCE(*p4dp, p4d); + set_p4d_kernel(p4dp, p4d); return; } =20 @@ -158,6 +163,11 @@ static inline void native_p4d_clear(p4d_t *p4d) native_set_p4d(p4d, native_make_p4d(0)); } =20 +static inline void set_pgd_kernel(pgd_t *pgdp, pgd_t pgd) +{ + WRITE_ONCE(*pgdp, pgd); +} + static inline void native_set_pgd(pgd_t *pgdp, pgd_t pgd) { WRITE_ONCE(*pgdp, pti_set_user_pgtbl(pgdp, pgd)); diff --git a/arch/x86/include/asm/sev.h b/arch/x86/include/asm/sev.h index 5b4a1ce3d368..e3b55bd15ce1 100644 --- a/arch/x86/include/asm/sev.h +++ b/arch/x86/include/asm/sev.h @@ -201,14 +201,14 @@ struct snp_guest_request_ioctl; void setup_ghcb(void); void __init early_snp_set_memory_private(unsigned long vaddr, unsigned lon= g paddr, unsigned long npages); -void __init early_snp_set_memory_shared(unsigned long vaddr, unsigned long= paddr, - unsigned long npages); +void early_snp_set_memory_shared(unsigned long vaddr, unsigned long paddr, + unsigned long npages); void __init snp_prep_memory(unsigned long paddr, unsigned int sz, enum psc= _op op); void snp_set_memory_shared(unsigned long vaddr, unsigned long npages); void snp_set_memory_private(unsigned long vaddr, unsigned long npages); void snp_set_wakeup_secondary_cpu(void); bool snp_init(struct boot_params *bp); -void __init __noreturn snp_abort(void); +void __noreturn snp_abort(void); int snp_issue_guest_request(u64 exit_code, struct snp_req_data *input, str= uct snp_guest_request_ioctl *rio); void snp_accept_memory(phys_addr_t start, phys_addr_t end); u64 snp_get_unsupported_features(u64 status); diff --git a/arch/x86/kernel/head64.c b/arch/x86/kernel/head64.c index 0ecd36f5326a..b014f81e0eac 100644 --- a/arch/x86/kernel/head64.c +++ b/arch/x86/kernel/head64.c @@ -91,16 +91,20 @@ static unsigned long __pitext sme_postprocess_startup(s= truct boot_params *bp, =20 for (; vaddr < vaddr_end; vaddr +=3D PMD_SIZE) { /* - * On SNP, transition the page to shared in the RMP table so that - * it is consistent with the page table attribute change. + * On SNP, transition the page to shared in the RMP + * table so that it is consistent with the page table + * attribute change. * - * __start_bss_decrypted has a virtual address in the high range - * mapping (kernel .text). PVALIDATE, by way of - * early_snp_set_memory_shared(), requires a valid virtual - * address but the kernel is currently running off of the identity - * mapping so use __pa() to get a *currently* valid virtual address. + * __start_bss_decrypted has a virtual address in the + * high range mapping (kernel .text). PVALIDATE, by way + * of early_snp_set_memory_shared(), requires a valid + * virtual address but the kernel is currently running + * off of the identity mapping so use __pa() to get a + * *currently* valid virtual address. */ - early_snp_set_memory_shared(__pa(vaddr), __pa(vaddr), PTRS_PER_PMD); + early_snp_set_memory_shared(__pa_nodebug(vaddr), + __pa_nodebug(vaddr), + PTRS_PER_PMD); =20 i =3D pmd_index(vaddr); pmd[i] -=3D sme_get_me_mask(); diff --git a/arch/x86/kernel/sev-shared.c b/arch/x86/kernel/sev-shared.c index 5db24d0fc557..481dbd009ce9 100644 --- a/arch/x86/kernel/sev-shared.c +++ b/arch/x86/kernel/sev-shared.c @@ -93,7 +93,8 @@ static bool __init sev_es_check_cpu_features(void) return true; } =20 -static void __noreturn sev_es_terminate(unsigned int set, unsigned int rea= son) +static __always_inline void __noreturn sev_es_terminate(unsigned int set, + unsigned int reason) { u64 val =3D GHCB_MSR_TERM_REQ; =20 @@ -226,10 +227,9 @@ static enum es_result verify_exception_info(struct ghc= b *ghcb, struct es_em_ctxt return ES_VMM_ERROR; } =20 -static enum es_result sev_es_ghcb_hv_call(struct ghcb *ghcb, - struct es_em_ctxt *ctxt, - u64 exit_code, u64 exit_info_1, - u64 exit_info_2) +static enum es_result __pitext +sev_es_ghcb_hv_call(struct ghcb *ghcb, struct es_em_ctxt *ctxt, + u64 exit_code, u64 exit_info_1, u64 exit_info_2) { /* Fill in protocol and format specifiers */ ghcb->protocol_version =3D ghcb_version; @@ -239,13 +239,13 @@ static enum es_result sev_es_ghcb_hv_call(struct ghcb= *ghcb, ghcb_set_sw_exit_info_1(ghcb, exit_info_1); ghcb_set_sw_exit_info_2(ghcb, exit_info_2); =20 - sev_es_wr_ghcb_msr(__pa(ghcb)); + sev_es_wr_ghcb_msr(__pa_nodebug(ghcb)); VMGEXIT(); =20 return verify_exception_info(ghcb, ctxt); } =20 -static int __sev_cpuid_hv(u32 fn, int reg_idx, u32 *reg) +static int __pitext __sev_cpuid_hv(u32 fn, int reg_idx, u32 *reg) { u64 val; =20 @@ -260,7 +260,7 @@ static int __sev_cpuid_hv(u32 fn, int reg_idx, u32 *reg) return 0; } =20 -static int __sev_cpuid_hv_msr(struct cpuid_leaf *leaf) +static int __pitext __sev_cpuid_hv_msr(struct cpuid_leaf *leaf) { int ret; =20 @@ -283,7 +283,9 @@ static int __sev_cpuid_hv_msr(struct cpuid_leaf *leaf) return ret; } =20 -static int __sev_cpuid_hv_ghcb(struct ghcb *ghcb, struct es_em_ctxt *ctxt,= struct cpuid_leaf *leaf) +static int __pitext __sev_cpuid_hv_ghcb(struct ghcb *ghcb, + struct es_em_ctxt *ctxt, + struct cpuid_leaf *leaf) { u32 cr4 =3D native_read_cr4(); int ret; @@ -316,7 +318,8 @@ static int __sev_cpuid_hv_ghcb(struct ghcb *ghcb, struc= t es_em_ctxt *ctxt, struc return ES_OK; } =20 -static int sev_cpuid_hv(struct ghcb *ghcb, struct es_em_ctxt *ctxt, struct= cpuid_leaf *leaf) +static int __pitext sev_cpuid_hv(struct ghcb *ghcb, struct es_em_ctxt *ctx= t, + struct cpuid_leaf *leaf) { return ghcb ? __sev_cpuid_hv_ghcb(ghcb, ctxt, leaf) : __sev_cpuid_hv_msr(leaf); @@ -395,7 +398,7 @@ static u32 snp_cpuid_calc_xsave_size(u64 xfeatures_en, = bool compacted) return xsave_size; } =20 -static bool +static bool __pitext snp_cpuid_get_validated_func(struct cpuid_leaf *leaf) { const struct snp_cpuid_table *cpuid_table =3D snp_cpuid_get_table(); @@ -431,14 +434,16 @@ snp_cpuid_get_validated_func(struct cpuid_leaf *leaf) return false; } =20 -static void snp_cpuid_hv(struct ghcb *ghcb, struct es_em_ctxt *ctxt, struc= t cpuid_leaf *leaf) +static void __pitext snp_cpuid_hv(struct ghcb *ghcb, struct es_em_ctxt *ct= xt, + struct cpuid_leaf *leaf) { if (sev_cpuid_hv(ghcb, ctxt, leaf)) sev_es_terminate(SEV_TERM_SET_LINUX, GHCB_TERM_CPUID_HV); } =20 -static int snp_cpuid_postprocess(struct ghcb *ghcb, struct es_em_ctxt *ctx= t, - struct cpuid_leaf *leaf) +static int __pitext snp_cpuid_postprocess(struct ghcb *ghcb, + struct es_em_ctxt *ctxt, + struct cpuid_leaf *leaf) { struct cpuid_leaf leaf_hv =3D *leaf; =20 @@ -532,7 +537,8 @@ static int snp_cpuid_postprocess(struct ghcb *ghcb, str= uct es_em_ctxt *ctxt, * Returns -EOPNOTSUPP if feature not enabled. Any other non-zero return v= alue * should be treated as fatal by caller. */ -static int snp_cpuid(struct ghcb *ghcb, struct es_em_ctxt *ctxt, struct cp= uid_leaf *leaf) +static int __pitext snp_cpuid(struct ghcb *ghcb, struct es_em_ctxt *ctxt, + struct cpuid_leaf *leaf) { const struct snp_cpuid_table *cpuid_table =3D snp_cpuid_get_table(); =20 @@ -574,7 +580,7 @@ static int snp_cpuid(struct ghcb *ghcb, struct es_em_ct= xt *ctxt, struct cpuid_le * page yet, so it only supports the MSR based communication with the * hypervisor and only the CPUID exit-code. */ -void __init do_vc_no_ghcb(struct pt_regs *regs, unsigned long exit_code) +void __pitext do_vc_no_ghcb(struct pt_regs *regs, unsigned long exit_code) { unsigned int subfn =3D lower_bits(regs->cx, 32); unsigned int fn =3D lower_bits(regs->ax, 32); @@ -1052,7 +1058,7 @@ static struct cc_blob_sev_info *find_cc_blob_setup_da= ta(struct boot_params *bp) * mapping needs to be updated in sync with all the changes to virtual mem= ory * layout and related mapping facilities throughout the boot process. */ -static void __init setup_cpuid_table(const struct cc_blob_sev_info *cc_inf= o) +static void __pitext setup_cpuid_table(const struct cc_blob_sev_info *cc_i= nfo) { const struct snp_cpuid_table *cpuid_table_fw, *cpuid_table; int i; diff --git a/arch/x86/kernel/sev.c b/arch/x86/kernel/sev.c index 1ec753331524..62981b463b76 100644 --- a/arch/x86/kernel/sev.c +++ b/arch/x86/kernel/sev.c @@ -682,8 +682,8 @@ static u64 __init get_jump_table_addr(void) return ret; } =20 -static void early_set_pages_state(unsigned long vaddr, unsigned long paddr, - unsigned long npages, enum psc_op op) +static void __pitext early_set_pages_state(unsigned long vaddr, unsigned l= ong paddr, + unsigned long npages, enum psc_op op) { unsigned long paddr_end; u64 val; @@ -758,8 +758,8 @@ void __init early_snp_set_memory_private(unsigned long = vaddr, unsigned long padd early_set_pages_state(vaddr, paddr, npages, SNP_PAGE_STATE_PRIVATE); } =20 -void __init early_snp_set_memory_shared(unsigned long vaddr, unsigned long= paddr, - unsigned long npages) +void __pitext early_snp_set_memory_shared(unsigned long vaddr, unsigned lo= ng paddr, + unsigned long npages) { /* * This can be invoked in early boot while running identity mapped, so @@ -2062,7 +2062,7 @@ bool __init handle_vc_boot_ghcb(struct pt_regs *regs) * * Scan for the blob in that order. */ -static __init struct cc_blob_sev_info *find_cc_blob(struct boot_params *bp) +static __pitext struct cc_blob_sev_info *find_cc_blob(struct boot_params *= bp) { struct cc_blob_sev_info *cc_info; =20 @@ -2088,7 +2088,7 @@ static __init struct cc_blob_sev_info *find_cc_blob(s= truct boot_params *bp) return cc_info; } =20 -bool __init snp_init(struct boot_params *bp) +bool __pitext snp_init(struct boot_params *bp) { struct cc_blob_sev_info *cc_info; =20 @@ -2110,7 +2110,7 @@ bool __init snp_init(struct boot_params *bp) return true; } =20 -void __init __noreturn snp_abort(void) +void __pitext __noreturn snp_abort(void) { sev_es_terminate(SEV_TERM_SET_GEN, GHCB_SNP_UNSUPPORTED); } diff --git a/arch/x86/lib/memcpy_64.S b/arch/x86/lib/memcpy_64.S index 0ae2e1712e2e..f56cb062d874 100644 --- a/arch/x86/lib/memcpy_64.S +++ b/arch/x86/lib/memcpy_64.S @@ -2,13 +2,14 @@ /* Copyright 2002 Andi Kleen */ =20 #include +#include #include #include #include #include #include =20 -.section .noinstr.text, "ax" + __PITEXT =20 /* * memcpy - Copy a memory block. diff --git a/arch/x86/lib/memset_64.S b/arch/x86/lib/memset_64.S index 0199d56cb479..455424dcadc0 100644 --- a/arch/x86/lib/memset_64.S +++ b/arch/x86/lib/memset_64.S @@ -2,11 +2,12 @@ /* Copyright 2002 Andi Kleen, SuSE Labs */ =20 #include +#include #include #include #include =20 -.section .noinstr.text, "ax" + __PITEXT =20 /* * ISO C memset - set a memory block to a byte value. This function uses f= ast diff --git a/arch/x86/mm/mem_encrypt_boot.S b/arch/x86/mm/mem_encrypt_boot.S index e25288ee33c2..f951f4f86e5c 100644 --- a/arch/x86/mm/mem_encrypt_boot.S +++ b/arch/x86/mm/mem_encrypt_boot.S @@ -7,6 +7,7 @@ * Author: Tom Lendacky */ =20 +#include #include #include #include @@ -14,7 +15,7 @@ #include #include =20 - .text + __PITEXT .code64 SYM_FUNC_START(sme_encrypt_execute) =20 diff --git a/arch/x86/mm/mem_encrypt_identity.c b/arch/x86/mm/mem_encrypt_i= dentity.c index 2e195866a7fe..bc39e04de980 100644 --- a/arch/x86/mm/mem_encrypt_identity.c +++ b/arch/x86/mm/mem_encrypt_identity.c @@ -85,7 +85,8 @@ struct sme_populate_pgd_data { */ static char sme_workarea[2 * PMD_SIZE] __section(".init.scratch"); =20 -static void __init sme_clear_pgd(struct sme_populate_pgd_data *ppd) + +static void __pitext sme_clear_pgd(struct sme_populate_pgd_data *ppd) { unsigned long pgd_start, pgd_end, pgd_size; pgd_t *pgd_p; @@ -100,7 +101,7 @@ static void __init sme_clear_pgd(struct sme_populate_pg= d_data *ppd) memset(pgd_p, 0, pgd_size); } =20 -static pud_t __init *sme_prepare_pgd(struct sme_populate_pgd_data *ppd) +static pud_t __pitext *sme_prepare_pgd(struct sme_populate_pgd_data *ppd) { pgd_t *pgd; p4d_t *p4d; @@ -112,7 +113,7 @@ static pud_t __init *sme_prepare_pgd(struct sme_populat= e_pgd_data *ppd) p4d =3D ppd->pgtable_area; memset(p4d, 0, sizeof(*p4d) * PTRS_PER_P4D); ppd->pgtable_area +=3D sizeof(*p4d) * PTRS_PER_P4D; - set_pgd(pgd, __pgd(PGD_FLAGS | __pa(p4d))); + set_pgd_kernel(pgd, __pgd(PGD_FLAGS | __pa(p4d))); } =20 p4d =3D p4d_offset(pgd, ppd->vaddr); @@ -120,7 +121,7 @@ static pud_t __init *sme_prepare_pgd(struct sme_populat= e_pgd_data *ppd) pud =3D ppd->pgtable_area; memset(pud, 0, sizeof(*pud) * PTRS_PER_PUD); ppd->pgtable_area +=3D sizeof(*pud) * PTRS_PER_PUD; - set_p4d(p4d, __p4d(P4D_FLAGS | __pa(pud))); + set_p4d_kernel(p4d, __p4d(P4D_FLAGS | __pa(pud))); } =20 pud =3D pud_offset(p4d, ppd->vaddr); @@ -137,7 +138,7 @@ static pud_t __init *sme_prepare_pgd(struct sme_populat= e_pgd_data *ppd) return pud; } =20 -static void __init sme_populate_pgd_large(struct sme_populate_pgd_data *pp= d) +static void __pitext sme_populate_pgd_large(struct sme_populate_pgd_data *= ppd) { pud_t *pud; pmd_t *pmd; @@ -153,7 +154,7 @@ static void __init sme_populate_pgd_large(struct sme_po= pulate_pgd_data *ppd) set_pmd(pmd, __pmd(ppd->paddr | ppd->pmd_flags)); } =20 -static void __init sme_populate_pgd(struct sme_populate_pgd_data *ppd) +static void __pitext sme_populate_pgd(struct sme_populate_pgd_data *ppd) { pud_t *pud; pmd_t *pmd; @@ -179,7 +180,7 @@ static void __init sme_populate_pgd(struct sme_populate= _pgd_data *ppd) set_pte(pte, __pte(ppd->paddr | ppd->pte_flags)); } =20 -static void __init __sme_map_range_pmd(struct sme_populate_pgd_data *ppd) +static void __pitext __sme_map_range_pmd(struct sme_populate_pgd_data *ppd) { while (ppd->vaddr < ppd->vaddr_end) { sme_populate_pgd_large(ppd); @@ -189,7 +190,7 @@ static void __init __sme_map_range_pmd(struct sme_popul= ate_pgd_data *ppd) } } =20 -static void __init __sme_map_range_pte(struct sme_populate_pgd_data *ppd) +static void __pitext __sme_map_range_pte(struct sme_populate_pgd_data *ppd) { while (ppd->vaddr < ppd->vaddr_end) { sme_populate_pgd(ppd); @@ -199,7 +200,7 @@ static void __init __sme_map_range_pte(struct sme_popul= ate_pgd_data *ppd) } } =20 -static void __init __sme_map_range(struct sme_populate_pgd_data *ppd, +static void __pitext __sme_map_range(struct sme_populate_pgd_data *ppd, pmdval_t pmd_flags, pteval_t pte_flags) { unsigned long vaddr_end; @@ -223,22 +224,22 @@ static void __init __sme_map_range(struct sme_populat= e_pgd_data *ppd, __sme_map_range_pte(ppd); } =20 -static void __init sme_map_range_encrypted(struct sme_populate_pgd_data *p= pd) +static void __pitext sme_map_range_encrypted(struct sme_populate_pgd_data = *ppd) { __sme_map_range(ppd, PMD_FLAGS_ENC, PTE_FLAGS_ENC); } =20 -static void __init sme_map_range_decrypted(struct sme_populate_pgd_data *p= pd) +static void __pitext sme_map_range_decrypted(struct sme_populate_pgd_data = *ppd) { __sme_map_range(ppd, PMD_FLAGS_DEC, PTE_FLAGS_DEC); } =20 -static void __init sme_map_range_decrypted_wp(struct sme_populate_pgd_data= *ppd) +static void __pitext sme_map_range_decrypted_wp(struct sme_populate_pgd_da= ta *ppd) { __sme_map_range(ppd, PMD_FLAGS_DEC_WP, PTE_FLAGS_DEC_WP); } =20 -static unsigned long __init sme_pgtable_calc(unsigned long len) +static unsigned long __pitext sme_pgtable_calc(unsigned long len) { unsigned long entries =3D 0, tables =3D 0; =20 @@ -275,7 +276,7 @@ static unsigned long __init sme_pgtable_calc(unsigned l= ong len) return entries + tables; } =20 -void __init sme_encrypt_kernel(struct boot_params *bp) +void __pitext sme_encrypt_kernel(struct boot_params *bp) { unsigned long workarea_start, workarea_end, workarea_len; unsigned long execute_start, execute_end, execute_len; @@ -310,8 +311,8 @@ void __init sme_encrypt_kernel(struct boot_params *bp) */ =20 /* Physical addresses gives us the identity mapped virtual addresses */ - kernel_start =3D __pa_symbol(_text); - kernel_end =3D ALIGN(__pa_symbol(_end), PMD_SIZE); + kernel_start =3D __pa(_text); + kernel_end =3D ALIGN(__pa(_end), PMD_SIZE); kernel_len =3D kernel_end - kernel_start; =20 initrd_start =3D 0; @@ -488,7 +489,7 @@ void __init sme_encrypt_kernel(struct boot_params *bp) native_write_cr3(__native_read_cr3()); } =20 -void __init sme_enable(struct boot_params *bp) +void __pitext sme_enable(struct boot_params *bp) { unsigned int eax, ebx, ecx, edx; unsigned long feature_mask; --=20 2.43.0.429.g432eaa2c6b-goog From nobody Wed Dec 24 01:33:42 2025 Received: from mail-yw1-f201.google.com (mail-yw1-f201.google.com [209.85.128.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D9C1715B0F2 for ; Mon, 29 Jan 2024 18:06:08 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706551570; cv=none; b=A1k5sqPFbf80CKEnCWZbxE2xCcFTavcfU0CDsfxzpQHnBbMCF75Ni9SFCUx3fJNfKUSW1MacOHcRbAl0nAdY4ZIJhp1NB9l3+Nc5hBBhF/dNaBb1A1LmOefZHJVpJcZH1uu/+8kr+qbmU8hTKGVepoRuw+tA3E9bYeool6Zx6tw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706551570; c=relaxed/simple; bh=hGhdgMicSLDQ/VaHKkAj5Vre9mOZCJGZnI0ycPa0ctU=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=J3FtkDgtyL8ziuPTzbJIOGE1uaDsdrKdbNE2pLIDw+wiN8U1fKkqqX7/2l6vCMpG0xd6gf3sTwt56Axp4rWKln2iXhPfllQ3u5BA9V3FTnBhaOdpfmrqzE24miIjaZqnSP+cLzEQA/g0Z7SlHybS058v7yq7qDWFNxK8Q+yq//8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=XbwiJPND; arc=none smtp.client-ip=209.85.128.201 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="XbwiJPND" Received: by mail-yw1-f201.google.com with SMTP id 00721157ae682-5ffa2bb4810so34296907b3.0 for ; Mon, 29 Jan 2024 10:06:08 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1706551568; x=1707156368; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=OvmdeW4/5l6h8clue2msSF5I8s+RaqVbElGLLzTfnIk=; b=XbwiJPNDWyHGc0IWoPOI2a1jWrHMXMQS1H/JWBiNiYH+ict5AV8k82cpOEu1tpYesX 12KLaUwS8JU1bFVHdiHtVkT3bCCzxp+pfg5kuCAdCk+773zEVlcAdtuUUlN+vzO2k2O5 3voCXta68R6WV5uQ784mD0UhhJg39VuC5AVjdzQTzwoOFFe95F9wzNfAlWFshxInDXQ5 EG/45KSqnS1t8ltyl0mNIX9fSEJWp4BJEaKJNH5L5K7JloaRrV3LN5n3yw/1MeaLtOdA b1wurvtFRzczj2Q9q7AMtdNL10K14wZ0ixelo2twZbggzqamVE2LRV1ZQmfTb3/HJKcK sGOw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1706551568; x=1707156368; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=OvmdeW4/5l6h8clue2msSF5I8s+RaqVbElGLLzTfnIk=; b=Sq9bie5Aw3TKvxpab0miYY+JVrn0UFAcxYLv7ozrJsKV5EvAyHTsyayFIdYatsi1px qg+/xu6e0WbjchukTIPEQBO0rD2RP0vfVZAliOfrqpDlquHOfqBiwQwlFLBxiXeZBoem tTjQnxh8Kei2E1z2lwWSXhZ1he17K2ni1BbfuAvfeUenf0Gs7jmStWV3RaH6ICpduJh+ Ar1qD/YE+gymSHKYE0VmzubHj4MEa59DsZb76t1toeJ/gOHgrWsSLb3+ys5VWfbnLZdW bUmmveIS1h5lp3Ibk4wB+yDzxGt6ZqI5AdQDAOqRjC3A9A409Ij9F05pbOCWU32xiFvM D/JQ== X-Gm-Message-State: AOJu0Yy1B1gxifGg4MtktwSCsCKJHd6xXRmdIivFtdnMQ/KMGOxiwzSX jlwYfaS3z/2bjf1MyAW9uanCBymETH2RSEUV/Wv1e+oTUlKNOxFdFhpSq/+xdykItLGyx948fkx 7j3Vami75I+Sj1wjc/vXgd2gXmIT0PGf6AJnb1bIwBcTnhwmcN6sTyvuyAQrq4lAZh5fd2xtTBz gACpOik3VWehEUtZiZWLc+encUdOsrDQ== X-Google-Smtp-Source: AGHT+IEvm5tXIhZK0iVzSFxmZq0xz25L/Fykdz68esiBebN0gmjK2KT5a1HMeCysY/4pBr4f9cq1K466 X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a05:6902:2206:b0:dc2:42fc:1366 with SMTP id dm6-20020a056902220600b00dc242fc1366mr424438ybb.9.1706551567626; Mon, 29 Jan 2024 10:06:07 -0800 (PST) Date: Mon, 29 Jan 2024 19:05:19 +0100 In-Reply-To: <20240129180502.4069817-21-ardb+git@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240129180502.4069817-21-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=1898; i=ardb@kernel.org; h=from:subject; bh=BAMQXD8G5KJ2RAZEnRCY7V9ynmVKgA6YQxKkMkZhfsc=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIXX7iwd2nB82KC89l5dabni8zdb++Mqo/SUlzOIzelh7A tI2J/Z2lLIwiHEwyIopsgjM/vtu5+mJUrXOs2Rh5rAygQxh4OIUgImckGb4Z9S36tFe1cvVWzbu ayl98UOoXKWc18nAMa1us+c8oyMr0xn++2XGVlQoGddc37R5goyOv+bk+2UVzxwrL7hu8HNk3C/ PAwA= X-Mailer: git-send-email 2.43.0.429.g432eaa2c6b-goog Message-ID: <20240129180502.4069817-37-ardb+git@google.com> Subject: [PATCH v3 16/19] x86/sev: Avoid WARN() in early code From: Ard Biesheuvel To: linux-kernel@vger.kernel.org Cc: Ard Biesheuvel , Kevin Loughlin , Tom Lendacky , Dionna Glaze , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , Andy Lutomirski , Arnd Bergmann , Nathan Chancellor , Nick Desaulniers , Justin Stitt , Kees Cook , Brian Gerst , linux-arch@vger.kernel.org, llvm@lists.linux.dev Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Ard Biesheuvel Drop uses of WARN() from code that is reachable from the early primary boot path which executes via the initial 1:1 mapping before the kernel page tables are populated. This is unsafe and mostly pointless, given that printk() does not actually work yet at this point. Signed-off-by: Ard Biesheuvel --- arch/x86/kernel/sev.c | 13 ++++--------- 1 file changed, 4 insertions(+), 9 deletions(-) diff --git a/arch/x86/kernel/sev.c b/arch/x86/kernel/sev.c index 62981b463b76..94bf054bbde3 100644 --- a/arch/x86/kernel/sev.c +++ b/arch/x86/kernel/sev.c @@ -698,7 +698,7 @@ static void __pitext early_set_pages_state(unsigned lon= g vaddr, unsigned long pa if (op =3D=3D SNP_PAGE_STATE_SHARED) { /* Page validation must be rescinded before changing to shared */ ret =3D pvalidate(vaddr, RMP_PG_SIZE_4K, false); - if (WARN(ret, "Failed to validate address 0x%lx ret %d", paddr, ret)) + if (ret) goto e_term; } =20 @@ -711,21 +711,16 @@ static void __pitext early_set_pages_state(unsigned l= ong vaddr, unsigned long pa =20 val =3D sev_es_rd_ghcb_msr(); =20 - if (WARN(GHCB_RESP_CODE(val) !=3D GHCB_MSR_PSC_RESP, - "Wrong PSC response code: 0x%x\n", - (unsigned int)GHCB_RESP_CODE(val))) + if (GHCB_RESP_CODE(val) !=3D GHCB_MSR_PSC_RESP) goto e_term; =20 - if (WARN(GHCB_MSR_PSC_RESP_VAL(val), - "Failed to change page state to '%s' paddr 0x%lx error 0x%llx\n", - op =3D=3D SNP_PAGE_STATE_PRIVATE ? "private" : "shared", - paddr, GHCB_MSR_PSC_RESP_VAL(val))) + if (GHCB_MSR_PSC_RESP_VAL(val)) goto e_term; =20 if (op =3D=3D SNP_PAGE_STATE_PRIVATE) { /* Page validation must be performed after changing to private */ ret =3D pvalidate(vaddr, RMP_PG_SIZE_4K, true); - if (WARN(ret, "Failed to validate address 0x%lx ret %d", paddr, ret)) + if (ret) goto e_term; } =20 --=20 2.43.0.429.g432eaa2c6b-goog From nobody Wed Dec 24 01:33:42 2025 Received: from mail-wm1-f74.google.com (mail-wm1-f74.google.com [209.85.128.74]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5C1B615B10B for ; Mon, 29 Jan 2024 18:06:11 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.74 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706551572; cv=none; b=PcJflShk5gGa1KxGvKMt6MqHD9wRZyGHROfQ9xHnYHEU9bs6hmOHUGeSy62WCV2JhD6/GDzO3gDKerWT8Lhbp/te/1+BLWgXyFGl/2Bs/p3Lj3XwdfVrw7sTbvbdDnWdCGg+EEW1AgIswiKJR1mI4bX/6t+EqPfYCIV4feyGYWY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706551572; c=relaxed/simple; bh=N7Dqbs6A3xIbAMqCtcMIQ29SguGASEykNJNGXXdKxFI=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=kNbCIcEEPPrJmYxWm0NkPP+bAOoKkRWTDzlmYE5E1ydf4xpirSSAm4hlVJyzKX4TFemEneH+DjhgxB+n2oeF4XS34z++wc+xxOfSYO55S/OFHJ1ZaV9N+NdAAXfgQIexbMi0NcdZiDn3yLlKI4TSQQiMhleAeabXoUtdSSwqzLw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=2I3ub+KA; arc=none smtp.client-ip=209.85.128.74 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="2I3ub+KA" Received: by mail-wm1-f74.google.com with SMTP id 5b1f17b1804b1-40eee438e92so15564265e9.1 for ; Mon, 29 Jan 2024 10:06:11 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1706551570; x=1707156370; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=Yw+flZGULMSavtLRoK2qSf3V9Gu3GyVCc2w7RxQ06Ro=; b=2I3ub+KA+rXs8X6vvGIrFkAe+Gv/fHigK+cknQ8Tq99AuS080zBsMsYdeF/IHzGKDs f/kX1tPo+JdTFWylmqmjE1WS7H/x9PlEbGTIPctIcJSSUtgj4HVkODUp1omxzKZS4Pp8 3bYOl6jSOMbOJ1F5Sd67BcJXP55dKmT5TYHlRHhsHwWZ05u+iffKBaKDDn3AYn/xYCnc ECDpukjhYD5vE+2XrgIe2DP6LJ9uG8fex8b+XHSDVm7sQge4/eIeXkYWJsxiKO/S/Ocw ABqIw6T0Agt2x0+jBf7g2bXhbvJnnwGCzxA57f/cpf+g1jl2/Yj9ddmIyU10EzJ9CHSv U5yw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1706551570; x=1707156370; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=Yw+flZGULMSavtLRoK2qSf3V9Gu3GyVCc2w7RxQ06Ro=; b=h7RxWA6B3SDzL1MWJ6JnNQP9ds7BHnMT9FYOKl6pTktzoVZxC5f3ockGROUGbJ8PFx bO98Q4ts36qFw6CbYdOtMLw18z4c9Dom7brzRIgCNAP6kKU64PNdt6IeqW2QPy0E0oHF Ywq+dNkcIF6dg4efSFIebHn1aA7MmNjvISzT+RI70wz5v0phsawwvQ4UtOHcepZl5KP+ EDEd9FmmsYl9OyOR+enZlqgll4XcUMsrOvopdAoS9ojPUvwmmRKnnrf/Jqk9rWw41wUo hs4Z5k7aSQrP2JmoVrbCH5aRVyGxu+xfubx8+kx0RmVEloqmGvXPl4aR0/lZtsqDrA9j CGlA== X-Gm-Message-State: AOJu0YwSlz+qU9QWw67sH/4DQRWRQVchHNeVXfHs2RbeK1ECok1Eq2Qw UmU9ObTntYXnO3QBx6lUH35X9SreiepULsg5zC+DemL5cFY6G7IA1++kuX2SkKzzOitzO33I9Qa iWyATG3ZWT9DtkhhbgTf/a78It6QQFX2Y+aNB2EjZ3wEFcrWcOoVFAB4oMq6SAMKjRU3Clkk4Qg BL+b26aU8ZDF6WOjr0Op9oq2pK3SlYYw== X-Google-Smtp-Source: AGHT+IENcq3eVgTctzbegGvOoo9ucEIbkmBu/hWVgKdlKYbqpqwTZckCmZpKMHsy8T2zaFDU+s/vXnzc X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a05:6000:1708:b0:33a:f0f9:bebc with SMTP id n8-20020a056000170800b0033af0f9bebcmr4262wrc.7.1706551569728; Mon, 29 Jan 2024 10:06:09 -0800 (PST) Date: Mon, 29 Jan 2024 19:05:20 +0100 In-Reply-To: <20240129180502.4069817-21-ardb+git@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240129180502.4069817-21-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=1945; i=ardb@kernel.org; h=from:subject; bh=gUz2qElRticSrvsvtLUDoN8B1p79LARk6mWyXvU24wE=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIXX7i4dqvH15HX5Wf/uN1EQe/hE6urU95U1q1p2ZgeqrZ 6o4OO7sKGVhEONgkBVTZBGY/ffdztMTpWqdZ8nCzGFlAhnCwMUpABOpm8DIsD7idciuqibGy28P sYtcmGQ4exvfPMk+4cUb/bhsFEIyljH8L3704rv+6ZWdMRP4FHZsWvXR8MaPtf27Pkm3P9t/x7U 8kBMA X-Mailer: git-send-email 2.43.0.429.g432eaa2c6b-goog Message-ID: <20240129180502.4069817-38-ardb+git@google.com> Subject: [PATCH v3 17/19] x86/sev: Use PIC codegen for early SEV startup code From: Ard Biesheuvel To: linux-kernel@vger.kernel.org Cc: Ard Biesheuvel , Kevin Loughlin , Tom Lendacky , Dionna Glaze , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , Andy Lutomirski , Arnd Bergmann , Nathan Chancellor , Nick Desaulniers , Justin Stitt , Kees Cook , Brian Gerst , linux-arch@vger.kernel.org, llvm@lists.linux.dev Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Ard Biesheuvel Use PIC codegen for the compilation units containing code that may be called very early during the boot, at which point the CPU still runs from the 1:1 mapping of memory. This is necessary to prevent the compiler from emitting absolute symbol references to addresses that are not mapped yet. Signed-off-by: Ard Biesheuvel --- arch/x86/kernel/Makefile | 2 ++ arch/x86/kernel/vmlinux.lds.S | 1 + arch/x86/mm/Makefile | 2 +- 3 files changed, 4 insertions(+), 1 deletion(-) diff --git a/arch/x86/kernel/Makefile b/arch/x86/kernel/Makefile index 42db41b04d8e..3819b65c64ec 100644 --- a/arch/x86/kernel/Makefile +++ b/arch/x86/kernel/Makefile @@ -24,7 +24,9 @@ endif # head64.c contains C code that may execute from a different virtual addre= ss # than it was linked at, so we always build it using PIE codegen CFLAGS_head64.o +=3D $(PIE_CFLAGS) +CFLAGS_sev.o +=3D $(PIE_CFLAGS) UBSAN_SANITIZE_head64.o :=3D n +UBSAN_SANITIZE_sev.o :=3D n =20 KASAN_SANITIZE_head$(BITS).o :=3D n KASAN_SANITIZE_dumpstack.o :=3D n diff --git a/arch/x86/kernel/vmlinux.lds.S b/arch/x86/kernel/vmlinux.lds.S index 77262e804250..bbdccb6362a9 100644 --- a/arch/x86/kernel/vmlinux.lds.S +++ b/arch/x86/kernel/vmlinux.lds.S @@ -182,6 +182,7 @@ SECTIONS =20 DATA_DATA CONSTRUCTORS + *(.data.rel .data.rel.*) =20 /* rarely changed data like cpu maps */ READ_MOSTLY_DATA(INTERNODE_CACHE_BYTES) diff --git a/arch/x86/mm/Makefile b/arch/x86/mm/Makefile index c80febc44cd2..f3bb8b415348 100644 --- a/arch/x86/mm/Makefile +++ b/arch/x86/mm/Makefile @@ -31,7 +31,7 @@ obj-y +=3D pat/ =20 # Make sure __phys_addr has no stackprotector CFLAGS_physaddr.o :=3D -fno-stack-protector -CFLAGS_mem_encrypt_identity.o :=3D -fno-stack-protector +CFLAGS_mem_encrypt_identity.o :=3D $(PIE_CFLAGS) =20 CFLAGS_fault.o :=3D -I $(srctree)/$(src)/../include/asm/trace =20 --=20 2.43.0.429.g432eaa2c6b-goog From nobody Wed Dec 24 01:33:42 2025 Received: from mail-wm1-f73.google.com (mail-wm1-f73.google.com [209.85.128.73]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5F5C415B2EB for ; Mon, 29 Jan 2024 18:06:13 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.73 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706551575; cv=none; b=ARFHUuvkpMCGrtINIJAbf97McxWRqXxxZMY6INHRixqCYP/8K7j5CrYyc7+LmpllRVNmL9s97sEMw1kyqlxtq5fsYxRwGZHM6pAwO5f7wDCR0iE6GN6cCVGp9Br4ZN1CoK8Roz54yBgKLwU/zdZ0WGgrzY8mvcquPZEr0EVnnHI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706551575; c=relaxed/simple; bh=+rYahPmzXDYe/IO3FyUv8LMIVG+yTpD3LpxEPcnxKUw=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=IVsM7QmjPGaA5OUPo1UwA/R78LciFRoPunB9Hav/5YZsNKpIghwzmWe5sRi7UYrYbLokJm6Ni18SvvkM4+ZtFCoAGup7aLjxdDFlzS5eI2eqLxahj/HMuYrgFLB9TKEDLIXPnF67FQUKxqr9aRIYkG0DjZpecen07RIQEtaMXqU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=NuBUM7eM; arc=none smtp.client-ip=209.85.128.73 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="NuBUM7eM" Received: by mail-wm1-f73.google.com with SMTP id 5b1f17b1804b1-40e8810b5f3so25162785e9.3 for ; Mon, 29 Jan 2024 10:06:13 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1706551572; x=1707156372; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=1we6nPeM2Zvq3sHnH1EZ2vSDoW7zEMVGBC8FJkE9Ugc=; b=NuBUM7eMy9VUEvE4klzVPO8suOqnNBJKEIlUbOR4EOJ0cy9n0cLWVhi9uqmckaOKV3 vKCTVzkhDdfk1NqdoZ1EO3WswEnzzSG3C5Lusb1koMY1n0V03xptgRjGSfgK6+zOV8nP xn44PGDrMEbzWx/eI9VTIzVXwmqerB5y++LD66/VpV95ZZ8iW2mBcjDt6CMKcoAGKw+/ tE+MyU0lNWRqCgE6Nri8xjSxn0luthIgwbSrKlMfS6o0bck+JUnM/CM67R2+kE51vSI+ lyRoB+Aqm8Qck5I4MsrUspl30qMTrlVvDmpb0EurAuHynhxfqGOgCVAfpbZpJ4iGn7Lx sAEg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1706551572; x=1707156372; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=1we6nPeM2Zvq3sHnH1EZ2vSDoW7zEMVGBC8FJkE9Ugc=; b=DjsOIELl6T3L7r1n2sAeSv06ZwGg51JHOWhMs0VHDJosisV7swOm7teLjt0f6GdUPU tGR2Klhh/e2sNnfL5nidQGUG7J/vNeSTxXLRMQe46AoZVQQUSlKk8qGCRSaS7A7sePzW 8XIQNA15VmvTNxkQZv+3eB3m5Xakz2cEHvZGgC5/eszX4rq8BwTIZ4ZeuRNRQjuugETd t1tyExU0wtAJ0yuRFE0sF2iTPGfWi20Fk31jBhjuzynm2MsfwnAXayk2yfSF1ukj79nQ ZpAQHlEcwOFfEeyW7EU4oATbdlaZBCZQvdtK890RXatqsCIeJg6wzRkvPjNqOM5kpFDX FtrQ== X-Gm-Message-State: AOJu0YzjAeqagg/viD3IYF+SaYIjKaCO1Ae/i8Cet5alATXPQYAcYzHf 2FflBWmO8CB+B508OampJze8B5RgAU0RoNP54aCvK3Shau+Rhdyuvj0W39P9F21CWmhXfw4iqzR O0gh1vJjB4Esr5q//xFr5Z1f2g67Yq0opk7CBaRtpr0CJzy8B6BEbnADMaHt5drRrnbxw7Vtnxf kYSd45DY/a3E8UQrbmy6Yqnl1NBu5DcQ== X-Google-Smtp-Source: AGHT+IHZlPWQchy0J38KpxwD5TmOBCD6HsGG1BLHK/s1BgvvkdecR9aOkvWWPLTFP29XujBFiYmoqpYE X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a05:600c:6012:b0:40e:fa6b:cda8 with SMTP id az18-20020a05600c601200b0040efa6bcda8mr10582wmb.7.1706551571814; Mon, 29 Jan 2024 10:06:11 -0800 (PST) Date: Mon, 29 Jan 2024 19:05:21 +0100 In-Reply-To: <20240129180502.4069817-21-ardb+git@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240129180502.4069817-21-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=2648; i=ardb@kernel.org; h=from:subject; bh=b16vPiy6Nf2YAluruWpP640zmYr0MIrigfShg9HjlpU=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIXX7i0evc6pO7Ltxi++Y3aTSOvsf+18vOOEd7XDs37FJk kwxfyOaO0pZGMQ4GGTFFFkEZv99t/P0RKla51myMHNYmUCGMHBxCsBENp1nZHho+2iy3+ZYHdd9 a8y1g98YMW3T67f1jI++f8Ps+2uORbmMDB2/ps69muN/7KifQ8rL5Sl2RxY49nk1iJ1N+VQpkCn mzAsA X-Mailer: git-send-email 2.43.0.429.g432eaa2c6b-goog Message-ID: <20240129180502.4069817-39-ardb+git@google.com> Subject: [PATCH v3 18/19] x86/sev: Drop inline asm LEA instructions for RIP-relative references From: Ard Biesheuvel To: linux-kernel@vger.kernel.org Cc: Ard Biesheuvel , Kevin Loughlin , Tom Lendacky , Dionna Glaze , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , Andy Lutomirski , Arnd Bergmann , Nathan Chancellor , Nick Desaulniers , Justin Stitt , Kees Cook , Brian Gerst , linux-arch@vger.kernel.org, llvm@lists.linux.dev Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Ard Biesheuvel The SEV code that may run early is now built with -fPIC and so there is no longer a need for explicit RIP-relative references in inline asm, given that is what the compiler will emit as well. Signed-off-by: Ard Biesheuvel --- arch/x86/kernel/sev-shared.c | 14 +------------- arch/x86/mm/mem_encrypt_identity.c | 11 +---------- 2 files changed, 2 insertions(+), 23 deletions(-) diff --git a/arch/x86/kernel/sev-shared.c b/arch/x86/kernel/sev-shared.c index 481dbd009ce9..1cfbc6d0df89 100644 --- a/arch/x86/kernel/sev-shared.c +++ b/arch/x86/kernel/sev-shared.c @@ -325,21 +325,9 @@ static int __pitext sev_cpuid_hv(struct ghcb *ghcb, st= ruct es_em_ctxt *ctxt, : __sev_cpuid_hv_msr(leaf); } =20 -/* - * This may be called early while still running on the initial identity - * mapping. Use RIP-relative addressing to obtain the correct address - * while running with the initial identity mapping as well as the - * switch-over to kernel virtual addresses later. - */ static const struct snp_cpuid_table *snp_cpuid_get_table(void) { - void *ptr; - - asm ("lea cpuid_table_copy(%%rip), %0" - : "=3Dr" (ptr) - : "p" (&cpuid_table_copy)); - - return ptr; + return &cpuid_table_copy; } =20 /* diff --git a/arch/x86/mm/mem_encrypt_identity.c b/arch/x86/mm/mem_encrypt_i= dentity.c index bc39e04de980..d01e6b1256c6 100644 --- a/arch/x86/mm/mem_encrypt_identity.c +++ b/arch/x86/mm/mem_encrypt_identity.c @@ -85,7 +85,6 @@ struct sme_populate_pgd_data { */ static char sme_workarea[2 * PMD_SIZE] __section(".init.scratch"); =20 - static void __pitext sme_clear_pgd(struct sme_populate_pgd_data *ppd) { unsigned long pgd_start, pgd_end, pgd_size; @@ -329,14 +328,6 @@ void __pitext sme_encrypt_kernel(struct boot_params *b= p) } #endif =20 - /* - * We're running identity mapped, so we must obtain the address to the - * SME encryption workarea using rip-relative addressing. - */ - asm ("lea sme_workarea(%%rip), %0" - : "=3Dr" (workarea_start) - : "p" (sme_workarea)); - /* * Calculate required number of workarea bytes needed: * executable encryption area size: @@ -346,7 +337,7 @@ void __pitext sme_encrypt_kernel(struct boot_params *bp) * pagetable structures for the encryption of the kernel * pagetable structures for workarea (in case not currently mapped) */ - execute_start =3D workarea_start; + execute_start =3D workarea_start =3D (unsigned long)sme_workarea; execute_end =3D execute_start + (PAGE_SIZE * 2) + PMD_SIZE; execute_len =3D execute_end - execute_start; =20 --=20 2.43.0.429.g432eaa2c6b-goog From nobody Wed Dec 24 01:33:42 2025 Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 29D5515B2FA for ; Mon, 29 Jan 2024 18:06:15 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.202 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706551576; cv=none; b=JhY4dVh4axXUirDpFJDVuF4Dj8oME89IwyBG0lhJlDwWXP1HcM2nOIhS1KZx8moKmceJEdeW467io+7oH6+NVY7QQ0bZqjeWsLRAR0HNpVinBvEtr13DWwESCAF2fQsA6TZI9SNiPNhnGXEelC2+1Izi1rje6o8Tg5Wu+oZu9VA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706551576; c=relaxed/simple; bh=q9aqq8TxofJ+IbL58aBuKCqKThJaXrgTFJRKYbTq++8=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=IlwHzdB5nrt1Cst46azIFL0FPN8XFNTWyOTwWN5/ePFYV8Kh9/F0njerUP2cCkY+Eb/KOTbiIBqHjeKI05JSIliVuURRwITFJJ2wPCy+O3GSK4Z5stkG/OVVTKaHZcOdDo8RA1tgGq3J8h7CmspDDXEWWVpsNF+6hub4WnmgK3A= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=PI2zoCmS; arc=none smtp.client-ip=209.85.128.202 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="PI2zoCmS" Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-5efe82b835fso61584467b3.0 for ; Mon, 29 Jan 2024 10:06:14 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1706551574; x=1707156374; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=M59gqHs6X4RPZ31jkw+MshCoOf2mD10mD/ghxicfQU0=; b=PI2zoCmSh14+b4AA5jJR2sX2z9H17Q65a3h8bGjuLx9dY9bwym/qVfeZOru9Wwm0lx wiohVPoecIwJ0lUbMKsVumbnStU26Ng9U9ds8aL5kS5HJoIoqSxYHtvBrjVI3Uqj0Xru h96xMahc+oVaWvAm3bOxq9VcC6XzxhrZaGwCupwwFmldsahlIGhZBnMtlNKJRjpvmWB/ S09VbtlXvSrTCWErgmoXiDi9vBZn5MQZyn6NLg/pLFj7Q0iD05slwkq4oEUsWaC1zXAU Y5zkQ6mD/1Kg6a7NSXJIHuKw7j+dv8AMXu4ZwBHxghuqhDjDHhpVHfMDbjbI8DBk2kfR ausA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1706551574; x=1707156374; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=M59gqHs6X4RPZ31jkw+MshCoOf2mD10mD/ghxicfQU0=; b=HwQovMp8Z8C/M4K7LZka+kXfEPY1t/nTO8oLVAMnrk7DRlDw2UhaRtCuF3HEEN7ph1 1WHenN2TizMqLiC8Zq+dagRpIVF693ekeT6Y17Vr+GyxDSMxDILYCOtnwactlg9Sf3ED mptA+lCqwjE3xvfK9YsRXVSc9rOFsdgXY/U2cY2MSS0LoFfOyAxYv1Y/VIQGGTG9vVUk gWOtJO4gooJOekqM1efZRFFBoQjC01XM3XXjiwm6yPLqC7W36pP4lCqYvm4el7aiBDYO IcM/2+LE2DI4geme/5FEB9HpqltvHvQBVypd6r/Y8I4e6zSDMZkQzp1SVJ8I9OLQV+bL WFwA== X-Gm-Message-State: AOJu0YxQ4i+3tDWD+fvjpI7ED+XSGfLtXVvPUoN9Y7Hh1l1LFtAafNfo yyHDs+saTK5FlLzhAW1vUZlGTswebCHjbqgzxz0fgtKUqSkfIh+GOD8hm6XYCuJwJPZT64l7Ooi WCs27A4K39cmXjGVeFJBc7t+Xa91J0x/KAx3wUPILkYXy6fH7W0lGeUlkSPBkhwt80+LwFM2+7Y 6LBeL6IaBSY1blgiMSGobHpY2mM1tcSQ== X-Google-Smtp-Source: AGHT+IFKsXEen3EWZg9GhnUEMC4esMp++s/hxM7yTCMDL5+CtA/pBJk6xsNb+SSidJj/K445XegU9it+ X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a81:9a88:0:b0:5fb:63fc:fac8 with SMTP id r130-20020a819a88000000b005fb63fcfac8mr2074385ywg.8.1706551574143; Mon, 29 Jan 2024 10:06:14 -0800 (PST) Date: Mon, 29 Jan 2024 19:05:22 +0100 In-Reply-To: <20240129180502.4069817-21-ardb+git@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240129180502.4069817-21-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=1282; i=ardb@kernel.org; h=from:subject; bh=ZbpL85fPGajjpfnYR9ri/bu07iizUhPZX6zRx/L0fxw=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIXX7iyezZ8/20v2498Hj1helL98vOvzjH/OrvzFPD01z6 WVvCTv1u6OUhUGMg0FWTJFFYPbfdztPT5SqdZ4lCzOHlQlkCAMXpwBMpDqI4a/I1ZMTrb8Ks7u2 Rx90kNBJu3H3p+ZT9ir3nhQO89melQIMf0WfiB3aZdCac9RvY2/CzGcL9fXyZjqd61Fv+sBuM7d mGQcA X-Mailer: git-send-email 2.43.0.429.g432eaa2c6b-goog Message-ID: <20240129180502.4069817-40-ardb+git@google.com> Subject: [PATCH v3 19/19] x86/startup_64: Don't bother setting up GS before the kernel is mapped From: Ard Biesheuvel To: linux-kernel@vger.kernel.org Cc: Ard Biesheuvel , Kevin Loughlin , Tom Lendacky , Dionna Glaze , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , Andy Lutomirski , Arnd Bergmann , Nathan Chancellor , Nick Desaulniers , Justin Stitt , Kees Cook , Brian Gerst , linux-arch@vger.kernel.org, llvm@lists.linux.dev Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Ard Biesheuvel The code that executes from the early 1:1 mapping of the kernel should set up the kernel page tables and nothing else. C code that is linked into this code path is severely restricted in what it can do, and is therefore required to remain uninstrumented. It also built with -fPIC and without stack protector support. This makes it unnecessary to enable per-CPU variable access this early, and for the boot CPU, the initialization that occurs in the common CPU startup path is sufficient. Signed-off-by: Ard Biesheuvel --- arch/x86/kernel/head_64.S | 7 ------- 1 file changed, 7 deletions(-) diff --git a/arch/x86/kernel/head_64.S b/arch/x86/kernel/head_64.S index e671caafd932..ae211cb62a1e 100644 --- a/arch/x86/kernel/head_64.S +++ b/arch/x86/kernel/head_64.S @@ -76,13 +76,6 @@ SYM_CODE_START_LOCAL(primary_startup_64) /* Set up the stack for verify_cpu() */ leaq (__end_init_task - PTREGS_SIZE)(%rip), %rsp =20 - /* Setup GSBASE to allow stack canary access for C code */ - movl $MSR_GS_BASE, %ecx - leaq INIT_PER_CPU_VAR(fixed_percpu_data)(%rip), %rdx - movl %edx, %eax - shrq $32, %rdx - wrmsr - call startup_64_setup_env =20 /* Now switch to __KERNEL_CS so IRET works reliably */ --=20 2.43.0.429.g432eaa2c6b-goog