From nobody Fri Dec 19 15:49:48 2025 Received: from mail.alien8.de (mail.alien8.de [65.109.113.108]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3CE2A145B2A for ; Sun, 31 Mar 2024 18:01:07 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=65.109.113.108 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711908070; cv=none; b=cLBan2OISenbZPUrxEIxKNP4mRd2KUCUVPepHIZiFswxNMpw/YTPY94vgwX1A1qkLylfs3F8EERcSOHqszrZ5Jqa3PUhSdJua0m3bL3k1H4ureiGh+vloY/OQucEyhnmOAb3xbDsSbCUEOFjbLDBIFX25vdG+QEnCJLvM4AIhqI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711908070; c=relaxed/simple; bh=p8woIZz9mSVPjyvCWDKt4OVUR1tVc5vbUSpsGV5vsKc=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=fCtEPWAcwSnqeo2vUl93wjC3THWHq/q89PMbFDwjJd50WmAF50TVoh0K10FgP+eMvDgXkIhaIdS5Q1JRt6qfiik4pXyRSaZeuv8PkMir/sQKRFHZ//FGzy56G78Dr9pUAWrv1+pUAlB9aBfnNuMb/pUx0DRPhFz0qU4Hyjta+yo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=alien8.de; spf=pass smtp.mailfrom=alien8.de; dkim=fail (4096-bit key) header.d=alien8.de header.i=@alien8.de header.b=Hwgi/YAO reason="signature verification failed"; arc=none smtp.client-ip=65.109.113.108 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=alien8.de Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=alien8.de Authentication-Results: smtp.subspace.kernel.org; dkim=fail reason="signature verification failed" (4096-bit key) header.d=alien8.de header.i=@alien8.de header.b="Hwgi/YAO" Received: from localhost (localhost.localdomain [127.0.0.1]) by mail.alien8.de (SuperMail on ZX Spectrum 128k) with ESMTP id 06A6D40E02A6; Sun, 31 Mar 2024 18:01:05 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at mail.alien8.de Authentication-Results: mail.alien8.de (amavisd-new); dkim=fail (4096-bit key) reason="fail (body has been altered)" header.d=alien8.de Received: from mail.alien8.de ([127.0.0.1]) by localhost (mail.alien8.de [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id a0RXuXejAVCe; Sun, 31 Mar 2024 18:01:00 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=alien8.de; s=alien8; t=1711908059; bh=Q1D4BZ2EvneFNiscjl0Lfx51FaajqA/nealgBM4NuXM=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=Hwgi/YAOIRhlAvdzUObLQ3/WiB3cdgzUqcUYMpDNLgUtx01reFZA2hUhkk6WCGbC8 JxXeBQC+Ij4iWhZPMkxDwUAEUXu+xyNlEvwwwjr/wHuIqTdRvFE9mg+RuEVPs+20fn I7gGF98XxWf4BPvi8bMRzN9NdmQ7Jun7V5ZNIt77f5LXAkiW5a+TDJJCaS5ifl1cnW qEdPj8F+ipQ/4uMnziNOKIM/Wx2Xb3vVST/+ibSwv/OVj90vYT/6DDfojKvvnZ2NTA 8TYvUxwXqlIxRU44adj75dsK3bGmnRLPcqnjyO0zRh0P5VClKhIjfT59ob00yEIixr PeABe/y9EpbBk66ZYPNVXa6Az0OWCCvSN4WMxBux7h1NFGg/t2KH4OI5G2D4Pe4jiE StZdKWw2YFcBbX994sLL2prNXVi69dctrsuYBBdlS0Dk5cWPZwQpk0QE5ij97E83Do OHXcZdouY3dRLb8c3vYAwvx1yRivgFpM/e11orrwsk758ISSiKgo1rMU4vhAXtOu7w SS9X5P+6BMn4ZXMQWZpqp71D6udGi9ZIbLwFr/oP11KPT+vhJY5JIpL76zYH+sSLQO xuZKASIyjvj5znImPbasCUYqC6Ne847vhy/74i4t0H/zcKV8zVLiB/3VU2XUT3XGQS faiCoBqew8gcznli9URwTLMw= Received: from zn.tnic (p5de8ecf7.dip0.t-ipconnect.de [93.232.236.247]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange ECDHE (P-256) server-signature ECDSA (P-256) server-digest SHA256) (No client certificate requested) by mail.alien8.de (SuperMail on ZX Spectrum 128k) with ESMTPSA id 91E8640E0202; Sun, 31 Mar 2024 18:00:57 +0000 (UTC) From: Borislav Petkov To: X86 ML Cc: LKML Subject: [PATCH v2 1/4] x86/alternatives: Use a temporary buffer when optimizing NOPs Date: Sun, 31 Mar 2024 20:00:39 +0200 Message-ID: <20240331180042.13933-2-bp@alien8.de> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240331180042.13933-1-bp@alien8.de> References: <20240331180042.13933-1-bp@alien8.de> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: "Borislav Petkov (AMD)" Instead of optimizing NOPs in-place, use a temporary buffer like the usual alternatives patching flow does. This obviates the need to grab locks when patching, see 6778977590da ("x86/alternatives: Disable interrupts and sync when optimiz= ing NOPs in place") While at it, add nomenclature definitions clarifying and simplifying the naming of function-local variables in the alternatives code. Signed-off-by: Borislav Petkov (AMD) --- arch/x86/include/asm/text-patching.h | 2 +- arch/x86/kernel/alternative.c | 84 +++++++++++++++------------- arch/x86/kernel/callthunks.c | 9 +-- 3 files changed, 49 insertions(+), 46 deletions(-) diff --git a/arch/x86/include/asm/text-patching.h b/arch/x86/include/asm/te= xt-patching.h index 345aafbc1964..6259f1937fe7 100644 --- a/arch/x86/include/asm/text-patching.h +++ b/arch/x86/include/asm/text-patching.h @@ -15,7 +15,7 @@ =20 extern void text_poke_early(void *addr, const void *opcode, size_t len); =20 -extern void apply_relocation(u8 *buf, size_t len, u8 *dest, u8 *src, size_= t src_len); +extern void apply_relocation(u8 *buf, const u8 * const instr, size_t instr= len, u8 *repl, size_t repl_len); =20 /* * Clear and restore the kernel write-protection flag on the local CPU. diff --git a/arch/x86/kernel/alternative.c b/arch/x86/kernel/alternative.c index 45a280f2161c..ec94f1359c00 100644 --- a/arch/x86/kernel/alternative.c +++ b/arch/x86/kernel/alternative.c @@ -124,6 +124,20 @@ const unsigned char * const x86_nops[ASM_NOP_MAX+1] = =3D #endif }; =20 +/* + * Nomenclature for variable names to simplify and clarify this code and e= ase + * any potential staring at it: + * + * @instr: source address of the original instructions in the kernel text = as + * generated by the compiler. + * + * @buf: temporary buffer on which the patching operates. This buffer is + * eventually text-poked into the kernel image. + * + * @replacement/@repl: pointer to the opcodes which are replacing @instr, = located + * in the .altinstr_replacement section. + */ + /* * Fill the buffer with a single effective instruction of size @len. * @@ -133,28 +147,28 @@ const unsigned char * const x86_nops[ASM_NOP_MAX+1] = =3D * each single-byte NOPs). If @len to fill out is > ASM_NOP_MAX, pad with = INT3 and * *jump* over instead of executing long and daft NOPs. */ -static void add_nop(u8 *instr, unsigned int len) +static void add_nop(u8 *buf, unsigned int len) { - u8 *target =3D instr + len; + u8 *target =3D buf + len; =20 if (!len) return; =20 if (len <=3D ASM_NOP_MAX) { - memcpy(instr, x86_nops[len], len); + memcpy(buf, x86_nops[len], len); return; } =20 if (len < 128) { - __text_gen_insn(instr, JMP8_INSN_OPCODE, instr, target, JMP8_INSN_SIZE); - instr +=3D JMP8_INSN_SIZE; + __text_gen_insn(buf, JMP8_INSN_OPCODE, buf, target, JMP8_INSN_SIZE); + buf +=3D JMP8_INSN_SIZE; } else { - __text_gen_insn(instr, JMP32_INSN_OPCODE, instr, target, JMP32_INSN_SIZE= ); - instr +=3D JMP32_INSN_SIZE; + __text_gen_insn(buf, JMP32_INSN_OPCODE, buf, target, JMP32_INSN_SIZE); + buf +=3D JMP32_INSN_SIZE; } =20 - for (;instr < target; instr++) - *instr =3D INT3_INSN_OPCODE; + for (;buf < target; buf++) + *buf =3D INT3_INSN_OPCODE; } =20 extern s32 __retpoline_sites[], __retpoline_sites_end[]; @@ -187,12 +201,12 @@ static bool insn_is_nop(struct insn *insn) * Find the offset of the first non-NOP instruction starting at @offset * but no further than @len. */ -static int skip_nops(u8 *instr, int offset, int len) +static int skip_nops(u8 *buf, int offset, int len) { struct insn insn; =20 for (; offset < len; offset +=3D insn.length) { - if (insn_decode_kernel(&insn, &instr[offset])) + if (insn_decode_kernel(&insn, &buf[offset])) break; =20 if (!insn_is_nop(&insn)) @@ -207,7 +221,7 @@ static int skip_nops(u8 *instr, int offset, int len) * to the end of the NOP sequence into a single NOP. */ static bool -__optimize_nops(u8 *instr, size_t len, struct insn *insn, int *next, int *= prev, int *target) +__optimize_nops(const u8 * const instr, u8 *buf, size_t len, struct insn *= insn, int *next, int *prev, int *target) { int i =3D *next - insn->length; =20 @@ -222,12 +236,12 @@ __optimize_nops(u8 *instr, size_t len, struct insn *i= nsn, int *next, int *prev, if (insn_is_nop(insn)) { int nop =3D i; =20 - *next =3D skip_nops(instr, *next, len); + *next =3D skip_nops(buf, *next, len); if (*target && *next =3D=3D *target) nop =3D *prev; =20 - add_nop(instr + nop, *next - nop); - DUMP_BYTES(ALT, instr, len, "%px: [%d:%d) optimized NOPs: ", instr, nop,= *next); + add_nop(buf + nop, *next - nop); + DUMP_BYTES(ALT, buf, len, "%px: [%d:%d) optimized NOPs: ", instr, nop, *= next); return true; } =20 @@ -239,32 +253,22 @@ __optimize_nops(u8 *instr, size_t len, struct insn *i= nsn, int *next, int *prev, * "noinline" to cause control flow change and thus invalidate I$ and * cause refetch after modification. */ -static void __init_or_module noinline optimize_nops(u8 *instr, size_t len) +static void __init_or_module noinline optimize_nops(const u8 * const instr= , u8 *buf, size_t len) { int prev, target =3D 0; =20 for (int next, i =3D 0; i < len; i =3D next) { struct insn insn; =20 - if (insn_decode_kernel(&insn, &instr[i])) + if (insn_decode_kernel(&insn, &buf[i])) return; =20 next =3D i + insn.length; =20 - __optimize_nops(instr, len, &insn, &next, &prev, &target); + __optimize_nops(instr, buf, len, &insn, &next, &prev, &target); } } =20 -static void __init_or_module noinline optimize_nops_inplace(u8 *instr, siz= e_t len) -{ - unsigned long flags; - - local_irq_save(flags); - optimize_nops(instr, len); - sync_core(); - local_irq_restore(flags); -} - /* * In this context, "source" is where the instructions are placed in the * section .altinstr_replacement, for example during kernel build by the @@ -335,11 +339,11 @@ bool need_reloc(unsigned long offset, u8 *src, size_t= src_len) return (target < src || target > src + src_len); } =20 -void apply_relocation(u8 *buf, size_t len, u8 *dest, u8 *src, size_t src_l= en) +void apply_relocation(u8 *buf, const u8 * const instr, size_t instrlen, u8= *repl, size_t repl_len) { int prev, target =3D 0; =20 - for (int next, i =3D 0; i < len; i =3D next) { + for (int next, i =3D 0; i < instrlen; i =3D next) { struct insn insn; =20 if (WARN_ON_ONCE(insn_decode_kernel(&insn, &buf[i]))) @@ -347,7 +351,7 @@ void apply_relocation(u8 *buf, size_t len, u8 *dest, u8= *src, size_t src_len) =20 next =3D i + insn.length; =20 - if (__optimize_nops(buf, len, &insn, &next, &prev, &target)) + if (__optimize_nops(instr, buf, instrlen, &insn, &next, &prev, &target)) continue; =20 switch (insn.opcode.bytes[0]) { @@ -361,10 +365,10 @@ void apply_relocation(u8 *buf, size_t len, u8 *dest, = u8 *src, size_t src_len) case JMP8_INSN_OPCODE: case JMP32_INSN_OPCODE: case CALL_INSN_OPCODE: - if (need_reloc(next + insn.immediate.value, src, src_len)) { + if (need_reloc(next + insn.immediate.value, repl, repl_len)) { apply_reloc(insn.immediate.nbytes, buf + i + insn_offset_immediate(&insn), - src - dest); + repl - instr); } =20 /* @@ -372,7 +376,7 @@ void apply_relocation(u8 *buf, size_t len, u8 *dest, u8= *src, size_t src_len) */ if (insn.opcode.bytes[0] =3D=3D JMP32_INSN_OPCODE) { s32 imm =3D insn.immediate.value; - imm +=3D src - dest; + imm +=3D repl - instr; imm +=3D JMP32_INSN_SIZE - JMP8_INSN_SIZE; if ((imm >> 31) =3D=3D (imm >> 7)) { buf[i+0] =3D JMP8_INSN_OPCODE; @@ -385,10 +389,10 @@ void apply_relocation(u8 *buf, size_t len, u8 *dest, = u8 *src, size_t src_len) } =20 if (insn_rip_relative(&insn)) { - if (need_reloc(next + insn.displacement.value, src, src_len)) { + if (need_reloc(next + insn.displacement.value, repl, repl_len)) { apply_reloc(insn.displacement.nbytes, buf + i + insn_offset_displacement(&insn), - src - dest); + repl - instr); } } } @@ -504,7 +508,9 @@ void __init_or_module noinline apply_alternatives(struc= t alt_instr *start, * patch if feature is *NOT* present. */ if (!boot_cpu_has(a->cpuid) =3D=3D !(a->flags & ALT_FLAG_NOT)) { - optimize_nops_inplace(instr, a->instrlen); + memcpy(insn_buff, instr, a->instrlen); + optimize_nops(instr, insn_buff, a->instrlen); + text_poke_early(instr, insn_buff, a->instrlen); continue; } =20 @@ -526,7 +532,7 @@ void __init_or_module noinline apply_alternatives(struc= t alt_instr *start, for (; insn_buff_sz < a->instrlen; insn_buff_sz++) insn_buff[insn_buff_sz] =3D 0x90; =20 - apply_relocation(insn_buff, a->instrlen, instr, replacement, a->replacem= entlen); + apply_relocation(insn_buff, instr, a->instrlen, replacement, a->replacem= entlen); =20 DUMP_BYTES(ALT, instr, a->instrlen, "%px: old_insn: ", instr); DUMP_BYTES(ALT, replacement, a->replacementlen, "%px: rpl_insn: ", rep= lacement); @@ -761,7 +767,7 @@ void __init_or_module noinline apply_retpolines(s32 *st= art, s32 *end) =20 len =3D patch_retpoline(addr, &insn, bytes); if (len =3D=3D insn.length) { - optimize_nops(bytes, len); + optimize_nops(addr, bytes, len); DUMP_BYTES(RETPOLINE, ((u8*)addr), len, "%px: orig: ", addr); DUMP_BYTES(RETPOLINE, ((u8*)bytes), len, "%px: repl: ", addr); text_poke_early(addr, bytes, len); diff --git a/arch/x86/kernel/callthunks.c b/arch/x86/kernel/callthunks.c index 30335182b6b0..771d95484453 100644 --- a/arch/x86/kernel/callthunks.c +++ b/arch/x86/kernel/callthunks.c @@ -185,8 +185,7 @@ static void *patch_dest(void *dest, bool direct) u8 *pad =3D dest - tsize; =20 memcpy(insn_buff, skl_call_thunk_template, tsize); - apply_relocation(insn_buff, tsize, pad, - skl_call_thunk_template, tsize); + apply_relocation(insn_buff, pad, tsize, skl_call_thunk_template, tsize); =20 /* Already patched? */ if (!bcmp(pad, insn_buff, tsize)) @@ -308,8 +307,7 @@ static bool is_callthunk(void *addr) pad =3D (void *)(dest - tmpl_size); =20 memcpy(insn_buff, skl_call_thunk_template, tmpl_size); - apply_relocation(insn_buff, tmpl_size, pad, - skl_call_thunk_template, tmpl_size); + apply_relocation(insn_buff, pad, tmpl_size, skl_call_thunk_template, tmpl= _size); =20 return !bcmp(pad, insn_buff, tmpl_size); } @@ -327,8 +325,7 @@ int x86_call_depth_emit_accounting(u8 **pprog, void *fu= nc) return 0; =20 memcpy(insn_buff, skl_call_thunk_template, tmpl_size); - apply_relocation(insn_buff, tmpl_size, *pprog, - skl_call_thunk_template, tmpl_size); + apply_relocation(insn_buff, *pprog, tmpl_size, skl_call_thunk_template, t= mpl_size); =20 memcpy(*pprog, insn_buff, tmpl_size); *pprog +=3D tmpl_size; --=20 2.43.0 From nobody Fri Dec 19 15:49:48 2025 Received: from mail.alien8.de (mail.alien8.de [65.109.113.108]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5B3C41DDCE for ; Sun, 31 Mar 2024 18:01:09 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=65.109.113.108 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711908071; cv=none; b=eh6E2aUccOJX3OnrO9aARnhStkeyzEqk/Tpa+PBwtKORnyNp9DpqsyZMm3fv1B7c94EeEoob10mtjd8TncaR7WwfsKqeroi9R59FcjeFx6AZ3tlwf5VrIETrbrBnA7yRdtMI9EK1dcl2TxJgdBk2B5q7LzkCQ0U+qomBdoR30ao= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711908071; c=relaxed/simple; bh=u6yw/6pNiS6XlXi0fw+StuO0q4IRQWyeHy0X63Dcn34=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=CFpHosd2XqJJTpyYg47+vL89gmOezQoPyu9h0DKKJuPr3Sx4GjCO1c1rERMQyqpuEyRAVDD2QH/Q/FDX7DQM8jpzBnSDA4J3hIB8+04uEqTPxaLpEgrI7/2B0nikGDaOtje43mcdJcvutaP1Ne7Yl0c7rGKXH+Rf2vKUwCC4pQY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=alien8.de; spf=pass smtp.mailfrom=alien8.de; dkim=fail (4096-bit key) header.d=alien8.de header.i=@alien8.de header.b=fYWTb4OR reason="signature verification failed"; arc=none smtp.client-ip=65.109.113.108 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=alien8.de Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=alien8.de Authentication-Results: smtp.subspace.kernel.org; dkim=fail reason="signature verification failed" (4096-bit key) header.d=alien8.de header.i=@alien8.de header.b="fYWTb4OR" Received: from localhost (localhost.localdomain [127.0.0.1]) by mail.alien8.de (SuperMail on ZX Spectrum 128k) with ESMTP id 87C0C40E02A7; Sun, 31 Mar 2024 18:01:07 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at mail.alien8.de Authentication-Results: mail.alien8.de (amavisd-new); dkim=fail (4096-bit key) reason="fail (body has been altered)" header.d=alien8.de Received: from mail.alien8.de ([127.0.0.1]) by localhost (mail.alien8.de [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id baCB6NFuj8Nw; Sun, 31 Mar 2024 18:01:05 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=alien8.de; s=alien8; t=1711908063; bh=/m+xg3eCZn7BMsmZutP2oDLl0SFWMQPzolR2d5YsX+8=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=fYWTb4ORgog9/JdWoCnaMBIKEb3AgBwPy9jmp/TKyjiC9NyBfs9nCGFhXzvB6ZK8g hbBXE8JFqxS/CkbjF/4YFwcOP7eGOpNxx5mmZV6Q2fKgyJbg6w2aOkqUW/U7qrRab/ 46X1byWJRAGvlRK0y2rrdq0crfR3H/JDj6OU7jwt+o0QBIl6HvgFEEwGGCNo6EO2Nr pqBJYPEu0YNJdLFgCvxmQXUVyen65HWMxx1ba6xIUzZgbLCzy/hqMddlZoFx9jS+/T +XUXWc5j70PFrBNG/lrBDD3etOGcvR2CuU7fer5oP3G+kR5W6/++WVOlMxtB1Qe+I1 asxk2gMTW81Ya0D+cMv1ZQb7VBPsnXwSt1C5IA1FblqXuIOapZ954sbrnVJ5V3utuX DbTZy88tm2J14/6SsxBRy8ACDNGPr0JXdYYWh/SZYUFTNpeeb35VmR3+s+syWpaaTp oUy5cSwTSo0JjR9dLvbTPLwCQgsaGzjS2BjB9OHV/Eck90+FPXUQnkvm94xa5qzAIy g1ckci0ZevmbjhQyDwQqN4NIVOezIHIjGk14RGO9o96we098WVHylygI+LXP8FUFgv kKPVbtiiJJrg0mU50PWn9VPWVfITnPll85kt/w7laTmc3CyWVbg3KjbgGvaTp354QV NKZ2OkMhCdQzmPs7O/hQej8E= Received: from zn.tnic (p5de8ecf7.dip0.t-ipconnect.de [93.232.236.247]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange ECDHE (P-256) server-signature ECDSA (P-256) server-digest SHA256) (No client certificate requested) by mail.alien8.de (SuperMail on ZX Spectrum 128k) with ESMTPSA id 6E76340E0028; Sun, 31 Mar 2024 18:01:01 +0000 (UTC) From: Borislav Petkov To: X86 ML Cc: LKML Subject: [PATCH v2 2/4] x86/alternatives: Get rid of __optimize_nops() Date: Sun, 31 Mar 2024 20:00:40 +0200 Message-ID: <20240331180042.13933-3-bp@alien8.de> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240331180042.13933-1-bp@alien8.de> References: <20240331180042.13933-1-bp@alien8.de> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: "Borislav Petkov (AMD)" There's no need to carve out bits of the NOP optimization functionality and look at JMP opcodes - simply do one more NOPs optimization pass at the end of patching. A lot simpler code. Signed-off-by: Borislav Petkov (AMD) --- arch/x86/kernel/alternative.c | 59 ++++++++++------------------------- 1 file changed, 16 insertions(+), 43 deletions(-) diff --git a/arch/x86/kernel/alternative.c b/arch/x86/kernel/alternative.c index ec94f1359c00..4b3378c71518 100644 --- a/arch/x86/kernel/alternative.c +++ b/arch/x86/kernel/alternative.c @@ -216,47 +216,12 @@ static int skip_nops(u8 *buf, int offset, int len) return offset; } =20 -/* - * Optimize a sequence of NOPs, possibly preceded by an unconditional jump - * to the end of the NOP sequence into a single NOP. - */ -static bool -__optimize_nops(const u8 * const instr, u8 *buf, size_t len, struct insn *= insn, int *next, int *prev, int *target) -{ - int i =3D *next - insn->length; - - switch (insn->opcode.bytes[0]) { - case JMP8_INSN_OPCODE: - case JMP32_INSN_OPCODE: - *prev =3D i; - *target =3D *next + insn->immediate.value; - return false; - } - - if (insn_is_nop(insn)) { - int nop =3D i; - - *next =3D skip_nops(buf, *next, len); - if (*target && *next =3D=3D *target) - nop =3D *prev; - - add_nop(buf + nop, *next - nop); - DUMP_BYTES(ALT, buf, len, "%px: [%d:%d) optimized NOPs: ", instr, nop, *= next); - return true; - } - - *target =3D 0; - return false; -} - /* * "noinline" to cause control flow change and thus invalidate I$ and * cause refetch after modification. */ -static void __init_or_module noinline optimize_nops(const u8 * const instr= , u8 *buf, size_t len) +static void noinline optimize_nops(const u8 * const instr, u8 *buf, size_t= len) { - int prev, target =3D 0; - for (int next, i =3D 0; i < len; i =3D next) { struct insn insn; =20 @@ -265,7 +230,14 @@ static void __init_or_module noinline optimize_nops(co= nst u8 * const instr, u8 * =20 next =3D i + insn.length; =20 - __optimize_nops(instr, buf, len, &insn, &next, &prev, &target); + if (insn_is_nop(&insn)) { + int nop =3D i; + + next =3D skip_nops(buf, next, len); + + add_nop(buf + nop, next - nop); + DUMP_BYTES(ALT, buf, len, "%px: [%d:%d) optimized NOPs: ", instr, nop, = next); + } } } =20 @@ -339,10 +311,8 @@ bool need_reloc(unsigned long offset, u8 *src, size_t = src_len) return (target < src || target > src + src_len); } =20 -void apply_relocation(u8 *buf, const u8 * const instr, size_t instrlen, u8= *repl, size_t repl_len) +static void __apply_relocation(u8 *buf, const u8 * const instr, size_t ins= trlen, u8 *repl, size_t repl_len) { - int prev, target =3D 0; - for (int next, i =3D 0; i < instrlen; i =3D next) { struct insn insn; =20 @@ -351,9 +321,6 @@ void apply_relocation(u8 *buf, const u8 * const instr, = size_t instrlen, u8 *repl =20 next =3D i + insn.length; =20 - if (__optimize_nops(instr, buf, instrlen, &insn, &next, &prev, &target)) - continue; - switch (insn.opcode.bytes[0]) { case 0x0f: if (insn.opcode.bytes[1] < 0x80 || @@ -398,6 +365,12 @@ void apply_relocation(u8 *buf, const u8 * const instr,= size_t instrlen, u8 *repl } } =20 +void apply_relocation(u8 *buf, const u8 * const instr, size_t instrlen, u8= *repl, size_t repl_len) +{ + __apply_relocation(buf, instr, instrlen, repl, repl_len); + optimize_nops(instr, buf, repl_len); +} + /* Low-level backend functions usable from alternative code replacements. = */ DEFINE_ASM_FUNC(nop_func, "", .entry.text); EXPORT_SYMBOL_GPL(nop_func); --=20 2.43.0 From nobody Fri Dec 19 15:49:48 2025 Received: from mail.alien8.de (mail.alien8.de [65.109.113.108]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 21AA5146010 for ; Sun, 31 Mar 2024 18:01:11 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=65.109.113.108 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711908073; cv=none; b=a8mUsc+/xHOBOSOcJltE7SG/tJ/rP8khklktFDqrP+ZntLnSz/XnPiliECmCouZkmS+OSIULI8DMkaj+H3+ejJTu0DiDnU1GsXQanduDqpSDRzMMYuLGhGrBFzQvqWdLe3mcAv8svCy4BGJhYvO8fNfVBwA5DIh96W+Cgpa1sSM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711908073; c=relaxed/simple; bh=IhzTBjgcSUxPoooKAQMtSfyuHxZVHsNUvUQ5FyB2k/w=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=cQagxF0d4jla9LY+rucff19mIRNg35YxMZTYo4Mph36dI0fOy5kp2UgEB9KOd22zs25YCtFKHpmNR8ntZAyvTmmcuf4ySYr9UH1DHk/Agy8p6hx5r8Yq0KYvaipnf1FcgB02AKDjBXUYXljnUX5JblH1trO9AJR2F4OtWwk/m4E= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=alien8.de; spf=pass smtp.mailfrom=alien8.de; dkim=fail (4096-bit key) header.d=alien8.de header.i=@alien8.de header.b=K9qn3z0c reason="signature verification failed"; arc=none smtp.client-ip=65.109.113.108 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=alien8.de Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=alien8.de Authentication-Results: smtp.subspace.kernel.org; dkim=fail reason="signature verification failed" (4096-bit key) header.d=alien8.de header.i=@alien8.de header.b="K9qn3z0c" Received: from localhost (localhost.localdomain [127.0.0.1]) by mail.alien8.de (SuperMail on ZX Spectrum 128k) with ESMTP id 58BE540E02A5; Sun, 31 Mar 2024 18:01:10 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at mail.alien8.de Authentication-Results: mail.alien8.de (amavisd-new); dkim=fail (4096-bit key) reason="fail (body has been altered)" header.d=alien8.de Received: from mail.alien8.de ([127.0.0.1]) by localhost (mail.alien8.de [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id JgU7a60AfwIv; Sun, 31 Mar 2024 18:01:07 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=alien8.de; s=alien8; t=1711908067; bh=ws0VdAWkXbMIBACYGflMkkVS5RnmQ4Ymb+wW5pLwxlI=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=K9qn3z0cErgG2fQK0eRAsEk/xh+rtwrfJum4UAEVoGvPQmGtfCgUUz6O2GazczbqA t/GFxksgI+UbutZr/56EglUnmYEnleIv5qnT8EKxSXIatIB4iIrB6dou80ewPlJdkv ZBSW621KvXFBF0iXLRn0KetG7Zv5e1kCmSM5xS00uo9P+whpnv9oDp89QWd2n6MCqI kQFAoeOg8LW42lWka5135uc+IXID8xFibgSA/zSmkU9q+yJkALaIkKcl/WkLoMMeuG Xi89RQ9ew7nQfwA4sIu3i0tKSwElv2PkwdX7QXVxtLOBeGQJiVDeQCJMkJ7MF646RU Dv4Fu0K7yAac6aATtwWdnNZbPoZ8SFqObQKAWyAvHCEGSf9f1rwrj1e05qJRjunJXS gBO1nTJ8l5HFUVyM7YgcMWOLilQR4riiWAF2tIhHQyk8bY2l/kaJtBTELYnBb4bY36 HoOD8EWkcIDGmKikUPpcHQgRx1goIClwdZuP0TLAyQM70k6LLGQnXBUAsy+7cR2/Uk 6t8cmG2U+Drk2lJl4/FnNLLFHAZUZ9x3qwtW4EtBiV1IoEBvpAUIfh3JvrcRXY4Ivq 7eSq0Zo0/Ty0uEIzyJAjiEECSupZ1ag40lpe0uNR2LFL8LLDojwHnf0I60QgaZlWWI heKv3yZ2It2iF+9lsSqWtSns= Received: from zn.tnic (p5de8ecf7.dip0.t-ipconnect.de [93.232.236.247]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange ECDHE (P-256) server-signature ECDSA (P-256) server-digest SHA256) (No client certificate requested) by mail.alien8.de (SuperMail on ZX Spectrum 128k) with ESMTPSA id 220E140E0202; Sun, 31 Mar 2024 18:01:05 +0000 (UTC) From: Borislav Petkov To: X86 ML Cc: LKML Subject: [PATCH v2 3/4] x86/alternatives: Optimize optimize_nops() Date: Sun, 31 Mar 2024 20:00:41 +0200 Message-ID: <20240331180042.13933-4-bp@alien8.de> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240331180042.13933-1-bp@alien8.de> References: <20240331180042.13933-1-bp@alien8.de> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: "Borislav Petkov (AMD)" Return early if NOPs have already been optimized. Signed-off-by: Borislav Petkov (AMD) --- arch/x86/kernel/alternative.c | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/arch/x86/kernel/alternative.c b/arch/x86/kernel/alternative.c index 4b3378c71518..67dd7c371d28 100644 --- a/arch/x86/kernel/alternative.c +++ b/arch/x86/kernel/alternative.c @@ -233,6 +233,10 @@ static void noinline optimize_nops(const u8 * const in= str, u8 *buf, size_t len) if (insn_is_nop(&insn)) { int nop =3D i; =20 + /* Has the NOP already been optimized? */ + if (i + insn.length =3D=3D len) + return; + next =3D skip_nops(buf, next, len); =20 add_nop(buf + nop, next - nop); --=20 2.43.0 From nobody Fri Dec 19 15:49:48 2025 Received: from mail.alien8.de (mail.alien8.de [65.109.113.108]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E6320145B24 for ; Sun, 31 Mar 2024 18:01:15 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=65.109.113.108 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711908077; cv=none; b=OpFAxWAXIBQ2XvecfuPG4cQH5mXIbw3J5v7RePx19sbf9DAkaSBzYBjJQap1EzO7M/lfH8WlC7ryzpqoC9/OAzo59IHoJ1V/w9UbhgfQtRu9eufWTPYdfYzDR48CT9WsOFXbT14A8cne5tOHh2oLlGpMKqAQfcRpJKjR5YU54D8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711908077; c=relaxed/simple; bh=wW/hM2suVHdvMVTfvZ04ENlpM59FopqQ2RH5VOfXQK8=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=shRjcb6FTcnl8DjXZHNb+VoO66+90HoED8xvp0LxiaJx7aoFsNsk3B2Id7NseEnUfwZwHFwL0fIfSCo/d1tRn4PxMKZyv6g/XTXn1QyCcwgGFJ4/GWuZOKdUU4FyJNViFzqr3ZCzZ1XHY2WTgsnbW6DqXlU9ie+imaItHjOmAAA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=alien8.de; spf=pass smtp.mailfrom=alien8.de; dkim=fail (4096-bit key) header.d=alien8.de header.i=@alien8.de header.b=VGbgEjZY reason="signature verification failed"; arc=none smtp.client-ip=65.109.113.108 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=alien8.de Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=alien8.de Authentication-Results: smtp.subspace.kernel.org; dkim=fail reason="signature verification failed" (4096-bit key) header.d=alien8.de header.i=@alien8.de header.b="VGbgEjZY" Received: from localhost (localhost.localdomain [127.0.0.1]) by mail.alien8.de (SuperMail on ZX Spectrum 128k) with ESMTP id 0ED4840E0202; Sun, 31 Mar 2024 18:01:14 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at mail.alien8.de Authentication-Results: mail.alien8.de (amavisd-new); dkim=fail (4096-bit key) reason="fail (body has been altered)" header.d=alien8.de Received: from mail.alien8.de ([127.0.0.1]) by localhost (mail.alien8.de [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id BRxOl6_4aL4i; Sun, 31 Mar 2024 18:01:11 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=alien8.de; s=alien8; t=1711908070; bh=kBucdMbXIcOk6bcPF3yfiq/poL2A4vPTnxg3GqSRyTs=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=VGbgEjZYIoRzaxGkxRcPoj976yA3EWtUzim+y8+ezm/awOsv3BO6GKEgjMMLCwX5Y ZA2looMQGCsOuzEn/xCGpaa6PHmS2YZSswIIu9C3fwbpe0rE5w0Jv6ZsvmUHpLZyWH xqnee0h1R2hIzUaS4wY5lW5jFABO3CKSNhTKbFyYwhi5GklMaE+s2aZQGY21zFWbVZ AsSpWfOFQWu1ACOqu4x4b5LByglxX3fxeqSisAms96ef+VHbpvTh9V+eMqoSXsPOBn tJkOH3B/k6X50QItHaTC+z0KVUomn3D3lGUbrTkfvCgAT35aFrvY+ftOg88qBh6L/V MOWLft+ta5XSao0kQA2LnQ27jxdpYiL5TBC7YfMSQdftGGpoA4mIgjPFRw0ydiwYK8 SOEqILduf2D9so2wJWnFnr9r30eZrTTDSe9Lo+XnTYeAx2AXqtlkF/KXXc2Eft8wl5 1061D5rEEhNQZO3otRXNknLdLfD+PzHtaHIixcxzTyTaxZ2qtiDom1bpPm3+3+Ovkm pjrxLEST4Ng0akL7vTnJkCaOV2q/zHmGJCy8E8PguYvK0BkNlOABDWHEltF6h6Z/vc a9z+JQ8KBHIGbKsSxBHXfol3t6oImvjVRkgrLDoi5zwP0R0IJRc9Mj15IPNxtaExqP 6fjCd/oAGaHNlOwqVBk1NKoc= Received: from zn.tnic (p5de8ecf7.dip0.t-ipconnect.de [93.232.236.247]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange ECDHE (P-256) server-signature ECDSA (P-256) server-digest SHA256) (No client certificate requested) by mail.alien8.de (SuperMail on ZX Spectrum 128k) with ESMTPSA id CAF6440E0028; Sun, 31 Mar 2024 18:01:08 +0000 (UTC) From: Borislav Petkov To: X86 ML Cc: LKML Subject: [PATCH v2 4/4] x86/alternatives: Sort local vars in apply_alternatives() Date: Sun, 31 Mar 2024 20:00:42 +0200 Message-ID: <20240331180042.13933-5-bp@alien8.de> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240331180042.13933-1-bp@alien8.de> References: <20240331180042.13933-1-bp@alien8.de> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: "Borislav Petkov (AMD)" In a reverse x-mas tree. No functional changes. Signed-off-by: Borislav Petkov (AMD) --- arch/x86/kernel/alternative.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/arch/x86/kernel/alternative.c b/arch/x86/kernel/alternative.c index 67dd7c371d28..7555c15b7183 100644 --- a/arch/x86/kernel/alternative.c +++ b/arch/x86/kernel/alternative.c @@ -445,9 +445,9 @@ static int alt_replace_call(u8 *instr, u8 *insn_buff, s= truct alt_instr *a) void __init_or_module noinline apply_alternatives(struct alt_instr *start, struct alt_instr *end) { - struct alt_instr *a; - u8 *instr, *replacement; u8 insn_buff[MAX_PATCH_LEN]; + u8 *instr, *replacement; + struct alt_instr *a; =20 DPRINTK(ALT, "alt table %px, -> %px", start, end); =20 --=20 2.43.0