From: Sohil Mehta <sohil.mehta@intel.com>
For patching, the kernel initializes a temporary mm area in the lower
half of the address range. See commit 4fc19708b165 ("x86/alternatives:
Initialize temporary mm for patching").
Disable LASS enforcement during patching to avoid triggering a #GP
fault.
The objtool warns due to a call to a non-allowed function that exists
outside of the stac/clac guard, or references to any function with a
dynamic function pointer inside the guard. See the Objtool warnings
section #9 in the document tools/objtool/Documentation/objtool.txt.
Considering that patching is usually small, replace the memcpy() and
memset() functions in the text poking functions with their open coded
versions.
Signed-off-by: Sohil Mehta <sohil.mehta@intel.com>
Signed-off-by: Alexander Shishkin <alexander.shishkin@linux.intel.com>
Signed-off-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com>
---
arch/x86/include/asm/smap.h | 33 +++++++++++++++++++++++++++++++--
arch/x86/kernel/alternative.c | 28 ++++++++++++++++++++++++++--
2 files changed, 57 insertions(+), 4 deletions(-)
diff --git a/arch/x86/include/asm/smap.h b/arch/x86/include/asm/smap.h
index 4f84d421d1cf..d0cc24348641 100644
--- a/arch/x86/include/asm/smap.h
+++ b/arch/x86/include/asm/smap.h
@@ -23,18 +23,47 @@
#else /* __ASSEMBLER__ */
+/*
+ * The CLAC/STAC instructions toggle the enforcement of X86_FEATURE_SMAP and
+ * X86_FEATURE_LASS.
+ *
+ * SMAP enforcement is based on the _PAGE_BIT_USER bit in the page tables: the
+ * kernel is not allowed to touch pages with the bit set unless the AC bit is
+ * set.
+ *
+ * LASS enforcement is based on bit 63 of the virtual address. The kernel is
+ * not allowed to touch memory in the lower half of the virtual address space
+ * unless the AC bit is set.
+ *
+ * Use stac()/clac() when accessing userspace (_PAGE_USER) mappings,
+ * regardless of location.
+ *
+ * Use lass_stac()/lass_clac() when accessing kernel mappings (!_PAGE_USER)
+ * in the lower half of the address space.
+ *
+ * Note: a barrier is implicit in alternative().
+ */
+
static __always_inline void clac(void)
{
- /* Note: a barrier is implicit in alternative() */
alternative("", "clac", X86_FEATURE_SMAP);
}
static __always_inline void stac(void)
{
- /* Note: a barrier is implicit in alternative() */
alternative("", "stac", X86_FEATURE_SMAP);
}
+static __always_inline void lass_clac(void)
+{
+ alternative("", "clac", X86_FEATURE_LASS);
+}
+
+static __always_inline void lass_stac(void)
+{
+ alternative("", "stac", X86_FEATURE_LASS);
+}
+
static __always_inline unsigned long smap_save(void)
{
unsigned long flags;
diff --git a/arch/x86/kernel/alternative.c b/arch/x86/kernel/alternative.c
index ea1d984166cd..992ece0e879a 100644
--- a/arch/x86/kernel/alternative.c
+++ b/arch/x86/kernel/alternative.c
@@ -2447,16 +2447,40 @@ void __init_or_module text_poke_early(void *addr, const void *opcode,
__ro_after_init struct mm_struct *text_poke_mm;
__ro_after_init unsigned long text_poke_mm_addr;
+/*
+ * Text poking creates and uses a mapping in the lower half of the
+ * address space. Relax LASS enforcement when accessing the poking
+ * address.
+ */
+
static void text_poke_memcpy(void *dst, const void *src, size_t len)
{
- memcpy(dst, src, len);
+ lass_stac();
+
+ /*
+ * Objtool is picky about what occurs within the STAC/CLAC region
+ * because this code runs with protection disabled. Objtool typically
+ * does not permit function calls in this area.
+ *
+ * Avoid using memcpy() here. Instead, open code it.
+ */
+ asm volatile("rep movsb"
+ : "+D" (dst), "+S" (src), "+c" (len) : : "memory");
+
+ lass_clac();
}
static void text_poke_memset(void *dst, const void *src, size_t len)
{
int c = *(const int *)src;
- memset(dst, c, len);
+ lass_stac();
+
+ /* Open code memset(): make objtool happy. See text_poke_memcpy(). */
+ asm volatile("rep stosb"
+ : "+D" (dst), "+c" (len) : "a" (c) : "memory");
+
+ lass_clac();
}
typedef void text_poke_f(void *dst, const void *src, size_t len);
--
2.47.2On Mon, 7 Jul 2025 11:03:02 +0300
"Kirill A. Shutemov" <kirill.shutemov@linux.intel.com> wrote:
> From: Sohil Mehta <sohil.mehta@intel.com>
>
> For patching, the kernel initializes a temporary mm area in the lower
> half of the address range. See commit 4fc19708b165 ("x86/alternatives:
> Initialize temporary mm for patching").
>
> Disable LASS enforcement during patching to avoid triggering a #GP
> fault.
>
> The objtool warns due to a call to a non-allowed function that exists
> outside of the stac/clac guard, or references to any function with a
> dynamic function pointer inside the guard. See the Objtool warnings
> section #9 in the document tools/objtool/Documentation/objtool.txt.
>
> Considering that patching is usually small, replace the memcpy() and
> memset() functions in the text poking functions with their open coded
> versions.
...
Or just write a byte copy loop in C with (eg) barrier() inside it
to stop gcc converting it to memcpy().
David
On July 28, 2025 12:11:37 PM PDT, David Laight <david.laight.linux@gmail.com> wrote:
>On Mon, 7 Jul 2025 11:03:02 +0300
>"Kirill A. Shutemov" <kirill.shutemov@linux.intel.com> wrote:
>
>> From: Sohil Mehta <sohil.mehta@intel.com>
>>
>> For patching, the kernel initializes a temporary mm area in the lower
>> half of the address range. See commit 4fc19708b165 ("x86/alternatives:
>> Initialize temporary mm for patching").
>>
>> Disable LASS enforcement during patching to avoid triggering a #GP
>> fault.
>>
>> The objtool warns due to a call to a non-allowed function that exists
>> outside of the stac/clac guard, or references to any function with a
>> dynamic function pointer inside the guard. See the Objtool warnings
>> section #9 in the document tools/objtool/Documentation/objtool.txt.
>>
>> Considering that patching is usually small, replace the memcpy() and
>> memset() functions in the text poking functions with their open coded
>> versions.
>...
>
>Or just write a byte copy loop in C with (eg) barrier() inside it
>to stop gcc converting it to memcpy().
>
> David
Great. It's rep movsb without any of the performance.
On Mon, 28 Jul 2025 12:28:33 -0700
"H. Peter Anvin" <hpa@zytor.com> wrote:
> On July 28, 2025 12:11:37 PM PDT, David Laight <david.laight.linux@gmail.com> wrote:
> >On Mon, 7 Jul 2025 11:03:02 +0300
> >"Kirill A. Shutemov" <kirill.shutemov@linux.intel.com> wrote:
> >
> >> From: Sohil Mehta <sohil.mehta@intel.com>
> >>
> >> For patching, the kernel initializes a temporary mm area in the lower
> >> half of the address range. See commit 4fc19708b165 ("x86/alternatives:
> >> Initialize temporary mm for patching").
> >>
> >> Disable LASS enforcement during patching to avoid triggering a #GP
> >> fault.
> >>
> >> The objtool warns due to a call to a non-allowed function that exists
> >> outside of the stac/clac guard, or references to any function with a
> >> dynamic function pointer inside the guard. See the Objtool warnings
> >> section #9 in the document tools/objtool/Documentation/objtool.txt.
> >>
> >> Considering that patching is usually small, replace the memcpy() and
> >> memset() functions in the text poking functions with their open coded
> >> versions.
> >...
> >
> >Or just write a byte copy loop in C with (eg) barrier() inside it
> >to stop gcc converting it to memcpy().
> >
> > David
>
> Great. It's rep movsb without any of the performance.
And without the massive setup overhead that dominates short copies.
Given the rest of the code I'm sure a byte copy loop won't make
any difference to the overall performance.
David
On 7/28/2025 12:38 PM, David Laight wrote:
>>> ...
>>>
>>> Or just write a byte copy loop in C with (eg) barrier() inside it
>>> to stop gcc converting it to memcpy().
>>>
>>> David
>>
>> Great. It's rep movsb without any of the performance.
>
> And without the massive setup overhead that dominates short copies.
> Given the rest of the code I'm sure a byte copy loop won't make
> any difference to the overall performance.
>
Wouldn't it be better to introduce a generic mechanism than something
customized for this scenario?
PeterZ had suggested that inline memcpy could have more usages:
https://lore.kernel.org/lkml/20241029113611.GS14555@noisy.programming.kicks-ass.net/
Is there a concern that the inline versions might get optimized into
standard memcpy/memset calls by GCC? Wouldn't the volatile keyword
prevent that?
static __always_inline void *__inline_memcpy(void *to, const void *from,
size_t len)
{
void *ret = to;
asm volatile("rep movsb"
: "+D" (to), "+S" (from), "+c" (len)
: : "memory");
return ret;
}
static __always_inline void *__inline_memset(void *s, int v, size_t n)
{
void *ret = s;
asm volatile("rep stosb"
: "+D" (s), "+c" (n)
: "a" ((uint8_t)v)
: "memory");
return ret;
}
On 7/7/25 01:03, Kirill A. Shutemov wrote:
> From: Sohil Mehta <sohil.mehta@intel.com>
>
> For patching, the kernel initializes a temporary mm area in the lower
> half of the address range. See commit 4fc19708b165 ("x86/alternatives:
> Initialize temporary mm for patching").
>
> Disable LASS enforcement during patching to avoid triggering a #GP
> fault.
>
> The objtool warns due to a call to a non-allowed function that exists
> outside of the stac/clac guard, or references to any function with a
> dynamic function pointer inside the guard. See the Objtool warnings
> section #9 in the document tools/objtool/Documentation/objtool.txt.
>
> Considering that patching is usually small, replace the memcpy() and
> memset() functions in the text poking functions with their open coded
> versions.
>
> Signed-off-by: Sohil Mehta <sohil.mehta@intel.com>
> Signed-off-by: Alexander Shishkin <alexander.shishkin@linux.intel.com>
> Signed-off-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com>
> ---
> arch/x86/include/asm/smap.h | 33 +++++++++++++++++++++++++++++++--
> arch/x86/kernel/alternative.c | 28 ++++++++++++++++++++++++++--
> 2 files changed, 57 insertions(+), 4 deletions(-)
>
> diff --git a/arch/x86/include/asm/smap.h b/arch/x86/include/asm/smap.h
> index 4f84d421d1cf..d0cc24348641 100644
> --- a/arch/x86/include/asm/smap.h
> +++ b/arch/x86/include/asm/smap.h
> @@ -23,18 +23,47 @@
>
> #else /* __ASSEMBLER__ */
>
> +/*
> + * The CLAC/STAC instructions toggle the enforcement of X86_FEATURE_SMAP and
> + * X86_FEATURE_LASS.
> + *
> + * SMAP enforcement is based on the _PAGE_BIT_USER bit in the page tables: the
> + * kernel is not allowed to touch pages with the bit set unless the AC bit is
> + * set.
> + *
> + * LASS enforcement is based on bit 63 of the virtual address. The kernel is
> + * not allowed to touch memory in the lower half of the virtual address space
> + * unless the AC bit is set.
> + *
> + * Use stac()/clac() when accessing userspace (_PAGE_USER) mappings,
> + * regardless of location.
> + *
> + * Use lass_stac()/lass_clac() when accessing kernel mappings (!_PAGE_USER)
> + * in the lower half of the address space.
> + *
> + * Note: a barrier is implicit in alternative().
> + */
> +
> static __always_inline void clac(void)
> {
> - /* Note: a barrier is implicit in alternative() */
> alternative("", "clac", X86_FEATURE_SMAP);
> }
>
> static __always_inline void stac(void)
> {
> - /* Note: a barrier is implicit in alternative() */
> alternative("", "stac", X86_FEATURE_SMAP);
> }
>
> +static __always_inline void lass_clac(void)
> +{
> + alternative("", "clac", X86_FEATURE_LASS);
> +}
> +
> +static __always_inline void lass_stac(void)
> +{
> + alternative("", "stac", X86_FEATURE_LASS);
> +}
Could we please move the comments about lass_*() closer to the LASS
functions?
> static __always_inline unsigned long smap_save(void)
> {
> unsigned long flags;
> diff --git a/arch/x86/kernel/alternative.c b/arch/x86/kernel/alternative.c
> index ea1d984166cd..992ece0e879a 100644
> --- a/arch/x86/kernel/alternative.c
> +++ b/arch/x86/kernel/alternative.c
> @@ -2447,16 +2447,40 @@ void __init_or_module text_poke_early(void *addr, const void *opcode,
> __ro_after_init struct mm_struct *text_poke_mm;
> __ro_after_init unsigned long text_poke_mm_addr;
>
> +/*
> + * Text poking creates and uses a mapping in the lower half of the
> + * address space. Relax LASS enforcement when accessing the poking
> + * address.
> + */
> +
> static void text_poke_memcpy(void *dst, const void *src, size_t len)
> {
> - memcpy(dst, src, len);
> + lass_stac();
> +
> + /*
> + * Objtool is picky about what occurs within the STAC/CLAC region
> + * because this code runs with protection disabled. Objtool typically
> + * does not permit function calls in this area.
> + *
> + * Avoid using memcpy() here. Instead, open code it.
> + */
> + asm volatile("rep movsb"
> + : "+D" (dst), "+S" (src), "+c" (len) : : "memory");
> +
> + lass_clac();
> }
This didn't turn out great. At the _very_ least, we could have a:
inline_memcpy_i_really_mean_it()
with the rep mov. Or even a #define if we were super paranoid the
compiler is out to get us.
But _actually_ open-coding inline assembly is far too ugly to live.
We can also be a bit more compact about the comments:
/*
* objtool enforces a strict policy of "no function calls within
* AC=1 regions". Adhere to the policy by doing a memcpy() that
* will never result in a function call.
*/
On 7/9/2025 9:58 AM, Dave Hansen wrote:
>> + * Avoid using memcpy() here. Instead, open code it.
>> + */
>> + asm volatile("rep movsb"
>> + : "+D" (dst), "+S" (src), "+c" (len) : : "memory");
>> +
>> + lass_clac();
>> }
>
> This didn't turn out great. At the _very_ least, we could have a:
>
> inline_memcpy_i_really_mean_it()
>
It looks like we should go back to __inline_memcpy()/_memset()
implementation that PeterZ had initially proposed. It seems to fit all
the requirements, right? Patch attached.
https://lore.kernel.org/lkml/20241028160917.1380714-3-alexander.shishkin@linux.intel.com/
> with the rep mov. Or even a #define if we were super paranoid the
> compiler is out to get us.
>
> But _actually_ open-coding inline assembly is far too ugly to live.
> From eb3b45b377df90d3b367e2b3fddfff1a72624a4e Mon Sep 17 00:00:00 2001
From: Peter Zijlstra <peterz@infradead.org>
Date: Mon, 28 Oct 2024 18:07:50 +0200
Subject: [PATCH] x86/asm: Introduce inline memcpy and memset
Provide inline memcpy and memset functions that can be used instead of
the GCC builtins whenever necessary.
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Signed-off-by: Sohil Mehta <sohil.mehta@intel.com>
---
arch/x86/include/asm/string.h | 26 ++++++++++++++++++++++++++
1 file changed, 26 insertions(+)
diff --git a/arch/x86/include/asm/string.h b/arch/x86/include/asm/string.h
index c3c2c1914d65..9cb5aae7fba9 100644
--- a/arch/x86/include/asm/string.h
+++ b/arch/x86/include/asm/string.h
@@ -1,6 +1,32 @@
/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef _ASM_X86_STRING_H
+#define _ASM_X86_STRING_H
+
#ifdef CONFIG_X86_32
# include <asm/string_32.h>
#else
# include <asm/string_64.h>
#endif
+
+static __always_inline void *__inline_memcpy(void *to, const void *from, size_t len)
+{
+ void *ret = to;
+
+ asm volatile("rep movsb"
+ : "+D" (to), "+S" (from), "+c" (len)
+ : : "memory");
+ return ret;
+}
+
+static __always_inline void *__inline_memset(void *s, int v, size_t n)
+{
+ void *ret = s;
+
+ asm volatile("rep stosb"
+ : "+D" (s), "+c" (n)
+ : "a" ((uint8_t)v)
+ : "memory");
+ return ret;
+}
+
+#endif /* _ASM_X86_STRING_H */
--
2.43.0
On 7/7/2025 1:03 AM, Kirill A. Shutemov wrote:
> From: Sohil Mehta <sohil.mehta@intel.com>
>
> For patching, the kernel initializes a temporary mm area in the lower
> half of the address range. See commit 4fc19708b165 ("x86/alternatives:
> Initialize temporary mm for patching").
>
> Disable LASS enforcement during patching to avoid triggering a #GP
> fault.
>
> The objtool warns due to a call to a non-allowed function that exists
> outside of the stac/clac guard, or references to any function with a
> dynamic function pointer inside the guard. See the Objtool warnings
> section #9 in the document tools/objtool/Documentation/objtool.txt.
>
> Considering that patching is usually small, replace the memcpy() and
> memset() functions in the text poking functions with their open coded
> versions.
>
> Signed-off-by: Sohil Mehta <sohil.mehta@intel.com>
> Signed-off-by: Alexander Shishkin <alexander.shishkin@linux.intel.com>
> Signed-off-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com>
Kirill, it might be worth adding your co-developed-by tag. The patch has
more changes than I can claim credit for.
> ---
> arch/x86/include/asm/smap.h | 33 +++++++++++++++++++++++++++++++--
> arch/x86/kernel/alternative.c | 28 ++++++++++++++++++++++++++--
> 2 files changed, 57 insertions(+), 4 deletions(-)
>
On Tue, Jul 08, 2025 at 06:08:18PM -0700, Sohil Mehta wrote:
> On 7/7/2025 1:03 AM, Kirill A. Shutemov wrote:
> > From: Sohil Mehta <sohil.mehta@intel.com>
> >
> > For patching, the kernel initializes a temporary mm area in the lower
> > half of the address range. See commit 4fc19708b165 ("x86/alternatives:
> > Initialize temporary mm for patching").
> >
> > Disable LASS enforcement during patching to avoid triggering a #GP
> > fault.
> >
> > The objtool warns due to a call to a non-allowed function that exists
> > outside of the stac/clac guard, or references to any function with a
> > dynamic function pointer inside the guard. See the Objtool warnings
> > section #9 in the document tools/objtool/Documentation/objtool.txt.
> >
> > Considering that patching is usually small, replace the memcpy() and
> > memset() functions in the text poking functions with their open coded
> > versions.
> >
> > Signed-off-by: Sohil Mehta <sohil.mehta@intel.com>
> > Signed-off-by: Alexander Shishkin <alexander.shishkin@linux.intel.com>
> > Signed-off-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com>
>
> Kirill, it might be worth adding your co-developed-by tag. The patch has
> more changes than I can claim credit for.
Okay. Will do.
--
Kiryl Shutsemau / Kirill A. Shutemov
© 2016 - 2025 Red Hat, Inc.