Hi,
This series implements Shadow Stacks for userspace using x86's Control-flow
Enforcement Technology (CET). CET consists of two related security
features: shadow stacks and indirect branch tracking. This series
implements just the shadow stack part of this feature, and just for
userspace.
The main use case for shadow stack is providing protection against return
oriented programming attacks. It works by maintaining a secondary (shadow)
stack using a special memory type that has protections against
modification. When executing a CALL instruction, the processor pushes the
return address to both the normal stack and to the special permission
shadow stack. Upon RET, the processor pops the shadow stack copy and
compares it to the normal stack copy. For more details, see the
coverletter from v1 [0].
Shadow Stack was rejected by Linus for 6.4 [1][2]. This is a new version
that addresses his concerns. In the months since the series was queued in
tip, there were also some non-critical things that turned up, that I was
planning do as fast follow ups. Since we are doing a re-spin, I thought to
just include them in the initial series. (see 3 and 4) Also, the whole
series has been re-ordered, after some comments from Linus prompted some
reflection.
Most of the patches are the same, so I’ll specifically list the patches
with changes to help focus review.
1. Redo of the pte_mkwrite() refactoring patches
------------------------------------------------
The point of these patches was to make pte_mkwrite() take a VMA.
Unfortunately the original version of this refactor had a bug. Linus
suggested an alternate way of doing the refactor that would be less error
prone. It sounds like this will be used by the riscv and possibly arm
shadow stack features. It would be great to collect some Reviewed-by tags
on these from anyone else that would depend on them.
Changed (and renamed) patches:
mm: Rename arch pte_mkwrite()'s to pte_mkwrite_novma()
mm: Move pte/pmd_mkwrite() callers with no VMA to _novma()
mm: Make pte_mkwrite() take a VMA
2. SavedDirty overhaul
----------------------
The Shadow Stack PTEs are defined by the HW as Write=0,Dirty=1. Since
Linux usually creates PTEs like that when it write-protects dirty memory,
the series introduced a SavedDirty software bit. When a PTE gets
write-protected the Dirty bit shifts to the SavedDirty bit so it won't
become shadow stack. When it is made writable, the opposite happens.
In the previously queued version, the SavedDirty bit was only used when
shadow stack was configured and available on the CPU. But this created two
versions of the Dirty bit behavior on x86, adding complexity. Linus
objected to this, and also the use of conditional control flow logic
instead of bit math. After some trial and error, I ended up with something
that tries to incorporate the feedback, but with some adjustments on the
specifics. I would like some feedback on the maintainability tradeoffs
taken and some scrutiny on the correctness as well.
First of all, switching to bit math for the SavedDirty dance really seems
to be a big improvement. The conditional part is now branchless and
isolated to two functions. Descriptions of the other changes follows, and
a little less of a clear win to me.
One thing that came up in this discussion was the performance impact of
the CMPXCHG loop added to ptep_set_wrprotect() in order to atomically do
the shift of Dirty to SavedDirty when a live PTE is being write-protected.
The concern was that this could impact the performance of fork() where it
is used to write-protect memory in the parent MM.
Linus had suggested to optimize the single threaded fork case to offset
the concerns of a performance impact. However, trying to stress this case,
I was unable to concoct a microbenchmark to show any slowdown of the LOCK
CMPXCHG loop vs the original LOCK AND. Hypothetically the CMPXCHG could
scale worse, but since I couldn’t actually entice it to show any slowdown,
I was thinking that the original worries might have been misplaced and we
could get away with the unconditional CMPXCHG loop.
As Dave previously mentioned, the other wrinkle in all of this is that on
CPUs that don’t support shadow stack, a CPU could rarely set Dirty=1 on a
PTE with Write=0. So the kernel logic needs to be robust to PTEs that have
Write=0,Dirty=1 PTEs in non-shadow stack cases on these CPUs. And also
should be able to handle Write=0,Dirty=1,SavedDirty=1 PTEs. Similarly, a
KNL (Xeon Phi platform) erratum can result in Dirty=1 bits getting set on
Present=0 PTEs.
In order to make the core-MM logic work correctly with shadow stack,
pte_write() needs to also return true for Write=0,Dirty=1 memory. Since
those older platforms can cause Dirty=1,Write=0 PTEs despite the kernels
efforts to not create any itself, the kernel can’t have this shadow stack
pte_write() logic when running on them. So the kernel can only check for
shadow stack memory in pte_write() when shadow stack is supported by the
CPU. Also, the kernel needs to make sure to not trigger any warnings when
it sees a Dirty=0,Write=1 PTE in an unexpected place. So these are also
only done when shadow stack is supported on the CPU. These checks are
isolated to single place in pte_shstk().
So in the end, we can shift the Dirty bit around unconditionally, but the
kernels behavior around the Dirty bit still needs to adjust depending on
the CPU actually supporting shadow stack.
Refactoring the SavedDirty<->Dirty setting logic into bitmath made it so
it would be a lot easier to compile the SavedDirty bit out if needed.
Basically it can be removed with only two checks in
mksaveddirty_shift()/clear_saveddirty_shift() (see “x86/mm: Introduce
_PAGE_SAVED_DIRTY”).
That all makes me wonder if it would still be better to disable SavedDirty
when shadow stack is not supported. One aspect of the idea to make
SavedDirty unconditional, was to unify the sets of rules to have to
reason about. But due to the behavior of the pre-shadow stack CPUs, this
also comes at the increased runtime behaviors we need to worry about. The
other point brought up was the increased testing of having SavedDirty used
more widely. But since we have to turn off the warnings and other logic,
this testing isn’t fully happening on these older platforms either. So I’m
not sure if the unconditional SavedDirty is really a win or not. I thought
it was slightly.
So in this version SavedDirty is turned on universally for x86. Even for
32 bit, which, while seeming a bit silly, allows there to be only one
version of the ptep_set_wrprotect() logic.
Changed patches:
x86/mm: Start actually marking _PAGE_SAVED_DIRTY
x86/mm: Update ptep/pmdp_set_wrprotect() for _PAGE_SAVED_DIRTY
x86/mm: Introduce _PAGE_SAVED_DIRTY
mm: Warn on shadow stack memory in wrong vma
x86/mm: Warn if create Write=0,Dirty=1 with raw prot
3. Shadow stack protections enhancements
----------------------------------------
The shadow stack signal frame format uses a special shadow stack frame
pattern that should not occur naturally in order to avoid forgery on
sigreturn. Two patches are added to strengthen the forgery checks. These
could have been squashed into the signal patch, but I thought leaving them
separate might make review easier for those familiar with the last series.
I was waffling on whether to postpone them to minimize changes from the
previously queued changed to v9. In the end, as long as the series was
already getting re-spun, I thought the extra protections were worth
starting with.
The new mmap maple tree code needs to be taught specifically about
VM_SHADOW_STACK, instead of just relying on vm_start_gap() like the old RB
stuff, so that is added as well to retain the start guard gap with maple
tree.
Added/changes patches:
x86/shstk: Check that SSP is aligned on sigreturn
x86/shstk: Check that signal frame is shadow stack mem
mm: Add guard pages around a shadow stack
4. Selftest enhancements
------------------------
A few miscellaneous selftest enhancements that accumulated since the old
series landed in tip. Added a test for the shadow stack guard gap and the
shadow stack ptrace interface. Also fixed a race that caused the uffd test
to sometimes hang.
Changed patches:
selftests/x86: Add shadow stack test
Since some of the changes were extensive in the modified patches, I
dropped some review tags. But I left testing tags, testers please retest.
Thanks,
Rick
[0] https://lore.kernel.org/lkml/20220130211838.8382-1-rick.p.edgecombe@intel.com/
[1] https://lore.kernel.org/lkml/CAHk-=wiuVXTfgapmjYQvrEDzn3naF2oYnHuky+feEJSj_G_yFQ@mail.gmail.com/
[2] https://lore.kernel.org/lkml/CAHk-=wiB0wy6oXOsPtYU4DSbqJAY8z5iNBKdjdOp2LP23khUoA@mail.gmail.com/
Mike Rapoport (1):
x86/shstk: Add ARCH_SHSTK_UNLOCK
Rick Edgecombe (38):
mm: Rename arch pte_mkwrite()'s to pte_mkwrite_novma()
mm: Move pte/pmd_mkwrite() callers with no VMA to _novma()
mm: Make pte_mkwrite() take a VMA
x86/shstk: Add Kconfig option for shadow stack
x86/traps: Move control protection handler to separate file
x86/cpufeatures: Add CPU feature flags for shadow stacks
x86/mm: Move pmd_write(), pud_write() up in the file
x86/mm: Introduce _PAGE_SAVED_DIRTY
x86/mm: Update ptep/pmdp_set_wrprotect() for _PAGE_SAVED_DIRTY
x86/mm: Start actually marking _PAGE_SAVED_DIRTY
x86/mm: Remove _PAGE_DIRTY from kernel RO pages
x86/mm: Check shadow stack page fault errors
mm: Add guard pages around a shadow stack.
mm: Warn on shadow stack memory in wrong vma
x86/mm: Warn if create Write=0,Dirty=1 with raw prot
mm/mmap: Add shadow stack pages to memory accounting
x86/mm: Introduce MAP_ABOVE4G
x86/mm: Teach pte_mkwrite() about stack memory
mm: Don't allow write GUPs to shadow stack memory
Documentation/x86: Add CET shadow stack description
x86/fpu/xstate: Introduce CET MSR and XSAVES supervisor states
x86/fpu: Add helper for modifying xstate
x86: Introduce userspace API for shadow stack
x86/shstk: Add user control-protection fault handler
x86/shstk: Add user-mode shadow stack support
x86/shstk: Handle thread shadow stack
x86/shstk: Introduce routines modifying shstk
x86/shstk: Handle signals for shadow stack
x86/shstk: Check that SSP is aligned on sigreturn
x86/shstk: Check that signal frame is shadow stack mem
x86/shstk: Introduce map_shadow_stack syscall
x86/shstk: Support WRSS for userspace
x86: Expose thread features in /proc/$PID/status
x86/shstk: Wire in shadow stack interface
x86/cpufeatures: Enable CET CR4 bit for shadow stack
selftests/x86: Add shadow stack test
x86: Add PTRACE interface for shadow stack
x86/shstk: Add ARCH_SHSTK_STATUS
Yu-cheng Yu (3):
mm: Re-introduce vm_flags to do_mmap()
mm: Move VM_UFFD_MINOR_BIT from 37 to 38
mm: Introduce VM_SHADOW_STACK for shadow stack memory
Documentation/arch/x86/index.rst | 1 +
Documentation/arch/x86/shstk.rst | 179 ++++
Documentation/filesystems/proc.rst | 1 +
Documentation/mm/arch_pgtable_helpers.rst | 12 +-
arch/Kconfig | 3 +
arch/alpha/include/asm/pgtable.h | 2 +-
arch/arc/include/asm/hugepage.h | 2 +-
arch/arc/include/asm/pgtable-bits-arcv2.h | 2 +-
arch/arm/include/asm/pgtable-3level.h | 2 +-
arch/arm/include/asm/pgtable.h | 2 +-
arch/arm/kernel/signal.c | 2 +-
arch/arm64/include/asm/pgtable.h | 4 +-
arch/arm64/kernel/signal.c | 2 +-
arch/arm64/kernel/signal32.c | 2 +-
arch/arm64/mm/trans_pgd.c | 4 +-
arch/csky/include/asm/pgtable.h | 2 +-
arch/hexagon/include/asm/pgtable.h | 2 +-
arch/ia64/include/asm/pgtable.h | 2 +-
arch/loongarch/include/asm/pgtable.h | 4 +-
arch/m68k/include/asm/mcf_pgtable.h | 2 +-
arch/m68k/include/asm/motorola_pgtable.h | 2 +-
arch/m68k/include/asm/sun3_pgtable.h | 2 +-
arch/microblaze/include/asm/pgtable.h | 2 +-
arch/mips/include/asm/pgtable.h | 6 +-
arch/nios2/include/asm/pgtable.h | 2 +-
arch/openrisc/include/asm/pgtable.h | 2 +-
arch/parisc/include/asm/pgtable.h | 2 +-
arch/powerpc/include/asm/book3s/32/pgtable.h | 2 +-
arch/powerpc/include/asm/book3s/64/pgtable.h | 4 +-
arch/powerpc/include/asm/nohash/32/pgtable.h | 4 +-
arch/powerpc/include/asm/nohash/32/pte-8xx.h | 4 +-
arch/powerpc/include/asm/nohash/64/pgtable.h | 2 +-
arch/riscv/include/asm/pgtable.h | 6 +-
arch/s390/include/asm/hugetlb.h | 2 +-
arch/s390/include/asm/pgtable.h | 4 +-
arch/s390/mm/pageattr.c | 4 +-
arch/sh/include/asm/pgtable_32.h | 4 +-
arch/sparc/include/asm/pgtable_32.h | 2 +-
arch/sparc/include/asm/pgtable_64.h | 6 +-
arch/sparc/kernel/signal32.c | 2 +-
arch/sparc/kernel/signal_64.c | 2 +-
arch/um/include/asm/pgtable.h | 2 +-
arch/x86/Kconfig | 24 +
arch/x86/Kconfig.assembler | 5 +
arch/x86/entry/syscalls/syscall_64.tbl | 1 +
arch/x86/include/asm/cpufeatures.h | 2 +
arch/x86/include/asm/disabled-features.h | 16 +-
arch/x86/include/asm/fpu/api.h | 9 +
arch/x86/include/asm/fpu/regset.h | 7 +-
arch/x86/include/asm/fpu/sched.h | 3 +-
arch/x86/include/asm/fpu/types.h | 16 +-
arch/x86/include/asm/fpu/xstate.h | 6 +-
arch/x86/include/asm/idtentry.h | 2 +-
arch/x86/include/asm/mmu_context.h | 2 +
arch/x86/include/asm/pgtable.h | 302 +++++-
arch/x86/include/asm/pgtable_types.h | 46 +-
arch/x86/include/asm/processor.h | 8 +
arch/x86/include/asm/shstk.h | 38 +
arch/x86/include/asm/special_insns.h | 13 +
arch/x86/include/asm/tlbflush.h | 3 +-
arch/x86/include/asm/trap_pf.h | 2 +
arch/x86/include/asm/traps.h | 12 +
arch/x86/include/uapi/asm/mman.h | 4 +
arch/x86/include/uapi/asm/prctl.h | 12 +
arch/x86/kernel/Makefile | 4 +
arch/x86/kernel/cet.c | 152 +++
arch/x86/kernel/cpu/common.c | 35 +-
arch/x86/kernel/cpu/cpuid-deps.c | 1 +
arch/x86/kernel/cpu/proc.c | 23 +
arch/x86/kernel/fpu/core.c | 54 +-
arch/x86/kernel/fpu/regset.c | 81 ++
arch/x86/kernel/fpu/xstate.c | 90 +-
arch/x86/kernel/idt.c | 2 +-
arch/x86/kernel/process.c | 21 +-
arch/x86/kernel/process_64.c | 8 +
arch/x86/kernel/ptrace.c | 12 +
arch/x86/kernel/shstk.c | 529 +++++++++++
arch/x86/kernel/signal.c | 1 +
arch/x86/kernel/signal_32.c | 2 +-
arch/x86/kernel/signal_64.c | 8 +-
arch/x86/kernel/sys_x86_64.c | 6 +-
arch/x86/kernel/traps.c | 87 --
arch/x86/mm/fault.c | 22 +
arch/x86/mm/pat/set_memory.c | 4 +-
arch/x86/mm/pgtable.c | 40 +
arch/x86/xen/enlighten_pv.c | 2 +-
arch/x86/xen/mmu_pv.c | 2 +-
arch/x86/xen/xen-asm.S | 2 +-
arch/xtensa/include/asm/pgtable.h | 2 +-
fs/aio.c | 2 +-
fs/proc/array.c | 6 +
fs/proc/task_mmu.c | 3 +
include/asm-generic/hugetlb.h | 2 +-
include/linux/mm.h | 67 +-
include/linux/mman.h | 4 +
include/linux/pgtable.h | 28 +
include/linux/proc_fs.h | 2 +
include/linux/syscalls.h | 1 +
include/uapi/asm-generic/siginfo.h | 3 +-
include/uapi/asm-generic/unistd.h | 2 +-
include/uapi/linux/elf.h | 2 +
ipc/shm.c | 2 +-
kernel/sys_ni.c | 1 +
mm/debug_vm_pgtable.c | 12 +-
mm/gup.c | 2 +-
mm/huge_memory.c | 11 +-
mm/internal.h | 4 +-
mm/memory.c | 5 +-
mm/migrate.c | 2 +-
mm/migrate_device.c | 2 +-
mm/mmap.c | 14 +-
mm/mprotect.c | 2 +-
mm/nommu.c | 4 +-
mm/userfaultfd.c | 2 +-
mm/util.c | 2 +-
tools/testing/selftests/x86/Makefile | 2 +-
.../testing/selftests/x86/test_shadow_stack.c | 884 ++++++++++++++++++
117 files changed, 2789 insertions(+), 307 deletions(-)
create mode 100644 Documentation/arch/x86/shstk.rst
create mode 100644 arch/x86/include/asm/shstk.h
create mode 100644 arch/x86/kernel/cet.c
create mode 100644 arch/x86/kernel/shstk.c
create mode 100644 tools/testing/selftests/x86/test_shadow_stack.c
--
2.34.1