From nobody Sun Feb 8 10:17:35 2026 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id AC6D11D0786 for ; Tue, 19 Nov 2024 16:32:05 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=90.155.50.34 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1732033927; cv=none; b=ijvgMt/tDvhhRt06ojvH38AB2vYR+OY1zWODDcQ13McIWOHFBKRK+qWg0bI2iR+2vNILQbXFVD+ZysZwFGD5tz6tMj6NkmD9x3TnHhUpWLpxkh8MG9DtYyDjdAWk85xKBkpUokmztGqLT77U5I5rCxCadmUyvBDxiyL72EBHmco= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1732033927; c=relaxed/simple; bh=Kll3naxKaRgx1hIF5ErmABSbuADQDLzscKsVmxNt9F0=; h=Message-Id:Date:From:To:Cc:Subject:References:MIME-Version: Content-Type; b=hRD+z4eep4fIB/GK8GNl53HPHEEe6/yrp1UWrlbcJ61GgppD2KW4IEcb6YZm7T/3MeDesF2b51hQWS5usXjhuI742RCin7s9CjB+dSsu1kr3vhnYepTJndN5tAPRpkzXwNHgJJIr8DNsDR3bVn7T/5+lU9+OGakfyoKVzpkS4rk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org; spf=none smtp.mailfrom=infradead.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b=qxQ0arek; arc=none smtp.client-ip=90.155.50.34 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=infradead.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="qxQ0arek" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-Id:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=vE/LC6yVFmhHKFcfrpK9WBndKxy/9zWNjg0UWLOfH/0=; b=qxQ0arekj2w0UoqvJqOcbcBI+W HdJPpSIND0nTvTLxOre5dY2Zdp7RdR/gc1x3bkeQFD8wSnOSO5BKs+s7UsDDTSvjzm2k7oO564j0D 0HcPXTFS++1RsqiQtb1u1Ks6Gm+xZpYvd9a42afVHWqAqyYwGfFnCTlscfRNB82CFUT6P6juPryzO asqOYSoiKQ/71lDXUHYQBHw5ssfgzmfip3ocQU4J28XhZAeyX8Jy97GQeggHS8z8Vpc17gi3MHRm3 94pgKjsT+vgvgjQVQ5cp8+fjvvAXKBy5leIpoUdp/Faemh/MEKjkza6w6oIajAMok/C1zbW6Qfbac KrNhbc0w==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by casper.infradead.org with esmtpsa (Exim 4.98 #2 (Red Hat Linux)) id 1tDR90-00000004IUF-2IZi; Tue, 19 Nov 2024 16:32:03 +0000 Received: by noisy.programming.kicks-ass.net (Postfix, from userid 0) id D903B300848; Tue, 19 Nov 2024 17:32:02 +0100 (CET) Message-Id: <20241119163035.322525475@infradead.org> User-Agent: quilt/0.65 Date: Tue, 19 Nov 2024 17:25:28 +0100 From: Peter Zijlstra To: x86@kernel.org, "To:riel"@surriel.com Cc: linux-kernel@vger.kernel.org, peterz@infradead.org Subject: [PATCH 1/7] x86/mm: Add mm argument to unuse_temporary_mm() References: <20241119162527.952745944@infradead.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" In commit 209954cbc7d0 ("x86/mm/tlb: Update mm_cpumask lazily") unuse_temporary_mm() grew the assumption that it gets used on poking_nn exclusively. While this is currently true, lets not hard code this assumption. Signed-off-by: Peter Zijlstra (Intel) --- arch/x86/kernel/alternative.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) --- a/arch/x86/kernel/alternative.c +++ b/arch/x86/kernel/alternative.c @@ -1828,14 +1828,14 @@ static inline temp_mm_state_t use_tempor __ro_after_init struct mm_struct *poking_mm; __ro_after_init unsigned long poking_addr; =20 -static inline void unuse_temporary_mm(temp_mm_state_t prev_state) +static inline void unuse_temporary_mm(struct mm_struct *mm, temp_mm_state_= t prev_state) { lockdep_assert_irqs_disabled(); =20 switch_mm_irqs_off(NULL, prev_state.mm, current); =20 /* Clear the cpumask, to indicate no TLB flushing is needed anywhere */ - cpumask_clear_cpu(raw_smp_processor_id(), mm_cpumask(poking_mm)); + cpumask_clear_cpu(raw_smp_processor_id(), mm_cpumask(mm)); =20 /* * Restore the breakpoints if they were disabled before the temporary mm @@ -1942,7 +1942,7 @@ static void *__text_poke(text_poke_f fun * instruction that already allows the core to see the updated version. * Xen-PV is assumed to serialize execution in a similar manner. */ - unuse_temporary_mm(prev); + unuse_temporary_mm(poking_mm, prev); =20 /* * Flushing the TLB might involve IPIs, which would require enabled From nobody Sun Feb 8 10:17:35 2026 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id AC7871D0B9B for ; Tue, 19 Nov 2024 16:32:05 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=90.155.50.34 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1732033927; cv=none; b=BezcbHbfbv78l2cIq4MxNWTDTtt7qzpQbWWIuxjIsYyJsoGlEf4n+vcKu5KjBHQUi66yIk5zk8bAG0UuixNzibFCkp9getpge9pwZxK29qr91tC0f14hgHhdsEirRHZLcn5oMnE+SuhVfxqdCxoeEDsyJ2kzMMpzvnAUdbQrqnA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1732033927; c=relaxed/simple; bh=3hTq3ao6mMjvQIMqDw14trvOszSouAAEXNR4qG7UVT4=; h=Message-Id:Date:From:To:Cc:Subject:References:MIME-Version: Content-Type; b=nUFUiK+Tt/DaEkwhp9DCK9O7HAjhBUdcXRX1+04wcfhTIDZNcnml8il67RlRkPZvpQNlCiOW80zNUxU7d4BMRWuVwXLzYkbo7yx6xBXOOzPDd2TE0tYIatjPCgoMsNcYEaIDE98QG82EESSsZjGlocOJ6SSKSqEtlLjGdbb3R3I= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org; spf=none smtp.mailfrom=infradead.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b=tP8gFJZk; arc=none smtp.client-ip=90.155.50.34 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=infradead.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="tP8gFJZk" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-Id:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=t4Vmjpv2X0PmjEDEF4K179IGaPm3OC/AOeYNwIwQvfQ=; b=tP8gFJZkXCnvGpbvlSASjgS6wA 6Ym2vzmkaQk9RD2jPdgMYCzSBF5CRwu4wh+d+4fImXrdsevZsTGxQXXoG22SPR/fSSwzkc234od/c 6K0gE2DyT0Cpy6q5+gJXXmmG2oHg/lavxD/hl9HDGHipfZOOi93mlaCRCSzICT2t8yvt28Uf6yinu 3N/TvmsEJnFzMdwae9ZaxxhsX5gN7m5YJ8yXuSx2mdLi3HYoZlMi7PeiRzJ+n+nYQs9J9Uc8i/hfP C4V/R/07FYpAI5bIqKWpAUT3vzjaPe/W6u9yrZrSHlDW0GWM2c+Sa8xlf4nlmt2GZzQctlQZKaV97 dXKaoXUg==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by casper.infradead.org with esmtpsa (Exim 4.98 #2 (Red Hat Linux)) id 1tDR90-00000004IUG-24Zk; Tue, 19 Nov 2024 16:32:03 +0000 Received: by noisy.programming.kicks-ass.net (Postfix, from userid 0) id DD8B7300AA5; Tue, 19 Nov 2024 17:32:02 +0100 (CET) Message-Id: <20241119163035.433533770@infradead.org> User-Agent: quilt/0.65 Date: Tue, 19 Nov 2024 17:25:29 +0100 From: Peter Zijlstra To: x86@kernel.org, "To:riel"@surriel.com Cc: linux-kernel@vger.kernel.org, peterz@infradead.org, Andy Lutomirski Subject: [PATCH 2/7] x86/events, x86/insn-eval: Remove incorrect active_mm references References: <20241119162527.952745944@infradead.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Andy Lutomirski When decoding an instruction or handling a perf event that references an LDT segment, if we don't have a valid user context, trying to access the LDT by any means other than SLDT is racy. Certainly, using current->active_mm is wrong, as active_mm can point to a real user mm when CR3 and LDTR no longer reference that mm. Clean up the code. If nmi_uaccess_okay() says we don't have a valid context, just fail. Otherwise use current->mm. Signed-off-by: Andy Lutomirski Signed-off-by: Peter Zijlstra (Intel) Link: https://lkml.kernel.org/r/d456e7da9dbd271aacd14812d4b9b74e7d7edd52.16= 41659630.git.luto@kernel.org --- arch/x86/events/core.c | 9 ++++++++- arch/x86/lib/insn-eval.c | 13 ++++++++++--- 2 files changed, 18 insertions(+), 4 deletions(-) --- a/arch/x86/events/core.c +++ b/arch/x86/events/core.c @@ -2798,8 +2798,15 @@ static unsigned long get_segment_base(un #ifdef CONFIG_MODIFY_LDT_SYSCALL struct ldt_struct *ldt; =20 + /* + * If we're not in a valid context with a real (not just lazy) + * user mm, then don't even try. + */ + if (!nmi_uaccess_okay()) + return 0; + /* IRQs are off, so this synchronizes with smp_store_release */ - ldt =3D READ_ONCE(current->active_mm->context.ldt); + ldt =3D smp_load_acquire(¤t->mm->context.ldt); if (!ldt || idx >=3D ldt->nr_entries) return 0; =20 --- a/arch/x86/lib/insn-eval.c +++ b/arch/x86/lib/insn-eval.c @@ -631,14 +631,21 @@ static bool get_desc(struct desc_struct /* Bits [15:3] contain the index of the desired entry. */ sel >>=3D 3; =20 - mutex_lock(¤t->active_mm->context.lock); - ldt =3D current->active_mm->context.ldt; + /* + * If we're not in a valid context with a real (not just lazy) + * user mm, then don't even try. + */ + if (!nmi_uaccess_okay()) + return false; + + mutex_lock(¤t->mm->context.lock); + ldt =3D current->mm->context.ldt; if (ldt && sel < ldt->nr_entries) { *out =3D ldt->entries[sel]; success =3D true; } =20 - mutex_unlock(¤t->active_mm->context.lock); + mutex_unlock(¤t->mm->context.lock); =20 return success; } From nobody Sun Feb 8 10:17:35 2026 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id AC7341D07AB for ; Tue, 19 Nov 2024 16:32:05 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=90.155.50.34 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1732033927; cv=none; b=isBCHCM8Bwdz40wQjolZ47P2YjBxClYSGnx1BtHwaJUWJMTZTlOg3F/RwpXO7elNyjh8ReEdbWPtaLpeXyFh1PFWf1GGSjxlKo9xeibEEkcN19fFJ+zOFMZ/3modYmcQe59A7LnQwK5P/meefkRKVvy9yHrZx/MWdOV3ehYJVIY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1732033927; c=relaxed/simple; bh=t2mAq+XSOk2pSgviatf3ysUM3qhgcmUBIzs/wMhv/M0=; h=Message-Id:Date:From:To:Cc:Subject:References:MIME-Version: Content-Type; b=TaMysKo+np0cabXcycS3l42XSh6+NHf6xNJJEnN594H9PTyhpWLPxwzez7uJPFiJ6EMBpa++PEf7dVBe0HtYV24rVcYbGrhmeSmbiJkewNZ7skhq991z6VgoDVUE9nCmKvV8P6YnYNqATqLyTf3CoUucl4+2Zng1hXRDqIg/HX8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org; spf=none smtp.mailfrom=infradead.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b=CgUv2Jaq; arc=none smtp.client-ip=90.155.50.34 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=infradead.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="CgUv2Jaq" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-Id:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=Dwx1+Re0t7sLEqRPtBA9teCYzo3QP8AF4KmF4gdWCLE=; b=CgUv2Jaq5SUgC9PBoB7sXsWOwr upjTY18zakAKjtSIIGWthq3xngfWl+CjzI7Xae1Oud9MOZccf4ngTo+zk4v8lYRZMnk90tQJaE4SV bawyJhV+m1T4QF3TT4+oy4xwdXPuAPQbT9AY4CclmvOYpkmpQQz/iP/rIS2HlnfSsDy6JvJBnqCGo ZtKTWdfBUvV09a27P+olZnawCw/T5tfEMOONfP80+XqOu5kwxwp2ZCNnCgPr+7A1Thum5YrdR8A/W UwyiSt43BGfI+21qNfKGSHljAnemq1dgKUyOGO6bIkwy+nfjlxZs80QeWO0JZlDrW9ZyYYDbOpXRV qsXty4AA==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by casper.infradead.org with esmtpsa (Exim 4.98 #2 (Red Hat Linux)) id 1tDR90-00000004IUH-2FKB; Tue, 19 Nov 2024 16:32:03 +0000 Received: by noisy.programming.kicks-ass.net (Postfix, from userid 0) id E264E300E95; Tue, 19 Nov 2024 17:32:02 +0100 (CET) Message-Id: <20241119163035.533822339@infradead.org> User-Agent: quilt/0.65 Date: Tue, 19 Nov 2024 17:25:30 +0100 From: Peter Zijlstra To: x86@kernel.org, "To:riel"@surriel.com Cc: linux-kernel@vger.kernel.org, peterz@infradead.org, Andy Lutomirski Subject: [PATCH 3/7] x86/mm: Make use/unuse_temporary_mm() non-static References: <20241119162527.952745944@infradead.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Andy Lutomirski This prepares them for use outside of the alternative machinery. The code is unchanged. Signed-off-by: Andy Lutomirski Signed-off-by: Peter Zijlstra (Intel) Link: https://lkml.kernel.org/r/d1205bc7e165e249c52b7fe8cb1254f06e8a0e2a.16= 41659630.git.luto@kernel.org --- arch/x86/include/asm/mmu_context.h | 7 +++ arch/x86/kernel/alternative.c | 68 --------------------------------= ----- arch/x86/mm/tlb.c | 63 ++++++++++++++++++++++++++++++++= ++ 3 files changed, 70 insertions(+), 68 deletions(-) --- a/arch/x86/include/asm/mmu_context.h +++ b/arch/x86/include/asm/mmu_context.h @@ -263,4 +263,11 @@ unsigned long __get_current_cr3_fast(voi =20 #include =20 +typedef struct { + struct mm_struct *mm; +} temp_mm_state_t; + +extern temp_mm_state_t use_temporary_mm(struct mm_struct *mm); +extern void unuse_temporary_mm(struct mm_struct *mm, temp_mm_state_t prev_= state); + #endif /* _ASM_X86_MMU_CONTEXT_H */ --- a/arch/x86/kernel/alternative.c +++ b/arch/x86/kernel/alternative.c @@ -1774,77 +1774,9 @@ void __init_or_module text_poke_early(vo } } =20 -typedef struct { - struct mm_struct *mm; -} temp_mm_state_t; - -/* - * Using a temporary mm allows to set temporary mappings that are not acce= ssible - * by other CPUs. Such mappings are needed to perform sensitive memory wri= tes - * that override the kernel memory protections (e.g., W^X), without exposi= ng the - * temporary page-table mappings that are required for these write operati= ons to - * other CPUs. Using a temporary mm also allows to avoid TLB shootdowns wh= en the - * mapping is torn down. - * - * Context: The temporary mm needs to be used exclusively by a single core= . To - * harden security IRQs must be disabled while the temporary mm is - * loaded, thereby preventing interrupt handler bugs from overrid= ing - * the kernel memory protection. - */ -static inline temp_mm_state_t use_temporary_mm(struct mm_struct *mm) -{ - temp_mm_state_t temp_state; - - lockdep_assert_irqs_disabled(); - - /* - * Make sure not to be in TLB lazy mode, as otherwise we'll end up - * with a stale address space WITHOUT being in lazy mode after - * restoring the previous mm. - */ - if (this_cpu_read(cpu_tlbstate_shared.is_lazy)) - leave_mm(); - - temp_state.mm =3D this_cpu_read(cpu_tlbstate.loaded_mm); - switch_mm_irqs_off(NULL, mm, current); - - /* - * If breakpoints are enabled, disable them while the temporary mm is - * used. Userspace might set up watchpoints on addresses that are used - * in the temporary mm, which would lead to wrong signals being sent or - * crashes. - * - * Note that breakpoints are not disabled selectively, which also causes - * kernel breakpoints (e.g., perf's) to be disabled. This might be - * undesirable, but still seems reasonable as the code that runs in the - * temporary mm should be short. - */ - if (hw_breakpoint_active()) - hw_breakpoint_disable(); - - return temp_state; -} - __ro_after_init struct mm_struct *poking_mm; __ro_after_init unsigned long poking_addr; =20 -static inline void unuse_temporary_mm(struct mm_struct *mm, temp_mm_state_= t prev_state) -{ - lockdep_assert_irqs_disabled(); - - switch_mm_irqs_off(NULL, prev_state.mm, current); - - /* Clear the cpumask, to indicate no TLB flushing is needed anywhere */ - cpumask_clear_cpu(raw_smp_processor_id(), mm_cpumask(mm)); - - /* - * Restore the breakpoints if they were disabled before the temporary mm - * was loaded. - */ - if (hw_breakpoint_active()) - hw_breakpoint_restore(); -} - static void text_poke_memcpy(void *dst, const void *src, size_t len) { memcpy(dst, src, len); --- a/arch/x86/mm/tlb.c +++ b/arch/x86/mm/tlb.c @@ -674,6 +674,69 @@ void enter_lazy_tlb(struct mm_struct *mm } =20 /* + * Using a temporary mm allows to set temporary mappings that are not acce= ssible + * by other CPUs. Such mappings are needed to perform sensitive memory wri= tes + * that override the kernel memory protections (e.g., W^X), without exposi= ng the + * temporary page-table mappings that are required for these write operati= ons to + * other CPUs. Using a temporary mm also allows to avoid TLB shootdowns wh= en the + * mapping is torn down. + * + * Context: The temporary mm needs to be used exclusively by a single core= . To + * harden security IRQs must be disabled while the temporary mm is + * loaded, thereby preventing interrupt handler bugs from overrid= ing + * the kernel memory protection. + */ +temp_mm_state_t use_temporary_mm(struct mm_struct *mm) +{ + temp_mm_state_t temp_state; + + lockdep_assert_irqs_disabled(); + + /* + * Make sure not to be in TLB lazy mode, as otherwise we'll end up + * with a stale address space WITHOUT being in lazy mode after + * restoring the previous mm. + */ + if (this_cpu_read(cpu_tlbstate_shared.is_lazy)) + leave_mm(); + + temp_state.mm =3D this_cpu_read(cpu_tlbstate.loaded_mm); + switch_mm_irqs_off(NULL, mm, current); + + /* + * If breakpoints are enabled, disable them while the temporary mm is + * used. Userspace might set up watchpoints on addresses that are used + * in the temporary mm, which would lead to wrong signals being sent or + * crashes. + * + * Note that breakpoints are not disabled selectively, which also causes + * kernel breakpoints (e.g., perf's) to be disabled. This might be + * undesirable, but still seems reasonable as the code that runs in the + * temporary mm should be short. + */ + if (hw_breakpoint_active()) + hw_breakpoint_disable(); + + return temp_state; +} + +void unuse_temporary_mm(struct mm_struct *mm, temp_mm_state_t prev_state) +{ + lockdep_assert_irqs_disabled(); + switch_mm_irqs_off(NULL, prev_state.mm, current); + + /* Clear the cpumask, to indicate no TLB flushing is needed anywhere */ + cpumask_clear_cpu(raw_smp_processor_id(), mm_cpumask(mm)); + + /* + * Restore the breakpoints if they were disabled before the temporary mm + * was loaded. + */ + if (hw_breakpoint_active()) + hw_breakpoint_restore(); +} + +/* * Call this when reinitializing a CPU. It fixes the following potential * problems: * From nobody Sun Feb 8 10:17:35 2026 Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5D9801D0E10 for ; Tue, 19 Nov 2024 16:32:06 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=90.155.92.199 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1732033929; cv=none; b=iyjcYV7+O3aS6Y738pa+RmaJ5WpBOciQEhGk5AvWCdAZEw3hARG7MEmXm6qvX9W92wcwjQPw/6mQl568w0TRtp/p+huOhOGDm8yxhLguNxVUYxn12XJSFbWZ7d3e5sLee4bGdRxU8rdVVjch05H/deR627ZIQc98u477qlfl9UY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1732033929; c=relaxed/simple; bh=Yka98/1DnkN8a40MMGm5+Yi0sw49U0PbphlXtS9eM/8=; h=Message-Id:Date:From:To:Cc:Subject:References:MIME-Version: Content-Type; b=tHjvwKX3C6e9jmsgbnNEwRekf3BrdCCWUGZgHH3tfexLjYLu9EyQcjCtQKGbK9k9xNHyTlYnLak7YQdApfe+8TzUGljb+5USchwIRYv5fauNIZ0fz3vJQrxqe1gW/luE5QLRTsfZRLxE7smu8n6xTJgg8eUdpx5NdBOUQyCpg1s= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org; spf=none smtp.mailfrom=infradead.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b=GKzbyzi9; arc=none smtp.client-ip=90.155.92.199 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=infradead.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="GKzbyzi9" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-Id:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=peG4b29NN3VgAAfzJZTqvQ0ALZJTfu1tNGwDQ2J9tYc=; b=GKzbyzi9LbtJ3E/7EQ1m4Ke98i q9ZgIy6mKBCPvg9nryo3ZgPraXS7sDSxFbdhQpBX0h9rDWzJfkClnO+hPl8/JiHdztwNHjmdOQIOo obL4ziROeUvJF7bdqXwtefgqolHG4gRyaRQn2kD+3NETHZgTz9CJDUmz6cNQgg2hVnf4P9GPGMGWb ijV6nq+Pj3I539RjJTPudq3YNi2BA8CtGfVhurEBIREZWXzKKOHHT3gX80jkM7dRRaHV2y6QDaO1L CdU6u40aKeeLwnJVUebBTxYxn296AmiVThFOjLvWFgf2XvI09EwKZln+OnE1iJ/FbokEVbOMimpqn Qg/CaAag==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by desiato.infradead.org with esmtpsa (Exim 4.98 #2 (Red Hat Linux)) id 1tDR91-00000000MTS-1Hx0; Tue, 19 Nov 2024 16:32:03 +0000 Received: by noisy.programming.kicks-ass.net (Postfix, from userid 0) id E677730119B; Tue, 19 Nov 2024 17:32:02 +0100 (CET) Message-Id: <20241119163035.648739178@infradead.org> User-Agent: quilt/0.65 Date: Tue, 19 Nov 2024 17:25:31 +0100 From: Peter Zijlstra To: x86@kernel.org, "To:riel"@surriel.com Cc: linux-kernel@vger.kernel.org, peterz@infradead.org Subject: [PATCH 4/7] x86/mm: Remove mm argument from unuse_temporary_mm() again References: <20241119162527.952745944@infradead.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Now that unuse_temporary_mm() lives in tlb.c it can access loaded_mm. Signed-off-by: Peter Zijlstra (Intel) --- arch/x86/include/asm/mmu_context.h | 2 +- arch/x86/kernel/alternative.c | 2 +- arch/x86/mm/tlb.c | 8 +++++--- 3 files changed, 7 insertions(+), 5 deletions(-) --- a/arch/x86/include/asm/mmu_context.h +++ b/arch/x86/include/asm/mmu_context.h @@ -268,6 +268,6 @@ typedef struct { } temp_mm_state_t; =20 extern temp_mm_state_t use_temporary_mm(struct mm_struct *mm); -extern void unuse_temporary_mm(struct mm_struct *mm, temp_mm_state_t prev_= state); +extern void unuse_temporary_mm(temp_mm_state_t prev_state); =20 #endif /* _ASM_X86_MMU_CONTEXT_H */ --- a/arch/x86/kernel/alternative.c +++ b/arch/x86/kernel/alternative.c @@ -1874,7 +1874,7 @@ static void *__text_poke(text_poke_f fun * instruction that already allows the core to see the updated version. * Xen-PV is assumed to serialize execution in a similar manner. */ - unuse_temporary_mm(poking_mm, prev); + unuse_temporary_mm(prev); =20 /* * Flushing the TLB might involve IPIs, which would require enabled --- a/arch/x86/mm/tlb.c +++ b/arch/x86/mm/tlb.c @@ -720,13 +720,15 @@ temp_mm_state_t use_temporary_mm(struct return temp_state; } =20 -void unuse_temporary_mm(struct mm_struct *mm, temp_mm_state_t prev_state) +void unuse_temporary_mm(temp_mm_state_t prev_state) { lockdep_assert_irqs_disabled(); - switch_mm_irqs_off(NULL, prev_state.mm, current); =20 /* Clear the cpumask, to indicate no TLB flushing is needed anywhere */ - cpumask_clear_cpu(raw_smp_processor_id(), mm_cpumask(mm)); + cpumask_clear_cpu(smp_processor_id(), + mm_cpumask(this_cpu_read(cpu_tlbstate.loaded_mm))); + + switch_mm_irqs_off(NULL, prev_state.mm, current); =20 /* * Restore the breakpoints if they were disabled before the temporary mm From nobody Sun Feb 8 10:17:35 2026 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id AC6331D04A5 for ; Tue, 19 Nov 2024 16:32:05 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=90.155.50.34 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1732033928; cv=none; b=eDdoRodnYsMTsd0fxHt4XRWQvzu7i2/EGwgQAmDlO0sinwqmD3+CvUYGdyB7o7gdGb/0vOSCeS4F4vlZ9VnYfOq2+wHF54GgtlK/FfgsZI4cV7sq1soamBklNmZ6+OhSKBPLJ8B0Uwzq9e/et9HXk3PeFumYGRYCecGcBo1c54g= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1732033928; c=relaxed/simple; bh=I8k0F0QiD8jJvP3np/mgV1ix1z/ehcWpSoNtSt5TcEQ=; h=Message-Id:Date:From:To:Cc:Subject:References:MIME-Version: Content-Type; b=P+DXPQluycJuIKMpZm3DnoCEkNCx4FPbQ6yCni2vhhsy4fCR9PRDc260YdBUwT70oHzyUEUamE5k6Z5iLE9NJTeQXnrItoNGGmRhAVD9CTgZnWsQO9LtYjnc2eHwjiZw595bjrCNdLYBhIXg9OopkYJUl6z+btAcz8efLvN29Ok= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org; spf=none smtp.mailfrom=infradead.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b=bP/W67pM; arc=none smtp.client-ip=90.155.50.34 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=infradead.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="bP/W67pM" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-Id:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=h/mnugnYftitJTxGPEXN/E3BRMTm4cTRJ4TLn82el4Y=; b=bP/W67pMX2DCXyaAz/cVAYD1KO 6dSCYQbG4p+79PtpZViWOMZve5PuH6ITBOzpKsRtuZ+vgFKFvRdCNgz7ISkULIomPue3FA2ZvwXws 535h+L4owZQq4lBigVhw2PlGV0ndjAF6xLibY2bEtnZXiX6D9PLDhkZHFVjDbx/PuNlhCJxK1FAGC Q8OS1lrLFr4rIRWwoBInUIwZPncA/uz7ivIgsA54qgXMjVrbzZfMPi76lxNEufZPVwsM2fGmuRyDv /Mg4mMiBF9xNZokZRhERjXT5m0y4a+RLu/oZtcl4uvpalyX+CNHFRHCIyyahSPsWGmmNpq6IUsisq QORQV3uQ==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by casper.infradead.org with esmtpsa (Exim 4.98 #2 (Red Hat Linux)) id 1tDR90-00000004IUI-3K1o; Tue, 19 Nov 2024 16:32:04 +0000 Received: by noisy.programming.kicks-ass.net (Postfix, from userid 0) id EAAA230119C; Tue, 19 Nov 2024 17:32:02 +0100 (CET) Message-Id: <20241119163035.758732080@infradead.org> User-Agent: quilt/0.65 Date: Tue, 19 Nov 2024 17:25:32 +0100 From: Peter Zijlstra To: x86@kernel.org, "To:riel"@surriel.com Cc: linux-kernel@vger.kernel.org, peterz@infradead.org, Andy Lutomirski Subject: [PATCH 5/7] x86/mm: Allow temporary mms when IRQs are on References: <20241119162527.952745944@infradead.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Andy Lutomirski EFI runtime services should use temporary mms, but EFI runtime services want IRQs on. Preemption must still be disabled in a temporary mm context. At some point, the entirely temporary mm mechanism should be moved out of arch code. Signed-off-by: Andy Lutomirski Signed-off-by: Peter Zijlstra (Intel) Link: https://lkml.kernel.org/r/a8a92ce490b57447ef56898c55133473e481896e.16= 41659630.git.luto@kernel.org --- arch/x86/mm/tlb.c | 19 ++++++++++++------- 1 file changed, 12 insertions(+), 7 deletions(-) --- a/arch/x86/mm/tlb.c +++ b/arch/x86/mm/tlb.c @@ -679,18 +679,23 @@ void enter_lazy_tlb(struct mm_struct *mm * that override the kernel memory protections (e.g., W^X), without exposi= ng the * temporary page-table mappings that are required for these write operati= ons to * other CPUs. Using a temporary mm also allows to avoid TLB shootdowns wh= en the - * mapping is torn down. + * mapping is torn down. Temporary mms can also be used for EFI runtime s= ervice + * calls or similar functionality. * - * Context: The temporary mm needs to be used exclusively by a single core= . To - * harden security IRQs must be disabled while the temporary mm is - * loaded, thereby preventing interrupt handler bugs from overrid= ing - * the kernel memory protection. + * It is illegal to schedule while using a temporary mm -- the context swi= tch + * code is unaware of the temporary mm and does not know how to context sw= itch. + * Use a real (non-temporary) mm in a kernel thread if you need to sleep. + * + * Note: For sensitive memory writes, the temporary mm needs to be used + * exclusively by a single core, and IRQs should be disabled while t= he + * temporary mm is loaded, thereby preventing interrupt handler bugs= from + * overriding the kernel memory protection. */ temp_mm_state_t use_temporary_mm(struct mm_struct *mm) { temp_mm_state_t temp_state; =20 - lockdep_assert_irqs_disabled(); + lockdep_assert_preemption_disabled(); =20 /* * Make sure not to be in TLB lazy mode, as otherwise we'll end up @@ -722,7 +727,7 @@ temp_mm_state_t use_temporary_mm(struct =20 void unuse_temporary_mm(temp_mm_state_t prev_state) { - lockdep_assert_irqs_disabled(); + lockdep_assert_preemption_disabled(); =20 /* Clear the cpumask, to indicate no TLB flushing is needed anywhere */ cpumask_clear_cpu(smp_processor_id(), From nobody Sun Feb 8 10:17:35 2026 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id AC6821D04A9 for ; Tue, 19 Nov 2024 16:32:05 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=90.155.50.34 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1732033927; cv=none; b=n+3C69Eb+9HMdzdFS9pQar1wIroZ/bgF5leIyWIzpjT3E/bT8bW2DxRXPzl0KYKZqCElDLJVswW6NIHtt+YXECwYVvId7meQ/n8KbJauI/S45gQdPPw8Msnsu6L14NzCWpxFyLmsTk+dVYpRLfk9eMnCaSpECZPjNePC/w0M+UQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1732033927; c=relaxed/simple; bh=aaLJwwXEQB5gZe/QJah9bx2V1s2F0e/ZLvGyqbZdTWY=; h=Message-Id:Date:From:To:Cc:Subject:References:MIME-Version: Content-Type; b=RgJc/ItEyoR5r2/FwDdT/BE8+h4PWmGOn5f6ayF70q1x4GdpErwnylX+i+H1EcHmWebSqCVR19gHSf43koTkbrTD9638iNpEMDTDxze3h/0QKJeKBZaiX7akFeHJ05ivZB0vgUatmfJotGzmrTrJWEu+sKIT5H50LNnKAmSEYOk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org; spf=none smtp.mailfrom=infradead.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b=L9nncVOG; arc=none smtp.client-ip=90.155.50.34 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=infradead.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="L9nncVOG" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-Id:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=iKHnr1Xsc+ED8W7SLp96C55Yt4mBqADdvf/siSHY3aM=; b=L9nncVOGYnDFMkaGvhplOHhiAb kXzkBlKj3Hvg+sdwoqqWV7CrXEdRf2a6AUJyH/rqypjd5TCUdQ4QqgfqqKGemesQ0bFMgyhz9n2Uv 49a2vBRqij+i8oemll4YeuoBzLzM9kwJisUBU5KtGOnZUPldSNc5FgtmIRdVCK28Qyha1I9v/dz1w G7/NGwXkS/8Yvs8PknAUdDOjrgXPehBn10LlfkgIYbWOj/1Ox9v+R4UEwQJHy4+kg+O0VrlNoonAn VwMRg+3snncL4CE8LQwTFnGjZqUGef9ibeBgsFhgcr/NMbjM5kKJQpfwPqWM2lrkyHTmFc4wpkZ81 wx9JMprg==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by casper.infradead.org with esmtpsa (Exim 4.98 #2 (Red Hat Linux)) id 1tDR90-00000004IUJ-3IZ5; Tue, 19 Nov 2024 16:32:04 +0000 Received: by noisy.programming.kicks-ass.net (Postfix, from userid 0) id EEE01301D99; Tue, 19 Nov 2024 17:32:02 +0100 (CET) Message-Id: <20241119163035.877939834@infradead.org> User-Agent: quilt/0.65 Date: Tue, 19 Nov 2024 17:25:33 +0100 From: Peter Zijlstra To: x86@kernel.org, "To:riel"@surriel.com Cc: linux-kernel@vger.kernel.org, peterz@infradead.org, Andy Lutomirski Subject: [PATCH 6/7] x86/efi: Make efi_enter/leave_mm use the temporary_mm machinery References: <20241119162527.952745944@infradead.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Andy Lutomirski This should be considerably more robust. It's also necessary for optimized for_each_possible_lazymm_cpu() on x86 -- without this patch, EFI calls in lazy context would remove the lazy mm from mm_cpumask(). Signed-off-by: Andy Lutomirski Signed-off-by: Peter Zijlstra (Intel) Link: https://lkml.kernel.org/r/3efc4cfd1d7c45a32752ced389d6666be15cde56.16= 41659630.git.luto@kernel.org --- arch/x86/platform/efi/efi_64.c | 9 +++------ 1 file changed, 3 insertions(+), 6 deletions(-) --- a/arch/x86/platform/efi/efi_64.c +++ b/arch/x86/platform/efi/efi_64.c @@ -54,7 +54,7 @@ * 0xffff_ffff_0000_0000 and limit EFI VA mapping space to 64G. */ static u64 efi_va =3D EFI_VA_START; -static struct mm_struct *efi_prev_mm; +static temp_mm_state_t efi_temp_mm_state; =20 /* * We need our own copy of the higher levels of the page tables @@ -476,15 +476,12 @@ void __init efi_dump_pagetable(void) */ static void efi_enter_mm(void) { - efi_prev_mm =3D current->active_mm; - current->active_mm =3D &efi_mm; - switch_mm(efi_prev_mm, &efi_mm, NULL); + efi_temp_mm_state =3D use_temporary_mm(&efi_mm); } =20 static void efi_leave_mm(void) { - current->active_mm =3D efi_prev_mm; - switch_mm(&efi_mm, efi_prev_mm, NULL); + unuse_temporary_mm(efi_temp_mm_state); } =20 void arch_efi_call_virt_setup(void) From nobody Sun Feb 8 10:17:35 2026 Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5D8F614A614 for ; Tue, 19 Nov 2024 16:32:06 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=90.155.92.199 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1732033928; cv=none; b=GYBDQQHg6cYZ5/NjeUI2hGclbP/qbEOC92u4obH1sQnAR92920OHODJZ9J+lXcxtgyyFrqakMBTZpL8GzWYwUegyougS8DArk3ZHJcf/+78sJlgt1u4eTo0oBxjbZcm/HAxSBHGhIAe/82c4+0nwTVUMu0S6vTNqxnCZEWkGHc4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1732033928; c=relaxed/simple; bh=mBPldd3yUicxrETCRUqOLGqOQNV9i/fAUwpKaw1SsI0=; h=Message-Id:Date:From:To:Cc:Subject:References:MIME-Version: Content-Type; b=T5BOwKIo4SaOqd5xCzxn36zz5VI5xtVYEnLhkGIr4LQjiIET4qKvhaHtw2vTcCzAobhnPZtIbKlvfGyc+BDzpLiZMzRdwKCrlkrI7B8yq5Wo3S9HLFAhGUqQDajCGAasqP0kkEevyoujGmItS3+DivMygQyCbeaMQztktMXeT80= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org; spf=none smtp.mailfrom=infradead.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b=g63RtYFx; arc=none smtp.client-ip=90.155.92.199 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=infradead.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="g63RtYFx" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-Id:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=bkFLe1usrwLIzhTY/+ijYnFXLDpAk9FZvP9TcIkqIHg=; b=g63RtYFxb944hivfZKh41nEC2y DgGWs/i0ec7s/51lPeONyDj2FryFVYYSmlq4R70Kt65MNS5RYNzalrFG6hDfXAxvj/hfjmEbH4R/4 LTI4HTMz8nmsM5HSkJPy1qKA/IyXLg5iK9EqPkbZ9+NfomSoqAL5KpqPl228F578B0YjGOfU/4AQg 5z11iRUuNQjwXsAPynmlshm630pFL2gNadFywh0XM8tL7rU+Gdnuyvp7R/zGOtpdDx3OFTP6xHBYx bOdSlSlfqEmQnWZcN3qs32FNhMMaccFmDa0sLsq6UpOKQChBT6uwa+jdANXDhTmJI4YVI5XO5KcvA Vx06WNQw==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by desiato.infradead.org with esmtpsa (Exim 4.98 #2 (Red Hat Linux)) id 1tDR92-00000000MTV-1Mfw; Tue, 19 Nov 2024 16:32:04 +0000 Received: by noisy.programming.kicks-ass.net (Postfix, from userid 0) id F28533021DA; Tue, 19 Nov 2024 17:32:02 +0100 (CET) Message-Id: <20241119163035.985203915@infradead.org> User-Agent: quilt/0.65 Date: Tue, 19 Nov 2024 17:25:34 +0100 From: Peter Zijlstra To: x86@kernel.org, "To:riel"@surriel.com Cc: linux-kernel@vger.kernel.org, peterz@infradead.org, Andy Lutomirski Subject: [PATCH 7/7] x86/mm: Opt in to IRQs-off activate_mm() References: <20241119162527.952745944@infradead.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Andy Lutomirski We gain nothing by having the core code enable IRQs right before calling activate_mm() only for us to turn them right back off again in switch_mm(). This will save a few cycles, so execve() should be blazingly fast with this patch applied! Signed-off-by: Andy Lutomirski Signed-off-by: Peter Zijlstra (Intel) Link: https://lkml.kernel.org/r/69c7d711f240cfec23e6024e940d31af2990db36.16= 41659630.git.luto@kernel.org --- arch/x86/Kconfig | 1 + arch/x86/include/asm/mmu_context.h | 2 +- 2 files changed, 2 insertions(+), 1 deletion(-) Index: linux-2.6/arch/x86/Kconfig =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D --- linux-2.6.orig/arch/x86/Kconfig +++ linux-2.6/arch/x86/Kconfig @@ -133,6 +133,7 @@ config X86 select ARCH_WANT_OPTIMIZE_HUGETLB_VMEMMAP if X86_64 select ARCH_WANTS_THP_SWAP if X86_64 select ARCH_HAS_PARANOID_L1D_FLUSH + select ARCH_WANT_IRQS_OFF_ACTIVATE_MM select BUILDTIME_TABLE_SORT select CLKEVT_I8253 select CLOCKSOURCE_VALIDATE_LAST_CYCLE Index: linux-2.6/arch/x86/include/asm/mmu_context.h =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D --- linux-2.6.orig/arch/x86/include/asm/mmu_context.h +++ linux-2.6/arch/x86/include/asm/mmu_context.h @@ -175,7 +175,7 @@ extern void switch_mm_irqs_off(struct mm #define activate_mm(prev, next) \ do { \ paravirt_enter_mmap(next); \ - switch_mm((prev), (next), NULL); \ + switch_mm_irqs_off((prev), (next), NULL); \ } while (0); =20 #ifdef CONFIG_X86_32