From nobody Sat Oct 4 01:45:00 2025 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 10EB0320CC6; Thu, 21 Aug 2025 12:42:48 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=90.155.50.34 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1755780172; cv=none; b=nzFyHvvW7UBgovxY8Wug1Y7FLtwaz0uEuW+s1264EE82YLL64O9aw4aEKP0yXpeunPwTKAItIzWGjz0JpBA43iz3H3t+kGpXrAykWUCSns7UOPpjgZ2Jui9nn/I4tOulLv8cO6uUXDep206O+6KgYEVMeX9XYT6yG6liw7LeuOI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1755780172; c=relaxed/simple; bh=fRuRyExtuleTQ0QZnw8BEriw7fy34Dn7Y/V6jbcJ+AE=; h=Message-ID:Date:From:To:Cc:Subject:References:MIME-Version: Content-Type; b=JsSlcVQBFrTLtr5nhiwqPPmxx9FdsWpAcc7gZdkVIQ8/9zka8tjCdFzm+ogoD60UfvZHBxImp6MJjuf48Xgjzjbv6C0vt0tvD57FqGBftzow5FUBn7lk2nXXJwN0lB6TyzeEvGT1cmD14zQnb34o4GeHTRH6YSRds8EBYJiLk1w= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org; spf=none smtp.mailfrom=infradead.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b=knmNnlNr; arc=none smtp.client-ip=90.155.50.34 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=infradead.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="knmNnlNr" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=68tGCHbm2uavV4Zx8UMMGxPgMa8MecvRI7OvCw9rTO8=; b=knmNnlNrEy4vgCCNy2oeO6yRNW YKmOGUjwHTeB2ZGLhJD7+CwCcUi5JUPHlA/Yimn6CzqLAOtAf+Q2iJcWGGjOCSlTaikb7pjv3WOqS WUrQB7mL/64nA89Yq8f01wFaJHAwSp1JBaxKRjdd2TNeUgQyb2Cam84NcS+GeGjX49++k6hv/IktA otGIoGKmc7yiB4ud+5pNH4DE73iO/OnXRryvaanNjYXZYW4nANW+Les5HvFPexybBg13U8H4rxYq/ f7IP9hI6Ol4BMDBrz1oZiEFYQ4gLbyWQxQD4e5FQCG5Ac6y6Lvoa366qszm4HE95DtRwcWow7tNz6 dJSJVOfQ==; Received: from 77-249-17-252.cable.dynamic.v4.ziggo.nl ([77.249.17.252] helo=noisy.programming.kicks-ass.net) by casper.infradead.org with esmtpsa (Exim 4.98.2 #2 (Red Hat Linux)) id 1up4cn-000000075gM-2lXi; Thu, 21 Aug 2025 12:42:38 +0000 Received: by noisy.programming.kicks-ass.net (Postfix, from userid 0) id 894CE30034B; Thu, 21 Aug 2025 14:42:37 +0200 (CEST) Message-ID: <20250821123656.705837806@infradead.org> User-Agent: quilt/0.68 Date: Thu, 21 Aug 2025 14:28:23 +0200 From: Peter Zijlstra To: jolsa@kernel.org, oleg@redhat.com, andrii@kernel.org, mhiramat@kernel.org Cc: linux-kernel@vger.kernel.org, peterz@infradead.org, alx@kernel.org, eyal.birger@gmail.com, kees@kernel.org, bpf@vger.kernel.org, linux-trace-kernel@vger.kernel.org, x86@kernel.org, songliubraving@fb.com, yhs@fb.com, john.fastabend@gmail.com, haoluo@google.com, rostedt@goodmis.org, alan.maguire@oracle.com, David.Laight@ACULAB.COM, thomas@t-8ch.de, mingo@kernel.org, rick.p.edgecombe@intel.com Subject: [PATCH 1/6] uprobes/x86: Add struct uretprobe_syscall_args References: <20250821122822.671515652@infradead.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Like uprobe_syscall_args; keep things consistent. Signed-off-by: Peter Zijlstra (Intel) Acked-by: Andrii Nakryiko Reviewed-by: Oleg Nesterov --- arch/x86/kernel/uprobes.c | 33 ++++++++++++++++++++------------- 1 file changed, 20 insertions(+), 13 deletions(-) --- a/arch/x86/kernel/uprobes.c +++ b/arch/x86/kernel/uprobes.c @@ -311,6 +311,12 @@ static int uprobe_init_insn(struct arch_ =20 #ifdef CONFIG_X86_64 =20 +struct uretprobe_syscall_args { + unsigned long r11; + unsigned long cx; + unsigned long ax; +}; + asm ( ".pushsection .rodata\n" ".global uretprobe_trampoline_entry\n" @@ -324,8 +330,8 @@ asm ( "uretprobe_syscall_check:\n" "popq %r11\n" "popq %rcx\n" - - /* The uretprobe syscall replaces stored %rax value with final + /* + * The uretprobe syscall replaces stored %rax value with final * return address, so we don't restore %rax in here and just * call ret. */ @@ -366,7 +372,8 @@ static unsigned long trampoline_check_ip SYSCALL_DEFINE0(uretprobe) { struct pt_regs *regs =3D task_pt_regs(current); - unsigned long err, ip, sp, r11_cx_ax[3], tramp; + struct uretprobe_syscall_args args; + unsigned long err, ip, sp, tramp; =20 /* If there's no trampoline, we are called from wrong place. */ tramp =3D uprobe_get_trampoline_vaddr(); @@ -377,15 +384,15 @@ SYSCALL_DEFINE0(uretprobe) if (unlikely(regs->ip !=3D trampoline_check_ip(tramp))) goto sigill; =20 - err =3D copy_from_user(r11_cx_ax, (void __user *)regs->sp, sizeof(r11_cx_= ax)); + err =3D copy_from_user(&args, (void __user *)regs->sp, sizeof(args)); if (err) goto sigill; =20 /* expose the "right" values of r11/cx/ax/sp to uprobe_consumer/s */ - regs->r11 =3D r11_cx_ax[0]; - regs->cx =3D r11_cx_ax[1]; - regs->ax =3D r11_cx_ax[2]; - regs->sp +=3D sizeof(r11_cx_ax); + regs->r11 =3D args.r11; + regs->cx =3D args.cx; + regs->ax =3D args.ax; + regs->sp +=3D sizeof(args); regs->orig_ax =3D -1; =20 ip =3D regs->ip; @@ -401,21 +408,21 @@ SYSCALL_DEFINE0(uretprobe) */ if (regs->sp !=3D sp || shstk_is_enabled()) return regs->ax; - regs->sp -=3D sizeof(r11_cx_ax); + regs->sp -=3D sizeof(args); =20 /* for the case uprobe_consumer has changed r11/cx */ - r11_cx_ax[0] =3D regs->r11; - r11_cx_ax[1] =3D regs->cx; + args.r11 =3D regs->r11; + args.cx =3D regs->cx; =20 /* * ax register is passed through as return value, so we can use * its space on stack for ip value and jump to it through the * trampoline's ret instruction */ - r11_cx_ax[2] =3D regs->ip; + args.ax =3D regs->ip; regs->ip =3D ip; =20 - err =3D copy_to_user((void __user *)regs->sp, r11_cx_ax, sizeof(r11_cx_ax= )); + err =3D copy_to_user((void __user *)regs->sp, &args, sizeof(args)); if (err) goto sigill; From nobody Sat Oct 4 01:45:00 2025 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 10F2C320CC7; Thu, 21 Aug 2025 12:42:48 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=90.155.50.34 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1755780171; cv=none; b=Bdnpz5ecsutESP9x6JhOqxDHfXLV9kRMLEk3tKpjXgy0oLFqGbA6g9uzJxDunk8XW6hrldXJSBBcsRvawa+udypMf7yhoM5i+y8Y7w3FMRae8oV6QiuxEOqMTLoMa6Q0yXuSfiU1WI0Cj3Dw8PVi3bUAzg99gDdpC9vjqWzDSyI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1755780171; c=relaxed/simple; bh=TaRH99rt334n31pVTVnJJSZiKxpreeOtyJH2Uj+N3vU=; h=Message-ID:Date:From:To:Cc:Subject:References:MIME-Version: Content-Type; b=NaDjgY0dNONtFO8pqyLBh++c7S2Lyrbd7Oz2hcxMd6LaHgs/P7rqG/P2ckwEFB4ES0xNcpVw4PO3nHVdD29zTLjRsUaJ2QxvJAymA8pE0G4G2R3SWox43ya5tE/a7tyUzjMTC7oU7RoPCx0g0uVgaK2y1A/xas8Xs5AC+JvFB+I= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org; spf=none smtp.mailfrom=infradead.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b=DeWNRATK; arc=none smtp.client-ip=90.155.50.34 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=infradead.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="DeWNRATK" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=v5IBmCyX7FHgg78Lka775Rmw7i2/O/zrHJ4DZcZBDXM=; b=DeWNRATK6vJLgggrQW6WLopcV2 q8/brORqIuusc1quz7WXjwHH/PPPNMNw3ZLUE2W4xRAxMzgkOdWqgwK6j9D1FWevNI2VoE1kNqZrt qPo9HUKOfMJ+S4HD57z6ZiWqFSn2CurOTl42vX4UIIby1sun4/K3c1CssJA2qprhRPCMUoiUTDlzk RF+jfM7Jxdb4HmReLul+BZ738RW0T8oKJ6G1Z3PlJVC88mr2ArAPRFXs7SXMlSYeDihf23AAdT424 /CkRHbd3Ysd8KObKtPpD3ZzJXbypIZe6vAH8dsKqEI+UOTvS6ankSOHh7Mj8Bbbpln45hOM6M5H/F 5W/X/DKg==; Received: from 77-249-17-252.cable.dynamic.v4.ziggo.nl ([77.249.17.252] helo=noisy.programming.kicks-ass.net) by casper.infradead.org with esmtpsa (Exim 4.98.2 #2 (Red Hat Linux)) id 1up4cn-000000075gQ-2lZA; Thu, 21 Aug 2025 12:42:38 +0000 Received: by noisy.programming.kicks-ass.net (Postfix, from userid 0) id 8DCD1302DA9; Thu, 21 Aug 2025 14:42:37 +0200 (CEST) Message-ID: <20250821123656.823296198@infradead.org> User-Agent: quilt/0.68 Date: Thu, 21 Aug 2025 14:28:24 +0200 From: Peter Zijlstra To: jolsa@kernel.org, oleg@redhat.com, andrii@kernel.org, mhiramat@kernel.org Cc: linux-kernel@vger.kernel.org, peterz@infradead.org, alx@kernel.org, eyal.birger@gmail.com, kees@kernel.org, bpf@vger.kernel.org, linux-trace-kernel@vger.kernel.org, x86@kernel.org, songliubraving@fb.com, yhs@fb.com, john.fastabend@gmail.com, haoluo@google.com, rostedt@goodmis.org, alan.maguire@oracle.com, David.Laight@ACULAB.COM, thomas@t-8ch.de, mingo@kernel.org, rick.p.edgecombe@intel.com Subject: [PATCH 2/6] uprobes/x86: Optimize is_optimize() References: <20250821122822.671515652@infradead.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Make is_optimized() return a tri-state and avoid return through argument. This simplifies things a little. Signed-off-by: Peter Zijlstra (Intel) Acked-by: Andrii Nakryiko Reviewed-by: Oleg Nesterov --- arch/x86/kernel/uprobes.c | 34 +++++++++++++--------------------- 1 file changed, 13 insertions(+), 21 deletions(-) --- a/arch/x86/kernel/uprobes.c +++ b/arch/x86/kernel/uprobes.c @@ -1047,7 +1047,7 @@ static bool __is_optimized(uprobe_opcode return __in_uprobe_trampoline(vaddr + 5 + call->raddr); } =20 -static int is_optimized(struct mm_struct *mm, unsigned long vaddr, bool *o= ptimized) +static int is_optimized(struct mm_struct *mm, unsigned long vaddr) { uprobe_opcode_t insn[5]; int err; @@ -1055,8 +1055,7 @@ static int is_optimized(struct mm_struct err =3D copy_from_vaddr(mm, vaddr, &insn, 5); if (err) return err; - *optimized =3D __is_optimized((uprobe_opcode_t *)&insn, vaddr); - return 0; + return __is_optimized((uprobe_opcode_t *)&insn, vaddr); } =20 static bool should_optimize(struct arch_uprobe *auprobe) @@ -1069,17 +1068,14 @@ int set_swbp(struct arch_uprobe *auprobe unsigned long vaddr) { if (should_optimize(auprobe)) { - bool optimized =3D false; - int err; - /* * We could race with another thread that already optimized the probe, * so let's not overwrite it with int3 again in this case. */ - err =3D is_optimized(vma->vm_mm, vaddr, &optimized); - if (err) - return err; - if (optimized) + int ret =3D is_optimized(vma->vm_mm, vaddr); + if (ret < 0) + return ret; + if (ret) return 0; } return uprobe_write_opcode(auprobe, vma, vaddr, UPROBE_SWBP_INSN, @@ -1090,17 +1086,13 @@ int set_orig_insn(struct arch_uprobe *au unsigned long vaddr) { if (test_bit(ARCH_UPROBE_FLAG_CAN_OPTIMIZE, &auprobe->flags)) { - struct mm_struct *mm =3D vma->vm_mm; - bool optimized =3D false; - int err; - - err =3D is_optimized(mm, vaddr, &optimized); - if (err) - return err; - if (optimized) { - err =3D swbp_unoptimize(auprobe, vma, vaddr); - WARN_ON_ONCE(err); - return err; + int ret =3D is_optimized(vma->vm_mm, vaddr); + if (ret < 0) + return ret; + if (ret) { + ret =3D swbp_unoptimize(auprobe, vma, vaddr); + WARN_ON_ONCE(ret); + return ret; } } return uprobe_write_opcode(auprobe, vma, vaddr, *(uprobe_opcode_t *)&aupr= obe->insn, From nobody Sat Oct 4 01:45:00 2025 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3538D320CCC; Thu, 21 Aug 2025 12:42:49 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=90.155.50.34 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1755780171; cv=none; b=ku3EVpWkYJDR2MRy5YF2/i+q5puqQkRfTDAONLiO3xsQDR6pfPA8qug4bjqE9MvzxAdVBNFiz1hNnsTi5XBXueDgl8h/hnRbfNIWKSG2eFOvRTZUsLbdDiLoWZ1xoBwHb7ARaV2en6Ika8D5RW2U7MQ1+oaYI6dufoRE7kUBtoM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1755780171; c=relaxed/simple; bh=DSCxnBqXlAjwBet02afoRqk6AmGkoQ+vOU3nqICgQCg=; h=Message-ID:Date:From:To:Cc:Subject:References:MIME-Version: Content-Type; b=Z5LQqlHm8L7MMLl/ZIpH3q2HOrZ9utzCOOoFkX3Z3JzhYYfvGlSSKng1CL1i4GlSfKw/b3PVwbJ0TdKMJvcGwWm5Gp9Ofdsmhb3UVs0nFUCSVIXVYs3SgmTAgjuzsMBP31Di5nAktbEwFR4M2pkWGzEMQF9W8f5rrQbuHdrN414= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org; spf=none smtp.mailfrom=infradead.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b=JqvNJyoX; arc=none smtp.client-ip=90.155.50.34 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=infradead.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="JqvNJyoX" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=sYpPlJumgdW96JxSCoblEXM9Ih3+Uh4D/QbsVncvLxw=; b=JqvNJyoXGsq7iDrp5ymVPpRqXy 2oL0ohe1mGPge36DHEE5SjraK3DA0fi5zczWoNcnJ/8hb+Y0UL4V3Lzv/GSC+p8LSkZEPqANvCPJX jtnLqF1IfpwoOsX0xMpYVZNGnCgzXgiJ8TTWatAdXfCiGUxjqZy69MpYvXZMsuPqg/pLich4VSK+X MH875ssYb2X5CAKyGVFTvsA9Ip+/p1r8UnJ+jBTiW3Q7Vu2oQ9WxkUo8ypxkUpZXMa1wnjKjrNvwd +BM+WIm5nVd/QIAZnPvikVbu9w0hvfneiBrObHdfq5TGVZuuJXk3nwTkjsZDWaEG5EJudvwpBVLRZ o4viswqw==; Received: from 77-249-17-252.cable.dynamic.v4.ziggo.nl ([77.249.17.252] helo=noisy.programming.kicks-ass.net) by casper.infradead.org with esmtpsa (Exim 4.98.2 #2 (Red Hat Linux)) id 1up4cn-000000075gS-2lF7; Thu, 21 Aug 2025 12:42:38 +0000 Received: by noisy.programming.kicks-ass.net (Postfix, from userid 0) id 91D42302E5F; Thu, 21 Aug 2025 14:42:37 +0200 (CEST) Message-ID: <20250821123656.935559566@infradead.org> User-Agent: quilt/0.68 Date: Thu, 21 Aug 2025 14:28:25 +0200 From: Peter Zijlstra To: jolsa@kernel.org, oleg@redhat.com, andrii@kernel.org, mhiramat@kernel.org Cc: linux-kernel@vger.kernel.org, peterz@infradead.org, alx@kernel.org, eyal.birger@gmail.com, kees@kernel.org, bpf@vger.kernel.org, linux-trace-kernel@vger.kernel.org, x86@kernel.org, songliubraving@fb.com, yhs@fb.com, john.fastabend@gmail.com, haoluo@google.com, rostedt@goodmis.org, alan.maguire@oracle.com, David.Laight@ACULAB.COM, thomas@t-8ch.de, mingo@kernel.org, rick.p.edgecombe@intel.com Subject: [PATCH 3/6] uprobes/x86: Accept more NOP forms References: <20250821122822.671515652@infradead.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Instead of only accepting the x86_64 nop5 chosen by the kernel, accept any x86_64 NOP or NOPL instruction that is 5 bytes. Notably, the x86_64 nop5 pattern is valid in 32bit apps and could get compiler generated when build for i686 (which introduced NOPL). Since the trampoline is x86_64 only, make sure to limit to x86_64 code. Signed-off-by: Peter Zijlstra (Intel) Acked-by: Andrii Nakryiko Reviewed-by: Oleg Nesterov --- arch/x86/kernel/uprobes.c | 37 ++++++++++++++++++++++++++++++++----- 1 file changed, 32 insertions(+), 5 deletions(-) --- a/arch/x86/kernel/uprobes.c +++ b/arch/x86/kernel/uprobes.c @@ -1157,10 +1157,37 @@ void arch_uprobe_optimize(struct arch_up mmap_write_unlock(mm); } =20 -static bool can_optimize(struct arch_uprobe *auprobe, unsigned long vaddr) +static bool insn_is_nop(struct insn *insn) { - if (memcmp(&auprobe->insn, x86_nops[5], 5)) + return insn->opcode.nbytes =3D=3D 1 && insn->opcode.bytes[0] =3D=3D 0x90; +} + +static bool insn_is_nopl(struct insn *insn) +{ + if (insn->opcode.nbytes !=3D 2) + return false; + + if (insn->opcode.bytes[0] !=3D 0x0f || insn->opcode.bytes[1] !=3D 0x1f) + return false; + + if (!insn->modrm.nbytes) + return false; + + if (X86_MODRM_REG(insn->modrm.bytes[0]) !=3D 0) + return false; + + /* 0f 1f /0 - NOPL */ + return true; +} + +static bool can_optimize(struct insn *insn, unsigned long vaddr) +{ + if (!insn->x86_64 || insn->length !=3D 5) return false; + + if (!insn_is_nop(insn) && !insn_is_nopl(insn)) + return false; + /* We can't do cross page atomic writes yet. */ return PAGE_SIZE - (vaddr & ~PAGE_MASK) >=3D 5; } @@ -1177,7 +1204,7 @@ static void riprel_pre_xol(struct arch_u static void riprel_post_xol(struct arch_uprobe *auprobe, struct pt_regs *r= egs) { } -static bool can_optimize(struct arch_uprobe *auprobe, unsigned long vaddr) +static bool can_optimize(struct insn *insn, unsigned long vaddr) { return false; } @@ -1539,15 +1566,15 @@ static int push_setup_xol_ops(struct arc */ int arch_uprobe_analyze_insn(struct arch_uprobe *auprobe, struct mm_struct= *mm, unsigned long addr) { - struct insn insn; u8 fix_ip_or_call =3D UPROBE_FIX_IP; + struct insn insn; int ret; =20 ret =3D uprobe_init_insn(auprobe, &insn, is_64bit_mm(mm)); if (ret) return ret; =20 - if (can_optimize(auprobe, addr)) + if (can_optimize(&insn, addr)) set_bit(ARCH_UPROBE_FLAG_CAN_OPTIMIZE, &auprobe->flags); =20 ret =3D branch_setup_xol_ops(auprobe, &insn); From nobody Sat Oct 4 01:45:00 2025 Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 16CAE320CCB; Thu, 21 Aug 2025 12:42:48 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=90.155.92.199 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1755780172; cv=none; b=htKX7Tdjc/fqKWb/gKVUoABFiRTF3MUbYlxwat3ndkKUYztCofYK+LjXGPp+z1DUMInqByLhZw+47Ft0jecKwmSMF74VR7chvA23tSf3fU2fuwTokQpbhMoK9a8n90iql6DGJlImPzjzjT+3o04aTkJErKYCJOUq9U8jGjcHoYY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1755780172; c=relaxed/simple; bh=oq/yG06XThH0TAVnnFJ3Ji2lCZ4WoreG3a3P5NttsvI=; h=Message-ID:Date:From:To:Cc:Subject:References:MIME-Version: Content-Type; b=pyC1OjTZ95ACHYXHJHNljv0GTB3Z+294u168bzbTt/HQOooyFbKwo/MtswW3f+u767iEI2g3JSfd1gxBvBjY5fRR+t9D2BzUCMmFiRMruEteFehTjfharbCJXdg4ezHZKxSYFeZng5t+/YCLSJXq0LTKfksieNmIWd775hwMGXA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org; spf=none smtp.mailfrom=infradead.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b=d7WUjo5s; arc=none smtp.client-ip=90.155.92.199 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=infradead.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="d7WUjo5s" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=y+aSTnr9180YGHPj/X/hxIPUO484ST3otwkCN+epUbY=; b=d7WUjo5ssFzkM2anLj57PJ+D4R nPKY0rpzTP73PzrnZvFZHl8RAomTWYwmabHUXo4ACjKl8g3HC3HHa4ncKA6goIZ/lAYL+Q+IXr7p4 wLQn6s69LZJz1HdOW+Ac5bb/gDeqmKRZOb+4HEjnS8VXxnw9pEmol3JqYAf/XuSs/GRpf/FmFVISD w5nWcz6KWVzWIOgMHUM70T/uFAZtHL0rTG1HN91PQAXt+62m8Vdtj8GG3JaddJaGbzCaX9tWqYXEf v2t5H3SUqaEBUkfJHn0Gcif2ln7gCIvtH6e/0GDp7NnlPvE01kZ/3rMAYuF5A3K3QQN/i+k0M7vqt dfavWUMw==; Received: from 77-249-17-252.cable.dynamic.v4.ziggo.nl ([77.249.17.252] helo=noisy.programming.kicks-ass.net) by desiato.infradead.org with esmtpsa (Exim 4.98.2 #2 (Red Hat Linux)) id 1up4co-00000000Y9O-1fjZ; Thu, 21 Aug 2025 12:42:38 +0000 Received: by noisy.programming.kicks-ass.net (Postfix, from userid 0) id 95A5C302E7B; Thu, 21 Aug 2025 14:42:37 +0200 (CEST) Message-ID: <20250821123657.055790090@infradead.org> User-Agent: quilt/0.68 Date: Thu, 21 Aug 2025 14:28:26 +0200 From: Peter Zijlstra To: jolsa@kernel.org, oleg@redhat.com, andrii@kernel.org, mhiramat@kernel.org Cc: linux-kernel@vger.kernel.org, peterz@infradead.org, alx@kernel.org, eyal.birger@gmail.com, kees@kernel.org, bpf@vger.kernel.org, linux-trace-kernel@vger.kernel.org, x86@kernel.org, songliubraving@fb.com, yhs@fb.com, john.fastabend@gmail.com, haoluo@google.com, rostedt@goodmis.org, alan.maguire@oracle.com, David.Laight@ACULAB.COM, thomas@t-8ch.de, mingo@kernel.org, rick.p.edgecombe@intel.com Subject: [PATCH 4/6] uprobes/x86: Fix uprobe syscall vs shadow stack References: <20250821122822.671515652@infradead.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" The uprobe syscall stores and strips the trampoline stack frame from the user context, to make it appear similar to an exception at the original instruction. It then restores the trampoline stack when it can exit using sysexit. Make sure to match the regular stack manipulation with shadow stack operations such that regular and shadow stack don't get out of sync and causes trouble. This enables using the optimization when shadow stack is in use. Signed-off-by: Peter Zijlstra (Intel) Acked-by: Andrii Nakryiko Reviewed-by: Oleg Nesterov --- arch/x86/include/asm/shstk.h | 4 ++++ arch/x86/kernel/shstk.c | 40 ++++++++++++++++++++++++++++++++++++++= ++ arch/x86/kernel/uprobes.c | 17 ++++++++--------- 3 files changed, 52 insertions(+), 9 deletions(-) --- a/arch/x86/include/asm/shstk.h +++ b/arch/x86/include/asm/shstk.h @@ -23,6 +23,8 @@ int setup_signal_shadow_stack(struct ksi int restore_signal_shadow_stack(void); int shstk_update_last_frame(unsigned long val); bool shstk_is_enabled(void); +int shstk_pop(u64 *val); +int shstk_push(u64 val); #else static inline long shstk_prctl(struct task_struct *task, int option, unsigned long arg2) { return -EINVAL; } @@ -35,6 +37,8 @@ static inline int setup_signal_shadow_st static inline int restore_signal_shadow_stack(void) { return 0; } static inline int shstk_update_last_frame(unsigned long val) { return 0; } static inline bool shstk_is_enabled(void) { return false; } +static inline int shstk_pop(u64 *val) { return -ENOTSUPP; } +static inline int shstk_push(u64 val) { return -ENOTSUPP; } #endif /* CONFIG_X86_USER_SHADOW_STACK */ =20 #endif /* __ASSEMBLER__ */ --- a/arch/x86/kernel/shstk.c +++ b/arch/x86/kernel/shstk.c @@ -246,6 +246,46 @@ static unsigned long get_user_shstk_addr return ssp; } =20 +int shstk_pop(u64 *val) +{ + int ret =3D 0; + u64 ssp; + + if (!features_enabled(ARCH_SHSTK_SHSTK)) + return -ENOTSUPP; + + fpregs_lock_and_load(); + + rdmsrq(MSR_IA32_PL3_SSP, ssp); + if (val && get_user(*val, (__user u64 *)ssp)) + ret =3D -EFAULT; + else + wrmsrq(MSR_IA32_PL3_SSP, ssp + SS_FRAME_SIZE); + fpregs_unlock(); + + return ret; +} + +int shstk_push(u64 val) +{ + u64 ssp; + int ret; + + if (!features_enabled(ARCH_SHSTK_SHSTK)) + return -ENOTSUPP; + + fpregs_lock_and_load(); + + rdmsrq(MSR_IA32_PL3_SSP, ssp); + ssp -=3D SS_FRAME_SIZE; + ret =3D write_user_shstk_64((__user void *)ssp, val); + if (!ret) + wrmsrq(MSR_IA32_PL3_SSP, ssp); + fpregs_unlock(); + + return ret; +} + #define SHSTK_DATA_BIT BIT(63) =20 static int put_shstk_data(u64 __user *addr, u64 data) --- a/arch/x86/kernel/uprobes.c +++ b/arch/x86/kernel/uprobes.c @@ -804,7 +804,7 @@ SYSCALL_DEFINE0(uprobe) { struct pt_regs *regs =3D task_pt_regs(current); struct uprobe_syscall_args args; - unsigned long ip, sp; + unsigned long ip, sp, sret; int err; =20 /* Allow execution only from uprobe trampolines. */ @@ -831,6 +831,10 @@ SYSCALL_DEFINE0(uprobe) =20 sp =3D regs->sp; =20 + err =3D shstk_pop((u64 *)&sret); + if (err =3D=3D -EFAULT || (!err && sret !=3D args.retaddr)) + goto sigill; + handle_syscall_uprobe(regs, regs->ip); =20 /* @@ -855,6 +859,9 @@ SYSCALL_DEFINE0(uprobe) if (args.retaddr - 5 !=3D regs->ip) args.retaddr =3D regs->ip; =20 + if (shstk_push(args.retaddr) =3D=3D -EFAULT) + goto sigill; + regs->ip =3D ip; =20 err =3D copy_to_user((void __user *)regs->sp, &args, sizeof(args)); @@ -1124,14 +1131,6 @@ void arch_uprobe_optimize(struct arch_up struct mm_struct *mm =3D current->mm; uprobe_opcode_t insn[5]; =20 - /* - * Do not optimize if shadow stack is enabled, the return address hijack - * code in arch_uretprobe_hijack_return_addr updates wrong frame when - * the entry uprobe is optimized and the shadow stack crashes the app. - */ - if (shstk_is_enabled()) - return; - if (!should_optimize(auprobe)) return; From nobody Sat Oct 4 01:45:00 2025 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 35414320CD0; Thu, 21 Aug 2025 12:42:49 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=90.155.50.34 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1755780171; cv=none; b=m3DPv9wKxRI4C9/Owim6W876lHEKSHDsM/DI9ODOCr0hL7GcSc5Da39uDYItl74bfAW+n97NPoubv/nUHTMIV/ERUDaMQ7zcS6OtsVtFWcwKdnrW3IMCqZjxDAVy4zjlsw1L3pS/JOC2H4SiDPI2NUfE7+0A6N9aaiWIT4wVpbs= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1755780171; c=relaxed/simple; bh=GKQWt6HCwxPBe3xBtOLxXC0NieLPfmFa+AwEWkXDNuE=; h=Message-ID:Date:From:To:Cc:Subject:References:MIME-Version: Content-Type; b=cH5vtYJHU3CsHBuvxBkC90KazXBwrw2mhdRKRozuayO58OleAOYddiRibRYOyi4oxGOY1q+2eLTpsa/dlx3UpoHeqOl4h0pkmGrmtsfiAAGedi3FNZHXEHaoXP4elhkLeoF8PGYAM8wvUSDt0vClMDlshE1yibB5zw3fKHIgjmg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org; spf=none smtp.mailfrom=infradead.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b=TNb+GwI3; arc=none smtp.client-ip=90.155.50.34 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=infradead.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="TNb+GwI3" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=yHvX80/S5wM5H42vWbAlSWklnG143m/4pBgyGGy+vjM=; b=TNb+GwI3IqbDOi+JiexIZm0s+s Lsud/ypc5SSHcv4XhB+AAgj/0QmW2GRUh4RkVwmaf0zIK1YyNV8juKGR1QlZKbbA3QGwAD1awUrPq TrWpJezJ+CXRKAin7gLB7geEfbp56ivQssMxgD2kcvRwL+KNwGYnYwmIVL1t7DgDfMdDKtLinox4D 11C3/vIIPNjQSkmo07LEJuUG9QwPsqAYwRMyb1uLo1fWTCWwfuAu0+A+X+300N3AwqUJECHJYhKK1 ngyQc9DlyoCZzpqcAIAWbrsrbhHSWVhU0+SJA8LX93OkUmNL9cW7xwl1pZ2MSYj7JhoTcQP3uBH+E 27v3OvKw==; Received: from 77-249-17-252.cable.dynamic.v4.ziggo.nl ([77.249.17.252] helo=noisy.programming.kicks-ass.net) by casper.infradead.org with esmtpsa (Exim 4.98.2 #2 (Red Hat Linux)) id 1up4co-000000075hO-1RZI; Thu, 21 Aug 2025 12:42:39 +0000 Received: by noisy.programming.kicks-ass.net (Postfix, from userid 0) id 99FF3302EA6; Thu, 21 Aug 2025 14:42:37 +0200 (CEST) Message-ID: <20250821123657.163417243@infradead.org> User-Agent: quilt/0.68 Date: Thu, 21 Aug 2025 14:28:27 +0200 From: Peter Zijlstra To: jolsa@kernel.org, oleg@redhat.com, andrii@kernel.org, mhiramat@kernel.org Cc: linux-kernel@vger.kernel.org, peterz@infradead.org, alx@kernel.org, eyal.birger@gmail.com, kees@kernel.org, bpf@vger.kernel.org, linux-trace-kernel@vger.kernel.org, x86@kernel.org, songliubraving@fb.com, yhs@fb.com, john.fastabend@gmail.com, haoluo@google.com, rostedt@goodmis.org, alan.maguire@oracle.com, David.Laight@ACULAB.COM, thomas@t-8ch.de, mingo@kernel.org, rick.p.edgecombe@intel.com Subject: [PATCH 5/6] uprobes/x86: Make asm style consistent References: <20250821122822.671515652@infradead.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" The asm syntax in uretprobe_trampoline and uprobe_trampoline differs in the use of operand size suffixes. Make them consistent and remove all size suffixes. Signed-off-by: Peter Zijlstra (Intel) Acked-by: Andrii Nakryiko Reviewed-by: Oleg Nesterov --- arch/x86/kernel/uprobes.c | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) --- a/arch/x86/kernel/uprobes.c +++ b/arch/x86/kernel/uprobes.c @@ -321,21 +321,21 @@ asm ( ".pushsection .rodata\n" ".global uretprobe_trampoline_entry\n" "uretprobe_trampoline_entry:\n" - "pushq %rax\n" - "pushq %rcx\n" - "pushq %r11\n" - "movq $" __stringify(__NR_uretprobe) ", %rax\n" + "push %rax\n" + "push %rcx\n" + "push %r11\n" + "mov $" __stringify(__NR_uretprobe) ", %rax\n" "syscall\n" ".global uretprobe_syscall_check\n" "uretprobe_syscall_check:\n" - "popq %r11\n" - "popq %rcx\n" + "pop %r11\n" + "pop %rcx\n" /* * The uretprobe syscall replaces stored %rax value with final * return address, so we don't restore %rax in here and just * call ret. */ - "retq\n" + "ret\n" ".global uretprobe_trampoline_end\n" "uretprobe_trampoline_end:\n" ".popsection\n" @@ -885,7 +885,7 @@ asm ( "push %rcx\n" "push %r11\n" "push %rax\n" - "movq $" __stringify(__NR_uprobe) ", %rax\n" + "mov $" __stringify(__NR_uprobe) ", %rax\n" "syscall\n" "pop %rax\n" "pop %r11\n" From nobody Sat Oct 4 01:45:00 2025 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A6FBD3112D7; Thu, 21 Aug 2025 12:42:48 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=90.155.50.34 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1755780170; cv=none; b=Elo2WXMiIkkfHuyFpqPcnjL/lLBz/odV6I3sq1BdFj2wH2q0YjpCOtZzCNkuqjp2in6AxSmT25SYNjOymS/OgYPj2kvnHDt1T3l7WcpaxDKxTMBJt/Fsn6lbIjBw262gfgcn1UWlIbQj02y2oUUx+ZbsoGjxtLKA4FbUDT3qXy8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1755780170; c=relaxed/simple; bh=diK4GdqSYOrMkc1PLXgYj9NZFB7FD5b1HcE9HTWHCYg=; h=Message-ID:Date:From:To:Cc:Subject:References:MIME-Version: Content-Type; b=IrjjmDDeaDqZcw1WIgBhrakssKaxKYimB/E4RRo58KWmhBCg0PKd/colNroQ68IQNXZILfTj1ukZ2N9YEWrew/rBgv948nW4QsnF5VzLZiSito8jxxslW8Dmjn6t5Lixq6MNqWlTl2K5llYXlOkhGJOr+Ww0wvaK7INYblgiUaw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org; spf=none smtp.mailfrom=infradead.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b=rd2tNdTr; arc=none smtp.client-ip=90.155.50.34 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=infradead.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="rd2tNdTr" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=nNty9DGaBRq1520f2c0+eIPDHwMGQWX/BIKoK9+6HOs=; b=rd2tNdTrVZnnAdrN8chRZLF18T Znxs3JqKnHHcPjUG2fLMN58eitM7CUQzByijcXFGMPNEzByj09bge9RUOd5pBsa3Cqj3DbPMUArwA JBwGlHCQykIL9bg+qIeo5Me5a5Lrx8X3JBymRiLRGsllCI6Iw/E2Kh7pvAxVDIHWuhKGt+CY7jHZC a0GN3UYq4QCfNZPJZ7LTbSl9kxRrtakz0DfIgLipKUwdUZFpqiYqP2XGQypjN4e5MJw8IOh54DUMP HoWaO/dQwk3FXJUrzVXad8+ySjiUF0Qad2otjoXuFs7K5dfk+4hSXWDS8r2hmIHHWCpiZMScwKjMo mN+GRjHg==; Received: from 77-249-17-252.cable.dynamic.v4.ziggo.nl ([77.249.17.252] helo=noisy.programming.kicks-ass.net) by casper.infradead.org with esmtpsa (Exim 4.98.2 #2 (Red Hat Linux)) id 1up4co-000000075hP-1Q0x; Thu, 21 Aug 2025 12:42:39 +0000 Received: by noisy.programming.kicks-ass.net (Postfix, from userid 0) id 9DFFD302EC2; Thu, 21 Aug 2025 14:42:37 +0200 (CEST) Message-ID: <20250821123657.277506098@infradead.org> User-Agent: quilt/0.68 Date: Thu, 21 Aug 2025 14:28:28 +0200 From: Peter Zijlstra To: jolsa@kernel.org, oleg@redhat.com, andrii@kernel.org, mhiramat@kernel.org Cc: linux-kernel@vger.kernel.org, peterz@infradead.org, alx@kernel.org, eyal.birger@gmail.com, kees@kernel.org, bpf@vger.kernel.org, linux-trace-kernel@vger.kernel.org, x86@kernel.org, songliubraving@fb.com, yhs@fb.com, john.fastabend@gmail.com, haoluo@google.com, rostedt@goodmis.org, alan.maguire@oracle.com, David.Laight@ACULAB.COM, thomas@t-8ch.de, mingo@kernel.org, rick.p.edgecombe@intel.com Subject: [PATCH 6/6] uprobes/x86: Add SLS mitigation to the trampolines References: <20250821122822.671515652@infradead.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" It is trivial; no reason not to. Signed-off-by: Peter Zijlstra (Intel) Acked-by: Andrii Nakryiko Acked-by: Masami Hiramatsu (Google) Reviewed-by: Oleg Nesterov --- arch/x86/kernel/uprobes.c | 2 ++ 1 file changed, 2 insertions(+) --- a/arch/x86/kernel/uprobes.c +++ b/arch/x86/kernel/uprobes.c @@ -336,6 +336,7 @@ asm ( * call ret. */ "ret\n" + "int3\n" ".global uretprobe_trampoline_end\n" "uretprobe_trampoline_end:\n" ".popsection\n" @@ -891,6 +892,7 @@ asm ( "pop %r11\n" "pop %rcx\n" "ret\n" + "int3\n" ".balign " __stringify(PAGE_SIZE) "\n" ".popsection\n" );