From nobody Sat Oct 4 22:33:05 2025 Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B34242ECD26 for ; Tue, 12 Aug 2025 10:46:16 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=90.155.92.199 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1754995578; cv=none; b=BrDE8gb2w5Gofdh6YvkDFC0ZFbS8E/jsm/PGp/zytPKhCp34tFaR7DMGA3N+9G34WjwfhZ+qmpeuGsJLyMlEcyzPm24RV7zdFarW8hO68v02NVRxAvoYm4nWiv7ZyMF31DYlFyqCCKA9nh2tU5vhbRWgLdfqstTYKxMMrAargk0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1754995578; c=relaxed/simple; bh=90A0POUVdjRmDsJdSa06F+cFnSb60quQYA/J8S4zWVM=; h=Message-ID:Date:From:To:Cc:Subject:References:MIME-Version: Content-Type; b=UHC0lAGJrKXqaSt44i/nEbncHKQqe3+2nMjYL9SYXO95mhoeUkaHZ5u4/cFOs1PwXfsqMKdEFJqCdbZa7mbanEqq9LuBzWLG6JnGrjSMQ6NVgP4h5XeGGtG1mGVo3Lg3k88hGxdA06Wevmh/eZbBfBX7NyJuYcg6xdZpf1kR0Rc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org; spf=none smtp.mailfrom=infradead.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b=eW78CtEI; arc=none smtp.client-ip=90.155.92.199 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=infradead.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="eW78CtEI" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=1/lNevcxT2DftRRLAkNmAg/Df1IFy3AMCXEmSlENCKY=; b=eW78CtEI4B01OKRS15fRLKNWqP vHNa7u0ZJSoJ8J/f4R7DSioTzoBrRDWId2TH7egNhpBWLqsEFGbaGAYfOmnrZJ5O2ofeoDdNobDQS pRF0vG8dE8gMPobL6+Xhi7lEGbsaNz+9Lhm0ab+DYOnRJS/sGYQeeHXkzcqn9yLIxJKSa3OOV1jod kHO76NdqrIiZ9z8Fhi4HqTcoV2QrLksoxJAs6vvC32vlUmEr3d8SyqvfvEXtfK89L3r9ePRjtamCu bzlnt9mgiKF6qibz+l3P8zLma43jzIT/lIHebLLGrAOZDGl8QtRmdoCi/ziTCeaTG/InpcISMTrUO 9/zfgJgQ==; Received: from 77-249-17-252.cable.dynamic.v4.ziggo.nl ([77.249.17.252] helo=noisy.programming.kicks-ass.net) by desiato.infradead.org with esmtpsa (Exim 4.98.2 #2 (Red Hat Linux)) id 1ulmWA-0000000Fmjx-1n8a; Tue, 12 Aug 2025 10:46:10 +0000 Received: by noisy.programming.kicks-ass.net (Postfix, from userid 0) id 8DAF030049D; Tue, 12 Aug 2025 12:46:09 +0200 (CEST) Message-ID: <20250812104018.424519320@infradead.org> User-Agent: quilt/0.68 Date: Tue, 12 Aug 2025 12:38:59 +0200 From: Peter Zijlstra To: tglx@linutronix.de Cc: linux-kernel@vger.kernel.org, peterz@infradead.org, torvalds@linuxfoundation.org, mingo@kernel.org, namhyung@kernel.org, acme@redhat.com, kees@kernel.org, Lorenzo Stoakes Subject: [PATCH v3 01/15] perf: Remove redundant condition for AUX buffer size References: <20250812103858.234850285@infradead.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Thomas Gleixner It is already checked whether the VMA size is the same as nr_pages * PAGE_SIZE, so later checking both: aux_size =3D=3D vma_size && aux_size =3D=3D nr_pages * PAGE_SIZE is redundant. Remove the vma_size check as nr_pages is what is actually used in the allocation function. That prepares for splitting out the buffer allocation into seperate functions, so that only nr_pages needs to be handed in. No functional change. Signed-off-by: Thomas Gleixner Signed-off-by: Peter Zijlstra (Intel) Reviewed-by: Lorenzo Stoakes Link: https://lkml.kernel.org/r/20250811070620.389961756@linutronix.de --- kernel/events/core.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) --- a/kernel/events/core.c +++ b/kernel/events/core.c @@ -7043,7 +7043,7 @@ static int perf_mmap(struct file *file, if (rb_has_aux(rb) && rb->aux_pgoff !=3D vma->vm_pgoff) goto aux_unlock; =20 - if (aux_size !=3D vma_size || aux_size !=3D nr_pages * PAGE_SIZE) + if (aux_size !=3D nr_pages * PAGE_SIZE) goto aux_unlock; =20 /* already mapped with a different size */ From nobody Sat Oct 4 22:33:05 2025 Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id EAAF12ECD30 for ; Tue, 12 Aug 2025 10:46:16 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=90.155.92.199 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1754995578; cv=none; b=DlUdWZBc7TUbOb8muH1+9757l5Yqrq0lFt0+3KEQxqSsNefm/k14Ya42hBQR2W9N/HCwToZEJXOKhxr9OoUbtKDlGHEBHbfGwhpnaLxjCYpRs42wYC1DlGV4zy6bO0lfBTMDysiyTL7pbi/TYdBXikmGCFQe7wQWbwiQ/FgWqmA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1754995578; c=relaxed/simple; bh=qPsyvgyVam1olZvZ1Um4MzvtbSbI8iOofWKD0HLFGao=; h=Message-ID:Date:From:To:Cc:Subject:References:MIME-Version: Content-Type; b=rwSrcOyIR2xTYP6Ls5uZykutKYurGh/uG9nV4PD9Y7xRQlt1v0vyKiAgx1sPk+WWMadH1hC3DQuLhqMjFxaGnVWdQynMZTTEchOHaGxrBkM7IU95NkxWZDt+0sgCqrE6QBIVun9mo5hDf2pGddIwN1Ff6dP3ArUjpECEQMsKaAY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org; spf=none smtp.mailfrom=infradead.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b=mMMzi/Xn; arc=none smtp.client-ip=90.155.92.199 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=infradead.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="mMMzi/Xn" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=ZY0s9fAhs5DDwQyDEeSs26F4GltZYgjOXs4uAdzONUA=; b=mMMzi/XnaLjETYpNeGG/KKAEUn HFA+fmO3VVBdchc/9gbEB6S2V90xYCjGmr0L+17dYL41yba4rO6zp7D0IRhrLB5gH78FORb2pJlDq FHrCkw7IqtU2GKtF7EX0SeJulC8bPe5jx2qNA4nwNVrlyANTcG72fVL+jsPrReBx5nTlJRrpEyICY hvZr8q0orSeR9j3yAiSPApGwtTrTeHggDiDFfKm7eXHQD58YqCtmAj8VzdnDR3U/Z5pDvjRMHSX7B O8Laf8/d46El77C0S0/qNRhuUU8Edl7+B4I2NToOKye4751L7RWNZK3s4GxrJAl6rcO8E9SOWsOnv WVPsSNfw==; Received: from 77-249-17-252.cable.dynamic.v4.ziggo.nl ([77.249.17.252] helo=noisy.programming.kicks-ass.net) by desiato.infradead.org with esmtpsa (Exim 4.98.2 #2 (Red Hat Linux)) id 1ulmWA-0000000Fmjy-1lhz; Tue, 12 Aug 2025 10:46:10 +0000 Received: by noisy.programming.kicks-ass.net (Postfix, from userid 0) id 927F230050D; Tue, 12 Aug 2025 12:46:09 +0200 (CEST) Message-ID: <20250812104018.541975109@infradead.org> User-Agent: quilt/0.68 Date: Tue, 12 Aug 2025 12:39:00 +0200 From: Peter Zijlstra To: tglx@linutronix.de Cc: linux-kernel@vger.kernel.org, peterz@infradead.org, torvalds@linuxfoundation.org, mingo@kernel.org, namhyung@kernel.org, acme@redhat.com, kees@kernel.org, Lorenzo Stoakes Subject: [PATCH v3 02/15] perf: Split out mlock limit handling References: <20250812103858.234850285@infradead.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Thomas Gleixner To prepare for splitting the buffer allocation out into seperate functions for the ring buffer and the AUX buffer, split out mlock limit handling into a helper function, which can be called from both. No functional change intended. Signed-off-by: Thomas Gleixner Signed-off-by: Peter Zijlstra (Intel) Reviewed-by: Lorenzo Stoakes Link: https://lkml.kernel.org/r/20250811070620.463634790@linutronix.de --- kernel/events/core.c | 77 +++++++++++++++++++++++++---------------------= ----- 1 file changed, 38 insertions(+), 39 deletions(-) --- a/kernel/events/core.c +++ b/kernel/events/core.c @@ -6927,17 +6927,49 @@ static int map_range(struct perf_buffer return err; } =20 +static bool perf_mmap_calc_limits(struct vm_area_struct *vma, long *user_e= xtra, long *extra) +{ + unsigned long user_locked, user_lock_limit, locked, lock_limit; + struct user_struct *user =3D current_user(); + + user_lock_limit =3D sysctl_perf_event_mlock >> (PAGE_SHIFT - 10); + /* Increase the limit linearly with more CPUs */ + user_lock_limit *=3D num_online_cpus(); + + user_locked =3D atomic_long_read(&user->locked_vm); + + /* + * sysctl_perf_event_mlock may have changed, so that + * user->locked_vm > user_lock_limit + */ + if (user_locked > user_lock_limit) + user_locked =3D user_lock_limit; + user_locked +=3D *user_extra; + + if (user_locked > user_lock_limit) { + /* + * charge locked_vm until it hits user_lock_limit; + * charge the rest from pinned_vm + */ + *extra =3D user_locked - user_lock_limit; + *user_extra -=3D *extra; + } + + lock_limit =3D rlimit(RLIMIT_MEMLOCK); + lock_limit >>=3D PAGE_SHIFT; + locked =3D atomic64_read(&vma->vm_mm->pinned_vm) + *extra; + + return locked <=3D lock_limit || !perf_is_paranoid() || capable(CAP_IPC_L= OCK); +} + static int perf_mmap(struct file *file, struct vm_area_struct *vma) { struct perf_event *event =3D file->private_data; - unsigned long user_locked, user_lock_limit; struct user_struct *user =3D current_user(); + unsigned long vma_size, nr_pages; + long user_extra =3D 0, extra =3D 0; struct mutex *aux_mutex =3D NULL; struct perf_buffer *rb =3D NULL; - unsigned long locked, lock_limit; - unsigned long vma_size; - unsigned long nr_pages; - long user_extra =3D 0, extra =3D 0; int ret, flags =3D 0; mapped_f mapped; =20 @@ -7063,38 +7093,7 @@ static int perf_mmap(struct file *file, } } =20 - user_lock_limit =3D sysctl_perf_event_mlock >> (PAGE_SHIFT - 10); - - /* - * Increase the limit linearly with more CPUs: - */ - user_lock_limit *=3D num_online_cpus(); - - user_locked =3D atomic_long_read(&user->locked_vm); - - /* - * sysctl_perf_event_mlock may have changed, so that - * user->locked_vm > user_lock_limit - */ - if (user_locked > user_lock_limit) - user_locked =3D user_lock_limit; - user_locked +=3D user_extra; - - if (user_locked > user_lock_limit) { - /* - * charge locked_vm until it hits user_lock_limit; - * charge the rest from pinned_vm - */ - extra =3D user_locked - user_lock_limit; - user_extra -=3D extra; - } - - lock_limit =3D rlimit(RLIMIT_MEMLOCK); - lock_limit >>=3D PAGE_SHIFT; - locked =3D atomic64_read(&vma->vm_mm->pinned_vm) + extra; - - if ((locked > lock_limit) && perf_is_paranoid() && - !capable(CAP_IPC_LOCK)) { + if (!perf_mmap_calc_limits(vma, &user_extra, &extra)) { ret =3D -EPERM; goto unlock; } From nobody Sat Oct 4 22:33:05 2025 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9196F2EBDC8 for ; Tue, 12 Aug 2025 10:46:13 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=90.155.50.34 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1754995575; cv=none; b=isGSALzIv5SchgzCQANkzfwxf3t6COuYURVofmMPaxEz2GtADs9XmHmlsi9Wu6RDpcsRCeuKl7c6o6DJWQ8ghCLH4TbOgnKly/Lzeb5rwD4dh4c1xOIaLP7wVWj5vDDX9JlXL9pceTZ2/ND1DhxP0iS8kTX7M9Ec3smlgJNeL54= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1754995575; c=relaxed/simple; bh=eRGEL7znStyvargRyOLTDHsdch5WLgEi3BY22hs1CZU=; h=Message-ID:Date:From:To:Cc:Subject:References:MIME-Version: Content-Type; b=EKznou9lA9lhKp+HMwe/BmBCrKa3K8QcTQYEMITjS1OaIhmlLtwTw01kdlW3wLVdn0QAQoZoyLEy6nFO3IFTeTyxVXKsOSZBurMx4j1+UISFMzF4lyJ9W/KbwHigcKSgDy6TWIhO4v2ZxDgxe97zeYlTydTPAl54LACwNEn3IP8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org; spf=none smtp.mailfrom=infradead.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b=s2wpy52W; arc=none smtp.client-ip=90.155.50.34 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=infradead.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="s2wpy52W" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=jEGbx+FK1IDMeSzqEno3xQ+nwzX57brmTQ8dOvKmm0I=; b=s2wpy52WXBGKfcygA2ksOZYXl1 ZtorkhYK5ka4rCvR8/Mi06Z8zrJNtUdqn87aWb9CEc1XdSAECW28ssDfvsmY7MdBil3ahViogQJI7 VcNrIyZ176e8ahDv2qVeeQBABE46hv/IUpxoab9sSC8aNtFUWPXUfFZEDZl8t1+LVSH+owgC1YtJq H45znyIVj9pRDEcNWRm6HCbSSHqUHvAYXHHG4BMKz0/vJnQZTn5rO7knrTHfxhakpFhOpjnjN1vmf 3E5KfCZkeUj/ZA1yKYnjpCHwqao9L9EZGU9HzrsqwK4wRgGc7TUzcR8YKsOXLJaMofwFG1IjbhM+n MWf0VzqQ==; Received: from 77-249-17-252.cable.dynamic.v4.ziggo.nl ([77.249.17.252] helo=noisy.programming.kicks-ass.net) by casper.infradead.org with esmtpsa (Exim 4.98.2 #2 (Red Hat Linux)) id 1ulmW9-0000000FQYf-3Rys; Tue, 12 Aug 2025 10:46:10 +0000 Received: by noisy.programming.kicks-ass.net (Postfix, from userid 0) id 96729300578; Tue, 12 Aug 2025 12:46:09 +0200 (CEST) Message-ID: <20250812104018.660347811@infradead.org> User-Agent: quilt/0.68 Date: Tue, 12 Aug 2025 12:39:01 +0200 From: Peter Zijlstra To: tglx@linutronix.de Cc: linux-kernel@vger.kernel.org, peterz@infradead.org, torvalds@linuxfoundation.org, mingo@kernel.org, namhyung@kernel.org, acme@redhat.com, kees@kernel.org, Lorenzo Stoakes Subject: [PATCH v3 03/15] perf: Split out VM accounting References: <20250812103858.234850285@infradead.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Thomas Gleixner Similary to the mlock limit calculation the VM accounting is required for both the ringbuffer and the AUX buffer allocations. To prepare for splitting them out into seperate functions, move the accounting into a helper function. Signed-off-by: Thomas Gleixner Signed-off-by: Peter Zijlstra (Intel) Reviewed-by: Lorenzo Stoakes Link: https://lkml.kernel.org/r/20250811070620.527392167@linutronix.de --- kernel/events/core.c | 13 +++++++++---- 1 file changed, 9 insertions(+), 4 deletions(-) --- a/kernel/events/core.c +++ b/kernel/events/core.c @@ -6962,10 +6962,17 @@ static bool perf_mmap_calc_limits(struct return locked <=3D lock_limit || !perf_is_paranoid() || capable(CAP_IPC_L= OCK); } =20 +static void perf_mmap_account(struct vm_area_struct *vma, long user_extra,= long extra) +{ + struct user_struct *user =3D current_user(); + + atomic_long_add(user_extra, &user->locked_vm); + atomic64_add(extra, &vma->vm_mm->pinned_vm); +} + static int perf_mmap(struct file *file, struct vm_area_struct *vma) { struct perf_event *event =3D file->private_data; - struct user_struct *user =3D current_user(); unsigned long vma_size, nr_pages; long user_extra =3D 0, extra =3D 0; struct mutex *aux_mutex =3D NULL; @@ -7136,9 +7143,7 @@ static int perf_mmap(struct file *file, =20 unlock: if (!ret) { - atomic_long_add(user_extra, &user->locked_vm); - atomic64_add(extra, &vma->vm_mm->pinned_vm); - + perf_mmap_account(vma, user_extra, extra); atomic_inc(&event->mmap_count); } else if (rb) { /* AUX allocation failed */ From nobody Sat Oct 4 22:33:05 2025 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7B25F2EBBBB for ; Tue, 12 Aug 2025 10:46:13 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=90.155.50.34 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1754995575; cv=none; b=rDfEO6lFoEommfZyqMLOlj4HWpjnaYRMgnZ6JxXOiAi0O24VU6K2hS7KWrQHugV5o8hzRoand9Ljq4y+nVDEc77oRzJjdTnHCR+j3i9K/heFWLQyNcRXn/qT2YOxZGtgcCvkm3F+g8cnebuBJeojKQiBlgWvmNzR2e4vhJPZ6Uw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1754995575; c=relaxed/simple; bh=+GqHX87jWLZqw+t0UZth4U8Z3fRwn4VwUt00AcsKB+g=; h=Message-ID:Date:From:To:Cc:Subject:References:MIME-Version: Content-Type; b=K8l1D7HGr8uBYrw6OTY+hBzNB3LxSGkIJjlmAlWbfr7cvnknxZl8NFDdoUhswSJIQQ8E1dgSDlZkEgrEEYJ7lUlnLxr6/Hws8rBjzo0EVGaPZECYIs6gdue6/nocg710hGOENLj1Qzg6kj7Rod5BxyeA1X4s1j3klWfBqu1Wkow= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org; spf=none smtp.mailfrom=infradead.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b=erqaHiiR; arc=none smtp.client-ip=90.155.50.34 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=infradead.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="erqaHiiR" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=Tnj+ZpbznC9f7uV46Bm+bU4G2ZhL0fK09wKDx1V/NPg=; b=erqaHiiRIFeTNzrEiadSKD0O7d wqjwyqwaiwIw0CJ3Of0PBa+9OPXLY7fEeoj32LdeoqChz6P9zraloS1e0ARcemSAzvrZBosIkKBrN hSS1PD74hFE2EHn5Ms3l0r9/BR65MkGKFLwqbY/OYyiaIV1iV3irsrpHZ6jwGlfxvkVwr1UTiVcl5 eZ4MSITOhzhLlbeYjaTUAtaoWvVgfsp4UF9ZNnpBAzWMATx6exZhqlpyAUmMRg10smnuRRZ3+lV+n X5Y7a4PtU7qNfmIRmIAS5XC81r02HwBp2pA+6Q+BFML9Sg7XH4NHZKVyEa1M2UiwJKdwlBMxp/+u2 INvvx47A==; Received: from 77-249-17-252.cable.dynamic.v4.ziggo.nl ([77.249.17.252] helo=noisy.programming.kicks-ass.net) by casper.infradead.org with esmtpsa (Exim 4.98.2 #2 (Red Hat Linux)) id 1ulmW9-0000000FQYg-3VrD; Tue, 12 Aug 2025 10:46:10 +0000 Received: by noisy.programming.kicks-ass.net (Postfix, from userid 0) id 9A83A300583; Tue, 12 Aug 2025 12:46:09 +0200 (CEST) Message-ID: <20250812104018.781244099@infradead.org> User-Agent: quilt/0.68 Date: Tue, 12 Aug 2025 12:39:02 +0200 From: Peter Zijlstra To: tglx@linutronix.de Cc: linux-kernel@vger.kernel.org, peterz@infradead.org, torvalds@linuxfoundation.org, mingo@kernel.org, namhyung@kernel.org, acme@redhat.com, kees@kernel.org Subject: [PATCH v3 04/15] perf: Move perf_mmap_calc_limits() into both rb and aux branches References: <20250812103858.234850285@infradead.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" if (cond) { A; } else { B; } C; into if (cond) { A; C; } else { B; C; } Suggested-by: Thomas Gleixner Signed-off-by: Peter Zijlstra (Intel) Reviewed-by: Lorenzo Stoakes --- kernel/events/core.c | 26 ++++++++++++++++++-------- 1 file changed, 18 insertions(+), 8 deletions(-) --- a/kernel/events/core.c +++ b/kernel/events/core.c @@ -7054,6 +7054,16 @@ static int perf_mmap(struct file *file, ring_buffer_attach(event, NULL); } =20 + if (!perf_mmap_calc_limits(vma, &user_extra, &extra)) { + ret =3D -EPERM; + goto unlock; + } + + WARN_ON(!rb && event->rb); + + if (vma->vm_flags & VM_WRITE) + flags |=3D RING_BUFFER_WRITABLE; + } else { /* * AUX area mapping: if rb->aux_nr_pages !=3D 0, it's already @@ -7100,17 +7110,17 @@ static int perf_mmap(struct file *file, ret =3D 0; goto unlock; } - } =20 - if (!perf_mmap_calc_limits(vma, &user_extra, &extra)) { - ret =3D -EPERM; - goto unlock; - } + if (!perf_mmap_calc_limits(vma, &user_extra, &extra)) { + ret =3D -EPERM; + goto unlock; + } =20 - WARN_ON(!rb && event->rb); + WARN_ON(!rb && event->rb); =20 - if (vma->vm_flags & VM_WRITE) - flags |=3D RING_BUFFER_WRITABLE; + if (vma->vm_flags & VM_WRITE) + flags |=3D RING_BUFFER_WRITABLE; + } =20 if (!rb) { rb =3D rb_alloc(nr_pages, From nobody Sat Oct 4 22:33:05 2025 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7B1822E7658 for ; Tue, 12 Aug 2025 10:46:13 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=90.155.50.34 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1754995576; cv=none; b=vGGhRF+egvzII/BeNoyEPBeb+Gvbh96TL/EoAEIapapkHgPdoNc+HvRkSIN6O8fNV/EWAaiYERTAvINMQByozkW08s2hSajHpnKk1sj9JgZ2vV9fuxbhicyNjkD1XDvSWWiq/owxdsHDTZkbXma434gRsr888j+YEYAtFWJ9VpU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1754995576; c=relaxed/simple; bh=6jXnp2rS7TSHsIuoRZxdD2f8WwXiq1oxKEXJqcaElw8=; h=Message-ID:Date:From:To:Cc:Subject:References:MIME-Version: Content-Type; b=tTR3HhUyKCCraBlbJ9zYvGu5epkTFZhnzrQ0+ktb3yuzBK/qa4O/kue54WuazJN34pJxqJLy/fdwbs8xImav40yFyB0FawoLLTDNFxD7CXMsnbeYZ+2j7b5xaVdGlBmWHqUC/wl5VJzJVoOwYlp4u5CJRmDx/xZvvlgASsCo0VE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org; spf=none smtp.mailfrom=infradead.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b=VNx85/um; arc=none smtp.client-ip=90.155.50.34 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=infradead.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="VNx85/um" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=fJdG7PKCgL19h78JrV/TWKTJ1AopejIhDsc/rZOBL8E=; b=VNx85/umE5XxKIS0zpGhghKEVk q2sFaHPk+NRP8FTzScZQUM0F2WwWwbXbF2nYHVXHDIJ1ifa3Y/a24lupoLB3IIwapybYIkkRSBlEf DL6SbRhV+bT/g/pb8Rf95XJ5pZD43WX/rNpq7n3vDn+ZQCzi1b5u2HdLAvs8fUdXYr9PDR9MsRM15 p5wzvQfsPKvtlYvlfqc1KxaA095kvo3E/pa8e1frEjlwLRS9Ybksp50v84NTXBHOqaG8A8/HLbv0f soQwXifQeJbAwldqWHfNgIKcOsa+u6gd45M3/sAOpadQuIXqInEBKuxU5CLPK3JxdJIg/M7cLh4EE NWoCw09w==; Received: from 77-249-17-252.cable.dynamic.v4.ziggo.nl ([77.249.17.252] helo=noisy.programming.kicks-ass.net) by casper.infradead.org with esmtpsa (Exim 4.98.2 #2 (Red Hat Linux)) id 1ulmWA-0000000FQZ9-1jPe; Tue, 12 Aug 2025 10:46:11 +0000 Received: by noisy.programming.kicks-ass.net (Postfix, from userid 0) id 9E6EB300708; Tue, 12 Aug 2025 12:46:09 +0200 (CEST) Message-ID: <20250812104018.900078502@infradead.org> User-Agent: quilt/0.68 Date: Tue, 12 Aug 2025 12:39:03 +0200 From: Peter Zijlstra To: tglx@linutronix.de Cc: linux-kernel@vger.kernel.org, peterz@infradead.org, torvalds@linuxfoundation.org, mingo@kernel.org, namhyung@kernel.org, acme@redhat.com, kees@kernel.org Subject: [PATCH v3 05/15] perf: Merge consecutive conditionals in perf_mmap() References: <20250812103858.234850285@infradead.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" if (cond) { A; } else { B; } if (cond) { C; } else { D; } into: if (cond) { A; C; } else { B; D; } Notably the conditions are not identical in form, but are equivalent. Suggested-by: Thomas Gleixner Signed-off-by: Peter Zijlstra (Intel) Reviewed-by: Lorenzo Stoakes --- kernel/events/core.c | 41 +++++++++++++++++++---------------------- 1 file changed, 19 insertions(+), 22 deletions(-) --- a/kernel/events/core.c +++ b/kernel/events/core.c @@ -7064,6 +7064,25 @@ static int perf_mmap(struct file *file, if (vma->vm_flags & VM_WRITE) flags |=3D RING_BUFFER_WRITABLE; =20 + rb =3D rb_alloc(nr_pages, + event->attr.watermark ? event->attr.wakeup_watermark : 0, + event->cpu, flags); + + if (!rb) { + ret =3D -ENOMEM; + goto unlock; + } + + atomic_set(&rb->mmap_count, 1); + rb->mmap_user =3D get_current_user(); + rb->mmap_locked =3D extra; + + ring_buffer_attach(event, rb); + + perf_event_update_time(event); + perf_event_init_userpage(event); + perf_event_update_userpage(event); + ret =3D 0; } else { /* * AUX area mapping: if rb->aux_nr_pages !=3D 0, it's already @@ -7120,29 +7139,7 @@ static int perf_mmap(struct file *file, =20 if (vma->vm_flags & VM_WRITE) flags |=3D RING_BUFFER_WRITABLE; - } =20 - if (!rb) { - rb =3D rb_alloc(nr_pages, - event->attr.watermark ? event->attr.wakeup_watermark : 0, - event->cpu, flags); - - if (!rb) { - ret =3D -ENOMEM; - goto unlock; - } - - atomic_set(&rb->mmap_count, 1); - rb->mmap_user =3D get_current_user(); - rb->mmap_locked =3D extra; - - ring_buffer_attach(event, rb); - - perf_event_update_time(event); - perf_event_init_userpage(event); - perf_event_update_userpage(event); - ret =3D 0; - } else { ret =3D rb_alloc_aux(rb, event, vma->vm_pgoff, nr_pages, event->attr.aux_watermark, flags); if (!ret) { From nobody Sat Oct 4 22:33:05 2025 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7B3542EBBBE for ; Tue, 12 Aug 2025 10:46:13 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=90.155.50.34 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1754995575; cv=none; b=OXkijRafH93LplX01aqVzi7TXhkEIvm0thJMmrPShhOp3fA5CL+/piet1Y136Qiw1MjVOAJ5g3phFN2jMyt6MIZJWqaQ4lHjkL2HtutsXinwY1FBO7FFoc+GbCD+U2ovzHZQ9mpT2hkdsrZzJSrM+fmrPQko3tENQLH35mZoZiI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1754995575; c=relaxed/simple; bh=2ba7Z58vhrDfj/ZlafZhaiDTlHwpRGpfmw/dwXTdt24=; h=Message-ID:Date:From:To:Cc:Subject:References:MIME-Version: Content-Type; b=jstUIulMd1/3LeNRDauaoXOj/UkRBlhIouQesIp7JgnD1MRhrCnKZLsPeblUOFMS7ArsdrNdRK0CP8mIc2in+56uxqKN+JiXgtD/+59+g9oyJDjXJ+TO6BaYvuUjry3jG7A2pScxIlCKsho/KxEpiEHogp3IsFawmQ4S/Cq7sQs= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org; spf=none smtp.mailfrom=infradead.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b=r/36OG1P; arc=none smtp.client-ip=90.155.50.34 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=infradead.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="r/36OG1P" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=ScUPih1zGAlOszyxaH0LrklvJs+SRhmKJ96YIxT4pFQ=; b=r/36OG1P9f57MsYKXNG1BAbjST 4wfVGE5iR+CDKP3B8umTY32VEMuPzk0/0j+NVV7DM6ypZYuOp0RmEin1iwTGavK5Tg4JYm9l3BvBf CR2Xlc7mEUBWsIFuPKIYSkGQ96JWnZUVmcjKjMeEystnQNnvzmCXCDTUY3Y8vXzlYA9ag0PIasvI0 amCvYZjz6cdqY1+lQgah6QKgjhuRol2jMJymRRV7kIYt4DSnXpk7LkFoS5RF+Z2ECy3zOJvE33nds WBoarcI9OEL/eW/4pP1G5CBhLGuPWkrtANP7ce0qGJQA5B22wJ/7CdAPZf3+pwI8MSyESRYIdW5kO YD9fC3uA==; Received: from 77-249-17-252.cable.dynamic.v4.ziggo.nl ([77.249.17.252] helo=noisy.programming.kicks-ass.net) by casper.infradead.org with esmtpsa (Exim 4.98.2 #2 (Red Hat Linux)) id 1ulmWA-0000000FQZB-1mSO; Tue, 12 Aug 2025 10:46:11 +0000 Received: by noisy.programming.kicks-ass.net (Postfix, from userid 0) id A23AA300B53; Tue, 12 Aug 2025 12:46:09 +0200 (CEST) Message-ID: <20250812104019.016252852@infradead.org> User-Agent: quilt/0.68 Date: Tue, 12 Aug 2025 12:39:04 +0200 From: Peter Zijlstra To: tglx@linutronix.de Cc: linux-kernel@vger.kernel.org, peterz@infradead.org, torvalds@linuxfoundation.org, mingo@kernel.org, namhyung@kernel.org, acme@redhat.com, kees@kernel.org Subject: [PATCH v3 06/15] perf: Move common code into both rb and aux branches References: <20250812103858.234850285@infradead.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" if (cond) { A; } else { B; } C; into if (cond) { A; C; } else { B; C; } Suggested-by: Thomas Gleixner Signed-off-by: Peter Zijlstra (Intel) Reviewed-by: Lorenzo Stoakes --- kernel/events/core.c | 25 +++++++++++++++---------- 1 file changed, 15 insertions(+), 10 deletions(-) --- a/kernel/events/core.c +++ b/kernel/events/core.c @@ -7043,6 +7043,8 @@ static int perf_mmap(struct file *file, ret =3D 0; /* We need the rb to map pages. */ rb =3D event->rb; + perf_mmap_account(vma, user_extra, extra); + atomic_inc(&event->mmap_count); goto unlock; } =20 @@ -7083,6 +7085,9 @@ static int perf_mmap(struct file *file, perf_event_init_userpage(event); perf_event_update_userpage(event); ret =3D 0; + + perf_mmap_account(vma, user_extra, extra); + atomic_inc(&event->mmap_count); } else { /* * AUX area mapping: if rb->aux_nr_pages !=3D 0, it's already @@ -7127,11 +7132,12 @@ static int perf_mmap(struct file *file, if (rb_has_aux(rb)) { atomic_inc(&rb->aux_mmap_count); ret =3D 0; - goto unlock; + goto aux_success; } =20 if (!perf_mmap_calc_limits(vma, &user_extra, &extra)) { ret =3D -EPERM; + atomic_dec(&rb->mmap_count); goto unlock; } =20 @@ -7142,20 +7148,19 @@ static int perf_mmap(struct file *file, =20 ret =3D rb_alloc_aux(rb, event, vma->vm_pgoff, nr_pages, event->attr.aux_watermark, flags); - if (!ret) { - atomic_set(&rb->aux_mmap_count, 1); - rb->aux_mmap_locked =3D extra; + if (ret) { + atomic_dec(&rb->mmap_count); + goto unlock; } - } =20 -unlock: - if (!ret) { + atomic_set(&rb->aux_mmap_count, 1); + rb->aux_mmap_locked =3D extra; +aux_success: perf_mmap_account(vma, user_extra, extra); atomic_inc(&event->mmap_count); - } else if (rb) { - /* AUX allocation failed */ - atomic_dec(&rb->mmap_count); } + +unlock: aux_unlock: if (aux_mutex) mutex_unlock(aux_mutex); From nobody Sat Oct 4 22:33:05 2025 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7B3D62EBBBF for ; Tue, 12 Aug 2025 10:46:13 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=90.155.50.34 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1754995575; cv=none; b=HuO+djXBo0jB3nkfc6xHkr7zopc78aNixWlXBD3uba0/by7wIkdX8gr2AwVmfv1Q58rQx++6CwAs8cMn52nRY/UT5UNNBpXmz5R+4x2IBW+sE6qvhDTAi/CoL/wVORp117WCNPSAonRLfPcOxkI0eT+EWkZaIrtWwvdPDYj0QTc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1754995575; c=relaxed/simple; bh=zK7o3HDuWU4BVKW+6lCrUa9WEjTn4oLRiZkoVTcXZOE=; h=Message-ID:Date:From:To:Cc:Subject:References:MIME-Version: Content-Type; b=YZOlj7o+op6pdbaUqXVm3fNfx0Z95JmdGlu4ffRVX3HkJkU75d1wyYa7n/lu6wV5uIYvtilTQXm+eb9PQt3aK3FMlHU3JQYAmIEQksnFU4sGKg56CpqTLNmwrGCXtT0ryOFQz0/MAQ8MulnoFmcjmfM8iEbC3jIcGGFMy4ihb2U= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org; spf=none smtp.mailfrom=infradead.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b=BG4vIY2H; arc=none smtp.client-ip=90.155.50.34 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=infradead.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="BG4vIY2H" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=PNZWAaiFwlYCPNM+xWarlBc9Xsa7ubjKBBskZjPIoTY=; b=BG4vIY2HxeHmMgzal0LZtHl0ta JG89u2/wFNUmXK6WswQHPmNqre6ToOmpY6uxUliY7GAeizKluiLqvoC4lTNwPMj9UYiNBpPBmhhkd indgbpvieVReJr8/TnfmOC3nixRmr0R6tZPSbwBpyDchtL6VWE8h/g2lwe0tlKnev+4F2x8hUUN/X vf7NPbOlFeGos29m0+dq+AFeaSJ6DIx8eXYgTd2F3IKwu6MfBKkvgcD/gXzdY2TX5zEej5nct314Y OeCd/64VMzhOk2wfXmapg0FNFyAq5rjTuVbN3JlHiP/5ou/tJyrZSH35HPBWxa+o1vIrU6sxTU5Rd 7dWTNs2Q==; Received: from 77-249-17-252.cable.dynamic.v4.ziggo.nl ([77.249.17.252] helo=noisy.programming.kicks-ass.net) by casper.infradead.org with esmtpsa (Exim 4.98.2 #2 (Red Hat Linux)) id 1ulmWA-0000000FQZA-1koU; Tue, 12 Aug 2025 10:46:11 +0000 Received: by noisy.programming.kicks-ass.net (Postfix, from userid 0) id A6CC83018AA; Tue, 12 Aug 2025 12:46:09 +0200 (CEST) Message-ID: <20250812104019.131293512@infradead.org> User-Agent: quilt/0.68 Date: Tue, 12 Aug 2025 12:39:05 +0200 From: Peter Zijlstra To: tglx@linutronix.de Cc: linux-kernel@vger.kernel.org, peterz@infradead.org, torvalds@linuxfoundation.org, mingo@kernel.org, namhyung@kernel.org, acme@redhat.com, kees@kernel.org Subject: [PATCH v3 07/15] perf: Remove redundant aux_unlock label References: <20250812103858.234850285@infradead.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" unlock and aux_unlock are now identical, remove the aux_unlock one. Signed-off-by: Peter Zijlstra (Intel) Reviewed-by: Lorenzo Stoakes --- kernel/events/core.c | 17 ++++++++--------- 1 file changed, 8 insertions(+), 9 deletions(-) --- a/kernel/events/core.c +++ b/kernel/events/core.c @@ -7095,7 +7095,7 @@ static int perf_mmap(struct file *file, =20 rb =3D event->rb; if (!rb) - goto aux_unlock; + goto unlock; =20 aux_mutex =3D &rb->aux_mutex; mutex_lock(aux_mutex); @@ -7104,27 +7104,27 @@ static int perf_mmap(struct file *file, aux_size =3D READ_ONCE(rb->user_page->aux_size); =20 if (aux_offset < perf_data_size(rb) + PAGE_SIZE) - goto aux_unlock; + goto unlock; =20 if (aux_offset !=3D vma->vm_pgoff << PAGE_SHIFT) - goto aux_unlock; + goto unlock; =20 /* already mapped with a different offset */ if (rb_has_aux(rb) && rb->aux_pgoff !=3D vma->vm_pgoff) - goto aux_unlock; + goto unlock; =20 if (aux_size !=3D nr_pages * PAGE_SIZE) - goto aux_unlock; + goto unlock; =20 /* already mapped with a different size */ if (rb_has_aux(rb) && rb->aux_nr_pages !=3D nr_pages) - goto aux_unlock; + goto unlock; =20 if (!is_power_of_2(nr_pages)) - goto aux_unlock; + goto unlock; =20 if (!atomic_inc_not_zero(&rb->mmap_count)) - goto aux_unlock; + goto unlock; =20 if (rb_has_aux(rb)) { atomic_inc(&rb->aux_mmap_count); @@ -7158,7 +7158,6 @@ static int perf_mmap(struct file *file, } =20 unlock: -aux_unlock: if (aux_mutex) mutex_unlock(aux_mutex); mutex_unlock(&event->mmap_mutex); From nobody Sat Oct 4 22:33:05 2025 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7B2E22EBBBC for ; Tue, 12 Aug 2025 10:46:13 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=90.155.50.34 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1754995575; cv=none; b=RcLK9CF/mp+OeISAqzm6fwrswmwzHDJsl4zjglltQtaNaPVJwj99v/IeV0Iknsj/BuVFW0jZVq962IDq1WsU4E1xaLtW3LzHBX1mD/Oz9fMjVkElPLFOYbjyjrHiEEpzcnqcGUg7WfX2fBGnhXLA15yPYFDH3H6Z5dGiwe+lWSI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1754995575; c=relaxed/simple; bh=huIGDtcKlRxAlBFW57UooF8d92AXgxH92M8j0MQ6Yo4=; h=Message-ID:Date:From:To:Cc:Subject:References:MIME-Version: Content-Type; b=l0POEbCuC/f6PJ+Kq/5IY+9vHGTm943jrBXlp81Kt33aouwKUED4tyRiCm6EnXsU2iZRJJETHu2YVIIaedewKY4j6DiGxrEYRyU4I9ZCgCwvlZZOmRMfb6GK1Di2zD4UVIcUwCW5ABn5tp0WCoPgJUZrZWufHlLxQxrPcvy0Ono= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org; spf=none smtp.mailfrom=infradead.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b=IrlMrvZt; arc=none smtp.client-ip=90.155.50.34 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=infradead.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="IrlMrvZt" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=vWI8sEWp8on09JLJtm1IzmG6jE0fexrCizV9pTCnHgg=; b=IrlMrvZtpYP3SL0RCR8SHq1EFP TqoaPsLp43RaOgTk+Bjnta/E08ZcEhRwiYfrtwoRMkocXZlG/J9iFU7js2nNOzLRCA7OxNyuA9Eif 0Qmcb20fzSOVmq1x0QIpmbj7uYTZPqjvwaLNGyMmmDgU2QM+aB3MmoMneFB35PcF8e/kcrSLzTOSR HBZ8imjgMkBBGCQKrKR7lTirEVr+xPYK2bVus5r3HbHhtGMJlUSFEphKEPjUsBGPx6+0zJ//2sNqz TPyg6tYGMbUgEERMlXPcmDQE8m5tNth18/ahUSkZBcHqCUztXnYM1IShExLCr4gtm3SwqyCiEIicK IMZCeJMQ==; Received: from 77-249-17-252.cable.dynamic.v4.ziggo.nl ([77.249.17.252] helo=noisy.programming.kicks-ass.net) by casper.infradead.org with esmtpsa (Exim 4.98.2 #2 (Red Hat Linux)) id 1ulmWA-0000000FQZC-1o3b; Tue, 12 Aug 2025 10:46:11 +0000 Received: by noisy.programming.kicks-ass.net (Postfix, from userid 0) id AA887302D9C; Tue, 12 Aug 2025 12:46:09 +0200 (CEST) Message-ID: <20250812104019.246250452@infradead.org> User-Agent: quilt/0.68 Date: Tue, 12 Aug 2025 12:39:06 +0200 From: Peter Zijlstra To: tglx@linutronix.de Cc: linux-kernel@vger.kernel.org, peterz@infradead.org, torvalds@linuxfoundation.org, mingo@kernel.org, namhyung@kernel.org, acme@redhat.com, kees@kernel.org Subject: [PATCH v3 08/15] perf: Use guard() for aux_mutex in perf_mmap() References: <20250812103858.234850285@infradead.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" After duplicating the common code into the rb/aux branches is it possible to use a simple guard() for the aux_mutex. Making the aux branch self-contained. Signed-off-by: Peter Zijlstra (Intel) Reviewed-by: Lorenzo Stoakes --- kernel/events/core.c | 6 +----- 1 file changed, 1 insertion(+), 5 deletions(-) --- a/kernel/events/core.c +++ b/kernel/events/core.c @@ -6975,7 +6975,6 @@ static int perf_mmap(struct file *file, struct perf_event *event =3D file->private_data; unsigned long vma_size, nr_pages; long user_extra =3D 0, extra =3D 0; - struct mutex *aux_mutex =3D NULL; struct perf_buffer *rb =3D NULL; int ret, flags =3D 0; mapped_f mapped; @@ -7098,8 +7097,7 @@ static int perf_mmap(struct file *file, if (!rb) goto unlock; =20 - aux_mutex =3D &rb->aux_mutex; - mutex_lock(aux_mutex); + guard(mutex)(&rb->aux_mutex); =20 aux_offset =3D READ_ONCE(rb->user_page->aux_offset); aux_size =3D READ_ONCE(rb->user_page->aux_size); @@ -7159,8 +7157,6 @@ static int perf_mmap(struct file *file, } =20 unlock: - if (aux_mutex) - mutex_unlock(aux_mutex); mutex_unlock(&event->mmap_mutex); =20 if (ret) From nobody Sat Oct 4 22:33:05 2025 Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 38B322EBDD3 for ; Tue, 12 Aug 2025 10:46:14 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=90.155.92.199 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1754995576; cv=none; b=F6tQMB4KMdtE9q3aoQVtO7gj5xhFe7wEDJz/AuF0SadNrd7gLDNxJD1naxMZsKyjuHSz3led4HSC0cNtvkG+n2ValOu7stk5WFCxdkE6gw9bmp/+leyAAEhmbj5qifqmOoCRt4JV+HyQJqn7iiivq2wGvyyI9LpLeAU7fEWtMEU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1754995576; c=relaxed/simple; bh=9dBKkxgVCaJlHroMXQ5F48AOP2I7LQc+qPDdhwLvAe8=; h=Message-ID:Date:From:To:Cc:Subject:References:MIME-Version: Content-Type; b=Ko6RDWD14NVYg8KD/Rmcwi7IjZY8uCbaszwjZl9Ad8GmY7HzsqH6qsf8j/IZ/7krrCigCczlUR9c83pKQudZZfGYtRosPG2+ygOWOUQZmUydhWjftMItq5kG+lBGON0m/TyGa7LnIugL7/FPoxk6+f/Y8LXV6PFBHR2oGs7V8KU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org; spf=none smtp.mailfrom=infradead.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b=I6+ssZgd; arc=none smtp.client-ip=90.155.92.199 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=infradead.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="I6+ssZgd" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=Hpirb6bCQAZ5IpK55f3SQ1KgVjNuGOPebF5RQWmGUGE=; b=I6+ssZgdFJlAS1khGRuQaaZ2YK Wb4XpWlOoZEliyRajphSdT+U0hyyqJKUzKiHh9Y4v/as8b7OjgNF2RwEZbKx2ZE99IJYz2VRPHjWj xM7lDbJPcgPbr1YsahCm9dXeqvuo+sellOaVqoZDEzaME9VKdB2VPOJ00c8IK/ffySHFSWyG9Vote JHiuG0Mkk0M2tRhLWU2XARxsddNA7ommOZ06V6l7zLlnOsVGNh1rmN7rUHdmfGhBWjPXdNyaidONa bveea0Fo5qy0Vus0xtz11iQpOUpQDIz4v3x2nmBEGx3Q8nNvCtUtCJ0ixg5VHtbD2Z52W2ynvLskz yTuhsDpQ==; Received: from 77-249-17-252.cable.dynamic.v4.ziggo.nl ([77.249.17.252] helo=noisy.programming.kicks-ass.net) by desiato.infradead.org with esmtpsa (Exim 4.98.2 #2 (Red Hat Linux)) id 1ulmWB-0000000Fmk9-1mNf; Tue, 12 Aug 2025 10:46:11 +0000 Received: by noisy.programming.kicks-ass.net (Postfix, from userid 0) id AE653302DA9; Tue, 12 Aug 2025 12:46:09 +0200 (CEST) Message-ID: <20250812104019.362581570@infradead.org> User-Agent: quilt/0.68 Date: Tue, 12 Aug 2025 12:39:07 +0200 From: Peter Zijlstra To: tglx@linutronix.de Cc: linux-kernel@vger.kernel.org, peterz@infradead.org, torvalds@linuxfoundation.org, mingo@kernel.org, namhyung@kernel.org, acme@redhat.com, kees@kernel.org Subject: [PATCH v3 09/15] perf: Reflow to get rid of aux_success label References: <20250812103858.234850285@infradead.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Mostly re-indent noise needed to get rid of that label. Signed-off-by: Peter Zijlstra (Intel) Reviewed-by: Lorenzo Stoakes --- kernel/events/core.c | 37 ++++++++++++++++++------------------- 1 file changed, 18 insertions(+), 19 deletions(-) --- a/kernel/events/core.c +++ b/kernel/events/core.c @@ -7127,30 +7127,29 @@ static int perf_mmap(struct file *file, if (rb_has_aux(rb)) { atomic_inc(&rb->aux_mmap_count); ret =3D 0; - goto aux_success; - } =20 - if (!perf_mmap_calc_limits(vma, &user_extra, &extra)) { - ret =3D -EPERM; - atomic_dec(&rb->mmap_count); - goto unlock; - } + } else { + if (!perf_mmap_calc_limits(vma, &user_extra, &extra)) { + ret =3D -EPERM; + atomic_dec(&rb->mmap_count); + goto unlock; + } =20 - WARN_ON(!rb && event->rb); + WARN_ON(!rb && event->rb); =20 - if (vma->vm_flags & VM_WRITE) - flags |=3D RING_BUFFER_WRITABLE; + if (vma->vm_flags & VM_WRITE) + flags |=3D RING_BUFFER_WRITABLE; =20 - ret =3D rb_alloc_aux(rb, event, vma->vm_pgoff, nr_pages, - event->attr.aux_watermark, flags); - if (ret) { - atomic_dec(&rb->mmap_count); - goto unlock; - } + ret =3D rb_alloc_aux(rb, event, vma->vm_pgoff, nr_pages, + event->attr.aux_watermark, flags); + if (ret) { + atomic_dec(&rb->mmap_count); + goto unlock; + } =20 - atomic_set(&rb->aux_mmap_count, 1); - rb->aux_mmap_locked =3D extra; -aux_success: + atomic_set(&rb->aux_mmap_count, 1); + rb->aux_mmap_locked =3D extra; + } perf_mmap_account(vma, user_extra, extra); atomic_inc(&event->mmap_count); } From nobody Sat Oct 4 22:33:05 2025 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7B4482EBDC1 for ; Tue, 12 Aug 2025 10:46:13 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=90.155.50.34 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1754995575; cv=none; b=akduSGqKv++XXAovp385QBVoLefhE6R7f7ni8GyHPPjRL2FDBrwBLCcWvLpV+vap1MbcbQmUnxFyzeoIhyzCWDfyLOule+kKZbG7jp8iWFRKa6VXd1VprLyE5chED+XgiKoYCjCBA7gBY2fPr/IcITiFzbHd6wMCncMgeLGp8nw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1754995575; c=relaxed/simple; bh=ZE5vynBKnwKYuQNYC3nwGNGwXONkUv7/Rbu9iVfkjz4=; h=Message-ID:Date:From:To:Cc:Subject:References:MIME-Version: Content-Type; b=daRz4APe+6Qp9FOAh6CfgBbICAg8i+1J8SXpiqIIhOhCXxQK2kKVq6TEY8j81oOwZsk9leZECfP3g2HmlwVJiNQ3Y2zrqdJKmUt9pFf9uN1ewj3kmmXVhE2plevaP4SaRg2lXG2Ox8rLbmPMyxyZbgkgcDMvupmEP5tDN85U4Ao= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org; spf=none smtp.mailfrom=infradead.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b=CF42ILGa; arc=none smtp.client-ip=90.155.50.34 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=infradead.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="CF42ILGa" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=MpOltfKavXG/lOsVZUtZqsNzyjSMLe7nbHwuyWd1j9Y=; b=CF42ILGatp97EQ0oy837nlljBU iAgvSCk0z2/810HmRa+lUmpd35wbDof9xwGKbPmY171ILGbKlmvwkLWH2ZoASs25akdNQYEvSE01P JfjJn1keK9bJSqv/5pVSNd8unb1r4kI7AVVJrWNXq7g6/eOxv+mIs76xVqH6N6gxVJ9ZPjMdIkv+9 iER9UjsOxNc2N61csSgsrRHUNKISi/IFeBSlxhb7/Xs0Vq0i9I3gP8aq9s/BcK3orj4orjWGgNHWJ nxUdeSjYxZ9LBYl4f0FLfp9HobFvGvSmY/eJ5EbXw+wLmtgwDuh7aCjJB/Hli9nLcnagOMU0YtnnV Xe9KVZ/w==; Received: from 77-249-17-252.cable.dynamic.v4.ziggo.nl ([77.249.17.252] helo=noisy.programming.kicks-ass.net) by casper.infradead.org with esmtpsa (Exim 4.98.2 #2 (Red Hat Linux)) id 1ulmWA-0000000FQZQ-3Sec; Tue, 12 Aug 2025 10:46:11 +0000 Received: by noisy.programming.kicks-ass.net (Postfix, from userid 0) id B290F302DFC; Tue, 12 Aug 2025 12:46:09 +0200 (CEST) Message-ID: <20250812104019.494205648@infradead.org> User-Agent: quilt/0.68 Date: Tue, 12 Aug 2025 12:39:08 +0200 From: Peter Zijlstra To: tglx@linutronix.de Cc: linux-kernel@vger.kernel.org, peterz@infradead.org, torvalds@linuxfoundation.org, mingo@kernel.org, namhyung@kernel.org, acme@redhat.com, kees@kernel.org Subject: [PATCH v3 10/15] perf: Split out the AUX buffer allocation References: <20250812103858.234850285@infradead.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Move the AUX buffer allocation branch into its own function. Originally-by: Thomas Gleixner Signed-off-by: Peter Zijlstra (Intel) Reviewed-by: Lorenzo Stoakes --- kernel/events/core.c | 144 +++++++++++++++++++++++++++-------------------= ----- 1 file changed, 77 insertions(+), 67 deletions(-) --- a/kernel/events/core.c +++ b/kernel/events/core.c @@ -6970,6 +6970,82 @@ static void perf_mmap_account(struct vm_ atomic64_add(extra, &vma->vm_mm->pinned_vm); } =20 +static int perf_mmap_aux(struct vm_area_struct *vma, struct perf_event *ev= ent, + unsigned long nr_pages) +{ + long extra =3D 0, user_extra =3D nr_pages; + u64 aux_offset, aux_size; + struct perf_buffer *rb; + int ret, rb_flags =3D 0; + + rb =3D event->rb; + if (!rb) + return -EINVAL; + + guard(mutex)(&rb->aux_mutex); + + /* + * AUX area mapping: if rb->aux_nr_pages !=3D 0, it's already + * mapped, all subsequent mappings should have the same size + * and offset. Must be above the normal perf buffer. + */ + aux_offset =3D READ_ONCE(rb->user_page->aux_offset); + aux_size =3D READ_ONCE(rb->user_page->aux_size); + + if (aux_offset < perf_data_size(rb) + PAGE_SIZE) + return -EINVAL; + + if (aux_offset !=3D vma->vm_pgoff << PAGE_SHIFT) + return -EINVAL; + + /* already mapped with a different offset */ + if (rb_has_aux(rb) && rb->aux_pgoff !=3D vma->vm_pgoff) + return -EINVAL; + + if (aux_size !=3D nr_pages * PAGE_SIZE) + return -EINVAL; + + /* already mapped with a different size */ + if (rb_has_aux(rb) && rb->aux_nr_pages !=3D nr_pages) + return -EINVAL; + + if (!is_power_of_2(nr_pages)) + return -EINVAL; + + if (!atomic_inc_not_zero(&rb->mmap_count)) + return -EINVAL; + + if (rb_has_aux(rb)) { + atomic_inc(&rb->aux_mmap_count); + + } else { + if (!perf_mmap_calc_limits(vma, &user_extra, &extra)) { + atomic_dec(&rb->mmap_count); + return -EPERM; + } + + WARN_ON(!rb && event->rb); + + if (vma->vm_flags & VM_WRITE) + rb_flags |=3D RING_BUFFER_WRITABLE; + + ret =3D rb_alloc_aux(rb, event, vma->vm_pgoff, nr_pages, + event->attr.aux_watermark, rb_flags); + if (ret) { + atomic_dec(&rb->mmap_count); + return ret; + } + + atomic_set(&rb->aux_mmap_count, 1); + rb->aux_mmap_locked =3D extra; + } + + perf_mmap_account(vma, user_extra, extra); + atomic_inc(&event->mmap_count); + + return 0; +} + static int perf_mmap(struct file *file, struct vm_area_struct *vma) { struct perf_event *event =3D file->private_data; @@ -7087,73 +7163,7 @@ static int perf_mmap(struct file *file, perf_mmap_account(vma, user_extra, extra); atomic_inc(&event->mmap_count); } else { - /* - * AUX area mapping: if rb->aux_nr_pages !=3D 0, it's already - * mapped, all subsequent mappings should have the same size - * and offset. Must be above the normal perf buffer. - */ - u64 aux_offset, aux_size; - - rb =3D event->rb; - if (!rb) - goto unlock; - - guard(mutex)(&rb->aux_mutex); - - aux_offset =3D READ_ONCE(rb->user_page->aux_offset); - aux_size =3D READ_ONCE(rb->user_page->aux_size); - - if (aux_offset < perf_data_size(rb) + PAGE_SIZE) - goto unlock; - - if (aux_offset !=3D vma->vm_pgoff << PAGE_SHIFT) - goto unlock; - - /* already mapped with a different offset */ - if (rb_has_aux(rb) && rb->aux_pgoff !=3D vma->vm_pgoff) - goto unlock; - - if (aux_size !=3D nr_pages * PAGE_SIZE) - goto unlock; - - /* already mapped with a different size */ - if (rb_has_aux(rb) && rb->aux_nr_pages !=3D nr_pages) - goto unlock; - - if (!is_power_of_2(nr_pages)) - goto unlock; - - if (!atomic_inc_not_zero(&rb->mmap_count)) - goto unlock; - - if (rb_has_aux(rb)) { - atomic_inc(&rb->aux_mmap_count); - ret =3D 0; - - } else { - if (!perf_mmap_calc_limits(vma, &user_extra, &extra)) { - ret =3D -EPERM; - atomic_dec(&rb->mmap_count); - goto unlock; - } - - WARN_ON(!rb && event->rb); - - if (vma->vm_flags & VM_WRITE) - flags |=3D RING_BUFFER_WRITABLE; - - ret =3D rb_alloc_aux(rb, event, vma->vm_pgoff, nr_pages, - event->attr.aux_watermark, flags); - if (ret) { - atomic_dec(&rb->mmap_count); - goto unlock; - } - - atomic_set(&rb->aux_mmap_count, 1); - rb->aux_mmap_locked =3D extra; - } - perf_mmap_account(vma, user_extra, extra); - atomic_inc(&event->mmap_count); + ret =3D perf_mmap_aux(vma, event, nr_pages); } =20 unlock: From nobody Sat Oct 4 22:33:05 2025 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7B1F32EBB97 for ; Tue, 12 Aug 2025 10:46:13 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=90.155.50.34 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1754995575; cv=none; b=s0+UdgJCtFnI2GPs0AGMcDxfocoIe96ml4JG3U/653FUy9FQw4mUmk/dAlSw8b0u6KaNqFxlRZVGM/5C8BcKtQMd8mbl1xGZHKzGVAWc1vYNEHnB1D3I7vb7HVurkt2k91QlNZp2FCNNjfd5HVpxq01hdNvPDbj/5kiVJI0tsaE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1754995575; c=relaxed/simple; bh=OdvD/3vzW42we+neywbF2Wl66RwgbZPElQB80Jczvys=; h=Message-ID:Date:From:To:Cc:Subject:References:MIME-Version: Content-Type; b=ezps5TAIahHBwomJY/mzzzL310uR1LVpLY9dHtkLCnK4KRBXuwpo1SfoRI8XMlHDqHaIEddSWufblWCxwBDEJv4ZT2dtNAMsC1sZXYMgXg/q6K/jX6O7cC41w7OKyA1VWj6MXRed/clSPcNQMbMP+wKZgxE2EOoUrWxQp0h3cK4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org; spf=none smtp.mailfrom=infradead.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b=eqgRjKvx; arc=none smtp.client-ip=90.155.50.34 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=infradead.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="eqgRjKvx" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=ejrb0r/YALyS6aGWfYb+zt6lZSFk7NAHKIFYQaIufYM=; b=eqgRjKvx/gGaahA31h7Aph+E0z wAzgoAcTJsddu7FX/jr2BGi7CDsiEdgBRdFBBdZ5P5kn1Agi3c+xOckoHcbdHe2qPvYLH8bn3Kq/u 1I3BhazDCyFXzLDTZldyMMRcUWgoquLOxmSK0DEt2rTLfM6+ceXhs0FlnukLWdlqdqWQINfdGRE3J jcWitLLc4VUc0zz4HcV/5VyutGMDDvFBhnSl/ZqNAwcTSyNxKU7M092Jx+Iuj+7XswOma5s4Fn0dA 9zt1Cnfy957GQIXZ2Ul7PWEhe0t/74cq5ffc99Knc0DabHfz85ZUb+V4gtynUneOf7lTKGfFQHE+r Q+bNUwHg==; Received: from 77-249-17-252.cable.dynamic.v4.ziggo.nl ([77.249.17.252] helo=noisy.programming.kicks-ass.net) by casper.infradead.org with esmtpsa (Exim 4.98.2 #2 (Red Hat Linux)) id 1ulmWA-0000000FQZS-3Seo; Tue, 12 Aug 2025 10:46:11 +0000 Received: by noisy.programming.kicks-ass.net (Postfix, from userid 0) id B68BF302E34; Tue, 12 Aug 2025 12:46:09 +0200 (CEST) Message-ID: <20250812104019.605285302@infradead.org> User-Agent: quilt/0.68 Date: Tue, 12 Aug 2025 12:39:09 +0200 From: Peter Zijlstra To: tglx@linutronix.de Cc: linux-kernel@vger.kernel.org, peterz@infradead.org, torvalds@linuxfoundation.org, mingo@kernel.org, namhyung@kernel.org, acme@redhat.com, kees@kernel.org Subject: [PATCH v3 11/15] perf: Make RB allocation branch self sufficient References: <20250812103858.234850285@infradead.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Ensure @rb usage doesn't extend out of the branch block. Signed-off-by: Peter Zijlstra (Intel) Reviewed-by: Lorenzo Stoakes --- kernel/events/core.c | 6 +----- 1 file changed, 1 insertion(+), 5 deletions(-) --- a/kernel/events/core.c +++ b/kernel/events/core.c @@ -7116,8 +7116,6 @@ static int perf_mmap(struct file *file, * multiple times. */ ret =3D 0; - /* We need the rb to map pages. */ - rb =3D event->rb; perf_mmap_account(vma, user_extra, extra); atomic_inc(&event->mmap_count); goto unlock; @@ -7136,8 +7134,6 @@ static int perf_mmap(struct file *file, goto unlock; } =20 - WARN_ON(!rb && event->rb); - if (vma->vm_flags & VM_WRITE) flags |=3D RING_BUFFER_WRITABLE; =20 @@ -7190,7 +7186,7 @@ static int perf_mmap(struct file *file, * full cleanup in this case and therefore does not invoke * vmops::close(). */ - ret =3D map_range(rb, vma); + ret =3D map_range(event->rb, vma); if (ret) perf_mmap_close(vma); From nobody Sat Oct 4 22:33:05 2025 Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 38C7E2EBDD5 for ; Tue, 12 Aug 2025 10:46:14 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=90.155.92.199 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1754995576; cv=none; b=kVi3vimJmUNTeCy2MVXlqo9QVfaEnwHLlvM3llxI7uuslnKANsWBvKl1j3T+PFeOYMenhKLVW5iuVhxlmxHfpVdgfgZ6e6NyTup2i6fyk3UpR0ltg+BoYkbKBA0VzfJmFNAQTaTA83T2gkfWp+RkdtpNo5oyMHbs0CL4dyZEmfE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1754995576; c=relaxed/simple; bh=zsUoYfmrwBZufVYd2oSJFDKq0MGsmEWxFkUupdw1aGI=; h=Message-ID:Date:From:To:Cc:Subject:References:MIME-Version: Content-Type; b=lMN2Q0Ry6YG/uTZvgb9lDFjucIh7S1y3lwVp6H/G/Og9ePMFohuGBJVIcjqNO0JVNatSGQOBQNyYBOb1j+JNEH8yNkyDxmULNiJIEALSdyf09Mg+7ixidwkZWWKH+8JLxgSAYX4s8HR5brVvMxKgocn5jcPYCprUoPhA1ZSfxRw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org; spf=none smtp.mailfrom=infradead.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b=jidkELdJ; arc=none smtp.client-ip=90.155.92.199 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=infradead.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="jidkELdJ" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=vH3yePd5DKCVSO1Y/zfCgO448JmB78RevnyS2SQs/eI=; b=jidkELdJqhmcWelCr6GAeNRiln KoG1RQHgfe9N3+rujY9rU0s+6UrY5BHueu5MYqJtZyWenhNvA0eo/AzckxcjcE5ALqBgeZrQqJpxN mcc8FQGZ/tMiVJp/yrvEvXCHeCecBO7oYdwwBeyoFhRVL7bAfqOEP2ZQLuhwxXO1/9wjcElhQLSCJ jOVjF9A/4ggMm33cZNxM7mJOFfer1PbdDOHHgzt3DZogkcqGIinG62G/6NwwrvPP0clhwAQsD+6oW 9Vfde9OVXAj0ko0Xhc4cwTFn2YdDtjo3afSnSetMbVNn4y88bXD7Sp82UTN+aNPDEK0OHPL4ES4RO YQ1SYtgw==; Received: from 77-249-17-252.cable.dynamic.v4.ziggo.nl ([77.249.17.252] helo=noisy.programming.kicks-ass.net) by desiato.infradead.org with esmtpsa (Exim 4.98.2 #2 (Red Hat Linux)) id 1ulmWB-0000000FmkA-21YP; Tue, 12 Aug 2025 10:46:11 +0000 Received: by noisy.programming.kicks-ass.net (Postfix, from userid 0) id BB4F9302E5F; Tue, 12 Aug 2025 12:46:09 +0200 (CEST) Message-ID: <20250812104019.722214699@infradead.org> User-Agent: quilt/0.68 Date: Tue, 12 Aug 2025 12:39:10 +0200 From: Peter Zijlstra To: tglx@linutronix.de Cc: linux-kernel@vger.kernel.org, peterz@infradead.org, torvalds@linuxfoundation.org, mingo@kernel.org, namhyung@kernel.org, acme@redhat.com, kees@kernel.org Subject: [PATCH v3 12/15] perf: Split out the RB allocation References: <20250812103858.234850285@infradead.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Move the RB buffer allocation branch into its own function. Originally-by: Thomas Gleixner Signed-off-by: Peter Zijlstra (Intel) Reviewed-by: Lorenzo Stoakes --- kernel/events/core.c | 145 +++++++++++++++++++++++++---------------------= ----- 1 file changed, 73 insertions(+), 72 deletions(-) --- a/kernel/events/core.c +++ b/kernel/events/core.c @@ -6970,6 +6970,75 @@ static void perf_mmap_account(struct vm_ atomic64_add(extra, &vma->vm_mm->pinned_vm); } =20 +static int perf_mmap_rb(struct vm_area_struct *vma, struct perf_event *eve= nt, + unsigned long nr_pages) +{ + long extra =3D 0, user_extra =3D nr_pages; + struct perf_buffer *rb; + int rb_flags =3D 0; + + nr_pages -=3D 1; + + /* + * If we have rb pages ensure they're a power-of-two number, so we + * can do bitmasks instead of modulo. + */ + if (nr_pages !=3D 0 && !is_power_of_2(nr_pages)) + return -EINVAL; + + WARN_ON_ONCE(event->ctx->parent_ctx); + + if (event->rb) { + if (data_page_nr(event->rb) !=3D nr_pages) + return -EINVAL; + + if (atomic_inc_not_zero(&event->rb->mmap_count)) { + /* + * Success -- managed to mmap() the same buffer + * multiple times. + */ + perf_mmap_account(vma, user_extra, extra); + atomic_inc(&event->mmap_count); + return 0; + } + + /* + * Raced against perf_mmap_close()'s + * atomic_dec_and_mutex_lock() remove the + * event and continue as if !event->rb + */ + ring_buffer_attach(event, NULL); + } + + if (!perf_mmap_calc_limits(vma, &user_extra, &extra)) + return -EPERM; + + if (vma->vm_flags & VM_WRITE) + rb_flags |=3D RING_BUFFER_WRITABLE; + + rb =3D rb_alloc(nr_pages, + event->attr.watermark ? event->attr.wakeup_watermark : 0, + event->cpu, rb_flags); + + if (!rb) + return -ENOMEM; + + atomic_set(&rb->mmap_count, 1); + rb->mmap_user =3D get_current_user(); + rb->mmap_locked =3D extra; + + ring_buffer_attach(event, rb); + + perf_event_update_time(event); + perf_event_init_userpage(event); + perf_event_update_userpage(event); + + perf_mmap_account(vma, user_extra, extra); + atomic_inc(&event->mmap_count); + + return 0; +} + static int perf_mmap_aux(struct vm_area_struct *vma, struct perf_event *ev= ent, unsigned long nr_pages) { @@ -7050,10 +7119,8 @@ static int perf_mmap(struct file *file, { struct perf_event *event =3D file->private_data; unsigned long vma_size, nr_pages; - long user_extra =3D 0, extra =3D 0; - struct perf_buffer *rb =3D NULL; - int ret, flags =3D 0; mapped_f mapped; + int ret; =20 /* * Don't allow mmap() of inherited per-task counters. This would @@ -7079,8 +7146,6 @@ static int perf_mmap(struct file *file, if (vma_size !=3D PAGE_SIZE * nr_pages) return -EINVAL; =20 - user_extra =3D nr_pages; - mutex_lock(&event->mmap_mutex); ret =3D -EINVAL; =20 @@ -7094,74 +7159,10 @@ static int perf_mmap(struct file *file, goto unlock; } =20 - if (vma->vm_pgoff =3D=3D 0) { - nr_pages -=3D 1; - - /* - * If we have rb pages ensure they're a power-of-two number, so we - * can do bitmasks instead of modulo. - */ - if (nr_pages !=3D 0 && !is_power_of_2(nr_pages)) - goto unlock; - - WARN_ON_ONCE(event->ctx->parent_ctx); - - if (event->rb) { - if (data_page_nr(event->rb) !=3D nr_pages) - goto unlock; - - if (atomic_inc_not_zero(&event->rb->mmap_count)) { - /* - * Success -- managed to mmap() the same buffer - * multiple times. - */ - ret =3D 0; - perf_mmap_account(vma, user_extra, extra); - atomic_inc(&event->mmap_count); - goto unlock; - } - - /* - * Raced against perf_mmap_close()'s - * atomic_dec_and_mutex_lock() remove the - * event and continue as if !event->rb - */ - ring_buffer_attach(event, NULL); - } - - if (!perf_mmap_calc_limits(vma, &user_extra, &extra)) { - ret =3D -EPERM; - goto unlock; - } - - if (vma->vm_flags & VM_WRITE) - flags |=3D RING_BUFFER_WRITABLE; - - rb =3D rb_alloc(nr_pages, - event->attr.watermark ? event->attr.wakeup_watermark : 0, - event->cpu, flags); - - if (!rb) { - ret =3D -ENOMEM; - goto unlock; - } - - atomic_set(&rb->mmap_count, 1); - rb->mmap_user =3D get_current_user(); - rb->mmap_locked =3D extra; - - ring_buffer_attach(event, rb); - - perf_event_update_time(event); - perf_event_init_userpage(event); - perf_event_update_userpage(event); - ret =3D 0; - - perf_mmap_account(vma, user_extra, extra); - atomic_inc(&event->mmap_count); - } else { + if (vma->vm_pgoff =3D=3D 0) + ret =3D perf_mmap_rb(vma, event, nr_pages); + else ret =3D perf_mmap_aux(vma, event, nr_pages); - } =20 unlock: mutex_unlock(&event->mmap_mutex); From nobody Sat Oct 4 22:33:05 2025 Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3895A2D59E8 for ; Tue, 12 Aug 2025 10:46:14 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=90.155.92.199 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1754995577; cv=none; b=aVbjmb3Mnf3/Xe6bMZkb+e2nAdTR8J7Qtix+KI5sm2ZqPTt8qF6ZNqRnjrr0X//NAU0ogYzVtsEj8ZuOcvfe6fSi3xkWMgQ+xNt+00jq2bxdguRK3H4vCbaT+W2MYLzYG/8izIG0kPgBty2X63M1mmw7DFLKylWOgcPJ+uXKQaw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1754995577; c=relaxed/simple; bh=mXlOmXXy918rASXvtZIVPDqyYAZmrU42vh530MenKMo=; h=Message-ID:Date:From:To:Cc:Subject:References:MIME-Version: Content-Type; b=UWoDY/XJBTReOLq5Xn3Kb12jXwRddKlO7baJmUiZUFbAJvLNtFhGGJyx2efXi9Knj0JobwnIOGxnPx79doOZ77xRCP6dxyTFukGnsREG3gf8H1VkB5mAyb3Wc9albbNyRYKUciTppFxr6R07LrBoByyj5hwqHLlKU2OATzJ5z2Y= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org; spf=none smtp.mailfrom=infradead.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b=pOu3TFT8; arc=none smtp.client-ip=90.155.92.199 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=infradead.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="pOu3TFT8" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=bxIeDsMrcQcfMnP4YcLQ9R8Rqxr0Wt8Vdq6pJjThJM0=; b=pOu3TFT8+SEFzc/rO/Yb2BXsuw 9kGtXmO0Fvc54BSwh3osSrCnDw3BTjogy5UwzfX+bPVLHczod7nruq5hSI9qHyKO9FySIaOQcAAaG xdtuDA9kRjSqpFoqlYxezQ8bp+X4ILjadZi4s9/ufYS7ZAihbJDrH1XS9UtR6iM+Hm4fjyDsvdiLQ Q4d9c7NVOFbrVBXIyu4bMaRa5P4QUIENMCYiIYn2hvEmac4CvYr8NzO0f5C7JNAwrilcq7ssxACGl Ru3B4Yso3xlu/lL59zbg7as9Iud5BWIZwmctjWdNpgBKy2+k82EwJ9BUGyb2BEZl3m2KD5v8CXO9n mUNx8Rzw==; Received: from 77-249-17-252.cable.dynamic.v4.ziggo.nl ([77.249.17.252] helo=noisy.programming.kicks-ass.net) by desiato.infradead.org with esmtpsa (Exim 4.98.2 #2 (Red Hat Linux)) id 1ulmWB-0000000FmkB-24E3; Tue, 12 Aug 2025 10:46:11 +0000 Received: by noisy.programming.kicks-ass.net (Postfix, from userid 0) id BF171302EA6; Tue, 12 Aug 2025 12:46:09 +0200 (CEST) Message-ID: <20250812104019.838047976@infradead.org> User-Agent: quilt/0.68 Date: Tue, 12 Aug 2025 12:39:11 +0200 From: Peter Zijlstra To: tglx@linutronix.de Cc: linux-kernel@vger.kernel.org, peterz@infradead.org, torvalds@linuxfoundation.org, mingo@kernel.org, namhyung@kernel.org, acme@redhat.com, kees@kernel.org Subject: [PATCH v3 13/15] perf: Use scoped_guard() for mmap_mutex in perf_mmap() References: <20250812103858.234850285@infradead.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Mostly just re-indent noise. Suggested-by: Thomas Gleixner Signed-off-by: Peter Zijlstra (Intel) Reviewed-by: Lorenzo Stoakes --- kernel/events/core.c | 32 ++++++++++++++------------------ 1 file changed, 14 insertions(+), 18 deletions(-) --- a/kernel/events/core.c +++ b/kernel/events/core.c @@ -7143,27 +7143,23 @@ static int perf_mmap(struct file *file, if (vma_size !=3D PAGE_SIZE * nr_pages) return -EINVAL; =20 - mutex_lock(&event->mmap_mutex); - ret =3D -EINVAL; + scoped_guard (mutex, &event->mmap_mutex) { + /* + * This relies on __pmu_detach_event() taking mmap_mutex after marking + * the event REVOKED. Either we observe the state, or __pmu_detach_event= () + * will detach the rb created here. + */ + if (event->state <=3D PERF_EVENT_STATE_REVOKED) { + ret =3D -ENODEV; + break; + } =20 - /* - * This relies on __pmu_detach_event() taking mmap_mutex after marking - * the event REVOKED. Either we observe the state, or __pmu_detach_event() - * will detach the rb created here. - */ - if (event->state <=3D PERF_EVENT_STATE_REVOKED) { - ret =3D -ENODEV; - goto unlock; + if (vma->vm_pgoff =3D=3D 0) + ret =3D perf_mmap_rb(vma, event, nr_pages); + else + ret =3D perf_mmap_aux(vma, event, nr_pages); } =20 - if (vma->vm_pgoff =3D=3D 0) - ret =3D perf_mmap_rb(vma, event, nr_pages); - else - ret =3D perf_mmap_aux(vma, event, nr_pages); - -unlock: - mutex_unlock(&event->mmap_mutex); - if (ret) return ret; From nobody Sat Oct 4 22:33:05 2025 Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 38A9C2EBDD2 for ; Tue, 12 Aug 2025 10:46:14 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=90.155.92.199 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1754995576; cv=none; b=RBqpJTXsXOx4o7foDgO9nJtstCWvwMSxgt+J9OhVgFeKYRBL2cLNeK0NyhDy5rm/abbUY9TMlSaHoWfsOSi95ifVqbQBxq+qBLCH0Vp1rb9HjFFsi0dlgCPLwfQ7xGgnwVBzwmrv61qxnPGcFAZYcsf3StYZbfbVWt+z3rnAOpg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1754995576; c=relaxed/simple; bh=1tsQSdG3yZOrZGZhMvetPvkheLSWSMiaRFMeyD+344g=; h=Message-ID:Date:From:To:Cc:Subject:References:MIME-Version: Content-Type; b=oQU8BphDvVaocWObRs9YRTegKkvkqR27dUa7IH9FuIYKI+/j8wDWhoOTtTXEYHB3c2wROhCNP2YZ49RIRCuLZ8Byv9UA4HdSj1z5cyDbsTGCXPV/M4IdRWOQHJ4TEZCBW3w6q+bs/IH846v4Ila9kzFUJXvqF3lrrKKtOVYWT1g= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org; spf=none smtp.mailfrom=infradead.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b=ioGtXGnO; arc=none smtp.client-ip=90.155.92.199 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=infradead.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="ioGtXGnO" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=p++m+D3TzbqZgFFa4l1hUbPQS1hbYRMWFqRcL82KSek=; b=ioGtXGnORxciG9LHA+ZahVlm93 lsapz6RaQ/xuHK73D0pU54NwGdMvHALWFKWujYviuZ8UslqtR+zePiAgPuKQBCRo2ZKWt5na6M+I6 FrOg0xSFcbViSJhqbCDzqBTJvCQO6mw1hmFX4lHC6At8oK/tc2TsjZ3guniw2vrHZvpspZDdP2TNG Lqvs475kGLNvbgMLFdJjtLQUYdAyUllN+x7WhEjX+iu8BOc+Io17+jYP27Gz2bOrfltBwXGVryCRj V4LMYTtKRW2BdNMVvb26QOG4+iQ+z1Au3QtOOWhAzRRqaj58lrthp1ytbrlJ/n/ktSpAGcxajkPak yjF70r9w==; Received: from 77-249-17-252.cable.dynamic.v4.ziggo.nl ([77.249.17.252] helo=noisy.programming.kicks-ass.net) by desiato.infradead.org with esmtpsa (Exim 4.98.2 #2 (Red Hat Linux)) id 1ulmWB-0000000FmkC-25jQ; Tue, 12 Aug 2025 10:46:11 +0000 Received: by noisy.programming.kicks-ass.net (Postfix, from userid 0) id C4E85302EC2; Tue, 12 Aug 2025 12:46:09 +0200 (CEST) Message-ID: <20250812104019.956479989@infradead.org> User-Agent: quilt/0.68 Date: Tue, 12 Aug 2025 12:39:12 +0200 From: Peter Zijlstra To: tglx@linutronix.de Cc: linux-kernel@vger.kernel.org, peterz@infradead.org, torvalds@linuxfoundation.org, mingo@kernel.org, namhyung@kernel.org, acme@redhat.com, kees@kernel.org Subject: [PATCH v3 14/15] perf: Identify the 0->1 transition for event::mmap_count References: <20250812103858.234850285@infradead.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Needed because refcount_inc() doesn't allow the 0->1 transition. Suggested-by: Thomas Gleixner Signed-off-by: Peter Zijlstra (Intel) Reviewed-by: Lorenzo Stoakes --- kernel/events/core.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) --- a/kernel/events/core.c +++ b/kernel/events/core.c @@ -7033,7 +7033,7 @@ static int perf_mmap_rb(struct vm_area_s perf_event_update_userpage(event); =20 perf_mmap_account(vma, user_extra, extra); - atomic_inc(&event->mmap_count); + atomic_set(&event->mmap_count, 1); =20 return 0; } From nobody Sat Oct 4 22:33:05 2025 Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 38BBB2EBDD4 for ; Tue, 12 Aug 2025 10:46:14 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=90.155.92.199 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1754995576; cv=none; b=CY6Z5HQORumQKSCz0NdUaM9Rr50xnMuMjhNzx7TSZ0TgPuhyXG1jy9tUZH5KF9ZGf3tWWfUYuRIv6iZwNNGmO2UbvrO1JcvuUhm7P503Aiuhhy3gdQrTNAgxa8lgU28ts/y4NZz8+0u1v87NsB0s6hkh+n8mO3M5R+j0B2nhR6k= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1754995576; c=relaxed/simple; bh=Spl98Pg3sChTHHYAUT6cfpeEUlAt9SX1VxXqypPEddA=; h=Message-ID:Date:From:To:Cc:Subject:References:MIME-Version: Content-Type; b=IMdjb+wH5iOQ/cyNi3dNc/ifmVZHS2Dv37leXuNkqEcxRTCj2gy3zg8kD7LBf9jMguARcPzuZZaM6rBDscAeJPOZIRXMhwSW4y4ljKdyqDA/Sjusx/q2Een+rfpsLn1041L0XMyzqEyWj+fJ6fWCOx0FMYrDRhrowH34DcGH/lg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org; spf=none smtp.mailfrom=infradead.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b=n9hfjeRj; arc=none smtp.client-ip=90.155.92.199 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=infradead.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="n9hfjeRj" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=YWkfcPUrzQAlnpEnb8Fccs5Qm2U3D45YEUaPiQ4jQPo=; b=n9hfjeRj/F48Ut3LQ0pAu4jGP1 w1HfvQY0UuYzUEDzYmq2MAqMv6Q6AWdKNeITohjlEWCcIo1rrNjpuEEO/IMrQxAuag/LtWJQ56qUo NUo2Mq3DrFrCecq7OwzlpYwjGPgbtt+CvOhlT1fNC86EExyIOzxcWv0gGRZU7dB3QIFgu0fHkrMbf ODYvRpmtb6RSbtDmYJNrncRFz3qYVpHRt8o0u5osdoAwuAsBf97o+5jBEuxRFmfRWXA/x4Fh9devX QYeWE7N6N2dFQUUnflZabtOLFWxuBDrc6XaGONu0VK5mFNCNs/4AJDwgSJkPHXtvYgTzZUssHc3Aw CzlJv4Gw==; Received: from 77-249-17-252.cable.dynamic.v4.ziggo.nl ([77.249.17.252] helo=noisy.programming.kicks-ass.net) by desiato.infradead.org with esmtpsa (Exim 4.98.2 #2 (Red Hat Linux)) id 1ulmWB-0000000FmkD-2yBv; Tue, 12 Aug 2025 10:46:12 +0000 Received: by noisy.programming.kicks-ass.net (Postfix, from userid 0) id CA184302EEA; Tue, 12 Aug 2025 12:46:09 +0200 (CEST) Message-ID: <20250812104020.071507932@infradead.org> User-Agent: quilt/0.68 Date: Tue, 12 Aug 2025 12:39:13 +0200 From: Peter Zijlstra To: tglx@linutronix.de Cc: linux-kernel@vger.kernel.org, peterz@infradead.org, torvalds@linuxfoundation.org, mingo@kernel.org, namhyung@kernel.org, acme@redhat.com, kees@kernel.org Subject: [PATCH v3 15/15] perf: Convert mmap() refcounts to refcount_t References: <20250812103858.234850285@infradead.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Thomas Gleixner The recently fixed reference count leaks could have been detected by using refcount_t and refcount_t would have mitigated the potential overflow at least. Now that the code is properly structured, convert the mmap() related mmap_count variants over to refcount_t. No functional change intended. Signed-off-by: Thomas Gleixner Signed-off-by: Peter Zijlstra (Intel) Link: https://lkml.kernel.org/r/20250811070620.716309215@linutronix.de Reviewed-by: Lorenzo Stoakes --- include/linux/perf_event.h | 2 +- kernel/events/core.c | 40 ++++++++++++++++++++-------------------- kernel/events/internal.h | 4 ++-- kernel/events/ring_buffer.c | 2 +- 4 files changed, 24 insertions(+), 24 deletions(-) --- a/include/linux/perf_event.h +++ b/include/linux/perf_event.h @@ -859,7 +859,7 @@ struct perf_event { =20 /* mmap bits */ struct mutex mmap_mutex; - atomic_t mmap_count; + refcount_t mmap_count; =20 struct perf_buffer *rb; struct list_head rb_entry; --- a/kernel/events/core.c +++ b/kernel/events/core.c @@ -3968,7 +3968,7 @@ static noinline int visit_groups_merge(s */ static inline bool event_update_userpage(struct perf_event *event) { - if (likely(!atomic_read(&event->mmap_count))) + if (likely(!refcount_read(&event->mmap_count))) return false; =20 perf_event_update_time(event); @@ -6704,11 +6704,11 @@ static void perf_mmap_open(struct vm_are struct perf_event *event =3D vma->vm_file->private_data; mapped_f mapped =3D get_mapped(event, event_mapped); =20 - atomic_inc(&event->mmap_count); - atomic_inc(&event->rb->mmap_count); + refcount_inc(&event->mmap_count); + refcount_inc(&event->rb->mmap_count); =20 if (vma->vm_pgoff) - atomic_inc(&event->rb->aux_mmap_count); + refcount_inc(&event->rb->aux_mmap_count); =20 if (mapped) mapped(event, vma->vm_mm); @@ -6743,7 +6743,7 @@ static void perf_mmap_close(struct vm_ar * to avoid complications. */ if (rb_has_aux(rb) && vma->vm_pgoff =3D=3D rb->aux_pgoff && - atomic_dec_and_mutex_lock(&rb->aux_mmap_count, &rb->aux_mutex)) { + refcount_dec_and_mutex_lock(&rb->aux_mmap_count, &rb->aux_mutex)) { /* * Stop all AUX events that are writing to this buffer, * so that we can free its AUX pages and corresponding PMU @@ -6763,10 +6763,10 @@ static void perf_mmap_close(struct vm_ar mutex_unlock(&rb->aux_mutex); } =20 - if (atomic_dec_and_test(&rb->mmap_count)) + if (refcount_dec_and_test(&rb->mmap_count)) detach_rest =3D true; =20 - if (!atomic_dec_and_mutex_lock(&event->mmap_count, &event->mmap_mutex)) + if (!refcount_dec_and_mutex_lock(&event->mmap_count, &event->mmap_mutex)) goto out_put; =20 ring_buffer_attach(event, NULL); @@ -6992,19 +6992,19 @@ static int perf_mmap_rb(struct vm_area_s if (data_page_nr(event->rb) !=3D nr_pages) return -EINVAL; =20 - if (atomic_inc_not_zero(&event->rb->mmap_count)) { + if (refcount_inc_not_zero(&event->rb->mmap_count)) { /* * Success -- managed to mmap() the same buffer * multiple times. */ perf_mmap_account(vma, user_extra, extra); - atomic_inc(&event->mmap_count); + refcount_inc(&event->mmap_count); return 0; } =20 /* * Raced against perf_mmap_close()'s - * atomic_dec_and_mutex_lock() remove the + * refcount_dec_and_mutex_lock() remove the * event and continue as if !event->rb */ ring_buffer_attach(event, NULL); @@ -7023,7 +7023,7 @@ static int perf_mmap_rb(struct vm_area_s if (!rb) return -ENOMEM; =20 - atomic_set(&rb->mmap_count, 1); + refcount_set(&rb->mmap_count, 1); rb->mmap_user =3D get_current_user(); rb->mmap_locked =3D extra; =20 @@ -7034,7 +7034,7 @@ static int perf_mmap_rb(struct vm_area_s perf_event_update_userpage(event); =20 perf_mmap_account(vma, user_extra, extra); - atomic_set(&event->mmap_count, 1); + refcount_set(&event->mmap_count, 1); =20 return 0; } @@ -7081,15 +7081,15 @@ static int perf_mmap_aux(struct vm_area_ if (!is_power_of_2(nr_pages)) return -EINVAL; =20 - if (!atomic_inc_not_zero(&rb->mmap_count)) + if (!refcount_inc_not_zero(&rb->mmap_count)) return -EINVAL; =20 if (rb_has_aux(rb)) { - atomic_inc(&rb->aux_mmap_count); + refcount_inc(&rb->aux_mmap_count); =20 } else { if (!perf_mmap_calc_limits(vma, &user_extra, &extra)) { - atomic_dec(&rb->mmap_count); + refcount_dec(&rb->mmap_count); return -EPERM; } =20 @@ -7101,16 +7101,16 @@ static int perf_mmap_aux(struct vm_area_ ret =3D rb_alloc_aux(rb, event, vma->vm_pgoff, nr_pages, event->attr.aux_watermark, rb_flags); if (ret) { - atomic_dec(&rb->mmap_count); + refcount_dec(&rb->mmap_count); return ret; } =20 - atomic_set(&rb->aux_mmap_count, 1); + refcount_set(&rb->aux_mmap_count, 1); rb->aux_mmap_locked =3D extra; } =20 perf_mmap_account(vma, user_extra, extra); - atomic_inc(&event->mmap_count); + refcount_inc(&event->mmap_count); =20 return 0; } @@ -13257,7 +13257,7 @@ perf_event_set_output(struct perf_event mutex_lock_double(&event->mmap_mutex, &output_event->mmap_mutex); set: /* Can't redirect output if we've got an active mmap() */ - if (atomic_read(&event->mmap_count)) + if (refcount_read(&event->mmap_count)) goto unlock; =20 if (output_event) { @@ -13270,7 +13270,7 @@ perf_event_set_output(struct perf_event goto unlock; =20 /* did we race against perf_mmap_close() */ - if (!atomic_read(&rb->mmap_count)) { + if (!refcount_read(&rb->mmap_count)) { ring_buffer_put(rb); goto unlock; } --- a/kernel/events/internal.h +++ b/kernel/events/internal.h @@ -35,7 +35,7 @@ struct perf_buffer { spinlock_t event_lock; struct list_head event_list; =20 - atomic_t mmap_count; + refcount_t mmap_count; unsigned long mmap_locked; struct user_struct *mmap_user; =20 @@ -47,7 +47,7 @@ struct perf_buffer { unsigned long aux_pgoff; int aux_nr_pages; int aux_overwrite; - atomic_t aux_mmap_count; + refcount_t aux_mmap_count; unsigned long aux_mmap_locked; void (*free_aux)(void *); refcount_t aux_refcount; --- a/kernel/events/ring_buffer.c +++ b/kernel/events/ring_buffer.c @@ -400,7 +400,7 @@ void *perf_aux_output_begin(struct perf_ * the same order, see perf_mmap_close. Otherwise we end up freeing * aux pages in this path, which is a bug, because in_atomic(). */ - if (!atomic_read(&rb->aux_mmap_count)) + if (!refcount_read(&rb->aux_mmap_count)) goto err; =20 if (!refcount_inc_not_zero(&rb->aux_refcount))