From nobody Sun Oct 5 07:20:29 2025 Received: from galois.linutronix.de (Galois.linutronix.de [193.142.43.55]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 63738218851 for ; Wed, 6 Aug 2025 20:12:57 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=193.142.43.55 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1754511179; cv=none; b=nKIeRUHPdEXBFTs4WScZYuvTh2QAXlgdJXJuFQDzvrxJm6hGq79TiKM9/LB5qqX1D9bYlIXEoUlDjB2lkYUvsqjyX1g10pz9tK8/NXID8UHDmaPpHBCUzMLPMHKRMndZb+w5xb+U1W+zHaN9mFQ0aQjHYeGQYoJtszb2HGC/EqY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1754511179; c=relaxed/simple; bh=ezyusppn6TC+HKOYmKP2XK4z1E2hbCIZce901l6bTzk=; h=Message-ID:From:To:Cc:Subject:References:MIME-Version: Content-Type:Date; b=o43RmqUxahmvobWnTVMFk/lYJ9iQDdOuSvKuVOoR4iO0Ki8m5hyebj6RxmTA5b3dfeiwo79fDNr30kQm/fspLmyn6d5L65WtnwafTAAWk3Xh6GwOTAK5X8ivQV8SRpHO6M62jYQTHD51Qg2l7jQCJE4ljHkEY3i9XHhhwOJQjxs= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linutronix.de; spf=pass smtp.mailfrom=linutronix.de; dkim=pass (2048-bit key) header.d=linutronix.de header.i=@linutronix.de header.b=jR//06CK; dkim=permerror (0-bit key) header.d=linutronix.de header.i=@linutronix.de header.b=GtfCjCIk; arc=none smtp.client-ip=193.142.43.55 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linutronix.de Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linutronix.de Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="jR//06CK"; dkim=permerror (0-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="GtfCjCIk" Message-ID: <20250806200617.258754928@linutronix.de> DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1754511174; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: references:references; bh=xtCc47izxKFF4G1h9XrAXBPLX8fBReX2herfs+JeM0A=; b=jR//06CKnmW1mmlrDIUAdr5S42M0JlttdXpNhXWrnQ7dnpDZ7RoeXFL0Qbq7ALukRxP++i RYqU97fXaxH7i4DFNQIM4Le/nsO8QlDTdCLnZx7lJJAa6dU5VtHm9wcCSvFhAYl8T+J/Ys OLe5Wb+mkWY5fTNM9tGDdlY4onjC7apKMNY4XgXOlrRB5hkdzSrFrW11FwF8kTDfCsn+Vp j7AyuiV6BSQD6/jDVV2KGq00cBBItzroxbprpahsrY+mO8mny/bhm1dkR8dytcSStwYmN2 KJjwI0r4ZcoVc/oAwWwaeP0CljMKK7scg8oKg1k9VA/1OMJeGT+wgonRm6AVCw== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1754511174; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: references:references; bh=xtCc47izxKFF4G1h9XrAXBPLX8fBReX2herfs+JeM0A=; b=GtfCjCIkzB+YHk//9Z3OSgPBPGCVPlM+yQBQplq6xek3xpfnKCxF3tMyWfJUL3uMAIn6YD PlJi7Kp8uEGviFDQ== From: Thomas Gleixner To: LKML Cc: Linus Torvalds , Peter Zijlstra , Ingo Molnar , Namhyung Kim , Arnaldo Carvalho de Melo , Lorenzo Stoakes , Kees Cook Subject: [patch 1/6] perf/core: Remove redundant condition for AUX buffer size References: <20250806195624.880096284@linutronix.de> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Date: Wed, 6 Aug 2025 22:12:54 +0200 (CEST) Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" It is already checked whether the VMA size is the same as nr_pages * PAGE_SIZE, so later checking both: aux_size =3D=3D vma_size && aux_size =3D=3D nr_pages * PAGE_SIZE is redundant. Remove the vma_size check as nr_pages is what is actually used in the allocation function. That prepares for splitting out the buffer allocation into seperate functions, so that only nr_pages needs to be handed in. No functional change. Signed-off-by: Thomas Gleixner Reviewed-by: Lorenzo Stoakes --- kernel/events/core.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) --- a/kernel/events/core.c +++ b/kernel/events/core.c @@ -7043,7 +7043,7 @@ static int perf_mmap(struct file *file, if (rb_has_aux(rb) && rb->aux_pgoff !=3D vma->vm_pgoff) goto aux_unlock; =20 - if (aux_size !=3D vma_size || aux_size !=3D nr_pages * PAGE_SIZE) + if (aux_size !=3D nr_pages * PAGE_SIZE) goto aux_unlock; =20 /* already mapped with a different size */ From nobody Sun Oct 5 07:20:29 2025 Received: from galois.linutronix.de (Galois.linutronix.de [193.142.43.55]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 636DC217704 for ; Wed, 6 Aug 2025 20:12:58 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=193.142.43.55 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1754511180; cv=none; b=JX4r6RJid/BY071O1sAYR8yoL/6HhAT9NU2WUhnKQK3e0SNLIe1pQexT9r2Rx3HRoyfyISkRDI9PX+FR/+vSXNoLuR/K0brCWyumDReslF2sUdRF2rCXWD8H5zpxfMhDUTjIGMePr1t+5pueqlDGRbZeIzYY4TzcoWWhXY++6Mk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1754511180; c=relaxed/simple; bh=72vMnJvciZSPM/VhGoT+SeCzUW0TeuZSkTad6DQiHv0=; h=Message-ID:From:To:Cc:Subject:References:MIME-Version: Content-Type:Date; b=iBTrEbsptMipsWFdtLS6bbYztBDqtq66YOG46udiGlqBru0zA4Ind0/CxS52i3it78bSjXlUkK7MV6Ra+SSN6sUeMFHwB5t1gn2qTw/ldvQ7RQESAM8hdx/qfUruZR4XeonifrXjZ3/ky+rH1/SGg736BH2Q6DNemdX1eWPPb8U= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linutronix.de; spf=pass smtp.mailfrom=linutronix.de; dkim=pass (2048-bit key) header.d=linutronix.de header.i=@linutronix.de header.b=zJcYxnG0; dkim=permerror (0-bit key) header.d=linutronix.de header.i=@linutronix.de header.b=E+dj7hFD; arc=none smtp.client-ip=193.142.43.55 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linutronix.de Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linutronix.de Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="zJcYxnG0"; dkim=permerror (0-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="E+dj7hFD" Message-ID: <20250806200617.323157553@linutronix.de> DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1754511176; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: references:references; bh=mpYOPyyu2yqbbjhSAYsp7OV9dua6EsvwR+28yq/NgKY=; b=zJcYxnG043EexAcCCW1s3hQHDqCUZLaMP74Tp7wL5C0WGczMLgm//qNQMwcuczBuKbcort gyCWVYuWCJ+QENouB7OKqQUsu7Gi1X/GMrBlTOcRWZuw5foywwmdEC90CkLax05xWfyw5y JKUgmnlgVsFQsFqG+M9+x/7HiBF+8eVkSa39sNGvht4V7QNikpiJL6CoSqcyRCFDTEvb1c 3RWRu1abE/GSRHIRIgEESCM8KcZoRhjZN1WNFX01taqI7fSU+bs1A/wnkJqT/pqnb9FkLA dOVyBEmaBCyZQGXOitddQG60Vn1stgoUf6jmgGr5qhbnivmRh2bDpeKKG5c5cw== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1754511176; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: references:references; bh=mpYOPyyu2yqbbjhSAYsp7OV9dua6EsvwR+28yq/NgKY=; b=E+dj7hFDiNOWLiDFheUf87NFwW1qwLrhgcB5dGyV15Fh3INUgHuHWc6MI6qqHTM02RK+TY YbVXJjH89TC7h+AQ== From: Thomas Gleixner To: LKML Cc: Linus Torvalds , Peter Zijlstra , Ingo Molnar , Namhyung Kim , Arnaldo Carvalho de Melo , Lorenzo Stoakes , Kees Cook Subject: [patch 2/6] perf/core: Split out mlock limit handling References: <20250806195624.880096284@linutronix.de> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Date: Wed, 6 Aug 2025 22:12:55 +0200 (CEST) Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" To prepare for splitting the buffer allocation out into seperate functions for the ring buffer and the AUX buffer, split out mlock limit handling into a helper function, which can be called from both. No functional change intended. Signed-off-by: Thomas Gleixner Reviewed-by: Lorenzo Stoakes --- kernel/events/core.c | 75 +++++++++++++++++++++++++---------------------= ----- 1 file changed, 38 insertions(+), 37 deletions(-) --- a/kernel/events/core.c +++ b/kernel/events/core.c @@ -6927,17 +6927,49 @@ static int map_range(struct perf_buffer return err; } =20 +static bool perf_mmap_calc_limits(struct vm_area_struct *vma, long *user_e= xtra, long *extra) +{ + unsigned long user_locked, user_lock_limit, locked, lock_limit; + struct user_struct *user =3D current_user(); + + user_lock_limit =3D sysctl_perf_event_mlock >> (PAGE_SHIFT - 10); + /* Increase the limit linearly with more CPUs */ + user_lock_limit *=3D num_online_cpus(); + + user_locked =3D atomic_long_read(&user->locked_vm); + + /* + * sysctl_perf_event_mlock may have changed, so that + * user->locked_vm > user_lock_limit + */ + if (user_locked > user_lock_limit) + user_locked =3D user_lock_limit; + user_locked +=3D *user_extra; + + if (user_locked > user_lock_limit) { + /* + * charge locked_vm until it hits user_lock_limit; + * charge the rest from pinned_vm + */ + *extra =3D user_locked - user_lock_limit; + *user_extra -=3D *extra; + } + + lock_limit =3D rlimit(RLIMIT_MEMLOCK); + lock_limit >>=3D PAGE_SHIFT; + locked =3D atomic64_read(&vma->vm_mm->pinned_vm) + *extra; + + return locked <=3D lock_limit || !perf_is_paranoid() || capable(CAP_IPC_L= OCK); +} + static int perf_mmap(struct file *file, struct vm_area_struct *vma) { struct perf_event *event =3D file->private_data; - unsigned long user_locked, user_lock_limit; struct user_struct *user =3D current_user(); + unsigned long vma_size, nr_pages; + long user_extra =3D 0, extra =3D 0; struct mutex *aux_mutex =3D NULL; struct perf_buffer *rb =3D NULL; - unsigned long locked, lock_limit; - unsigned long vma_size; - unsigned long nr_pages; - long user_extra =3D 0, extra =3D 0; int ret, flags =3D 0; mapped_f mapped; =20 @@ -7063,38 +7095,7 @@ static int perf_mmap(struct file *file, } } =20 - user_lock_limit =3D sysctl_perf_event_mlock >> (PAGE_SHIFT - 10); - - /* - * Increase the limit linearly with more CPUs: - */ - user_lock_limit *=3D num_online_cpus(); - - user_locked =3D atomic_long_read(&user->locked_vm); - - /* - * sysctl_perf_event_mlock may have changed, so that - * user->locked_vm > user_lock_limit - */ - if (user_locked > user_lock_limit) - user_locked =3D user_lock_limit; - user_locked +=3D user_extra; - - if (user_locked > user_lock_limit) { - /* - * charge locked_vm until it hits user_lock_limit; - * charge the rest from pinned_vm - */ - extra =3D user_locked - user_lock_limit; - user_extra -=3D extra; - } - - lock_limit =3D rlimit(RLIMIT_MEMLOCK); - lock_limit >>=3D PAGE_SHIFT; - locked =3D atomic64_read(&vma->vm_mm->pinned_vm) + extra; - - if ((locked > lock_limit) && perf_is_paranoid() && - !capable(CAP_IPC_LOCK)) { + if (!perf_mmap_calc_limits(vma, &user_extra, &extra)) { ret =3D -EPERM; goto unlock; } From nobody Sun Oct 5 07:20:29 2025 Received: from galois.linutronix.de (Galois.linutronix.de [193.142.43.55]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9B6D321ABD5 for ; Wed, 6 Aug 2025 20:13:00 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=193.142.43.55 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1754511182; cv=none; b=Vt4B2NdmlnxpMnZzFSDiCXxuJbOpTLGAQkxdNkYj9Y8DD+VI7NSay9PBJXyoGe0Iz+A/7a5MhKNYI19TdnxWng3VrXMifEB4nyaV7gOhQdZcsbcVUijbwafEtfMHnGevxU6qwe5gBHZpmB8KF5gbHvMfd7m9F6GfdTlmdBODj48= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1754511182; c=relaxed/simple; bh=Nc2kKlgpRkhpPpvOyx8gG4nYfydMc4CeOuhggFgi5JM=; h=Message-ID:From:To:Cc:Subject:References:MIME-Version: Content-Type:Date; b=gRsdw+NdmkWqkQmWoYd9R5JfWIWIkm/eSix3a9SYJzll5Vtl+VikBnMRaEu+miJmHku413x4I/I/1y29Z5gPOjxbs1mzCBkjrTRnFB8tpqRP058mfRTOJCuZPVMHbp0wLrd3rA0BPNJQeKxY/n243c0IGjHPtck0Can9PyaNBXY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linutronix.de; spf=pass smtp.mailfrom=linutronix.de; dkim=pass (2048-bit key) header.d=linutronix.de header.i=@linutronix.de header.b=lUrzqdhv; dkim=permerror (0-bit key) header.d=linutronix.de header.i=@linutronix.de header.b=ygD5uaA7; arc=none smtp.client-ip=193.142.43.55 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linutronix.de Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linutronix.de Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="lUrzqdhv"; dkim=permerror (0-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="ygD5uaA7" Message-ID: <20250806200617.386748791@linutronix.de> DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1754511178; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: references:references; bh=u9lmEu5NM3cZYehFdtl8Oe0xuOKsLEY0QXX8YtLCDZs=; b=lUrzqdhvW1zZxOSlghhPU4xXSDUmLre8fJnifoiR+YztFeZmWmMigOU1HezWcNOdwfmUG/ +dH/+dOq51w6Y3HpMyE1vr/DOMjVZspVRsOxbWSfrs+G0GshG2TLm4B9Sr36rIcWtjuyqB IpPNpLlJ4b58fNOIui4tmbSe5Lzyc4lzS1y2BKv1vOt91WWpXtuDk9rN24SHiPFoDxfeqR p8md3Ra5dMvfnIslJJEw7vYl2V5QQMdzBHHDhwLMFVS1fU4danIVGtF8gLCInyM42wTfSe NA2Z1oEBJ6b5b7jTU+l7V71uqcHA+aP+RBsd/id/TiKY6iAKrSnKTwz6lq9TQg== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1754511178; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: references:references; bh=u9lmEu5NM3cZYehFdtl8Oe0xuOKsLEY0QXX8YtLCDZs=; b=ygD5uaA7tI08yA3vL2ssM1J+hZ/5JP1GbDWkRKYN4VrswB7top/1Y994Dp0rCWDohiBMS/ LNnYVk8zfUtz8/Dw== From: Thomas Gleixner To: LKML Cc: Linus Torvalds , Peter Zijlstra , Ingo Molnar , Namhyung Kim , Arnaldo Carvalho de Melo , Lorenzo Stoakes , Kees Cook Subject: [patch 3/6] perf/core: Split out VM accounting References: <20250806195624.880096284@linutronix.de> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Date: Wed, 6 Aug 2025 22:12:57 +0200 (CEST) Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Similary to the mlock limit calculation the VM accounting is required for both the ringbuffer and the AUX buffer allocations. To prepare for splitting them out into seperate functions, move the accounting into a helper function. Signed-off-by: Thomas Gleixner Reviewed-by: Lorenzo Stoakes --- kernel/events/core.c | 13 +++++++++---- 1 file changed, 9 insertions(+), 4 deletions(-) --- a/kernel/events/core.c +++ b/kernel/events/core.c @@ -6962,10 +6962,17 @@ static bool perf_mmap_calc_limits(struct return locked <=3D lock_limit || !perf_is_paranoid() || capable(CAP_IPC_L= OCK); } =20 +static void perf_mmap_account(struct vm_area_struct *vma, long user_extra,= long extra) +{ + struct user_struct *user =3D current_user(); + + atomic_long_add(user_extra, &user->locked_vm); + atomic64_add(extra, &vma->vm_mm->pinned_vm); +} + static int perf_mmap(struct file *file, struct vm_area_struct *vma) { struct perf_event *event =3D file->private_data; - struct user_struct *user =3D current_user(); unsigned long vma_size, nr_pages; long user_extra =3D 0, extra =3D 0; struct mutex *aux_mutex =3D NULL; @@ -7136,9 +7143,7 @@ static int perf_mmap(struct file *file, =20 unlock: if (!ret) { - atomic_long_add(user_extra, &user->locked_vm); - atomic64_add(extra, &vma->vm_mm->pinned_vm); - + perf_mmap_account(vma, user_extra, extra); atomic_inc(&event->mmap_count); } else if (rb) { /* AUX allocation failed */ From nobody Sun Oct 5 07:20:29 2025 Received: from galois.linutronix.de (Galois.linutronix.de [193.142.43.55]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6FEBD1C862D for ; Wed, 6 Aug 2025 20:13:01 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=193.142.43.55 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1754511183; cv=none; b=GmQ3iTv0jv1QYUr9MW9dX/6V35RSwTckKKX50J5b/hb7fwXvVsVkVv+jNhX7trxEFbnUsC98VAf8U/63yQeyfTH4083xhG+0WrcPqpaUMP9HAGv5jR0FseP3US3rU4NELU54/6L58L2DWTGuVNJ+6cO/t5o6pc+NPAl2fsNhx+s= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1754511183; c=relaxed/simple; bh=mptlW4QvppdOxcML4TpyWb/m8EUMoIUssMOWjbfA3d8=; h=Message-ID:From:To:Cc:Subject:References:MIME-Version: Content-Type:Date; b=GGJfGbss96NTqAsqNMIVasjQcLTiZFNq2XDSjek9EzgytjvzitKWIA5g8v3R2BQyB0e+KZeRNSe1Xo+GgaamxBymrjI1V7kcefAVKopzu119QjiMvOlO8yF0Jr+WXFnAcQv5W+2rJz1hi8kfM2sA2/c/WTZ1PbU8+/0xcAHJgGc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linutronix.de; spf=pass smtp.mailfrom=linutronix.de; dkim=pass (2048-bit key) header.d=linutronix.de header.i=@linutronix.de header.b=CH6KhZEb; dkim=permerror (0-bit key) header.d=linutronix.de header.i=@linutronix.de header.b=Kw3urDcB; arc=none smtp.client-ip=193.142.43.55 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linutronix.de Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linutronix.de Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="CH6KhZEb"; dkim=permerror (0-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="Kw3urDcB" Message-ID: <20250806200617.450396385@linutronix.de> DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1754511179; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: references:references; bh=Q8ibW5m28of82Bs979FQ4uHADNle8n1fHPOg+oWk1mo=; b=CH6KhZEbInrHnzjopUeycuoGdw7RTrV/yaA10H5yaQEuMagU3q29dkGGfBOuRMta6+jW7Z 2lSxdeP/DS3j3V6t7vcJ4E1fdSFoA40W3si7ep1f0mLw1xZqVwmqkpQiNfyMTJiCCXAJkS dL+UBkxmmdjE7B5n+z2LuiW6o1M13dN9z8tmsx4K2k6/hEZOhXWp3c+CROFM3l9gARZU3d yl91z39R+P+LVKW4OJh5fijYzvw7EPWCsMb/A7A/mGbVkIVP8zVJ6R/BOaMCSLiCEBVI9V ih/r80OQlcLV1ExfuoY4hONymLBS+IRefI4S8X8sqp+CW813iPbyrd6e+5TKUQ== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1754511179; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: references:references; bh=Q8ibW5m28of82Bs979FQ4uHADNle8n1fHPOg+oWk1mo=; b=Kw3urDcBeZGqwAjYC39VZbU+HvsjmiltVoojaTRUHx5WUjvbrS0Hyp8RzkYl2PKYWLl/LK rtKPW7rBm1wIrIBQ== From: Thomas Gleixner To: LKML Cc: Linus Torvalds , Peter Zijlstra , Ingo Molnar , Namhyung Kim , Arnaldo Carvalho de Melo , Lorenzo Stoakes , Kees Cook Subject: [patch 4/6] perf/core: Split out ringbuffer allocation References: <20250806195624.880096284@linutronix.de> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Date: Wed, 6 Aug 2025 22:12:58 +0200 (CEST) Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" The code logic in perf_mmap() is incomprehensible and has been source of subtle bugs in the past. It makes it impossible to convert the atomic_t reference counts to refcount_t. There is not really much, which is shared between the ringbuffer and AUX buffer allocation code since the mlock limit calculation and the accounting has been split out into helper functions. Move the AUX buffer allocation code out and integrate the call with a momentary workaround to allow skipping the remaining ringbuffer related code completely. That workaround will be removed once the ringbuffer allocation is moved to its own function as well. No functional change. Signed-off-by: Thomas Gleixner --- kernel/events/core.c | 134 ++++++++++++++++++++++++++++------------------= ----- 1 file changed, 75 insertions(+), 59 deletions(-) --- a/kernel/events/core.c +++ b/kernel/events/core.c @@ -6970,12 +6970,76 @@ static void perf_mmap_account(struct vm_ atomic64_add(extra, &vma->vm_mm->pinned_vm); } =20 +static int perf_mmap_aux(struct vm_area_struct *vma, struct perf_event *ev= ent, + unsigned long nr_pages) +{ + long user_extra =3D nr_pages, extra =3D 0; + struct perf_buffer *rb =3D event->rb; + u64 aux_offset, aux_size; + int ret, rb_flags =3D 0; + + /* + * AUX area mapping: if rb->aux_nr_pages !=3D 0, it's already + * mapped, all subsequent mappings should have the same size + * and offset. Must be above the normal perf buffer. + */ + aux_offset =3D READ_ONCE(rb->user_page->aux_offset); + aux_size =3D READ_ONCE(rb->user_page->aux_size); + + if (aux_offset < perf_data_size(rb) + PAGE_SIZE) + return -EINVAL; + + if (aux_offset !=3D vma->vm_pgoff << PAGE_SHIFT) + return -EINVAL; + + /* Already mapped with a different offset */ + if (rb_has_aux(rb) && rb->aux_pgoff !=3D vma->vm_pgoff) + return -EINVAL; + + if (aux_size !=3D nr_pages * PAGE_SIZE) + return -EINVAL; + + /* Already mapped with a different size */ + if (rb_has_aux(rb) && rb->aux_nr_pages !=3D nr_pages) + return -EINVAL; + + if (!is_power_of_2(nr_pages)) + return -EINVAL; + + /* If this succeeds, subsequent failures have to undo it */ + if (!atomic_inc_not_zero(&rb->mmap_count)) + return -EINVAL; + + /* If mapped, attach to it */ + if (rb_has_aux(rb)) { + atomic_inc(&rb->aux_mmap_count); + return 0; + } + + if (!perf_mmap_calc_limits(vma, &user_extra, &extra)) { + atomic_dec(&rb->mmap_count); + return -EPERM; + } + + ret =3D rb_alloc_aux(rb, event, vma->vm_pgoff, nr_pages, + event->attr.aux_watermark, rb_flags); + if (ret) { + atomic_dec(&rb->mmap_count); + return ret; + } + + atomic_set(&rb->aux_mmap_count, 1); + rb->aux_mmap_locked =3D extra; + perf_mmap_account(vma, user_extra, extra); + atomic_inc(&event->mmap_count); + return 0; +} + static int perf_mmap(struct file *file, struct vm_area_struct *vma) { struct perf_event *event =3D file->private_data; unsigned long vma_size, nr_pages; long user_extra =3D 0, extra =3D 0; - struct mutex *aux_mutex =3D NULL; struct perf_buffer *rb =3D NULL; int ret, flags =3D 0; mapped_f mapped; @@ -7055,51 +7119,15 @@ static int perf_mmap(struct file *file, } =20 } else { - /* - * AUX area mapping: if rb->aux_nr_pages !=3D 0, it's already - * mapped, all subsequent mappings should have the same size - * and offset. Must be above the normal perf buffer. - */ - u64 aux_offset, aux_size; - - rb =3D event->rb; - if (!rb) - goto aux_unlock; - - aux_mutex =3D &rb->aux_mutex; - mutex_lock(aux_mutex); - - aux_offset =3D READ_ONCE(rb->user_page->aux_offset); - aux_size =3D READ_ONCE(rb->user_page->aux_size); - - if (aux_offset < perf_data_size(rb) + PAGE_SIZE) - goto aux_unlock; - - if (aux_offset !=3D vma->vm_pgoff << PAGE_SHIFT) - goto aux_unlock; - - /* already mapped with a different offset */ - if (rb_has_aux(rb) && rb->aux_pgoff !=3D vma->vm_pgoff) - goto aux_unlock; - - if (aux_size !=3D nr_pages * PAGE_SIZE) - goto aux_unlock; - - /* already mapped with a different size */ - if (rb_has_aux(rb) && rb->aux_nr_pages !=3D nr_pages) - goto aux_unlock; - - if (!is_power_of_2(nr_pages)) - goto aux_unlock; - - if (!atomic_inc_not_zero(&rb->mmap_count)) - goto aux_unlock; - - if (rb_has_aux(rb)) { - atomic_inc(&rb->aux_mmap_count); - ret =3D 0; - goto unlock; + if (event->rb) { + ret =3D -EINVAL; + } else { + scoped_guard(mutex, &event->rb->aux_mutex) + ret =3D perf_mmap_aux(vma, event, nr_pages); } + // Temporary workaround to split out AUX handling first + mutex_unlock(&event->mmap_mutex); + goto out; } =20 if (!perf_mmap_calc_limits(vma, &user_extra, &extra)) { @@ -7132,28 +7160,16 @@ static int perf_mmap(struct file *file, perf_event_init_userpage(event); perf_event_update_userpage(event); ret =3D 0; - } else { - ret =3D rb_alloc_aux(rb, event, vma->vm_pgoff, nr_pages, - event->attr.aux_watermark, flags); - if (!ret) { - atomic_set(&rb->aux_mmap_count, 1); - rb->aux_mmap_locked =3D extra; - } } - unlock: if (!ret) { perf_mmap_account(vma, user_extra, extra); atomic_inc(&event->mmap_count); - } else if (rb) { - /* AUX allocation failed */ - atomic_dec(&rb->mmap_count); } -aux_unlock: - if (aux_mutex) - mutex_unlock(aux_mutex); mutex_unlock(&event->mmap_mutex); =20 +// Temporary until RB allocation is split out. +out: if (ret) return ret; From nobody Sun Oct 5 07:20:29 2025 Received: from galois.linutronix.de (Galois.linutronix.de [193.142.43.55]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D5D802C08CA for ; Wed, 6 Aug 2025 20:13:02 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=193.142.43.55 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1754511184; cv=none; b=QUu4P6DkHY2pFJGeJXIx2EMiGa1IhEPRGtIJr5ld9Rxjpe6T2sqSj6CciH8+Xb+qoOc8RSfB/b4UMjrLg8F1ndlxxUosxIPP0iDM6zf5yHohqUQyHMOCS0jNQPR2li00qnyv5WVX9phj3wMuLZCZoCMoFp1mt5pVm4xWf7cjlAg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1754511184; c=relaxed/simple; bh=EvekjKimv6DPVa4oXAQouJKYBitznonUqyeEHOjkb5U=; h=Message-ID:From:To:Cc:Subject:References:MIME-Version: Content-Type:Date; b=XE7i9uWz6AidwZ9a4J14KvBCtF+GZvMfcY9mL0YBWYfUHxj+PBw6NKVJNEmf5Ga4AJVwQya1nOYQXyEo1SP8XA2b/NiCvku7WMzlcvsgwrneivgMAABdc97HD9rwYg/NjnGXzqP++slRpEu4y4jBXG1dS7C6UMm6x0v8B8rkg9g= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linutronix.de; spf=pass smtp.mailfrom=linutronix.de; dkim=pass (2048-bit key) header.d=linutronix.de header.i=@linutronix.de header.b=JkbIJAtm; dkim=permerror (0-bit key) header.d=linutronix.de header.i=@linutronix.de header.b=S0Thr4zo; arc=none smtp.client-ip=193.142.43.55 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linutronix.de Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linutronix.de Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="JkbIJAtm"; dkim=permerror (0-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="S0Thr4zo" Message-ID: <20250806200617.513959766@linutronix.de> DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1754511181; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: references:references; bh=5impsAP87RWuwsK32+xW8Xa7uCwljYhrxKJgzH5kwI4=; b=JkbIJAtmSC+HFIn2FJbeJfoM2Gqq40Hfv7BvxnVbNrABVI3TdF3KqdFWioPEZLnxOXMvmE 2ummQemjkoApFLAAxE+2QhVCsuE5rriIbeXLXK0xjXcShbyW1UgKg6Iu5Yqbw/UJVLU3PP dEbf4vs2fkVaoF0kDE7qEW+CIzRMQKbCVshy2OllY2nas1zH07v6YoRPSqB+y6VlNF6vGQ OIIeduGaqLDTOKi9IpOLBe5G9PZbZBS7AW5i1UnpcP3YIUsPTThAXGHh/FWBLiWr8JYCij 1HCoBPxW4puSKMwWit8+gREgM1HsN9sdNqTiPP2HeGixmHt5Hvddu2/C/vB9vg== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1754511181; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: references:references; bh=5impsAP87RWuwsK32+xW8Xa7uCwljYhrxKJgzH5kwI4=; b=S0Thr4zo1plI/cxzCWEPZlFBsV6LVvpTaTIWbv4i3anmg8oHux4qVkjOXzqPCgEIOKmuB5 qdouZVcYk/9rsRBQ== From: Thomas Gleixner To: LKML Cc: Linus Torvalds , Peter Zijlstra , Ingo Molnar , Namhyung Kim , Arnaldo Carvalho de Melo , Lorenzo Stoakes , Kees Cook Subject: [patch 5/6] perf/core: Split the ringbuffer mmap() and allocation code out References: <20250806195624.880096284@linutronix.de> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Date: Wed, 6 Aug 2025 22:13:00 +0200 (CEST) Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" The code logic in perf_mmap() is incomprehensible and has been source of subtle bugs in the past. It makes it impossible to convert the atomic_t reference counts to refcount_t. Now that the AUX buffer mapping and allocation code is in it's own function apply the same treatment to the ringbuffer part and remove the temporary workarounds created by the AUX split out. No functional change intended. Signed-off-by: Thomas Gleixner --- kernel/events/core.c | 175 ++++++++++++++++++++++------------------------= ----- 1 file changed, 77 insertions(+), 98 deletions(-) --- a/kernel/events/core.c +++ b/kernel/events/core.c @@ -6970,6 +6970,69 @@ static void perf_mmap_account(struct vm_ atomic64_add(extra, &vma->vm_mm->pinned_vm); } =20 +static int perf_mmap_rb(struct vm_area_struct *vma, struct perf_event *eve= nt, + unsigned long nr_pages) +{ + long user_extra =3D nr_pages, extra =3D 0; + struct perf_buffer *rb =3D event->rb; + int rb_flags =3D 0; + + /* + * If we have rb pages ensure they're a power-of-two number, so we + * can do bitmasks instead of modulo. + */ + if (--nr_pages !=3D 0 && !is_power_of_2(nr_pages)) + return -EINVAL; + + WARN_ON_ONCE(event->ctx->parent_ctx); + + if (rb) { + if (data_page_nr(rb) !=3D nr_pages) + return -EINVAL; + + if (atomic_inc_not_zero(&event->rb->mmap_count)) { + /* + * Success -- managed to mmap() the same buffer + * multiple times. + */ + atomic_inc(&event->mmap_count); + return 0; + } + /* + * Raced against perf_mmap_close()'s + * atomic_dec_and_mutex_lock() remove the event and + * continue as if !event->rb + */ + ring_buffer_attach(event, NULL); + } + + if (!perf_mmap_calc_limits(vma, &user_extra, &extra)) + return -EPERM; + + if (vma->vm_flags & VM_WRITE) + rb_flags |=3D RING_BUFFER_WRITABLE; + + rb =3D rb_alloc(nr_pages, event->attr.watermark ? event->attr.wakeup_wate= rmark : 0, + event->cpu, rb_flags); + + if (!rb) + return -ENOMEM; + + atomic_set(&rb->mmap_count, 1); + rb->mmap_user =3D get_current_user(); + rb->mmap_locked =3D extra; + + ring_buffer_attach(event, rb); + + perf_event_update_time(event); + perf_event_init_userpage(event); + perf_event_update_userpage(event); + + perf_mmap_account(vma, user_extra, extra); + atomic_set(&event->mmap_count, 1); + return 0; +} + static int perf_mmap_aux(struct vm_area_struct *vma, struct perf_event *ev= ent, unsigned long nr_pages) { @@ -7039,10 +7102,8 @@ static int perf_mmap(struct file *file, { struct perf_event *event =3D file->private_data; unsigned long vma_size, nr_pages; - long user_extra =3D 0, extra =3D 0; - struct perf_buffer *rb =3D NULL; - int ret, flags =3D 0; mapped_f mapped; + int ret; =20 /* * Don't allow mmap() of inherited per-task counters. This would @@ -7068,114 +7129,32 @@ static int perf_mmap(struct file *file, if (vma_size !=3D PAGE_SIZE * nr_pages) return -EINVAL; =20 - user_extra =3D nr_pages; - - mutex_lock(&event->mmap_mutex); - ret =3D -EINVAL; - - /* - * This relies on __pmu_detach_event() taking mmap_mutex after marking - * the event REVOKED. Either we observe the state, or __pmu_detach_event() - * will detach the rb created here. - */ - if (event->state <=3D PERF_EVENT_STATE_REVOKED) { - ret =3D -ENODEV; - goto unlock; - } - - if (vma->vm_pgoff =3D=3D 0) { - nr_pages -=3D 1; - + scoped_guard(mutex, &event->mmap_mutex) { /* - * If we have rb pages ensure they're a power-of-two number, so we - * can do bitmasks instead of modulo. + * This relies on __pmu_detach_event() taking mmap_mutex + * after marking the event REVOKED. Either we observe the + * state, or __pmu_detach_event() will detach the rb + * created here. */ - if (nr_pages !=3D 0 && !is_power_of_2(nr_pages)) - goto unlock; - - WARN_ON_ONCE(event->ctx->parent_ctx); - - if (event->rb) { - if (data_page_nr(event->rb) !=3D nr_pages) - goto unlock; - - if (atomic_inc_not_zero(&event->rb->mmap_count)) { - /* - * Success -- managed to mmap() the same buffer - * multiple times. - */ - ret =3D 0; - /* We need the rb to map pages. */ - rb =3D event->rb; - goto unlock; - } - - /* - * Raced against perf_mmap_close()'s - * atomic_dec_and_mutex_lock() remove the - * event and continue as if !event->rb - */ - ring_buffer_attach(event, NULL); - } + if (event->state <=3D PERF_EVENT_STATE_REVOKED) + return -ENODEV; =20 - } else { - if (event->rb) { - ret =3D -EINVAL; + if (vma->vm_pgoff =3D=3D 0) { + ret =3D perf_mmap_rb(vma, event, nr_pages); } else { + if (!event->rb) + return -EINVAL; scoped_guard(mutex, &event->rb->aux_mutex) ret =3D perf_mmap_aux(vma, event, nr_pages); } - // Temporary workaround to split out AUX handling first - mutex_unlock(&event->mmap_mutex); - goto out; - } - - if (!perf_mmap_calc_limits(vma, &user_extra, &extra)) { - ret =3D -EPERM; - goto unlock; - } - - WARN_ON(!rb && event->rb); - - if (vma->vm_flags & VM_WRITE) - flags |=3D RING_BUFFER_WRITABLE; - - if (!rb) { - rb =3D rb_alloc(nr_pages, - event->attr.watermark ? event->attr.wakeup_watermark : 0, - event->cpu, flags); - - if (!rb) { - ret =3D -ENOMEM; - goto unlock; - } - - atomic_set(&rb->mmap_count, 1); - rb->mmap_user =3D get_current_user(); - rb->mmap_locked =3D extra; - - ring_buffer_attach(event, rb); - - perf_event_update_time(event); - perf_event_init_userpage(event); - perf_event_update_userpage(event); - ret =3D 0; - } -unlock: - if (!ret) { - perf_mmap_account(vma, user_extra, extra); - atomic_inc(&event->mmap_count); } - mutex_unlock(&event->mmap_mutex); =20 -// Temporary until RB allocation is split out. -out: if (ret) return ret; =20 /* * Since pinned accounting is per vm we cannot allow fork() to copy our - * vma. + * VMA. The VMA is fixed size and must not be included in dumps. */ vm_flags_set(vma, VM_DONTCOPY | VM_DONTEXPAND | VM_DONTDUMP); vma->vm_ops =3D &perf_mmap_vmops; @@ -7190,7 +7169,7 @@ static int perf_mmap(struct file *file, * full cleanup in this case and therefore does not invoke * vmops::close(). */ - ret =3D map_range(rb, vma); + ret =3D map_range(event->rb, vma); if (ret) perf_mmap_close(vma); From nobody Sun Oct 5 07:20:29 2025 Received: from galois.linutronix.de (Galois.linutronix.de [193.142.43.55]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8983E2D192B for ; Wed, 6 Aug 2025 20:13:04 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=193.142.43.55 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1754511186; cv=none; b=PzYlXM9c4ZTbanNSVo+hAaANrJQ0Qrc3+ZUPNpPJv3Jv1f2TBGcfG+D8vQmp7aAOScuaTv0CkWUvg53tIf3AKuqnRpTG8SdHzZzsrPeaYYYYGGla3gIt85AuFBpYYtA6UYaoeN3ojStMyTyCYNT3QV7LWuFKiySmBbzzgs3OwEk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1754511186; c=relaxed/simple; bh=LM3h/QBg46ucDhqNkB/XC02efGvnIxjitHO3MZLDhus=; h=Message-ID:From:To:Cc:Subject:References:MIME-Version: Content-Type:Date; b=V7vK7vL6nCE+mMRxEdlGoY1lbppfZtUYjSZ2FxQ94Ac2VtjV7DvD500uvqM3mrJQ3u+sQcCntOaeRxwnzM+bN0lUCknzdxk6SDAA0UmjruVZkDHrJvKo0v+MTZkm+GfBHzZy0NKkTlxMZEUe/QEDS4IyIRI4vujcgkNu8YhyZYY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linutronix.de; spf=pass smtp.mailfrom=linutronix.de; dkim=pass (2048-bit key) header.d=linutronix.de header.i=@linutronix.de header.b=VzwSij8J; dkim=permerror (0-bit key) header.d=linutronix.de header.i=@linutronix.de header.b=9Z0fWs7+; arc=none smtp.client-ip=193.142.43.55 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linutronix.de Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linutronix.de Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="VzwSij8J"; dkim=permerror (0-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="9Z0fWs7+" Message-ID: <20250806200617.576749592@linutronix.de> DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1754511182; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: references:references; bh=XMxUxbXM63hGqWPvFcSsFgioRmubeAKlsk/MQh1rq/g=; b=VzwSij8JiOmwhSB0782EgyOtO6o5VN/39XkX3Kelz+lgqRI9SnODEUeZcpf5YDv3zOcHC8 0FxQEiGQ8S8x8Y1vpwW74JuRTfeLkaub0YAObRhxl/to8SRCzeENVyTR9GtJcsMrhZjVnk H0zLIEZ4cwuHiAVKUjSRgyIlF5I0VbWjArF559WpkTvwCJWjGYzKYvMm+xxfqCVmh+vkmu wFsLK5ROu0fFxz6UPulgi6lyLM6hJAhkIXVgrj6XQFfqTu7uuw3oslB3x/THMEeb0JRrlO eWJVa8rOZzZaqDwhY6Kqa2Z4Txp6IXHw26KciQPCoj56tOESJPUZsT4IvoXn4g== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1754511182; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: references:references; bh=XMxUxbXM63hGqWPvFcSsFgioRmubeAKlsk/MQh1rq/g=; b=9Z0fWs7+eYJN3TDS8rxtMqwZt3HyurTslws8wv6Y6IZgM551+uF7KihVwwkNhRwnHgT2Dr 6dHm56P48kOV/yAQ== From: Thomas Gleixner To: LKML Cc: Linus Torvalds , Peter Zijlstra , Ingo Molnar , Namhyung Kim , Arnaldo Carvalho de Melo , Lorenzo Stoakes , Kees Cook Subject: [patch 6/6] perf/core: Convert mmap() refcounts to refcount_t References: <20250806195624.880096284@linutronix.de> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Date: Wed, 6 Aug 2025 22:13:02 +0200 (CEST) Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" The recently fixed reference count leaks could have been detected by using refcount_t and refcount_t would have mitigated the potential overflow at least. Now that the code is properly structured, convert the mmap() related mmap_count variants over to refcount_t. No functional change intended. Signed-off-by: Thomas Gleixner --- include/linux/perf_event.h | 2 +- kernel/events/core.c | 40 ++++++++++++++++++++-------------------- kernel/events/internal.h | 4 ++-- kernel/events/ring_buffer.c | 2 +- 4 files changed, 24 insertions(+), 24 deletions(-) --- a/include/linux/perf_event.h +++ b/include/linux/perf_event.h @@ -859,7 +859,7 @@ struct perf_event { =20 /* mmap bits */ struct mutex mmap_mutex; - atomic_t mmap_count; + refcount_t mmap_count; =20 struct perf_buffer *rb; struct list_head rb_entry; --- a/kernel/events/core.c +++ b/kernel/events/core.c @@ -3968,7 +3968,7 @@ static noinline int visit_groups_merge(s */ static inline bool event_update_userpage(struct perf_event *event) { - if (likely(!atomic_read(&event->mmap_count))) + if (likely(!refcount_read(&event->mmap_count))) return false; =20 perf_event_update_time(event); @@ -6704,11 +6704,11 @@ static void perf_mmap_open(struct vm_are struct perf_event *event =3D vma->vm_file->private_data; mapped_f mapped =3D get_mapped(event, event_mapped); =20 - atomic_inc(&event->mmap_count); - atomic_inc(&event->rb->mmap_count); + refcount_inc(&event->mmap_count); + refcount_inc(&event->rb->mmap_count); =20 if (vma->vm_pgoff) - atomic_inc(&event->rb->aux_mmap_count); + refcount_inc(&event->rb->aux_mmap_count); =20 if (mapped) mapped(event, vma->vm_mm); @@ -6743,7 +6743,7 @@ static void perf_mmap_close(struct vm_ar * to avoid complications. */ if (rb_has_aux(rb) && vma->vm_pgoff =3D=3D rb->aux_pgoff && - atomic_dec_and_mutex_lock(&rb->aux_mmap_count, &rb->aux_mutex)) { + refcount_dec_and_mutex_lock(&rb->aux_mmap_count, &rb->aux_mutex)) { /* * Stop all AUX events that are writing to this buffer, * so that we can free its AUX pages and corresponding PMU @@ -6763,10 +6763,10 @@ static void perf_mmap_close(struct vm_ar mutex_unlock(&rb->aux_mutex); } =20 - if (atomic_dec_and_test(&rb->mmap_count)) + if (refcount_dec_and_test(&rb->mmap_count)) detach_rest =3D true; =20 - if (!atomic_dec_and_mutex_lock(&event->mmap_count, &event->mmap_mutex)) + if (!refcount_dec_and_mutex_lock(&event->mmap_count, &event->mmap_mutex)) goto out_put; =20 ring_buffer_attach(event, NULL); @@ -6990,17 +6990,17 @@ static int perf_mmap_rb(struct vm_area_s if (data_page_nr(rb) !=3D nr_pages) return -EINVAL; =20 - if (atomic_inc_not_zero(&event->rb->mmap_count)) { + if (refcount_inc_not_zero(&event->rb->mmap_count)) { /* * Success -- managed to mmap() the same buffer * multiple times. */ - atomic_inc(&event->mmap_count); + refcount_inc(&event->mmap_count); return 0; } /* * Raced against perf_mmap_close()'s - * atomic_dec_and_mutex_lock() remove the event and + * refcount_dec_and_mutex_lock() remove the event and * continue as if !event->rb */ ring_buffer_attach(event, NULL); @@ -7018,7 +7018,7 @@ static int perf_mmap_rb(struct vm_area_s if (!rb) return -ENOMEM; =20 - atomic_set(&rb->mmap_count, 1); + refcount_set(&rb->mmap_count, 1); rb->mmap_user =3D get_current_user(); rb->mmap_locked =3D extra; =20 @@ -7029,7 +7029,7 @@ static int perf_mmap_rb(struct vm_area_s perf_event_update_userpage(event); =20 perf_mmap_account(vma, user_extra, extra); - atomic_set(&event->mmap_count, 1); + refcount_set(&event->mmap_count, 1); return 0; } =20 @@ -7070,31 +7070,31 @@ static int perf_mmap_aux(struct vm_area_ return -EINVAL; =20 /* If this succeeds, subsequent failures have to undo it */ - if (!atomic_inc_not_zero(&rb->mmap_count)) + if (!refcount_inc_not_zero(&rb->mmap_count)) return -EINVAL; =20 /* If mapped, attach to it */ if (rb_has_aux(rb)) { - atomic_inc(&rb->aux_mmap_count); + refcount_inc(&rb->aux_mmap_count); return 0; } =20 if (!perf_mmap_calc_limits(vma, &user_extra, &extra)) { - atomic_dec(&rb->mmap_count); + refcount_dec(&rb->mmap_count); return -EPERM; } =20 ret =3D rb_alloc_aux(rb, event, vma->vm_pgoff, nr_pages, event->attr.aux_watermark, rb_flags); if (ret) { - atomic_dec(&rb->mmap_count); + refcount_dec(&rb->mmap_count); return ret; } =20 - atomic_set(&rb->aux_mmap_count, 1); + refcount_set(&rb->aux_mmap_count, 1); rb->aux_mmap_locked =3D extra; perf_mmap_account(vma, user_extra, extra); - atomic_inc(&event->mmap_count); + refcount_inc(&event->mmap_count); return 0; } =20 @@ -13243,7 +13243,7 @@ perf_event_set_output(struct perf_event mutex_lock_double(&event->mmap_mutex, &output_event->mmap_mutex); set: /* Can't redirect output if we've got an active mmap() */ - if (atomic_read(&event->mmap_count)) + if (refcount_read(&event->mmap_count)) goto unlock; =20 if (output_event) { @@ -13256,7 +13256,7 @@ perf_event_set_output(struct perf_event goto unlock; =20 /* did we race against perf_mmap_close() */ - if (!atomic_read(&rb->mmap_count)) { + if (!refcount_read(&rb->mmap_count)) { ring_buffer_put(rb); goto unlock; } --- a/kernel/events/internal.h +++ b/kernel/events/internal.h @@ -35,7 +35,7 @@ struct perf_buffer { spinlock_t event_lock; struct list_head event_list; =20 - atomic_t mmap_count; + refcount_t mmap_count; unsigned long mmap_locked; struct user_struct *mmap_user; =20 @@ -47,7 +47,7 @@ struct perf_buffer { unsigned long aux_pgoff; int aux_nr_pages; int aux_overwrite; - atomic_t aux_mmap_count; + refcount_t aux_mmap_count; unsigned long aux_mmap_locked; void (*free_aux)(void *); refcount_t aux_refcount; --- a/kernel/events/ring_buffer.c +++ b/kernel/events/ring_buffer.c @@ -400,7 +400,7 @@ void *perf_aux_output_begin(struct perf_ * the same order, see perf_mmap_close. Otherwise we end up freeing * aux pages in this path, which is a bug, because in_atomic(). */ - if (!atomic_read(&rb->aux_mmap_count)) + if (!refcount_read(&rb->aux_mmap_count)) goto err; =20 if (!refcount_inc_not_zero(&rb->aux_refcount))