From nobody Sun Feb 8 09:32:55 2026 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1E20B22F173 for ; Wed, 5 Feb 2025 10:25:42 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=90.155.50.34 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738751144; cv=none; b=cB4X8nPhCqjEgJRtYKEkosrGLvkODfTY5hoW9N7GyDp4umzjeiBdGSxlqr50URIjrKrdZDODkUBLCxk2v5EhDK3LDPzJZH+aKd15WIPm2bDqfQ7CLDdi4O5e+qo+i1H7BmZhiQ/6RT8BtOoHZ/6bYaQXwrJyXLxf3d1z68+uj3Y= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738751144; c=relaxed/simple; bh=jkluaH4XEYEQte8xGB2hv1h2ykK+4gRIqt1RXfpWLbg=; h=Message-ID:Date:From:To:Cc:Subject:References:MIME-Version: Content-Type; b=dN+syEgdwPXpaOwNOFPxRGvoopIeamTVhQA4ZO9vOlUJmHpRArVtUIDcY9Ac1sGluLuYcaqrXQ+4AQyw4yFAUR6RiUwLN4sMzYIkm303Sis0GBFSU1J4slSUFtv9g1fzJ02F1gexZoeeoXHMQFG6aURdYqjAhJ/bpG6IE+g2jg0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org; spf=none smtp.mailfrom=infradead.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b=RBVWViR0; arc=none smtp.client-ip=90.155.50.34 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=infradead.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="RBVWViR0" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=ae6cNYVBi7drCixIq97d0s/5QVUjP8FLagtyIO73Xkc=; b=RBVWViR07Kf9g8zQn/4BaWuRoR setm5JM/ZlhmOaZq79WO5hFJljs2/4PxLqmm9v8kQt+UmKgZ9WOY55MtR2zRhLbjuxiIooms6/P75 BsZXD+YrMqu8jAQ8KNOp3qgyGX6ydVFY11P8Z8EDbf4Pl4uArzIthqZTmI5CGMKr0no1nyk5U0y9O NkD1SOE27z632qiAyrX76er/2CEtjz4Ctl77P8F5afTOWNKCbl9aI39xsNRxPnRsS5+ADIkxd+yU+ 0scqmYrQ/ox210VRUnNRcY3hnlcEOpIwv+ECnGzn3MIMTGYQEEWrWkGPq4RrdBeIkzwCBlKdsVnv5 0SnLL1Ow==; Received: from 77-249-17-89.cable.dynamic.v4.ziggo.nl ([77.249.17.89] helo=noisy.programming.kicks-ass.net) by casper.infradead.org with esmtpsa (Exim 4.98 #2 (Red Hat Linux)) id 1tfcb2-00000004GPR-2bak; Wed, 05 Feb 2025 10:25:28 +0000 Received: by noisy.programming.kicks-ass.net (Postfix, from userid 0) id B0F703003E1; Wed, 5 Feb 2025 11:25:27 +0100 (CET) Message-ID: <20250205102448.348146549@infradead.org> User-Agent: quilt/0.66 Date: Wed, 05 Feb 2025 11:21:21 +0100 From: Peter Zijlstra To: mingo@kernel.org, ravi.bangoria@amd.com, lucas.demarchi@intel.com Cc: linux-kernel@vger.kernel.org, peterz@infradead.org, willy@infradead.org, acme@kernel.org, namhyung@kernel.org, mark.rutland@arm.com, alexander.shishkin@linux.intel.com, jolsa@kernel.org, irogers@google.com, adrian.hunter@intel.com, kan.liang@linux.intel.com, David Hildenbrand Subject: [PATCH v2 01/24] lockdep: Fix might_fault() References: <20250205102120.531585416@infradead.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Turns out that commit 9ec23531fd48 ("sched/preempt, mm/fault: Trigger might_sleep() in might_fault() with disabled pagefaults") accidentally (and unnessecarily) put the lockdep part of __might_fault() under CONFIG_DEBUG_ATOMIC_SLEEP. Cc: David Hildenbrand Fixes: 9ec23531fd48 ("sched/preempt, mm/fault: Trigger might_sleep() in mig= ht_fault() with disabled pagefaults") Signed-off-by: Peter Zijlstra (Intel) Reviewed-by: Ravi Bangoria --- mm/memory.c | 2 -- 1 file changed, 2 deletions(-) --- a/mm/memory.c +++ b/mm/memory.c @@ -6695,10 +6695,8 @@ void __might_fault(const char *file, int if (pagefault_disabled()) return; __might_sleep(file, line); -#if defined(CONFIG_DEBUG_ATOMIC_SLEEP) if (current->mm) might_lock_read(¤t->mm->mmap_lock); -#endif } EXPORT_SYMBOL(__might_fault); #endif From nobody Sun Feb 8 09:32:55 2026 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 83F0722C32D for ; Wed, 5 Feb 2025 10:25:38 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=90.155.50.34 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738751140; cv=none; b=dE5hr5goKRV7pZ7zxy570QiqkbRdW4WBVVdbq0BrG2dU7jIR+Z/XTh2vZIU0ETGFmye7ZLDvKfIKvvsIX+dSDDZjxrSDqQnix4OCXsuzHxQusw1uMX/b8cjyFVHPhoIgAPzszZrsNNO+z1OlnPUrh749D8pE7B9HKtwGPpe/c2M= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738751140; c=relaxed/simple; bh=ejCK5vxQbMv7jatQ4Hem5aLISpgvatcVYDYPSTmRmrA=; h=Message-ID:Date:From:To:Cc:Subject:References:MIME-Version: Content-Type; b=IjoHcy9UEe0H9vz43UYQpKvMTr7SWAovEVdeMVSQcY3/eIybKeBA/N6X2cNM2Uj4cWzbELdDAN68axIUhVuzQxbVdQl4WZg0Ah5e/gSY3B6uIru2St2Fch6YAmem59c+GwqYOQHSIV02TQT+mtNDIWWgnxK9Lq6KSzKgBH0az9w= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org; spf=none smtp.mailfrom=infradead.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b=HL+TPPiP; arc=none smtp.client-ip=90.155.50.34 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=infradead.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="HL+TPPiP" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=tN2WqQdsW+7vvSTtR2P1++oVZtUzPhTpdLXdVmnWZ8A=; b=HL+TPPiP0w1pEuRLpID87rC5gr sO/toELS3ke0IW2IsXEmrw1Bp8MjIeDxUjcyjwnsqR0EuxImPER+Fkt2vCaOUJ7+eLTjP31uzzwon ACetxHE9QMf1uk/XkVu7mGiOq9VAnaKnvxYyIzqhKMChPF8LFCTNiEdfkfbshkEDYw8VTMAnsHpuy n7edfUceu1nwpjpRWEhgYb1I+F9E99ubFojr7SVOIo4IUb4hEJogabcB9dEgd7Bv96pS2cg0ICLh5 7t0BtfGA0OQN4t4FK/c3RjYXetxZ/oQsh6teN6Mj3I1w8xD8eOzAt9CP52BSFYJ7VM+9Wm5fSFm8q 7v4cKF8g==; Received: from 77-249-17-89.cable.dynamic.v4.ziggo.nl ([77.249.17.89] helo=noisy.programming.kicks-ass.net) by casper.infradead.org with esmtpsa (Exim 4.98 #2 (Red Hat Linux)) id 1tfcb2-00000004GPS-2hZU; Wed, 05 Feb 2025 10:25:28 +0000 Received: by noisy.programming.kicks-ass.net (Postfix, from userid 0) id B55D530050D; Wed, 5 Feb 2025 11:25:27 +0100 (CET) Message-ID: <20250205102448.458857702@infradead.org> User-Agent: quilt/0.66 Date: Wed, 05 Feb 2025 11:21:22 +0100 From: Peter Zijlstra To: mingo@kernel.org, ravi.bangoria@amd.com, lucas.demarchi@intel.com Cc: linux-kernel@vger.kernel.org, peterz@infradead.org, willy@infradead.org, acme@kernel.org, namhyung@kernel.org, mark.rutland@arm.com, alexander.shishkin@linux.intel.com, jolsa@kernel.org, irogers@google.com, adrian.hunter@intel.com, kan.liang@linux.intel.com Subject: [PATCH v2 02/24] perf: Ensure bpf_perf_link path is properly serialized References: <20250205102120.531585416@infradead.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Ravi reported that the bpf_perf_link_attach() usage of perf_event_set_bpf_prog() is not serialized by ctx->mutex, unlike the PERF_EVENT_IOC_SET_BPF case. Reported-by: Ravi Bangoria Signed-off-by: Peter Zijlstra (Intel) Reviewed-by: Ravi Bangoria --- kernel/events/core.c | 34 ++++++++++++++++++++++++++++++---- 1 file changed, 30 insertions(+), 4 deletions(-) --- a/kernel/events/core.c +++ b/kernel/events/core.c @@ -6001,6 +6001,9 @@ static int perf_event_set_output(struct static int perf_event_set_filter(struct perf_event *event, void __user *ar= g); static int perf_copy_attr(struct perf_event_attr __user *uattr, struct perf_event_attr *attr); +static int __perf_event_set_bpf_prog(struct perf_event *event, + struct bpf_prog *prog, + u64 bpf_cookie); =20 static long _perf_ioctl(struct perf_event *event, unsigned int cmd, unsign= ed long arg) { @@ -6063,7 +6066,7 @@ static long _perf_ioctl(struct perf_even if (IS_ERR(prog)) return PTR_ERR(prog); =20 - err =3D perf_event_set_bpf_prog(event, prog, 0); + err =3D __perf_event_set_bpf_prog(event, prog, 0); if (err) { bpf_prog_put(prog); return err; @@ -10790,8 +10793,9 @@ static inline bool perf_event_is_tracing return false; } =20 -int perf_event_set_bpf_prog(struct perf_event *event, struct bpf_prog *pro= g, - u64 bpf_cookie) +static int __perf_event_set_bpf_prog(struct perf_event *event, + struct bpf_prog *prog, + u64 bpf_cookie) { bool is_kprobe, is_uprobe, is_tracepoint, is_syscall_tp; =20 @@ -10829,6 +10833,20 @@ int perf_event_set_bpf_prog(struct perf_ return perf_event_attach_bpf_prog(event, prog, bpf_cookie); } =20 +int perf_event_set_bpf_prog(struct perf_event *event, + struct bpf_prog *prog, + u64 bpf_cookie) +{ + struct perf_event_context *ctx; + int ret; + + ctx =3D perf_event_ctx_lock(event); + ret =3D __perf_event_set_bpf_prog(event, prog, bpf_cookie); + perf_event_ctx_unlock(event, ctx); + + return ret; +} + void perf_event_free_bpf_prog(struct perf_event *event) { if (!perf_event_is_tracing(event)) { @@ -10848,7 +10866,15 @@ static void perf_event_free_filter(struc { } =20 -int perf_event_set_bpf_prog(struct perf_event *event, struct bpf_prog *pro= g, +static int __perf_event_set_bpf_prog(struct perf_event *event, + struct bpf_prog *prog, + u64 bpf_cookie) +{ + return -ENOENT; +} + +int perf_event_set_bpf_prog(struct perf_event *event, + struct bpf_prog *prog, u64 bpf_cookie) { return -ENOENT; From nobody Sun Feb 8 09:32:55 2026 Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B53BC2063D8 for ; Wed, 5 Feb 2025 10:25:33 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=90.155.92.199 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738751135; cv=none; b=B/5Km54MWvZhNmfcCkMgcy431CqA2mzcBcPMNBCll+Uu3o3iHe78yYbOyaf4xuAPzC62WCqlqRfP/6Q0JS+xRdVmPr1umq2ussUER9YkbsuGoKq6RlogwY1B9FxtbtQIh64af6kQOIaQeeaG+LH/aoFpT+mJMY2eNwN+kVi9VkA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738751135; c=relaxed/simple; bh=2W3RwCqdWrSbI3AnIU7O9V1/QqhgCRodoKt377zeFEM=; h=Message-ID:Date:From:To:Cc:Subject:References:MIME-Version: Content-Type; b=RgpffQmO6tN/Wg2Z06qnRWY/wTuMZnqMc3FaZDASOPi/KcK4Lne+ppaOjjcr8ffk+FmKZxbyS8J7Rbp7X52TGoJZcuYaX9PDPWPSwJcybMRw5aqPaOyOXKLyBE5K4mRKYRQV3YFunCXcMpKvRrtRfy1kVvr4WgIv3DdTBFnrB0o= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org; spf=none smtp.mailfrom=infradead.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b=nUpUfYO+; arc=none smtp.client-ip=90.155.92.199 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=infradead.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="nUpUfYO+" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=Hz7UCE6SpGqMPyTo25jsb7HnrcJHQ4a5YG8n6VtwtD0=; b=nUpUfYO+2FDb4M3TjV8sLJdP4C 5c0QznLIJ9Gh3vPhJV8pLp6ZhIlXK7RClAwo5YqCs8oiTJ1c1LpWkLAgjEnlOkzkyjD5efi39503h 6Sk0Y91oHLycdwRtaHxdJy8eHuntLQME/f6ByGeMlUHz2KfuHdNwvYydTmHLxOmfvy3nqd9hcXTLU BFjFeaI6VyK8T6pqFqBEbT43cjZuFfx7lJOyloeNOnzayH84jO1pRMyED9quROIcqYjAfUWb5r/WW eBF7+4NJ2I1qlK/C0qhMkqd4H050VUfpEcrl2SIv7hwv89jWzYrVgRFcNvnKAE1rO/nye7WH7UK83 Hh+EOTpQ==; Received: from 77-249-17-89.cable.dynamic.v4.ziggo.nl ([77.249.17.89] helo=noisy.programming.kicks-ass.net) by desiato.infradead.org with esmtpsa (Exim 4.98 #2 (Red Hat Linux)) id 1tfcb2-0000000Gbm6-132m; Wed, 05 Feb 2025 10:25:28 +0000 Received: by noisy.programming.kicks-ass.net (Postfix, from userid 0) id B928930078C; Wed, 5 Feb 2025 11:25:27 +0100 (CET) Message-ID: <20250205102448.563246589@infradead.org> User-Agent: quilt/0.66 Date: Wed, 05 Feb 2025 11:21:23 +0100 From: Peter Zijlstra To: mingo@kernel.org, ravi.bangoria@amd.com, lucas.demarchi@intel.com Cc: linux-kernel@vger.kernel.org, peterz@infradead.org, willy@infradead.org, acme@kernel.org, namhyung@kernel.org, mark.rutland@arm.com, alexander.shishkin@linux.intel.com, jolsa@kernel.org, irogers@google.com, adrian.hunter@intel.com, kan.liang@linux.intel.com Subject: [PATCH v2 03/24] perf: Simplify child event tear-down References: <20250205102120.531585416@infradead.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Currently perf_event_release_kernel() will iterate the child events and att= empt tear-down. However, it removes them from the child_list using list_move(), notably skipping the state management done by perf_child_detach(). Crucially, it fails to clear PERF_ATTACH_CHILD, which opens the door for a concurrent perf_remove_from_context() to race. This way child_list management stays fully serialized using child_mutex. Signed-off-by: Peter Zijlstra (Intel) Reviewed-by: Ravi Bangoria --- kernel/events/core.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) --- a/kernel/events/core.c +++ b/kernel/events/core.c @@ -5526,8 +5526,8 @@ int perf_event_release_kernel(struct per tmp =3D list_first_entry_or_null(&event->child_list, struct perf_event, child_list); if (tmp =3D=3D child) { - perf_remove_from_context(child, DETACH_GROUP); - list_move(&child->child_list, &free_list); + perf_remove_from_context(child, DETACH_GROUP | DETACH_CHILD); + list_add(&child->child_list, &free_list); /* * This matches the refcount bump in inherit_event(); * this can't be the last reference. From nobody Sun Feb 8 09:32:55 2026 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id ED5FC22B8C8 for ; Wed, 5 Feb 2025 10:25:37 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=90.155.50.34 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738751139; cv=none; b=qObhOSna6PAiBugiHZyEtA1af9NyWF/phOASOyB5+jtCltWjCP+ZaiY/uNr2GuO5aKSijms5qM9d1ZRzCpFdljWsiymdCK78UbSDmztvFLjaRHmoSUBjD8Q8SEzc5iEPbQSiB1tMHyyHfxuGOU4WgSbg0pv9Cho0RfbtBU+/LYo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738751139; c=relaxed/simple; bh=t+nkHao6oa2/XUVBMrLN2l1wwJK0Yjp4YK/DtpLwCAs=; h=Message-ID:Date:From:To:Cc:Subject:References:MIME-Version: Content-Type; b=SCZV796boY6S9kya8UIsPqYjzyKyCvyfSfl9fHNCUpyCvmrxyIl2h+x7grlsNAp4Z2SBJ90NtcXFaRz/98tkvVCMdv8a8s/jN9ZG6czbusmtJbmuRic73ACBVnZTIYF2GO7702wI1+pKx80MEMNrMgJyOPKV2BjBwcsXv1x+91A= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org; spf=none smtp.mailfrom=infradead.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b=ZFv74E6g; arc=none smtp.client-ip=90.155.50.34 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=infradead.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="ZFv74E6g" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=hr6tiJEA7PUuuL2CfFFjilBKdlpHbB0gBHXJT1lSU6Y=; b=ZFv74E6gei7ac7JUh1QwVEuTg5 ujc4Mkm5w9isa3F+Rj4zeIhclRWVwSJWgRYqSPI4wn5MUNjrpVIaZ+cA/Iuj4SNGMLGtip2a+f4xZ TwOynxRHSNclh8a+HJFYREMtcZlhaIPiE7KgtV/j3cnTKuf1+p/AN1hcXZW21bamG3AS00uru3PS5 R+obNUNd7JSfi0cBCkNgZDKRmgjACJhliyKO2SLyUj6V3987vq6eMBIEHO+9hkBmrPBz1sBpy018+ fTtAaCR6va4lLU5CXDyKZ4yVlsOLbMS7y6gLvDOiOh7iFxAId7GEzM96k2R8M1x1NQAS9GZbrIgFz 5OYij8+g==; Received: from 77-249-17-89.cable.dynamic.v4.ziggo.nl ([77.249.17.89] helo=noisy.programming.kicks-ass.net) by casper.infradead.org with esmtpsa (Exim 4.98 #2 (Red Hat Linux)) id 1tfcb2-00000004GPT-2YR0; Wed, 05 Feb 2025 10:25:28 +0000 Received: by noisy.programming.kicks-ass.net (Postfix, from userid 0) id BCF1D30083E; Wed, 5 Feb 2025 11:25:27 +0100 (CET) Message-ID: <20250205102448.680187551@infradead.org> User-Agent: quilt/0.66 Date: Wed, 05 Feb 2025 11:21:24 +0100 From: Peter Zijlstra To: mingo@kernel.org, ravi.bangoria@amd.com, lucas.demarchi@intel.com Cc: linux-kernel@vger.kernel.org, peterz@infradead.org, willy@infradead.org, acme@kernel.org, namhyung@kernel.org, mark.rutland@arm.com, alexander.shishkin@linux.intel.com, jolsa@kernel.org, irogers@google.com, adrian.hunter@intel.com, kan.liang@linux.intel.com Subject: [PATCH v2 04/24] perf: Simplify perf_event_free_task() wait References: <20250205102120.531585416@infradead.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Signed-off-by: Peter Zijlstra (Intel) Reviewed-by: Ravi Bangoria --- kernel/events/core.c | 32 ++++++++------------------------ 1 file changed, 8 insertions(+), 24 deletions(-) --- a/kernel/events/core.c +++ b/kernel/events/core.c @@ -1223,8 +1223,14 @@ static void put_ctx(struct perf_event_co if (refcount_dec_and_test(&ctx->refcount)) { if (ctx->parent_ctx) put_ctx(ctx->parent_ctx); - if (ctx->task && ctx->task !=3D TASK_TOMBSTONE) - put_task_struct(ctx->task); + if (ctx->task) { + if (ctx->task =3D=3D TASK_TOMBSTONE) { + smp_mb(); /* pairs with wait_var_event() */ + wake_up_var(&ctx->refcount); + } else { + put_task_struct(ctx->task); + } + } call_rcu(&ctx->rcu_head, free_ctx); } } @@ -5492,8 +5498,6 @@ int perf_event_release_kernel(struct per again: mutex_lock(&event->child_mutex); list_for_each_entry(child, &event->child_list, child_list) { - void *var =3D NULL; - /* * Cannot change, child events are not migrated, see the * comment with perf_event_ctx_lock_nested(). @@ -5533,39 +5537,19 @@ int perf_event_release_kernel(struct per * this can't be the last reference. */ put_event(event); - } else { - var =3D &ctx->refcount; } =20 mutex_unlock(&event->child_mutex); mutex_unlock(&ctx->mutex); put_ctx(ctx); =20 - if (var) { - /* - * If perf_event_free_task() has deleted all events from the - * ctx while the child_mutex got released above, make sure to - * notify about the preceding put_ctx(). - */ - smp_mb(); /* pairs with wait_var_event() */ - wake_up_var(var); - } goto again; } mutex_unlock(&event->child_mutex); =20 list_for_each_entry_safe(child, tmp, &free_list, child_list) { - void *var =3D &child->ctx->refcount; - list_del(&child->child_list); free_event(child); - - /* - * Wake any perf_event_free_task() waiting for this event to be - * freed. - */ - smp_mb(); /* pairs with wait_var_event() */ - wake_up_var(var); } =20 no_ctx: From nobody Sun Feb 8 09:32:55 2026 Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4818D22AE45 for ; Wed, 5 Feb 2025 10:25:35 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=90.155.92.199 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738751136; cv=none; b=YcZu8eq8vTajoLmoBHyDYdah3PNu+unaS/eZ/NmFZl9DgKwoSjqDDrZoF3coxVCRfLPsOdjqwnqOZz+egz5s7Kp8+t6DvSWbMmFQ1M+jyJx1+QNf7XxUbr65mhi9wpqnAzdTsIXkRQtiXHHjnO97KOaIB52tWdhjQgHurN4qn1g= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738751136; c=relaxed/simple; bh=7kv+ohn4+JMZ1rgZ5wrp+qFbM4WdOU9IkQEezJWEEA4=; h=Message-ID:Date:From:To:Cc:Subject:References:MIME-Version: Content-Type; b=Dr4Y/Xd84HgbrfdRRECOmPi5cSSfmjl5XmdiIc5yhxemfk8Wol5ncihz43NtRp+FhAf3eqH9SxJwV1bChOLsYaCm1CYgCJE7pbod06k/hVFG7tT0PjDY+Z7IWhGysv4P2yfZmuyLDDtNVtkwtGqIST7GcITtMu3LiloNVd5ZcP4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org; spf=none smtp.mailfrom=infradead.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b=OyzVMoxh; arc=none smtp.client-ip=90.155.92.199 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=infradead.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="OyzVMoxh" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=muUtDZ871e3BKh4vPnVNv+6PbMqja4ougcT0oszeQUw=; b=OyzVMoxhEvY+5dE39XSngFPzAI iPDdFIen8iE+zrnwT0smH8XB405zPTe4tbNUv6rAAs3nfObF7y/pB3J3sGPkRCJ44SJddRG61o5P2 xE1fVtSuAYO3ih1GznSThGNW4lL4Fh7S7SV+s2TSEQcoPtL8NaA78OERzIVFWNeyaUfzNHUdthRq7 bjO89Ul80pvhTq5IZa4gGR7CnN0nODgR30slcKFwolzdDkKegbpWXnLm2bXYERdEp300aHFA+528x nP17Zh47sQe6N3ymxFiK3KUev0suMhg6GbUOomz6FDi+/czbQ/Uuo7bS/Gzdaw0IMS5ZxedkGkQNh WbElW3Hw==; Received: from 77-249-17-89.cable.dynamic.v4.ziggo.nl ([77.249.17.89] helo=noisy.programming.kicks-ass.net) by desiato.infradead.org with esmtpsa (Exim 4.98 #2 (Red Hat Linux)) id 1tfcb2-0000000Gbm9-40Ll; Wed, 05 Feb 2025 10:25:29 +0000 Received: by noisy.programming.kicks-ass.net (Postfix, from userid 0) id C0F6B302E7A; Wed, 5 Feb 2025 11:25:27 +0100 (CET) Message-ID: <20250205102448.789958355@infradead.org> User-Agent: quilt/0.66 Date: Wed, 05 Feb 2025 11:21:25 +0100 From: Peter Zijlstra To: mingo@kernel.org, ravi.bangoria@amd.com, lucas.demarchi@intel.com Cc: linux-kernel@vger.kernel.org, peterz@infradead.org, willy@infradead.org, acme@kernel.org, namhyung@kernel.org, mark.rutland@arm.com, alexander.shishkin@linux.intel.com, jolsa@kernel.org, irogers@google.com, adrian.hunter@intel.com, kan.liang@linux.intel.com Subject: [PATCH v2 05/24] perf: Simplify perf_event_release_kernel() References: <20250205102120.531585416@infradead.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Get rid of the free_list... Signed-off-by: Peter Zijlstra (Intel) Reviewed-by: Ravi Bangoria --- kernel/events/core.c | 12 +++++------- 1 file changed, 5 insertions(+), 7 deletions(-) --- a/kernel/events/core.c +++ b/kernel/events/core.c @@ -5462,7 +5462,6 @@ int perf_event_release_kernel(struct per { struct perf_event_context *ctx =3D event->ctx; struct perf_event *child, *tmp; - LIST_HEAD(free_list); =20 /* * If we got here through err_alloc: free_event(event); we will not @@ -5531,27 +5530,26 @@ int perf_event_release_kernel(struct per struct perf_event, child_list); if (tmp =3D=3D child) { perf_remove_from_context(child, DETACH_GROUP | DETACH_CHILD); - list_add(&child->child_list, &free_list); /* * This matches the refcount bump in inherit_event(); * this can't be the last reference. */ put_event(event); + } else { + child =3D NULL; } =20 mutex_unlock(&event->child_mutex); mutex_unlock(&ctx->mutex); + + if (child) + free_event(child); put_ctx(ctx); =20 goto again; } mutex_unlock(&event->child_mutex); =20 - list_for_each_entry_safe(child, tmp, &free_list, child_list) { - list_del(&child->child_list); - free_event(child); - } - no_ctx: put_event(event); /* Must be the 'last' reference */ return 0; From nobody Sun Feb 8 09:32:55 2026 Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3B4BC22B587 for ; Wed, 5 Feb 2025 10:25:36 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=90.155.92.199 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738751139; cv=none; b=lRPl0v6GitPdrb6mQD6o1jDhvSHkzz8jBrtiKquqpyThO2l0GJb64gVbx7QIU6d+YH279HFyJwIqRQVQGUv14HrOVOwMUZORlL8iM7zhNCpGAh50ZOujI7AYnqYDm0BJb0foZhfOpbvtcSnElOmIcjSEHiQhuJpBj6z9UZicUjo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738751139; c=relaxed/simple; bh=k/cG38pmzyjUIZk1609SZCN25ncHk8XKRnc054U+rLI=; h=Message-ID:Date:From:To:Cc:Subject:References:MIME-Version: Content-Type; b=Uf2g/flvWUBRuZs1ExFq70+ZpLJtzjgIx/D7duig5dtYH6iagPqjKIB5CjcGtQkcLUA9mJKB4rjC9g3QaKM7Bve86n1aZq3e58ZaIQ1ElOBGxRwMJMIZhXtWcBdopfl3ZXwD3jmX7pIkP6hPYgxrFraOBp1j4nDebaeWzWhTHIU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org; spf=none smtp.mailfrom=infradead.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b=CLqmmzEl; arc=none smtp.client-ip=90.155.92.199 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=infradead.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="CLqmmzEl" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=awRwc3C7jyT26FBW24uhR95uNITqcA5r2W8hvYtalrM=; b=CLqmmzElantDocWicWXo46ZDoM ZxwKXd5pR5JlWruytj68BVCKyWw4bsDzgc/KpL+YC3QI7PodPo6QUofwDzYwonIKj0MnB/B+6/EkJ iyP+9A8hIXlohpDN6xSpemFPt6yXVIN670dC9/ujEqoncOp0q+qO/36eE9AvxqA85LhDAboE3GM3o TxI+4gv1kO3BZ+H1viYBho6yul/4xUr1xrgHdIup4gPZM0bN86042f/tCss9c3RJdJw6NS+N5W3PI 0kjImxhoTTOCTE5UOdxn7srHvPjnk/zr53Lb6Wz5fU8Yus9xoh81s+5hDBJMfb4/Gd108EQ4PiJYK bt5Mf6kA==; Received: from 77-249-17-89.cable.dynamic.v4.ziggo.nl ([77.249.17.89] helo=noisy.programming.kicks-ass.net) by desiato.infradead.org with esmtpsa (Exim 4.98 #2 (Red Hat Linux)) id 1tfcb2-0000000GbmA-41ew; Wed, 05 Feb 2025 10:25:30 +0000 Received: by noisy.programming.kicks-ass.net (Postfix, from userid 0) id C4C33302E9C; Wed, 5 Feb 2025 11:25:27 +0100 (CET) Message-ID: <20250205102448.898435571@infradead.org> User-Agent: quilt/0.66 Date: Wed, 05 Feb 2025 11:21:26 +0100 From: Peter Zijlstra To: mingo@kernel.org, ravi.bangoria@amd.com, lucas.demarchi@intel.com Cc: linux-kernel@vger.kernel.org, peterz@infradead.org, willy@infradead.org, acme@kernel.org, namhyung@kernel.org, mark.rutland@arm.com, alexander.shishkin@linux.intel.com, jolsa@kernel.org, irogers@google.com, adrian.hunter@intel.com, kan.liang@linux.intel.com Subject: [PATCH v2 06/24] perf: Fix pmus_lock vs pmus_srcu ordering References: <20250205102120.531585416@infradead.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Commit a63fbed776c7 ("perf/tracing/cpuhotplug: Fix locking order") placed pmus_lock inside pmus_srcu, this makes perf_pmu_unregister() trip lockdep. Move the locking about such that only pmu_idr and pmus (list) are modified while holding pmus_lock. This avoids doing synchronize_srcu() while holding pmus_lock and all is well again. Fixes: a63fbed776c7 ("perf/tracing/cpuhotplug: Fix locking order") Signed-off-by: Peter Zijlstra (Intel) Reviewed-by: Ravi Bangoria Tested-by: Lucas De Marchi --- kernel/events/core.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) --- a/kernel/events/core.c +++ b/kernel/events/core.c @@ -11836,6 +11836,8 @@ void perf_pmu_unregister(struct pmu *pmu { mutex_lock(&pmus_lock); list_del_rcu(&pmu->entry); + idr_remove(&pmu_idr, pmu->type); + mutex_unlock(&pmus_lock); =20 /* * We dereference the pmu list under both SRCU and regular RCU, so @@ -11845,7 +11847,6 @@ void perf_pmu_unregister(struct pmu *pmu synchronize_rcu(); =20 free_percpu(pmu->pmu_disable_count); - idr_remove(&pmu_idr, pmu->type); if (pmu_bus_running && pmu->dev && pmu->dev !=3D PMU_NULL_DEV) { if (pmu->nr_addr_filters) device_remove_file(pmu->dev, &dev_attr_nr_addr_filters); @@ -11853,7 +11854,6 @@ void perf_pmu_unregister(struct pmu *pmu put_device(pmu->dev); } free_pmu_context(pmu); - mutex_unlock(&pmus_lock); } EXPORT_SYMBOL_GPL(perf_pmu_unregister); From nobody Sun Feb 8 09:32:55 2026 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id DEA012309B3 for ; Wed, 5 Feb 2025 10:25:50 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=90.155.50.34 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738751152; cv=none; b=B5dVxgkTPK5Uf1wn1k9eWvwUSZgSFLWsBY5QUfS7citewkqI/w45lm1/gk86GPOZcp1eNObijTY/396/Y5X46/Ag9uIlgzlLmx/pfEPZorlhS8zG7surfFze3L0yO6ocoqFW0U4tubUQR21XOVLTMXoK3NDhhXv/zn+oJv5UKyo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738751152; c=relaxed/simple; bh=AeZhM/tJiHVa4QhSBs3iTgAN3KRu0glOTLTyr6soPK8=; h=Message-ID:Date:From:To:Cc:Subject:References:MIME-Version: Content-Type; b=H2iexXxyKy/ZW8/H/2XGjvmOMMQ96hRdfayZ7glUVZLtFej/1/mcvPCAgfQq8SNlvDQqCnh+05XOqu7jhvUAWnFq2dKcc53AzccMYwEa3MhfSoIpsKRPFMazEwuDvSVKK6dUEhrYtTx/elixFC9LIZkURZUXtolb3fGiB8CqkXg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org; spf=none smtp.mailfrom=infradead.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b=vBRiqrBp; arc=none smtp.client-ip=90.155.50.34 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=infradead.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="vBRiqrBp" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=rqt7ldGfsvkJgM5c0RTcP/CKPPdkVrAGzemxz4WCgPU=; b=vBRiqrBp1uzhzfBZdCI+dV/C8F iBlVwCOWgmuC0cKOVHJ7buPKxzUW9NNhmivEM39KkcFktD1IUIBhlmq1ELXx1hYmaKOlQa22nCunj c6VuXjGey6RV8qhaBICOdQp4q2dF/V0P4F56Iy1rJ9yDOe9bKKeBwlRlnHyKzme+HGDBZZy3wyXm3 kEWmucFniSQw1lNlwNCgvIhhAI8wMcNICH0cDXa6Y+/SUefaTzdrcObVKy6u53uz6yN0kG6iofwMg 3ReYJAjGmq6f7y1371yZRZm8lqALSHLt7SOYfvEVSGojiwZ8cEWREkX7pL0e/KITKbzpB3Y2ltShL mc7deYdQ==; Received: from 77-249-17-89.cable.dynamic.v4.ziggo.nl ([77.249.17.89] helo=noisy.programming.kicks-ass.net) by casper.infradead.org with esmtpsa (Exim 4.98 #2 (Red Hat Linux)) id 1tfcb3-00000004GPY-1wdb; Wed, 05 Feb 2025 10:25:29 +0000 Received: by noisy.programming.kicks-ass.net (Postfix, from userid 0) id C8B113031DE; Wed, 5 Feb 2025 11:25:27 +0100 (CET) Message-ID: <20250205102449.006102855@infradead.org> User-Agent: quilt/0.66 Date: Wed, 05 Feb 2025 11:21:27 +0100 From: Peter Zijlstra To: mingo@kernel.org, ravi.bangoria@amd.com, lucas.demarchi@intel.com Cc: linux-kernel@vger.kernel.org, peterz@infradead.org, willy@infradead.org, acme@kernel.org, namhyung@kernel.org, mark.rutland@arm.com, alexander.shishkin@linux.intel.com, jolsa@kernel.org, irogers@google.com, adrian.hunter@intel.com, kan.liang@linux.intel.com Subject: [PATCH v2 07/24] perf: Fix perf_pmu_register() vs perf_init_event() References: <20250205102120.531585416@infradead.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" There is a fairly obvious race between perf_init_event() doing idr_find() and perf_pmu_register() doing idr_alloc() with an incompletely initialized pmu pointer. Avoid by doing idr_alloc() on a NULL pointer to register the id, and swizzling the real pmu pointer at the end using idr_replace(). Also making sure to not set pmu members after publishing the pmu, duh. [ introduce idr_cmpxchg() in order to better handle the idr_replace() error case -- if it were to return an unexpected pointer, it will already have replaced the value and there is no going back. ] Signed-off-by: Peter Zijlstra (Intel) Reviewed-by: Ravi Bangoria --- kernel/events/core.c | 28 ++++++++++++++++++++++++++-- 1 file changed, 26 insertions(+), 2 deletions(-) --- a/kernel/events/core.c +++ b/kernel/events/core.c @@ -11739,6 +11739,21 @@ static int pmu_dev_alloc(struct pmu *pmu static struct lock_class_key cpuctx_mutex; static struct lock_class_key cpuctx_lock; =20 +static bool idr_cmpxchg(struct idr *idr, unsigned long id, void *old, void= *new) +{ + void *tmp, *val =3D idr_find(idr, id); + + if (val !=3D old) + return false; + + tmp =3D idr_replace(idr, new, id); + if (IS_ERR(tmp)) + return false; + + WARN_ON_ONCE(tmp !=3D val); + return true; +} + int perf_pmu_register(struct pmu *pmu, const char *name, int type) { int cpu, ret, max =3D PERF_TYPE_MAX; @@ -11765,7 +11780,7 @@ int perf_pmu_register(struct pmu *pmu, c if (type >=3D 0) max =3D type; =20 - ret =3D idr_alloc(&pmu_idr, pmu, max, 0, GFP_KERNEL); + ret =3D idr_alloc(&pmu_idr, NULL, max, 0, GFP_KERNEL); if (ret < 0) goto free_pdc; =20 @@ -11773,6 +11788,7 @@ int perf_pmu_register(struct pmu *pmu, c =20 type =3D ret; pmu->type =3D type; + atomic_set(&pmu->exclusive_cnt, 0); =20 if (pmu_bus_running && !pmu->dev) { ret =3D pmu_dev_alloc(pmu); @@ -11821,14 +11837,22 @@ int perf_pmu_register(struct pmu *pmu, c if (!pmu->event_idx) pmu->event_idx =3D perf_event_idx_default; =20 + /* + * Now that the PMU is complete, make it visible to perf_try_init_event(). + */ + if (!idr_cmpxchg(&pmu_idr, pmu->type, NULL, pmu)) + goto free_context; list_add_rcu(&pmu->entry, &pmus); - atomic_set(&pmu->exclusive_cnt, 0); + ret =3D 0; unlock: mutex_unlock(&pmus_lock); =20 return ret; =20 +free_context: + free_percpu(pmu->cpu_pmu_context); + free_dev: if (pmu->dev && pmu->dev !=3D PMU_NULL_DEV) { device_del(pmu->dev); From nobody Sun Feb 8 09:32:55 2026 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id AAD6722F399 for ; Wed, 5 Feb 2025 10:25:43 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=90.155.50.34 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738751145; cv=none; b=J6Q1yt9HsFk3/tKZ2sF99xMaLeXmObdb4dabqJqQT/mD8VcGUQlto6w412mL8/wN3X3ADoEa/naD5Ostp6gk62vZAYimeMuyp+VIBe2tLGIyX7o51DHpDV2S9QLS4fWLXi9K9BS4NqH8sIQF4+Boanl2ugOqtKjG5fZeGslDv6Y= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738751145; c=relaxed/simple; bh=V5gRc9xaSn1o1EJddIrGLWliyFJt9NbqAOiRFB5WSpI=; h=Message-ID:Date:From:To:Cc:Subject:References:MIME-Version: Content-Type; b=MNPfjnOtvN1kUr7yMyJ2/UiSEN6+uomCgYChTGoI7hFYzT62izdn2+bZ69fjIB6lVat33qgddkrUpTbxPZdkSCuYGiK8MIl5QOe2E3oRprHA0+opNjkamWigEfvFj/IlhMd6soNqU4cUAgabMTJQ1+E4BkB6UtvD+YRaJyudxvk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org; spf=none smtp.mailfrom=infradead.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b=rvVT9Y6Q; arc=none smtp.client-ip=90.155.50.34 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=infradead.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="rvVT9Y6Q" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=caaHUBwU3FjXVkr/jMxyi0Xl6jcwm1jZ/HPyaNVGz1c=; b=rvVT9Y6QWlCkTCvxMkqE1VeA9T i1JZFZO+NBH72LciNsxNNhiERji24+Emb7gdfGg8CeZHjfaTFxqHNnfJytzdyJJElcUwBko74iv2d j4IlVXCuyeMfnTTJ+CjN6sULirxzXwQMJO7bGYiXp78dS1uzpOyRl7i6I63RqZ9dBYTioLcCQYevK bKttlyLVu9A2TdJu8rvPMOh3Ak0lYYADhOsJo5ugy7BqfEIfRi22C88Cjr7VpBknoKM2euWlOZtJ6 bsD0QlbWgLLDphQP0FLWYdu0Xitl0fclSDWPsXKPiq4djky3WPUjtU5mVJppWaR9WuOmrQeN1iuZ8 pI4UYmGw==; Received: from 77-249-17-89.cable.dynamic.v4.ziggo.nl ([77.249.17.89] helo=noisy.programming.kicks-ass.net) by casper.infradead.org with esmtpsa (Exim 4.98 #2 (Red Hat Linux)) id 1tfcb3-00000004GPZ-24yU; Wed, 05 Feb 2025 10:25:29 +0000 Received: by noisy.programming.kicks-ass.net (Postfix, from userid 0) id CC50530619B; Wed, 5 Feb 2025 11:25:27 +0100 (CET) Message-ID: <20250205102449.110145835@infradead.org> User-Agent: quilt/0.66 Date: Wed, 05 Feb 2025 11:21:28 +0100 From: Peter Zijlstra To: mingo@kernel.org, ravi.bangoria@amd.com, lucas.demarchi@intel.com Cc: linux-kernel@vger.kernel.org, peterz@infradead.org, willy@infradead.org, acme@kernel.org, namhyung@kernel.org, mark.rutland@arm.com, alexander.shishkin@linux.intel.com, jolsa@kernel.org, irogers@google.com, adrian.hunter@intel.com, kan.liang@linux.intel.com Subject: [PATCH v2 08/24] perf: Cleanup perf_try_init_event() References: <20250205102120.531585416@infradead.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Make sure that perf_try_init_event() doesn't leave event->pmu nor event->destroy set on failure. Signed-off-by: Peter Zijlstra (Intel) Reviewed-by: Ravi Bangoria --- kernel/events/core.c | 69 ++++++++++++++++++++++++++++++----------------= ----- 1 file changed, 41 insertions(+), 28 deletions(-) --- a/kernel/events/core.c +++ b/kernel/events/core.c @@ -12019,38 +12019,51 @@ static int perf_try_init_event(struct pm if (ctx) perf_event_ctx_unlock(event->group_leader, ctx); =20 - if (!ret) { - if (!(pmu->capabilities & PERF_PMU_CAP_EXTENDED_REGS) && - has_extended_regs(event)) - ret =3D -EOPNOTSUPP; - - if (pmu->capabilities & PERF_PMU_CAP_NO_EXCLUDE && - event_has_any_exclude_flag(event)) - ret =3D -EINVAL; - - if (pmu->scope !=3D PERF_PMU_SCOPE_NONE && event->cpu >=3D 0) { - const struct cpumask *cpumask =3D perf_scope_cpu_topology_cpumask(pmu->= scope, event->cpu); - struct cpumask *pmu_cpumask =3D perf_scope_cpumask(pmu->scope); - int cpu; - - if (pmu_cpumask && cpumask) { - cpu =3D cpumask_any_and(pmu_cpumask, cpumask); - if (cpu >=3D nr_cpu_ids) - ret =3D -ENODEV; - else - event->event_caps |=3D PERF_EV_CAP_READ_SCOPE; - } else { - ret =3D -ENODEV; - } - } + if (ret) + goto err_pmu; =20 - if (ret && event->destroy) - event->destroy(event); + if (!(pmu->capabilities & PERF_PMU_CAP_EXTENDED_REGS) && + has_extended_regs(event)) { + ret =3D -EOPNOTSUPP; + goto err_destroy; } =20 - if (ret) - module_put(pmu->module); + if (pmu->capabilities & PERF_PMU_CAP_NO_EXCLUDE && + event_has_any_exclude_flag(event)) { + ret =3D -EINVAL; + goto err_destroy; + } + + if (pmu->scope !=3D PERF_PMU_SCOPE_NONE && event->cpu >=3D 0) { + const struct cpumask *cpumask; + struct cpumask *pmu_cpumask; + int cpu; + + cpumask =3D perf_scope_cpu_topology_cpumask(pmu->scope, event->cpu); + pmu_cpumask =3D perf_scope_cpumask(pmu->scope); + + ret =3D -ENODEV; + if (!pmu_cpumask || !cpumask) + goto err_destroy; + + cpu =3D cpumask_any_and(pmu_cpumask, cpumask); + if (cpu >=3D nr_cpu_ids) + goto err_destroy; + + event->event_caps |=3D PERF_EV_CAP_READ_SCOPE; + } + + return 0; + +err_destroy: + if (event->destroy) { + event->destroy(event); + event->destroy =3D NULL; + } =20 +err_pmu: + event->pmu =3D NULL; + module_put(pmu->module); return ret; } From nobody Sun Feb 8 09:32:55 2026 Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 97D2922AE71 for ; Wed, 5 Feb 2025 10:25:35 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=90.155.92.199 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738751137; cv=none; b=qIeaQ5cP3xJy3/djMsKEfPM6Thls/TuS4ZTHwSiMnfqB6lt6o8PhrO7HulvK2YMM1kYGrkPfpPR6KkfNBwVBowBTRLocazlpudmKTb1DckDnJayeTUTPhN3jRndSiAoji8M7XZF9qpcvAVYDsnub4RkdEZpYL53fUntEI2lL6MM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738751137; c=relaxed/simple; bh=lYXRXPP6HLwg/Bp8nCAcHAAa3/kzpOc+P0uFm2dk0qY=; h=Message-ID:Date:From:To:Cc:Subject:References:MIME-Version: Content-Type; b=rV1mjIjnbxU7lsTmdAHd/p+8Exz09vzZFapDS3z3pyVBYmQe2G006uDNu9EC8n0befL20q7fHoincTSfwJLM28uMQLPgagxjy4cgzGGpkZSXaIHKBFd/1//It8m6LpDZKmf/ngE/BOxBBxb8Qc3OjqD4YeINB96Ow0KOJ8KxTZ0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org; spf=none smtp.mailfrom=infradead.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b=Sm8UQ8fQ; arc=none smtp.client-ip=90.155.92.199 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=infradead.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="Sm8UQ8fQ" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=WAF72F0vl5OegJiTt7utxhY0fac2CVe/BE2y5cW/mMs=; b=Sm8UQ8fQ/WuHYbGLDVWldUILRR iX46Hm0IIvmdJb1HydK3ojcaZNuvazjyoFMGV4XM7RVepjF6oaw4Cw7P3ww43BYxZCUKi4IPmLO5a vS1w5BjeXEDrHJEL39m707GDY1AaW12KLdiPsddReCq+uquitbjqsof61dgYmqZxt8RCpu2e38gn0 0qh/YpvCg0n81JRT6BtDGulK40wT8L9/yb5Znymmuw80SeUaUSxTbDyjGn49byZuZbanpjBlf9fB2 lNjhijLaG/9weTj+M1WgwgM1RyFsoBFA2MnaaI2QzFm8WI1TNreDQb6j3pMP3zaz7BHqE525tkdqc IPDuWiug==; Received: from 77-249-17-89.cable.dynamic.v4.ziggo.nl ([77.249.17.89] helo=noisy.programming.kicks-ass.net) by desiato.infradead.org with esmtpsa (Exim 4.98 #2 (Red Hat Linux)) id 1tfcb3-0000000GbmD-2SoP; Wed, 05 Feb 2025 10:25:32 +0000 Received: by noisy.programming.kicks-ass.net (Postfix, from userid 0) id CFD153061E4; Wed, 5 Feb 2025 11:25:27 +0100 (CET) Message-ID: <20250205102449.230417308@infradead.org> User-Agent: quilt/0.66 Date: Wed, 05 Feb 2025 11:21:29 +0100 From: Peter Zijlstra To: mingo@kernel.org, ravi.bangoria@amd.com, lucas.demarchi@intel.com Cc: linux-kernel@vger.kernel.org, peterz@infradead.org, willy@infradead.org, acme@kernel.org, namhyung@kernel.org, mark.rutland@arm.com, alexander.shishkin@linux.intel.com, jolsa@kernel.org, irogers@google.com, adrian.hunter@intel.com, kan.liang@linux.intel.com Subject: [PATCH v2 09/24] perf: Simplify perf_event_alloc() error path References: <20250205102120.531585416@infradead.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" The error cleanup sequence in perf_event_alloc() is a subset of the existing _free_event() function (it must of course be). Split this out into __free_event() and simplify the error path. Signed-off-by: Peter Zijlstra (Intel) Reviewed-by: Ravi Bangoria --- include/linux/perf_event.h | 16 +++-- kernel/events/core.c | 134 ++++++++++++++++++++++------------------= ----- 2 files changed, 75 insertions(+), 75 deletions(-) --- a/include/linux/perf_event.h +++ b/include/linux/perf_event.h @@ -673,13 +673,15 @@ struct swevent_hlist { struct rcu_head rcu_head; }; =20 -#define PERF_ATTACH_CONTEXT 0x01 -#define PERF_ATTACH_GROUP 0x02 -#define PERF_ATTACH_TASK 0x04 -#define PERF_ATTACH_TASK_DATA 0x08 -#define PERF_ATTACH_ITRACE 0x10 -#define PERF_ATTACH_SCHED_CB 0x20 -#define PERF_ATTACH_CHILD 0x40 +#define PERF_ATTACH_CONTEXT 0x0001 +#define PERF_ATTACH_GROUP 0x0002 +#define PERF_ATTACH_TASK 0x0004 +#define PERF_ATTACH_TASK_DATA 0x0008 +#define PERF_ATTACH_ITRACE 0x0010 +#define PERF_ATTACH_SCHED_CB 0x0020 +#define PERF_ATTACH_CHILD 0x0040 +#define PERF_ATTACH_EXCLUSIVE 0x0080 +#define PERF_ATTACH_CALLCHAIN 0x0100 =20 struct bpf_prog; struct perf_cgroup; --- a/kernel/events/core.c +++ b/kernel/events/core.c @@ -5245,6 +5245,8 @@ static int exclusive_event_init(struct p return -EBUSY; } =20 + event->attach_state |=3D PERF_ATTACH_EXCLUSIVE; + return 0; } =20 @@ -5252,14 +5254,13 @@ static void exclusive_event_destroy(stru { struct pmu *pmu =3D event->pmu; =20 - if (!is_exclusive_pmu(pmu)) - return; - /* see comment in exclusive_event_init() */ if (event->attach_state & PERF_ATTACH_TASK) atomic_dec(&pmu->exclusive_cnt); else atomic_inc(&pmu->exclusive_cnt); + + event->attach_state &=3D ~PERF_ATTACH_EXCLUSIVE; } =20 static bool exclusive_event_match(struct perf_event *e1, struct perf_event= *e2) @@ -5318,40 +5319,20 @@ static void perf_pending_task_sync(struc rcuwait_wait_event(&event->pending_work_wait, !event->pending_work, TASK_= UNINTERRUPTIBLE); } =20 -static void _free_event(struct perf_event *event) +/* vs perf_event_alloc() error */ +static void __free_event(struct perf_event *event) { - irq_work_sync(&event->pending_irq); - irq_work_sync(&event->pending_disable_irq); - perf_pending_task_sync(event); - - unaccount_event(event); + if (event->attach_state & PERF_ATTACH_CALLCHAIN) + put_callchain_buffers(); =20 - security_perf_event_free(event); + kfree(event->addr_filter_ranges); =20 - if (event->rb) { - /* - * Can happen when we close an event with re-directed output. - * - * Since we have a 0 refcount, perf_mmap_close() will skip - * over us; possibly making our ring_buffer_put() the last. - */ - mutex_lock(&event->mmap_mutex); - ring_buffer_attach(event, NULL); - mutex_unlock(&event->mmap_mutex); - } + if (event->attach_state & PERF_ATTACH_EXCLUSIVE) + exclusive_event_destroy(event); =20 if (is_cgroup_event(event)) perf_detach_cgroup(event); =20 - if (!event->parent) { - if (event->attr.sample_type & PERF_SAMPLE_CALLCHAIN) - put_callchain_buffers(); - } - - perf_event_free_bpf_prog(event); - perf_addr_filters_splice(event, NULL); - kfree(event->addr_filter_ranges); - if (event->destroy) event->destroy(event); =20 @@ -5362,22 +5343,58 @@ static void _free_event(struct perf_even if (event->hw.target) put_task_struct(event->hw.target); =20 - if (event->pmu_ctx) + if (event->pmu_ctx) { + /* + * put_pmu_ctx() needs an event->ctx reference, because of + * epc->ctx. + */ + WARN_ON_ONCE(!event->ctx); + WARN_ON_ONCE(event->pmu_ctx->ctx !=3D event->ctx); put_pmu_ctx(event->pmu_ctx); + } =20 /* - * perf_event_free_task() relies on put_ctx() being 'last', in particular - * all task references must be cleaned up. + * perf_event_free_task() relies on put_ctx() being 'last', in + * particular all task references must be cleaned up. */ if (event->ctx) put_ctx(event->ctx); =20 - exclusive_event_destroy(event); - module_put(event->pmu->module); + if (event->pmu) + module_put(event->pmu->module); =20 call_rcu(&event->rcu_head, free_event_rcu); } =20 +/* vs perf_event_alloc() success */ +static void _free_event(struct perf_event *event) +{ + irq_work_sync(&event->pending_irq); + irq_work_sync(&event->pending_disable_irq); + perf_pending_task_sync(event); + + unaccount_event(event); + + security_perf_event_free(event); + + if (event->rb) { + /* + * Can happen when we close an event with re-directed output. + * + * Since we have a 0 refcount, perf_mmap_close() will skip + * over us; possibly making our ring_buffer_put() the last. + */ + mutex_lock(&event->mmap_mutex); + ring_buffer_attach(event, NULL); + mutex_unlock(&event->mmap_mutex); + } + + perf_event_free_bpf_prog(event); + perf_addr_filters_splice(event, NULL); + + __free_event(event); +} + /* * Used to free events which have a known refcount of 1, such as in error = paths * where the event isn't exposed yet and inherited events. @@ -12390,7 +12407,7 @@ perf_event_alloc(struct perf_event_attr * See perf_output_read(). */ if (has_inherit_and_sample_read(attr) && !(attr->sample_type & PERF_SAMPL= E_TID)) - goto err_ns; + goto err; =20 if (!has_branch_stack(event)) event->attr.branch_sample_type =3D 0; @@ -12398,7 +12415,7 @@ perf_event_alloc(struct perf_event_attr pmu =3D perf_init_event(event); if (IS_ERR(pmu)) { err =3D PTR_ERR(pmu); - goto err_ns; + goto err; } =20 /* @@ -12408,25 +12425,25 @@ perf_event_alloc(struct perf_event_attr */ if (pmu->task_ctx_nr =3D=3D perf_invalid_context && (task || cgroup_fd != =3D -1)) { err =3D -EINVAL; - goto err_pmu; + goto err; } =20 if (event->attr.aux_output && (!(pmu->capabilities & PERF_PMU_CAP_AUX_OUTPUT) || event->attr.aux_pause || event->attr.aux_resume)) { err =3D -EOPNOTSUPP; - goto err_pmu; + goto err; } =20 if (event->attr.aux_pause && event->attr.aux_resume) { err =3D -EINVAL; - goto err_pmu; + goto err; } =20 if (event->attr.aux_start_paused) { if (!(pmu->capabilities & PERF_PMU_CAP_AUX_PAUSE)) { err =3D -EOPNOTSUPP; - goto err_pmu; + goto err; } event->hw.aux_paused =3D 1; } @@ -12434,12 +12451,12 @@ perf_event_alloc(struct perf_event_attr if (cgroup_fd !=3D -1) { err =3D perf_cgroup_connect(cgroup_fd, event, attr, group_leader); if (err) - goto err_pmu; + goto err; } =20 err =3D exclusive_event_init(event); if (err) - goto err_pmu; + goto err; =20 if (has_addr_filter(event)) { event->addr_filter_ranges =3D kcalloc(pmu->nr_addr_filters, @@ -12447,7 +12464,7 @@ perf_event_alloc(struct perf_event_attr GFP_KERNEL); if (!event->addr_filter_ranges) { err =3D -ENOMEM; - goto err_per_task; + goto err; } =20 /* @@ -12472,41 +12489,22 @@ perf_event_alloc(struct perf_event_attr if (event->attr.sample_type & PERF_SAMPLE_CALLCHAIN) { err =3D get_callchain_buffers(attr->sample_max_stack); if (err) - goto err_addr_filters; + goto err; + event->attach_state |=3D PERF_ATTACH_CALLCHAIN; } } =20 err =3D security_perf_event_alloc(event); if (err) - goto err_callchain_buffer; + goto err; =20 /* symmetric to unaccount_event() in _free_event() */ account_event(event); =20 return event; =20 -err_callchain_buffer: - if (!event->parent) { - if (event->attr.sample_type & PERF_SAMPLE_CALLCHAIN) - put_callchain_buffers(); - } -err_addr_filters: - kfree(event->addr_filter_ranges); - -err_per_task: - exclusive_event_destroy(event); - -err_pmu: - if (is_cgroup_event(event)) - perf_detach_cgroup(event); - if (event->destroy) - event->destroy(event); - module_put(pmu->module); -err_ns: - if (event->hw.target) - put_task_struct(event->hw.target); - call_rcu(&event->rcu_head, free_event_rcu); - +err: + __free_event(event); return ERR_PTR(err); } From nobody Sun Feb 8 09:32:55 2026 Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 34DD222B584 for ; Wed, 5 Feb 2025 10:25:36 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=90.155.92.199 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738751138; cv=none; b=bzkfe9E6b1CBPhIoBx41gs9rdrndsKI4p5Qn1HN57zrXvkF00beNDWhFHf2EUPqLRUp8UWGatmrGAewJ84qoYljbwShC5laYiw8R26AFrolwYdKvVgFgnDyBhNOa3+gRP2YHd9BCWehxlung2RAKmXfcpiMJAdQ5wUB3CSwG6d4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738751138; c=relaxed/simple; bh=Uo3iRwMo30qUuwd+fK7rUxoC49gNZhQz6oTLxcvKrjw=; h=Message-ID:Date:From:To:Cc:Subject:References:MIME-Version: Content-Type; b=e2IubtZardAlouXBmbahD1X4Go1F6hgBLMfIqxtxMNoQZVAUMR5mQurfWqiI2lLhju5W2offX/4s/Zwo35X6j6SkioM4Lcg6RdxKjuF420p3pfKgjYyKooUJ2WigjKkt5Hbmn1yqUE69m2n1mIMTOxhhCtAUzFi3LpUcohKnlPw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org; spf=none smtp.mailfrom=infradead.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b=egFaWgvu; arc=none smtp.client-ip=90.155.92.199 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=infradead.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="egFaWgvu" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=0c5KDEJ1EMCTvwH6BbR0x0uKuseQOdbaYO4MmFZng44=; b=egFaWgvumZPBW/cOOY2E4oXebI zc10uTwSyWLGuRUeNBdleXsnVcwgO72cTxGNbANZ2hlMUxBS7yFqYnorXnUOMUhNraNUH3ixcCPS3 c+1VKb7w69g3nt1EvpBTMClR4NyHSvwPRNN0vrPTmNsbbI4N6UKnCnqIvmkR+vwhS8o8ndsVLTtXy sKscBHGLRDVrakmbtWlyjsQZB9A7Q/9z1u/QZX/l4X1LaoRUyJVpxoD7laHfqp6vdfqSr1vwa79om 989I3TahhyrbhWiYJtMeIo5hm7RpWuMh9LvgJN1mVMTiidHVvMCqsQ9t6JegDU8zcaECYIgRGdr2u i4M4aPsQ==; Received: from 77-249-17-89.cable.dynamic.v4.ziggo.nl ([77.249.17.89] helo=noisy.programming.kicks-ass.net) by desiato.infradead.org with esmtpsa (Exim 4.98 #2 (Red Hat Linux)) id 1tfcb3-0000000GbmC-23Lr; Wed, 05 Feb 2025 10:25:29 +0000 Received: by noisy.programming.kicks-ass.net (Postfix, from userid 0) id D34A23061EE; Wed, 5 Feb 2025 11:25:27 +0100 (CET) Message-ID: <20250205102449.342213679@infradead.org> User-Agent: quilt/0.66 Date: Wed, 05 Feb 2025 11:21:30 +0100 From: Peter Zijlstra To: mingo@kernel.org, ravi.bangoria@amd.com, lucas.demarchi@intel.com Cc: linux-kernel@vger.kernel.org, peterz@infradead.org, willy@infradead.org, acme@kernel.org, namhyung@kernel.org, mark.rutland@arm.com, alexander.shishkin@linux.intel.com, jolsa@kernel.org, irogers@google.com, adrian.hunter@intel.com, kan.liang@linux.intel.com Subject: [PATCH v2 10/24] perf: Simplify perf_pmu_register() error path References: <20250205102120.531585416@infradead.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" The error path of perf_pmu_register() is of course very similar to a subset of perf_pmu_unregister(). Extract this common part in perf_pmu_free() and simplify things. Signed-off-by: Peter Zijlstra (Intel) Reviewed-by: Ravi Bangoria --- kernel/events/core.c | 67 ++++++++++++++++++++++------------------------= ----- 1 file changed, 30 insertions(+), 37 deletions(-) --- a/kernel/events/core.c +++ b/kernel/events/core.c @@ -11605,11 +11605,6 @@ static int perf_event_idx_default(struct return 0; } =20 -static void free_pmu_context(struct pmu *pmu) -{ - free_percpu(pmu->cpu_pmu_context); -} - /* * Let userspace know that this PMU supports address range filtering: */ @@ -11815,6 +11810,7 @@ static int pmu_dev_alloc(struct pmu *pmu =20 free_dev: put_device(pmu->dev); + pmu->dev =3D NULL; goto out; } =20 @@ -11836,25 +11832,38 @@ static bool idr_cmpxchg(struct idr *idr, return true; } =20 +static void perf_pmu_free(struct pmu *pmu) +{ + free_percpu(pmu->pmu_disable_count); + if (pmu_bus_running && pmu->dev && pmu->dev !=3D PMU_NULL_DEV) { + if (pmu->nr_addr_filters) + device_remove_file(pmu->dev, &dev_attr_nr_addr_filters); + device_del(pmu->dev); + put_device(pmu->dev); + } + free_percpu(pmu->cpu_pmu_context); +} + int perf_pmu_register(struct pmu *pmu, const char *name, int type) { int cpu, ret, max =3D PERF_TYPE_MAX; =20 + pmu->type =3D -1; + mutex_lock(&pmus_lock); ret =3D -ENOMEM; pmu->pmu_disable_count =3D alloc_percpu(int); if (!pmu->pmu_disable_count) goto unlock; =20 - pmu->type =3D -1; if (WARN_ONCE(!name, "Can not register anonymous pmu.\n")) { ret =3D -EINVAL; - goto free_pdc; + goto free; } =20 if (WARN_ONCE(pmu->scope >=3D PERF_PMU_MAX_SCOPE, "Can not register a pmu= with an invalid scope.\n")) { ret =3D -EINVAL; - goto free_pdc; + goto free; } =20 pmu->name =3D name; @@ -11864,24 +11873,23 @@ int perf_pmu_register(struct pmu *pmu, c =20 ret =3D idr_alloc(&pmu_idr, NULL, max, 0, GFP_KERNEL); if (ret < 0) - goto free_pdc; + goto free; =20 WARN_ON(type >=3D 0 && ret !=3D type); =20 - type =3D ret; - pmu->type =3D type; + pmu->type =3D ret; atomic_set(&pmu->exclusive_cnt, 0); =20 if (pmu_bus_running && !pmu->dev) { ret =3D pmu_dev_alloc(pmu); if (ret) - goto free_idr; + goto free; } =20 ret =3D -ENOMEM; pmu->cpu_pmu_context =3D alloc_percpu(struct perf_cpu_pmu_context); if (!pmu->cpu_pmu_context) - goto free_dev; + goto free; =20 for_each_possible_cpu(cpu) { struct perf_cpu_pmu_context *cpc; @@ -11922,8 +11930,10 @@ int perf_pmu_register(struct pmu *pmu, c /* * Now that the PMU is complete, make it visible to perf_try_init_event(). */ - if (!idr_cmpxchg(&pmu_idr, pmu->type, NULL, pmu)) - goto free_context; + if (!idr_cmpxchg(&pmu_idr, pmu->type, NULL, pmu)) { + ret =3D -EINVAL; + goto free; + } list_add_rcu(&pmu->entry, &pmus); =20 ret =3D 0; @@ -11932,20 +11942,10 @@ int perf_pmu_register(struct pmu *pmu, c =20 return ret; =20 -free_context: - free_percpu(pmu->cpu_pmu_context); - -free_dev: - if (pmu->dev && pmu->dev !=3D PMU_NULL_DEV) { - device_del(pmu->dev); - put_device(pmu->dev); - } - -free_idr: - idr_remove(&pmu_idr, pmu->type); - -free_pdc: - free_percpu(pmu->pmu_disable_count); +free: + if (pmu->type >=3D 0) + idr_remove(&pmu_idr, pmu->type); + perf_pmu_free(pmu); goto unlock; } EXPORT_SYMBOL_GPL(perf_pmu_register); @@ -11964,14 +11964,7 @@ void perf_pmu_unregister(struct pmu *pmu synchronize_srcu(&pmus_srcu); synchronize_rcu(); =20 - free_percpu(pmu->pmu_disable_count); - if (pmu_bus_running && pmu->dev && pmu->dev !=3D PMU_NULL_DEV) { - if (pmu->nr_addr_filters) - device_remove_file(pmu->dev, &dev_attr_nr_addr_filters); - device_del(pmu->dev); - put_device(pmu->dev); - } - free_pmu_context(pmu); + perf_pmu_free(pmu); } EXPORT_SYMBOL_GPL(perf_pmu_unregister); From nobody Sun Feb 8 09:32:55 2026 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4B55822B8D9 for ; Wed, 5 Feb 2025 10:25:37 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=90.155.50.34 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738751140; cv=none; b=tChwClG/LCEMVlCLGQFbK2UwBIW2bcEjcQIBRil+ImSLZjnWgLYiZ+TfhaWJGRvXFMHaaRv60117WwAnyxC76os38fmQoLEuOn8RiWX3mzSXzanT5p0vvBdoRB/gwcRwKcKl8PU/oF/CpNFjNT2XmHZ0LwLMjF6Plonne9frcGo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738751140; c=relaxed/simple; bh=fb4yx1mLvfIivb42zEJ6sQ3mn7X1TYb8g7pYQvyHERQ=; h=Message-ID:Date:From:To:Cc:Subject:References:MIME-Version: Content-Type; b=IMdqhZaOkQDYO0DJWFMJMEI4oyO63Godz9gW76P+5zrYWBdwHVbfT+vi7FlVwq3fgWR3I+4IJDdkWXtwv/ML8xv3szHubYEZrxT7nsoBTq6blVGIxZB4tP3eZ6jjjKlI225VGVxWzjlxRGH9Dj/02xr5uP3bYp/+wAoOl1nMsoU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org; spf=none smtp.mailfrom=infradead.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b=BOaNDxB+; arc=none smtp.client-ip=90.155.50.34 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=infradead.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="BOaNDxB+" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=et62BZe26TSNPT9m7zmj3JkW+qnq8ZueXYHt8xkqpL4=; b=BOaNDxB+8qhm1WOH11iyiecxJn fVFA/AfK8L/sNXJUpsGOsUpM1QTERb3e91fXdlp5KgPVQku1O9sAloZrvn43heerci7C9nxx/Ruiz /1CxaRweBbHnR/rS+0tYmJ5BIGKmORcvN57X0b8vfRe5L7V+F66NWt5D3GWsroF0hN0ZfCQsKCR8A hSnMnezrJqzAtDQtTj+fnxuk1EeXXt01EOyAGI2Yq0XoSmTJr7waCBjP4NcqYsNF6h7Wc3Az3VpPc fQUpSvpU7jUqhZRifr2UQ+OmORjbXTYTan0YAyRqBQQIpMszBAGVucUibW4q7/88FSinCmyhzsvMT 4pjwMd8g==; Received: from 77-249-17-89.cable.dynamic.v4.ziggo.nl ([77.249.17.89] helo=noisy.programming.kicks-ass.net) by casper.infradead.org with esmtpsa (Exim 4.98 #2 (Red Hat Linux)) id 1tfcb3-00000004GPa-26uI; Wed, 05 Feb 2025 10:25:29 +0000 Received: by noisy.programming.kicks-ass.net (Postfix, from userid 0) id D6E323061F6; Wed, 5 Feb 2025 11:25:27 +0100 (CET) Message-ID: <20250205102449.451984422@infradead.org> User-Agent: quilt/0.66 Date: Wed, 05 Feb 2025 11:21:31 +0100 From: Peter Zijlstra To: mingo@kernel.org, ravi.bangoria@amd.com, lucas.demarchi@intel.com Cc: linux-kernel@vger.kernel.org, peterz@infradead.org, willy@infradead.org, acme@kernel.org, namhyung@kernel.org, mark.rutland@arm.com, alexander.shishkin@linux.intel.com, jolsa@kernel.org, irogers@google.com, adrian.hunter@intel.com, kan.liang@linux.intel.com Subject: [PATCH v2 11/24] perf: Simplify perf_pmu_register() References: <20250205102120.531585416@infradead.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Using the previously introduced perf_pmu_free() and a new IDR helper, simplify the perf_pmu_register error paths. Signed-off-by: Peter Zijlstra (Intel) Reviewed-by: Ravi Bangoria --- include/linux/idr.h | 17 ++++++++++++ kernel/events/core.c | 71 ++++++++++++++++++++--------------------------= ----- 2 files changed, 46 insertions(+), 42 deletions(-) --- a/include/linux/idr.h +++ b/include/linux/idr.h @@ -15,6 +15,7 @@ #include #include #include +#include =20 struct idr { struct radix_tree_root idr_rt; @@ -124,6 +125,22 @@ void *idr_get_next_ul(struct idr *, unsi void *idr_replace(struct idr *, void *, unsigned long id); void idr_destroy(struct idr *); =20 +struct __class_idr { + struct idr *idr; + int id; +}; + +#define idr_null ((struct __class_idr){ NULL, -1 }) +#define take_idr_id(id) __get_and_null(id, idr_null) + +DEFINE_CLASS(idr_alloc, struct __class_idr, + if (_T.id >=3D 0) idr_remove(_T.idr, _T.id), + ((struct __class_idr){ + .idr =3D idr, + .id =3D idr_alloc(idr, ptr, start, end, gfp), + }), + struct idr *idr, void *ptr, int start, int end, gfp_t gfp); + /** * idr_init_base() - Initialise an IDR. * @idr: IDR handle. --- a/kernel/events/core.c +++ b/kernel/events/core.c @@ -11778,52 +11778,49 @@ static void perf_pmu_free(struct pmu *pm free_percpu(pmu->cpu_pmu_context); } =20 -int perf_pmu_register(struct pmu *pmu, const char *name, int type) +DEFINE_FREE(pmu_unregister, struct pmu *, if (_T) perf_pmu_free(_T)) + +int perf_pmu_register(struct pmu *_pmu, const char *name, int type) { - int cpu, ret, max =3D PERF_TYPE_MAX; + int cpu, max =3D PERF_TYPE_MAX; =20 - pmu->type =3D -1; + struct pmu *pmu __free(pmu_unregister) =3D _pmu; + guard(mutex)(&pmus_lock); =20 - mutex_lock(&pmus_lock); - ret =3D -ENOMEM; pmu->pmu_disable_count =3D alloc_percpu(int); if (!pmu->pmu_disable_count) - goto unlock; + return -ENOMEM; =20 - if (WARN_ONCE(!name, "Can not register anonymous pmu.\n")) { - ret =3D -EINVAL; - goto free; - } + if (WARN_ONCE(!name, "Can not register anonymous pmu.\n")) + return -EINVAL; =20 - if (WARN_ONCE(pmu->scope >=3D PERF_PMU_MAX_SCOPE, "Can not register a pmu= with an invalid scope.\n")) { - ret =3D -EINVAL; - goto free; - } + if (WARN_ONCE(pmu->scope >=3D PERF_PMU_MAX_SCOPE, + "Can not register a pmu with an invalid scope.\n")) + return -EINVAL; =20 pmu->name =3D name; =20 if (type >=3D 0) max =3D type; =20 - ret =3D idr_alloc(&pmu_idr, NULL, max, 0, GFP_KERNEL); - if (ret < 0) - goto free; + CLASS(idr_alloc, pmu_type)(&pmu_idr, NULL, max, 0, GFP_KERNEL); + if (pmu_type.id < 0) + return pmu_type.id; =20 - WARN_ON(type >=3D 0 && ret !=3D type); + WARN_ON(type >=3D 0 && pmu_type.id !=3D type); =20 - pmu->type =3D ret; + pmu->type =3D pmu_type.id; atomic_set(&pmu->exclusive_cnt, 0); =20 if (pmu_bus_running && !pmu->dev) { - ret =3D pmu_dev_alloc(pmu); + int ret =3D pmu_dev_alloc(pmu); if (ret) - goto free; + return ret; } =20 - ret =3D -ENOMEM; pmu->cpu_pmu_context =3D alloc_percpu(struct perf_cpu_pmu_context); if (!pmu->cpu_pmu_context) - goto free; + return -ENOMEM; =20 for_each_possible_cpu(cpu) { struct perf_cpu_pmu_context *cpc; @@ -11864,32 +11861,22 @@ int perf_pmu_register(struct pmu *pmu, c /* * Now that the PMU is complete, make it visible to perf_try_init_event(). */ - if (!idr_cmpxchg(&pmu_idr, pmu->type, NULL, pmu)) { - ret =3D -EINVAL; - goto free; - } + if (!idr_cmpxchg(&pmu_idr, pmu->type, NULL, pmu)) + return -EINVAL; list_add_rcu(&pmu->entry, &pmus); =20 - ret =3D 0; -unlock: - mutex_unlock(&pmus_lock); - - return ret; - -free: - if (pmu->type >=3D 0) - idr_remove(&pmu_idr, pmu->type); - perf_pmu_free(pmu); - goto unlock; + take_idr_id(pmu_type); + _pmu =3D no_free_ptr(pmu); // let it rip + return 0; } EXPORT_SYMBOL_GPL(perf_pmu_register); =20 void perf_pmu_unregister(struct pmu *pmu) { - mutex_lock(&pmus_lock); - list_del_rcu(&pmu->entry); - idr_remove(&pmu_idr, pmu->type); - mutex_unlock(&pmus_lock); + scoped_guard (mutex, &pmus_lock) { + list_del_rcu(&pmu->entry); + idr_remove(&pmu_idr, pmu->type); + } =20 /* * We dereference the pmu list under both SRCU and regular RCU, so From nobody Sun Feb 8 09:32:55 2026 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D747022FDE0 for ; Wed, 5 Feb 2025 10:25:44 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=90.155.50.34 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738751146; cv=none; b=qtbCoPQ5XC9oYZi3DZrNhjh+nSPb8hazZ2eG0Q8gQUPN38N+Dp9Rwigzw1QWSNawdFSx8ivr0clJqb8r0EgiRER9zxl/W32/c4rYQUZag9JDxoXtIBMa8xtPboAANxlcg/wsD1MlXRnc+efCZGvOYBnCcGh/M7GdVdFFif2mx5g= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738751146; c=relaxed/simple; bh=f5dpMh9AhzPjGGmswmBsQrHH1XA3sp2ngO2q5ZzvHoo=; h=Message-ID:Date:From:To:Cc:Subject:References:MIME-Version: Content-Type; b=qQ40Ri2p7Ha46MPC7t38qfu45rEKLWRmrlISQ9A8rbrPelwDADvcEiPZ/ZxkviFzHQ5AULNHIkUe66ZavT39tCpNfHYairHDg/7v2SN6auLfQQ6xOC0+b9MR2/MzLQJ3cw9UsbXTua5gRFp6B9IQaYk2LjzCt7iYSehfsSeloOg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org; spf=none smtp.mailfrom=infradead.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b=BI0PO8sG; arc=none smtp.client-ip=90.155.50.34 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=infradead.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="BI0PO8sG" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=mO36VRNVxK68SLi3Hxx0gi8+RzNHKpmki4ffDW+Vt6g=; b=BI0PO8sG8f/04Bu+03mBn0tg8/ 6qTsRXSFvATfaPeQR2W8SihBK8thion04pKNaikCGNrq9VUVK+nwNNrdnGlqMSoYk2gPwFo1k7AHi 7U8DcslOIWSWOLmjYAKzFRRqbtp0cUKO8ZctA1NGHJN+UY8HF4pqZpnWq5kCy3ctr+i0+vvkQFn5S SYWpzSX1clujYqL6PmZH/CnFJ697CLxlI2Ni730K1+fktFBfBZDLWYVQXY7lF7ArYAuMAm/go/kfd XWsaqdrlFQxT3Za638epspI1ODlqk4190aDD09feJWp6wc04bijJvSVZRfDnTcjoqvg3MHm6mkdwT gTkO5oag==; Received: from 77-249-17-89.cable.dynamic.v4.ziggo.nl ([77.249.17.89] helo=noisy.programming.kicks-ass.net) by casper.infradead.org with esmtpsa (Exim 4.98 #2 (Red Hat Linux)) id 1tfcb3-00000004GPb-27OX; Wed, 05 Feb 2025 10:25:29 +0000 Received: by noisy.programming.kicks-ass.net (Postfix, from userid 0) id DAE713064EF; Wed, 5 Feb 2025 11:25:27 +0100 (CET) Message-ID: <20250205102449.560323038@infradead.org> User-Agent: quilt/0.66 Date: Wed, 05 Feb 2025 11:21:32 +0100 From: Peter Zijlstra To: mingo@kernel.org, ravi.bangoria@amd.com, lucas.demarchi@intel.com Cc: linux-kernel@vger.kernel.org, peterz@infradead.org, willy@infradead.org, acme@kernel.org, namhyung@kernel.org, mark.rutland@arm.com, alexander.shishkin@linux.intel.com, jolsa@kernel.org, irogers@google.com, adrian.hunter@intel.com, kan.liang@linux.intel.com Subject: [PATCH v2 12/24] perf: Simplify perf_init_event() References: <20250205102120.531585416@infradead.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Signed-off-by: Peter Zijlstra (Intel) Reviewed-by: Ravi Bangoria --- kernel/events/core.c | 31 ++++++++++++------------------- 1 file changed, 12 insertions(+), 19 deletions(-) --- a/kernel/events/core.c +++ b/kernel/events/core.c @@ -11942,10 +11942,10 @@ static int perf_try_init_event(struct pm static struct pmu *perf_init_event(struct perf_event *event) { bool extended_type =3D false; - int idx, type, ret; struct pmu *pmu; + int type, ret; =20 - idx =3D srcu_read_lock(&pmus_srcu); + guard(srcu)(&pmus_srcu); =20 /* * Save original type before calling pmu->event_init() since certain @@ -11958,7 +11958,7 @@ static struct pmu *perf_init_event(struc pmu =3D event->parent->pmu; ret =3D perf_try_init_event(pmu, event); if (!ret) - goto unlock; + return pmu; } =20 /* @@ -11977,13 +11977,12 @@ static struct pmu *perf_init_event(struc } =20 again: - rcu_read_lock(); - pmu =3D idr_find(&pmu_idr, type); - rcu_read_unlock(); + scoped_guard (rcu) + pmu =3D idr_find(&pmu_idr, type); if (pmu) { if (event->attr.type !=3D type && type !=3D PERF_TYPE_RAW && !(pmu->capabilities & PERF_PMU_CAP_EXTENDED_HW_TYPE)) - goto fail; + return ERR_PTR(-ENOENT); =20 ret =3D perf_try_init_event(pmu, event); if (ret =3D=3D -ENOENT && event->attr.type !=3D type && !extended_type) { @@ -11992,27 +11991,21 @@ static struct pmu *perf_init_event(struc } =20 if (ret) - pmu =3D ERR_PTR(ret); + return ERR_PTR(ret); =20 - goto unlock; + return pmu; } =20 list_for_each_entry_rcu(pmu, &pmus, entry, lockdep_is_held(&pmus_srcu)) { ret =3D perf_try_init_event(pmu, event); if (!ret) - goto unlock; + return pmu; =20 - if (ret !=3D -ENOENT) { - pmu =3D ERR_PTR(ret); - goto unlock; - } + if (ret !=3D -ENOENT) + return ERR_PTR(ret); } -fail: - pmu =3D ERR_PTR(-ENOENT); -unlock: - srcu_read_unlock(&pmus_srcu, idx); =20 - return pmu; + return ERR_PTR(-ENOENT); } =20 static void attach_sb_event(struct perf_event *event) From nobody Sun Feb 8 09:32:55 2026 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id BF1C422C35E for ; Wed, 5 Feb 2025 10:25:38 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=90.155.50.34 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738751140; cv=none; b=Zdkjsqu7pyMaRodbsbZ2tTV62BpLIyyINZV+OOi8QYhBFql0sQ4gAJS/EXKraQWV/Bdg1W5HE2lPzN2jFNjIROFxAYV3zcq9bGTskuXQYJT4YUm0pRAXKoEFQ35vp4fEkBlO1AgWKhXxhODSeTjei0o3W/OdPrG5a5708ZOt3vo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738751140; c=relaxed/simple; bh=VyZRdBAMs530KrDX/IvA+7ZtQx5J6svRBc1QlxxrPn4=; h=Message-ID:Date:From:To:Cc:Subject:References:MIME-Version: Content-Type; b=uIk5KgtJ8JgN4E9X4Kl+FBcYMRP9eWu8iaCM+rTEBf2n/Ue93I8Q2RMS/pszlsUCGPcKveEH+zy0D81VgQJ3NFowBuEWyrP//BnC3LDpdLyU9k+xqENBw6jZs8AoaxOAkFIDQW0utg2L6TmRO+aqEvjQgGmMMjRK2qhoGjpfppI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org; spf=none smtp.mailfrom=infradead.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b=UlswYVN8; arc=none smtp.client-ip=90.155.50.34 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=infradead.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="UlswYVN8" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=eu5ZvhRBqGbSk6AaLgfzXy2QPVH2kRteBU9D7EL3a1k=; b=UlswYVN8AYGp3Qft4WUC8TZ31z Q92UHUTApwYK7JqqUtGo265pLOuA5CYDJWcEKFH85ENAHSetAQUujLokh77gWL9ymz6wkacL4CvFG SvKTPHx1Axfbg17rfKfeiAwUyfWgA6ghdtnogGhM8tRP5V8+Wqd7XCkvPdq5NhFvb6u+9kpSooWSw 5BZYHN58RNuNFMUcuH4Vt/t5hi29bCA0txiA9fHrxx21YBp2TwXWd9G4GWEgj/3UGcs66Wm2vnD3m P+fn3oSNjLUhP8/9y/0TLKeMBPX+ptPLFRnHYoivcBcRlmlqjmjWl09C3OWQAQ2mJUuhBYBl5aK4+ wOvxis/g==; Received: from 77-249-17-89.cable.dynamic.v4.ziggo.nl ([77.249.17.89] helo=noisy.programming.kicks-ass.net) by casper.infradead.org with esmtpsa (Exim 4.98 #2 (Red Hat Linux)) id 1tfcb3-00000004GPe-2FDa; Wed, 05 Feb 2025 10:25:30 +0000 Received: by noisy.programming.kicks-ass.net (Postfix, from userid 0) id DEA513078A4; Wed, 5 Feb 2025 11:25:27 +0100 (CET) Message-ID: <20250205102449.682083733@infradead.org> User-Agent: quilt/0.66 Date: Wed, 05 Feb 2025 11:21:33 +0100 From: Peter Zijlstra To: mingo@kernel.org, ravi.bangoria@amd.com, lucas.demarchi@intel.com Cc: linux-kernel@vger.kernel.org, peterz@infradead.org, willy@infradead.org, acme@kernel.org, namhyung@kernel.org, mark.rutland@arm.com, alexander.shishkin@linux.intel.com, jolsa@kernel.org, irogers@google.com, adrian.hunter@intel.com, kan.liang@linux.intel.com Subject: [PATCH v2 13/24] perf: Simplify perf_event_alloc() References: <20250205102120.531585416@infradead.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Using the previous simplifications, transition perf_event_alloc() to the cleanup way of things -- reducing error path magic. Signed-off-by: Peter Zijlstra (Intel) Reviewed-by: Ravi Bangoria --- kernel/events/core.c | 60 +++++++++++++++++++---------------------------= ----- 1 file changed, 23 insertions(+), 37 deletions(-) --- a/kernel/events/core.c +++ b/kernel/events/core.c @@ -5360,6 +5361,8 @@ static void __free_event(struct perf_eve call_rcu(&event->rcu_head, free_event_rcu); } =20 +DEFINE_FREE(__free_event, struct perf_event *, if (_T) __free_event(_T)) + /* vs perf_event_alloc() success */ static void _free_event(struct perf_event *event) { @@ -12238,7 +12241,6 @@ perf_event_alloc(struct perf_event_attr void *context, int cgroup_fd) { struct pmu *pmu; - struct perf_event *event; struct hw_perf_event *hwc; long err =3D -EINVAL; int node; @@ -12253,8 +12255,8 @@ perf_event_alloc(struct perf_event_attr } =20 node =3D (cpu >=3D 0) ? cpu_to_node(cpu) : -1; - event =3D kmem_cache_alloc_node(perf_event_cache, GFP_KERNEL | __GFP_ZERO, - node); + struct perf_event *event __free(__free_event) =3D + kmem_cache_alloc_node(perf_event_cache, GFP_KERNEL | __GFP_ZERO, node); if (!event) return ERR_PTR(-ENOMEM); =20 @@ -12361,65 +12363,53 @@ perf_event_alloc(struct perf_event_attr * See perf_output_read(). */ if (has_inherit_and_sample_read(attr) && !(attr->sample_type & PERF_SAMPL= E_TID)) - goto err; + return ERR_PTR(-EINVAL); =20 if (!has_branch_stack(event)) event->attr.branch_sample_type =3D 0; =20 pmu =3D perf_init_event(event); - if (IS_ERR(pmu)) { - err =3D PTR_ERR(pmu); - goto err; - } + if (IS_ERR(pmu)) + return (void*)pmu; =20 /* * Disallow uncore-task events. Similarly, disallow uncore-cgroup * events (they don't make sense as the cgroup will be different * on other CPUs in the uncore mask). */ - if (pmu->task_ctx_nr =3D=3D perf_invalid_context && (task || cgroup_fd != =3D -1)) { - err =3D -EINVAL; - goto err; - } + if (pmu->task_ctx_nr =3D=3D perf_invalid_context && (task || cgroup_fd != =3D -1)) + return ERR_PTR(-EINVAL); =20 if (event->attr.aux_output && (!(pmu->capabilities & PERF_PMU_CAP_AUX_OUTPUT) || - event->attr.aux_pause || event->attr.aux_resume)) { - err =3D -EOPNOTSUPP; - goto err; - } + event->attr.aux_pause || event->attr.aux_resume)) + return ERR_PTR(-EOPNOTSUPP); =20 - if (event->attr.aux_pause && event->attr.aux_resume) { - err =3D -EINVAL; - goto err; - } + if (event->attr.aux_pause && event->attr.aux_resume) + return ERR_PTR(-EINVAL); =20 if (event->attr.aux_start_paused) { - if (!(pmu->capabilities & PERF_PMU_CAP_AUX_PAUSE)) { - err =3D -EOPNOTSUPP; - goto err; - } + if (!(pmu->capabilities & PERF_PMU_CAP_AUX_PAUSE)) + return ERR_PTR(-EOPNOTSUPP); event->hw.aux_paused =3D 1; } =20 if (cgroup_fd !=3D -1) { err =3D perf_cgroup_connect(cgroup_fd, event, attr, group_leader); if (err) - goto err; + return ERR_PTR(err); } =20 err =3D exclusive_event_init(event); if (err) - goto err; + return ERR_PTR(err); =20 if (has_addr_filter(event)) { event->addr_filter_ranges =3D kcalloc(pmu->nr_addr_filters, sizeof(struct perf_addr_filter_range), GFP_KERNEL); - if (!event->addr_filter_ranges) { - err =3D -ENOMEM; - goto err; - } + if (!event->addr_filter_ranges) + return ERR_PTR(-ENOMEM); =20 /* * Clone the parent's vma offsets: they are valid until exec() @@ -12443,23 +12433,19 @@ perf_event_alloc(struct perf_event_attr if (event->attr.sample_type & PERF_SAMPLE_CALLCHAIN) { err =3D get_callchain_buffers(attr->sample_max_stack); if (err) - goto err; + return ERR_PTR(err); event->attach_state |=3D PERF_ATTACH_CALLCHAIN; } } =20 err =3D security_perf_event_alloc(event); if (err) - goto err; + return ERR_PTR(err); =20 /* symmetric to unaccount_event() in _free_event() */ account_event(event); =20 - return event; - -err: - __free_event(event); - return ERR_PTR(err); + return_ptr(event); } =20 static int perf_copy_attr(struct perf_event_attr __user *uattr, From nobody Sun Feb 8 09:32:55 2026 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id BE68223098F for ; Wed, 5 Feb 2025 10:25:47 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=90.155.50.34 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738751149; cv=none; b=ugOiGp9yDrdbDrBrZrgAlpOVJm35hYXeC6b4rQyCXMgHNSR9yWJVYXuLqPCk6jPhtl5Kntbpchby1M1yNMjAUSpLCPdMLPgH4AIDOxvVR8meXuO2GsQttfmudgbAdCNHQeEKPwUSgrSLRGqkmIKSCYxzWZzHU/wxFBk9BQPuYXQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738751149; c=relaxed/simple; bh=41Y9iiBFIWa+xKSz6JdAshpsBvIL4g8JIQnoVc7pb6g=; h=Message-ID:Date:From:To:Cc:Subject:References:MIME-Version: Content-Type; b=j433LXajJX0PEUQmKLr6Nux7OsKJC4CecTMarCpfv335AJNupe/n+Pmm7N5ceHB2iELKPf6U4d3iW/Pyb8DUg0wXLgIL3fuTmqm0sJJiPzM3CD9U6lFn2aebT1nYWU0f/ifPSjSsMkG4wOSD4/A3ILnnCZa1YtW5BtmJOOD3fj4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org; spf=none smtp.mailfrom=infradead.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b=hLIOl2vz; arc=none smtp.client-ip=90.155.50.34 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=infradead.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="hLIOl2vz" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=Bm6lxT+iF6YGPyZbFX7MMSqDd/wgHbIFbh5l5hoRNJg=; b=hLIOl2vzAkNabmGYA4aJFmCT6p bIePL8i7YjExx2HO3sqnJoSFz78EVSE7Z6QF2zMGrTAL8mKOPQgg2KWyZbZ0IB7Igh0TrVRVS4p5/ SnC/3Eh6qJsUYldipD/F0/R90G2UKrhXAaFFtnrfGuFdl/KUynUczKBkTPZ2uVd0ctnPdQddj2vcr VCanr5NKRywGkMxOD2Q78L8MDPkcyJ+OWIzBUZ92x9v3H1InHejx4WeZkXOax2mLubrxBIeydYtY3 lnmYNKGe/StmqmHenqipHEfAZM63WmFIKH+Tdld6oFhj0axKl3D3apYE5c40NQEanaK1XGdehSrCO yoZ/X2Lw==; Received: from 77-249-17-89.cable.dynamic.v4.ziggo.nl ([77.249.17.89] helo=noisy.programming.kicks-ass.net) by casper.infradead.org with esmtpsa (Exim 4.98 #2 (Red Hat Linux)) id 1tfcb3-00000004GPf-2HJO; Wed, 05 Feb 2025 10:25:30 +0000 Received: by noisy.programming.kicks-ass.net (Postfix, from userid 0) id E2228307FA8; Wed, 5 Feb 2025 11:25:27 +0100 (CET) Message-ID: <20250205102449.797431411@infradead.org> User-Agent: quilt/0.66 Date: Wed, 05 Feb 2025 11:21:34 +0100 From: Peter Zijlstra To: mingo@kernel.org, ravi.bangoria@amd.com, lucas.demarchi@intel.com Cc: linux-kernel@vger.kernel.org, peterz@infradead.org, willy@infradead.org, acme@kernel.org, namhyung@kernel.org, mark.rutland@arm.com, alexander.shishkin@linux.intel.com, jolsa@kernel.org, irogers@google.com, adrian.hunter@intel.com, kan.liang@linux.intel.com Subject: [PATCH v2 14/24] perf: Merge pmu_disable_count into cpu_pmu_context References: <20250205102120.531585416@infradead.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Because it makes no sense to have two per-cpu allocations per pmu. Signed-off-by: Peter Zijlstra (Intel) Reviewed-by: Ravi Bangoria --- include/linux/perf_event.h | 2 +- kernel/events/core.c | 12 ++++-------- 2 files changed, 5 insertions(+), 9 deletions(-) --- a/include/linux/perf_event.h +++ b/include/linux/perf_event.h @@ -343,7 +343,6 @@ struct pmu { */ unsigned int scope; =20 - int __percpu *pmu_disable_count; struct perf_cpu_pmu_context __percpu *cpu_pmu_context; atomic_t exclusive_cnt; /* < 0: cpu; > 0: tsk */ int task_ctx_nr; @@ -1031,6 +1030,7 @@ struct perf_cpu_pmu_context { =20 int active_oncpu; int exclusive; + int pmu_disable_count; =20 raw_spinlock_t hrtimer_lock; struct hrtimer hrtimer; --- a/kernel/events/core.c +++ b/kernel/events/core.c @@ -1174,21 +1174,22 @@ static int perf_mux_hrtimer_restart_ipi( =20 void perf_pmu_disable(struct pmu *pmu) { - int *count =3D this_cpu_ptr(pmu->pmu_disable_count); + int *count =3D &this_cpu_ptr(pmu->cpu_pmu_context)->pmu_disable_count; if (!(*count)++) pmu->pmu_disable(pmu); } =20 void perf_pmu_enable(struct pmu *pmu) { - int *count =3D this_cpu_ptr(pmu->pmu_disable_count); + int *count =3D &this_cpu_ptr(pmu->cpu_pmu_context)->pmu_disable_count; if (!--(*count)) pmu->pmu_enable(pmu); } =20 static void perf_assert_pmu_disabled(struct pmu *pmu) { - WARN_ON_ONCE(*this_cpu_ptr(pmu->pmu_disable_count) =3D=3D 0); + int *count =3D &this_cpu_ptr(pmu->cpu_pmu_context)->pmu_disable_count; + WARN_ON_ONCE(*count =3D=3D 0); } =20 static inline void perf_pmu_read(struct perf_event *event) @@ -11860,7 +11861,6 @@ static bool idr_cmpxchg(struct idr *idr, =20 static void perf_pmu_free(struct pmu *pmu) { - free_percpu(pmu->pmu_disable_count); if (pmu_bus_running && pmu->dev && pmu->dev !=3D PMU_NULL_DEV) { if (pmu->nr_addr_filters) device_remove_file(pmu->dev, &dev_attr_nr_addr_filters); @@ -11879,10 +11879,6 @@ int perf_pmu_register(struct pmu *_pmu, struct pmu *pmu __free(pmu_unregister) =3D _pmu; guard(mutex)(&pmus_lock); =20 - pmu->pmu_disable_count =3D alloc_percpu(int); - if (!pmu->pmu_disable_count) - return -ENOMEM; - if (WARN_ONCE(!name, "Can not register anonymous pmu.\n")) return -EINVAL; From nobody Sun Feb 8 09:32:55 2026 Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A264E22B59D for ; Wed, 5 Feb 2025 10:25:36 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=90.155.92.199 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738751138; cv=none; b=gp0OyyQpNviVFR7noUN4s+3IBmuW4qGbm6vVngY8a8oRZ5un6zZYkOtho6ZPRBiLXlmL8W/QhT2tdtXO3LC3UepuaFqW5d3tIomi66Jnc9OZaYSnz0dnbRJ6N9io9wdX1bfwto8Jc5qeW0StLAgU13jViZetmSAznvQH8uTANZQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738751138; c=relaxed/simple; bh=2zscDwLR0ZjhieKGNemGCr5W89z17A7bETmSG8j0smg=; h=Message-ID:Date:From:To:Cc:Subject:References:MIME-Version: Content-Type; b=exSneeW7erMe1diWc+xzMogkTEV0apEN7qsUqykyZS75U0AJZA9xsVJQdgOkc4aaaJWhVhXqRVR3Spy9f5gWEQBDQPP7hoEZAHG+nkLQ0nfqX3Vl/OEOu318vsiXBnmPtnQN6Suq/kmHZP/beljl2QI+Q2cFeAPPcGzep5Y50+E= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org; spf=none smtp.mailfrom=infradead.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b=IpcY51AP; arc=none smtp.client-ip=90.155.92.199 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=infradead.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="IpcY51AP" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=UnuX1XoQTtG9dRc73ChG71Cgdr+YdzLA5XtqNeptKjc=; b=IpcY51APqo79GkJhgk/sAgb8mD bCti6WPHKIX8iZm9XQc04e2Rvp+Z45OAlHvfgiF84LoxfbONC9WTnBIJbXPmEdFLPsQSsbYas+/oo 8D0yM7TNIKnOX+gWElxJFqPHEw/i4VOMflFQxB7BSAVfWfA4Rt9Gv+wfeU1Y3uvtX0Ems1V7AIOmg 8O1ecEgUZ8C/r7hAOtWrsVrizoBX60Wr1+jghi8X5C1PSIUITowvfSP42ELFEC0HzlDJBK2xL5wif veTLI2qc6/9CBraZbPj83Xmv8flKuxi/GDiLVOBgQy+6wGXFBO2Zcq+DrtfO9RgPmbD0pqNl9+Jsr 7THMfBdQ==; Received: from 77-249-17-89.cable.dynamic.v4.ziggo.nl ([77.249.17.89] helo=noisy.programming.kicks-ass.net) by desiato.infradead.org with esmtpsa (Exim 4.98 #2 (Red Hat Linux)) id 1tfcb3-0000000GbmF-347e; Wed, 05 Feb 2025 10:25:30 +0000 Received: by noisy.programming.kicks-ass.net (Postfix, from userid 0) id E5F61307FAE; Wed, 5 Feb 2025 11:25:27 +0100 (CET) Message-ID: <20250205102449.906046341@infradead.org> User-Agent: quilt/0.66 Date: Wed, 05 Feb 2025 11:21:35 +0100 From: Peter Zijlstra To: mingo@kernel.org, ravi.bangoria@amd.com, lucas.demarchi@intel.com Cc: linux-kernel@vger.kernel.org, peterz@infradead.org, willy@infradead.org, acme@kernel.org, namhyung@kernel.org, mark.rutland@arm.com, alexander.shishkin@linux.intel.com, jolsa@kernel.org, irogers@google.com, adrian.hunter@intel.com, kan.liang@linux.intel.com Subject: [PATCH v2 15/24] perf: Add this_cpc() helper References: <20250205102120.531585416@infradead.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" As a preparation for adding yet another indirection. Signed-off-by: Peter Zijlstra (Intel) Reviewed-by: Ravi Bangoria --- kernel/events/core.c | 34 ++++++++++++++++++---------------- 1 file changed, 18 insertions(+), 16 deletions(-) --- a/kernel/events/core.c +++ b/kernel/events/core.c @@ -1176,23 +1176,28 @@ static int perf_mux_hrtimer_restart_ipi( return perf_mux_hrtimer_restart(arg); } =20 +static __always_inline struct perf_cpu_pmu_context *this_cpc(struct pmu *p= mu) +{ + return this_cpu_ptr(pmu->cpu_pmu_context); +} + void perf_pmu_disable(struct pmu *pmu) { - int *count =3D &this_cpu_ptr(pmu->cpu_pmu_context)->pmu_disable_count; + int *count =3D &this_cpc(pmu)->pmu_disable_count; if (!(*count)++) pmu->pmu_disable(pmu); } =20 void perf_pmu_enable(struct pmu *pmu) { - int *count =3D &this_cpu_ptr(pmu->cpu_pmu_context)->pmu_disable_count; + int *count =3D &this_cpc(pmu)->pmu_disable_count; if (!--(*count)) pmu->pmu_enable(pmu); } =20 static void perf_assert_pmu_disabled(struct pmu *pmu) { - int *count =3D &this_cpu_ptr(pmu->cpu_pmu_context)->pmu_disable_count; + int *count =3D &this_cpc(pmu)->pmu_disable_count; WARN_ON_ONCE(*count =3D=3D 0); } =20 @@ -2304,7 +2309,7 @@ static void event_sched_out(struct perf_event *event, struct perf_event_context *ctx) { struct perf_event_pmu_context *epc =3D event->pmu_ctx; - struct perf_cpu_pmu_context *cpc =3D this_cpu_ptr(epc->pmu->cpu_pmu_conte= xt); + struct perf_cpu_pmu_context *cpc =3D this_cpc(epc->pmu); enum perf_event_state state =3D PERF_EVENT_STATE_INACTIVE; =20 // XXX cpc serialization, probably per-cpu IRQ disabled @@ -2445,9 +2450,8 @@ __perf_remove_from_context(struct perf_e pmu_ctx->rotate_necessary =3D 0; =20 if (ctx->task && ctx->is_active) { - struct perf_cpu_pmu_context *cpc; + struct perf_cpu_pmu_context *cpc =3D this_cpc(pmu_ctx->pmu); =20 - cpc =3D this_cpu_ptr(pmu_ctx->pmu->cpu_pmu_context); WARN_ON_ONCE(cpc->task_epc && cpc->task_epc !=3D pmu_ctx); cpc->task_epc =3D NULL; } @@ -2585,7 +2589,7 @@ static int event_sched_in(struct perf_event *event, struct perf_event_context *ctx) { struct perf_event_pmu_context *epc =3D event->pmu_ctx; - struct perf_cpu_pmu_context *cpc =3D this_cpu_ptr(epc->pmu->cpu_pmu_conte= xt); + struct perf_cpu_pmu_context *cpc =3D this_cpc(epc->pmu); int ret =3D 0; =20 WARN_ON_ONCE(event->ctx !=3D ctx); @@ -2692,7 +2696,7 @@ group_sched_in(struct perf_event *group_ static int group_can_go_on(struct perf_event *event, int can_add_hw) { struct perf_event_pmu_context *epc =3D event->pmu_ctx; - struct perf_cpu_pmu_context *cpc =3D this_cpu_ptr(epc->pmu->cpu_pmu_conte= xt); + struct perf_cpu_pmu_context *cpc =3D this_cpc(epc->pmu); =20 /* * Groups consisting entirely of software events can always go on. @@ -3315,9 +3319,8 @@ static void __pmu_ctx_sched_out(struct p struct pmu *pmu =3D pmu_ctx->pmu; =20 if (ctx->task && !(ctx->is_active & EVENT_ALL)) { - struct perf_cpu_pmu_context *cpc; + struct perf_cpu_pmu_context *cpc =3D this_cpc(pmu); =20 - cpc =3D this_cpu_ptr(pmu->cpu_pmu_context); WARN_ON_ONCE(cpc->task_epc && cpc->task_epc !=3D pmu_ctx); cpc->task_epc =3D NULL; } @@ -3565,7 +3568,7 @@ static void perf_ctx_sched_task_cb(struc struct perf_cpu_pmu_context *cpc; =20 list_for_each_entry(pmu_ctx, &ctx->pmu_ctx_list, pmu_ctx_entry) { - cpc =3D this_cpu_ptr(pmu_ctx->pmu->cpu_pmu_context); + cpc =3D this_cpc(pmu_ctx->pmu); =20 if (cpc->sched_cb_usage && pmu_ctx->pmu->sched_task) pmu_ctx->pmu->sched_task(pmu_ctx, sched_in); @@ -3674,7 +3677,7 @@ static DEFINE_PER_CPU(int, perf_sched_cb =20 void perf_sched_cb_dec(struct pmu *pmu) { - struct perf_cpu_pmu_context *cpc =3D this_cpu_ptr(pmu->cpu_pmu_context); + struct perf_cpu_pmu_context *cpc =3D this_cpc(pmu); =20 this_cpu_dec(perf_sched_cb_usages); barrier(); @@ -3686,7 +3689,7 @@ void perf_sched_cb_dec(struct pmu *pmu) =20 void perf_sched_cb_inc(struct pmu *pmu) { - struct perf_cpu_pmu_context *cpc =3D this_cpu_ptr(pmu->cpu_pmu_context); + struct perf_cpu_pmu_context *cpc =3D this_cpc(pmu); =20 if (!cpc->sched_cb_usage++) list_add(&cpc->sched_cb_entry, this_cpu_ptr(&sched_cb_list)); @@ -3810,7 +3813,7 @@ static void __link_epc(struct perf_event if (!pmu_ctx->ctx->task) return; =20 - cpc =3D this_cpu_ptr(pmu_ctx->pmu->cpu_pmu_context); + cpc =3D this_cpc(pmu_ctx->pmu); WARN_ON_ONCE(cpc->task_epc && cpc->task_epc !=3D pmu_ctx); cpc->task_epc =3D pmu_ctx; } @@ -3939,10 +3942,9 @@ static int merge_sched_in(struct perf_ev perf_cgroup_event_disable(event, ctx); perf_event_set_state(event, PERF_EVENT_STATE_ERROR); } else { - struct perf_cpu_pmu_context *cpc; + struct perf_cpu_pmu_context *cpc =3D this_cpc(event->pmu_ctx->pmu); =20 event->pmu_ctx->rotate_necessary =3D 1; - cpc =3D this_cpu_ptr(event->pmu_ctx->pmu->cpu_pmu_context); perf_mux_hrtimer_restart(cpc); group_update_userpage(event); } From nobody Sun Feb 8 09:32:55 2026 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8CBE822ACEF for ; Wed, 5 Feb 2025 10:25:34 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=90.155.50.34 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738751136; cv=none; b=Hl7nH/bmXgFGHN4TlslwaOoZ7FwAUTW2hfCKm0EhLXyxbBTLM/wiBYP2Qdkh4zHEaHKYpRTXgeLJZMpxrRX7z4/Yi0VYO6e1nJZ4i1A6ruOsHbxCsAG+WPZ+MKklIuvYX6e9gBgd4wYbw2dOCUXgJBfonH2NVWKV01Zj6uU85SQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738751136; c=relaxed/simple; bh=1kQdT7RNP30VSQRhVbLbDXk7u0BCc/CqkwU4buF82HI=; h=Message-ID:Date:From:To:Cc:Subject:References:MIME-Version: Content-Type; b=GyldpZkuTUbua3h3g0RMq4FfgML0vC1YmJkOKIMX/xBwbvLbvSN5McMD/6J6vkkNLbqFyYuo+zjbjEkvnSZsq09htonHh1ZBIEfc0OCoeFg9Q01lIjF4RJ3hRE0/3EUTMtgRo88/uUYrtx+IHY2MyscieBATW/KrKf4PV4CAdY8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org; spf=none smtp.mailfrom=infradead.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b=UCnKCpJa; arc=none smtp.client-ip=90.155.50.34 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=infradead.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="UCnKCpJa" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=Wwd6VbMcmGqdP94/vtno7cnLFnb1TlambeNcu37V/zM=; b=UCnKCpJaplDgiCBcHb+48fhhPo wkNRN/6jjzxBknsakXe4zcsBPed5tZ5sccNToZEXMSBQXmtAaiynpNK8REJeNPNF2cnUqcSZ3HExz cMz1PmG980uUwgrVQ/2dkbVZvGIQVo5rhoTTVDfpDYM+c4gl9AwGTjtfUkyoj7IaG3ICDTPsHtUsE d7A8RodJV+WjKTDBgGf9kBcGY7q4tExci8xTLT1O33yoDUzugdUVAmIaRYQ9qf9lUKiLhdZ0sQWmn YjIaJL0K4ANWf/QfWBpi7seFf5wKXEA40GNEgQ7uxm0AKHqmzpEK2edioVZ6L1xUhAD/cGsLAVxpR fNgan5uA==; Received: from 77-249-17-89.cable.dynamic.v4.ziggo.nl ([77.249.17.89] helo=noisy.programming.kicks-ass.net) by casper.infradead.org with esmtpsa (Exim 4.98 #2 (Red Hat Linux)) id 1tfcb3-00000004GPj-35EO; Wed, 05 Feb 2025 10:25:30 +0000 Received: by noisy.programming.kicks-ass.net (Postfix, from userid 0) id E9436308076; Wed, 5 Feb 2025 11:25:27 +0100 (CET) Message-ID: <20250205102450.016081954@infradead.org> User-Agent: quilt/0.66 Date: Wed, 05 Feb 2025 11:21:36 +0100 From: Peter Zijlstra To: mingo@kernel.org, ravi.bangoria@amd.com, lucas.demarchi@intel.com Cc: linux-kernel@vger.kernel.org, peterz@infradead.org, willy@infradead.org, acme@kernel.org, namhyung@kernel.org, mark.rutland@arm.com, alexander.shishkin@linux.intel.com, jolsa@kernel.org, irogers@google.com, adrian.hunter@intel.com, kan.liang@linux.intel.com Subject: [PATCH v2 16/24] perf: Detach perf_cpu_pmu_context and pmu lifetimes References: <20250205102120.531585416@infradead.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" In prepration for being able to unregister a pmu with existing events, it becomes important to detach struct perf_cpu_pmu_context lifetimes from that of struct pmu. Notably perf_cpu_pmu_context embeds a perf_event_pmu_context that can stay referenced until the last event goes. Signed-off-by: Peter Zijlstra (Intel) Reviewed-by: Ravi Bangoria --- include/linux/perf_event.h | 4 +-- kernel/events/core.c | 56 +++++++++++++++++++++++++++++++++++++---= ----- 2 files changed, 49 insertions(+), 11 deletions(-) --- a/include/linux/perf_event.h +++ b/include/linux/perf_event.h @@ -336,7 +336,7 @@ struct pmu { */ unsigned int scope; =20 - struct perf_cpu_pmu_context __percpu *cpu_pmu_context; + struct perf_cpu_pmu_context __percpu **cpu_pmu_context; atomic_t exclusive_cnt; /* < 0: cpu; > 0: tsk */ int task_ctx_nr; int hrtimer_interval_ms; @@ -901,7 +901,7 @@ struct perf_event_pmu_context { struct list_head pinned_active; struct list_head flexible_active; =20 - /* Used to avoid freeing per-cpu perf_event_pmu_context */ + /* Used to identify the per-cpu perf_event_pmu_context */ unsigned int embedded : 1; =20 unsigned int nr_events; --- a/kernel/events/core.c +++ b/kernel/events/core.c @@ -1178,7 +1178,7 @@ static int perf_mux_hrtimer_restart_ipi( =20 static __always_inline struct perf_cpu_pmu_context *this_cpc(struct pmu *p= mu) { - return this_cpu_ptr(pmu->cpu_pmu_context); + return *this_cpu_ptr(pmu->cpu_pmu_context); } =20 void perf_pmu_disable(struct pmu *pmu) @@ -4971,11 +4971,14 @@ find_get_pmu_context(struct pmu *pmu, st */ struct perf_cpu_pmu_context *cpc; =20 - cpc =3D per_cpu_ptr(pmu->cpu_pmu_context, event->cpu); + cpc =3D *per_cpu_ptr(pmu->cpu_pmu_context, event->cpu); epc =3D &cpc->epc; raw_spin_lock_irq(&ctx->lock); if (!epc->ctx) { - atomic_set(&epc->refcount, 1); + /* + * One extra reference for the pmu; see perf_pmu_free(). + */ + atomic_set(&epc->refcount, 2); epc->embedded =3D 1; list_add(&epc->pmu_ctx_entry, &ctx->pmu_ctx_list); epc->ctx =3D ctx; @@ -5044,6 +5047,15 @@ static void get_pmu_ctx(struct perf_even WARN_ON_ONCE(!atomic_inc_not_zero(&epc->refcount)); } =20 +static void free_cpc_rcu(struct rcu_head *head) +{ + struct perf_cpu_pmu_context *cpc =3D + container_of(head, typeof(*cpc), epc.rcu_head); + + kfree(cpc->epc.task_ctx_data); + kfree(cpc); +} + static void free_epc_rcu(struct rcu_head *head) { struct perf_event_pmu_context *epc =3D container_of(head, typeof(*epc), r= cu_head); @@ -5078,8 +5090,10 @@ static void put_pmu_ctx(struct perf_even =20 raw_spin_unlock_irqrestore(&ctx->lock, flags); =20 - if (epc->embedded) + if (epc->embedded) { + call_rcu(&epc->rcu_head, free_cpc_rcu); return; + } =20 call_rcu(&epc->rcu_head, free_epc_rcu); } @@ -11595,7 +11609,7 @@ perf_event_mux_interval_ms_store(struct cpus_read_lock(); for_each_online_cpu(cpu) { struct perf_cpu_pmu_context *cpc; - cpc =3D per_cpu_ptr(pmu->cpu_pmu_context, cpu); + cpc =3D *per_cpu_ptr(pmu->cpu_pmu_context, cpu); cpc->hrtimer_interval =3D ns_to_ktime(NSEC_PER_MSEC * timer); =20 cpu_function_call(cpu, perf_mux_hrtimer_restart_ipi, cpc); @@ -11767,7 +11781,25 @@ static void perf_pmu_free(struct pmu *pm device_del(pmu->dev); put_device(pmu->dev); } - free_percpu(pmu->cpu_pmu_context); + + if (pmu->cpu_pmu_context) { + int cpu; + + for_each_possible_cpu(cpu) { + struct perf_cpu_pmu_context *cpc; + + cpc =3D *per_cpu_ptr(pmu->cpu_pmu_context, cpu); + if (!cpc) + continue; + if (cpc->epc.embedded) { + /* refcount managed */ + put_pmu_ctx(&cpc->epc); + continue; + } + kfree(cpc); + } + free_percpu(pmu->cpu_pmu_context); + } } =20 DEFINE_FREE(pmu_unregister, struct pmu *, if (_T) perf_pmu_free(_T)) @@ -11806,14 +11838,20 @@ int perf_pmu_register(struct pmu *_pmu, return ret; } =20 - pmu->cpu_pmu_context =3D alloc_percpu(struct perf_cpu_pmu_context); + pmu->cpu_pmu_context =3D alloc_percpu(struct perf_cpu_pmu_context *); if (!pmu->cpu_pmu_context) return -ENOMEM; =20 for_each_possible_cpu(cpu) { - struct perf_cpu_pmu_context *cpc; + struct perf_cpu_pmu_context *cpc =3D + kmalloc_node(sizeof(struct perf_cpu_pmu_context), + GFP_KERNEL | __GFP_ZERO, + cpu_to_node(cpu)); + + if (!cpc) + return -ENOMEM; =20 - cpc =3D per_cpu_ptr(pmu->cpu_pmu_context, cpu); + *per_cpu_ptr(pmu->cpu_pmu_context, cpu) =3D cpc; __perf_init_event_pmu_context(&cpc->epc, pmu); __perf_mux_hrtimer_init(cpc, cpu); } From nobody Sun Feb 8 09:32:55 2026 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id EA1DD22ACF1 for ; Wed, 5 Feb 2025 10:25:34 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=90.155.50.34 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738751136; cv=none; b=izB7hiWcPqj62MZ0w1PikM8rh9K4Vr7cG7+2tfGpD1Mfuq+WzFuEVmyqNYeqrHGba+NtWDGnC0BumrDpYHBBVsq8tManzZxdOXM03f0qAISGzvlgu6PliOA2+QviEbhuu1G5Z7sDsobGVGUS5BU/6W8EoB3lzwOjaOwXGjBaBc0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738751136; c=relaxed/simple; bh=axSv5kNHfFatY3qrMWazkxF0Xp0A0k5ornDlIv1p4P8=; h=Message-ID:Date:From:To:Cc:Subject:References:MIME-Version: Content-Type; b=Ka/L42hJ5XzDn6HxItIcauunrV8tvP82eHdNk4WC9UREs81RXU8YR8xnmoBt278Cd+W0sn3x4m3H0xJLkZLU3tnggth6MPy2QnjQe21ibtfvqnWZqNOuCA88X068VcjA9uET+F+2KzxyavQRm4AnB2T6B0A3+4FYalrdlPJWv9M= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org; spf=none smtp.mailfrom=infradead.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b=HlgKGx+G; arc=none smtp.client-ip=90.155.50.34 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=infradead.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="HlgKGx+G" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=Fgq+swJWnqWBfSBKjOMZU1+pyDensX+uKGeMorHPpsI=; b=HlgKGx+Gd7R8YsbqwX/nofzc3O 9IlAAg6P3+ly/i5nj69BYisE+pufuAduqZ966g/9yo1G2R7jUNL2GvBtrBt5xk4eNg7WiT+AFq1mn Z4GETiBzQl+xxZq2UDC9najXbUqdJ0DMF2fiGeHuZA/tkS5zJnEuze24dNWPucdhAPX4kmjzXMJHW SVu7tfrTdDq1AREO9jHAfi9MYamo+M0BFh95M7v4cs5PYixuRA28FV9F66+il8i6jv5dnmU+0XZ89 TFfxnhnkGu8DdTwiGAVioFGL+0ZO3gKwqYKRW8FvHwNZBBOkTRlmaRrxVudT+QsvlHlNVMwg2jRzj 5W1ZdFJQ==; Received: from 77-249-17-89.cable.dynamic.v4.ziggo.nl ([77.249.17.89] helo=noisy.programming.kicks-ass.net) by casper.infradead.org with esmtpsa (Exim 4.98 #2 (Red Hat Linux)) id 1tfcb4-00000004GQ6-1wMt; Wed, 05 Feb 2025 10:25:30 +0000 Received: by noisy.programming.kicks-ass.net (Postfix, from userid 0) id ED27A3080B3; Wed, 5 Feb 2025 11:25:27 +0100 (CET) Message-ID: <20250205102450.119262657@infradead.org> User-Agent: quilt/0.66 Date: Wed, 05 Feb 2025 11:21:37 +0100 From: Peter Zijlstra To: mingo@kernel.org, ravi.bangoria@amd.com, lucas.demarchi@intel.com Cc: linux-kernel@vger.kernel.org, peterz@infradead.org, willy@infradead.org, acme@kernel.org, namhyung@kernel.org, mark.rutland@arm.com, alexander.shishkin@linux.intel.com, jolsa@kernel.org, irogers@google.com, adrian.hunter@intel.com, kan.liang@linux.intel.com Subject: [PATCH v2 17/24] perf: Introduce perf_free_addr_filters() References: <20250205102120.531585416@infradead.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Replace _free_event()'s use of perf_addr_filters_splice()s use with an explicit perf_free_addr_filters() with the explicit propery that it is able to be called a second time without ill effect. Most notable, referencing event->pmu must be avoided when there are no filters left (from eg a previous call). Signed-off-by: Peter Zijlstra (Intel) Reviewed-by: Ravi Bangoria --- kernel/events/core.c | 19 ++++++++++++++++--- 1 file changed, 16 insertions(+), 3 deletions(-) --- a/kernel/events/core.c +++ b/kernel/events/core.c @@ -5307,8 +5307,7 @@ static bool exclusive_event_installable( return true; } =20 -static void perf_addr_filters_splice(struct perf_event *event, - struct list_head *head); +static void perf_free_addr_filters(struct perf_event *event); =20 static void perf_pending_task_sync(struct perf_event *event) { @@ -5407,7 +5406,7 @@ static void _free_event(struct perf_even } =20 perf_event_free_bpf_prog(event); - perf_addr_filters_splice(event, NULL); + perf_free_addr_filters(event); =20 __free_event(event); } @@ -10880,6 +10882,17 @@ static void perf_addr_filters_splice(str free_filters_list(&list); } =20 +static void perf_free_addr_filters(struct perf_event *event) +{ + /* + * Used during free paths, there is no concurrency. + */ + if (list_empty(&event->addr_filters.list)) + return; + + perf_addr_filters_splice(event, NULL); +} + /* * Scan through mm's vmas and see if one of them matches the * @filter; if so, adjust filter's address range. From nobody Sun Feb 8 09:32:55 2026 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8CB5E22ACDF for ; Wed, 5 Feb 2025 10:25:34 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=90.155.50.34 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738751136; cv=none; b=ia+KVBd6iQX5lYIydoxnEWgJnrCYlOEjj3EKREUUGTTjqfBLxGjmmL0oJBwaopBfstigroF3fqjUpKPTZJonh72nezy70hXJkg2k310oc6iraI9C885+ugYMzR+mxUGzZbcZHrdAiYn3dOsh3hBSo5WHjQ3d1O9r9SZsP2skWrk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738751136; c=relaxed/simple; bh=iU1mfq4vL+87Qe1N+xJ3oAYSLpoDgo2jjmOnpa8z3AY=; h=Message-ID:Date:From:To:Cc:Subject:References:MIME-Version: Content-Type; b=Q5/IyD00EE/tarordfn2r5LvY8Xlq1L5I900lVJ2ZqDDWk+qhIUcC0HVK7D6PjOL67jfHv7dNU9ujKnmvJ7lqvMWC3zyHkGMzsC32ZzrAWm7BemPIHJAwUdqHz0DFixJ4IJTjYoDdnQH2GrW5J8zZ8wZ+33q2cqajce9VFaxmBQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org; spf=none smtp.mailfrom=infradead.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b=V6hg/fMP; arc=none smtp.client-ip=90.155.50.34 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=infradead.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="V6hg/fMP" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=Jt1bq5imd9I+0bQfTxJMRZoYW9GAQBZvZ0BHkv1KS+o=; b=V6hg/fMPGZl8AToiVn8RDyOEHv 1XojFLZynbTq5BLYoy99bqJZb0xZYeDtpSFgah/A+s4psx8Cu08YUuZJ5VbJ31iBs/bYJnwEaTiCT emfVEpODspmNyNH3Ao798Yj+YIKAx0t6YhZxKkRRgtdGwTzE6By5HEx9JZQK2GAVjEVju917pUW8G TuoMMdnxDUg8ATAZ6gpVdYap8R+05NQsg16hX/CvKCUmnmJHr8X40kEXbMI0hbJ7Lj1/Iqbd6X1FG KkgGr3DoyESajRaF/l65omcAgxTiBuvdKV18HnVcFDyBGlvpMLZrrUpsLA5CI8DIrGDsNKBOE6h7x Yrq+Xzww==; Received: from 77-249-17-89.cable.dynamic.v4.ziggo.nl ([77.249.17.89] helo=noisy.programming.kicks-ass.net) by casper.infradead.org with esmtpsa (Exim 4.98 #2 (Red Hat Linux)) id 1tfcb4-00000004GQ7-24ak; Wed, 05 Feb 2025 10:25:30 +0000 Received: by noisy.programming.kicks-ass.net (Postfix, from userid 0) id F08D9308136; Wed, 5 Feb 2025 11:25:27 +0100 (CET) Message-ID: <20250205102450.240970193@infradead.org> User-Agent: quilt/0.66 Date: Wed, 05 Feb 2025 11:21:38 +0100 From: Peter Zijlstra To: mingo@kernel.org, ravi.bangoria@amd.com, lucas.demarchi@intel.com Cc: linux-kernel@vger.kernel.org, peterz@infradead.org, willy@infradead.org, acme@kernel.org, namhyung@kernel.org, mark.rutland@arm.com, alexander.shishkin@linux.intel.com, jolsa@kernel.org, irogers@google.com, adrian.hunter@intel.com, kan.liang@linux.intel.com Subject: [PATCH v2 18/24] perf: Robustify perf_event_free_bpf_prog() References: <20250205102120.531585416@infradead.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Ensure perf_event_free_bpf_prog() is safe to call a second time; notably without making any references to event->pmu when there is no prog left. Signed-off-by: Peter Zijlstra (Intel) Reviewed-by: Ravi Bangoria --- kernel/events/core.c | 19 ++++++++++++++++--- 1 file changed, 16 insertions(+), 3 deletions(-) --- a/kernel/events/core.c +++ b/kernel/events/core.c @@ -10782,6 +10781,9 @@ int perf_event_set_bpf_prog(struct perf_ =20 void perf_event_free_bpf_prog(struct perf_event *event) { + if (!event->prog) + return; + if (!perf_event_is_tracing(event)) { perf_event_free_bpf_handler(event); return; From nobody Sun Feb 8 09:32:55 2026 Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3498222B582 for ; Wed, 5 Feb 2025 10:25:35 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=90.155.92.199 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738751138; cv=none; b=Zsqabap6guORZY1Jm6TFAege6FIgT8nllfWc/Ldfk0k70+3cHPCa+d3+PybEkh15dFfq6oG66NogVBH8DfnNwX5yYguBzS5zAkVkJl+aLBnoPI+pigWgj/qtp+mpxkzHOxHSf6t1ONN3Oc3VD5AeD4KzNk3q90+ZxO+oMIXXOeQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738751138; c=relaxed/simple; bh=G/8gpuRBjgFB9ukLyeqX5GcoznpbBpChcH0nQSZO2mU=; h=Message-ID:Date:From:To:Cc:Subject:References:MIME-Version: Content-Type; b=SlmrdhCpbUyGiuyglOWcF96p9arNibVmiPIgyIK2lJC1bBB88AwTfr6msjQeZz/inVGIjz0O8hTXYcSq1wVqbUiyGsM9Un5g1pCddBwtdY5sD+F2Nn8vBEtd8HXtLS9hidOp7XSgBiga3YIDX7xHR3BQFEQUhuL7ysGi8sKwIzI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org; spf=none smtp.mailfrom=infradead.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b=Yajd7LFB; arc=none smtp.client-ip=90.155.92.199 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=infradead.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="Yajd7LFB" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=iNUen6OBM+L20CA6MxshEiynG2hCrEafEcpdg8s7hL0=; b=Yajd7LFBPuJF3PQ+otJlvWBaVc taWsSL9qpjr6x7RI5VpB0mFGQQ4Y/sJqSsgz2av6gQhnnwhekd8pxFpFwe8/L75+VhICxDGCSPQbF vN+8UrWTLc2IHjZtEFGuZf4fV5u7P1WzzuvXM9/dBH83UgJkHhBC/geDT1gLkvmoqwyrlF36dIGD6 12k9Vd6LfhP80TzW2eB9Z7zwpZ2i+6aFxg6avzaLP77gV7QPsBUXYpjb56s5N43tc6LhInykX4PWI otbjbqnY+IuaV5JbhtOQ6UbQWkX5HwQ/oAMFPFsklhqQIvdnQx5o4Ad3aCKsF2aPPTXL2uzB5evGa W6Cv6eUQ==; Received: from 77-249-17-89.cable.dynamic.v4.ziggo.nl ([77.249.17.89] helo=noisy.programming.kicks-ass.net) by desiato.infradead.org with esmtpsa (Exim 4.98 #2 (Red Hat Linux)) id 1tfcb4-0000000GbmP-2JIz; Wed, 05 Feb 2025 10:25:31 +0000 Received: by noisy.programming.kicks-ass.net (Postfix, from userid 0) id F403530813A; Wed, 5 Feb 2025 11:25:27 +0100 (CET) Message-ID: <20250205102450.350989371@infradead.org> User-Agent: quilt/0.66 Date: Wed, 05 Feb 2025 11:21:39 +0100 From: Peter Zijlstra To: mingo@kernel.org, ravi.bangoria@amd.com, lucas.demarchi@intel.com Cc: linux-kernel@vger.kernel.org, peterz@infradead.org, willy@infradead.org, acme@kernel.org, namhyung@kernel.org, mark.rutland@arm.com, alexander.shishkin@linux.intel.com, jolsa@kernel.org, irogers@google.com, adrian.hunter@intel.com, kan.liang@linux.intel.com Subject: [PATCH v2 19/24] perf: Simplify perf_mmap() control flow References: <20250205102120.531585416@infradead.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" if (c) { X1; } else { Y; goto l; } X2; l: into: if (c) { X1; X2; } else { Y; } Signed-off-by: Peter Zijlstra (Intel) Reviewed-by: Ravi Bangoria --- kernel/events/core.c | 73 ++++++++++++++++++++++++----------------------= ----- 1 file changed, 35 insertions(+), 38 deletions(-) --- a/kernel/events/core.c +++ b/kernel/events/core.c @@ -6664,6 +6664,41 @@ static int perf_mmap(struct file *file, =20 if (vma->vm_pgoff =3D=3D 0) { nr_pages =3D (vma_size / PAGE_SIZE) - 1; + + /* + * If we have rb pages ensure they're a power-of-two number, so we + * can do bitmasks instead of modulo. + */ + if (nr_pages !=3D 0 && !is_power_of_2(nr_pages)) + return -EINVAL; + + if (vma_size !=3D PAGE_SIZE * (1 + nr_pages)) + return -EINVAL; + + WARN_ON_ONCE(event->ctx->parent_ctx); +again: + mutex_lock(&event->mmap_mutex); + if (event->rb) { + if (data_page_nr(event->rb) !=3D nr_pages) { + ret =3D -EINVAL; + goto unlock; + } + + if (!atomic_inc_not_zero(&event->rb->mmap_count)) { + /* + * Raced against perf_mmap_close(); remove the + * event and try again. + */ + ring_buffer_attach(event, NULL); + mutex_unlock(&event->mmap_mutex); + goto again; + } + + rb =3D event->rb; + goto unlock; + } + + user_extra =3D nr_pages + 1; } else { /* * AUX area mapping: if rb->aux_nr_pages !=3D 0, it's already @@ -6723,47 +6758,9 @@ static int perf_mmap(struct file *file, =20 atomic_set(&rb->aux_mmap_count, 1); user_extra =3D nr_pages; - - goto accounting; } =20 - /* - * If we have rb pages ensure they're a power-of-two number, so we - * can do bitmasks instead of modulo. - */ - if (nr_pages !=3D 0 && !is_power_of_2(nr_pages)) - return -EINVAL; - - if (vma_size !=3D PAGE_SIZE * (1 + nr_pages)) - return -EINVAL; - - WARN_ON_ONCE(event->ctx->parent_ctx); -again: - mutex_lock(&event->mmap_mutex); - if (event->rb) { - if (data_page_nr(event->rb) !=3D nr_pages) { - ret =3D -EINVAL; - goto unlock; - } - - if (!atomic_inc_not_zero(&event->rb->mmap_count)) { - /* - * Raced against perf_mmap_close(); remove the - * event and try again. - */ - ring_buffer_attach(event, NULL); - mutex_unlock(&event->mmap_mutex); - goto again; - } - /* We need the rb to map pages. */ - rb =3D event->rb; - goto unlock; - } - - user_extra =3D nr_pages + 1; - -accounting: user_lock_limit =3D sysctl_perf_event_mlock >> (PAGE_SHIFT - 10); =20 /* From nobody Sun Feb 8 09:32:55 2026 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A23BA22B59C for ; Wed, 5 Feb 2025 10:25:36 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=90.155.50.34 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738751138; cv=none; b=RX83Ld5pZG0EYWWh699aYqiGPZdJ/61JbxsVwcG62Hy2DaDUeSZWdAwYMPpBt2beDx1bshDMjsatvTK7LxjpnwVow3VaeR6sj2xo6yzmQtugNGt1zhj4eJf3aceZ4raWzOr9cKxoZ1ZTKinNYpDohDebemjNYV7Mdu87Q1k9ITg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738751138; c=relaxed/simple; bh=Vh/ANEgyY8n5fImb081ab42mhkj0zQrjYMF1NAwI1mQ=; h=Message-ID:Date:From:To:Cc:Subject:References:MIME-Version: Content-Type; b=eu/sbzbooNeN6XxV8M/BugUa2UIetZFV/mJqJEMC7smgpD7HbfqPtTWsk644CTF+d6TMuLTFol8Wdzwg57+RtTuMCe+LS8PxXCGsOqirDDAmKwnCCpESaTN3bEO6MnxmrOwNbbZvuJwUsYjTZAqcdyCK5FucNoKmi5wOFrufwZk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org; spf=none smtp.mailfrom=infradead.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b=ZP4lFCpd; arc=none smtp.client-ip=90.155.50.34 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=infradead.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="ZP4lFCpd" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=gXbe8TsfFkdLWBnFeHNPgw4BSHCBxSKL0V50l6k/6n8=; b=ZP4lFCpdOvAnzPIJYu92SqLUYo A3T4p1chIChfUGzX7PfOZWmSNVQuEkEHgYivzGQzeq42s/XwPxeXxRd3cz5NIAE0W2ziITYEUdnqA k2E5rkh+X/BCTYPfee2LZlXWK9KHVW86Pyd2tqZjVGqacBPHIKPbvg4/+Eu7CwycoiZ0U2gfLPnf8 Fg1Hjd75Cfl/3zL8Qdw0RCDWurTlyPCvML+AArntnIuEjsC8tbZjbHxKrcYqmrnEpknPd4RVeB7A5 oi5/wMt4D4d17wrFyB9Z1RVz62D07ZYoGC6keT/PpWO9Xr8IlKe3wjXsuvee32xHEqRJa0P0wf3un qnCNJB4w==; Received: from 77-249-17-89.cable.dynamic.v4.ziggo.nl ([77.249.17.89] helo=noisy.programming.kicks-ass.net) by casper.infradead.org with esmtpsa (Exim 4.98 #2 (Red Hat Linux)) id 1tfcb4-00000004GQ8-2J4c; Wed, 05 Feb 2025 10:25:30 +0000 Received: by noisy.programming.kicks-ass.net (Postfix, from userid 0) id 033BB3085D1; Wed, 5 Feb 2025 11:25:28 +0100 (CET) Message-ID: <20250205102450.458085859@infradead.org> User-Agent: quilt/0.66 Date: Wed, 05 Feb 2025 11:21:40 +0100 From: Peter Zijlstra To: mingo@kernel.org, ravi.bangoria@amd.com, lucas.demarchi@intel.com Cc: linux-kernel@vger.kernel.org, peterz@infradead.org, willy@infradead.org, acme@kernel.org, namhyung@kernel.org, mark.rutland@arm.com, alexander.shishkin@linux.intel.com, jolsa@kernel.org, irogers@google.com, adrian.hunter@intel.com, kan.liang@linux.intel.com Subject: [PATCH v2 20/24] perf: Fix perf_mmap() failure path References: <20250205102120.531585416@infradead.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" When f_ops->mmap() returns failure, m_ops->close() is *not* called. Signed-off-by: Peter Zijlstra (Intel) Reviewed-by: Ravi Bangoria --- kernel/events/core.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) --- a/kernel/events/core.c +++ b/kernel/events/core.c @@ -6852,7 +6852,7 @@ static int perf_mmap(struct file *file, if (!ret) ret =3D map_range(rb, vma); =20 - if (event->pmu->event_mapped) + if (!ret && event->pmu->event_mapped) event->pmu->event_mapped(event, vma->vm_mm); =20 return ret; From nobody Sun Feb 8 09:32:55 2026 Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3D70422B590 for ; Wed, 5 Feb 2025 10:25:36 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=90.155.92.199 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738751138; cv=none; b=JReEFF6wqBguyhES+bIvwbP6DQ229TWWFEJIUMdJnWb7GmQauUMAB1b+7K+2FfYsS0QWBljgPDHm3PorZNB4JTQcF0TG8Ls74Du8kK4rNHMlpKYWPknlyXUuJ2wkwfVCnE/mGiPdoMozh2ypwsYiLmdCkIZ3T+GuHuzht0oxTqM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738751138; c=relaxed/simple; bh=rrbiOmz2jF0M7L1i0onuQaUMQEHoe8XCSP0wQvcLi6c=; h=Message-ID:Date:From:To:Cc:Subject:References:MIME-Version: Content-Type; b=CvrI5TgTENUJ97pXYRE8qiwmwQMtPTK7KGSmJmHCxKoOek7OjPv3bp8vA83bbCbrgdvnRn4oHUsUEuZsjAMPBFW/1mGIV5+/ZcFGHLJ6mO4dQNThqe7t6k4lktdmByQccwJK/KcmF50aW64t6XkyZfLGHcxoEG7gpl0f4eCTycE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org; spf=none smtp.mailfrom=infradead.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b=oT7ehJRI; arc=none smtp.client-ip=90.155.92.199 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=infradead.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="oT7ehJRI" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=pfiwW0dLUfHVNchZgL9hTztc6fyT30FB5T+P5wRGV0k=; b=oT7ehJRIlW8qvL/xMPkpDVw95M FTlpyl6ZoQw4+7Zg6xRrwR/5V2MlPRAmlDcaxLGR0KhKj3VqAI/CY2T31v+oEFG6Nm/SgY0niEVfU FmWzcB9BJ/sEMC8e41exIEbWfKp86AHVzZMD02LefN0ooy6mTlNVmSFIEHXMJyOWDXn77Z5lx1Ba0 XwM/2XIo4W+ZGCa3fRjz+3L16juCfNOBxwuPniKbMnHw0c5tIEzmuCfbwKj5yOc/szwe0VL+2ETfX GXfP3WXpq0L5KziDJsqYYOOaQVdzGj7krEAPkOFOTe3MFS3zNWGbBetn6GBQ9c4P1BhZrEmZJuvNN +hQVvEvA==; Received: from 77-249-17-89.cable.dynamic.v4.ziggo.nl ([77.249.17.89] helo=noisy.programming.kicks-ass.net) by desiato.infradead.org with esmtpsa (Exim 4.98 #2 (Red Hat Linux)) id 1tfcb4-0000000GbmQ-2Xiy; Wed, 05 Feb 2025 10:25:32 +0000 Received: by noisy.programming.kicks-ass.net (Postfix, from userid 0) id 066ED3085D7; Wed, 5 Feb 2025 11:25:28 +0100 (CET) Message-ID: <20250205102450.566454649@infradead.org> User-Agent: quilt/0.66 Date: Wed, 05 Feb 2025 11:21:41 +0100 From: Peter Zijlstra To: mingo@kernel.org, ravi.bangoria@amd.com, lucas.demarchi@intel.com Cc: linux-kernel@vger.kernel.org, peterz@infradead.org, willy@infradead.org, acme@kernel.org, namhyung@kernel.org, mark.rutland@arm.com, alexander.shishkin@linux.intel.com, jolsa@kernel.org, irogers@google.com, adrian.hunter@intel.com, kan.liang@linux.intel.com Subject: [PATCH v2 21/24] perf: Further simplify perf_mmap() References: <20250205102120.531585416@infradead.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Signed-off-by: Peter Zijlstra (Intel) Reviewed-by: Ravi Bangoria --- kernel/events/core.c | 21 ++++++++++----------- 1 file changed, 10 insertions(+), 11 deletions(-) --- a/kernel/events/core.c +++ b/kernel/events/core.c @@ -6661,9 +6661,18 @@ static int perf_mmap(struct file *file, return ret; =20 vma_size =3D vma->vm_end - vma->vm_start; + nr_pages =3D vma_size / PAGE_SIZE; + + if (nr_pages > INT_MAX) + return -ENOMEM; + + if (vma_size !=3D PAGE_SIZE * nr_pages) + return -EINVAL; + + user_extra =3D nr_pages; =20 if (vma->vm_pgoff =3D=3D 0) { - nr_pages =3D (vma_size / PAGE_SIZE) - 1; + nr_pages -=3D 1; =20 /* * If we have rb pages ensure they're a power-of-two number, so we @@ -6672,9 +6681,6 @@ static int perf_mmap(struct file *file, if (nr_pages !=3D 0 && !is_power_of_2(nr_pages)) return -EINVAL; =20 - if (vma_size !=3D PAGE_SIZE * (1 + nr_pages)) - return -EINVAL; - WARN_ON_ONCE(event->ctx->parent_ctx); again: mutex_lock(&event->mmap_mutex); @@ -6697,8 +6703,6 @@ static int perf_mmap(struct file *file, rb =3D event->rb; goto unlock; } - - user_extra =3D nr_pages + 1; } else { /* * AUX area mapping: if rb->aux_nr_pages !=3D 0, it's already @@ -6710,10 +6714,6 @@ static int perf_mmap(struct file *file, if (!event->rb) return -EINVAL; =20 - nr_pages =3D vma_size / PAGE_SIZE; - if (nr_pages > INT_MAX) - return -ENOMEM; - mutex_lock(&event->mmap_mutex); ret =3D -EINVAL; =20 @@ -6757,7 +6757,6 @@ static int perf_mmap(struct file *file, } =20 atomic_set(&rb->aux_mmap_count, 1); - user_extra =3D nr_pages; } =20 /* We need the rb to map pages. */ From nobody Sun Feb 8 09:32:55 2026 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8E90F22AE65 for ; Wed, 5 Feb 2025 10:25:35 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=90.155.50.34 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738751137; cv=none; b=e9NBS1RRYBYD3L7y++3rJACJMH/0Y8i75bii0qDgckxG7IkzWzbRLZz+88PmfLS8coKcs2rm17VhHACXfBe4WB9Dt3RqxuZEMEnDbkGx62N5EBhOfN/Muv/H2LSJCtLUTbiIOcdf8tRI8fUJjCkRO5mAqglBvD7XaeFSdzA/xQg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738751137; c=relaxed/simple; bh=nl/E7IbifIHY4saIGLrcUyqLZKw5Y4dVMLE4DYop58Y=; h=Message-ID:Date:From:To:Cc:Subject:References:MIME-Version: Content-Type; b=lVugHw5EslAkKuPeszwceVGWSlLDP7LvNGh1EuDUDBVp/5Fc2yKInZQHbFD6SUrqA3bPFAE4sFb/oGR6tTphfEabhTzPTNU5AYANYzVeMOBQySK2adK/5KqlEbzqsD4zCjbJsg5gXZtXP2MCb77tYTbVKEHy278K3Bh1PktbnL8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org; spf=none smtp.mailfrom=infradead.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b=pza+EoGK; arc=none smtp.client-ip=90.155.50.34 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=infradead.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="pza+EoGK" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=zvAjMRogCzZXBg0SQLthP0VbH9NxSHSRwZYyo2uheI4=; b=pza+EoGKR8cpT/KN8sT5/+L4pD 6Zp2YqczUFeL6npG5/mfBsDhyGZGdPholkSKZpc0NV6diavYr8G6KgkT6KrVrLZifPQBKqSCngLGk 8MLiT3hqDty7zA5MmwPT3w/3UBdZNY5vgkJ1bPF9PR3YtGHUTBrxvlVCI4aVlnOjF4zlpjxcSnjV0 7rGwqkg0kx1CqBQKqBX9bgQPWYvMa6lYlLabQZ/h5tnOfM3Tmoth9COyPByTSGfM3W5JBOSm4CBRD FR4cO6D7x7MubuzV+4MWbVlLDFc4/P5bxXuHNQsEQww3sXUDnVEpwdwl67/G84FAUiiNMFaJ7nNvj RNfWi5hQ==; Received: from 77-249-17-89.cable.dynamic.v4.ziggo.nl ([77.249.17.89] helo=noisy.programming.kicks-ass.net) by casper.infradead.org with esmtpsa (Exim 4.98 #2 (Red Hat Linux)) id 1tfcb4-00000004GQ9-2U9P; Wed, 05 Feb 2025 10:25:30 +0000 Received: by noisy.programming.kicks-ass.net (Postfix, from userid 0) id 0A3863085E6; Wed, 5 Feb 2025 11:25:28 +0100 (CET) Message-ID: <20250205102450.672685519@infradead.org> User-Agent: quilt/0.66 Date: Wed, 05 Feb 2025 11:21:42 +0100 From: Peter Zijlstra To: mingo@kernel.org, ravi.bangoria@amd.com, lucas.demarchi@intel.com Cc: linux-kernel@vger.kernel.org, peterz@infradead.org, willy@infradead.org, acme@kernel.org, namhyung@kernel.org, mark.rutland@arm.com, alexander.shishkin@linux.intel.com, jolsa@kernel.org, irogers@google.com, adrian.hunter@intel.com, kan.liang@linux.intel.com Subject: [PATCH v2 22/24] perf: Remove retry loop from perf_mmap() References: <20250205102120.531585416@infradead.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" AFAICT there is no actual benefit from the mutex drop on re-try. The 'worst' case scenario is that we instantly re-gain the mutex without perf_mmap_close() getting it. So might as well make that the normal case. Reflow the code to make the ring buffer detach case naturally flow into the no ring buffer case. Signed-off-by: Peter Zijlstra (Intel) Reviewed-by: Ravi Bangoria --- kernel/events/core.c | 23 ++++++++++++++--------- 1 file changed, 14 insertions(+), 9 deletions(-) --- a/kernel/events/core.c +++ b/kernel/events/core.c @@ -6667,27 +6667,32 @@ static int perf_mmap(struct file *file, return -EINVAL; =20 WARN_ON_ONCE(event->ctx->parent_ctx); -again: mutex_lock(&event->mmap_mutex); + if (event->rb) { if (data_page_nr(event->rb) !=3D nr_pages) { ret =3D -EINVAL; goto unlock; } =20 - if (!atomic_inc_not_zero(&event->rb->mmap_count)) { + if (atomic_inc_not_zero(&event->rb->mmap_count)) { /* - * Raced against perf_mmap_close(); remove the - * event and try again. + * Success -- managed to mmap() the same buffer + * multiple times. */ - ring_buffer_attach(event, NULL); - mutex_unlock(&event->mmap_mutex); - goto again; + rb =3D event->rb; + ret =3D 0; + goto unlock; } =20 - rb =3D event->rb; - goto unlock; + /* + * Raced against perf_mmap_close()'s + * atomic_dec_and_mutex_lock() remove the + * event and continue as if !event->rb + */ + ring_buffer_attach(event, NULL); } + } else { /* * AUX area mapping: if rb->aux_nr_pages !=3D 0, it's already From nobody Sun Feb 8 09:32:55 2026 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 96BFE22B8B9 for ; Wed, 5 Feb 2025 10:25:37 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=90.155.50.34 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738751139; cv=none; b=VVb8VkIa+Imya9UUoxep/4rejk0Qilg6WxTQOiHHEc5zRWH6d65GE8Ji4VlhEjxb2c2LcXLxjDHcDceb/yQnGlLrFOmJ7lYzZrMv1KIXyJS83IVlcjFeNdPUi28sH4pvdqsoauHcQAFbWjKpO88ALvnaC9ec+1ETpPOx4O5/RhA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738751139; c=relaxed/simple; bh=G9kQZVQBtt+RsD9ItrMsGJ3XXGNJo6t6g0PxG20qb1I=; h=Message-ID:Date:From:To:Cc:Subject:References:MIME-Version: Content-Type; b=rCUgqQJV/s69vapiKpeERppukljzNxGsGxxMnkH6LoEa3Qx3CQTRtT/KD/b/bT2FMwF59lO65qYI3rz2mUgpWmpGPtS5y9TeB+Y1s5rP3bM9JayRgjJoxx9JYt6I/8kv0MhRGf+/bQ+SFjxNRO5KQ+JxW/LHB8LJR7lKqYLTJjk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org; spf=none smtp.mailfrom=infradead.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b=Ll8QTfYk; arc=none smtp.client-ip=90.155.50.34 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=infradead.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="Ll8QTfYk" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=XRZxkRqsf7mtFnenUXbwbTX3ORyj+dl5pWe7RLTsNck=; b=Ll8QTfYk2zz3e5BCmrrN9jI1uf 7Ho3vr/XB08zfhvDg7afq/+P3TFC0niBsAKY7l6DUoCMnOfxkTjiMgdPbmx16CeB0KMjaEqynWeoA s5yUrl3Ylv6qdikh3L4fnCKcSDY+fJJF0sSkCOC45cfYhkv3Tpk7FtHp3XpyhjOxytlSNlY+bfsxx IrNUZCUS5u9qbkiL037s/URZNd5e6yQmgPBICC63jqiV4uIhAYIuxX9TfrWkJNYU6IXu04BYvKMF6 jELHDM/FVmxESSO7oGwRjmlvTSg9hNr/eP31BbTXpMndeAhdALTvVbRYiR4BGOaJC6XUEMNHxcCEG UAsvHylw==; Received: from 77-249-17-89.cable.dynamic.v4.ziggo.nl ([77.249.17.89] helo=noisy.programming.kicks-ass.net) by casper.infradead.org with esmtpsa (Exim 4.98 #2 (Red Hat Linux)) id 1tfcb4-00000004GQA-2Z6r; Wed, 05 Feb 2025 10:25:30 +0000 Received: by noisy.programming.kicks-ass.net (Postfix, from userid 0) id 0DCAE3085FC; Wed, 5 Feb 2025 11:25:28 +0100 (CET) Message-ID: <20250205102450.781721189@infradead.org> User-Agent: quilt/0.66 Date: Wed, 05 Feb 2025 11:21:43 +0100 From: Peter Zijlstra To: mingo@kernel.org, ravi.bangoria@amd.com, lucas.demarchi@intel.com Cc: linux-kernel@vger.kernel.org, peterz@infradead.org, willy@infradead.org, acme@kernel.org, namhyung@kernel.org, mark.rutland@arm.com, alexander.shishkin@linux.intel.com, jolsa@kernel.org, irogers@google.com, adrian.hunter@intel.com, kan.liang@linux.intel.com Subject: [PATCH v2 23/24] perf: Lift event->mmap_mutex in perf_mmap() References: <20250205102120.531585416@infradead.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" This puts 'all' of perf_mmap() under single event->mmap_mutex. Signed-off-by: Peter Zijlstra (Intel) Reviewed-by: Ravi Bangoria --- kernel/events/core.c | 20 ++++++++------------ 1 file changed, 8 insertions(+), 12 deletions(-) --- a/kernel/events/core.c +++ b/kernel/events/core.c @@ -6643,7 +6643,7 @@ static int perf_mmap(struct file *file, unsigned long vma_size; unsigned long nr_pages; long user_extra =3D 0, extra =3D 0; - int ret =3D 0, flags =3D 0; + int ret, flags =3D 0; =20 /* * Don't allow mmap() of inherited per-task counters. This would @@ -6671,6 +6671,9 @@ static int perf_mmap(struct file *file, =20 user_extra =3D nr_pages; =20 + mutex_lock(&event->mmap_mutex); + ret =3D -EINVAL; + if (vma->vm_pgoff =3D=3D 0) { nr_pages -=3D 1; =20 @@ -6679,16 +6682,13 @@ static int perf_mmap(struct file *file, * can do bitmasks instead of modulo. */ if (nr_pages !=3D 0 && !is_power_of_2(nr_pages)) - return -EINVAL; + goto unlock; =20 WARN_ON_ONCE(event->ctx->parent_ctx); - mutex_lock(&event->mmap_mutex); =20 if (event->rb) { - if (data_page_nr(event->rb) !=3D nr_pages) { - ret =3D -EINVAL; + if (data_page_nr(event->rb) !=3D nr_pages) goto unlock; - } =20 if (atomic_inc_not_zero(&event->rb->mmap_count)) { /* @@ -6716,12 +6716,6 @@ static int perf_mmap(struct file *file, */ u64 aux_offset, aux_size; =20 - if (!event->rb) - return -EINVAL; - - mutex_lock(&event->mmap_mutex); - ret =3D -EINVAL; - rb =3D event->rb; if (!rb) goto aux_unlock; @@ -6832,6 +6826,8 @@ static int perf_mmap(struct file *file, rb->aux_mmap_locked =3D extra; } =20 + ret =3D 0; + unlock: if (!ret) { atomic_long_add(user_extra, &user->locked_vm); From nobody Sun Feb 8 09:32:55 2026 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id AEE8922AE75 for ; Wed, 5 Feb 2025 10:25:35 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=90.155.50.34 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738751138; cv=none; b=T88W7jNnZ+j5yqkVyF0lMxJoGeozj3tNMn3LcouTgNm1ZCbfmqeax3dRgzJ8QbmxAQGYWpS0oe9l55rVO2d4WIPZVDpvZrWnEDkOG0rO4yGdgAw/7yW23eb8ummALw+fLPyqeornnr+avgeDyD0MgIh0nYuzDLuhTLRz6oFpI0E= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738751138; c=relaxed/simple; bh=9OBLsNR3PYTymWxQWcmGEQSr3TQ3Y8EXhcc7Z+5g7ms=; h=Message-ID:Date:From:To:Cc:Subject:References:MIME-Version: Content-Type; b=e+hPLtPmUQm2sx7WgNqzzJ6j9UAnsyed9vxoPYEtpLq9T3qbQDwVB2wLYfL5oc6qHplSAUGGTs1/Jrig+sRr0rl2XwU+coKwL2tQ89PZerfPknMtmh67zOyNKmyyioadHIj4enBbSFhTeGiHMxfil4MSpNwmHJ2FpZy2S88qWbQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org; spf=none smtp.mailfrom=infradead.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b=bQ9GDGbp; arc=none smtp.client-ip=90.155.50.34 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=infradead.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="bQ9GDGbp" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=agZeE/nn7pvItqKMTgeZGYABvMsC9TD7nkyisRSC2e8=; b=bQ9GDGbpTRB8Ro8j26IWApg/uU bv9UR8wl+51h38ilFEeY4T3xVVJxp3xBtNno0VwdGl2XQYpmhhyno/V3nRCrnLkc74z73ignOQE2T QzzXeo0QvDJU46TRBoklvdmDghC9giSEinsyNqu1ND5e8eNj3r3zo4Z/0MFosDiEkzNsZknYI8/+M 5Ip4mRTO64G45zbe1mcSQG7i59sDOOpsSIi2L2c2LkHyQ5ugOcx8+EjVLQXo1UgbzBXN7n4KUZRvV 9fxnNxq19JJawMl9wevZrAc7lB+jBb8Nzqt82BpXMcWrHAjgMYRe9KI34MIQCQl7YXT8rGt4vZHyz UXhji26A==; Received: from 77-249-17-89.cable.dynamic.v4.ziggo.nl ([77.249.17.89] helo=noisy.programming.kicks-ass.net) by casper.infradead.org with esmtpsa (Exim 4.98 #2 (Red Hat Linux)) id 1tfcb4-00000004GQB-2Wd7; Wed, 05 Feb 2025 10:25:30 +0000 Received: by noisy.programming.kicks-ass.net (Postfix, from userid 0) id 115F8308607; Wed, 5 Feb 2025 11:25:28 +0100 (CET) Message-ID: <20250205102450.888979808@infradead.org> User-Agent: quilt/0.66 Date: Wed, 05 Feb 2025 11:21:44 +0100 From: Peter Zijlstra To: mingo@kernel.org, ravi.bangoria@amd.com, lucas.demarchi@intel.com Cc: linux-kernel@vger.kernel.org, peterz@infradead.org, willy@infradead.org, acme@kernel.org, namhyung@kernel.org, mark.rutland@arm.com, alexander.shishkin@linux.intel.com, jolsa@kernel.org, irogers@google.com, adrian.hunter@intel.com, kan.liang@linux.intel.com Subject: [PATCH v2 24/24] perf: Make perf_pmu_unregister() useable References: <20250205102120.531585416@infradead.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Previously it was only safe to call perf_pmu_unregister() if there were no active events of that pmu around -- which was impossible to guarantee since it races all sorts against perf_init_event(). Rework the whole thing by: - keeping track of all events for a given pmu - 'hiding' the pmu from perf_init_event() - waiting for the appropriate (s)rcu grace periods such that all prior references to the PMU will be completed - detaching all still existing events of that pmu (see first point) and moving them to a new REVOKED state. - actually freeing the pmu data. Where notably the new REVOKED state must inhibit all event actions from reaching code that wants to use event->pmu. Signed-off-by: Peter Zijlstra (Intel) Reviewed-by: Ravi Bangoria --- include/linux/perf_event.h | 15 +- kernel/events/core.c | 266 ++++++++++++++++++++++++++++++++++++++++= ----- 2 files changed, 249 insertions(+), 32 deletions(-) --- a/include/linux/perf_event.h +++ b/include/linux/perf_event.h @@ -325,6 +325,9 @@ struct perf_output_handle; struct pmu { struct list_head entry; =20 + spinlock_t events_lock; + struct list_head events; + struct module *module; struct device *dev; struct device *parent; @@ -632,9 +635,10 @@ struct perf_addr_filter_range { * enum perf_event_state - the states of an event: */ enum perf_event_state { - PERF_EVENT_STATE_DEAD =3D -4, - PERF_EVENT_STATE_EXIT =3D -3, - PERF_EVENT_STATE_ERROR =3D -2, + PERF_EVENT_STATE_DEAD =3D -5, + PERF_EVENT_STATE_REVOKED =3D -4, /* pmu gone, must not touch */ + PERF_EVENT_STATE_EXIT =3D -3, /* task died, still inherit */ + PERF_EVENT_STATE_ERROR =3D -2, /* scheduling error, can enable */ PERF_EVENT_STATE_OFF =3D -1, PERF_EVENT_STATE_INACTIVE =3D 0, PERF_EVENT_STATE_ACTIVE =3D 1, @@ -875,6 +879,7 @@ struct perf_event { void *security; #endif struct list_head sb_list; + struct list_head pmu_list; =20 /* * Certain events gets forwarded to another pmu internally by over- @@ -1126,7 +1131,7 @@ extern void perf_aux_output_flag(struct extern void perf_event_itrace_started(struct perf_event *event); =20 extern int perf_pmu_register(struct pmu *pmu, const char *name, int type); -extern void perf_pmu_unregister(struct pmu *pmu); +extern int perf_pmu_unregister(struct pmu *pmu); =20 extern void __perf_event_task_sched_in(struct task_struct *prev, struct task_struct *task); @@ -1737,7 +1742,7 @@ static inline bool needs_branch_stack(st =20 static inline bool has_aux(struct perf_event *event) { - return event->pmu->setup_aux; + return event->pmu && event->pmu->setup_aux; } =20 static inline bool has_aux_action(struct perf_event *event) --- a/kernel/events/core.c +++ b/kernel/events/core.c @@ -2418,7 +2418,9 @@ ctx_time_update_event(struct perf_event_ =20 #define DETACH_GROUP 0x01UL #define DETACH_CHILD 0x02UL -#define DETACH_DEAD 0x04UL +#define DETACH_EXIT 0x04UL +#define DETACH_REVOKE 0x08UL +#define DETACH_DEAD 0x10UL =20 /* * Cross CPU call to remove a performance event @@ -2433,6 +2435,7 @@ __perf_remove_from_context(struct perf_e void *info) { struct perf_event_pmu_context *pmu_ctx =3D event->pmu_ctx; + enum perf_event_state state =3D PERF_EVENT_STATE_OFF; unsigned long flags =3D (unsigned long)info; =20 ctx_time_update(cpuctx, ctx); @@ -2441,16 +2444,22 @@ __perf_remove_from_context(struct perf_e * Ensure event_sched_out() switches to OFF, at the very least * this avoids raising perf_pending_task() at this time. */ - if (flags & DETACH_DEAD) + if (flags & DETACH_EXIT) + state =3D PERF_EVENT_STATE_EXIT; + if (flags & DETACH_REVOKE) + state =3D PERF_EVENT_STATE_REVOKED; + if (flags & DETACH_DEAD) { event->pending_disable =3D 1; + state =3D PERF_EVENT_STATE_DEAD; + } event_sched_out(event, ctx); if (flags & DETACH_GROUP) perf_group_detach(event); if (flags & DETACH_CHILD) perf_child_detach(event); list_del_event(event, ctx); - if (flags & DETACH_DEAD) - event->state =3D PERF_EVENT_STATE_DEAD; + + event->state =3D min(event->state, state); =20 if (!pmu_ctx->nr_events) { pmu_ctx->rotate_necessary =3D 0; @@ -4510,7 +4519,8 @@ static void perf_event_enable_on_exec(st =20 static void perf_remove_from_owner(struct perf_event *event); static void perf_event_exit_event(struct perf_event *event, - struct perf_event_context *ctx); + struct perf_event_context *ctx, + bool revoke); =20 /* * Removes all events from the current task that have been marked @@ -4537,7 +4547,7 @@ static void perf_event_remove_on_exec(st =20 modified =3D true; =20 - perf_event_exit_event(event, ctx); + perf_event_exit_event(event, ctx, false); } =20 raw_spin_lock_irqsave(&ctx->lock, flags); @@ -5137,6 +5147,7 @@ static bool is_sb_event(struct perf_even attr->context_switch || attr->text_poke || attr->bpf_event) return true; + return false; } =20 @@ -5338,6 +5349,8 @@ static void perf_pending_task_sync(struc /* vs perf_event_alloc() error */ static void __free_event(struct perf_event *event) { + struct pmu *pmu =3D event->pmu; + if (event->attach_state & PERF_ATTACH_CALLCHAIN) put_callchain_buffers(); =20 @@ -5364,6 +5377,7 @@ static void __free_event(struct perf_eve * put_pmu_ctx() needs an event->ctx reference, because of * epc->ctx. */ + WARN_ON_ONCE(!pmu); WARN_ON_ONCE(!event->ctx); WARN_ON_ONCE(event->pmu_ctx->ctx !=3D event->ctx); put_pmu_ctx(event->pmu_ctx); @@ -5376,8 +5390,13 @@ static void __free_event(struct perf_eve if (event->ctx) put_ctx(event->ctx); =20 - if (event->pmu) - module_put(event->pmu->module); + if (pmu) { + module_put(pmu->module); + scoped_guard (spinlock, &pmu->events_lock) { + list_del(&event->pmu_list); + wake_up_var(pmu); + } + } =20 call_rcu(&event->rcu_head, free_event_rcu); } @@ -5525,7 +5544,11 @@ int perf_event_release_kernel(struct per * Thus this guarantees that we will in fact observe and kill _ALL_ * child events. */ - perf_remove_from_context(event, DETACH_GROUP|DETACH_DEAD); + if (event->state > PERF_EVENT_STATE_REVOKED) { + perf_remove_from_context(event, DETACH_GROUP|DETACH_DEAD); + } else { + event->state =3D PERF_EVENT_STATE_DEAD; + } =20 perf_event_ctx_unlock(event, ctx); =20 @@ -5578,7 +5601,7 @@ int perf_event_release_kernel(struct per mutex_unlock(&ctx->mutex); =20 if (child) - free_event(child); + put_event(child); put_ctx(ctx); =20 goto again; @@ -5813,7 +5836,7 @@ __perf_read(struct perf_event *event, ch * error state (i.e. because it was pinned but it couldn't be * scheduled on to the CPU at some point). */ - if (event->state =3D=3D PERF_EVENT_STATE_ERROR) + if (event->state <=3D PERF_EVENT_STATE_ERROR) return 0; =20 if (count < event->read_size) @@ -5852,8 +5875,14 @@ static __poll_t perf_poll(struct file *f struct perf_buffer *rb; __poll_t events =3D EPOLLHUP; =20 + if (event->state <=3D PERF_EVENT_STATE_REVOKED) + return EPOLLERR; + poll_wait(file, &event->waitq, wait); =20 + if (event->state <=3D PERF_EVENT_STATE_REVOKED) + return EPOLLERR; + if (is_event_hup(event)) return events; =20 @@ -6027,6 +6056,9 @@ static long _perf_ioctl(struct perf_even void (*func)(struct perf_event *); u32 flags =3D arg; =20 + if (event->state <=3D PERF_EVENT_STATE_REVOKED) + return -ENODEV; + switch (cmd) { case PERF_EVENT_IOC_ENABLE: func =3D _perf_event_enable; @@ -6412,7 +6444,7 @@ static void perf_mmap_open(struct vm_are if (vma->vm_pgoff) atomic_inc(&event->rb->aux_mmap_count); =20 - if (event->pmu->event_mapped) + if (event->pmu && event->pmu->event_mapped) event->pmu->event_mapped(event, vma->vm_mm); } =20 @@ -6435,7 +6467,8 @@ static void perf_mmap_close(struct vm_ar unsigned long size =3D perf_data_size(rb); bool detach_rest =3D false; =20 - if (event->pmu->event_unmapped) + /* FIXIES vs perf_pmu_unregister() */ + if (event->pmu && event->pmu->event_unmapped) event->pmu->event_unmapped(event, vma->vm_mm); =20 /* @@ -6659,6 +6692,16 @@ static int perf_mmap(struct file *file, mutex_lock(&event->mmap_mutex); ret =3D -EINVAL; =20 + /* + * This relies on __pmu_detach_event() taking mmap_mutex after marking + * the event REVOKED. Either we observe the state, or __pmu_detach_event() + * will detach the rb created here. + */ + if (event->state <=3D PERF_EVENT_STATE_REVOKED) { + ret =3D -ENODEV; + goto unlock; + } + if (vma->vm_pgoff =3D=3D 0) { nr_pages -=3D 1; =20 @@ -6837,7 +6880,7 @@ static int perf_mmap(struct file *file, if (!ret) ret =3D map_range(rb, vma); =20 - if (!ret && event->pmu->event_mapped) + if (!ret && event->pmu && event->pmu->event_mapped) event->pmu->event_mapped(event, vma->vm_mm); =20 return ret; @@ -6849,6 +6892,9 @@ static int perf_fasync(int fd, struct fi struct perf_event *event =3D filp->private_data; int retval; =20 + if (event->state <=3D PERF_EVENT_STATE_REVOKED) + return -ENODEV; + inode_lock(inode); retval =3D fasync_helper(fd, filp, on, &event->fasync); inode_unlock(inode); @@ -10813,6 +10859,9 @@ static int __perf_event_set_bpf_prog(str { bool is_kprobe, is_uprobe, is_tracepoint, is_syscall_tp; =20 + if (event->state <=3D PERF_EVENT_STATE_REVOKED) + return -ENODEV; + if (!perf_event_is_tracing(event)) return perf_event_set_bpf_handler(event, prog, bpf_cookie); =20 @@ -11997,6 +12046,9 @@ int perf_pmu_register(struct pmu *_pmu, if (!pmu->event_idx) pmu->event_idx =3D perf_event_idx_default; =20 + INIT_LIST_HEAD(&pmu->events); + spin_lock_init(&pmu->events_lock); + /* * Now that the PMU is complete, make it visible to perf_try_init_event(). */ @@ -12010,21 +12062,143 @@ int perf_pmu_register(struct pmu *_pmu, } EXPORT_SYMBOL_GPL(perf_pmu_register); =20 -void perf_pmu_unregister(struct pmu *pmu) +static void __pmu_detach_event(struct pmu *pmu, struct perf_event *event, + struct perf_event_context *ctx) +{ + /* + * De-schedule the event and mark it REVOKED. + */ + perf_event_exit_event(event, ctx, true); + + /* + * All _free_event() bits that rely on event->pmu: + * + * Notably, perf_mmap() relies on the ordering here. + */ + scoped_guard (mutex, &event->mmap_mutex) { + WARN_ON_ONCE(pmu->event_unmapped); + /* + * Mostly an empty lock sequence, such that perf_mmap(), which + * relies on mmap_mutex, is sure to observe the state change. + */ + } + + perf_event_free_bpf_prog(event); + perf_free_addr_filters(event); + + if (event->destroy) { + event->destroy(event); + event->destroy =3D NULL; + } + + if (event->pmu_ctx) { + put_pmu_ctx(event->pmu_ctx); + event->pmu_ctx =3D NULL; + } + + exclusive_event_destroy(event); + module_put(pmu->module); + + event->pmu =3D NULL; /* force fault instead of UAF */ +} + +static void pmu_detach_event(struct pmu *pmu, struct perf_event *event) +{ + struct perf_event_context *ctx; + + ctx =3D perf_event_ctx_lock(event); + __pmu_detach_event(pmu, event, ctx); + perf_event_ctx_unlock(event, ctx); + + scoped_guard (spinlock, &pmu->events_lock) + list_del(&event->pmu_list); +} + +static struct perf_event *pmu_get_event(struct pmu *pmu) +{ + struct perf_event *event; + + guard(spinlock)(&pmu->events_lock); + list_for_each_entry(event, &pmu->events, pmu_list) { + if (atomic_long_inc_not_zero(&event->refcount)) + return event; + } + + return NULL; +} + +static bool pmu_empty(struct pmu *pmu) +{ + guard(spinlock)(&pmu->events_lock); + return list_empty(&pmu->events); +} + +static void pmu_detach_events(struct pmu *pmu) +{ + struct perf_event *event; + + for (;;) { + event =3D pmu_get_event(pmu); + if (!event) + break; + + pmu_detach_event(pmu, event); + put_event(event); + } + + /* + * wait for pending _free_event()s + */ + wait_var_event(pmu, pmu_empty(pmu)); +} + +int perf_pmu_unregister(struct pmu *pmu) { scoped_guard (mutex, &pmus_lock) { + if (!idr_cmpxchg(&pmu_idr, pmu->type, pmu, NULL)) + return -EINVAL; + list_del_rcu(&pmu->entry); - idr_remove(&pmu_idr, pmu->type); } =20 /* * We dereference the pmu list under both SRCU and regular RCU, so * synchronize against both of those. + * + * Notably, the entirety of event creation, from perf_init_event() + * (which will now fail, because of the above) until + * perf_install_in_context() should be under SRCU such that + * this synchronizes against event creation. This avoids trying to + * detach events that are not fully formed. */ synchronize_srcu(&pmus_srcu); synchronize_rcu(); =20 + if (pmu->event_unmapped && !pmu_empty(pmu)) { + /* + * Can't force remove events when pmu::event_unmapped() + * is used in perf_mmap_close(). + */ + guard(mutex)(&pmus_lock); + idr_cmpxchg(&pmu_idr, pmu->type, NULL, pmu); + list_add_rcu(&pmu->entry, &pmus); + return -EBUSY; + } + + scoped_guard (mutex, &pmus_lock) + idr_remove(&pmu_idr, pmu->type); + + /* + * PMU is removed from the pmus list, so no new events will + * be created, now take care of the existing ones. + */ + pmu_detach_events(pmu); + + /* + * PMU is unused, make it go away. + */ perf_pmu_free(pmu); + return 0; } EXPORT_SYMBOL_GPL(perf_pmu_unregister); =20 @@ -12107,7 +12281,7 @@ static struct pmu *perf_init_event(struc struct pmu *pmu; int type, ret; =20 - guard(srcu)(&pmus_srcu); + guard(srcu)(&pmus_srcu); /* pmu idr/list access */ =20 /* * Save original type before calling pmu->event_init() since certain @@ -12331,6 +12505,7 @@ perf_event_alloc(struct perf_event_attr INIT_LIST_HEAD(&event->active_entry); INIT_LIST_HEAD(&event->addr_filters.list); INIT_HLIST_NODE(&event->hlist_entry); + INIT_LIST_HEAD(&event->pmu_list); =20 =20 init_waitqueue_head(&event->waitq); @@ -12498,6 +12673,13 @@ perf_event_alloc(struct perf_event_attr /* symmetric to unaccount_event() in _free_event() */ account_event(event); =20 + /* + * Event creation should be under SRCU, see perf_pmu_unregister(). + */ + lockdep_assert_held(&pmus_srcu); + scoped_guard (spinlock, &pmu->events_lock) + list_add(&event->pmu_list, &pmu->events); + return_ptr(event); } =20 @@ -12697,6 +12879,9 @@ perf_event_set_output(struct perf_event goto unlock; =20 if (output_event) { + if (output_event->state <=3D PERF_EVENT_STATE_REVOKED) + goto unlock; + /* get the rb we want to redirect to */ rb =3D ring_buffer_get(output_event); if (!rb) @@ -12878,6 +13063,11 @@ SYSCALL_DEFINE5(perf_event_open, if (event_fd < 0) return event_fd; =20 + /* + * Event creation should be under SRCU, see perf_pmu_unregister(). + */ + guard(srcu)(&pmus_srcu); + CLASS(fd, group)(group_fd); // group_fd =3D=3D -1 =3D> empty if (group_fd !=3D -1) { if (!is_perf_file(group)) { @@ -12885,6 +13075,10 @@ SYSCALL_DEFINE5(perf_event_open, goto err_fd; } group_leader =3D fd_file(group)->private_data; + if (group_leader->state <=3D PERF_EVENT_STATE_REVOKED) { + err =3D -ENODEV; + goto err_fd; + } if (flags & PERF_FLAG_FD_OUTPUT) output_event =3D group_leader; if (flags & PERF_FLAG_FD_NO_GROUP) @@ -13218,6 +13412,11 @@ perf_event_create_kernel_counter(struct if (attr->aux_output || attr->aux_action) return ERR_PTR(-EINVAL); =20 + /* + * Event creation should be under SRCU, see perf_pmu_unregister(). + */ + guard(srcu)(&pmus_srcu); + event =3D perf_event_alloc(attr, cpu, task, NULL, NULL, overflow_handler, context, -1); if (IS_ERR(event)) { @@ -13429,10 +13628,11 @@ static void sync_child_event(struct perf } =20 static void -perf_event_exit_event(struct perf_event *event, struct perf_event_context = *ctx) +perf_event_exit_event(struct perf_event *event, + struct perf_event_context *ctx, bool revoke) { struct perf_event *parent_event =3D event->parent; - unsigned long detach_flags =3D 0; + unsigned long detach_flags =3D DETACH_EXIT; =20 if (parent_event) { /* @@ -13447,16 +13647,14 @@ perf_event_exit_event(struct perf_event * Do destroy all inherited groups, we don't care about those * and being thorough is better. */ - detach_flags =3D DETACH_GROUP | DETACH_CHILD; + detach_flags |=3D DETACH_GROUP | DETACH_CHILD; mutex_lock(&parent_event->child_mutex); } =20 - perf_remove_from_context(event, detach_flags); + if (revoke) + detach_flags |=3D DETACH_GROUP | DETACH_REVOKE; =20 - raw_spin_lock_irq(&ctx->lock); - if (event->state > PERF_EVENT_STATE_EXIT) - perf_event_set_state(event, PERF_EVENT_STATE_EXIT); - raw_spin_unlock_irq(&ctx->lock); + perf_remove_from_context(event, detach_flags); =20 /* * Child events can be freed. @@ -13467,7 +13665,13 @@ perf_event_exit_event(struct perf_event * Kick perf_poll() for is_event_hup(); */ perf_event_wakeup(parent_event); - free_event(event); + /* + * pmu_detach_event() will have an extra refcount. + */ + if (revoke) + put_event(event); + else + free_event(event); put_event(parent_event); return; } @@ -13532,7 +13736,7 @@ static void perf_event_exit_task_context perf_event_task(child, child_ctx, 0); =20 list_for_each_entry_safe(child_event, next, &child_ctx->event_list, event= _entry) - perf_event_exit_event(child_event, child_ctx); + perf_event_exit_event(child_event, child_ctx, false); =20 mutex_unlock(&child_ctx->mutex); =20 @@ -13722,6 +13926,14 @@ inherit_event(struct perf_event *parent_ if (parent_event->parent) parent_event =3D parent_event->parent; =20 + if (parent_event->state <=3D PERF_EVENT_STATE_REVOKED) + return NULL; + + /* + * Event creation should be under SRCU, see perf_pmu_unregister(). + */ + guard(srcu)(&pmus_srcu); + child_event =3D perf_event_alloc(&parent_event->attr, parent_event->cpu, child,