From nobody Wed Dec 17 17:57:50 2025 Received: from galois.linutronix.de (Galois.linutronix.de [193.142.43.55]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B3D6621771F; Mon, 9 Dec 2024 10:32:20 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=193.142.43.55 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1733740342; cv=none; b=ptmuKRVg9aukKz5F+XihLjk9MkXhPdS+hntqDx41z7Lggits73PNVnZyOtYsy3prSKIWU8lKuxB+bhwADpEgV3r4DzTLNkDG3tMQqWYDaasN6c9alUuZTzxsgOBbEhKsFNQoSu6PPeUqOJpdcuzkVi48s/xEgTAaJN7Qm9/jdh8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1733740342; c=relaxed/simple; bh=BAcGImtkq4rhzU9nEcLugGTLBzdtnoxtOKKiySB6v0M=; h=Date:From:To:Subject:Cc:In-Reply-To:References:MIME-Version: Message-ID:Content-Type; b=UFlNJ0ZoheC6LexcxZjfFjmBkYTifZWZIREwvoEc5nX8uF7hS48trb5tyHfN/4vDK4lUs9FnjUazhhAlf4W3BohBiixZ3HJ8dA9u07EDQNposJsDhWohRBXOBOdR0N8mhgdXQj16lYWgJYOKd4QIE2fu7GBqkuPDm1qvitBgJZ0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linutronix.de; spf=pass smtp.mailfrom=linutronix.de; dkim=pass (2048-bit key) header.d=linutronix.de header.i=@linutronix.de header.b=TKnG7n+2; dkim=permerror (0-bit key) header.d=linutronix.de header.i=@linutronix.de header.b=FWgxZ65r; arc=none smtp.client-ip=193.142.43.55 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linutronix.de Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linutronix.de Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="TKnG7n+2"; dkim=permerror (0-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="FWgxZ65r" Date: Mon, 09 Dec 2024 10:32:12 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1733740333; h=from:from:sender:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=3Ceb1xH+Rb+AT+GAWSdRrfbN7bhMvN2NPjXdmasqi9s=; b=TKnG7n+2wvoOk5tpBvmztvXgo8sdpU0K7XsPUBH/O5+//u29BLrhwiIwSAYoLskQrRZL4O X63rHhtwejadQj87ETKYG+2IsTkmqQ98r2FJb2ppMpxGkP8d70S7dvzEM4zx+eNV+frEnt FJNMVzNxT1D0H8lNwnsjMJOWNRo3sQjb3y48ACSGKM8M61HWX17yiwdAcxhrJaRH40WTAN yAqXD4t5/0KrvXKyaDeHVu3Wpavee6fGhk9SFZys/UJhG6ghLApfXA3tB0Cv2BTrAiAwbK gmwVjL/nswNR1CaBlPjSCD6XCkmVAtUfUiE8QPH8sN1s97PUGgMBAlX3nZ7WnQ== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1733740333; h=from:from:sender:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=3Ceb1xH+Rb+AT+GAWSdRrfbN7bhMvN2NPjXdmasqi9s=; b=FWgxZ65rRD9czQYoNxHwLRjeSoC6hQ3scY5AO9vT4hbypYz48DMqZ+EJLsHuknGlt9y0VJ vdCzzrCfoQGXKNBg== From: "tip-bot2 for Andrii Nakryiko" Sender: tip-bot2@linutronix.de Reply-to: linux-kernel@vger.kernel.org To: linux-tip-commits@vger.kernel.org Subject: [tip: perf/core] uprobes: Simplify session consumer tracking Cc: Andrii Nakryiko , Ingo Molnar , Masami Hiramatsu , Peter Zijlstra , Oleg Nesterov , x86@kernel.org, linux-kernel@vger.kernel.org In-Reply-To: <20241206002417.3295533-2-andrii@kernel.org> References: <20241206002417.3295533-2-andrii@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Message-ID: <173374033240.412.4008851577756475967.tip-bot2@tip-bot2> Robot-ID: Robot-Unsubscribe: Contact to get blacklisted from these emails Precedence: bulk Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable The following commit has been merged into the perf/core branch of tip: Commit-ID: f6299532535e62171f545ac59cb2b441ba8786fc Gitweb: https://git.kernel.org/tip/f6299532535e62171f545ac59cb2b441b= a8786fc Author: Andrii Nakryiko AuthorDate: Thu, 05 Dec 2024 16:24:14 -08:00 Committer: Peter Zijlstra CommitterDate: Mon, 09 Dec 2024 11:18:08 +01:00 uprobes: Simplify session consumer tracking In practice, each return_instance will typically contain either zero or one return_consumer, depending on whether it has any uprobe session consumer attached or not. It's highly unlikely that more than one uprobe session consumers will be attached to any given uprobe, so there is no need to optimize for that case. But the way we currently do memory allocation and accounting is by pre-allocating the space for 4 session consumers in contiguous block of memory next to struct return_instance fixed part. This is unnecessarily wasteful. This patch changes this to keep struct return_instance fixed-sized with one pre-allocated return_consumer, while (in a highly unlikely scenario) allowing for more session consumers in a separate dynamically allocated and reallocated array. We also simplify accounting a bit by not maintaining a separate temporary capacity for consumers array, and, instead, relying on krealloc() to be a no-op if underlying memory can accommodate a slightly bigger allocation (but again, it's very uncommon scenario to even have to do this reallocation). All this gets rid of ri_size(), simplifies push_consumer() and removes confusing ri->consumers_cnt re-assignment, while containing this singular preallocated consumer logic contained within a few simple preexisting helpers. Having fixed-sized struct return_instance simplifies and speeds up return_instance reuse that we ultimately add later in this patch set, see follow up patches. Signed-off-by: Andrii Nakryiko Signed-off-by: Ingo Molnar Cc: Masami Hiramatsu Cc: Peter Zijlstra Cc: Oleg Nesterov Link: https://lore.kernel.org/r/20241206002417.3295533-2-andrii@kernel.org --- include/linux/uprobes.h | 10 ++++-- kernel/events/uprobes.c | 72 ++++++++++++++++++++-------------------- 2 files changed, 45 insertions(+), 37 deletions(-) diff --git a/include/linux/uprobes.h b/include/linux/uprobes.h index e0a4c20..1d44997 100644 --- a/include/linux/uprobes.h +++ b/include/linux/uprobes.h @@ -154,12 +154,18 @@ struct return_instance { unsigned long stack; /* stack pointer */ unsigned long orig_ret_vaddr; /* original return address */ bool chained; /* true, if instance is nested */ - int consumers_cnt; + int cons_cnt; /* total number of session consumers */ =20 struct return_instance *next; /* keep as stack */ struct rcu_head rcu; =20 - struct return_consumer consumers[] __counted_by(consumers_cnt); + /* singular pre-allocated return_consumer instance for common case */ + struct return_consumer consumer; + /* + * extra return_consumer instances for rare cases of multiple session con= sumers, + * contains (cons_cnt - 1) elements + */ + struct return_consumer *extra_consumers; } ____cacheline_aligned; =20 enum rp_check { diff --git a/kernel/events/uprobes.c b/kernel/events/uprobes.c index daf4314..6beac52 100644 --- a/kernel/events/uprobes.c +++ b/kernel/events/uprobes.c @@ -1899,6 +1899,7 @@ static struct return_instance *free_ret_instance(stru= ct return_instance *ri, boo hprobe_finalize(&ri->hprobe, hstate); } =20 + kfree(ri->extra_consumers); kfree_rcu(ri, rcu); return next; } @@ -1974,32 +1975,34 @@ static struct uprobe_task *get_utask(void) return current->utask; } =20 -static size_t ri_size(int consumers_cnt) -{ - struct return_instance *ri; - - return sizeof(*ri) + sizeof(ri->consumers[0]) * consumers_cnt; -} - -#define DEF_CNT 4 - static struct return_instance *alloc_return_instance(void) { struct return_instance *ri; =20 - ri =3D kzalloc(ri_size(DEF_CNT), GFP_KERNEL); + ri =3D kzalloc(sizeof(*ri), GFP_KERNEL); if (!ri) return ZERO_SIZE_PTR; =20 - ri->consumers_cnt =3D DEF_CNT; return ri; } =20 static struct return_instance *dup_return_instance(struct return_instance = *old) { - size_t size =3D ri_size(old->consumers_cnt); + struct return_instance *ri; + + ri =3D kmemdup(old, sizeof(*ri), GFP_KERNEL); + + if (unlikely(old->cons_cnt > 1)) { + ri->extra_consumers =3D kmemdup(old->extra_consumers, + sizeof(ri->extra_consumers[0]) * (old->cons_cnt - 1), + GFP_KERNEL); + if (!ri->extra_consumers) { + kfree(ri); + return NULL; + } + } =20 - return kmemdup(old, size, GFP_KERNEL); + return ri; } =20 static int dup_utask(struct task_struct *t, struct uprobe_task *o_utask) @@ -2369,25 +2372,28 @@ static struct uprobe *find_active_uprobe_rcu(unsign= ed long bp_vaddr, int *is_swb return uprobe; } =20 -static struct return_instance* -push_consumer(struct return_instance *ri, int idx, __u64 id, __u64 cookie) +static struct return_instance *push_consumer(struct return_instance *ri, _= _u64 id, __u64 cookie) { + struct return_consumer *ric; + if (unlikely(ri =3D=3D ZERO_SIZE_PTR)) return ri; =20 - if (unlikely(idx >=3D ri->consumers_cnt)) { - struct return_instance *old_ri =3D ri; - - ri->consumers_cnt +=3D DEF_CNT; - ri =3D krealloc(old_ri, ri_size(old_ri->consumers_cnt), GFP_KERNEL); - if (!ri) { - kfree(old_ri); + if (unlikely(ri->cons_cnt > 0)) { + ric =3D krealloc(ri->extra_consumers, sizeof(*ric) * ri->cons_cnt, GFP_K= ERNEL); + if (!ric) { + kfree(ri->extra_consumers); + kfree_rcu(ri, rcu); return ZERO_SIZE_PTR; } + ri->extra_consumers =3D ric; } =20 - ri->consumers[idx].id =3D id; - ri->consumers[idx].cookie =3D cookie; + ric =3D likely(ri->cons_cnt =3D=3D 0) ? &ri->consumer : &ri->extra_consum= ers[ri->cons_cnt - 1]; + ric->id =3D id; + ric->cookie =3D cookie; + + ri->cons_cnt++; return ri; } =20 @@ -2395,14 +2401,17 @@ static struct return_consumer * return_consumer_find(struct return_instance *ri, int *iter, int id) { struct return_consumer *ric; - int idx =3D *iter; + int idx; =20 - for (ric =3D &ri->consumers[idx]; idx < ri->consumers_cnt; idx++, ric++) { + for (idx =3D *iter; idx < ri->cons_cnt; idx++) + { + ric =3D likely(idx =3D=3D 0) ? &ri->consumer : &ri->extra_consumers[idx = - 1]; if (ric->id =3D=3D id) { *iter =3D idx + 1; return ric; } } + return NULL; } =20 @@ -2416,7 +2425,6 @@ static void handler_chain(struct uprobe *uprobe, stru= ct pt_regs *regs) struct uprobe_consumer *uc; bool has_consumers =3D false, remove =3D true; struct return_instance *ri =3D NULL; - int push_idx =3D 0; =20 current->utask->auprobe =3D &uprobe->arch; =20 @@ -2441,18 +2449,12 @@ static void handler_chain(struct uprobe *uprobe, st= ruct pt_regs *regs) ri =3D alloc_return_instance(); =20 if (session) - ri =3D push_consumer(ri, push_idx++, uc->id, cookie); + ri =3D push_consumer(ri, uc->id, cookie); } current->utask->auprobe =3D NULL; =20 - if (!ZERO_OR_NULL_PTR(ri)) { - /* - * The push_idx value has the final number of return consumers, - * and ri->consumers_cnt has number of allocated consumers. - */ - ri->consumers_cnt =3D push_idx; + if (!ZERO_OR_NULL_PTR(ri)) prepare_uretprobe(uprobe, regs, ri); - } =20 if (remove && has_consumers) { down_read(&uprobe->register_rwsem);