From nobody Thu Nov 28 11:26:31 2024 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4A5EA1C5796 for ; Tue, 1 Oct 2024 14:25:33 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.129.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727792735; cv=none; b=jSSy75BBO7WWI7i60zCrQrPuWZDTFIsbqWEgAk+EpQ32gMn9KsOrytXT6XZb9/y+utApQyETHdrOj4MWy5gFPkQEnde4oN9m4gDGnRpRoi4/1K2v+MP5rnV2MB9h7oQPfr2gzcNYHK0nCF8nqePxeOGXzvPXvoJLHeM/4XOKfbQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727792735; c=relaxed/simple; bh=gs8sCIWGVClZmQGRAjTinJEKiYPTNdveadZOaLKKc6A=; h=Date:From:To:Cc:Subject:Message-ID:MIME-Version:Content-Type: Content-Disposition:In-Reply-To; b=agsFramjr1FcrRncnQMM0EKpjALlgv4GBa+16G1xU4ITTG5Cqpw3Ghfkenr2ameYNcgK1yQif1U9wYyu2mnVPRwC1GdhhhY4lxKtm+y/ZiLf30yDN+8ixjb7G6JbCQZdMExuZJS9+ZEkZ1zfBQNY4/VXaHvzCShuaUUEfKFHqqM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=a2o67cWo; arc=none smtp.client-ip=170.10.129.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="a2o67cWo" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1727792732; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to; bh=OakJ4msEqx8tZ0UDaJqBnUchDvgKLocyq9Ke5f2R7+Y=; b=a2o67cWoBhDlD/6DH3otgOOVdxBd4PpE21Ysi3zCqGjJIQSB9CxH7+fPKYc/QlAdGXwBd7 RzTNRSVbT6roAq8YPz/HDQAZ8msr74x7BJnQW82zWd5JiP/3vSFY5c3VH7VPPF+1jwUByV j1OpKJdyeZh8q7X5DmFb7lR4idocUo8= Received: from mx-prod-mc-04.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-627-KDe-G_QzMwanQAZpMA0bDA-1; Tue, 01 Oct 2024 10:25:27 -0400 X-MC-Unique: KDe-G_QzMwanQAZpMA0bDA-1 Received: from mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com (unknown [10.30.177.4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-04.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id E7A321916BD7; Tue, 1 Oct 2024 14:25:23 +0000 (UTC) Received: from dhcp-27-174.brq.redhat.com (unknown [10.45.225.88]) by mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with SMTP id B34183011D3D; Tue, 1 Oct 2024 14:25:13 +0000 (UTC) Received: by dhcp-27-174.brq.redhat.com (nbSMTP-1.00) for uid 1000 oleg@redhat.com; Tue, 1 Oct 2024 16:25:02 +0200 (CEST) Date: Tue, 1 Oct 2024 16:24:59 +0200 From: Oleg Nesterov To: Andrii Nakryiko , Jiri Olsa , Masami Hiramatsu , Peter Zijlstra Cc: Liao Chang , linux-kernel@vger.kernel.org, linux-trace-kernel@vger.kernel.org Subject: [PATCH 1/2] uprobes: kill xol_area->slot_count Message-ID: <20241001142458.GA13629@redhat.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <20241001142416.GA13599@redhat.com> User-Agent: Mutt/1.5.24 (2015-08-30) X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.4 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Add the new helper, xol_get_slot_nr() which does find_first_zero_bit() + test_and_set_bit(). xol_take_insn_slot() can wait for the "xol_get_slot_nr() < UINSNS_PER_PAGE" event instead of "area->slot_count < UINSNS_PER_PAGE". So we can kill area->slot_count and avoid atomic_inc() + atomic_dec(), this simplifies the code and can slightly improve the performance. Signed-off-by: Oleg Nesterov Acked-by: Andrii Nakryiko --- kernel/events/uprobes.c | 29 +++++++++++++++-------------- 1 file changed, 15 insertions(+), 14 deletions(-) diff --git a/kernel/events/uprobes.c b/kernel/events/uprobes.c index 20c58b6ee1ad..38d91e10d455 100644 --- a/kernel/events/uprobes.c +++ b/kernel/events/uprobes.c @@ -100,7 +100,6 @@ static LIST_HEAD(delayed_uprobe_list); */ struct xol_area { wait_queue_head_t wq; /* if all slots are busy */ - atomic_t slot_count; /* number of in-use slots */ unsigned long *bitmap; /* 0 =3D free slot */ =20 struct page *page; @@ -1559,7 +1558,6 @@ static struct xol_area *__create_xol_area(unsigned lo= ng vaddr) init_waitqueue_head(&area->wq); /* Reserve the 1st slot for get_trampoline_vaddr() */ set_bit(0, area->bitmap); - atomic_set(&area->slot_count, 1); insns =3D arch_uprobe_trampoline(&insns_size); arch_uprobe_copy_ixol(area->page, 0, insns, insns_size); =20 @@ -1632,24 +1630,28 @@ void uprobe_dup_mmap(struct mm_struct *oldmm, struc= t mm_struct *newmm) } } =20 +static unsigned long xol_get_slot_nr(struct xol_area *area) +{ + unsigned long slot_nr; + + slot_nr =3D find_first_zero_bit(area->bitmap, UINSNS_PER_PAGE); + if (slot_nr < UINSNS_PER_PAGE) { + if (!test_and_set_bit(slot_nr, area->bitmap)) + return slot_nr; + } + + return UINSNS_PER_PAGE; +} + /* * - search for a free slot. */ static unsigned long xol_take_insn_slot(struct xol_area *area) { - unsigned int slot_nr; + unsigned long slot_nr; =20 - for (;;) { - slot_nr =3D find_first_zero_bit(area->bitmap, UINSNS_PER_PAGE); - if (slot_nr < UINSNS_PER_PAGE) { - if (!test_and_set_bit(slot_nr, area->bitmap)) - break; - continue; - } - wait_event(area->wq, (atomic_read(&area->slot_count) < UINSNS_PER_PAGE)); - } + wait_event(area->wq, (slot_nr =3D xol_get_slot_nr(area)) < UINSNS_PER_PAG= E); =20 - atomic_inc(&area->slot_count); return area->vaddr + slot_nr * UPROBE_XOL_SLOT_BYTES; } =20 @@ -1685,7 +1687,6 @@ static void xol_free_insn_slot(struct uprobe_task *ut= ask) =20 slot_nr =3D offset / UPROBE_XOL_SLOT_BYTES; clear_bit(slot_nr, area->bitmap); - atomic_dec(&area->slot_count); smp_mb__after_atomic(); /* pairs with prepare_to_wait() */ if (waitqueue_active(&area->wq)) wake_up(&area->wq); --=20 2.25.1.362.g51ebf55 From nobody Thu Nov 28 11:26:31 2024 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4F07319D06E for ; Tue, 1 Oct 2024 14:25:32 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.133.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727792734; cv=none; b=EfZ9rLSLmFqYQ/66DYXuGyLnvpWIIPiHWzOmNE6NBXK4OJSsj9uFl0EKueSAeb9fdHRR9yp8Tv2Xwr1GAxYLSD/1DOsczT26244J6cHGhCX1SX4Z6UaoAX0K5+BKGmMaovb25C9nQvr+S+Mo3Sxwg7xttVnABD8JehwMKUIJUwY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727792734; c=relaxed/simple; bh=kB/vCOQVk1IMdjsq58HX7OYOPky84hE7fvfuCRDEbaE=; h=Date:From:To:Cc:Subject:Message-ID:MIME-Version:Content-Type: Content-Disposition:In-Reply-To; b=c9zUfTyyFJMcDwD98RbQ6AmH8tcGXPgJRvoRZfltv+4C8dS9IzwxxAL7vJuQjfuJkzQHUmrZJiSGZGyPuSVeDdJoOq0TvLNkZqsSrr93sdI6JacvDui0PyVBwbEiSy0yzwLqhxwdvFQxERrrEYivGFR3cU2pe/8fs1YHAO650q0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=KQIc2GGB; arc=none smtp.client-ip=170.10.133.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="KQIc2GGB" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1727792732; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to; bh=CAhIXQw4AKu0nqECcGYlfSjZV7z+K8tHHRI3flDnkjE=; b=KQIc2GGBRprM2B+2VfPKuvlhlMUUoOHvkeY5cl7w7oI7wnCh0msCcUgmaPyxYSAzOSRcQb HTW6+iRHRpuh6PS/rPELk16CaXMz2/2dH2/6ymXt8tLp+HqCWlHd2/F3bKfOoUncj7KkNu ul04jRvPm55xCkaEz+yvdKfDm58ocEE= Received: from mx-prod-mc-03.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-34-3yGILTEaMIiaAB3D8evZpg-1; Tue, 01 Oct 2024 10:25:26 -0400 X-MC-Unique: 3yGILTEaMIiaAB3D8evZpg-1 Received: from mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com (unknown [10.30.177.4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-03.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id BF7CA18D9F11; Tue, 1 Oct 2024 14:25:23 +0000 (UTC) Received: from dhcp-27-174.brq.redhat.com (unknown [10.45.225.88]) by mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with SMTP id 861EA301228D; Tue, 1 Oct 2024 14:25:18 +0000 (UTC) Received: by dhcp-27-174.brq.redhat.com (nbSMTP-1.00) for uid 1000 oleg@redhat.com; Tue, 1 Oct 2024 16:25:07 +0200 (CEST) Date: Tue, 1 Oct 2024 16:25:03 +0200 From: Oleg Nesterov To: Andrii Nakryiko , Jiri Olsa , Masami Hiramatsu , Peter Zijlstra Cc: Liao Chang , linux-kernel@vger.kernel.org, linux-trace-kernel@vger.kernel.org Subject: [PATCH 2/2] uprobes: fold xol_take_insn_slot() into xol_get_insn_slot() Message-ID: <20241001142503.GA13633@redhat.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <20241001142416.GA13599@redhat.com> User-Agent: Mutt/1.5.24 (2015-08-30) X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.4 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" After the previous change xol_take_insn_slot() becomes trivial, kill it. Signed-off-by: Oleg Nesterov Acked-by: Andrii Nakryiko --- kernel/events/uprobes.c | 17 ++++------------- 1 file changed, 4 insertions(+), 13 deletions(-) diff --git a/kernel/events/uprobes.c b/kernel/events/uprobes.c index 38d91e10d455..40ecab0971ff 100644 --- a/kernel/events/uprobes.c +++ b/kernel/events/uprobes.c @@ -1643,29 +1643,20 @@ static unsigned long xol_get_slot_nr(struct xol_are= a *area) return UINSNS_PER_PAGE; } =20 -/* - * - search for a free slot. - */ -static unsigned long xol_take_insn_slot(struct xol_area *area) -{ - unsigned long slot_nr; - - wait_event(area->wq, (slot_nr =3D xol_get_slot_nr(area)) < UINSNS_PER_PAG= E); - - return area->vaddr + slot_nr * UPROBE_XOL_SLOT_BYTES; -} - /* * xol_get_insn_slot - allocate a slot for xol. */ static bool xol_get_insn_slot(struct uprobe *uprobe, struct uprobe_task *u= task) { struct xol_area *area =3D get_xol_area(); + unsigned long slot_nr; =20 if (!area) return false; =20 - utask->xol_vaddr =3D xol_take_insn_slot(area); + wait_event(area->wq, (slot_nr =3D xol_get_slot_nr(area)) < UINSNS_PER_PAG= E); + + utask->xol_vaddr =3D area->vaddr + slot_nr * UPROBE_XOL_SLOT_BYTES; arch_uprobe_copy_ixol(area->page, utask->xol_vaddr, &uprobe->arch.ixol, sizeof(uprobe->arch.ixol)); return true; --=20 2.25.1.362.g51ebf55