From nobody Mon Jun 22 14:26:16 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id CE004C433F5 for ; Tue, 22 Mar 2022 12:10:04 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234666AbiCVMLa (ORCPT ); Tue, 22 Mar 2022 08:11:30 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38434 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234656AbiCVML0 (ORCPT ); Tue, 22 Mar 2022 08:11:26 -0400 Received: from mail-pj1-x102a.google.com (mail-pj1-x102a.google.com [IPv6:2607:f8b0:4864:20::102a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 190C783035 for ; Tue, 22 Mar 2022 05:09:53 -0700 (PDT) Received: by mail-pj1-x102a.google.com with SMTP id mj15-20020a17090b368f00b001c637aa358eso2484056pjb.0 for ; Tue, 22 Mar 2022 05:09:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20210112.gappssmtp.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=Tev5pfuhtnVPwxEGApA0jaXuM/+4Um3B3QdIHAHr3/o=; b=kaUArhFMQU/+L16rrX9b2jwV0DFyg/7KKILiG0ZcxIQskBESNFBLP7M3gFx+uVlCVj avO9dxMpPgatOr+Ffo/TR0MOF8c2/5UegzIOcLKGj4qT7X4XxfcxjMyV1cBum9bQkl8q mFoBfBvsnTLsMaiVBg/lTPhSqNiTBW97Po5pEJ1u95yUdeZzyXm7yYS18y497xYfAiF8 8ZAhj59tvq+uulTJk6AptR9ACeoWfvJfZBpxv/FBmY+O8WMSgYU/lFLS/QHwHMk9zLD7 lKsdw3/EpcD/2PpntQ4j++GyllOtfUUGldnfK+knqV2+iJUo6gD9H22KzjOun/YSri9R bSlA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=Tev5pfuhtnVPwxEGApA0jaXuM/+4Um3B3QdIHAHr3/o=; b=A8EoGgrHhosCd6JpUSP33LKhs01acEPORY73SmkU5aT8kaUR7S4sfyTOLkfQlTm5Ve Ntq1nmYGxRd397rqdxtsbdCbQJxKK5rujw9uXAPBNOhWzCiW3FOtzmcEXDj3Mnb3JhWs CsEDOhOI6O3sl0mDm9FMpZIBtWZiCed4frTUg5uU2M/umg1VO2s8ltH4lTQJC7jZLcUv xdANpLQU8g8aXkiisnxmBy0uuKPcRgDPSlE1naWEtfH65vuCdkDo7kq63ryBxbgAKox5 ggdTkt8YimYOwPYKdAki/dTLh81wfQ2Rb5RvFYj99lxOODgOtD0o7wtR1b+9hPHHFERq 9vCA== X-Gm-Message-State: AOAM530mtFSsjiY3+FzBk+q5kcqIFPLMI1hDM+siYg22KGFraQtHP1z8 wIJowcbK5h5MnuKOchUb1cSpaQ== X-Google-Smtp-Source: ABdhPJy1fYqWhk/nMQc7eASo1t7vayaZA8LoskEZQ75OPP6cBjFvwfhrjLO2mfLltmADfoi4BqIwSw== X-Received: by 2002:a17:90b:380d:b0:1c7:223:c0d with SMTP id mq13-20020a17090b380d00b001c702230c0dmr4792212pjb.94.1647950992975; Tue, 22 Mar 2022 05:09:52 -0700 (PDT) Received: from localhost.localdomain ([2409:8a28:e62:3990:75d0:8573:b73c:e6b4]) by smtp.gmail.com with ESMTPSA id q9-20020a056a00088900b004e03b051040sm24833850pfj.112.2022.03.22.05.09.45 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 22 Mar 2022 05:09:52 -0700 (PDT) From: Chengming Zhou To: peterz@infradead.org, mingo@redhat.com, acme@kernel.org, mark.rutland@arm.com, alexander.shishkin@linux.intel.com, jolsa@kernel.org, namhyung@kernel.org, eranian@google.com Cc: linux-perf-users@vger.kernel.org, linux-kernel@vger.kernel.org, duanxiongchun@bytedance.com, songmuchun@bytedance.com, Chengming Zhou Subject: [PATCH v2 1/6] perf/core: Fix incosistency between cgroup sched_out and sched_in Date: Tue, 22 Mar 2022 20:08:29 +0800 Message-Id: <20220322120834.98637-2-zhouchengming@bytedance.com> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20220322120834.98637-1-zhouchengming@bytedance.com> References: <20220322120834.98637-1-zhouchengming@bytedance.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" There is a race problem that can trigger WARN_ON_ONCE(cpuctx->cgrp) in perf_cgroup_switch(). CPU1 CPU2 (in context_switch) (attach running task) perf_cgroup_sched_out(prev, next) cgrp1 =3D=3D cgrp2 is True next->cgroups =3D cgrp3 perf_cgroup_attach() perf_cgroup_sched_in(prev, next) cgrp1 =3D=3D cgrp3 is False The commit a8d757ef076f ("perf events: Fix slow and broken cgroup context switch code") would save cpuctx switch out/in when the perf_cgroup of "prev" and "next" are the same. But perf_cgroup of task can change in concurrent with context_switch. If cgrp1 =3D=3D cgrp2 in sched_out(), cpuctx won't do switch out. Then task perf_cgroup changed cause cgrp1 !=3D cgrp2 in sched_in(), cpuctx will do switch in, and trigger WARN_ON_ONCE(cpuctx->cgrp). Even though perf_cgroup_switch will be synchronized as the context switch disables the interrupt, it still can see the task->cgroups is changing in the middle. So this patch combine perf_cgroup_sched_in() into perf_cgroup_sched_out(), rename to perf_cgroup_sched_switch(), to fix the incosistency between cgroup sched_out and sched_in. Fixes: a8d757ef076f ("perf events: Fix slow and broken cgroup context switc= h code") Signed-off-by: Chengming Zhou --- kernel/events/core.c | 101 ++++++++++--------------------------------- 1 file changed, 24 insertions(+), 77 deletions(-) diff --git a/kernel/events/core.c b/kernel/events/core.c index 6859229497b1..8b5cf2aedfe6 100644 --- a/kernel/events/core.c +++ b/kernel/events/core.c @@ -828,16 +828,10 @@ perf_cgroup_set_timestamp(struct task_struct *task, =20 static DEFINE_PER_CPU(struct list_head, cgrp_cpuctx_list); =20 -#define PERF_CGROUP_SWOUT 0x1 /* cgroup switch out every event */ -#define PERF_CGROUP_SWIN 0x2 /* cgroup switch in events based on task */ - /* * reschedule events based on the cgroup constraint of task. - * - * mode SWOUT : schedule out everything - * mode SWIN : schedule in based on cgroup for next */ -static void perf_cgroup_switch(struct task_struct *task, int mode) +static void perf_cgroup_switch(struct task_struct *task) { struct perf_cpu_context *cpuctx, *tmp; struct list_head *list; @@ -856,28 +850,22 @@ static void perf_cgroup_switch(struct task_struct *ta= sk, int mode) perf_ctx_lock(cpuctx, cpuctx->task_ctx); perf_pmu_disable(cpuctx->ctx.pmu); =20 - if (mode & PERF_CGROUP_SWOUT) { - cpu_ctx_sched_out(cpuctx, EVENT_ALL); - /* - * must not be done before ctxswout due - * to event_filter_match() in event_sched_out() - */ - cpuctx->cgrp =3D NULL; - } + cpu_ctx_sched_out(cpuctx, EVENT_ALL); + /* + * must not be done before ctxswout due + * to event_filter_match() in event_sched_out() + */ + cpuctx->cgrp =3D perf_cgroup_from_task(task, + &cpuctx->ctx); + /* + * set cgrp before ctxsw in to allow + * event_filter_match() to not have to pass + * task around + * we pass the cpuctx->ctx to perf_cgroup_from_task() + * because cgroup events are only per-cpu + */ + cpu_ctx_sched_in(cpuctx, EVENT_ALL, task); =20 - if (mode & PERF_CGROUP_SWIN) { - WARN_ON_ONCE(cpuctx->cgrp); - /* - * set cgrp before ctxsw in to allow - * event_filter_match() to not have to pass - * task around - * we pass the cpuctx->ctx to perf_cgroup_from_task() - * because cgorup events are only per-cpu - */ - cpuctx->cgrp =3D perf_cgroup_from_task(task, - &cpuctx->ctx); - cpu_ctx_sched_in(cpuctx, EVENT_ALL, task); - } perf_pmu_enable(cpuctx->ctx.pmu); perf_ctx_unlock(cpuctx, cpuctx->task_ctx); } @@ -885,8 +873,8 @@ static void perf_cgroup_switch(struct task_struct *task= , int mode) local_irq_restore(flags); } =20 -static inline void perf_cgroup_sched_out(struct task_struct *task, - struct task_struct *next) +static inline void perf_cgroup_sched_switch(struct task_struct *task, + struct task_struct *next) { struct perf_cgroup *cgrp1; struct perf_cgroup *cgrp2 =3D NULL; @@ -906,33 +894,7 @@ static inline void perf_cgroup_sched_out(struct task_s= truct *task, * do no touch the cgroup events. */ if (cgrp1 !=3D cgrp2) - perf_cgroup_switch(task, PERF_CGROUP_SWOUT); - - rcu_read_unlock(); -} - -static inline void perf_cgroup_sched_in(struct task_struct *prev, - struct task_struct *task) -{ - struct perf_cgroup *cgrp1; - struct perf_cgroup *cgrp2 =3D NULL; - - rcu_read_lock(); - /* - * we come here when we know perf_cgroup_events > 0 - * we do not need to pass the ctx here because we know - * we are holding the rcu lock - */ - cgrp1 =3D perf_cgroup_from_task(task, NULL); - cgrp2 =3D perf_cgroup_from_task(prev, NULL); - - /* - * only need to schedule in cgroup events if we are changing - * cgroup during ctxsw. Cgroup events were not scheduled - * out of ctxsw out if that was not the case. - */ - if (cgrp1 !=3D cgrp2) - perf_cgroup_switch(task, PERF_CGROUP_SWIN); + perf_cgroup_switch(task); =20 rcu_read_unlock(); } @@ -1100,13 +1062,8 @@ static inline void update_cgrp_time_from_cpuctx(stru= ct perf_cpu_context *cpuctx, { } =20 -static inline void perf_cgroup_sched_out(struct task_struct *task, - struct task_struct *next) -{ -} - -static inline void perf_cgroup_sched_in(struct task_struct *prev, - struct task_struct *task) +static inline void perf_cgroup_sched_switch(struct task_struct *task, + struct task_struct *next) { } =20 @@ -1124,7 +1081,7 @@ perf_cgroup_set_timestamp(struct task_struct *task, } =20 static inline void -perf_cgroup_switch(struct task_struct *task, struct task_struct *next) +perf_cgroup_sched_switch(struct task_struct *task, struct task_struct *nex= t) { } =20 @@ -3668,7 +3625,7 @@ void __perf_event_task_sched_out(struct task_struct *= task, * cgroup event are system-wide mode only */ if (atomic_read(this_cpu_ptr(&perf_cgroup_events))) - perf_cgroup_sched_out(task, next); + perf_cgroup_sched_switch(task, next); } =20 /* @@ -3984,16 +3941,6 @@ void __perf_event_task_sched_in(struct task_struct *= prev, struct perf_event_context *ctx; int ctxn; =20 - /* - * If cgroup events exist on this CPU, then we need to check if we have - * to switch in PMU state; cgroup event are system-wide mode only. - * - * Since cgroup events are CPU events, we must schedule these in before - * we schedule in the task events. - */ - if (atomic_read(this_cpu_ptr(&perf_cgroup_events))) - perf_cgroup_sched_in(prev, task); - for_each_task_context_nr(ctxn) { ctx =3D task->perf_event_ctxp[ctxn]; if (likely(!ctx)) @@ -13564,7 +13511,7 @@ static int __perf_cgroup_move(void *info) { struct task_struct *task =3D info; rcu_read_lock(); - perf_cgroup_switch(task, PERF_CGROUP_SWOUT | PERF_CGROUP_SWIN); + perf_cgroup_switch(task); rcu_read_unlock(); return 0; } --=20 2.20.1 From nobody Mon Jun 22 14:26:16 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7410CC433F5 for ; Tue, 22 Mar 2022 12:10:11 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234701AbiCVMLg (ORCPT ); Tue, 22 Mar 2022 08:11:36 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38708 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234664AbiCVMLa (ORCPT ); Tue, 22 Mar 2022 08:11:30 -0400 Received: from mail-pf1-x42d.google.com (mail-pf1-x42d.google.com [IPv6:2607:f8b0:4864:20::42d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A428282D14 for ; Tue, 22 Mar 2022 05:10:02 -0700 (PDT) Received: by mail-pf1-x42d.google.com with SMTP id s8so17926883pfk.12 for ; Tue, 22 Mar 2022 05:10:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20210112.gappssmtp.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=aWAdje7WFQnGpq+Kzbn3GFOKshIOo6IyB5vnvj78tbI=; b=eOeCiva+YFC5KtCUmdL7oCrHAm0F5xnJHvxasgCVLhR2oKzsPlmqbCsN6WokyH2cPE tXIXGeR04d8WAfEPmG0Npz1hUgzLXWLcqaIglC1ZDXNWaSnNADdQZLBdUOESRTU/eLh5 uuONzqNv2NOmfZmLkQ2JSsksRj9VDmHnV3HULmOnkUkP64/eA7oQk4LTFQilSvrj7NYT ngxTCWNaR2BqyBX4prH/K49v5Tva17Vaq25gGteX1aZfgNhrAEvf8PBhSXSPm49OziaL dEDC1JR2fbTBaeowrAOYg4XB9Nw4SCxM8jetPxTXTHJt2iJpKVDdpD4oo26yMRz+D9wn RSfw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=aWAdje7WFQnGpq+Kzbn3GFOKshIOo6IyB5vnvj78tbI=; b=gAUrmr4A9+cR/aP3xcFAenDEjgdEp/Oa+dHXYf8bOqdZfsxSHJj5O226In/vzA5ieB x57BYLcC4jE3oRclFAEFObZ1DZG+ix0KehJ6kV9Tp/tjnYk5ftUD6CKSZIVe+Gn2jhBv 74I11p6kpJBbQDesQ2pq/ujiKUQY/gkznz5Pp1258uWw82J780GavkRIS8S6wINolmVE V/uHcZFsneSJEPVlzG4VRrsP3x/Rvf8HTX5T6NnhXERm4E6uuUG1q000dHRIWGwcRwxy HzmlFz4LlkcKWkgdUZM64kFhvGL5L222pg7+zZigo+qpFn/nzJWSLhQBqAlSmhAHifPq vRzQ== X-Gm-Message-State: AOAM5328/YeXgKinxLsd4x0rWENnCAucxaR7utbIbQZcm4ByOSBwBKep TLiPfEV2V4G2IsjVoMHtLFm0sQ== X-Google-Smtp-Source: ABdhPJx+oCjdIGMKD+idwjDEPk2ce0on0bR+XsYYx/UwuotY0mRWSSSEXnV/dWWYRThCqC4ZOy3qcQ== X-Received: by 2002:a62:f205:0:b0:4fa:8461:421f with SMTP id m5-20020a62f205000000b004fa8461421fmr15556436pfh.4.1647951001679; Tue, 22 Mar 2022 05:10:01 -0700 (PDT) Received: from localhost.localdomain ([2409:8a28:e62:3990:75d0:8573:b73c:e6b4]) by smtp.gmail.com with ESMTPSA id q9-20020a056a00088900b004e03b051040sm24833850pfj.112.2022.03.22.05.09.53 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 22 Mar 2022 05:10:00 -0700 (PDT) From: Chengming Zhou To: peterz@infradead.org, mingo@redhat.com, acme@kernel.org, mark.rutland@arm.com, alexander.shishkin@linux.intel.com, jolsa@kernel.org, namhyung@kernel.org, eranian@google.com Cc: linux-perf-users@vger.kernel.org, linux-kernel@vger.kernel.org, duanxiongchun@bytedance.com, songmuchun@bytedance.com, Chengming Zhou Subject: [PATCH v2 2/6] perf/core: Introduce percpu perf_cgroup Date: Tue, 22 Mar 2022 20:08:30 +0800 Message-Id: <20220322120834.98637-3-zhouchengming@bytedance.com> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20220322120834.98637-1-zhouchengming@bytedance.com> References: <20220322120834.98637-1-zhouchengming@bytedance.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Although we don't have incosistency problem any more, we can have other problem like: CPU1 CPU2 (in context_switch) (attach running task) prev->cgroups =3D cgrp2 perf_cgroup_sched_switch(prev, next) cgrp2 =3D=3D cgrp2 is True If perf_cgroup of prev task changes from cgrp1 to cgrp2, perf_cgroup_sched_switch() will skip perf_cgroup_switch(), so the CPU would still schedule the cgrp1 events, but we should schedule the cgrp2 events. The reason of this problem is that we shouldn't use the changeable prev->cgroups to decide whether skip perf_cgroup_switch(). This patch introduces a percpu perf_cgroup to cache the perf_cgroup that scheduled in cpuctxes, which later used to compare with the perf_cgroup of next task to decide whether skip perf_cgroup_switch(). Since the perf_cgroup_switch() can be called after the context switch, the cgroup events might be scheduled already. So we put the comparison of perf_cgroups in perf_cgroup_switch(), and delete the unused function perf_cgroup_sched_switch(). We must clear the percpu perf_cgroup cache when the last cgroup event disabled. Fixes: a8d757ef076f ("perf events: Fix slow and broken cgroup context switc= h code") Signed-off-by: Chengming Zhou --- kernel/events/core.c | 63 ++++++++++++++++---------------------------- 1 file changed, 22 insertions(+), 41 deletions(-) diff --git a/kernel/events/core.c b/kernel/events/core.c index 8b5cf2aedfe6..848a3bfa9513 100644 --- a/kernel/events/core.c +++ b/kernel/events/core.c @@ -826,6 +826,7 @@ perf_cgroup_set_timestamp(struct task_struct *task, } } =20 +static DEFINE_PER_CPU(struct perf_cgroup *, cpu_perf_cgroup); static DEFINE_PER_CPU(struct list_head, cgrp_cpuctx_list); =20 /* @@ -833,6 +834,7 @@ static DEFINE_PER_CPU(struct list_head, cgrp_cpuctx_lis= t); */ static void perf_cgroup_switch(struct task_struct *task) { + struct perf_cgroup *cgrp; struct perf_cpu_context *cpuctx, *tmp; struct list_head *list; unsigned long flags; @@ -843,11 +845,21 @@ static void perf_cgroup_switch(struct task_struct *ta= sk) */ local_irq_save(flags); =20 + cgrp =3D perf_cgroup_from_task(task, NULL); + if (cgrp =3D=3D __this_cpu_read(cpu_perf_cgroup)) + goto out; + + __this_cpu_write(cpu_perf_cgroup, cgrp); + list =3D this_cpu_ptr(&cgrp_cpuctx_list); list_for_each_entry_safe(cpuctx, tmp, list, cgrp_cpuctx_entry) { WARN_ON_ONCE(cpuctx->ctx.nr_cgroups =3D=3D 0); =20 perf_ctx_lock(cpuctx, cpuctx->task_ctx); + + if (cpuctx->cgrp =3D=3D cgrp) + continue; + perf_pmu_disable(cpuctx->ctx.pmu); =20 cpu_ctx_sched_out(cpuctx, EVENT_ALL); @@ -855,14 +867,11 @@ static void perf_cgroup_switch(struct task_struct *ta= sk) * must not be done before ctxswout due * to event_filter_match() in event_sched_out() */ - cpuctx->cgrp =3D perf_cgroup_from_task(task, - &cpuctx->ctx); + cpuctx->cgrp =3D cgrp; /* * set cgrp before ctxsw in to allow * event_filter_match() to not have to pass * task around - * we pass the cpuctx->ctx to perf_cgroup_from_task() - * because cgroup events are only per-cpu */ cpu_ctx_sched_in(cpuctx, EVENT_ALL, task); =20 @@ -870,35 +879,10 @@ static void perf_cgroup_switch(struct task_struct *ta= sk) perf_ctx_unlock(cpuctx, cpuctx->task_ctx); } =20 +out: local_irq_restore(flags); } =20 -static inline void perf_cgroup_sched_switch(struct task_struct *task, - struct task_struct *next) -{ - struct perf_cgroup *cgrp1; - struct perf_cgroup *cgrp2 =3D NULL; - - rcu_read_lock(); - /* - * we come here when we know perf_cgroup_events > 0 - * we do not need to pass the ctx here because we know - * we are holding the rcu lock - */ - cgrp1 =3D perf_cgroup_from_task(task, NULL); - cgrp2 =3D perf_cgroup_from_task(next, NULL); - - /* - * only schedule out current cgroup events if we know - * that we are switching to a different cgroup. Otherwise, - * do no touch the cgroup events. - */ - if (cgrp1 !=3D cgrp2) - perf_cgroup_switch(task); - - rcu_read_unlock(); -} - static int perf_cgroup_ensure_storage(struct perf_event *event, struct cgroup_subsys_state *css) { @@ -1035,6 +1019,9 @@ perf_cgroup_event_disable(struct perf_event *event, s= truct perf_event_context *c cpuctx->cgrp =3D NULL; =20 list_del(&cpuctx->cgrp_cpuctx_entry); + + if (list_empty(per_cpu_ptr(&cgrp_cpuctx_list, event->cpu))) + __this_cpu_write(cpu_perf_cgroup, NULL); } =20 #else /* !CONFIG_CGROUP_PERF */ @@ -1062,11 +1049,6 @@ static inline void update_cgrp_time_from_cpuctx(stru= ct perf_cpu_context *cpuctx, { } =20 -static inline void perf_cgroup_sched_switch(struct task_struct *task, - struct task_struct *next) -{ -} - static inline int perf_cgroup_connect(pid_t pid, struct perf_event *event, struct perf_event_attr *attr, struct perf_event *group_leader) @@ -1080,11 +1062,6 @@ perf_cgroup_set_timestamp(struct task_struct *task, { } =20 -static inline void -perf_cgroup_sched_switch(struct task_struct *task, struct task_struct *nex= t) -{ -} - static inline u64 perf_cgroup_event_time(struct perf_event *event) { return 0; @@ -1104,6 +1081,10 @@ static inline void perf_cgroup_event_disable(struct perf_event *event, struct perf_event_cont= ext *ctx) { } + +static void perf_cgroup_switch(struct task_struct *task) +{ +} #endif =20 /* @@ -3625,7 +3606,7 @@ void __perf_event_task_sched_out(struct task_struct *= task, * cgroup event are system-wide mode only */ if (atomic_read(this_cpu_ptr(&perf_cgroup_events))) - perf_cgroup_sched_switch(task, next); + perf_cgroup_switch(next); } =20 /* --=20 2.20.1 From nobody Mon Jun 22 14:26:16 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 391DAC4332F for ; Tue, 22 Mar 2022 12:10:24 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234656AbiCVMLs (ORCPT ); Tue, 22 Mar 2022 08:11:48 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39036 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234674AbiCVMLi (ORCPT ); Tue, 22 Mar 2022 08:11:38 -0400 Received: from mail-pf1-x42b.google.com (mail-pf1-x42b.google.com [IPv6:2607:f8b0:4864:20::42b]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 680DB84EFA for ; Tue, 22 Mar 2022 05:10:10 -0700 (PDT) Received: by mail-pf1-x42b.google.com with SMTP id p5so11859821pfo.5 for ; Tue, 22 Mar 2022 05:10:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20210112.gappssmtp.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=2AtraGYpELN2VSwkCm9o6LS6piIeW98lmVH595mjWM4=; b=hwJKy5xOXdff+V1T2YYlazuDSchivl9TpZ2ozJ1RZQs/ERArDQv0yeF6WPm1cJ542b d10soCXHyIn6EAHqTvFn1FP2T15MMb0u4qEb5WLEOa9gi0FUud9YBI/OSdWLSxoFIEE9 G+GljbjomWdOmzgJx17mpZYEzVy1kAvj/iBTF64811SizHRg3pAaioRGjI19SOS6GcNI m/LViYBxwCz5XpfggYDE8sDzSTGxczOQfSdbmE2aDTBIablPSyPNfabZa+PWRjq3nBrT WppiL6Ydz3eO10GKfEELDYW9mlSFYeJl7exkx9R6qybF9EqDj5annknKWxBw0v+fJTsw 7DDQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=2AtraGYpELN2VSwkCm9o6LS6piIeW98lmVH595mjWM4=; b=sCPdNuVmwKkTfd1JM5TnV0rcCTE2dMeS6iVLBNUTDemV/UIH6PbF2HsDvPRZ2jDntR 6EtYuq4mLOyrOlL5zSt5fMzNDEPiYN+Z0ORQKl3S3ZywbQoSgge/WEtmU3FB0bMg3XbX k6IzWGsq8sMON2t5JhGi7og4ZRD2HN2cQmStMkrdmpwH//2fS6IFL0h+vgWn0FDPQVT8 u9R+Q1FTrq6NRh37mAQRnExYPtoXt6xsM0D2gpXRKyHd6GsUMu9xwulteAu6OkFrg5tY FTo/4ya9dCRPV2MVATIOkbLmVt7Zf/8f/rEC213zMVQ9OB3pHGg0mXEpgLwzuXfJyoyZ UwWw== X-Gm-Message-State: AOAM530LE+ufjfjWnB1Yo48Hw0+Ut89RdTnWSmlKvaf63WiZcqIK7Ja2 pSlPeFI5h6CZ8Z6RaFcp2vrOug== X-Google-Smtp-Source: ABdhPJxRZK3FL6qRyIV7r/JYlN57sohicd6xyx5VBOQnwCSmiucPv4tlAasC8Zcv/0WezxpGHtdg0Q== X-Received: by 2002:a63:5547:0:b0:374:4ec0:bad0 with SMTP id f7-20020a635547000000b003744ec0bad0mr21603410pgm.169.1647951009847; Tue, 22 Mar 2022 05:10:09 -0700 (PDT) Received: from localhost.localdomain ([2409:8a28:e62:3990:75d0:8573:b73c:e6b4]) by smtp.gmail.com with ESMTPSA id q9-20020a056a00088900b004e03b051040sm24833850pfj.112.2022.03.22.05.10.02 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 22 Mar 2022 05:10:09 -0700 (PDT) From: Chengming Zhou To: peterz@infradead.org, mingo@redhat.com, acme@kernel.org, mark.rutland@arm.com, alexander.shishkin@linux.intel.com, jolsa@kernel.org, namhyung@kernel.org, eranian@google.com Cc: linux-perf-users@vger.kernel.org, linux-kernel@vger.kernel.org, duanxiongchun@bytedance.com, songmuchun@bytedance.com, Chengming Zhou Subject: [PATCH v2 3/6] perf/core: Don't pass task around when ctx sched in Date: Tue, 22 Mar 2022 20:08:31 +0800 Message-Id: <20220322120834.98637-4-zhouchengming@bytedance.com> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20220322120834.98637-1-zhouchengming@bytedance.com> References: <20220322120834.98637-1-zhouchengming@bytedance.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" The current code pass task around for ctx_sched_in(), only to get perf_cgroup of the task, then update the timestamp of it and its ancestors and set them to active. But we can use cpuctx->cgrp to get active perf_cgroup and its ancestors since cpuctx->cgrp has been set before ctx_sched_in(). This patch remove the task argument in ctx_sched_in() and cleanup related code. Signed-off-by: Chengming Zhou --- kernel/events/core.c | 54 ++++++++++++++++++++------------------------ 1 file changed, 24 insertions(+), 30 deletions(-) diff --git a/kernel/events/core.c b/kernel/events/core.c index 848a3bfa9513..e27c06628bad 100644 --- a/kernel/events/core.c +++ b/kernel/events/core.c @@ -574,8 +574,7 @@ static void cpu_ctx_sched_out(struct perf_cpu_context *= cpuctx, enum event_type_t event_type); =20 static void cpu_ctx_sched_in(struct perf_cpu_context *cpuctx, - enum event_type_t event_type, - struct task_struct *task); + enum event_type_t event_type); =20 static void update_context_time(struct perf_event_context *ctx); static u64 perf_event_time(struct perf_event *event); @@ -801,10 +800,10 @@ static inline void update_cgrp_time_from_event(struct= perf_event *event) } =20 static inline void -perf_cgroup_set_timestamp(struct task_struct *task, - struct perf_event_context *ctx) +perf_cgroup_set_timestamp(struct perf_cpu_context *cpuctx) { - struct perf_cgroup *cgrp; + struct perf_event_context *ctx =3D &cpuctx->ctx; + struct perf_cgroup *cgrp =3D cpuctx->cgrp; struct perf_cgroup_info *info; struct cgroup_subsys_state *css; =20 @@ -813,10 +812,10 @@ perf_cgroup_set_timestamp(struct task_struct *task, * ensure we do not access cgroup data * unless we have the cgroup pinned (css_get) */ - if (!task || !ctx->nr_cgroups) + if (!cgrp) return; =20 - cgrp =3D perf_cgroup_from_task(task, ctx); + WARN_ON_ONCE(!ctx->nr_cgroups); =20 for (css =3D &cgrp->css; css; css =3D css->parent) { cgrp =3D container_of(css, struct perf_cgroup, css); @@ -873,7 +872,7 @@ static void perf_cgroup_switch(struct task_struct *task) * event_filter_match() to not have to pass * task around */ - cpu_ctx_sched_in(cpuctx, EVENT_ALL, task); + cpu_ctx_sched_in(cpuctx, EVENT_ALL); =20 perf_pmu_enable(cpuctx->ctx.pmu); perf_ctx_unlock(cpuctx, cpuctx->task_ctx); @@ -1057,8 +1056,7 @@ static inline int perf_cgroup_connect(pid_t pid, stru= ct perf_event *event, } =20 static inline void -perf_cgroup_set_timestamp(struct task_struct *task, - struct perf_event_context *ctx) +perf_cgroup_set_timestamp(struct perf_cpu_context *cpuctx) { } =20 @@ -2651,8 +2649,7 @@ static void ctx_sched_out(struct perf_event_context *= ctx, static void ctx_sched_in(struct perf_event_context *ctx, struct perf_cpu_context *cpuctx, - enum event_type_t event_type, - struct task_struct *task); + enum event_type_t event_type); =20 static void task_ctx_sched_out(struct perf_cpu_context *cpuctx, struct perf_event_context *ctx, @@ -2668,15 +2665,14 @@ static void task_ctx_sched_out(struct perf_cpu_cont= ext *cpuctx, } =20 static void perf_event_sched_in(struct perf_cpu_context *cpuctx, - struct perf_event_context *ctx, - struct task_struct *task) + struct perf_event_context *ctx) { - cpu_ctx_sched_in(cpuctx, EVENT_PINNED, task); + cpu_ctx_sched_in(cpuctx, EVENT_PINNED); if (ctx) - ctx_sched_in(ctx, cpuctx, EVENT_PINNED, task); - cpu_ctx_sched_in(cpuctx, EVENT_FLEXIBLE, task); + ctx_sched_in(ctx, cpuctx, EVENT_PINNED); + cpu_ctx_sched_in(cpuctx, EVENT_FLEXIBLE); if (ctx) - ctx_sched_in(ctx, cpuctx, EVENT_FLEXIBLE, task); + ctx_sched_in(ctx, cpuctx, EVENT_FLEXIBLE); } =20 /* @@ -2726,7 +2722,7 @@ static void ctx_resched(struct perf_cpu_context *cpuc= tx, else if (ctx_event_type & EVENT_PINNED) cpu_ctx_sched_out(cpuctx, EVENT_FLEXIBLE); =20 - perf_event_sched_in(cpuctx, task_ctx, current); + perf_event_sched_in(cpuctx, task_ctx); perf_pmu_enable(cpuctx->ctx.pmu); } =20 @@ -2949,7 +2945,7 @@ static void __perf_event_enable(struct perf_event *ev= ent, return; =20 if (!event_filter_match(event)) { - ctx_sched_in(ctx, cpuctx, EVENT_TIME, current); + ctx_sched_in(ctx, cpuctx, EVENT_TIME); return; } =20 @@ -2958,7 +2954,7 @@ static void __perf_event_enable(struct perf_event *ev= ent, * then don't put it on unless the group is on. */ if (leader !=3D event && leader->state !=3D PERF_EVENT_STATE_ACTIVE) { - ctx_sched_in(ctx, cpuctx, EVENT_TIME, current); + ctx_sched_in(ctx, cpuctx, EVENT_TIME); return; } =20 @@ -3803,8 +3799,7 @@ ctx_flexible_sched_in(struct perf_event_context *ctx, static void ctx_sched_in(struct perf_event_context *ctx, struct perf_cpu_context *cpuctx, - enum event_type_t event_type, - struct task_struct *task) + enum event_type_t event_type) { int is_active =3D ctx->is_active; =20 @@ -3816,7 +3811,7 @@ ctx_sched_in(struct perf_event_context *ctx, if (is_active ^ EVENT_TIME) { /* start ctx time */ __update_context_time(ctx, false); - perf_cgroup_set_timestamp(task, ctx); + perf_cgroup_set_timestamp(cpuctx); /* * CPU-release for the below ->is_active store, * see __load_acquire() in perf_event_time_now() @@ -3847,12 +3842,11 @@ ctx_sched_in(struct perf_event_context *ctx, } =20 static void cpu_ctx_sched_in(struct perf_cpu_context *cpuctx, - enum event_type_t event_type, - struct task_struct *task) + enum event_type_t event_type) { struct perf_event_context *ctx =3D &cpuctx->ctx; =20 - ctx_sched_in(ctx, cpuctx, event_type, task); + ctx_sched_in(ctx, cpuctx, event_type); } =20 static void perf_event_context_sched_in(struct perf_event_context *ctx, @@ -3894,7 +3888,7 @@ static void perf_event_context_sched_in(struct perf_e= vent_context *ctx, */ if (!RB_EMPTY_ROOT(&ctx->pinned_groups.tree)) cpu_ctx_sched_out(cpuctx, EVENT_FLEXIBLE); - perf_event_sched_in(cpuctx, ctx, task); + perf_event_sched_in(cpuctx, ctx); =20 if (cpuctx->sched_cb_usage && pmu->sched_task) pmu->sched_task(cpuctx->task_ctx, true); @@ -4195,7 +4189,7 @@ static bool perf_rotate_context(struct perf_cpu_conte= xt *cpuctx) if (cpu_event) rotate_ctx(&cpuctx->ctx, cpu_event); =20 - perf_event_sched_in(cpuctx, task_ctx, current); + perf_event_sched_in(cpuctx, task_ctx); =20 perf_pmu_enable(cpuctx->ctx.pmu); perf_ctx_unlock(cpuctx, cpuctx->task_ctx); @@ -4267,7 +4261,7 @@ static void perf_event_enable_on_exec(int ctxn) clone_ctx =3D unclone_ctx(ctx); ctx_resched(cpuctx, ctx, event_type); } else { - ctx_sched_in(ctx, cpuctx, EVENT_TIME, current); + ctx_sched_in(ctx, cpuctx, EVENT_TIME); } perf_ctx_unlock(cpuctx, ctx); =20 --=20 2.20.1 From nobody Mon Jun 22 14:26:16 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id A1D17C433EF for ; Tue, 22 Mar 2022 12:10:30 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234674AbiCVMLz (ORCPT ); Tue, 22 Mar 2022 08:11:55 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39986 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234712AbiCVMLr (ORCPT ); Tue, 22 Mar 2022 08:11:47 -0400 Received: from mail-pl1-x632.google.com (mail-pl1-x632.google.com [IPv6:2607:f8b0:4864:20::632]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A2FDA85650 for ; Tue, 22 Mar 2022 05:10:18 -0700 (PDT) Received: by mail-pl1-x632.google.com with SMTP id q11so15250203pln.11 for ; Tue, 22 Mar 2022 05:10:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20210112.gappssmtp.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=gwfN6VcDBeRjK7V61dOwWD2urkXJySgC4BmAO7ZvYbM=; b=Z6UWSD/PAs1653AL5+vTkN/dlBXFDOTgqXlQMSnGGGyYMAFx+d/4rMGckEn18A9qOl BhD/DlvKFWKZugK5n2yqWtz3cKE+FA2pm5AkQHe80adyXS4c8II+HiBMvRArqxrMC1JK 7rJWBWyVdCjT447KnmJqMNeI4SO32C0W2ObBI91MHNi2M94CHAsVYDtvpWCmZaVVuESr uIQsOlsdWIjgwAzXB8+OtSJeApSqjYqmBmtj1cnCQZ+R5x5XwGbIBJTyehP1vKbP80za r1HJMWiiMSqE+kcKFsovPokqFn/WJNdKk1tLgiXrGZ30gEX5xKps2YZEjiVuJT1TfQLv EYjA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=gwfN6VcDBeRjK7V61dOwWD2urkXJySgC4BmAO7ZvYbM=; b=qU05Ib7tbLrV6ZTgByAzsmHc0aaVeGkoFY5sWBrr/Rea8q15sgkNpDfC+61b+T0QZV jW1oQgJvYXifivkQjv6iZZzzkH7Z3gM2a8X+Lj09JWLWGzjGki/TWUO1Ui7God2SMJ0V C2+wd/aFgU5VGlv1f3H4Bxgx6JlYYAWKZ5F5U9avsviHyc5PYsMYRXvaOkqDjzlyqPoQ optFNSRH4gl7LJHYgrlw2k22rww0faJOOFktBv1Co8IbDBgvwXLVnj3JCSpCispdYveO AqbH6yv6p6eXAqg2/tloivOD1wO9DIylVFXyidkI+pWSlEFaqnRDMLqlpw3whpkC5EDN 3NRw== X-Gm-Message-State: AOAM531KG4ssD9kiLEX6TX2lIVFm1dn+fy1sp4u0T44tDuQfuEmJLX6g Q/hN1yJE8VSog23CECWcgXzzeg== X-Google-Smtp-Source: ABdhPJxHTeehJ7FigPtFgGA/wMvkEgVkxOff44GlHdN9UXiFH4IurvHVl5hUKeYxuPiAR4OmSOzQ6w== X-Received: by 2002:a17:902:b941:b0:14d:af72:3f23 with SMTP id h1-20020a170902b94100b0014daf723f23mr18182992pls.6.1647951017987; Tue, 22 Mar 2022 05:10:17 -0700 (PDT) Received: from localhost.localdomain ([2409:8a28:e62:3990:75d0:8573:b73c:e6b4]) by smtp.gmail.com with ESMTPSA id q9-20020a056a00088900b004e03b051040sm24833850pfj.112.2022.03.22.05.10.10 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 22 Mar 2022 05:10:17 -0700 (PDT) From: Chengming Zhou To: peterz@infradead.org, mingo@redhat.com, acme@kernel.org, mark.rutland@arm.com, alexander.shishkin@linux.intel.com, jolsa@kernel.org, namhyung@kernel.org, eranian@google.com Cc: linux-perf-users@vger.kernel.org, linux-kernel@vger.kernel.org, duanxiongchun@bytedance.com, songmuchun@bytedance.com, Chengming Zhou Subject: [PATCH v2 4/6] perf/core: Use stable cpuctx->cgrp when update perf cgroup time Date: Tue, 22 Mar 2022 20:08:32 +0800 Message-Id: <20220322120834.98637-5-zhouchengming@bytedance.com> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20220322120834.98637-1-zhouchengming@bytedance.com> References: <20220322120834.98637-1-zhouchengming@bytedance.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" The current code use the changeable task->cgroups when update the perf cgroup time, which maybe not the active perf_cgroup that sched_in on the CPU. This patch change to use the stable cpuctx->cgrp and only update time when event is matched with cpuctx->cgrp. Signed-off-by: Chengming Zhou --- kernel/events/core.c | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/kernel/events/core.c b/kernel/events/core.c index e27c06628bad..849a81299906 100644 --- a/kernel/events/core.c +++ b/kernel/events/core.c @@ -780,7 +780,6 @@ static inline void update_cgrp_time_from_cpuctx(struct = perf_cpu_context *cpuctx, static inline void update_cgrp_time_from_event(struct perf_event *event) { struct perf_cgroup_info *info; - struct perf_cgroup *cgrp; =20 /* * ensure we access cgroup data only when needed and @@ -789,11 +788,10 @@ static inline void update_cgrp_time_from_event(struct= perf_event *event) if (!is_cgroup_event(event)) return; =20 - cgrp =3D perf_cgroup_from_task(current, event->ctx); /* - * Do not update time when cgroup is not active + * Only update time when event is matched with cpuctx cgrp */ - if (cgroup_is_descendant(cgrp->css.cgroup, event->cgrp->css.cgroup)) { + if (perf_cgroup_match(event)) { info =3D this_cpu_ptr(event->cgrp->info); __update_cgrp_time(info, perf_clock(), true); } --=20 2.20.1 From nobody Mon Jun 22 14:26:16 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 375CDC433F5 for ; Tue, 22 Mar 2022 12:10:44 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234757AbiCVMMI (ORCPT ); Tue, 22 Mar 2022 08:12:08 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40612 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234702AbiCVMME (ORCPT ); Tue, 22 Mar 2022 08:12:04 -0400 Received: from mail-pg1-x536.google.com (mail-pg1-x536.google.com [IPv6:2607:f8b0:4864:20::536]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 39A1F85659 for ; Tue, 22 Mar 2022 05:10:27 -0700 (PDT) Received: by mail-pg1-x536.google.com with SMTP id e6so12473009pgn.2 for ; Tue, 22 Mar 2022 05:10:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20210112.gappssmtp.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=uePDHamTg8Aaje9cV7GXZKI4UAeGuCXRnl2ENUUEXxk=; b=FsRTO4RlcjR5SvPCtQOCwI3kpAD3W7w49KGuO99dqINQ6uOWZ5gOIMsgNsI6jxy9Uz QSCMtlPoMeim6YGW8OkevyocYuF/DBt28S6HI1X+CAidRFCDU9fJZc/XNpuAAXEslwpm MD1wTijilB06RSk200EmVDcbKzgWhOF3W3kILccXJCG46DTFvKTbFTHYC4tgb/ONeh/h G0l+INhk24k4SzX4HTlxzbuqMok45TzQGdon9rrL6losB9H0wWpTEArGsFSqIKO3OuWb UxSgpXULxwsvVnj0i/ff5ZkKHE9EpnJbUO+MjLpDReoR0eUp6kCZJjB9EBleJC/iiR6z zm9w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=uePDHamTg8Aaje9cV7GXZKI4UAeGuCXRnl2ENUUEXxk=; b=xLSKS4vbcThOk6hNwNZimYIPKL/VYgCmznvKHHJ/+TM9zeMdC0A2UXOuf8Dm7BSkI1 r5brJSbl5DKRaKM6heu9uNNW76OA4OiGrZjcuZlbd6h9m1DphkJV4HOLA/baHbUtVVW4 XsTAupf3ZlNR38nHE8J2ka4bhCjQxRdk/PbVtfsox2pzW/fjGz1EaALrKAxKLbRleBp+ WurkdrZ8hxJQi2pwqSr13RxEF682wQGLEJyeaQK10McFd2YwndgsTpgaSo2kd0zw/lTP OZfwrDYppzGjS39YQUThLyPLur14B8qxr6u/hz+I8Y6uB7XNO+UQP2Nv4kUTK1UTQSKb oCcQ== X-Gm-Message-State: AOAM532TazW103CIooqDPovFISIoIQhM1cWSzcNHqV+txDZog01XRK6r 8tXcC0fYQBDiVuUvhJC2Hb//et3FHpXRS63S X-Google-Smtp-Source: ABdhPJzdsnGKnRy+ZLFU+RCMb/c6l9MPuXmGqraFSCQJyAYNskxfSG2QUnuOF5/VdE382cVcf+ByYQ== X-Received: by 2002:a05:6a00:26cf:b0:4f6:fc52:7b6a with SMTP id p15-20020a056a0026cf00b004f6fc527b6amr28841152pfw.39.1647951026185; Tue, 22 Mar 2022 05:10:26 -0700 (PDT) Received: from localhost.localdomain ([2409:8a28:e62:3990:75d0:8573:b73c:e6b4]) by smtp.gmail.com with ESMTPSA id q9-20020a056a00088900b004e03b051040sm24833850pfj.112.2022.03.22.05.10.18 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 22 Mar 2022 05:10:25 -0700 (PDT) From: Chengming Zhou To: peterz@infradead.org, mingo@redhat.com, acme@kernel.org, mark.rutland@arm.com, alexander.shishkin@linux.intel.com, jolsa@kernel.org, namhyung@kernel.org, eranian@google.com Cc: linux-perf-users@vger.kernel.org, linux-kernel@vger.kernel.org, duanxiongchun@bytedance.com, songmuchun@bytedance.com, Chengming Zhou Subject: [PATCH v2 5/6] perf/core: Always set cpuctx cgrp when enable cgroup event Date: Tue, 22 Mar 2022 20:08:33 +0800 Message-Id: <20220322120834.98637-6-zhouchengming@bytedance.com> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20220322120834.98637-1-zhouchengming@bytedance.com> References: <20220322120834.98637-1-zhouchengming@bytedance.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" When enable a cgroup event, cpuctx->cgrp setting is conditional on the current task cgrp matching the event's cgroup, so have to do it for every new event. It brings complexity but no advantage. To keep it simple, this patch would always set cpuctx->cgrp when enable the first cgroup event, and reset to NULL when disable the last cgroup event. In this way, perf_cgroup_match() won't see cpuctx->cgrp =3D=3D NULL if it's a cgroup event, so add a WARN_ON_ONCE(!cpuctx->cgrp) there. Signed-off-by: Chengming Zhou --- kernel/events/core.c | 20 +++----------------- 1 file changed, 3 insertions(+), 17 deletions(-) diff --git a/kernel/events/core.c b/kernel/events/core.c index 849a81299906..4c8657b08301 100644 --- a/kernel/events/core.c +++ b/kernel/events/core.c @@ -703,7 +703,7 @@ perf_cgroup_match(struct perf_event *event) return true; =20 /* wants specific cgroup scope but @cpuctx isn't associated with any */ - if (!cpuctx->cgrp) + if (WARN_ON_ONCE(!cpuctx->cgrp)) return false; =20 /* @@ -975,22 +975,10 @@ perf_cgroup_event_enable(struct perf_event *event, st= ruct perf_event_context *ct */ cpuctx =3D container_of(ctx, struct perf_cpu_context, ctx); =20 - /* - * Since setting cpuctx->cgrp is conditional on the current @cgrp - * matching the event's cgroup, we must do this for every new event, - * because if the first would mismatch, the second would not try again - * and we would leave cpuctx->cgrp unset. - */ - if (ctx->is_active && !cpuctx->cgrp) { - struct perf_cgroup *cgrp =3D perf_cgroup_from_task(current, ctx); - - if (cgroup_is_descendant(cgrp->css.cgroup, event->cgrp->css.cgroup)) - cpuctx->cgrp =3D cgrp; - } - if (ctx->nr_cgroups++) return; =20 + cpuctx->cgrp =3D perf_cgroup_from_task(current, ctx); list_add(&cpuctx->cgrp_cpuctx_entry, per_cpu_ptr(&cgrp_cpuctx_list, event->cpu)); } @@ -1012,9 +1000,7 @@ perf_cgroup_event_disable(struct perf_event *event, s= truct perf_event_context *c if (--ctx->nr_cgroups) return; =20 - if (ctx->is_active && cpuctx->cgrp) - cpuctx->cgrp =3D NULL; - + cpuctx->cgrp =3D NULL; list_del(&cpuctx->cgrp_cpuctx_entry); =20 if (list_empty(per_cpu_ptr(&cgrp_cpuctx_list, event->cpu))) --=20 2.20.1 From nobody Mon Jun 22 14:26:16 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 89B3EC433EF for ; Tue, 22 Mar 2022 12:10:48 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234721AbiCVMMN (ORCPT ); Tue, 22 Mar 2022 08:12:13 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41488 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234693AbiCVMME (ORCPT ); Tue, 22 Mar 2022 08:12:04 -0400 Received: from mail-pj1-x102c.google.com (mail-pj1-x102c.google.com [IPv6:2607:f8b0:4864:20::102c]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C654D8567A for ; Tue, 22 Mar 2022 05:10:34 -0700 (PDT) Received: by mail-pj1-x102c.google.com with SMTP id mj15-20020a17090b368f00b001c637aa358eso2486900pjb.0 for ; Tue, 22 Mar 2022 05:10:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20210112.gappssmtp.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=O/7NZbomHJNevRDviLAq47WogxJXAVWPPfKEsGLaM2Q=; b=u+N3p9kTmRnGrXL314WMH5VKZQw2nEgT4xoBO1regbTW10qO9UIqkNynJKqT6hJTQI GfE7w63Iyvv6FpRThYj5q1G6+loA64m2+EqaBgdw4tNHxFu2u7d90d5asSWZXnxF23bG ZsMTHS7CN4uLkCKYbBNYFmTP8sr4Ok1yodh7ZhdNGts6aWPMBVE4Lz2wIc0PBRP1g1Cx aHFwhyYcQNpwDnmqNls6pMAVG50I1fcCb1R4B+V0kMZVJdSbE98mqY0MkmRyBGyepID5 u0ST1DHgNNRuXMPo0APBXThcgYAs8w3cnhfGw7CscQ/Ah9fyysaw89ORGlgdwn5Fh2PI vqTA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=O/7NZbomHJNevRDviLAq47WogxJXAVWPPfKEsGLaM2Q=; b=G1lYuS/4digQgOz++fkNtrGmD2YgeYUTj3L6z0LVtoRDnxVSVu8RC9UraAKU2CuNNd YhYvQyODcuEkO6EXcDQ2vBMtfaVwEUPCOv67Z3is0zJo45rJ7RzK7bFJYXtC87Hti2Y4 vGvyIu3wLIaK5HGd21Bp3P/j8rzMdPo0PQEpE27UmRQAELc39Yj9d1Cj8t9U3Y7lV+v0 fTdHOP0YS5VrMtMBeu0yvvvX8lLL4GbXTFYKCxmdzRxUwBiI1YFRXT3NFxEDLESOvRJy 0S6bHSw0ss3KJyRxOk53xjFbOrRtqiyo/B0s0r04tUz3dEMyJzi97CEgtvZwN60FVfVA GLVg== X-Gm-Message-State: AOAM532tSVEqiNzz4BL1R1PjCMtQw1X0kV1F2tsRt7bUT0+J2nhpN3ir b/EMIUM8L8W6qIflMoEUwF+whg== X-Google-Smtp-Source: ABdhPJxjr0Mbq/yQ/Er0makurPOudwZPK8zCr3U/6Orououfw/iFmcSMXQY4it/+KgCKOLrzKVLcFw== X-Received: by 2002:a17:90a:1197:b0:1bf:65ff:f542 with SMTP id e23-20020a17090a119700b001bf65fff542mr4610157pja.5.1647951034367; Tue, 22 Mar 2022 05:10:34 -0700 (PDT) Received: from localhost.localdomain ([2409:8a28:e62:3990:75d0:8573:b73c:e6b4]) by smtp.gmail.com with ESMTPSA id q9-20020a056a00088900b004e03b051040sm24833850pfj.112.2022.03.22.05.10.26 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 22 Mar 2022 05:10:34 -0700 (PDT) From: Chengming Zhou To: peterz@infradead.org, mingo@redhat.com, acme@kernel.org, mark.rutland@arm.com, alexander.shishkin@linux.intel.com, jolsa@kernel.org, namhyung@kernel.org, eranian@google.com Cc: linux-perf-users@vger.kernel.org, linux-kernel@vger.kernel.org, duanxiongchun@bytedance.com, songmuchun@bytedance.com, Chengming Zhou Subject: [PATCH v2 6/6] perf/core: Don't need event_filter_match when merge_sched_in() Date: Tue, 22 Mar 2022 20:08:34 +0800 Message-Id: <20220322120834.98637-7-zhouchengming@bytedance.com> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20220322120834.98637-1-zhouchengming@bytedance.com> References: <20220322120834.98637-1-zhouchengming@bytedance.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" There are two obselete comments in perf_cgroup_switch(), since we don't use event_filter_match() when event_sched_out(). And found we needn't to use event_filter_match() when sched_in too. Because now we use the perf_event groups RB-tree to get the exact matched perf_events, don't need to go through the event_filter_match() to check if matched. We can remove it in merge_sched_in() actually, but this patch changes it to a WARN_ON_ONCE for debug purpose, and found no warning in our stress test. Signed-off-by: Chengming Zhou --- kernel/events/core.c | 10 +++------- 1 file changed, 3 insertions(+), 7 deletions(-) diff --git a/kernel/events/core.c b/kernel/events/core.c index 4c8657b08301..744078fe2819 100644 --- a/kernel/events/core.c +++ b/kernel/events/core.c @@ -860,15 +860,11 @@ static void perf_cgroup_switch(struct task_struct *ta= sk) perf_pmu_disable(cpuctx->ctx.pmu); =20 cpu_ctx_sched_out(cpuctx, EVENT_ALL); - /* - * must not be done before ctxswout due - * to event_filter_match() in event_sched_out() - */ cpuctx->cgrp =3D cgrp; /* * set cgrp before ctxsw in to allow - * event_filter_match() to not have to pass - * task around + * visit_groups_merge() to find matched + * cgroup events */ cpu_ctx_sched_in(cpuctx, EVENT_ALL); =20 @@ -3729,7 +3725,7 @@ static int merge_sched_in(struct perf_event *event, v= oid *data) if (event->state <=3D PERF_EVENT_STATE_OFF) return 0; =20 - if (!event_filter_match(event)) + if (WARN_ON_ONCE(!event_filter_match(event))) return 0; =20 if (group_can_go_on(event, cpuctx, *can_add_hw)) { --=20 2.20.1