From nobody Fri Dec 19 21:12:40 2025 Received: from sender4-pp-f112.zoho.com (sender4-pp-f112.zoho.com [136.143.188.112]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id EB2BBDF6C for ; Wed, 3 Dec 2025 13:56:57 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=pass smtp.client-ip=136.143.188.112 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1764770219; cv=pass; b=unsmCFX/pJ8AbFZgjz6+b0BXLm3xF4VvQpZY7MIWmwI6lBIqFH/ZJIn2b21WGBqL5jmnPQArJoGA2gMh59bspMOjHI2E8V5Lf/+d3RRy4xqeYsHtkhfsq3+7b6LuYTqG+Dg/SK5X7yq7YKQARO5YdNfwwJbn9TH3TdFL3foZWeg= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1764770219; c=relaxed/simple; bh=B006ZXSd72/jP/YobCGvyhNNItsk7oZh0+O9WTnbAvI=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=OMaE089EijIZWGC30Loj/q00fujUqQmBdd0ZsGlI1rMgs2F+B3sJC7Y16CQImOEtl3V/lzAm3YfmB7tnh5zK9ZWE0HA2QOriuPVqabgxuRuEyo5wVvKh48NEKdqNhnqwnicuzkqhCgzue1J6GRz1bkdQdgqhQ7nqeYff/MA+UII= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=collabora.com; spf=pass smtp.mailfrom=collabora.com; dkim=pass (1024-bit key) header.d=collabora.com header.i=nicolas.frattaroli@collabora.com header.b=HrHOlml2; arc=pass smtp.client-ip=136.143.188.112 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=collabora.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=collabora.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=collabora.com header.i=nicolas.frattaroli@collabora.com header.b="HrHOlml2" ARC-Seal: i=1; a=rsa-sha256; t=1764770204; cv=none; d=zohomail.com; s=zohoarc; b=CSOAJ8aJW0jn2SFif2Ydk1uQyyx6OEm7IJwciO3Sg6SROYVwJ/RRuewWYdPTsYmahAnJ2SmplKzq4IhJCs8NW7GpYaADtcXOBjuvrJyx38ZrhsbRaYLpcBuazfK8oa0HVHzcMI796Nz3wTnJJ9ZQQF6TsG1NtOBW7SS9orRkxps= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1764770204; h=Content-Type:Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:MIME-Version:Message-ID:References:Subject:Subject:To:To:Message-Id:Reply-To; bh=BafTXFGVFmIDDsJ0hvlCVkL5yw5qTMNyy0t2dgnoYMY=; b=IvM5C3Pu8NCdCFMKgrlOkvqIwAKijPDvajO/jw5dQzapX48tBhSuNEZoU5w0xQYlEef004uR4diZgsgBCoCfAP4382ukzoDXFMg+GxuUQjVGtjosTWbXHUvv5fT6jm2KPFpviXPiAqIkUy5T4Fvxn5VLobl9fK10JtiQehp8Apg= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass header.i=collabora.com; spf=pass smtp.mailfrom=nicolas.frattaroli@collabora.com; dmarc=pass header.from= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; t=1764770204; s=zohomail; d=collabora.com; i=nicolas.frattaroli@collabora.com; h=From:From:Date:Date:Subject:Subject:MIME-Version:Content-Type:Content-Transfer-Encoding:Message-Id:Message-Id:References:In-Reply-To:To:To:Cc:Cc:Reply-To; bh=BafTXFGVFmIDDsJ0hvlCVkL5yw5qTMNyy0t2dgnoYMY=; b=HrHOlml2eCgEgPkZ1bObztMnXMkWea5+PIxlEZ7ntgnS0NMh2ZN9IPItYVvNG74y V/GyBb2VBTdP2fAawrhK5aAgkBl7wCMqYtoEvvhjk6EqXc6lS1+EznPJaJgZntZH7l+ 01lB1860OHGbTiKqOtY+jsVtk+SIlLUFETd+Mybc= Received: by mx.zohomail.com with SMTPS id 1764770203279936.6526689272895; Wed, 3 Dec 2025 05:56:43 -0800 (PST) From: Nicolas Frattaroli Date: Wed, 03 Dec 2025 14:56:22 +0100 Subject: [PATCH 1/2] drm/panthor: Add tracepoint for hardware utilisation changes Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20251203-panthor-tracepoints-v1-1-871c8917e084@collabora.com> References: <20251203-panthor-tracepoints-v1-0-871c8917e084@collabora.com> In-Reply-To: <20251203-panthor-tracepoints-v1-0-871c8917e084@collabora.com> To: Boris Brezillon , Steven Price , Liviu Dudau , Maarten Lankhorst , Maxime Ripard , Thomas Zimmermann , David Airlie , Simona Vetter Cc: kernel@collabora.com, linux-kernel@vger.kernel.org, dri-devel@lists.freedesktop.org, Nicolas Frattaroli X-Mailer: b4 0.14.3 Mali GPUs have three registers that indicate which parts of the hardware are powered and active at any moment. These take the form of bitmaps. In the case of SHADER_PWRACTIVE for example, a high bit indicates that the shader core corresponding to that bit index is active. These bitmaps aren't solely contiguous bits, as it's common to have holes in the sequence of shader core indices, and the actual set of which cores are present is defined by the "shader present" register. When the GPU finishes a power state transition, it fires a GPU_IRQ_POWER_CHANGED_ALL interrupt. After such an interrupt is received, the PWRACTIVE registers will likely contain interesting new information. This is not to be confused with the PWR_IRQ_POWER_CHANGED_ALL interrupt, which is something related to Mali v14+'s power control logic. The PWRACTIVE registers and corresponding interrupts are already available in v9 and onwards. Expose this as a tracepoint to userspace. This allows users to debug various scenarios and gather interesting information, such as: knowing how much hardware is lit up at any given time, correlating graphics corruption with a specific active shader core, measuring when hardware is allowed to go to an inactive state again, and so on. Signed-off-by: Nicolas Frattaroli --- drivers/gpu/drm/panthor/panthor_device.c | 1 + drivers/gpu/drm/panthor/panthor_gpu.c | 9 ++++++++ drivers/gpu/drm/panthor/panthor_trace.h | 38 ++++++++++++++++++++++++++++= ++++ 3 files changed, 48 insertions(+) diff --git a/drivers/gpu/drm/panthor/panthor_device.c b/drivers/gpu/drm/pan= thor/panthor_device.c index e133b1e0ad6d..a3cb934104b8 100644 --- a/drivers/gpu/drm/panthor/panthor_device.c +++ b/drivers/gpu/drm/panthor/panthor_device.c @@ -548,6 +548,7 @@ int panthor_device_resume(struct device *dev) DRM_PANTHOR_USER_MMIO_OFFSET, 0, 1); atomic_set(&ptdev->pm.state, PANTHOR_DEVICE_PM_STATE_ACTIVE); mutex_unlock(&ptdev->pm.mmio_lock); + return 0; =20 err_suspend_devfreq: diff --git a/drivers/gpu/drm/panthor/panthor_gpu.c b/drivers/gpu/drm/pantho= r/panthor_gpu.c index 9cb5dee93212..8830aa9a5c4b 100644 --- a/drivers/gpu/drm/panthor/panthor_gpu.c +++ b/drivers/gpu/drm/panthor/panthor_gpu.c @@ -22,6 +22,9 @@ #include "panthor_hw.h" #include "panthor_regs.h" =20 +#define CREATE_TRACE_POINTS +#include "panthor_trace.h" + /** * struct panthor_gpu - GPU block management data. */ @@ -46,6 +49,7 @@ struct panthor_gpu { (GPU_IRQ_FAULT | \ GPU_IRQ_PROTM_FAULT | \ GPU_IRQ_RESET_COMPLETED | \ + GPU_IRQ_POWER_CHANGED_ALL | \ GPU_IRQ_CLEAN_CACHES_COMPLETED) =20 static void panthor_gpu_coherency_set(struct panthor_device *ptdev) @@ -97,6 +101,11 @@ static void panthor_gpu_irq_handler(struct panthor_devi= ce *ptdev, u32 status) wake_up_all(&ptdev->gpu->reqs_acked); } spin_unlock(&ptdev->gpu->reqs_lock); + + if (status & GPU_IRQ_POWER_CHANGED_ALL) + trace_gpu_power_active(gpu_read64(ptdev, SHADER_PWRACTIVE), + gpu_read64(ptdev, TILER_PWRACTIVE), + gpu_read64(ptdev, L2_PWRACTIVE)); } PANTHOR_IRQ_HANDLER(gpu, GPU, panthor_gpu_irq_handler); =20 diff --git a/drivers/gpu/drm/panthor/panthor_trace.h b/drivers/gpu/drm/pant= hor/panthor_trace.h new file mode 100644 index 000000000000..01013f81e68a --- /dev/null +++ b/drivers/gpu/drm/panthor/panthor_trace.h @@ -0,0 +1,38 @@ +/* SPDX-License-Identifier: GPL-2.0 or MIT */ +/* Copyright 2025 Collabora ltd. */ + +#undef TRACE_SYSTEM +#define TRACE_SYSTEM panthor + +#if !defined(__PANTHOR_TRACE_H__) || defined(TRACE_HEADER_MULTI_READ) +#define __PANTHOR_TRACE_H__ + +#include +#include + +TRACE_EVENT(gpu_power_active, + TP_PROTO(u64 shader_bitmap, u64 tiler_bitmap, u64 l2_bitmap), + TP_ARGS(shader_bitmap, tiler_bitmap, l2_bitmap), + TP_STRUCT__entry( + __field(u64, shader_bitmap) + __field(u64, tiler_bitmap) + __field(u64, l2_bitmap) + ), + TP_fast_assign( + __entry->shader_bitmap =3D shader_bitmap; + __entry->tiler_bitmap =3D tiler_bitmap; + __entry->l2_bitmap =3D l2_bitmap; + ), + TP_printk("shader_bitmap=3D0x%llx tiler_bitmap=3D0x%llx l2_bitmap=3D0x%ll= x", + __entry->shader_bitmap, __entry->tiler_bitmap, __entry->l2_bitmap + ) +); + +#endif /* __PANTHOR_TRACE_H__ */ + +#undef TRACE_INCLUDE_PATH +#define TRACE_INCLUDE_PATH . +#undef TRACE_INCLUDE_FILE +#define TRACE_INCLUDE_FILE panthor_trace + +#include --=20 2.52.0 From nobody Fri Dec 19 21:12:40 2025 Received: from sender3-pp-f112.zoho.com (sender3-pp-f112.zoho.com [136.143.184.112]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1DDAC2FBE09 for ; Wed, 3 Dec 2025 13:57:00 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=pass smtp.client-ip=136.143.184.112 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1764770222; cv=pass; b=rdZH9tMnmbOt57KjUJS2zDh9BC3Ey22oazDZp8LqIIUfWkwCIMZAnNDmVizHpVicvmrnHmKRcjjg9rh1oa9wCZ4FFU35+jelAICk4HPn0lGQFbM5NnFuvBzaLFW67ioBBqe6gZd5bE1uddhtEUN+u6fVJmgEAnjayOpvSn/E+t8= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1764770222; c=relaxed/simple; bh=3BDuAtlKh/TU4DqE1rce/XMJK+2zYQhuNDZln8V/Gr0=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=dMf8yjKgiyg0tH3QWTvD9Psq+aLOKNjo6iHk9ZhLogcwdTN1tSZhi1BZkmaMI7b547c8QVRz+yfALH5sggML0q53l8/n5TDdjWO15fDpO83USKeunsegTyqHEkCekJhjtYEgt4jP4Deh2t3/UZUhJhh5QhXaMML6n2emn59fxgc= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=collabora.com; spf=pass smtp.mailfrom=collabora.com; dkim=pass (1024-bit key) header.d=collabora.com header.i=nicolas.frattaroli@collabora.com header.b=YyySJYRI; arc=pass smtp.client-ip=136.143.184.112 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=collabora.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=collabora.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=collabora.com header.i=nicolas.frattaroli@collabora.com header.b="YyySJYRI" ARC-Seal: i=1; a=rsa-sha256; t=1764770206; cv=none; d=zohomail.com; s=zohoarc; b=h62kQXc4e96tpDLzSxY0RPBPevu5sOF1NHK81Aa+14kJUg/eMIfDte5td9nLe24geenRoKTDl79kLALtszhgJRxxsejcm5pacKT0fIq3l2VD3ETLI9HOetA3v0wI/7WcfNQOrrZMC2eOICB/mZiOqnp6+cb1d9GLH2YRwNeB/Bo= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1764770206; h=Content-Type:Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:MIME-Version:Message-ID:References:Subject:Subject:To:To:Message-Id:Reply-To; bh=5wxRLdAjYIRm8juaxC/XPb9kwZMWrxfeLsXk8TZcU3Y=; b=dk+luQxmg8ESnGkAA3aBj6FAwEt9gKAbIVkmhn9GK4J95d33cbLkZcN2hANodY4w+pYRVnuHjK+weameUzF2FqAq2RNQ7AJJEY5djv6shcNxcHfpkv5tSReRDkDVS2mLuqRzI4eQexGLDuaE109dkc4G0RjlgTS86h/nGVdRyUc= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass header.i=collabora.com; spf=pass smtp.mailfrom=nicolas.frattaroli@collabora.com; dmarc=pass header.from= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; t=1764770206; s=zohomail; d=collabora.com; i=nicolas.frattaroli@collabora.com; h=From:From:Date:Date:Subject:Subject:MIME-Version:Content-Type:Content-Transfer-Encoding:Message-Id:Message-Id:References:In-Reply-To:To:To:Cc:Cc:Reply-To; bh=5wxRLdAjYIRm8juaxC/XPb9kwZMWrxfeLsXk8TZcU3Y=; b=YyySJYRIxAYmd3pnORqR8IHVcnaruTLs4LyOu9msyvZhRd7V6cf1HArc+HWO4Lb/ Y2mYkjIlJylVy4c/QrVG8WfS4pN1QBWnEGhEHe9pJs72F8EsP+MwdUDYifOj1Px912f DQolHhHdbnYeiTHBcFtk8xN4u0pjpGRux21utwJw= Received: by mx.zohomail.com with SMTPS id 1764770206036391.4795067537668; Wed, 3 Dec 2025 05:56:46 -0800 (PST) From: Nicolas Frattaroli Date: Wed, 03 Dec 2025 14:56:23 +0100 Subject: [PATCH 2/2] drm/panthor: Add gpu_job_irq tracepoint Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20251203-panthor-tracepoints-v1-2-871c8917e084@collabora.com> References: <20251203-panthor-tracepoints-v1-0-871c8917e084@collabora.com> In-Reply-To: <20251203-panthor-tracepoints-v1-0-871c8917e084@collabora.com> To: Boris Brezillon , Steven Price , Liviu Dudau , Maarten Lankhorst , Maxime Ripard , Thomas Zimmermann , David Airlie , Simona Vetter Cc: kernel@collabora.com, linux-kernel@vger.kernel.org, dri-devel@lists.freedesktop.org, Nicolas Frattaroli X-Mailer: b4 0.14.3 Mali's CSF firmware triggers the job IRQ whenever there's new firmware events for processing. While this can be a global event (BIT(31) of the status register), it's usually an event relating to a command stream group (the other bit indices). Panthor throws these events onto a workqueue for processing outside the IRQ handler. It's therefore useful to have an instrumented tracepoint that goes beyond the generic IRQ tracepoint for this specific case, as it can be augmented with additional data, namely the events bit mask. This can then be used to debug problems relating to GPU jobs events not being processed quickly enough. The duration_ns field can be used to work backwards from when the tracepoint fires (at the end of the IRQ handler) to figure out when the interrupt itself landed, providing not just information on how long the work queueing took, but also when the actual interrupt itself arrived. With this information in hand, the IRQ handler itself being slow can be excluded as a possible source of problems, and attention can be directed to the workqueue processing instead. Signed-off-by: Nicolas Frattaroli --- drivers/gpu/drm/panthor/panthor_fw.c | 13 +++++++++++++ drivers/gpu/drm/panthor/panthor_trace.h | 24 ++++++++++++++++++++++++ 2 files changed, 37 insertions(+) diff --git a/drivers/gpu/drm/panthor/panthor_fw.c b/drivers/gpu/drm/panthor= /panthor_fw.c index 94a3cd6dfa6d..df07f6435cda 100644 --- a/drivers/gpu/drm/panthor/panthor_fw.c +++ b/drivers/gpu/drm/panthor/panthor_fw.c @@ -26,6 +26,7 @@ #include "panthor_mmu.h" #include "panthor_regs.h" #include "panthor_sched.h" +#include "panthor_trace.h" =20 #define CSF_FW_NAME "mali_csffw.bin" =20 @@ -1059,6 +1060,12 @@ static void panthor_fw_init_global_iface(struct pant= hor_device *ptdev) =20 static void panthor_job_irq_handler(struct panthor_device *ptdev, u32 stat= us) { + u32 duration; + u64 start; + + if (tracepoint_enabled(gpu_job_irq)) + start =3D ktime_get_ns(); + gpu_write(ptdev, JOB_INT_CLEAR, status); =20 if (!ptdev->fw->booted && (status & JOB_INT_GLOBAL_IF)) @@ -1071,6 +1078,12 @@ static void panthor_job_irq_handler(struct panthor_d= evice *ptdev, u32 status) return; =20 panthor_sched_report_fw_events(ptdev, status); + + if (tracepoint_enabled(gpu_job_irq)) { + if (check_sub_overflow(ktime_get_ns(), start, &duration)) + duration =3D U32_MAX; + trace_gpu_job_irq(status, duration); + } } PANTHOR_IRQ_HANDLER(job, JOB, panthor_job_irq_handler); =20 diff --git a/drivers/gpu/drm/panthor/panthor_trace.h b/drivers/gpu/drm/pant= hor/panthor_trace.h index 01013f81e68a..fcddfdb6ffef 100644 --- a/drivers/gpu/drm/panthor/panthor_trace.h +++ b/drivers/gpu/drm/panthor/panthor_trace.h @@ -28,6 +28,30 @@ TRACE_EVENT(gpu_power_active, ) ); =20 +/** + * gpu_job_irq - called after a job interrupt from firmware completes + * @events: bitmask of BIT(CSG id) | BIT(31) for a global event + * @duration_ns: Nanoseconds between job IRQ handler entry and exit + * + * The panthor_job_irq_handler() function instrumented by this tracepoint = exits + * once it has queued the firmware interrupts for processing, not when the + * firmware interrupts are fully processed. This tracepoint allows for deb= ugging + * issues with delays in the workqueue's processing of events. + */ +TRACE_EVENT(gpu_job_irq, + TP_PROTO(u32 events, u32 duration_ns), + TP_ARGS(events, duration_ns), + TP_STRUCT__entry( + __field(u32, events) + __field(u32, duration_ns) + ), + TP_fast_assign( + __entry->events =3D events; + __entry->duration_ns =3D duration_ns; + ), + TP_printk("events=3D0x%x duration_ns=3D%d", __entry->events, __entry->dur= ation_ns) +); + #endif /* __PANTHOR_TRACE_H__ */ =20 #undef TRACE_INCLUDE_PATH --=20 2.52.0