From nobody Fri Nov 29 00:51:18 2024 Received: from mail-yw1-f201.google.com (mail-yw1-f201.google.com [209.85.128.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 31BC7185082 for ; Thu, 26 Sep 2024 17:36:33 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727372195; cv=none; b=dGByd4fcTUjCkqS0ClVRMogQe+95v9pHwWf6bI3fgcK0dU4tXsF5pDtdaEajMj7q94F8QoKSNrg5mTjvUve/iypLxAfnZsOZH2isOQTko9FyZo9rK5SlxDdMLYPccML7YT5h0078ZVt8JMGeNlka3LYt6JJ2qBe4CBDcjzs9Tyo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727372195; c=relaxed/simple; bh=blAc9G6OebTxuItqauXCFyKk/ED91E1EMUyf5Yy8yWw=; h=Date:In-Reply-To:Message-Id:Mime-Version:References:Subject:From: To:Content-Type; b=YnzQcdTLrMNr4HgBdmvMyegZ9237JMsjctYaZsp3Uwyb1u9pb/JaOfh/y1UqKzEDyflHChwGwMdBUTqmzDAi9wjzy+Ymk4WorCEuc5mI6xcACJBf32glJfhdiN42H3sBLZgrjDJIVb9pTP0Krs86NEhB8vYDS+Iior/p4xw09TA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--irogers.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=MeLuJvAc; arc=none smtp.client-ip=209.85.128.201 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--irogers.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="MeLuJvAc" Received: by mail-yw1-f201.google.com with SMTP id 00721157ae682-6d9e31e66eeso30807147b3.1 for ; Thu, 26 Sep 2024 10:36:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1727372193; x=1727976993; darn=vger.kernel.org; h=to:from:subject:references:mime-version:message-id:in-reply-to:date :from:to:cc:subject:date:message-id:reply-to; bh=EJsPdONTfaeiSG4hvXGB1buTusBhDadyMVVcT3k3om0=; b=MeLuJvAcnqAlvxl/TNKjWJJ3dkhzVCqLIZ85DMjVwARRAfD9c/cE5pTUp1GjwuN2rl hvPV0DSng7MrpTz2e7hwfzk5ZJHFXRx/A2FyWpeyxDaupLuV6rIISK26nurzIgcfSPGq 6ewRBPRiGf4gpingWH2BaVgOGkV9366owaDb6GSUNlOhYvZilNwErvGJXbAfLy7YYL9r JaZntzjWTq9edaV+zMg8vCWNulWGupFEHjM1rjMMB9rHyW7UklmIS+ppAoWK7BOsy2dQ gZGOsWihNY5ArZejY1DTM1e/JKecQgfgMkCyjTYNBzBjvy5zGlIcxRkkoOLtRSPGgsGN 6fCg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1727372193; x=1727976993; h=to:from:subject:references:mime-version:message-id:in-reply-to:date :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=EJsPdONTfaeiSG4hvXGB1buTusBhDadyMVVcT3k3om0=; b=K0VdfmoKsE79rLs+07K8+UzaPxLX1oD8iQ3qQvrwDq2AvmvMd9RVRHr7ZCi2adoyoq w3zZuhJKHJwAKzhtLbfF9ROvbkWXId0FFcpDQJgU3uW2l1qL2yTUo8W8POcO0CHtSLXr yuO/4HiFFXUeWHRxPDTM1VNy1WZDZA+Ji8+/2iRDqZGtTekh8WvHO+ShvM+Z4vMxj9/z ta8/Lp0YnkPEXLL9pCEM0Bhcv0zV7i6LQ9XYHnMSstkZJZzHGVo3sSi2Mjq+5oZF5etp LX9x3eF5y3mxNiJOLH0cUoG+wljstazBIokqCbljkc4WlqC+/1IglNcSf8xv66mn3Iho zinQ== X-Forwarded-Encrypted: i=1; AJvYcCWFnibBkM0jRc/ccTeQUzvnR2f0J6BaKfuVSUxgFfwYpxEBQa28OCSjXF2EZK+GNXPLDfMLwN7NLqgRZcQ=@vger.kernel.org X-Gm-Message-State: AOJu0YwayfWI31+es/bB4ymaKqF/LtEyUrlsBvqKqYrqK6s4SdON6hxU VzP0wNg5PaguzVshsOWdMbNHi6OycKG3rfx4pgIKkGOC2LRZWbx1XyFnjh6Ci96XE1pMm6NVVQH b4JZqDw== X-Google-Smtp-Source: AGHT+IHpxdTYHncEHsDg61Q+sSYLnuPuOSiEh1LvRBvpsC4/tju0BMh1GG4g2X8BoXY+xExwarwZnLL9TBUV X-Received: from irogers.svl.corp.google.com ([2620:15c:2a3:200:8872:3585:18ed:a056]) (user=irogers job=sendgmr) by 2002:a25:690b:0:b0:e24:f10f:7d99 with SMTP id 3f1490d57ef6-e2604b2dfdcmr119276.4.1727372192970; Thu, 26 Sep 2024 10:36:32 -0700 (PDT) Date: Thu, 26 Sep 2024 10:35:53 -0700 In-Reply-To: <20240926173554.404411-1-irogers@google.com> Message-Id: <20240926173554.404411-12-irogers@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240926173554.404411-1-irogers@google.com> X-Mailer: git-send-email 2.46.1.824.gd892dcdcdd-goog Subject: [PATCH v4 11/12] perf jevents: Build support for generating metrics from python From: Ian Rogers To: Peter Zijlstra , Ingo Molnar , Arnaldo Carvalho de Melo , Namhyung Kim , Mark Rutland , Alexander Shishkin , Jiri Olsa , Ian Rogers , Adrian Hunter , Kan Liang , John Garry , Jing Zhang , Sandipan Das , Benjamin Gray , Xu Yang , linux-kernel@vger.kernel.org, linux-perf-users@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Generate extra-metrics.json and extra-metricgroups.json from python architecture specific scripts. The metrics themselves will be added in later patches. If a build takes place in tools/perf/ then extra-metrics.json and extra-metricgroups.json are generated in that directory and so added to .gitignore. If there is an OUTPUT directory then the tools/perf/pmu-events/arch files are copied to it so the generated extra-metrics.json and extra-metricgroups.json can be added/generated there. Signed-off-by: Ian Rogers --- tools/perf/.gitignore | 2 + tools/perf/Makefile.perf | 23 +++++++--- tools/perf/pmu-events/Build | 62 ++++++++++++++++++++++++-- tools/perf/pmu-events/amd_metrics.py | 38 ++++++++++++++++ tools/perf/pmu-events/arm64_metrics.py | 39 ++++++++++++++++ tools/perf/pmu-events/intel_metrics.py | 38 ++++++++++++++++ 6 files changed, 193 insertions(+), 9 deletions(-) create mode 100755 tools/perf/pmu-events/amd_metrics.py create mode 100755 tools/perf/pmu-events/arm64_metrics.py create mode 100755 tools/perf/pmu-events/intel_metrics.py diff --git a/tools/perf/.gitignore b/tools/perf/.gitignore index f5b81d439387..c9a8da5bfc56 100644 --- a/tools/perf/.gitignore +++ b/tools/perf/.gitignore @@ -39,6 +39,8 @@ trace/beauty/generated/ pmu-events/pmu-events.c pmu-events/jevents pmu-events/metric_test.log +pmu-events/arch/**/extra-metrics.json +pmu-events/arch/**/extra-metricgroups.json tests/shell/*.shellcheck_log tests/shell/coresight/*.shellcheck_log tests/shell/lib/*.shellcheck_log diff --git a/tools/perf/Makefile.perf b/tools/perf/Makefile.perf index 9dd2e8d3f3c9..75b278497526 100644 --- a/tools/perf/Makefile.perf +++ b/tools/perf/Makefile.perf @@ -1265,9 +1265,26 @@ endif # CONFIG_PERF_BPF_SKEL bpf-skel-clean: $(call QUIET_CLEAN, bpf-skel) $(RM) -r $(SKEL_TMP_OUT) $(SKELETONS) $(SKE= L_OUT)/vmlinux.h =20 +pmu-events-clean: +ifeq ($(OUTPUT),) + $(call QUIET_CLEAN, pmu-events) $(RM) \ + pmu-events/pmu-events.c \ + pmu-events/metric_test.log \ + pmu-events/test-empty-pmu-events.c \ + pmu-events/empty-pmu-events.log + $(Q)find pmu-events/arch -name 'extra-metrics.json' -delete -o \ + -name 'extra-metricgroups.json' -delete +else + $(call QUIET_CLEAN, pmu-events) $(RM) -r $(OUTPUT)pmu-events/arch \ + $(OUTPUT)pmu-events/pmu-events.c \ + $(OUTPUT)pmu-events/metric_test.log \ + $(OUTPUT)pmu-events/test-empty-pmu-events.c \ + $(OUTPUT)pmu-events/empty-pmu-events.log +endif + clean:: $(LIBAPI)-clean $(LIBBPF)-clean $(LIBSUBCMD)-clean $(LIBSYMBOL)-cl= ean $(LIBPERF)-clean \ arm64-sysreg-defs-clean fixdep-clean python-clean bpf-skel-clean \ - tests-coresight-targets-clean + tests-coresight-targets-clean pmu-events-clean $(call QUIET_CLEAN, core-objs) $(RM) $(LIBPERF_A) $(OUTPUT)perf-archive \ $(OUTPUT)perf-iostat $(LANG_BINDINGS) $(Q)find $(or $(OUTPUT),.) -name '*.o' -delete -o -name '*.a' -delete -o \ @@ -1280,10 +1297,6 @@ clean:: $(LIBAPI)-clean $(LIBBPF)-clean $(LIBSUBCMD)= -clean $(LIBSYMBOL)-clean $( $(OUTPUT)FEATURE-DUMP $(OUTPUT)util/*-bison* $(OUTPUT)util/*-flex* \ $(OUTPUT)util/intel-pt-decoder/inat-tables.c \ $(OUTPUT)tests/llvm-src-{base,kbuild,prologue,relocation}.c \ - $(OUTPUT)pmu-events/pmu-events.c \ - $(OUTPUT)pmu-events/test-empty-pmu-events.c \ - $(OUTPUT)pmu-events/empty-pmu-events.log \ - $(OUTPUT)pmu-events/metric_test.log \ $(OUTPUT)$(fadvise_advice_array) \ $(OUTPUT)$(fsconfig_arrays) \ $(OUTPUT)$(fsmount_arrays) \ diff --git a/tools/perf/pmu-events/Build b/tools/perf/pmu-events/Build index d941bc9d16e9..d89275c4f485 100644 --- a/tools/perf/pmu-events/Build +++ b/tools/perf/pmu-events/Build @@ -1,7 +1,6 @@ pmu-events-y +=3D pmu-events.o JDIR =3D pmu-events/arch/$(SRCARCH) -JSON =3D $(shell [ -d $(JDIR) ] && \ - find $(JDIR) -name '*.json' -o -name 'mapfile.csv') +JSON =3D $(shell find pmu-events/arch -name *.json -o -name *.csv) JDIR_TEST =3D pmu-events/arch/test JSON_TEST =3D $(shell [ -d $(JDIR_TEST) ] && \ find $(JDIR_TEST) -name '*.json') @@ -29,6 +28,61 @@ $(PMU_EVENTS_C): $(EMPTY_PMU_EVENTS_C) $(call rule_mkdir) $(Q)$(call echo-cmd,gen)cp $< $@ else +# Extract the model from a extra-metrics.json or extra-metricgroups.json p= ath +model_name =3D $(shell echo $(1)|sed -e 's@.\+/\(.*\)/extra-metric.*\.json= @\1@') +vendor_name =3D $(shell echo $(1)|sed -e 's@.\+/\(.*\)/[^/]*/extra-metric.= *\.json@\1@') + +# Copy checked-in json for generation. +$(OUTPUT)pmu-events/arch/%: pmu-events/arch/% + $(call rule_mkdir) + $(Q)$(call echo-cmd,gen)cp $< $@ + +GEN_METRIC_DEPS :=3D pmu-events/metric.py + +# Generate AMD Json +ZENS =3D $(shell ls -d pmu-events/arch/x86/amdzen*) +ZEN_METRICS =3D $(foreach x,$(ZENS),$(OUTPUT)$(x)/extra-metrics.json) +ZEN_METRICGROUPS =3D $(foreach x,$(ZENS),$(OUTPUT)$(x)/extra-metricgroups.= json) + +$(ZEN_METRICS): pmu-events/amd_metrics.py $(GEN_METRIC_DEPS) + $(call rule_mkdir) + $(Q)$(call echo-cmd,gen)$(PYTHON) $< $(call model_name,$@) arch > $@ + +$(ZEN_METRICGROUPS): pmu-events/amd_metrics.py $(GEN_METRIC_DEPS) + $(call rule_mkdir) + $(Q)$(call echo-cmd,gen)$(PYTHON) $< -metricgroups $(call model_name,$@) = arch > $@ + +# Generate ARM Json +ARMS =3D $(shell ls -d pmu-events/arch/arm64/arm/*) +ARM_METRICS =3D $(foreach x,$(ARMS),$(OUTPUT)$(x)/extra-metrics.json) +ARM_METRICGROUPS =3D $(foreach x,$(ARMS),$(OUTPUT)$(x)/extra-metricgroups.= json) + +$(ARM_METRICS): pmu-events/arm64_metrics.py $(GEN_METRIC_DEPS) + $(call rule_mkdir) + $(Q)$(call echo-cmd,gen)$(PYTHON) $< $(call vendor_name,$@) $(call model_= name,$@) arch > $@ + +$(ARM_METRICGROUPS): pmu-events/arm64_metrics.py $(GEN_METRIC_DEPS) + $(call rule_mkdir) + $(Q)$(call echo-cmd,gen)$(PYTHON) $< -metricgroups $(call vendor_name,$@)= $(call model_name,$@) arch > $@ + +# Generate Intel Json +INTELS =3D $(shell ls -d pmu-events/arch/x86/*|grep -v amdzen|grep -v mapf= ile.csv) +INTEL_METRICS =3D $(foreach x,$(INTELS),$(OUTPUT)$(x)/extra-metrics.json) +INTEL_METRICGROUPS =3D $(foreach x,$(INTELS),$(OUTPUT)$(x)/extra-metricgro= ups.json) + +$(INTEL_METRICS): pmu-events/intel_metrics.py $(GEN_METRIC_DEPS) + $(call rule_mkdir) + $(Q)$(call echo-cmd,gen)$(PYTHON) $< $(call model_name,$@) arch > $@ + +$(INTEL_METRICGROUPS): pmu-events/intel_metrics.py $(GEN_METRIC_DEPS) + $(call rule_mkdir) + $(Q)$(call echo-cmd,gen)$(PYTHON) $< -metricgroups $(call model_name,$@) = arch > $@ + +GEN_JSON =3D $(patsubst %,$(OUTPUT)%,$(JSON)) \ + $(ZEN_METRICS) $(ZEN_METRICGROUPS) \ + $(ARM_METRICS) $(ARM_METRICGROUPS) \ + $(INTEL_METRICS) $(INTEL_METRICGROUPS) + $(METRIC_TEST_LOG): $(METRIC_TEST_PY) $(METRIC_PY) $(call rule_mkdir) $(Q)$(call echo-cmd,test)$(PYTHON) $< 2> $@ || (cat $@ && false) @@ -41,9 +95,9 @@ $(EMPTY_PMU_EVENTS_TEST_LOG): $(EMPTY_PMU_EVENTS_C) $(TES= T_EMPTY_PMU_EVENTS_C) $(call rule_mkdir) $(Q)$(call echo-cmd,test)diff -u $^ 2> $@ || (cat $@ && false) =20 -$(PMU_EVENTS_C): $(JSON) $(JSON_TEST) $(JEVENTS_PY) $(METRIC_PY) $(METRIC_= TEST_LOG) $(EMPTY_PMU_EVENTS_TEST_LOG) +$(PMU_EVENTS_C): $(GEN_JSON) $(JSON_TEST) $(JEVENTS_PY) $(METRIC_PY) $(MET= RIC_TEST_LOG) $(EMPTY_PMU_EVENTS_TEST_LOG) $(call rule_mkdir) - $(Q)$(call echo-cmd,gen)$(PYTHON) $(JEVENTS_PY) $(JEVENTS_ARCH) $(JEVENTS= _MODEL) pmu-events/arch $@ + $(Q)$(call echo-cmd,gen)$(PYTHON) $(JEVENTS_PY) $(JEVENTS_ARCH) $(JEVENTS= _MODEL) $(OUTPUT)pmu-events/arch $@ endif =20 # pmu-events.c file is generated in the OUTPUT directory so it needs a diff --git a/tools/perf/pmu-events/amd_metrics.py b/tools/perf/pmu-events/a= md_metrics.py new file mode 100755 index 000000000000..7ab2ee4fdb17 --- /dev/null +++ b/tools/perf/pmu-events/amd_metrics.py @@ -0,0 +1,38 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: (LGPL-2.1 OR BSD-2-Clause) +from metric import (JsonEncodeMetric, JsonEncodeMetricGroupDescriptions, M= etricGroup) +import argparse +import json +import os + +# Global command line arguments. +_args =3D None + +def main() -> None: + global _args + + def dir_path(path: str) -> str: + """Validate path is a directory for argparse.""" + if os.path.isdir(path): + return path + raise argparse.ArgumentTypeError(f'\'{path}\' is not a valid directory= ') + + parser =3D argparse.ArgumentParser(description=3D"AMD perf json generato= r") + parser.add_argument("-metricgroups", help=3D"Generate metricgroups data"= , action=3D'store_true') + parser.add_argument("model", help=3D"e.g. amdzen[123]") + parser.add_argument( + 'events_path', + type=3Ddir_path, + help=3D'Root of tree containing architecture directories containing = json files' + ) + _args =3D parser.parse_args() + + all_metrics =3D MetricGroup("",[]) + + if _args.metricgroups: + print(JsonEncodeMetricGroupDescriptions(all_metrics)) + else: + print(JsonEncodeMetric(all_metrics)) + +if __name__ =3D=3D '__main__': + main() diff --git a/tools/perf/pmu-events/arm64_metrics.py b/tools/perf/pmu-events= /arm64_metrics.py new file mode 100755 index 000000000000..a9f0e6bc751b --- /dev/null +++ b/tools/perf/pmu-events/arm64_metrics.py @@ -0,0 +1,39 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: (LGPL-2.1 OR BSD-2-Clause) +from metric import (JsonEncodeMetric, JsonEncodeMetricGroupDescriptions, M= etricGroup) +import argparse +import json +import os + +# Global command line arguments. +_args =3D None + +def main() -> None: + global _args + + def dir_path(path: str) -> str: + """Validate path is a directory for argparse.""" + if os.path.isdir(path): + return path + raise argparse.ArgumentTypeError(f'\'{path}\' is not a valid directory= ') + + parser =3D argparse.ArgumentParser(description=3D"ARM perf json generato= r") + parser.add_argument("-metricgroups", help=3D"Generate metricgroups data"= , action=3D'store_true') + parser.add_argument("vendor", help=3D"e.g. arm") + parser.add_argument("model", help=3D"e.g. neoverse-n1") + parser.add_argument( + 'events_path', + type=3Ddir_path, + help=3D'Root of tree containing architecture directories containing = json files' + ) + _args =3D parser.parse_args() + + all_metrics =3D MetricGroup("",[]) + + if _args.metricgroups: + print(JsonEncodeMetricGroupDescriptions(all_metrics)) + else: + print(JsonEncodeMetric(all_metrics)) + +if __name__ =3D=3D '__main__': + main() diff --git a/tools/perf/pmu-events/intel_metrics.py b/tools/perf/pmu-events= /intel_metrics.py new file mode 100755 index 000000000000..f004c27640d2 --- /dev/null +++ b/tools/perf/pmu-events/intel_metrics.py @@ -0,0 +1,38 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: (LGPL-2.1 OR BSD-2-Clause) +from metric import (JsonEncodeMetric, JsonEncodeMetricGroupDescriptions, M= etricGroup) +import argparse +import json +import os + +# Global command line arguments. +_args =3D None + +def main() -> None: + global _args + + def dir_path(path: str) -> str: + """Validate path is a directory for argparse.""" + if os.path.isdir(path): + return path + raise argparse.ArgumentTypeError(f'\'{path}\' is not a valid directory= ') + + parser =3D argparse.ArgumentParser(description=3D"Intel perf json genera= tor") + parser.add_argument("-metricgroups", help=3D"Generate metricgroups data"= , action=3D'store_true') + parser.add_argument("model", help=3D"e.g. skylakex") + parser.add_argument( + 'events_path', + type=3Ddir_path, + help=3D'Root of tree containing architecture directories containing = json files' + ) + _args =3D parser.parse_args() + + all_metrics =3D MetricGroup("",[]) + + if _args.metricgroups: + print(JsonEncodeMetricGroupDescriptions(all_metrics)) + else: + print(JsonEncodeMetric(all_metrics)) + +if __name__ =3D=3D '__main__': + main() --=20 2.46.1.824.gd892dcdcdd-goog