[v2] perf jevents: Handle deleted JSONS in out of source builds

[PATCH v2] perf jevents: Handle deleted JSONS in out of source builds

Posted by James Clark 2 weeks, 3 days ago

Make the source folders a dependency for the generated folder root so
that whenever a file is deleted from the source it will force a new
fresh copy of all the JSON files and avoid stale deleted files.

JSON_DIRS_OUTPUT_ROOT needs to be a dependency of LEGACY_CACHE_JSON so
that the root folder doesn't get cleaned after the legacy JSON is
generated. But this is a no-op with in-source builds as
JSON_DIRS_OUTPUT_ROOT is unset.

JSON_DIRS is added as a dependency of PMU_EVENTS_C which also forces a
re-build for in source builds when JSON files are deleted. This could
have also resulted in stale builds, but never a broken one.

Reported-by: Mark Brown <broonie@kernel.org>
Closes: https://lore.kernel.org/linux-next/aW5XSAo88_LBPSYI@sirena.org.uk/
Fixes: 4bb55de4ff03 ("perf jevents: Support copying the source json files to OUTPUT")
Signed-off-by: James Clark <james.clark@linaro.org>
---
This is a bit of a hack and I thought that making jevents.py handle
multiple input folders would be a much better solution than this. Then
we could have "gen-pmu-events" for only generated files and "pmu-events"
for only in-tree input files. It would be very clear what's generated
and what's not and all copying rules and special clean rules just
disappear (and this isn't the first time these rules have caused build
issues).

Unfortunately, after a while of trying to modify the script I thought it
was too invasive for now. The script does output per-file at the very
bottom of the logic in process_one_file(), so adding files in another
folder ends up re-emitting section headers when another chunk is output.
Although other parts of the script do build things up in memory before
outputting so it was possible to make those parts work with multiple
folders transparently.
---
Changes in v2:
- Drop dependency on rsync by wiping the output folder any time a source
  folder is updated.
- Also fix the same issue for in source builds by adding the source
  folders as a dependency of PMU_EVENTS_C.
- Link to v1: https://lore.kernel.org/r/20260120-james-perf-json-delete-fix-v1-1-bae2194b1dcf@linaro.org
---
 tools/perf/pmu-events/Build | 14 +++++++++++---
 1 file changed, 11 insertions(+), 3 deletions(-)

diff --git a/tools/perf/pmu-events/Build b/tools/perf/pmu-events/Build
index a46ab7b612df..4f9ef624ba70 100644
--- a/tools/perf/pmu-events/Build
+++ b/tools/perf/pmu-events/Build
@@ -1,5 +1,6 @@
 pmu-events-y	+= pmu-events.o
 JSON		=  $(shell find pmu-events/arch -name '*.json' -o -name '*.csv')
+JSON_DIRS	=  $(shell find pmu-events/arch -type d)
 JDIR_TEST	=  pmu-events/arch/test
 JSON_TEST	=  $(shell [ -d $(JDIR_TEST) ] &&			\
 			find $(JDIR_TEST) -name '*.json')
@@ -31,16 +32,23 @@ $(PMU_EVENTS_C): $(EMPTY_PMU_EVENTS_C)
 else
 # Copy checked-in json to OUTPUT for generation if it's an out of source build
 ifneq ($(OUTPUT),)
-$(OUTPUT)pmu-events/arch/%: pmu-events/arch/%
+# Remove all output directories when any source directory timestamp changes
+# so there are no stale deleted files
+JSON_DIRS_ROOT = $(OUTPUT)pmu-events/arch/
+$(JSON_DIRS_ROOT): $(JSON_DIRS)
+	$(Q)$(call echo-cmd,gen)rm -rf $@
+	$(Q)mkdir -p $@
+
+$(OUTPUT)pmu-events/arch/%: pmu-events/arch/% $(JSON_DIRS_ROOT)
 	$(call rule_mkdir)
 	$(Q)$(call echo-cmd,gen)cp $< $@
 endif
 
-$(LEGACY_CACHE_JSON): $(LEGACY_CACHE_PY)
+$(LEGACY_CACHE_JSON): $(LEGACY_CACHE_PY) $(JSON_DIRS_ROOT)
 	$(call rule_mkdir)
 	$(Q)$(call echo-cmd,gen)$(PYTHON) $(LEGACY_CACHE_PY) > $@
 
-GEN_JSON = $(patsubst %,$(OUTPUT)%,$(JSON)) $(LEGACY_CACHE_JSON)
+GEN_JSON = $(patsubst %,$(OUTPUT)%,$(JSON)) $(LEGACY_CACHE_JSON) $(JSON_DIRS)
 
 $(METRIC_TEST_LOG): $(METRIC_TEST_PY) $(METRIC_PY)
 	$(call rule_mkdir)

---
base-commit: 571d29baa07e83e637075239f379f91353c24ec9
change-id: 20260120-james-perf-json-delete-fix-71553b379e8e

Best regards,
-- 
James Clark <james.clark@linaro.org>

Re: [PATCH v2] perf jevents: Handle deleted JSONS in out of source builds

Posted by Arnaldo Carvalho de Melo 1 week, 4 days ago

On Wed, Jan 21, 2026 at 04:19:40PM +0000, James Clark wrote:
> Make the source folders a dependency for the generated folder root so
> that whenever a file is deleted from the source it will force a new
> fresh copy of all the JSON files and avoid stale deleted files.
> 
> JSON_DIRS_OUTPUT_ROOT needs to be a dependency of LEGACY_CACHE_JSON so
> that the root folder doesn't get cleaned after the legacy JSON is
> generated. But this is a no-op with in-source builds as
> JSON_DIRS_OUTPUT_ROOT is unset.
> 
> JSON_DIRS is added as a dependency of PMU_EVENTS_C which also forces a
> re-build for in source builds when JSON files are deleted. This could
> have also resulted in stale builds, but never a broken one.

Thanks, applied to perf-tools-next,

- Arnaldo
 
> Reported-by: Mark Brown <broonie@kernel.org>
> Closes: https://lore.kernel.org/linux-next/aW5XSAo88_LBPSYI@sirena.org.uk/
> Fixes: 4bb55de4ff03 ("perf jevents: Support copying the source json files to OUTPUT")
> Signed-off-by: James Clark <james.clark@linaro.org>
> ---
> This is a bit of a hack and I thought that making jevents.py handle
> multiple input folders would be a much better solution than this. Then
> we could have "gen-pmu-events" for only generated files and "pmu-events"
> for only in-tree input files. It would be very clear what's generated
> and what's not and all copying rules and special clean rules just
> disappear (and this isn't the first time these rules have caused build
> issues).
> 
> Unfortunately, after a while of trying to modify the script I thought it
> was too invasive for now. The script does output per-file at the very
> bottom of the logic in process_one_file(), so adding files in another
> folder ends up re-emitting section headers when another chunk is output.
> Although other parts of the script do build things up in memory before
> outputting so it was possible to make those parts work with multiple
> folders transparently.
> ---
> Changes in v2:
> - Drop dependency on rsync by wiping the output folder any time a source
>   folder is updated.
> - Also fix the same issue for in source builds by adding the source
>   folders as a dependency of PMU_EVENTS_C.
> - Link to v1: https://lore.kernel.org/r/20260120-james-perf-json-delete-fix-v1-1-bae2194b1dcf@linaro.org
> ---
>  tools/perf/pmu-events/Build | 14 +++++++++++---
>  1 file changed, 11 insertions(+), 3 deletions(-)
> 
> diff --git a/tools/perf/pmu-events/Build b/tools/perf/pmu-events/Build
> index a46ab7b612df..4f9ef624ba70 100644
> --- a/tools/perf/pmu-events/Build
> +++ b/tools/perf/pmu-events/Build
> @@ -1,5 +1,6 @@
>  pmu-events-y	+= pmu-events.o
>  JSON		=  $(shell find pmu-events/arch -name '*.json' -o -name '*.csv')
> +JSON_DIRS	=  $(shell find pmu-events/arch -type d)
>  JDIR_TEST	=  pmu-events/arch/test
>  JSON_TEST	=  $(shell [ -d $(JDIR_TEST) ] &&			\
>  			find $(JDIR_TEST) -name '*.json')
> @@ -31,16 +32,23 @@ $(PMU_EVENTS_C): $(EMPTY_PMU_EVENTS_C)
>  else
>  # Copy checked-in json to OUTPUT for generation if it's an out of source build
>  ifneq ($(OUTPUT),)
> -$(OUTPUT)pmu-events/arch/%: pmu-events/arch/%
> +# Remove all output directories when any source directory timestamp changes
> +# so there are no stale deleted files
> +JSON_DIRS_ROOT = $(OUTPUT)pmu-events/arch/
> +$(JSON_DIRS_ROOT): $(JSON_DIRS)
> +	$(Q)$(call echo-cmd,gen)rm -rf $@
> +	$(Q)mkdir -p $@
> +
> +$(OUTPUT)pmu-events/arch/%: pmu-events/arch/% $(JSON_DIRS_ROOT)
>  	$(call rule_mkdir)
>  	$(Q)$(call echo-cmd,gen)cp $< $@
>  endif
>  
> -$(LEGACY_CACHE_JSON): $(LEGACY_CACHE_PY)
> +$(LEGACY_CACHE_JSON): $(LEGACY_CACHE_PY) $(JSON_DIRS_ROOT)
>  	$(call rule_mkdir)
>  	$(Q)$(call echo-cmd,gen)$(PYTHON) $(LEGACY_CACHE_PY) > $@
>  
> -GEN_JSON = $(patsubst %,$(OUTPUT)%,$(JSON)) $(LEGACY_CACHE_JSON)
> +GEN_JSON = $(patsubst %,$(OUTPUT)%,$(JSON)) $(LEGACY_CACHE_JSON) $(JSON_DIRS)
>  
>  $(METRIC_TEST_LOG): $(METRIC_TEST_PY) $(METRIC_PY)
>  	$(call rule_mkdir)
> 
> ---
> base-commit: 571d29baa07e83e637075239f379f91353c24ec9
> change-id: 20260120-james-perf-json-delete-fix-71553b379e8e
> 
> Best regards,
> -- 
> James Clark <james.clark@linaro.org>