[v1] perf jevents: Handle deleted JSONS in out of source builds

[PATCH] perf jevents: Handle deleted JSONS in out of source builds

Posted by James Clark 2 weeks, 4 days ago

The cp command here doesn't remove files that have been removed from the
sourcetree. That means incremental builds can either succeed with stale
events or will fail completely if a stale json file has a broken
reference in it.

Fix it by using rsync instead of cp. legacy-cache.json has to be
excluded as this is a generated file isn't present in the source tree.

This only happens when deleting a JSON file, which has only happened
once since the linked commit. The fixes commit is marked as the origin
of the problem in case any future changes that delete JSONs are back
ported, rather than the first commit that deleted a JSON file.

Reported-by: Mark Brown <broonie@kernel.org>
Closes: https://lore.kernel.org/linux-next/aW5XSAo88_LBPSYI@sirena.org.uk/
Fixes: 4bb55de4ff03 ("perf jevents: Support copying the source json files to OUTPUT")
Signed-off-by: James Clark <james.clark@linaro.org>
---
This is a bit of a hack and I thought that making jevents.py handle
multiple input folders would be a much better solution than this. Then
we could have "gen-pmu-events" for only generated files and "pmu-events"
for only in-tree input files. It would be very clear what's generated
and what's not and all copying rules and special clean rules just
disappear (and this isn't the first time these rules have caused build
issues).

Unfortunately, after a while of trying to modify the script I thought it
was too invasive for now. The script does output per-file at the very
bottom of the logic in process_one_file(), so adding files in another
folder ends up re-emitting section headers when another chunk is output.
Although other parts of the script do build things up in memory before
outputting so it was possible to make those parts work with multiple
folders transparently.
---
 tools/perf/pmu-events/Build | 16 +++++++++++-----
 1 file changed, 11 insertions(+), 5 deletions(-)

diff --git a/tools/perf/pmu-events/Build b/tools/perf/pmu-events/Build
index a46ab7b612df..079d04c4f7d8 100644
--- a/tools/perf/pmu-events/Build
+++ b/tools/perf/pmu-events/Build
@@ -12,13 +12,16 @@ METRIC_TEST_LOG	=  $(OUTPUT)pmu-events/metric_test.log
 TEST_EMPTY_PMU_EVENTS_C = $(OUTPUT)pmu-events/test-empty-pmu-events.c
 EMPTY_PMU_EVENTS_TEST_LOG = $(OUTPUT)pmu-events/empty-pmu-events.log
 LEGACY_CACHE_PY	=  pmu-events/make_legacy_cache.py
-LEGACY_CACHE_JSON = $(OUTPUT)pmu-events/arch/common/common/legacy-cache.json
+LEGACY_CACHE_SYNC_PATH = arch/common/common/legacy-cache.json
+LEGACY_CACHE_JSON = $(OUTPUT)pmu-events/$(LEGACY_CACHE_SYNC_PATH)
 
 ifeq ($(JEVENTS_ARCH),)
 JEVENTS_ARCH=$(SRCARCH)
 endif
 JEVENTS_MODEL ?= all
 
+GEN_JSON = $(patsubst %,$(OUTPUT)%,$(JSON)) $(LEGACY_CACHE_JSON)
+
 #
 # Locate/process JSON files in pmu-events/arch/
 # directory and create tables in pmu-events.c.
@@ -31,17 +34,20 @@ $(PMU_EVENTS_C): $(EMPTY_PMU_EVENTS_C)
 else
 # Copy checked-in json to OUTPUT for generation if it's an out of source build
 ifneq ($(OUTPUT),)
-$(OUTPUT)pmu-events/arch/%: pmu-events/arch/%
+SYNC_PMU_EVENTS := $(OUTPUT)pmu-events/.synced
+$(SYNC_PMU_EVENTS): $(JSON)
 	$(call rule_mkdir)
-	$(Q)$(call echo-cmd,gen)cp $< $@
+	$(Q)$(call echo-cmd,gen)rsync -a --delete --exclude=$(LEGACY_CACHE_SYNC_PATH) \
+		--include="*.json" --include="*.csv" pmu-events/arch $(OUTPUT)pmu-events/
+	$(Q)touch $@
+
+$(GEN_JSON): $(SYNC_PMU_EVENTS)
 endif
 
 $(LEGACY_CACHE_JSON): $(LEGACY_CACHE_PY)
 	$(call rule_mkdir)
 	$(Q)$(call echo-cmd,gen)$(PYTHON) $(LEGACY_CACHE_PY) > $@
 
-GEN_JSON = $(patsubst %,$(OUTPUT)%,$(JSON)) $(LEGACY_CACHE_JSON)
-
 $(METRIC_TEST_LOG): $(METRIC_TEST_PY) $(METRIC_PY)
 	$(call rule_mkdir)
 	$(Q)$(call echo-cmd,test)$(PYTHON) $< 2> $@ || (cat $@ && false)

---
base-commit: 571d29baa07e83e637075239f379f91353c24ec9
change-id: 20260120-james-perf-json-delete-fix-71553b379e8e

Best regards,
-- 
James Clark <james.clark@linaro.org>

Re: [PATCH] perf jevents: Handle deleted JSONS in out of source builds

Posted by Ian Rogers 2 weeks, 4 days ago

On Tue, Jan 20, 2026 at 7:39 AM James Clark <james.clark@linaro.org> wrote:
>
> The cp command here doesn't remove files that have been removed from the
> sourcetree. That means incremental builds can either succeed with stale
> events or will fail completely if a stale json file has a broken
> reference in it.
>
> Fix it by using rsync instead of cp. legacy-cache.json has to be
> excluded as this is a generated file isn't present in the source tree.
>
> This only happens when deleting a JSON file, which has only happened
> once since the linked commit. The fixes commit is marked as the origin
> of the problem in case any future changes that delete JSONs are back
> ported, rather than the first commit that deleted a JSON file.
>
> Reported-by: Mark Brown <broonie@kernel.org>
> Closes: https://lore.kernel.org/linux-next/aW5XSAo88_LBPSYI@sirena.org.uk/
> Fixes: 4bb55de4ff03 ("perf jevents: Support copying the source json files to OUTPUT")
> Signed-off-by: James Clark <james.clark@linaro.org>
> ---
> This is a bit of a hack and I thought that making jevents.py handle
> multiple input folders would be a much better solution than this. Then
> we could have "gen-pmu-events" for only generated files and "pmu-events"
> for only in-tree input files. It would be very clear what's generated
> and what's not and all copying rules and special clean rules just
> disappear (and this isn't the first time these rules have caused build
> issues).
>
> Unfortunately, after a while of trying to modify the script I thought it
> was too invasive for now. The script does output per-file at the very
> bottom of the logic in process_one_file(), so adding files in another
> folder ends up re-emitting section headers when another chunk is output.
> Although other parts of the script do build things up in memory before
> outputting so it was possible to make those parts work with multiple
> folders transparently.

Thanks James!
Acked-by: Ian Rogers <irogers@google.com>
I see other rsync uses in:
tools/testing/selftests/sparc64/Makefile
tools/testing/selftests/bpf/Makefile
but they aren't the most compelling mainstream uses. I wonder whether
we can test for rsync's availability and if not fall back on cp?

Thanks,
Ian

> ---
>  tools/perf/pmu-events/Build | 16 +++++++++++-----
>  1 file changed, 11 insertions(+), 5 deletions(-)
>
> diff --git a/tools/perf/pmu-events/Build b/tools/perf/pmu-events/Build
> index a46ab7b612df..079d04c4f7d8 100644
> --- a/tools/perf/pmu-events/Build
> +++ b/tools/perf/pmu-events/Build
> @@ -12,13 +12,16 @@ METRIC_TEST_LOG     =  $(OUTPUT)pmu-events/metric_test.log
>  TEST_EMPTY_PMU_EVENTS_C = $(OUTPUT)pmu-events/test-empty-pmu-events.c
>  EMPTY_PMU_EVENTS_TEST_LOG = $(OUTPUT)pmu-events/empty-pmu-events.log
>  LEGACY_CACHE_PY        =  pmu-events/make_legacy_cache.py
> -LEGACY_CACHE_JSON = $(OUTPUT)pmu-events/arch/common/common/legacy-cache.json
> +LEGACY_CACHE_SYNC_PATH = arch/common/common/legacy-cache.json
> +LEGACY_CACHE_JSON = $(OUTPUT)pmu-events/$(LEGACY_CACHE_SYNC_PATH)
>
>  ifeq ($(JEVENTS_ARCH),)
>  JEVENTS_ARCH=$(SRCARCH)
>  endif
>  JEVENTS_MODEL ?= all
>
> +GEN_JSON = $(patsubst %,$(OUTPUT)%,$(JSON)) $(LEGACY_CACHE_JSON)
> +
>  #
>  # Locate/process JSON files in pmu-events/arch/
>  # directory and create tables in pmu-events.c.
> @@ -31,17 +34,20 @@ $(PMU_EVENTS_C): $(EMPTY_PMU_EVENTS_C)
>  else
>  # Copy checked-in json to OUTPUT for generation if it's an out of source build
>  ifneq ($(OUTPUT),)
> -$(OUTPUT)pmu-events/arch/%: pmu-events/arch/%
> +SYNC_PMU_EVENTS := $(OUTPUT)pmu-events/.synced
> +$(SYNC_PMU_EVENTS): $(JSON)
>         $(call rule_mkdir)
> -       $(Q)$(call echo-cmd,gen)cp $< $@
> +       $(Q)$(call echo-cmd,gen)rsync -a --delete --exclude=$(LEGACY_CACHE_SYNC_PATH) \
> +               --include="*.json" --include="*.csv" pmu-events/arch $(OUTPUT)pmu-events/
> +       $(Q)touch $@
> +
> +$(GEN_JSON): $(SYNC_PMU_EVENTS)
>  endif
>
>  $(LEGACY_CACHE_JSON): $(LEGACY_CACHE_PY)
>         $(call rule_mkdir)
>         $(Q)$(call echo-cmd,gen)$(PYTHON) $(LEGACY_CACHE_PY) > $@
>
> -GEN_JSON = $(patsubst %,$(OUTPUT)%,$(JSON)) $(LEGACY_CACHE_JSON)
> -
>  $(METRIC_TEST_LOG): $(METRIC_TEST_PY) $(METRIC_PY)
>         $(call rule_mkdir)
>         $(Q)$(call echo-cmd,test)$(PYTHON) $< 2> $@ || (cat $@ && false)
>
> ---
> base-commit: 571d29baa07e83e637075239f379f91353c24ec9
> change-id: 20260120-james-perf-json-delete-fix-71553b379e8e
>
> Best regards,
> --
> James Clark <james.clark@linaro.org>
>

Re: [PATCH] perf jevents: Handle deleted JSONS in out of source builds

Posted by Arnaldo Carvalho de Melo 2 weeks, 4 days ago

On Tue, Jan 20, 2026 at 10:01:52AM -0800, Ian Rogers wrote:
> On Tue, Jan 20, 2026 at 7:39 AM James Clark <james.clark@linaro.org> wrote:
> >
> > The cp command here doesn't remove files that have been removed from the
> > sourcetree. That means incremental builds can either succeed with stale
> > events or will fail completely if a stale json file has a broken
> > reference in it.
> >
> > Fix it by using rsync instead of cp. legacy-cache.json has to be
> > excluded as this is a generated file isn't present in the source tree.
> >
> > This only happens when deleting a JSON file, which has only happened
> > once since the linked commit. The fixes commit is marked as the origin
> > of the problem in case any future changes that delete JSONs are back
> > ported, rather than the first commit that deleted a JSON file.
> >
> > Reported-by: Mark Brown <broonie@kernel.org>
> > Closes: https://lore.kernel.org/linux-next/aW5XSAo88_LBPSYI@sirena.org.uk/
> > Fixes: 4bb55de4ff03 ("perf jevents: Support copying the source json files to OUTPUT")
> > Signed-off-by: James Clark <james.clark@linaro.org>
> > ---
> > This is a bit of a hack and I thought that making jevents.py handle
> > multiple input folders would be a much better solution than this. Then
> > we could have "gen-pmu-events" for only generated files and "pmu-events"
> > for only in-tree input files. It would be very clear what's generated
> > and what's not and all copying rules and special clean rules just
> > disappear (and this isn't the first time these rules have caused build
> > issues).
> >
> > Unfortunately, after a while of trying to modify the script I thought it
> > was too invasive for now. The script does output per-file at the very
> > bottom of the logic in process_one_file(), so adding files in another
> > folder ends up re-emitting section headers when another chunk is output.
> > Although other parts of the script do build things up in memory before
> > outputting so it was possible to make those parts work with multiple
> > folders transparently.
> 
> Thanks James!
> Acked-by: Ian Rogers <irogers@google.com>
> I see other rsync uses in:
> tools/testing/selftests/sparc64/Makefile
> tools/testing/selftests/bpf/Makefile
> but they aren't the most compelling mainstream uses. I wonder whether
> we can test for rsync's availability and if not fall back on cp?

It is not mentioned at all in Documentation, so probably its best not to
add a requirement for it?

- Arnaldo

Re: [PATCH] perf jevents: Handle deleted JSONS in out of source builds

Posted by James Clark 2 weeks, 4 days ago


On 1/20/26 6:54 PM, Arnaldo Carvalho de Melo wrote:
> On Tue, Jan 20, 2026 at 10:01:52AM -0800, Ian Rogers wrote:
>> On Tue, Jan 20, 2026 at 7:39 AM James Clark <james.clark@linaro.org> wrote:
>>>
>>> The cp command here doesn't remove files that have been removed from the
>>> sourcetree. That means incremental builds can either succeed with stale
>>> events or will fail completely if a stale json file has a broken
>>> reference in it.
>>>
>>> Fix it by using rsync instead of cp. legacy-cache.json has to be
>>> excluded as this is a generated file isn't present in the source tree.
>>>
>>> This only happens when deleting a JSON file, which has only happened
>>> once since the linked commit. The fixes commit is marked as the origin
>>> of the problem in case any future changes that delete JSONs are back
>>> ported, rather than the first commit that deleted a JSON file.
>>>
>>> Reported-by: Mark Brown <broonie@kernel.org>
>>> Closes: https://lore.kernel.org/linux-next/aW5XSAo88_LBPSYI@sirena.org.uk/
>>> Fixes: 4bb55de4ff03 ("perf jevents: Support copying the source json files to OUTPUT")
>>> Signed-off-by: James Clark <james.clark@linaro.org>
>>> ---
>>> This is a bit of a hack and I thought that making jevents.py handle
>>> multiple input folders would be a much better solution than this. Then
>>> we could have "gen-pmu-events" for only generated files and "pmu-events"
>>> for only in-tree input files. It would be very clear what's generated
>>> and what's not and all copying rules and special clean rules just
>>> disappear (and this isn't the first time these rules have caused build
>>> issues).
>>>
>>> Unfortunately, after a while of trying to modify the script I thought it
>>> was too invasive for now. The script does output per-file at the very
>>> bottom of the logic in process_one_file(), so adding files in another
>>> folder ends up re-emitting section headers when another chunk is output.
>>> Although other parts of the script do build things up in memory before
>>> outputting so it was possible to make those parts work with multiple
>>> folders transparently.
>>
>> Thanks James!
>> Acked-by: Ian Rogers <irogers@google.com>
>> I see other rsync uses in:
>> tools/testing/selftests/sparc64/Makefile
>> tools/testing/selftests/bpf/Makefile
>> but they aren't the most compelling mainstream uses. I wonder whether
>> we can test for rsync's availability and if not fall back on cp?

That will work if we completely wipe the destination directory every 
time. But it would be an untested path, because in reality everyone is 
going to have rsync. It might be better to re-implement rsync and then 
at least the same thing runs everywhere.

> 
> It is not mentioned at all in Documentation, so probably its best not to
> add a requirement for it?
> 
> - Arnaldo

I can hack together something that does the delete in a few lines of 
bash or make then? I did consider that originally but I thought that's 
what rsync does so I'll just use it.

Re: [PATCH] perf jevents: Handle deleted JSONS in out of source builds

Posted by Arnaldo Carvalho de Melo 1 week, 5 days ago

On Wed, Jan 21, 2026 at 09:51:30AM +0000, James Clark wrote:
> On 1/20/26 6:54 PM, Arnaldo Carvalho de Melo wrote:
> > On Tue, Jan 20, 2026 at 10:01:52AM -0800, Ian Rogers wrote:
> > > On Tue, Jan 20, 2026 at 7:39 AM James Clark <james.clark@linaro.org> wrote:
> > > > The cp command here doesn't remove files that have been removed from the
> > > > sourcetree. That means incremental builds can either succeed with stale
> > > > events or will fail completely if a stale json file has a broken
> > > > reference in it.

> > > > Fix it by using rsync instead of cp. legacy-cache.json has to be
> > > > excluded as this is a generated file isn't present in the source tree.

> > > > This only happens when deleting a JSON file, which has only happened
> > > > once since the linked commit. The fixes commit is marked as the origin
> > > > of the problem in case any future changes that delete JSONs are back
> > > > ported, rather than the first commit that deleted a JSON file.

> > > > Reported-by: Mark Brown <broonie@kernel.org>
> > > > Closes: https://lore.kernel.org/linux-next/aW5XSAo88_LBPSYI@sirena.org.uk/
> > > > Fixes: 4bb55de4ff03 ("perf jevents: Support copying the source json files to OUTPUT")
> > > > Signed-off-by: James Clark <james.clark@linaro.org>
> > > > ---
> > > > This is a bit of a hack and I thought that making jevents.py handle
> > > > multiple input folders would be a much better solution than this. Then
> > > > we could have "gen-pmu-events" for only generated files and "pmu-events"
> > > > for only in-tree input files. It would be very clear what's generated
> > > > and what's not and all copying rules and special clean rules just
> > > > disappear (and this isn't the first time these rules have caused build
> > > > issues).
> > > > 
> > > > Unfortunately, after a while of trying to modify the script I thought it
> > > > was too invasive for now. The script does output per-file at the very
> > > > bottom of the logic in process_one_file(), so adding files in another
> > > > folder ends up re-emitting section headers when another chunk is output.
> > > > Although other parts of the script do build things up in memory before
> > > > outputting so it was possible to make those parts work with multiple
> > > > folders transparently.
> > > 
> > > Thanks James!
> > > Acked-by: Ian Rogers <irogers@google.com>
> > > I see other rsync uses in:
> > > tools/testing/selftests/sparc64/Makefile
> > > tools/testing/selftests/bpf/Makefile
> > > but they aren't the most compelling mainstream uses. I wonder whether
> > > we can test for rsync's availability and if not fall back on cp?
 
> That will work if we completely wipe the destination directory every time.
> But it would be an untested path, because in reality everyone is going to
> have rsync. It might be better to re-implement rsync and then at least the
> same thing runs everywhere.
 
> > It is not mentioned at all in Documentation, so probably its best not to
> > add a requirement for it?
 
> I can hack together something that does the delete in a few lines of bash or
> make then? I did consider that originally but I thought that's what rsync
> does so I'll just use it.

Did this progress?

- Arnaldo

Re: [PATCH] perf jevents: Handle deleted JSONS in out of source builds

Posted by James Clark 1 week, 5 days ago


On 26/01/2026 8:45 pm, Arnaldo Carvalho de Melo wrote:
> On Wed, Jan 21, 2026 at 09:51:30AM +0000, James Clark wrote:
>> On 1/20/26 6:54 PM, Arnaldo Carvalho de Melo wrote:
>>> On Tue, Jan 20, 2026 at 10:01:52AM -0800, Ian Rogers wrote:
>>>> On Tue, Jan 20, 2026 at 7:39 AM James Clark <james.clark@linaro.org> wrote:
>>>>> The cp command here doesn't remove files that have been removed from the
>>>>> sourcetree. That means incremental builds can either succeed with stale
>>>>> events or will fail completely if a stale json file has a broken
>>>>> reference in it.
> 
>>>>> Fix it by using rsync instead of cp. legacy-cache.json has to be
>>>>> excluded as this is a generated file isn't present in the source tree.
> 
>>>>> This only happens when deleting a JSON file, which has only happened
>>>>> once since the linked commit. The fixes commit is marked as the origin
>>>>> of the problem in case any future changes that delete JSONs are back
>>>>> ported, rather than the first commit that deleted a JSON file.
> 
>>>>> Reported-by: Mark Brown <broonie@kernel.org>
>>>>> Closes: https://lore.kernel.org/linux-next/aW5XSAo88_LBPSYI@sirena.org.uk/
>>>>> Fixes: 4bb55de4ff03 ("perf jevents: Support copying the source json files to OUTPUT")
>>>>> Signed-off-by: James Clark <james.clark@linaro.org>
>>>>> ---
>>>>> This is a bit of a hack and I thought that making jevents.py handle
>>>>> multiple input folders would be a much better solution than this. Then
>>>>> we could have "gen-pmu-events" for only generated files and "pmu-events"
>>>>> for only in-tree input files. It would be very clear what's generated
>>>>> and what's not and all copying rules and special clean rules just
>>>>> disappear (and this isn't the first time these rules have caused build
>>>>> issues).
>>>>>
>>>>> Unfortunately, after a while of trying to modify the script I thought it
>>>>> was too invasive for now. The script does output per-file at the very
>>>>> bottom of the logic in process_one_file(), so adding files in another
>>>>> folder ends up re-emitting section headers when another chunk is output.
>>>>> Although other parts of the script do build things up in memory before
>>>>> outputting so it was possible to make those parts work with multiple
>>>>> folders transparently.
>>>>
>>>> Thanks James!
>>>> Acked-by: Ian Rogers <irogers@google.com>
>>>> I see other rsync uses in:
>>>> tools/testing/selftests/sparc64/Makefile
>>>> tools/testing/selftests/bpf/Makefile
>>>> but they aren't the most compelling mainstream uses. I wonder whether
>>>> we can test for rsync's availability and if not fall back on cp?
>   
>> That will work if we completely wipe the destination directory every time.
>> But it would be an untested path, because in reality everyone is going to
>> have rsync. It might be better to re-implement rsync and then at least the
>> same thing runs everywhere.
>   
>>> It is not mentioned at all in Documentation, so probably its best not to
>>> add a requirement for it?
>   
>> I can hack together something that does the delete in a few lines of bash or
>> make then? I did consider that originally but I thought that's what rsync
>> does so I'll just use it.
> 
> Did this progress?
> 
> - Arnaldo

Yes I sent a v2 here: 
https://lore.kernel.org/linux-perf-users/20260121-james-perf-json-delete-fix-v2-1-2e10f895bfd7@linaro.org/

If it looks ok to you it should be good to apply.