[PATCH v4 00/24] Tracefs support for pKVM

Vincent Donnefort posted 24 patches 7 months, 2 weeks ago
There is a newer version of this series
arch/arm64/include/asm/kvm_asm.h              |   8 +
arch/arm64/include/asm/kvm_define_hypevents.h |  21 +
arch/arm64/include/asm/kvm_hyp.h              |   1 -
arch/arm64/include/asm/kvm_hypevents.h        |  41 +
arch/arm64/include/asm/kvm_hyptrace.h         |  26 +
arch/arm64/include/asm/sections.h             |   1 +
arch/arm64/kernel/image-vars.h                |   6 +
arch/arm64/kernel/vmlinux.lds.S               |  36 +-
arch/arm64/kvm/Kconfig                        |  18 +
arch/arm64/kvm/Makefile                       |   2 +
arch/arm64/kvm/arm.c                          |  11 +
arch/arm64/kvm/hyp/include/nvhe/arm-smccc.h   |  13 +
arch/arm64/kvm/hyp/include/nvhe/clock.h       |  16 +
.../kvm/hyp/include/nvhe/define_events.h      |  21 +
arch/arm64/kvm/hyp/include/nvhe/trace.h       |  59 ++
arch/arm64/kvm/hyp/nvhe/Makefile              |   1 +
arch/arm64/kvm/hyp/nvhe/clock.c               |  65 ++
arch/arm64/kvm/hyp/nvhe/events.c              |  36 +
arch/arm64/kvm/hyp/nvhe/ffa.c                 |   2 +-
arch/arm64/kvm/hyp/nvhe/hyp-main.c            |  86 ++
arch/arm64/kvm/hyp/nvhe/hyp.lds.S             |   8 +
arch/arm64/kvm/hyp/nvhe/mm.c                  |   2 +-
arch/arm64/kvm/hyp/nvhe/psci-relay.c          |   5 +-
arch/arm64/kvm/hyp/nvhe/setup.c               |   4 +
arch/arm64/kvm/hyp/nvhe/switch.c              |   5 +-
arch/arm64/kvm/hyp/nvhe/trace.c               | 286 ++++++
arch/arm64/kvm/hyp_trace.c                    | 385 ++++++++
arch/arm64/kvm/hyp_trace.h                    |  11 +
arch/arm64/kvm/pkvm.c                         |   1 +
include/linux/ring_buffer.h                   | 102 +-
include/linux/simple_ring_buffer.h            |  57 ++
include/linux/trace_remote.h                  |  27 +
include/linux/trace_remote_event.h            |  33 +
include/trace/define_remote_events.h          |  73 ++
kernel/trace/Kconfig                          |  14 +
kernel/trace/Makefile                         |  19 +
kernel/trace/remote_test.c                    | 259 +++++
kernel/trace/remote_test_events.h             |  10 +
kernel/trace/ring_buffer.c                    | 254 ++++-
kernel/trace/simple_ring_buffer.c             | 404 ++++++++
kernel/trace/trace.c                          |   2 +-
kernel/trace/trace.h                          |   6 +
kernel/trace/trace_remote.c                   | 920 ++++++++++++++++++
.../ftrace/test.d/remotes/buffer_size.tc      |  24 +
.../selftests/ftrace/test.d/remotes/functions |  33 +
.../ftrace/test.d/remotes/pkvm/buffer_size.tc |  10 +
.../ftrace/test.d/remotes/pkvm/reset.tc       |  10 +
.../ftrace/test.d/remotes/pkvm/trace_pipe.tc  |  10 +
.../ftrace/test.d/remotes/pkvm/unloading.tc   |  10 +
.../selftests/ftrace/test.d/remotes/reset.tc  | 105 ++
.../ftrace/test.d/remotes/trace_pipe.tc       |  57 ++
.../ftrace/test.d/remotes/unloading.tc        |  36 +
52 files changed, 3599 insertions(+), 53 deletions(-)
create mode 100644 arch/arm64/include/asm/kvm_define_hypevents.h
create mode 100644 arch/arm64/include/asm/kvm_hypevents.h
create mode 100644 arch/arm64/include/asm/kvm_hyptrace.h
create mode 100644 arch/arm64/kvm/hyp/include/nvhe/arm-smccc.h
create mode 100644 arch/arm64/kvm/hyp/include/nvhe/clock.h
create mode 100644 arch/arm64/kvm/hyp/include/nvhe/define_events.h
create mode 100644 arch/arm64/kvm/hyp/include/nvhe/trace.h
create mode 100644 arch/arm64/kvm/hyp/nvhe/clock.c
create mode 100644 arch/arm64/kvm/hyp/nvhe/events.c
create mode 100644 arch/arm64/kvm/hyp/nvhe/trace.c
create mode 100644 arch/arm64/kvm/hyp_trace.c
create mode 100644 arch/arm64/kvm/hyp_trace.h
create mode 100644 include/linux/simple_ring_buffer.h
create mode 100644 include/linux/trace_remote.h
create mode 100644 include/linux/trace_remote_event.h
create mode 100644 include/trace/define_remote_events.h
create mode 100644 kernel/trace/remote_test.c
create mode 100644 kernel/trace/remote_test_events.h
create mode 100644 kernel/trace/simple_ring_buffer.c
create mode 100644 kernel/trace/trace_remote.c
create mode 100644 tools/testing/selftests/ftrace/test.d/remotes/buffer_size.tc
create mode 100644 tools/testing/selftests/ftrace/test.d/remotes/functions
create mode 100644 tools/testing/selftests/ftrace/test.d/remotes/pkvm/buffer_size.tc
create mode 100644 tools/testing/selftests/ftrace/test.d/remotes/pkvm/reset.tc
create mode 100644 tools/testing/selftests/ftrace/test.d/remotes/pkvm/trace_pipe.tc
create mode 100644 tools/testing/selftests/ftrace/test.d/remotes/pkvm/unloading.tc
create mode 100644 tools/testing/selftests/ftrace/test.d/remotes/reset.tc
create mode 100644 tools/testing/selftests/ftrace/test.d/remotes/trace_pipe.tc
create mode 100644 tools/testing/selftests/ftrace/test.d/remotes/unloading.tc
[PATCH v4 00/24] Tracefs support for pKVM
Posted by Vincent Donnefort 7 months, 2 weeks ago
The growing set of features supported by the hypervisor in protected
mode necessitates debugging and profiling tools. Tracefs is the
ideal candidate for this task:

  * It is simple to use and to script.

  * It is supported by various tools, from the trace-cmd CLI to the
    Android web-based perfetto.

  * The ring-buffer, where are stored trace events consists of linked
    pages, making it an ideal structure for sharing between kernel and
    hypervisor.

This series first introduces a new generic way of creating remote events and
remote buffers. Then it adds support to the pKVM hypervisor.

1. ring-buffer
--------------

To setup the per-cpu ring-buffers, a new interface is created:

  ring_buffer_remote:	Describes what the kernel needs to know about the
			remote writer, that is, the set of pages forming the
			ring-buffer and a callback for the reader/head
			swapping (enables consuming read)

  ring_buffer_remote():	Creates a read-only ring-buffer from a
			ring_buffer_remote.

To keep the internals of `struct ring_buffer` in sync with the remote,
the meta-page is used. It was originally introduced to enable user-space
mapping of the ring-buffer [1]. In this case, the kernel is not the
producer anymore but the reader. The function to read that meta-page is:

  ring_buffer_poll_remote():
			Update `struct ring_buffer` based on the remote
			meta-page. Wake-up readers if necessary.

The kernel has to poll the meta-page to be notified of newly written
events.

2. Tracefs
----------

This series introduce a new trace_remote that does the link between
tracefs and the remote ring-buffer.

The interface is found in the remotes/ directory at the root of the
tracefs mount point. Each remote is like an instance and you'll find
there a subset of the regular Tracefs user-space interface:

  remotes/test/
     buffer_size_kb
     trace_clock
     trace_pipe
     trace
     per_cpu/
             cpuX/
                 trace
                 trace_pipe
     events/

            test/
                selftest/
                          enable
                          id

Behind the scenes, kernel/trace/trace_remote.c creates this tracefs
hierarchy without relying on kernel/trace/trace.c. This is due to
fundamental differences:

  * Remote tracing doesn't support trace_array's system-specific
    features (snapshots, tracers, etc.).

  * Logged event formats differ (e.g., no PID for remote events).

  * Buffer operations require specific remote interactions.

3. Simple Ring-Buffer
---------------------

As the current ring-buffer.c implementation has too many dependencies to
be used directly by the pKVM hypervisor. A new simple implementation is
created and can be found in kernel/trace/simple-ring-buffer.c.

This implementation is write-only and is used by both the pKVM
hypervisor and a trace_remote test module.

4. Events
---------

A new REMOTE_EVENT() macro is added to simplify the creation of events
on the kernel side. As remote tracing buffer are read only, only the
event structure and a way of printing must be declared. The prototype of
the macro is very similar to the well-known TRACE_EVENT()

 REMOTE_EVENT(my_event, id,
     RE_STRUCT(
         re_field(u64, foobar)
     ),
     RE_PRINTK("foobar=%lld", __entry->foobar)
     )
  )

5. pKVM
-------

The pKVM support simply creates a "hypervisor" trace_remote on the
kernel side and inherits from simple-ring-buffer.c on the hypervisor
side.

A new event macro is created HYP_EVENT() that is under the hood re-using
REMOTE_EVENT() (defined in the previous paragaph) as well as generate
hypervisor specific struct and trace_<event>() functions.

5. Limitations:
---------------

Non-consuming reading of the buffer isn't supported (i.e. cat trace ->
-EPERM) due to current the lack of support in the ring-buffer meta-page.

[1] https://tracingsummit.org/ts/2022/hypervisortracing/
[2] https://lore.kernel.org/all/20240510140435.3550353-1-vdonnefort@google.com/

Changes since v3 (https://lore.kernel.org/all/20250224121353.98697-1-vdonnefort@google.com/)

  - Move tracefs support from kvm/hyp_trace.c into a generic trace_remote.c.
  - Move ring-buffer implementation from nvhe/trace.c into  a generic
    simple-ring-buffer.c
  - Rebase on 6.15-rc4.

Changes since v2 (https://lore.kernel.org/all/20250108114536.627715-1-vdonnefort@google.com/)

  - Fix ring-buffer remote reset
  - Fix fast-forward in rb_page_desc()
  - Refactor nvhe/trace.c
  - struct hyp_buffer_page more compact
  - Add a struct_len to trace_page_desc
  - Extend reset testing
  - Rebase on 6.14-rc3

Changes since v1 (https://lore.kernel.org/all/20240911093029.3279154-1-vdonnefort@google.com/)

  - Add 128-bits mult fallback in the unlikely event of an overflow. (John)
  - Fix ELF section sort.
  - __always_inline trace_* event macros.
  - Fix events/<event>/enable permissions.
  - Rename ring-buffer "writer" to "remote".
  - Rename CONFIG_PROTECTED_NVHE_TESTING to PKVM_SELFTEST to align with
    Quentin's upcoming selftest
  - Rebase on 6.13-rc3.

Changes since RFC (https://lore.kernel.org/all/20240805173234.3542917-1-vdonnefort@google.com/)

  - hypervisor trace clock:
     - mult/shift computed in hyp_trace.c. (John)
     - Update clock when it deviates from kernel boot clock. (John)
     - Add trace_clock file.
     - Separate patch for better readability.
  - Add a proper reset interface which does not need to teardown the
    tracing buffers. (Steven)
  - Return -EPERM on trace access. (Steven)
  - Add per-cpu trace file.
  - Automatically teardown and free the tracing buffer when it is empty,
    without readers and not currently tracing.
  - Show in buffer_size_kb if the buffer is loaded in the hypervisor or
    not.
  - Extend tests to cover reset and unload.
  - CC timekeeping folks on relevant patches (Marc)

Vincent Donnefort (24):
  ring-buffer: Introduce ring-buffer remotes
  tracing: Introduce trace remotes
  tracing: Add reset to trace remotes
  tracing: Add init callback to trace remotes
  tracing: Add events to trace remotes
  tracing: Add events/ root files to trace remotes
  tracing: Add helpers to create trace remote events
  ring-buffer: Expose buffer_data_page material
  tracing: Introduce simple_ring_buffer
  tracing: Add a trace remote module for testing
  tracing: selftests: Add trace remote tests
  tracing: load/unload page callbacks for simple_ring_buffer
  tracing: Check for undefined symbols in simple_ring_buffer
  KVM: arm64: Support unaligned fixmap in the nVHE hyp
  KVM: arm64: Add .hyp.data section
  KVM: arm64: Add clock support for the pKVM hyp
  KVM: arm64: Add tracing capability for the pKVM hyp
  KVM: arm64: Add trace remote for the pKVM hyp
  KVM: arm64: Sync boot clock with the pKVM hyp
  KVM: arm64: Add trace reset to the pKVM hyp
  KVM: arm64: Add event support to the pKVM hyp and trace remote
  KVM: arm64: Add hyp_enter/hyp_exit events to pKVM hyp
  KVM: arm64: Add selftest event support to pKVM hyp
  tracing: selftests: Add pKVM trace remote tests

 arch/arm64/include/asm/kvm_asm.h              |   8 +
 arch/arm64/include/asm/kvm_define_hypevents.h |  21 +
 arch/arm64/include/asm/kvm_hyp.h              |   1 -
 arch/arm64/include/asm/kvm_hypevents.h        |  41 +
 arch/arm64/include/asm/kvm_hyptrace.h         |  26 +
 arch/arm64/include/asm/sections.h             |   1 +
 arch/arm64/kernel/image-vars.h                |   6 +
 arch/arm64/kernel/vmlinux.lds.S               |  36 +-
 arch/arm64/kvm/Kconfig                        |  18 +
 arch/arm64/kvm/Makefile                       |   2 +
 arch/arm64/kvm/arm.c                          |  11 +
 arch/arm64/kvm/hyp/include/nvhe/arm-smccc.h   |  13 +
 arch/arm64/kvm/hyp/include/nvhe/clock.h       |  16 +
 .../kvm/hyp/include/nvhe/define_events.h      |  21 +
 arch/arm64/kvm/hyp/include/nvhe/trace.h       |  59 ++
 arch/arm64/kvm/hyp/nvhe/Makefile              |   1 +
 arch/arm64/kvm/hyp/nvhe/clock.c               |  65 ++
 arch/arm64/kvm/hyp/nvhe/events.c              |  36 +
 arch/arm64/kvm/hyp/nvhe/ffa.c                 |   2 +-
 arch/arm64/kvm/hyp/nvhe/hyp-main.c            |  86 ++
 arch/arm64/kvm/hyp/nvhe/hyp.lds.S             |   8 +
 arch/arm64/kvm/hyp/nvhe/mm.c                  |   2 +-
 arch/arm64/kvm/hyp/nvhe/psci-relay.c          |   5 +-
 arch/arm64/kvm/hyp/nvhe/setup.c               |   4 +
 arch/arm64/kvm/hyp/nvhe/switch.c              |   5 +-
 arch/arm64/kvm/hyp/nvhe/trace.c               | 286 ++++++
 arch/arm64/kvm/hyp_trace.c                    | 385 ++++++++
 arch/arm64/kvm/hyp_trace.h                    |  11 +
 arch/arm64/kvm/pkvm.c                         |   1 +
 include/linux/ring_buffer.h                   | 102 +-
 include/linux/simple_ring_buffer.h            |  57 ++
 include/linux/trace_remote.h                  |  27 +
 include/linux/trace_remote_event.h            |  33 +
 include/trace/define_remote_events.h          |  73 ++
 kernel/trace/Kconfig                          |  14 +
 kernel/trace/Makefile                         |  19 +
 kernel/trace/remote_test.c                    | 259 +++++
 kernel/trace/remote_test_events.h             |  10 +
 kernel/trace/ring_buffer.c                    | 254 ++++-
 kernel/trace/simple_ring_buffer.c             | 404 ++++++++
 kernel/trace/trace.c                          |   2 +-
 kernel/trace/trace.h                          |   6 +
 kernel/trace/trace_remote.c                   | 920 ++++++++++++++++++
 .../ftrace/test.d/remotes/buffer_size.tc      |  24 +
 .../selftests/ftrace/test.d/remotes/functions |  33 +
 .../ftrace/test.d/remotes/pkvm/buffer_size.tc |  10 +
 .../ftrace/test.d/remotes/pkvm/reset.tc       |  10 +
 .../ftrace/test.d/remotes/pkvm/trace_pipe.tc  |  10 +
 .../ftrace/test.d/remotes/pkvm/unloading.tc   |  10 +
 .../selftests/ftrace/test.d/remotes/reset.tc  | 105 ++
 .../ftrace/test.d/remotes/trace_pipe.tc       |  57 ++
 .../ftrace/test.d/remotes/unloading.tc        |  36 +
 52 files changed, 3599 insertions(+), 53 deletions(-)
 create mode 100644 arch/arm64/include/asm/kvm_define_hypevents.h
 create mode 100644 arch/arm64/include/asm/kvm_hypevents.h
 create mode 100644 arch/arm64/include/asm/kvm_hyptrace.h
 create mode 100644 arch/arm64/kvm/hyp/include/nvhe/arm-smccc.h
 create mode 100644 arch/arm64/kvm/hyp/include/nvhe/clock.h
 create mode 100644 arch/arm64/kvm/hyp/include/nvhe/define_events.h
 create mode 100644 arch/arm64/kvm/hyp/include/nvhe/trace.h
 create mode 100644 arch/arm64/kvm/hyp/nvhe/clock.c
 create mode 100644 arch/arm64/kvm/hyp/nvhe/events.c
 create mode 100644 arch/arm64/kvm/hyp/nvhe/trace.c
 create mode 100644 arch/arm64/kvm/hyp_trace.c
 create mode 100644 arch/arm64/kvm/hyp_trace.h
 create mode 100644 include/linux/simple_ring_buffer.h
 create mode 100644 include/linux/trace_remote.h
 create mode 100644 include/linux/trace_remote_event.h
 create mode 100644 include/trace/define_remote_events.h
 create mode 100644 kernel/trace/remote_test.c
 create mode 100644 kernel/trace/remote_test_events.h
 create mode 100644 kernel/trace/simple_ring_buffer.c
 create mode 100644 kernel/trace/trace_remote.c
 create mode 100644 tools/testing/selftests/ftrace/test.d/remotes/buffer_size.tc
 create mode 100644 tools/testing/selftests/ftrace/test.d/remotes/functions
 create mode 100644 tools/testing/selftests/ftrace/test.d/remotes/pkvm/buffer_size.tc
 create mode 100644 tools/testing/selftests/ftrace/test.d/remotes/pkvm/reset.tc
 create mode 100644 tools/testing/selftests/ftrace/test.d/remotes/pkvm/trace_pipe.tc
 create mode 100644 tools/testing/selftests/ftrace/test.d/remotes/pkvm/unloading.tc
 create mode 100644 tools/testing/selftests/ftrace/test.d/remotes/reset.tc
 create mode 100644 tools/testing/selftests/ftrace/test.d/remotes/trace_pipe.tc
 create mode 100644 tools/testing/selftests/ftrace/test.d/remotes/unloading.tc


base-commit: 02ddfb981de88a2c15621115dd7be2431252c568
-- 
2.49.0.967.g6a0df3ecc3-goog
Re: [PATCH v4 00/24] Tracefs support for pKVM
Posted by Steven Rostedt 7 months, 1 week ago
On Tue,  6 May 2025 17:47:56 +0100
Vincent Donnefort <vdonnefort@google.com> wrote:

> The growing set of features supported by the hypervisor in protected
> mode necessitates debugging and profiling tools. Tracefs is the
> ideal candidate for this task:
> 
>   * It is simple to use and to script.
> 
>   * It is supported by various tools, from the trace-cmd CLI to the
>     Android web-based perfetto.
> 
>   * The ring-buffer, where are stored trace events consists of linked
>     pages, making it an ideal structure for sharing between kernel and
>     hypervisor.
> 
> This series first introduces a new generic way of creating remote events and
> remote buffers. Then it adds support to the pKVM hypervisor.
> 
> 1. ring-buffer
> --------------
> 
> To setup the per-cpu ring-buffers, a new interface is created:
> 
>   ring_buffer_remote:	Describes what the kernel needs to know about the
> 			remote writer, that is, the set of pages forming the
> 			ring-buffer and a callback for the reader/head
> 			swapping (enables consuming read)
> 
>   ring_buffer_remote():	Creates a read-only ring-buffer from a
> 			ring_buffer_remote.
> 
> To keep the internals of `struct ring_buffer` in sync with the remote,
> the meta-page is used. It was originally introduced to enable user-space
> mapping of the ring-buffer [1]. In this case, the kernel is not the
> producer anymore but the reader. The function to read that meta-page is:
> 
>   ring_buffer_poll_remote():
> 			Update `struct ring_buffer` based on the remote
> 			meta-page. Wake-up readers if necessary.
> 
> The kernel has to poll the meta-page to be notified of newly written
> events.
> 
> 2. Tracefs
> ----------
> 
> This series introduce a new trace_remote that does the link between
> tracefs and the remote ring-buffer.
> 
> The interface is found in the remotes/ directory at the root of the
> tracefs mount point. Each remote is like an instance and you'll find
> there a subset of the regular Tracefs user-space interface:
> 
>   remotes/test/
>      buffer_size_kb
>      trace_clock
>      trace_pipe
>      trace
>      per_cpu/
>              cpuX/
>                  trace
>                  trace_pipe
>      events/
> 
>             test/
>                 selftest/
>                           enable
>                           id
> 
> Behind the scenes, kernel/trace/trace_remote.c creates this tracefs
> hierarchy without relying on kernel/trace/trace.c. This is due to
> fundamental differences:
> 
>   * Remote tracing doesn't support trace_array's system-specific
>     features (snapshots, tracers, etc.).
> 
>   * Logged event formats differ (e.g., no PID for remote events).
> 
>   * Buffer operations require specific remote interactions.
> 
> 3. Simple Ring-Buffer
> ---------------------
> 
> As the current ring-buffer.c implementation has too many dependencies to
> be used directly by the pKVM hypervisor. A new simple implementation is
> created and can be found in kernel/trace/simple-ring-buffer.c.
> 
> This implementation is write-only and is used by both the pKVM
> hypervisor and a trace_remote test module.
> 
> 4. Events
> ---------
> 
> A new REMOTE_EVENT() macro is added to simplify the creation of events
> on the kernel side. As remote tracing buffer are read only, only the
> event structure and a way of printing must be declared. The prototype of
> the macro is very similar to the well-known TRACE_EVENT()
> 
>  REMOTE_EVENT(my_event, id,
>      RE_STRUCT(
>          re_field(u64, foobar)
>      ),
>      RE_PRINTK("foobar=%lld", __entry->foobar)
>      )
>   )
> 
> 5. pKVM
> -------
> 
> The pKVM support simply creates a "hypervisor" trace_remote on the
> kernel side and inherits from simple-ring-buffer.c on the hypervisor
> side.
> 
> A new event macro is created HYP_EVENT() that is under the hood re-using
> REMOTE_EVENT() (defined in the previous paragaph) as well as generate
> hypervisor specific struct and trace_<event>() functions.
> 
> 5. Limitations:
> ---------------
> 
> Non-consuming reading of the buffer isn't supported (i.e. cat trace ->
> -EPERM) due to current the lack of support in the ring-buffer meta-page.
> 
> [1] https://tracingsummit.org/ts/2022/hypervisortracing/
> [2] https://lore.kernel.org/all/20240510140435.3550353-1-vdonnefort@google.com/
> 

BTW,  I tried to build this series and it fails.

  CALL    /work/git/test-linux.git/scripts/checksyscalls.sh
  CC      kernel/trace/simple_ring_buffer.o
In file included from ./arch/x86/include/generated/asm/rwonce.h:1,
                 from /work/git/test-linux.git/include/linux/compiler.h:390,
                 from /work/git/test-linux.git/arch/x86/include/asm/atomic.h:5,
                 from /work/git/test-linux.git/include/linux/atomic.h:7,
                 from /work/git/test-linux.git/kernel/trace/simple_ring_buffer.c:7:
/work/git/test-linux.git/kernel/trace/simple_ring_buffer.c: In function ‘simple_rb_move_tail’:
/work/git/test-linux.git/include/asm-generic/rwonce.h:55:37: error: assignment to ‘struct list_head *’ from ‘long unsigned int’ makes pointer from integer without a cast [-Wint-conversion]
   55 |         *(volatile typeof(x) *)&(x) = (val);                            \
      |                                     ^
/work/git/test-linux.git/include/asm-generic/rwonce.h:61:9: note: in expansion of macro ‘__WRITE_ONCE’
   61 |         __WRITE_ONCE(x, val);                                           \
      |         ^~~~~~~~~~~~
/work/git/test-linux.git/arch/x86/include/asm/barrier.h:63:9: note: in expansion of macro ‘WRITE_ONCE’
   63 |         WRITE_ONCE(*p, v);                                              \
      |         ^~~~~~~~~~
/work/git/test-linux.git/include/asm-generic/barrier.h:172:55: note: in expansion of macro ‘__smp_store_release’
  172 | #define smp_store_release(p, v) do { kcsan_release(); __smp_store_release(p, v); } while (0)
      |                                                       ^~~~~~~~~~~~~~~~~~~
/work/git/test-linux.git/kernel/trace/simple_ring_buffer.c:129:17: note: in expansion of macro ‘smp_store_release’
  129 |                 smp_store_release(&new_tail->list.next,
      |                 ^~~~~~~~~~~~~~~~~
make[5]: *** [/work/git/test-linux.git/scripts/Makefile.build:203: kernel/trace/simple_ring_buffer.o] Error 1
make[4]: *** [/work/git/test-linux.git/scripts/Makefile.build:461: kernel/trace] Error 2
make[3]: *** [/work/git/test-linux.git/scripts/Makefile.build:461: kernel] Error 2
make[2]: *** [/work/git/test-linux.git/Makefile:2004: .] Error 2
make[1]: *** [/work/git/test-linux.git/Makefile:248: __sub-make] Error 2
make[1]: Leaving directory '/work/build/trace/nobackup/debiantesting-x86-64'

Even when I fixed this, it then failed with the building of the sample module.

I think you need something like:

obj-$(CONFIG_TRACE_REMOTE_TEST) += remote_test_mod.o

remote_test_mod-y := simple_ring_buffer.o remote_test.o trace_remote.o

If the module needs more than one object file. Then the module should be
called something that doesn't have a .c file and use that name with ".o" to
add all the objects.

I think this could work, but this still had issues with functions not exported.

-- Steve
Re: [PATCH v4 00/24] Tracefs support for pKVM
Posted by Vincent Donnefort 7 months, 1 week ago
On Wed, May 14, 2025 at 01:38:15PM -0400, Steven Rostedt wrote:
> On Tue,  6 May 2025 17:47:56 +0100
> Vincent Donnefort <vdonnefort@google.com> wrote:
> 
> > The growing set of features supported by the hypervisor in protected
> > mode necessitates debugging and profiling tools. Tracefs is the
> > ideal candidate for this task:
> > 
> >   * It is simple to use and to script.
> > 
> >   * It is supported by various tools, from the trace-cmd CLI to the
> >     Android web-based perfetto.
> > 
> >   * The ring-buffer, where are stored trace events consists of linked
> >     pages, making it an ideal structure for sharing between kernel and
> >     hypervisor.
> > 
> > This series first introduces a new generic way of creating remote events and
> > remote buffers. Then it adds support to the pKVM hypervisor.
> > 
> > 1. ring-buffer
> > --------------
> > 
> > To setup the per-cpu ring-buffers, a new interface is created:
> > 
> >   ring_buffer_remote:	Describes what the kernel needs to know about the
> > 			remote writer, that is, the set of pages forming the
> > 			ring-buffer and a callback for the reader/head
> > 			swapping (enables consuming read)
> > 
> >   ring_buffer_remote():	Creates a read-only ring-buffer from a
> > 			ring_buffer_remote.
> > 
> > To keep the internals of `struct ring_buffer` in sync with the remote,
> > the meta-page is used. It was originally introduced to enable user-space
> > mapping of the ring-buffer [1]. In this case, the kernel is not the
> > producer anymore but the reader. The function to read that meta-page is:
> > 
> >   ring_buffer_poll_remote():
> > 			Update `struct ring_buffer` based on the remote
> > 			meta-page. Wake-up readers if necessary.
> > 
> > The kernel has to poll the meta-page to be notified of newly written
> > events.
> > 
> > 2. Tracefs
> > ----------
> > 
> > This series introduce a new trace_remote that does the link between
> > tracefs and the remote ring-buffer.
> > 
> > The interface is found in the remotes/ directory at the root of the
> > tracefs mount point. Each remote is like an instance and you'll find
> > there a subset of the regular Tracefs user-space interface:
> > 
> >   remotes/test/
> >      buffer_size_kb
> >      trace_clock
> >      trace_pipe
> >      trace
> >      per_cpu/
> >              cpuX/
> >                  trace
> >                  trace_pipe
> >      events/
> > 
> >             test/
> >                 selftest/
> >                           enable
> >                           id
> > 
> > Behind the scenes, kernel/trace/trace_remote.c creates this tracefs
> > hierarchy without relying on kernel/trace/trace.c. This is due to
> > fundamental differences:
> > 
> >   * Remote tracing doesn't support trace_array's system-specific
> >     features (snapshots, tracers, etc.).
> > 
> >   * Logged event formats differ (e.g., no PID for remote events).
> > 
> >   * Buffer operations require specific remote interactions.
> > 
> > 3. Simple Ring-Buffer
> > ---------------------
> > 
> > As the current ring-buffer.c implementation has too many dependencies to
> > be used directly by the pKVM hypervisor. A new simple implementation is
> > created and can be found in kernel/trace/simple-ring-buffer.c.
> > 
> > This implementation is write-only and is used by both the pKVM
> > hypervisor and a trace_remote test module.
> > 
> > 4. Events
> > ---------
> > 
> > A new REMOTE_EVENT() macro is added to simplify the creation of events
> > on the kernel side. As remote tracing buffer are read only, only the
> > event structure and a way of printing must be declared. The prototype of
> > the macro is very similar to the well-known TRACE_EVENT()
> > 
> >  REMOTE_EVENT(my_event, id,
> >      RE_STRUCT(
> >          re_field(u64, foobar)
> >      ),
> >      RE_PRINTK("foobar=%lld", __entry->foobar)
> >      )
> >   )
> > 
> > 5. pKVM
> > -------
> > 
> > The pKVM support simply creates a "hypervisor" trace_remote on the
> > kernel side and inherits from simple-ring-buffer.c on the hypervisor
> > side.
> > 
> > A new event macro is created HYP_EVENT() that is under the hood re-using
> > REMOTE_EVENT() (defined in the previous paragaph) as well as generate
> > hypervisor specific struct and trace_<event>() functions.
> > 
> > 5. Limitations:
> > ---------------
> > 
> > Non-consuming reading of the buffer isn't supported (i.e. cat trace ->
> > -EPERM) due to current the lack of support in the ring-buffer meta-page.
> > 
> > [1] https://tracingsummit.org/ts/2022/hypervisortracing/
> > [2] https://lore.kernel.org/all/20240510140435.3550353-1-vdonnefort@google.com/
> > 
> 
> BTW,  I tried to build this series and it fails.


Yes, appologies, I've started applying your comments today and I've figured out
I haven't tried building for x86.

I already have fixes locally for what's below.

I probably can send a v5 this week if you wish, unless you prefer to wait a bit
more for more comments?

> 
>   CALL    /work/git/test-linux.git/scripts/checksyscalls.sh
>   CC      kernel/trace/simple_ring_buffer.o
> In file included from ./arch/x86/include/generated/asm/rwonce.h:1,
>                  from /work/git/test-linux.git/include/linux/compiler.h:390,
>                  from /work/git/test-linux.git/arch/x86/include/asm/atomic.h:5,
>                  from /work/git/test-linux.git/include/linux/atomic.h:7,
>                  from /work/git/test-linux.git/kernel/trace/simple_ring_buffer.c:7:
> /work/git/test-linux.git/kernel/trace/simple_ring_buffer.c: In function ‘simple_rb_move_tail’:
> /work/git/test-linux.git/include/asm-generic/rwonce.h:55:37: error: assignment to ‘struct list_head *’ from ‘long unsigned int’ makes pointer from integer without a cast [-Wint-conversion]
>    55 |         *(volatile typeof(x) *)&(x) = (val);                            \
>       |                                     ^
> /work/git/test-linux.git/include/asm-generic/rwonce.h:61:9: note: in expansion of macro ‘__WRITE_ONCE’
>    61 |         __WRITE_ONCE(x, val);                                           \
>       |         ^~~~~~~~~~~~
> /work/git/test-linux.git/arch/x86/include/asm/barrier.h:63:9: note: in expansion of macro ‘WRITE_ONCE’
>    63 |         WRITE_ONCE(*p, v);                                              \
>       |         ^~~~~~~~~~
> /work/git/test-linux.git/include/asm-generic/barrier.h:172:55: note: in expansion of macro ‘__smp_store_release’
>   172 | #define smp_store_release(p, v) do { kcsan_release(); __smp_store_release(p, v); } while (0)
>       |                                                       ^~~~~~~~~~~~~~~~~~~
> /work/git/test-linux.git/kernel/trace/simple_ring_buffer.c:129:17: note: in expansion of macro ‘smp_store_release’
>   129 |                 smp_store_release(&new_tail->list.next,
>       |                 ^~~~~~~~~~~~~~~~~
> make[5]: *** [/work/git/test-linux.git/scripts/Makefile.build:203: kernel/trace/simple_ring_buffer.o] Error 1
> make[4]: *** [/work/git/test-linux.git/scripts/Makefile.build:461: kernel/trace] Error 2
> make[3]: *** [/work/git/test-linux.git/scripts/Makefile.build:461: kernel] Error 2
> make[2]: *** [/work/git/test-linux.git/Makefile:2004: .] Error 2
> make[1]: *** [/work/git/test-linux.git/Makefile:248: __sub-make] Error 2
> make[1]: Leaving directory '/work/build/trace/nobackup/debiantesting-x86-64'
> 
> Even when I fixed this, it then failed with the building of the sample module.
> 
> I think you need something like:
> 
> obj-$(CONFIG_TRACE_REMOTE_TEST) += remote_test_mod.o
> 
> remote_test_mod-y := simple_ring_buffer.o remote_test.o trace_remote.o
> 
> If the module needs more than one object file. Then the module should be
> called something that doesn't have a .c file and use that name with ".o" to
> add all the objects.
> 
> I think this could work, but this still had issues with functions not exported.
> 
> -- Steve
Re: [PATCH v4 00/24] Tracefs support for pKVM
Posted by Steven Rostedt 7 months, 1 week ago
On Wed, 14 May 2025 19:13:37 +0100
Vincent Donnefort <vdonnefort@google.com> wrote:

> I probably can send a v5 this week if you wish, unless you prefer to wait a bit
> more for more comments?

Feel free to send them this week, but I'll be on PTO starting tonight, so I
won't look at them until Tuesday.

-- Steve