[v2] kunit: enable hardware virtualization

[PATCH v2 2/2] kunit: enable hardware acceleration when available

Posted by Tamir Duberstein 1 year, 3 months ago

Use KVM or HVF if supported by the QEMU binary and available on the
system.

This produces a nice improvement on my Apple M3 Pro running macOS 14.7:

Before:
./tools/testing/kunit/kunit.py exec --arch arm64
[HH:MM:SS] Elapsed time: 10.145s

After:
./tools/testing/kunit/kunit.py exec --arch arm64
[HH:MM:SS] Elapsed time: 1.773s

Signed-off-by: Tamir Duberstein <tamird@gmail.com>
---
 tools/testing/kunit/kunit_kernel.py       | 3 +++
 tools/testing/kunit/qemu_configs/arm64.py | 2 +-
 2 files changed, 4 insertions(+), 1 deletion(-)

diff --git a/tools/testing/kunit/kunit_kernel.py b/tools/testing/kunit/kunit_kernel.py
index 61931c4926fd6645f2c62dd13f9842a432ec4167..3146acb884ecf0bcff94d5938535aabd4486fe82 100644
--- a/tools/testing/kunit/kunit_kernel.py
+++ b/tools/testing/kunit/kunit_kernel.py
@@ -123,6 +123,9 @@ class LinuxSourceTreeOperationsQemu(LinuxSourceTreeOperations):
 				'-append', ' '.join(params + [self._kernel_command_line]),
 				'-no-reboot',
 				'-nographic',
+				'-accel', 'kvm',
+				'-accel', 'hvf',
+				'-accel', 'tcg',
 				'-serial', self._serial] + self._extra_qemu_params
 		# Note: shlex.join() does what we want, but requires python 3.8+.
 		print('Running tests with:\n$', ' '.join(shlex.quote(arg) for arg in qemu_command))
diff --git a/tools/testing/kunit/qemu_configs/arm64.py b/tools/testing/kunit/qemu_configs/arm64.py
index d3ff27024755411441f910799be30399295c9541..5c44d3a87e6dd2cd6b086138186a277a1473585b 100644
--- a/tools/testing/kunit/qemu_configs/arm64.py
+++ b/tools/testing/kunit/qemu_configs/arm64.py
@@ -9,4 +9,4 @@ CONFIG_SERIAL_AMBA_PL011_CONSOLE=y''',
 			   qemu_arch='aarch64',
 			   kernel_path='arch/arm64/boot/Image.gz',
 			   kernel_command_line='console=ttyAMA0',
-			   extra_qemu_params=['-machine', 'virt', '-cpu', 'max,pauth-impdef=on'])
+			   extra_qemu_params=['-machine', 'virt', '-cpu', 'max'])

-- 
2.47.0

Re: [PATCH v2 2/2] kunit: enable hardware acceleration when available

Posted by Kristina Martsenko 1 year, 3 months ago

On 02/11/2024 12:09, Tamir Duberstein wrote:
> Use KVM or HVF if supported by the QEMU binary and available on the
> system.
> 
> This produces a nice improvement on my Apple M3 Pro running macOS 14.7:
> 
> Before:
> ./tools/testing/kunit/kunit.py exec --arch arm64
> [HH:MM:SS] Elapsed time: 10.145s
> 
> After:
> ./tools/testing/kunit/kunit.py exec --arch arm64
> [HH:MM:SS] Elapsed time: 1.773s
> 
> Signed-off-by: Tamir Duberstein <tamird@gmail.com>
> ---
>  tools/testing/kunit/kunit_kernel.py       | 3 +++
>  tools/testing/kunit/qemu_configs/arm64.py | 2 +-
>  2 files changed, 4 insertions(+), 1 deletion(-)
> 
> diff --git a/tools/testing/kunit/kunit_kernel.py b/tools/testing/kunit/kunit_kernel.py
> index 61931c4926fd6645f2c62dd13f9842a432ec4167..3146acb884ecf0bcff94d5938535aabd4486fe82 100644
> --- a/tools/testing/kunit/kunit_kernel.py
> +++ b/tools/testing/kunit/kunit_kernel.py
> @@ -123,6 +123,9 @@ class LinuxSourceTreeOperationsQemu(LinuxSourceTreeOperations):
>  				'-append', ' '.join(params + [self._kernel_command_line]),
>  				'-no-reboot',
>  				'-nographic',
> +				'-accel', 'kvm',
> +				'-accel', 'hvf',
> +				'-accel', 'tcg',
>  				'-serial', self._serial] + self._extra_qemu_params
>  		# Note: shlex.join() does what we want, but requires python 3.8+.
>  		print('Running tests with:\n$', ' '.join(shlex.quote(arg) for arg in qemu_command))
> diff --git a/tools/testing/kunit/qemu_configs/arm64.py b/tools/testing/kunit/qemu_configs/arm64.py
> index d3ff27024755411441f910799be30399295c9541..5c44d3a87e6dd2cd6b086138186a277a1473585b 100644
> --- a/tools/testing/kunit/qemu_configs/arm64.py
> +++ b/tools/testing/kunit/qemu_configs/arm64.py
> @@ -9,4 +9,4 @@ CONFIG_SERIAL_AMBA_PL011_CONSOLE=y''',
>  			   qemu_arch='aarch64',
>  			   kernel_path='arch/arm64/boot/Image.gz',
>  			   kernel_command_line='console=ttyAMA0',
> -			   extra_qemu_params=['-machine', 'virt', '-cpu', 'max,pauth-impdef=on'])
> +			   extra_qemu_params=['-machine', 'virt', '-cpu', 'max'])

Would it be possible to keep 'pauth-impdef=on' for TCG emulation? Otherwise
performance regresses by about 20%.

Before this patch:
./tools/testing/kunit/kunit.py exec --arch=arm64 --cross_compile=aarch64-linux-
[11:03:38] Elapsed time: 15.494s

After this patch:
./tools/testing/kunit/kunit.py exec --arch=arm64 --cross_compile=aarch64-linux-
[11:10:47] Elapsed time: 19.099s

Thanks,
Kristina

Re: [PATCH v2 2/2] kunit: enable hardware acceleration when available

Posted by Tamir Duberstein 1 year, 3 months ago

On Tue, Nov 5, 2024 at 8:36 AM Kristina Martsenko
<kristina.martsenko@arm.com> wrote:
>
> On 02/11/2024 12:09, Tamir Duberstein wrote:
> > Use KVM or HVF if supported by the QEMU binary and available on the
> > system.
> >
> > This produces a nice improvement on my Apple M3 Pro running macOS 14.7:
> >
> > Before:
> > ./tools/testing/kunit/kunit.py exec --arch arm64
> > [HH:MM:SS] Elapsed time: 10.145s
> >
> > After:
> > ./tools/testing/kunit/kunit.py exec --arch arm64
> > [HH:MM:SS] Elapsed time: 1.773s
> >
> > Signed-off-by: Tamir Duberstein <tamird@gmail.com>
> > ---
> >  tools/testing/kunit/kunit_kernel.py       | 3 +++
> >  tools/testing/kunit/qemu_configs/arm64.py | 2 +-
> >  2 files changed, 4 insertions(+), 1 deletion(-)
> >
> > diff --git a/tools/testing/kunit/kunit_kernel.py b/tools/testing/kunit/kunit_kernel.py
> > index 61931c4926fd6645f2c62dd13f9842a432ec4167..3146acb884ecf0bcff94d5938535aabd4486fe82 100644
> > --- a/tools/testing/kunit/kunit_kernel.py
> > +++ b/tools/testing/kunit/kunit_kernel.py
> > @@ -123,6 +123,9 @@ class LinuxSourceTreeOperationsQemu(LinuxSourceTreeOperations):
> >                               '-append', ' '.join(params + [self._kernel_command_line]),
> >                               '-no-reboot',
> >                               '-nographic',
> > +                             '-accel', 'kvm',
> > +                             '-accel', 'hvf',
> > +                             '-accel', 'tcg',
> >                               '-serial', self._serial] + self._extra_qemu_params
> >               # Note: shlex.join() does what we want, but requires python 3.8+.
> >               print('Running tests with:\n$', ' '.join(shlex.quote(arg) for arg in qemu_command))
> > diff --git a/tools/testing/kunit/qemu_configs/arm64.py b/tools/testing/kunit/qemu_configs/arm64.py
> > index d3ff27024755411441f910799be30399295c9541..5c44d3a87e6dd2cd6b086138186a277a1473585b 100644
> > --- a/tools/testing/kunit/qemu_configs/arm64.py
> > +++ b/tools/testing/kunit/qemu_configs/arm64.py
> > @@ -9,4 +9,4 @@ CONFIG_SERIAL_AMBA_PL011_CONSOLE=y''',
> >                          qemu_arch='aarch64',
> >                          kernel_path='arch/arm64/boot/Image.gz',
> >                          kernel_command_line='console=ttyAMA0',
> > -                        extra_qemu_params=['-machine', 'virt', '-cpu', 'max,pauth-impdef=on'])
> > +                        extra_qemu_params=['-machine', 'virt', '-cpu', 'max'])
>
> Would it be possible to keep 'pauth-impdef=on' for TCG emulation? Otherwise
> performance regresses by about 20%.
>
> Before this patch:
> ./tools/testing/kunit/kunit.py exec --arch=arm64 --cross_compile=aarch64-linux-
> [11:03:38] Elapsed time: 15.494s
>
> After this patch:
> ./tools/testing/kunit/kunit.py exec --arch=arm64 --cross_compile=aarch64-linux-
> [11:10:47] Elapsed time: 19.099s
>
> Thanks,
> Kristina

Hi Kristina, thanks for pointing that out. I'm able to reproduce the
regression. I poked around and I can't find a way to enable
`pauth-impdef` only for TCG, and the problem is that enabling it
globally produces:

tools/testing/kunit/kunit.py exec --arch arm64 --make_options LLVM=1
--raw_output=all
[15:34:05] Starting KUnit Kernel (1/1)...
Running tests with:
$ qemu-system-aarch64 -nodefaults -m 1024 -kernel
.kunit/arch/arm64/boot/Image.gz -append 'kunit.enable=1
console=ttyAMA0 kunit_shutdown=reboot' -no-reboot -nographic -accel
kvm -accel hvf -accel tcg -serial stdio -machine virt -cpu
max,pauth-impdef=on
qemu-system-aarch64: -accel kvm: invalid accelerator kvm
qemu-system-aarch64: falling back to HVF
qemu-system-aarch64: can't apply global max-arm-cpu.pauth-impdef=on:
Property 'max-arm-cpu.pauth-impdef' not found

This behavior is at least somewhat intentional[0]. I have filed a bug
with qemu[1]. If someone can conceive of a way to achieve this, I'd be
delighted to send a v3.

Link: https://gitlab.com/qemu-project/qemu/-/commit/92d6528dbb20c6aec4022dfd63c7ffee44f19f77
[0]
Link: https://gitlab.com/qemu-project/qemu/-/issues/2656 [1]

Re: [PATCH v2 2/2] kunit: enable hardware acceleration when available

Posted by Tamir Duberstein 1 year, 2 months ago

On Tue, Nov 5, 2024 at 2:51 PM Tamir Duberstein <tamird@gmail.com> wrote:
>
> This behavior is at least somewhat intentional[0]. I have filed a bug
> with qemu[1]. If someone can conceive of a way to achieve this, I'd be
> delighted to send a v3.
>
> [cut]
>
> Link: https://gitlab.com/qemu-project/qemu/-/issues/2656 [1]

There hasn't been movement on this matter here or on the upstream bug.
We can either proceed with this patch as-is or restore the manual
feature detection that was present in v1
(https://lore.kernel.org/all/20241025-kunit-qemu-accel-macos-v1-2-2f30c26192d4@gmail.com/)
in order to retain `pauth-impdef` for TCG. Could I get some guidance
from maintainers please?

Tamir

Re: [PATCH v2 2/2] kunit: enable hardware acceleration when available

Posted by David Gow 1 year, 3 months ago

On Sat, 2 Nov 2024 at 20:10, Tamir Duberstein <tamird@gmail.com> wrote:
>
> Use KVM or HVF if supported by the QEMU binary and available on the
> system.
>
> This produces a nice improvement on my Apple M3 Pro running macOS 14.7:
>
> Before:
> ./tools/testing/kunit/kunit.py exec --arch arm64
> [HH:MM:SS] Elapsed time: 10.145s
>
> After:
> ./tools/testing/kunit/kunit.py exec --arch arm64
> [HH:MM:SS] Elapsed time: 1.773s
>
> Signed-off-by: Tamir Duberstein <tamird@gmail.com>
> ---

Thanks a lot.

I finally managed to dig up an arm64 machine and test this, and I can
reproduce the performance improvement.

Reviewed-by: David Gow <davidgow@google.com>

Cheers,
-- David

>  tools/testing/kunit/kunit_kernel.py       | 3 +++
>  tools/testing/kunit/qemu_configs/arm64.py | 2 +-
>  2 files changed, 4 insertions(+), 1 deletion(-)
>
> diff --git a/tools/testing/kunit/kunit_kernel.py b/tools/testing/kunit/kunit_kernel.py
> index 61931c4926fd6645f2c62dd13f9842a432ec4167..3146acb884ecf0bcff94d5938535aabd4486fe82 100644
> --- a/tools/testing/kunit/kunit_kernel.py
> +++ b/tools/testing/kunit/kunit_kernel.py
> @@ -123,6 +123,9 @@ class LinuxSourceTreeOperationsQemu(LinuxSourceTreeOperations):
>                                 '-append', ' '.join(params + [self._kernel_command_line]),
>                                 '-no-reboot',
>                                 '-nographic',
> +                               '-accel', 'kvm',
> +                               '-accel', 'hvf',
> +                               '-accel', 'tcg',
>                                 '-serial', self._serial] + self._extra_qemu_params
>                 # Note: shlex.join() does what we want, but requires python 3.8+.
>                 print('Running tests with:\n$', ' '.join(shlex.quote(arg) for arg in qemu_command))
> diff --git a/tools/testing/kunit/qemu_configs/arm64.py b/tools/testing/kunit/qemu_configs/arm64.py
> index d3ff27024755411441f910799be30399295c9541..5c44d3a87e6dd2cd6b086138186a277a1473585b 100644
> --- a/tools/testing/kunit/qemu_configs/arm64.py
> +++ b/tools/testing/kunit/qemu_configs/arm64.py
> @@ -9,4 +9,4 @@ CONFIG_SERIAL_AMBA_PL011_CONSOLE=y''',
>                            qemu_arch='aarch64',
>                            kernel_path='arch/arm64/boot/Image.gz',
>                            kernel_command_line='console=ttyAMA0',
> -                          extra_qemu_params=['-machine', 'virt', '-cpu', 'max,pauth-impdef=on'])
> +                          extra_qemu_params=['-machine', 'virt', '-cpu', 'max'])
>
> --
> 2.47.0
>