[Qemu-devel] [RISU PATCH v3 00/18] Support for generating x86 SIMD test images

Jan Bobek posted 18 patches 13 weeks ago
Failed in applying to current master (apply log)
risugen                    |   27 +-
risugen_arm.pm             |    6 +-
risugen_common.pm          |  117 +-
risugen_m68k.pm            |    3 +-
risugen_ppc64.pm           |    6 +-
risugen_x86.pm             |  518 +++++
risugen_x86_asm.pm         |  918 ++++++++
risugen_x86_constraints.pm |  154 ++
risugen_x86_memory.pm      |   87 +
x86.risu                   | 4499 ++++++++++++++++++++++++++++++++++++
10 files changed, 6293 insertions(+), 42 deletions(-)
create mode 100644 risugen_x86.pm
create mode 100644 risugen_x86_asm.pm
create mode 100644 risugen_x86_constraints.pm
create mode 100644 risugen_x86_memory.pm
create mode 100644 x86.risu

[Qemu-devel] [RISU PATCH v3 00/18] Support for generating x86 SIMD test images

Posted by Jan Bobek 13 weeks ago
This is v3 of the patch series posted in [1] and [2]. Note that this
is the first fully-featured patch series implementing all desired
functionality, including (V)LDMXCSR and VSIB-based instructions like
VGATHER*.

While implementing the last bits required in order to support VGATHERx
instructions, I ran into problems which required a larger redesign;
namely, there are no more !emit blocks as their functionality is now
implemented in regular !constraints blocks. Also, memory constraints
are specified in !memory blocks, similarly to other architectures.

I tested these changes on my machine; both master and slave modes work
in both 32-bit and 64-bit modes.

Cheers,
 -Jan

Changes since v2:
  Too many to be listed individually; this patch series might be
  better reviewed on its own.

References:
  1. https://lists.nongnu.org/archive/html/qemu-devel/2019-06/msg04123.html
  2. https://lists.nongnu.org/archive/html/qemu-devel/2019-07/msg00001.html

Jan Bobek (18):
  risugen_common: add helper functions insnv, randint
  risugen_common: split eval_with_fields into extract_fields and
    eval_block
  risugen_x86_asm: add module
  risugen_x86_constraints: add module
  risugen_x86_memory: add module
  risugen_x86: add module
  risugen: allow all byte-aligned instructions
  risugen: add command-line flag --x86_64
  risugen: add --xfeatures option for x86
  x86.risu: add MMX instructions
  x86.risu: add SSE instructions
  x86.risu: add SSE2 instructions
  x86.risu: add SSE3 instructions
  x86.risu: add SSSE3 instructions
  x86.risu: add SSE4.1 and SSE4.2 instructions
  x86.risu: add AES and PCLMULQDQ instructions
  x86.risu: add AVX instructions
  x86.risu: add AVX2 instructions

 risugen                    |   27 +-
 risugen_arm.pm             |    6 +-
 risugen_common.pm          |  117 +-
 risugen_m68k.pm            |    3 +-
 risugen_ppc64.pm           |    6 +-
 risugen_x86.pm             |  518 +++++
 risugen_x86_asm.pm         |  918 ++++++++
 risugen_x86_constraints.pm |  154 ++
 risugen_x86_memory.pm      |   87 +
 x86.risu                   | 4499 ++++++++++++++++++++++++++++++++++++
 10 files changed, 6293 insertions(+), 42 deletions(-)
 create mode 100644 risugen_x86.pm
 create mode 100644 risugen_x86_asm.pm
 create mode 100644 risugen_x86_constraints.pm
 create mode 100644 risugen_x86_memory.pm
 create mode 100644 x86.risu

-- 
2.20.1


Re: [Qemu-devel] [RISU PATCH v3 00/18] Support for generating x86 SIMD test images

Posted by Alex Bennée 13 weeks ago
Jan Bobek <jan.bobek@gmail.com> writes:

> This is v3 of the patch series posted in [1] and [2]. Note that this
> is the first fully-featured patch series implementing all desired
> functionality, including (V)LDMXCSR and VSIB-based instructions like
> VGATHER*.
>
> While implementing the last bits required in order to support VGATHERx
> instructions, I ran into problems which required a larger redesign;
> namely, there are no more !emit blocks as their functionality is now
> implemented in regular !constraints blocks. Also, memory constraints
> are specified in !memory blocks, similarly to other architectures.
>
> I tested these changes on my machine; both master and slave modes work
> in both 32-bit and 64-bit modes.

Two things I've noticed:

  ./contrib/generate_all.sh -n 1 x86.risu testcases.x86

takes a very long time. I wonder if this is a consequence of constantly
needing to re-query the random number generator?

The other is:

  set -x RISU ./build/i686-linux-gnu/risu
  ./contrib/record_traces.sh testcases.x86/*.risu.bin

fails on the first trace when validating the playback. Might want to
check why that is.

>
> Cheers,
>  -Jan
>
> Changes since v2:
>   Too many to be listed individually; this patch series might be
>   better reviewed on its own.
>
> References:
>   1. https://lists.nongnu.org/archive/html/qemu-devel/2019-06/msg04123.html
>   2. https://lists.nongnu.org/archive/html/qemu-devel/2019-07/msg00001.html
>
> Jan Bobek (18):
>   risugen_common: add helper functions insnv, randint
>   risugen_common: split eval_with_fields into extract_fields and
>     eval_block
>   risugen_x86_asm: add module
>   risugen_x86_constraints: add module
>   risugen_x86_memory: add module
>   risugen_x86: add module
>   risugen: allow all byte-aligned instructions
>   risugen: add command-line flag --x86_64
>   risugen: add --xfeatures option for x86
>   x86.risu: add MMX instructions
>   x86.risu: add SSE instructions
>   x86.risu: add SSE2 instructions
>   x86.risu: add SSE3 instructions
>   x86.risu: add SSSE3 instructions
>   x86.risu: add SSE4.1 and SSE4.2 instructions
>   x86.risu: add AES and PCLMULQDQ instructions
>   x86.risu: add AVX instructions
>   x86.risu: add AVX2 instructions
>
>  risugen                    |   27 +-
>  risugen_arm.pm             |    6 +-
>  risugen_common.pm          |  117 +-
>  risugen_m68k.pm            |    3 +-
>  risugen_ppc64.pm           |    6 +-
>  risugen_x86.pm             |  518 +++++
>  risugen_x86_asm.pm         |  918 ++++++++
>  risugen_x86_constraints.pm |  154 ++
>  risugen_x86_memory.pm      |   87 +
>  x86.risu                   | 4499 ++++++++++++++++++++++++++++++++++++
>  10 files changed, 6293 insertions(+), 42 deletions(-)
>  create mode 100644 risugen_x86.pm
>  create mode 100644 risugen_x86_asm.pm
>  create mode 100644 risugen_x86_constraints.pm
>  create mode 100644 risugen_x86_memory.pm
>  create mode 100644 x86.risu


--
Alex Bennée

Re: [Qemu-devel] [RISU PATCH v3 00/18] Support for generating x86 SIMD test images

Posted by Jan Bobek 13 weeks ago
On 7/12/19 9:34 AM, Alex Bennée wrote:
> 
> Jan Bobek <jan.bobek@gmail.com> writes:
> 
>> This is v3 of the patch series posted in [1] and [2]. Note that this
>> is the first fully-featured patch series implementing all desired
>> functionality, including (V)LDMXCSR and VSIB-based instructions like
>> VGATHER*.
>>
>> While implementing the last bits required in order to support VGATHERx
>> instructions, I ran into problems which required a larger redesign;
>> namely, there are no more !emit blocks as their functionality is now
>> implemented in regular !constraints blocks. Also, memory constraints
>> are specified in !memory blocks, similarly to other architectures.
>>
>> I tested these changes on my machine; both master and slave modes work
>> in both 32-bit and 64-bit modes.
> 
> Two things I've noticed:
> 
>   ./contrib/generate_all.sh -n 1 x86.risu testcases.x86
> 
> takes a very long time. I wonder if this is a consequence of constantly
> needing to re-query the random number generator?

I believe so. While other architectures can be as cheap as a single rand()
call per instruction, x86 does more like 5-10.

Even worse, there are some instructions which cannot be generated in
32-bit mode (those requiring REX.W prefix, e.g. MMX MOVQ). When I let
the script run for a little bit, risugen would get stuck in an
infinite loop, because it could only choose from a single instruction
which wasn't valid for 32-bit....

> The other is:
> 
>   set -x RISU ./build/i686-linux-gnu/risu
>   ./contrib/record_traces.sh testcases.x86/*.risu.bin
> 
> fails on the first trace when validating the playback. Might want to
> check why that is.

The SIMD registers aren't getting initialized; both master and
apprentice need an --xfeatures=XXX parameter for that. Right now the
default is 'none'; unless the instructions are filtered, you'd need
--xfeatures=avx (or --xfeatures=sse, and that only works because on my
laptop, the upper part of ymm registers seems to be always zeroed when
risu starts).

>>
>> Cheers,
>>  -Jan
>>
>> Changes since v2:
>>   Too many to be listed individually; this patch series might be
>>   better reviewed on its own.
>>
>> References:
>>   1. https://lists.nongnu.org/archive/html/qemu-devel/2019-06/msg04123.html
>>   2. https://lists.nongnu.org/archive/html/qemu-devel/2019-07/msg00001.html
>>
>> Jan Bobek (18):
>>   risugen_common: add helper functions insnv, randint
>>   risugen_common: split eval_with_fields into extract_fields and
>>     eval_block
>>   risugen_x86_asm: add module
>>   risugen_x86_constraints: add module
>>   risugen_x86_memory: add module
>>   risugen_x86: add module
>>   risugen: allow all byte-aligned instructions
>>   risugen: add command-line flag --x86_64
>>   risugen: add --xfeatures option for x86
>>   x86.risu: add MMX instructions
>>   x86.risu: add SSE instructions
>>   x86.risu: add SSE2 instructions
>>   x86.risu: add SSE3 instructions
>>   x86.risu: add SSSE3 instructions
>>   x86.risu: add SSE4.1 and SSE4.2 instructions
>>   x86.risu: add AES and PCLMULQDQ instructions
>>   x86.risu: add AVX instructions
>>   x86.risu: add AVX2 instructions
>>
>>  risugen                    |   27 +-
>>  risugen_arm.pm             |    6 +-
>>  risugen_common.pm          |  117 +-
>>  risugen_m68k.pm            |    3 +-
>>  risugen_ppc64.pm           |    6 +-
>>  risugen_x86.pm             |  518 +++++
>>  risugen_x86_asm.pm         |  918 ++++++++
>>  risugen_x86_constraints.pm |  154 ++
>>  risugen_x86_memory.pm      |   87 +
>>  x86.risu                   | 4499 ++++++++++++++++++++++++++++++++++++
>>  10 files changed, 6293 insertions(+), 42 deletions(-)
>>  create mode 100644 risugen_x86.pm
>>  create mode 100644 risugen_x86_asm.pm
>>  create mode 100644 risugen_x86_constraints.pm
>>  create mode 100644 risugen_x86_memory.pm
>>  create mode 100644 x86.risu
> 
> 
> --
> Alex Bennée
> 

Re: [Qemu-devel] [RISU PATCH v3 00/18] Support for generating x86 SIMD test images

Posted by Alex Bennée 13 weeks ago
Jan Bobek <jan.bobek@gmail.com> writes:

> On 7/12/19 9:34 AM, Alex Bennée wrote:
>>
>> Jan Bobek <jan.bobek@gmail.com> writes:
>>
>>> This is v3 of the patch series posted in [1] and [2]. Note that this
>>> is the first fully-featured patch series implementing all desired
>>> functionality, including (V)LDMXCSR and VSIB-based instructions like
>>> VGATHER*.
>>>
>>> While implementing the last bits required in order to support VGATHERx
>>> instructions, I ran into problems which required a larger redesign;
>>> namely, there are no more !emit blocks as their functionality is now
>>> implemented in regular !constraints blocks. Also, memory constraints
>>> are specified in !memory blocks, similarly to other architectures.
>>>
>>> I tested these changes on my machine; both master and slave modes work
>>> in both 32-bit and 64-bit modes.
>>
>> Two things I've noticed:
>>
>>   ./contrib/generate_all.sh -n 1 x86.risu testcases.x86
>>
>> takes a very long time. I wonder if this is a consequence of constantly
>> needing to re-query the random number generator?
>
> I believe so. While other architectures can be as cheap as a single rand()
> call per instruction, x86 does more like 5-10.

OK

> Even worse, there are some instructions which cannot be generated in
> 32-bit mode (those requiring REX.W prefix, e.g. MMX MOVQ). When I let
> the script run for a little bit, risugen would get stuck in an
> infinite loop, because it could only choose from a single instruction
> which wasn't valid for 32-bit....

The first instruction I see hang is:

  Running: /home/alex/lsrc/tests/risu.git/risugen --xfeatures avx  --pattern CVTSD2SI_64 x86.risu testcases.x86/insn_CVTSD2SI_64__INC.risu.bin
  Generating code using patterns: CVTSD2SI_64 SSE2...
  [                                                                            ]

I wonder if this means we should split the x86.risu by mode? Or some
other way of filtering out patterns that are invalid for a mode?

We do have the concept of classes, see the @ annotations in
aarch64.risu. I guess the generate_all script doesn't handle that nicely
yet though. For now I lumped them all together with:

  ./risugen --pattern "MMX" --xfeatures=sse x86.risu all_mmx.risu.bin
  ./risugen --pattern "SSE" --xfeatures=sse x86.risu all_sse.risu.bin
  ./risugen --pattern "SSE2" --xfeatures=sse x86.risu all_sse2.risu.bin
  ./risugen --pattern "SSE3" --xfeatures=sse x86.risu all_sse3.risu.bin
  ./risugen --pattern "AVX" --xfeatures=avx x86.risu all_avx.risu.bin
  ./risugen --pattern "AVX2" --xfeatures=avx x86.risu all_avx2.risu.bin

>
>> The other is:
>>
>>   set -x RISU ./build/i686-linux-gnu/risu
>>   ./contrib/record_traces.sh testcases.x86/*.risu.bin
>>
>> fails on the first trace when validating the playback. Might want to
>> check why that is.
>
> The SIMD registers aren't getting initialized; both master and
> apprentice need an --xfeatures=XXX parameter for that. Right now the
> default is 'none'; unless the instructions are filtered, you'd need
> --xfeatures=avx (or --xfeatures=sse, and that only works because on my
> laptop, the upper part of ymm registers seems to be always zeroed when
> risu starts).

Ahh OK, I did a lot better with:

  ./contrib/generate_all.sh -n 1 x86.risu testcases.x86 -- --xfeatures avx
  set -x RISU ./build/i686-linux-gnu/risu --xfeatures=avx
  ./contrib/record_traces.sh testcases.x86-avx/*.risu.bin

There are enough failures when you run those against QEMU for now that
we don't need to worry too much about coverage yet ;-)

>>>
>>> Cheers,
>>>  -Jan
>>>
>>> Changes since v2:
>>>   Too many to be listed individually; this patch series might be
>>>   better reviewed on its own.
>>>
>>> References:
>>>   1. https://lists.nongnu.org/archive/html/qemu-devel/2019-06/msg04123.html
>>>   2. https://lists.nongnu.org/archive/html/qemu-devel/2019-07/msg00001.html
>>>
>>> Jan Bobek (18):
>>>   risugen_common: add helper functions insnv, randint
>>>   risugen_common: split eval_with_fields into extract_fields and
>>>     eval_block
>>>   risugen_x86_asm: add module
>>>   risugen_x86_constraints: add module
>>>   risugen_x86_memory: add module
>>>   risugen_x86: add module
>>>   risugen: allow all byte-aligned instructions
>>>   risugen: add command-line flag --x86_64
>>>   risugen: add --xfeatures option for x86
>>>   x86.risu: add MMX instructions
>>>   x86.risu: add SSE instructions
>>>   x86.risu: add SSE2 instructions
>>>   x86.risu: add SSE3 instructions
>>>   x86.risu: add SSSE3 instructions
>>>   x86.risu: add SSE4.1 and SSE4.2 instructions
>>>   x86.risu: add AES and PCLMULQDQ instructions
>>>   x86.risu: add AVX instructions
>>>   x86.risu: add AVX2 instructions
>>>
>>>  risugen                    |   27 +-
>>>  risugen_arm.pm             |    6 +-
>>>  risugen_common.pm          |  117 +-
>>>  risugen_m68k.pm            |    3 +-
>>>  risugen_ppc64.pm           |    6 +-
>>>  risugen_x86.pm             |  518 +++++
>>>  risugen_x86_asm.pm         |  918 ++++++++
>>>  risugen_x86_constraints.pm |  154 ++
>>>  risugen_x86_memory.pm      |   87 +
>>>  x86.risu                   | 4499 ++++++++++++++++++++++++++++++++++++
>>>  10 files changed, 6293 insertions(+), 42 deletions(-)
>>>  create mode 100644 risugen_x86.pm
>>>  create mode 100644 risugen_x86_asm.pm
>>>  create mode 100644 risugen_x86_constraints.pm
>>>  create mode 100644 risugen_x86_memory.pm
>>>  create mode 100644 x86.risu
>>
>>
>> --
>> Alex Bennée
>>


--
Alex Bennée