This code was never updated when the 32bit build of Xen was dropped.
* Expand the now-redundant ptr_reg macro.
* The number of iterations in the loop can be halfed by using 64bit writes,
without consuming any extra execution resource in the pipeline. Adjust all
numbers/offsets appropriately.
* Replace dec with sub to avoid a eflags stall, and position it to be
macro-fused with the related jmp.
* With no need to preserve eflags across the body of the loop, replace lea
with add which has 1/3'rd the latency on basically all 64bit hardware.
A quick userspace perf test on my Haswell dev box indicates that the old
version takes ~1385 cycles on average (ignoring outliers), and the new version
takes ~1060 cyles, or about 77% of the time.
Reported-by: Edwin Török <edvin.torok@citrix.com>
Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
---
CC: Jan Beulich <JBeulich@suse.com>
CC: Wei Liu <wl@xen.org>
CC: Roger Pau Monné <roger.pau@citrix.com>
CC: Edwin Török <edvin.torok@citrix.com>
There is almost certainly better room for improvement, especially now that we
have alternatives, but this is substantial improvement which is very safe for
backport.
---
xen/arch/x86/clear_page.S | 16 +++++++---------
1 file changed, 7 insertions(+), 9 deletions(-)
diff --git a/xen/arch/x86/clear_page.S b/xen/arch/x86/clear_page.S
index 243a767..0817610 100644
--- a/xen/arch/x86/clear_page.S
+++ b/xen/arch/x86/clear_page.S
@@ -2,18 +2,16 @@
#include <asm/page.h>
-#define ptr_reg %rdi
-
ENTRY(clear_page_sse2)
- mov $PAGE_SIZE/16, %ecx
+ mov $PAGE_SIZE/32, %ecx
xor %eax,%eax
-0: dec %ecx
- movnti %eax, (ptr_reg)
- movnti %eax, 4(ptr_reg)
- movnti %eax, 8(ptr_reg)
- movnti %eax, 12(ptr_reg)
- lea 16(ptr_reg), ptr_reg
+0: movnti %rax, 0(%rdi)
+ movnti %rax, 8(%rdi)
+ movnti %rax, 16(%rdi)
+ movnti %rax, 24(%rdi)
+ add $32, %rdi
+ sub $1, %ecx
jnz 0b
sfence
--
2.1.4
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-develOn 17/06/2019 20:49, Andrew Cooper wrote: > This code was never updated when the 32bit build of Xen was dropped. > > * Expand the now-redundant ptr_reg macro. > * The number of iterations in the loop can be halfed by using 64bit writes, > without consuming any extra execution resource in the pipeline. Adjust all > numbers/offsets appropriately. > * Replace dec with sub to avoid a eflags stall, and position it to be > macro-fused with the related jmp. > * With no need to preserve eflags across the body of the loop, replace lea > with add which has 1/3'rd the latency on basically all 64bit hardware. > > A quick userspace perf test on my Haswell dev box indicates that the old > version takes ~1385 cycles on average (ignoring outliers), and the new version > takes ~1060 cyles, or about 77% of the time. And just for giggles, a rep stosq loop on this hardware is ~180 cycles, which is more than 5 times better than the result of this patch (16% of the time). In some copious free time, clear/copy page ought to be come runtime dependent on FastString being enabled, but I don't have time to organise this right now. ~Andrew _______________________________________________ Xen-devel mailing list Xen-devel@lists.xenproject.org https://lists.xenproject.org/mailman/listinfo/xen-devel
>>> On 17.06.19 at 21:49, <andrew.cooper3@citrix.com> wrote: > This code was never updated when the 32bit build of Xen was dropped. > > * Expand the now-redundant ptr_reg macro. > * The number of iterations in the loop can be halfed by using 64bit writes, > without consuming any extra execution resource in the pipeline. Adjust all > numbers/offsets appropriately. > * Replace dec with sub to avoid a eflags stall, and position it to be > macro-fused with the related jmp. > * With no need to preserve eflags across the body of the loop, replace lea > with add which has 1/3'rd the latency on basically all 64bit hardware. > > A quick userspace perf test on my Haswell dev box indicates that the old > version takes ~1385 cycles on average (ignoring outliers), and the new version > takes ~1060 cyles, or about 77% of the time. > > Reported-by: Edwin Török <edvin.torok@citrix.com> > Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com> Assuming you want this to go in despite your REP STOSQ remark later on, Reviewed-by: Jan Beulich <jbeulich@suse.com> with one remark: > --- a/xen/arch/x86/clear_page.S > +++ b/xen/arch/x86/clear_page.S > @@ -2,18 +2,16 @@ > > #include <asm/page.h> > > -#define ptr_reg %rdi > - > ENTRY(clear_page_sse2) > - mov $PAGE_SIZE/16, %ecx > + mov $PAGE_SIZE/32, %ecx > xor %eax,%eax > > -0: dec %ecx > - movnti %eax, (ptr_reg) > - movnti %eax, 4(ptr_reg) > - movnti %eax, 8(ptr_reg) > - movnti %eax, 12(ptr_reg) > - lea 16(ptr_reg), ptr_reg > +0: movnti %rax, 0(%rdi) Could I talk you into leaving out this 0? Rather old gas actually emits an 8-bit displacement when it's spelled like this. Jan _______________________________________________ Xen-devel mailing list Xen-devel@lists.xenproject.org https://lists.xenproject.org/mailman/listinfo/xen-devel
On 18/06/2019 11:33, Jan Beulich wrote: >>>> On 17.06.19 at 21:49, <andrew.cooper3@citrix.com> wrote: >> This code was never updated when the 32bit build of Xen was dropped. >> >> * Expand the now-redundant ptr_reg macro. >> * The number of iterations in the loop can be halfed by using 64bit writes, >> without consuming any extra execution resource in the pipeline. Adjust all >> numbers/offsets appropriately. >> * Replace dec with sub to avoid a eflags stall, and position it to be >> macro-fused with the related jmp. >> * With no need to preserve eflags across the body of the loop, replace lea >> with add which has 1/3'rd the latency on basically all 64bit hardware. >> >> A quick userspace perf test on my Haswell dev box indicates that the old >> version takes ~1385 cycles on average (ignoring outliers), and the new version >> takes ~1060 cyles, or about 77% of the time. >> >> Reported-by: Edwin Török <edvin.torok@citrix.com> >> Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com> > Assuming you want this to go in despite your REP STOSQ remark > later on, > Reviewed-by: Jan Beulich <jbeulich@suse.com> > with one remark: > >> --- a/xen/arch/x86/clear_page.S >> +++ b/xen/arch/x86/clear_page.S >> @@ -2,18 +2,16 @@ >> >> #include <asm/page.h> >> >> -#define ptr_reg %rdi >> - >> ENTRY(clear_page_sse2) >> - mov $PAGE_SIZE/16, %ecx >> + mov $PAGE_SIZE/32, %ecx >> xor %eax,%eax >> >> -0: dec %ecx >> - movnti %eax, (ptr_reg) >> - movnti %eax, 4(ptr_reg) >> - movnti %eax, 8(ptr_reg) >> - movnti %eax, 12(ptr_reg) >> - lea 16(ptr_reg), ptr_reg >> +0: movnti %rax, 0(%rdi) > Could I talk you into leaving out this 0? Rather old gas actually emits > an 8-bit displacement when it's spelled like this. Oh ok. I'll still align the (%rdi) though to make the column easier to read. I'll put this in now, and see if I can find some time before 4.13 ships to make some alternatives-based better options. ~Andrew _______________________________________________ Xen-devel mailing list Xen-devel@lists.xenproject.org https://lists.xenproject.org/mailman/listinfo/xen-devel
© 2016 - 2026 Red Hat, Inc.