From nobody Fri Mar 29 08:18:09 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of groups.io designates 66.175.222.12 as permitted sender) client-ip=66.175.222.12; envelope-from=bounce+27952+66094+1787277+3901457@groups.io; helo=web01.groups.io; Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of groups.io designates 66.175.222.12 as permitted sender) smtp.mailfrom=bounce+27952+66094+1787277+3901457@groups.io; dmarc=fail(p=none dis=none) header.from=intel.com ARC-Seal: i=1; a=rsa-sha256; t=1602300513; cv=none; d=zohomail.com; s=zohoarc; b=nEFokDT4i8bW1xRe4gEzw95UFZsE85RNvDNBC0z0nnsmR+jZYlwp/gS/rT6WhDMBm3rCb/2rM/gEDyzUM6DbbxtmS6CixbMRD3+OaI8hIt+t9BJxngcybv95n0TxZjQjoxDLsIgJNP6ZVGw5sgGl0UKZ0gt+KJd0re1DdLChC3s= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1602300513; h=Content-Type:Date:From:List-Id:List-Unsubscribe:MIME-Version:Message-ID:Reply-To:Sender:Subject:To; bh=zLrGn4I0I05WdGAqbYkcD/qW3p1Q4mWtfzlE9bPuLUQ=; b=JcWC5woA9jt1bDxHyysmIufE3Z8IHxlh4s//1UVg4VxfBhG7FMk1aZquaf9rATaVMjXgmcK7BxRGzWobQvhI/zuHv6x2MeisvR/r3PVklLh89PLEux4qF5zXiry0bv92retxA0nNUzK5CMQt1O8bE4Qae52NcptVpoptgdTnJiw= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of groups.io designates 66.175.222.12 as permitted sender) smtp.mailfrom=bounce+27952+66094+1787277+3901457@groups.io; dmarc=fail header.from= (p=none dis=none) header.from= Received: from web01.groups.io (web01.groups.io [66.175.222.12]) by mx.zohomail.com with SMTPS id 1602300513511743.3158398025878; Fri, 9 Oct 2020 20:28:33 -0700 (PDT) Return-Path: X-Received: by 127.0.0.2 with SMTP id o3W3YY1788612xMlcTdfuP8y; Fri, 09 Oct 2020 20:28:33 -0700 X-Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by mx.groups.io with SMTP id smtpd.web11.2051.1602276878261351835 for ; Fri, 09 Oct 2020 13:54:38 -0700 IronPort-SDR: ajFhVQgFKICIelqoBIUIYa5T6gZzXsqGWqBJPu+9KwVE1dZKrTV54q8P7KlrjBdd+5/D6sh/Nx AEpEDAokXabA== X-IronPort-AV: E=McAfee;i="6000,8403,9769"; a="164748887" X-IronPort-AV: E=Sophos;i="5.77,356,1596524400"; d="scan'208";a="164748887" X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False X-Received: from fmsmga007.fm.intel.com ([10.253.24.52]) by fmsmga103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 09 Oct 2020 13:54:37 -0700 IronPort-SDR: XuDmKIO82++Fsbrx2V1jpmKK4hMUXjZCxbnV29ZfK/GaSA41G9lZFRLTXun10BDbQM8x1VrsDn AUYSQuvQo/hA== X-IronPort-AV: E=Sophos;i="5.77,356,1596524400"; d="scan'208";a="298433178" X-Received: from jcompost-mobl.ch.intel.com ([10.209.179.250]) by fmsmga007-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 09 Oct 2020 13:54:37 -0700 From: "Compostella, Jeremy" To: Subject: [edk2-devel] [PATCH] BaseMemoryLibSse2: Take advantage of write combining buffers Organization: Intel Corportation - 2200 Mission College Blvd. Santa Clara, CA 95052. USA User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.3 (gnu/linux) Date: Fri, 09 Oct 2020 13:54:36 -0700 Message-ID: <87r1q76rxv.fsf@jcompost-mobl.ch.intel.com> MIME-Version: 1.0 Precedence: Bulk List-Unsubscribe: Sender: devel@edk2.groups.io List-Id: Mailing-List: list devel@edk2.groups.io; contact devel+owner@edk2.groups.io Reply-To: devel@edk2.groups.io,jeremy.compostella@intel.com X-Gm-Message-State: qnmizRsA9BOO2lmvp2vs1zw6x1787277AA= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=groups.io; q=dns/txt; s=20140610; t=1602300513; bh=jqoqmmKQKAisejPdqXk/rHdzufLJXHH7u+VqKbUs6DU=; h=Content-Type:Date:From:Reply-To:Subject:To; b=JedOsqPeXLhj3c4+gVZjsAt13zPizK4nZPell9YnAgqwsntHhg3+XErcehA/3tzDmm5 kJBEGINKPMiitdChmwe1gDBEK1xUhHlcb/302m3ud9bm9/e2XC7o2qDWvOmD0vSEBIKr7 IGZ7Cy/hWFUawnez0/NvnqqxutHC4PaOLZ4= X-ZohoMail-DKIM: pass (identity @groups.io) Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" The current SSE2 implementation of the ZeroMem(), SetMem(), SetMem16(), SetMem32() and SetMem64() functions is writing 16 bytes per 16 bytes. It hurts the performances so bad that this is even slower than a simple 'rep stos' (4% slower) in regular RAM. To take full advantages of the 'movntdq' instruction it is better to "queue" a total of 64 bytes in the write combining buffers. This patch implements such a change. Below is a table where I measured (with 'rdtsc') the time to write an entire 100MB RAM buffer. These functions operate almost twice faster. | Function | Arch | Untouched | 64 bytes | Result | |----------+------+-----------+----------+--------| | ZeroMem | Ia32 | 17765947 | 9136062 | 1.945x | | ZeroMem | X64 | 17525170 | 9233391 | 1.898x | | SetMem | Ia32 | 17522291 | 9137272 | 1.918x | | SetMem | X64 | 17949261 | 9176978 | 1.956x | | SetMem16 | Ia32 | 18219673 | 9372062 | 1.944x | | SetMem16 | X64 | 17523331 | 9275184 | 1.889x | | SetMem32 | Ia32 | 18495036 | 9273053 | 1.994x | | SetMem32 | X64 | 17368864 | 9285885 | 1.870x | | SetMem64 | Ia32 | 18564473 | 9241362 | 2.009x | | SetMem64 | X64 | 17506951 | 9280148 | 1.886x | Signed-off-by: Jeremy Compostella --- .../BaseMemoryLibSse2/Ia32/SetMem.nasm | 11 ++++++---- .../BaseMemoryLibSse2/Ia32/SetMem16.nasm | 11 ++++++---- .../BaseMemoryLibSse2/Ia32/SetMem32.nasm | 9 ++++++--- .../BaseMemoryLibSse2/Ia32/SetMem64.nasm | 20 +++++++++++++++---- .../BaseMemoryLibSse2/Ia32/ZeroMem.nasm | 11 ++++++---- .../Library/BaseMemoryLibSse2/X64/SetMem.nasm | 9 ++++++--- .../BaseMemoryLibSse2/X64/SetMem16.nasm | 11 ++++++---- .../BaseMemoryLibSse2/X64/SetMem32.nasm | 9 ++++++--- .../BaseMemoryLibSse2/X64/SetMem64.nasm | 19 ++++++++++++++---- .../BaseMemoryLibSse2/X64/ZeroMem.nasm | 13 +++++++----- 10 files changed, 85 insertions(+), 38 deletions(-) diff --git a/MdePkg/Library/BaseMemoryLibSse2/Ia32/SetMem.nasm b/MdePkg/Lib= rary/BaseMemoryLibSse2/Ia32/SetMem.nasm index 24313cb4b3..a8744300c6 100644 --- a/MdePkg/Library/BaseMemoryLibSse2/Ia32/SetMem.nasm +++ b/MdePkg/Library/BaseMemoryLibSse2/Ia32/SetMem.nasm @@ -34,7 +34,7 @@ ASM_PFX(InternalMemSetMem): mov al, [esp + 16] ; al <- Value xor ecx, ecx sub ecx, edi - and ecx, 15 ; ecx + edi aligns on 16-byte boun= dary + and ecx, 63 ; ecx + edi aligns on 16-byte boun= dary jz .0 cmp ecx, edx cmova ecx, edx @@ -42,8 +42,8 @@ ASM_PFX(InternalMemSetMem): rep stosb .0: mov ecx, edx - and edx, 15 - shr ecx, 4 ; ecx <- # of DQwords to set + and edx, 63 + shr ecx, 6 ; ecx <- # of DQwords to set jz @SetBytes mov ah, al ; ax <- Value | (Value << 8) add esp, -16 @@ -53,7 +53,10 @@ ASM_PFX(InternalMemSetMem): movlhps xmm0, xmm0 ; xmm0 <- Value repeats 16 times .1: movntdq [edi], xmm0 ; edi should be 16-byte aligned - add edi, 16 + movntdq [edi + 16], xmm0 + movntdq [edi + 32], xmm0 + movntdq [edi + 48], xmm0 + add edi, 64 loop .1 mfence movdqu xmm0, [esp] ; restore xmm0 diff --git a/MdePkg/Library/BaseMemoryLibSse2/Ia32/SetMem16.nasm b/MdePkg/L= ibrary/BaseMemoryLibSse2/Ia32/SetMem16.nasm index 6e308b5594..d461ee086c 100644 --- a/MdePkg/Library/BaseMemoryLibSse2/Ia32/SetMem16.nasm +++ b/MdePkg/Library/BaseMemoryLibSse2/Ia32/SetMem16.nasm @@ -33,7 +33,7 @@ ASM_PFX(InternalMemSetMem16): mov edi, [esp + 8] xor ecx, ecx sub ecx, edi - and ecx, 15 ; ecx + edi aligns on 16-byte boun= dary + and ecx, 63 ; ecx + edi aligns on 16-byte boun= dary mov eax, [esp + 16] jz .0 shr ecx, 1 @@ -43,15 +43,18 @@ ASM_PFX(InternalMemSetMem16): rep stosw .0: mov ecx, edx - and edx, 7 - shr ecx, 3 + and edx, 31 + shr ecx, 5 jz @SetWords movd xmm0, eax pshuflw xmm0, xmm0, 0 movlhps xmm0, xmm0 .1: movntdq [edi], xmm0 ; edi should be 16-byte aligned - add edi, 16 + movntdq [edi + 16], xmm0 + movntdq [edi + 32], xmm0 + movntdq [edi + 48], xmm0 + add edi, 64 loop .1 mfence @SetWords: diff --git a/MdePkg/Library/BaseMemoryLibSse2/Ia32/SetMem32.nasm b/MdePkg/L= ibrary/BaseMemoryLibSse2/Ia32/SetMem32.nasm index 2cfc8cb0dd..3ffdcd07d7 100644 --- a/MdePkg/Library/BaseMemoryLibSse2/Ia32/SetMem32.nasm +++ b/MdePkg/Library/BaseMemoryLibSse2/Ia32/SetMem32.nasm @@ -43,14 +43,17 @@ ASM_PFX(InternalMemSetMem32): rep stosd .0: mov ecx, edx - and edx, 3 - shr ecx, 2 + and edx, 15 + shr ecx, 4 jz @SetDwords movd xmm0, eax pshufd xmm0, xmm0, 0 .1: movntdq [edi], xmm0 - add edi, 16 + movntdq [edi + 16], xmm0 + movntdq [edi + 32], xmm0 + movntdq [edi + 48], xmm0 + add edi, 64 loop .1 mfence @SetDwords: diff --git a/MdePkg/Library/BaseMemoryLibSse2/Ia32/SetMem64.nasm b/MdePkg/L= ibrary/BaseMemoryLibSse2/Ia32/SetMem64.nasm index e153495a68..cd000648ae 100644 --- a/MdePkg/Library/BaseMemoryLibSse2/Ia32/SetMem64.nasm +++ b/MdePkg/Library/BaseMemoryLibSse2/Ia32/SetMem64.nasm @@ -38,17 +38,29 @@ ASM_PFX(InternalMemSetMem64): add edx, 8 dec ecx .0: - shr ecx, 1 + push ebx + mov ebx, ecx + and ebx, 7 + shr ecx, 3 jz @SetQwords movlhps xmm0, xmm0 .1: movntdq [edx], xmm0 - lea edx, [edx + 16] + movntdq [edx + 16], xmm0 + movntdq [edx + 32], xmm0 + movntdq [edx + 48], xmm0 + lea edx, [edx + 64] loop .1 mfence @SetQwords: - jnc .2 + test ebx, ebx + jz .3 + mov ecx, ebx +.2 movq qword [edx], xmm0 -.2: + lea edx, [edx + 8] + loop .2 +.3: + pop ebx ret =20 diff --git a/MdePkg/Library/BaseMemoryLibSse2/Ia32/ZeroMem.nasm b/MdePkg/Li= brary/BaseMemoryLibSse2/Ia32/ZeroMem.nasm index cd34006f59..0e0828551b 100644 --- a/MdePkg/Library/BaseMemoryLibSse2/Ia32/ZeroMem.nasm +++ b/MdePkg/Library/BaseMemoryLibSse2/Ia32/ZeroMem.nasm @@ -33,7 +33,7 @@ ASM_PFX(InternalMemZeroMem): xor ecx, ecx sub ecx, edi xor eax, eax - and ecx, 15 + and ecx, 63 jz .0 cmp ecx, edx cmova ecx, edx @@ -41,13 +41,16 @@ ASM_PFX(InternalMemZeroMem): rep stosb .0: mov ecx, edx - and edx, 15 - shr ecx, 4 + and edx, 63 + shr ecx, 6 jz @ZeroBytes pxor xmm0, xmm0 .1: movntdq [edi], xmm0 - add edi, 16 + movntdq [edi + 16], xmm0 + movntdq [edi + 32], xmm0 + movntdq [edi + 48], xmm0 + add edi, 64 loop .1 mfence @ZeroBytes: diff --git a/MdePkg/Library/BaseMemoryLibSse2/X64/SetMem.nasm b/MdePkg/Libr= ary/BaseMemoryLibSse2/X64/SetMem.nasm index 5bd1c2262d..28b11ee586 100644 --- a/MdePkg/Library/BaseMemoryLibSse2/X64/SetMem.nasm +++ b/MdePkg/Library/BaseMemoryLibSse2/X64/SetMem.nasm @@ -42,8 +42,8 @@ ASM_PFX(InternalMemSetMem): rep stosb .0: mov rcx, rdx - and rdx, 15 - shr rcx, 4 + and rdx, 63 + shr rcx, 6 jz @SetBytes mov ah, al ; ax <- Value repeats twice movdqa [rsp + 0x10], xmm0 ; save xmm0 @@ -52,7 +52,10 @@ ASM_PFX(InternalMemSetMem): movlhps xmm0, xmm0 ; xmm0 <- Value repeats 16 times .1: movntdq [rdi], xmm0 ; rdi should be 16-byte aligned - add rdi, 16 + movntdq [rdi + 16], xmm0 + movntdq [rdi + 32], xmm0 + movntdq [rdi + 48], xmm0 + add rdi, 64 loop .1 mfence movdqa xmm0, [rsp + 0x10] ; restore xmm0 diff --git a/MdePkg/Library/BaseMemoryLibSse2/X64/SetMem16.nasm b/MdePkg/Li= brary/BaseMemoryLibSse2/X64/SetMem16.nasm index 90d159820a..375be19313 100644 --- a/MdePkg/Library/BaseMemoryLibSse2/X64/SetMem16.nasm +++ b/MdePkg/Library/BaseMemoryLibSse2/X64/SetMem16.nasm @@ -33,7 +33,7 @@ ASM_PFX(InternalMemSetMem16): mov r9, rdi xor rcx, rcx sub rcx, rdi - and rcx, 15 + and rcx, 63 mov rax, r8 jz .0 shr rcx, 1 @@ -43,15 +43,18 @@ ASM_PFX(InternalMemSetMem16): rep stosw .0: mov rcx, rdx - and edx, 7 - shr rcx, 3 + and edx, 31 + shr rcx, 5 jz @SetWords movd xmm0, eax pshuflw xmm0, xmm0, 0 movlhps xmm0, xmm0 .1: movntdq [rdi], xmm0 - add rdi, 16 + movntdq [rdi + 16], xmm0 + movntdq [rdi + 32], xmm0 + movntdq [rdi + 48], xmm0 + add rdi, 64 loop .1 mfence @SetWords: diff --git a/MdePkg/Library/BaseMemoryLibSse2/X64/SetMem32.nasm b/MdePkg/Li= brary/BaseMemoryLibSse2/X64/SetMem32.nasm index 928e086889..5d12beaa9a 100644 --- a/MdePkg/Library/BaseMemoryLibSse2/X64/SetMem32.nasm +++ b/MdePkg/Library/BaseMemoryLibSse2/X64/SetMem32.nasm @@ -43,14 +43,17 @@ ASM_PFX(InternalMemSetMem32): rep stosd .0: mov rcx, rdx - and edx, 3 - shr rcx, 2 + and edx, 15 + shr rcx, 4 jz @SetDwords movd xmm0, eax pshufd xmm0, xmm0, 0 .1: movntdq [rdi], xmm0 - add rdi, 16 + movntdq [rdi + 16], xmm0 + movntdq [rdi + 32], xmm0 + movntdq [rdi + 48], xmm0 + add rdi, 64 loop .1 mfence @SetDwords: diff --git a/MdePkg/Library/BaseMemoryLibSse2/X64/SetMem64.nasm b/MdePkg/Li= brary/BaseMemoryLibSse2/X64/SetMem64.nasm index d771810542..265983d5ad 100644 --- a/MdePkg/Library/BaseMemoryLibSse2/X64/SetMem64.nasm +++ b/MdePkg/Library/BaseMemoryLibSse2/X64/SetMem64.nasm @@ -37,17 +37,28 @@ ASM_PFX(InternalMemSetMem64): add rdx, 8 dec rcx .0: - shr rcx, 1 + push rbx + mov rbx, rcx + and rbx, 7 + shr rcx, 3 jz @SetQwords movlhps xmm0, xmm0 .1: movntdq [rdx], xmm0 - lea rdx, [rdx + 16] + movntdq [rdx + 16], xmm0 + movntdq [rdx + 32], xmm0 + movntdq [rdx + 48], xmm0 + lea rdx, [rdx + 64] loop .1 mfence @SetQwords: - jnc .2 - mov [rdx], r8 + push rdi + mov rcx, rbx + mov rax, r8 + mov rdi, rdx + rep stosq + pop rdi=09 .2: + pop rbx ret =20 diff --git a/MdePkg/Library/BaseMemoryLibSse2/X64/ZeroMem.nasm b/MdePkg/Lib= rary/BaseMemoryLibSse2/X64/ZeroMem.nasm index 5ddcae9ca5..21f504e3b7 100644 --- a/MdePkg/Library/BaseMemoryLibSse2/X64/ZeroMem.nasm +++ b/MdePkg/Library/BaseMemoryLibSse2/X64/ZeroMem.nasm @@ -32,7 +32,7 @@ ASM_PFX(InternalMemZeroMem): xor rcx, rcx xor eax, eax sub rcx, rdi - and rcx, 15 + and rcx, 63 mov r8, rdi jz .0 cmp rcx, rdx @@ -41,13 +41,16 @@ ASM_PFX(InternalMemZeroMem): rep stosb .0: mov rcx, rdx - and edx, 15 - shr rcx, 4 + and edx, 63 + shr rcx, 6 jz @ZeroBytes pxor xmm0, xmm0 .1: - movntdq [rdi], xmm0 ; rdi should be 16-byte aligned - add rdi, 16 + movntdq [rdi], xmm0 + movntdq [rdi + 16], xmm0 + movntdq [rdi + 32], xmm0 + movntdq [rdi + 48], xmm0 + add rdi, 64 loop .1 mfence @ZeroBytes: --=20 2.25.3 -=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D- Groups.io Links: You receive all messages sent to this group. View/Reply Online (#66094): https://edk2.groups.io/g/devel/message/66094 Mute This Topic: https://groups.io/mt/77417951/1787277 Group Owner: devel+owner@edk2.groups.io Unsubscribe: https://edk2.groups.io/g/devel/unsub [importer@patchew.org] -=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-