From nobody Thu Nov 7 16:38:19 2024 Received: from mail-pj1-f48.google.com (mail-pj1-f48.google.com [209.85.216.48]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8657321B451 for ; Thu, 17 Oct 2024 19:00:41 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.48 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1729191643; cv=none; b=jFlyP6686P0CWdgV3XHWdD3L6FMqPAoXLMaKUBt25LW4N0LieYpreY0sPHdS2OQDrxXdQ3+2r7uwZpHuyvtCh/n0aW4Vp2mYjSNRmH+AKMCo8E22kL1BrHq9864hBz+OiydPw2mIo3j3seoC7ihYVB3CcG5SvS7QgHiV16XI4XE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1729191643; c=relaxed/simple; bh=GWkK7tZeIaEdt2CifhxPtw86GQswYXso/LoJx0oCOHY=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=TJ0bEbGaAJOkEOOJw66CiuGN9EFEgkJsxVwxlx8ibj9S4V/fMDiiLjE8oKVr5741F3rplFm88zldjJo16eWFhyHVILaZKj0lo5vOc/+GZDTZqlO/jbwQLUM3/Az4/+CAZdnh5zl7k3UoiY8947HBKwD85YsoxsZ6FCTaAAuScfI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=rivosinc.com; spf=pass smtp.mailfrom=rivosinc.com; dkim=pass (2048-bit key) header.d=rivosinc-com.20230601.gappssmtp.com header.i=@rivosinc-com.20230601.gappssmtp.com header.b=nO/NttKM; arc=none smtp.client-ip=209.85.216.48 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=rivosinc.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=rivosinc.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=rivosinc-com.20230601.gappssmtp.com header.i=@rivosinc-com.20230601.gappssmtp.com header.b="nO/NttKM" Received: by mail-pj1-f48.google.com with SMTP id 98e67ed59e1d1-2e2ad9825a7so947052a91.0 for ; Thu, 17 Oct 2024 12:00:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1729191641; x=1729796441; darn=vger.kernel.org; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :mime-version:subject:date:from:from:to:cc:subject:date:message-id :reply-to; bh=G/BXEqMIqSYxMOqT7xN76pMOEAL23LidRxAEAWgsy/I=; b=nO/NttKM6Wdf3oEsKGUzTJJzVrxk4xlCtd12PlBweUpVElJnTQtXZ3QT9f8Jd/6dHP Udk51cj6ZhHfk69AHfPTQehT/axKJKDrk3n+3Y+UbBU/ijv207XbZJoa525IeXZYje0s L0wbqrYXfyCZJhVm0Wd/5+OBQyrpMLio5vfVmNDNA1fi+YZ2B54rNiKzSJasxx5w+i6s FGlJh6FdUtBjKiHOKnGxmuCaxwjPIpVPJ5vf0adf/JfpRz2gLtvtytS+cI08PsWnC+h0 uTLzUF6aZIlulKEOnqAN6VZElR2ANozFGVXJ4JV712wTiiJBdgbLzAlN30057Wj6KYpD gPeg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1729191641; x=1729796441; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :mime-version:subject:date:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=G/BXEqMIqSYxMOqT7xN76pMOEAL23LidRxAEAWgsy/I=; b=Vox2rbr2gRXIXxjhBHlCWFR0dBut/D/2JXx6g8j08dH+CjoNLtxfnERNvkaBUWPeID morQETK7o6+QY3c2pDH5p7Jd2a4pEmlOwb/5v3YslvJvqhg72FimdKdz+NntnQuJCpuw gS9GKqr7tDuEK40W1y3toR+wVVb+pbSEXkMuYCVnrEiqNpCf9kIf5foswl04EYhW3ruF GBvjpS+VZzaNnYu9I5jyVqnmmgzzPO+Ph8qR3Gp18M3KB8sSe1NhANXB7tnkvhedZiWU xsB563xqPPhawBZalc6x3ahFxz2JiykDa4YFqdp1yxc77qn5TqOSeRWAFTENFSXC72F9 m5fg== X-Forwarded-Encrypted: i=1; AJvYcCW7CdJw8/GjjcgopZPg+dHPO0ZWN96qlZJRyGIDmq4+W58zTERP6qvdGvbFesyNpz3QoCkQYVjZ/tx8NQI=@vger.kernel.org X-Gm-Message-State: AOJu0Yyw745u03nYTd4Z3ruxd88/zRtMXUwLS1btAqDtnS46qbf17FSW 39hn+YHieOJzIe+i3uKw1XNVAWJj1ZS3o2n4y/uttUqRWrh3kIG+dFu8RpSAM4nO7y/UgeM7cDl E X-Google-Smtp-Source: AGHT+IHCgs2HWOhUoTUe59n89YrWVQE0OBgi1zoEPAV99GmBsqSsvTiwAnnbOwVic5dJBGKcLna6Lg== X-Received: by 2002:a17:90a:d90c:b0:2e5:5ab5:ba52 with SMTP id 98e67ed59e1d1-2e56165ea76mr7694a91.20.1729191640739; Thu, 17 Oct 2024 12:00:40 -0700 (PDT) Received: from charlie.ba.rivosinc.com ([64.71.180.162]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-2e55d7bf9a8sm217766a91.13.2024.10.17.12.00.39 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 17 Oct 2024 12:00:39 -0700 (PDT) From: Charlie Jenkins Date: Thu, 17 Oct 2024 12:00:22 -0700 Subject: [PATCH v10 5/6] RISC-V: Report vector unaligned access speed hwprobe Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20241017-jesse_unaligned_vector-v10-5-5b33500160f8@rivosinc.com> References: <20241017-jesse_unaligned_vector-v10-0-5b33500160f8@rivosinc.com> In-Reply-To: <20241017-jesse_unaligned_vector-v10-0-5b33500160f8@rivosinc.com> To: Paul Walmsley , Palmer Dabbelt , Albert Ou , =?utf-8?q?Cl=C3=A9ment_L=C3=A9ger?= , Evan Green , Jonathan Corbet Cc: Palmer Dabbelt , linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, Charlie Jenkins , Jesse Taube X-Mailer: b4 0.13.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=11094; i=charlie@rivosinc.com; h=from:subject:message-id; bh=j/dbDwoe5QvLFXzszJPgbeYRVdxzj8+aKPKL6Ox7jPQ=; b=owGbwMvMwCHWx5hUnlvL8Y3xtFoSQ7pg3Nnbvx62bcvady/ZOWmKrn9T4Mvrnuv85z+vam7u3 vLrzn//jlIWBjEOBlkxRRaeaw3MrXf0y46Klk2AmcPKBDKEgYtTACbiysbIcFtF5ZoJV530BsNs r9laM1KPeP2PffEoY6+j2G3bpxvzTjP8Yg78oWC69EragVrrCUenbtXY9+HJYfH1s7Qlv55e/sz 0Kw8A X-Developer-Key: i=charlie@rivosinc.com; a=openpgp; fpr=7D834FF11B1D8387E61C776FFB10D1F27D6B1354 From: Jesse Taube Detect if vector misaligned accesses are faster or slower than equivalent vector byte accesses. This is useful for usermode to know whether vector byte accesses or vector misaligned accesses have a better bandwidth for operations like memcpy. Signed-off-by: Jesse Taube Reviewed-by: Charlie Jenkins --- arch/riscv/Kconfig | 18 ++++ arch/riscv/kernel/Makefile | 3 +- arch/riscv/kernel/copy-unaligned.h | 5 + arch/riscv/kernel/sys_hwprobe.c | 6 ++ arch/riscv/kernel/unaligned_access_speed.c | 141 +++++++++++++++++++++++++= +++- arch/riscv/kernel/vec-copy-unaligned.S | 58 ++++++++++++ 6 files changed, 228 insertions(+), 3 deletions(-) diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig index 93f9a2958de7..c33311fdfc8c 100644 --- a/arch/riscv/Kconfig +++ b/arch/riscv/Kconfig @@ -856,6 +856,24 @@ config RISCV_PROBE_VECTOR_UNALIGNED_ACCESS will dynamically determine the speed of vector unaligned accesses on the underlying system if they are supported. =20 +config RISCV_SLOW_VECTOR_UNALIGNED_ACCESS + bool "Assume the system supports slow vector unaligned memory accesses" + depends on NONPORTABLE + help + Assume that the system supports slow vector unaligned memory accesses. = The + kernel and userspace programs may not be able to run at all on systems + that do not support unaligned memory accesses. + +config RISCV_EFFICIENT_VECTOR_UNALIGNED_ACCESS + bool "Assume the system supports fast vector unaligned memory accesses" + depends on NONPORTABLE + help + Assume that the system supports fast vector unaligned memory accesses. = When + enabled, this option improves the performance of the kernel on such + systems. However, the kernel and userspace programs will run much more + slowly, or will not be able to run at all, on systems that do not + support efficient unaligned memory accesses. + endchoice =20 source "arch/riscv/Kconfig.vendor" diff --git a/arch/riscv/kernel/Makefile b/arch/riscv/kernel/Makefile index 7f88cc4931f5..30db92672ada 100644 --- a/arch/riscv/kernel/Makefile +++ b/arch/riscv/kernel/Makefile @@ -70,7 +70,8 @@ obj-$(CONFIG_MMU) +=3D vdso.o vdso/ =20 obj-$(CONFIG_RISCV_MISALIGNED) +=3D traps_misaligned.o obj-$(CONFIG_RISCV_MISALIGNED) +=3D unaligned_access_speed.o -obj-$(CONFIG_RISCV_PROBE_UNALIGNED_ACCESS) +=3D copy-unaligned.o +obj-$(CONFIG_RISCV_PROBE_UNALIGNED_ACCESS) +=3D copy-unaligned.o +obj-$(CONFIG_RISCV_PROBE_VECTOR_UNALIGNED_ACCESS) +=3D vec-copy-unaligned.o =20 obj-$(CONFIG_FPU) +=3D fpu.o obj-$(CONFIG_FPU) +=3D kernel_mode_fpu.o diff --git a/arch/riscv/kernel/copy-unaligned.h b/arch/riscv/kernel/copy-un= aligned.h index e3d70d35b708..85d4d11450cb 100644 --- a/arch/riscv/kernel/copy-unaligned.h +++ b/arch/riscv/kernel/copy-unaligned.h @@ -10,4 +10,9 @@ void __riscv_copy_words_unaligned(void *dst, const void *src, size_t size); void __riscv_copy_bytes_unaligned(void *dst, const void *src, size_t size); =20 +#ifdef CONFIG_RISCV_PROBE_VECTOR_UNALIGNED_ACCESS +void __riscv_copy_vec_words_unaligned(void *dst, const void *src, size_t s= ize); +void __riscv_copy_vec_bytes_unaligned(void *dst, const void *src, size_t s= ize); +#endif + #endif /* __RISCV_KERNEL_COPY_UNALIGNED_H */ diff --git a/arch/riscv/kernel/sys_hwprobe.c b/arch/riscv/kernel/sys_hwprob= e.c index 6441baada36b..6673278e84d5 100644 --- a/arch/riscv/kernel/sys_hwprobe.c +++ b/arch/riscv/kernel/sys_hwprobe.c @@ -228,6 +228,12 @@ static u64 hwprobe_vec_misaligned(const struct cpumask= *cpus) #else static u64 hwprobe_vec_misaligned(const struct cpumask *cpus) { + if (IS_ENABLED(CONFIG_RISCV_EFFICIENT_VECTOR_UNALIGNED_ACCESS)) + return RISCV_HWPROBE_MISALIGNED_VECTOR_FAST; + + if (IS_ENABLED(CONFIG_RISCV_SLOW_VECTOR_UNALIGNED_ACCESS)) + return RISCV_HWPROBE_MISALIGNED_VECTOR_SLOW; + return RISCV_HWPROBE_MISALIGNED_VECTOR_UNKNOWN; } #endif diff --git a/arch/riscv/kernel/unaligned_access_speed.c b/arch/riscv/kernel= /unaligned_access_speed.c index 0b8b5e17453a..91f189cf1611 100644 --- a/arch/riscv/kernel/unaligned_access_speed.c +++ b/arch/riscv/kernel/unaligned_access_speed.c @@ -6,11 +6,13 @@ #include #include #include +#include #include #include #include #include #include +#include =20 #include "copy-unaligned.h" =20 @@ -268,12 +270,147 @@ static int check_unaligned_access_speed_all_cpus(voi= d) } #endif =20 +#ifdef CONFIG_RISCV_PROBE_VECTOR_UNALIGNED_ACCESS +static void check_vector_unaligned_access(struct work_struct *work __alway= s_unused) +{ + int cpu =3D smp_processor_id(); + u64 start_cycles, end_cycles; + u64 word_cycles; + u64 byte_cycles; + int ratio; + unsigned long start_jiffies, now; + struct page *page; + void *dst; + void *src; + long speed =3D RISCV_HWPROBE_MISALIGNED_VECTOR_SLOW; + + if (per_cpu(vector_misaligned_access, cpu) !=3D RISCV_HWPROBE_MISALIGNED_= VECTOR_UNKNOWN) + return; + + page =3D alloc_pages(GFP_KERNEL, MISALIGNED_BUFFER_ORDER); + if (!page) { + pr_warn("Allocation failure, not measuring vector misaligned performance= \n"); + return; + } + + /* Make an unaligned destination buffer. */ + dst =3D (void *)((unsigned long)page_address(page) | 0x1); + /* Unalign src as well, but differently (off by 1 + 2 =3D 3). */ + src =3D dst + (MISALIGNED_BUFFER_SIZE / 2); + src +=3D 2; + word_cycles =3D -1ULL; + + /* Do a warmup. */ + kernel_vector_begin(); + __riscv_copy_vec_words_unaligned(dst, src, MISALIGNED_COPY_SIZE); + + start_jiffies =3D jiffies; + while ((now =3D jiffies) =3D=3D start_jiffies) + cpu_relax(); + + /* + * For a fixed amount of time, repeatedly try the function, and take + * the best time in cycles as the measurement. + */ + while (time_before(jiffies, now + (1 << MISALIGNED_ACCESS_JIFFIES_LG2))) { + start_cycles =3D get_cycles64(); + /* Ensure the CSR read can't reorder WRT to the copy. */ + mb(); + __riscv_copy_vec_words_unaligned(dst, src, MISALIGNED_COPY_SIZE); + /* Ensure the copy ends before the end time is snapped. */ + mb(); + end_cycles =3D get_cycles64(); + if ((end_cycles - start_cycles) < word_cycles) + word_cycles =3D end_cycles - start_cycles; + } + + byte_cycles =3D -1ULL; + __riscv_copy_vec_bytes_unaligned(dst, src, MISALIGNED_COPY_SIZE); + start_jiffies =3D jiffies; + while ((now =3D jiffies) =3D=3D start_jiffies) + cpu_relax(); + + while (time_before(jiffies, now + (1 << MISALIGNED_ACCESS_JIFFIES_LG2))) { + start_cycles =3D get_cycles64(); + /* Ensure the CSR read can't reorder WRT to the copy. */ + mb(); + __riscv_copy_vec_bytes_unaligned(dst, src, MISALIGNED_COPY_SIZE); + /* Ensure the copy ends before the end time is snapped. */ + mb(); + end_cycles =3D get_cycles64(); + if ((end_cycles - start_cycles) < byte_cycles) + byte_cycles =3D end_cycles - start_cycles; + } + + kernel_vector_end(); + + /* Don't divide by zero. */ + if (!word_cycles || !byte_cycles) { + pr_warn("cpu%d: rdtime lacks granularity needed to measure unaligned vec= tor access speed\n", + cpu); + + return; + } + + if (word_cycles < byte_cycles) + speed =3D RISCV_HWPROBE_MISALIGNED_VECTOR_FAST; + + ratio =3D div_u64((byte_cycles * 100), word_cycles); + pr_info("cpu%d: Ratio of vector byte access time to vector unaligned word= access is %d.%02d, unaligned accesses are %s\n", + cpu, + ratio / 100, + ratio % 100, + (speed =3D=3D RISCV_HWPROBE_MISALIGNED_VECTOR_FAST) ? "fast" : "slow"); + + per_cpu(vector_misaligned_access, cpu) =3D speed; +} + +static int riscv_online_cpu_vec(unsigned int cpu) +{ + if (!has_vector()) + return 0; + + if (per_cpu(vector_misaligned_access, cpu) !=3D RISCV_HWPROBE_MISALIGNED_= VECTOR_UNSUPPORTED) + return 0; + + check_vector_unaligned_access_emulated(NULL); + check_vector_unaligned_access(NULL); + return 0; +} + +/* Measure unaligned access speed on all CPUs present at boot in parallel.= */ +static int vec_check_unaligned_access_speed_all_cpus(void *unused __always= _unused) +{ + schedule_on_each_cpu(check_vector_unaligned_access); + + /* + * Setup hotplug callbacks for any new CPUs that come online or go + * offline. + */ + cpuhp_setup_state_nocalls(CPUHP_AP_ONLINE_DYN, "riscv:online", + riscv_online_cpu_vec, NULL); + + return 0; +} +#else /* CONFIG_RISCV_PROBE_VECTOR_UNALIGNED_ACCESS */ +static int vec_check_unaligned_access_speed_all_cpus(void *unused __always= _unused) +{ + return 0; +} +#endif + static int check_unaligned_access_all_cpus(void) { - bool all_cpus_emulated; + bool all_cpus_emulated, all_cpus_vec_unsupported; =20 all_cpus_emulated =3D check_unaligned_access_emulated_all_cpus(); - check_vector_unaligned_access_emulated_all_cpus(); + all_cpus_vec_unsupported =3D check_vector_unaligned_access_emulated_all_c= pus(); + + if (!all_cpus_vec_unsupported && + IS_ENABLED(CONFIG_RISCV_PROBE_VECTOR_UNALIGNED_ACCESS)) { + kthread_run(vec_check_unaligned_access_speed_all_cpus, + NULL, "vec_check_unaligned_access_speed_all_cpus"); + } =20 if (!all_cpus_emulated) return check_unaligned_access_speed_all_cpus(); diff --git a/arch/riscv/kernel/vec-copy-unaligned.S b/arch/riscv/kernel/vec= -copy-unaligned.S new file mode 100644 index 000000000000..d16f19f1b3b6 --- /dev/null +++ b/arch/riscv/kernel/vec-copy-unaligned.S @@ -0,0 +1,58 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* Copyright (C) 2024 Rivos Inc. */ + +#include +#include +#include + + .text + +#define WORD_EEW 32 + +#define WORD_SEW CONCATENATE(e, WORD_EEW) +#define VEC_L CONCATENATE(vle, WORD_EEW).v +#define VEC_S CONCATENATE(vle, WORD_EEW).v + +/* void __riscv_copy_vec_words_unaligned(void *, const void *, size_t) */ +/* Performs a memcpy without aligning buffers, using word loads and stores= . */ +/* Note: The size is truncated to a multiple of WORD_EEW */ +SYM_FUNC_START(__riscv_copy_vec_words_unaligned) + andi a4, a2, ~(WORD_EEW-1) + beqz a4, 2f + add a3, a1, a4 + .option push + .option arch, +zve32x +1: + vsetivli t0, 8, WORD_SEW, m8, ta, ma + VEC_L v0, (a1) + VEC_S v0, (a0) + addi a0, a0, WORD_EEW + addi a1, a1, WORD_EEW + bltu a1, a3, 1b + +2: + .option pop + ret +SYM_FUNC_END(__riscv_copy_vec_words_unaligned) + +/* void __riscv_copy_vec_bytes_unaligned(void *, const void *, size_t) */ +/* Performs a memcpy without aligning buffers, using only byte accesses. */ +/* Note: The size is truncated to a multiple of 8 */ +SYM_FUNC_START(__riscv_copy_vec_bytes_unaligned) + andi a4, a2, ~(8-1) + beqz a4, 2f + add a3, a1, a4 + .option push + .option arch, +zve32x +1: + vsetivli t0, 8, e8, m8, ta, ma + vle8.v v0, (a1) + vse8.v v0, (a0) + addi a0, a0, 8 + addi a1, a1, 8 + bltu a1, a3, 1b + +2: + .option pop + ret +SYM_FUNC_END(__riscv_copy_vec_bytes_unaligned) --=20 2.45.0