From nobody Thu Nov 7 16:51:05 2024 Received: from mail-pj1-f49.google.com (mail-pj1-f49.google.com [209.85.216.49]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0C18B21C16D for ; Thu, 17 Oct 2024 19:00:39 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.49 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1729191643; cv=none; b=GYvT5gpwg9kQPaszHvctyDTGKg+n4F6gx3XcI9zwHJietypB47Co54AHV9D/2+Cc/Atyo4z2vogF+ksZNiPch1qIYVMYuUllAaX468OiPZ5ZfAMXgMKbeNhN5xsNS4Wpdo3kdfMelJH53YBaY4yE/wDUYPTcKyI33o5O+i8ID+M= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1729191643; c=relaxed/simple; bh=Az7W+wt6uT29Whn1yI/Cp0xJjX8tW0mabXSDx1cU7Aw=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=LfYIX2aeN+0dBqcZSRHyLUkoPN1HI6u+RZnoc/DK+sLUwaNXBPDxIgDfWZJSA1XDJuLB2J7zo1YpCWU0G0E/GUkXU1BX2DtOJLZT3AhQMvPHY8bhTJONk5HR2nvz294smtVtJIH6hbXbBduNYD/lkIauxqq/OgRZTrZNDkRWjAk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=rivosinc.com; spf=pass smtp.mailfrom=rivosinc.com; dkim=pass (2048-bit key) header.d=rivosinc-com.20230601.gappssmtp.com header.i=@rivosinc-com.20230601.gappssmtp.com header.b=qW/kZGAP; arc=none smtp.client-ip=209.85.216.49 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=rivosinc.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=rivosinc.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=rivosinc-com.20230601.gappssmtp.com header.i=@rivosinc-com.20230601.gappssmtp.com header.b="qW/kZGAP" Received: by mail-pj1-f49.google.com with SMTP id 98e67ed59e1d1-2e31af47681so1041708a91.2 for ; Thu, 17 Oct 2024 12:00:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1729191639; x=1729796439; darn=vger.kernel.org; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :mime-version:subject:date:from:from:to:cc:subject:date:message-id :reply-to; bh=KsZdRYs87XBY7bpyyAW292fo0y859KGq52kYcY8Yvrc=; b=qW/kZGAPw614hx1aXJo2LEEwSn4r3zv18KJraY7FaVL6BbY2+nNX6FCMFzjLX/B0Ae tILLM/ZG2al3k9+cZkX7QqcPChxLBx2/tFo/U7yiK4yJcfYkl42+EGLcTjvfK2AysAFw cYLqXrJF9qg7UjeJ6i70gO0LOKCorEfNvWIksgdJ2DRMtkB4CAzE0BU/jSr6r28ExPLh ipLQrWuWlui+dbYUodDcTTBibqEJy5XUh/CbfUOF0k1k/cuzEiK/VgAAdAnNcroRocVK TnxfYR0FlEZbpv8YI+dmH2DNAJOazYBGnze6Bz6YWGJKXbyavgTZUthJUWHdAl1Tqkz5 G6Pw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1729191639; x=1729796439; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :mime-version:subject:date:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=KsZdRYs87XBY7bpyyAW292fo0y859KGq52kYcY8Yvrc=; b=B8kSChgZ9AWmaFVpMZCQdy2+wY1Z1exdby7pNNVeZnkq3AgYOZ2fPFecKoL5wonb05 3KFoA+0B13ssFwmpxMdkmQ2nMRA6RmsD6cl+YzyjNQVytBDIlyAgM0ztC0dOi2SxDXxf alSRMLS8EV7TPH6SQPkuupWQQcfBho49L/MBwC+qnRKShlMkHM8bgZ0B78ffZJ7zyO9V jcg6ntbWBPyVK/AhoFcJO3lqzgF9nT+2D9VVkPfWghhMQ+rVJcwDTLKUwX0P0X3Y+b/x 6S/74Tzy18hrtiqvg59iD49griF9kVA2OwfWuJoAfXnJobtsTqpPy+z9elLG4QF3ijs9 sRsg== X-Forwarded-Encrypted: i=1; AJvYcCXkK36M+IifMGJw7fXWCBULKnmr6XzHkBMr/O3qZGLrGHbQqdpLuKrPMVcNuofMMtkF9LVYBcHvfwzW7ZI=@vger.kernel.org X-Gm-Message-State: AOJu0YwOPkH361AkR6FAjYvIHaTxwX4xxNNVRekzTOQ0KduooTaj5Zgv C8AL3Toz7+ozdMnjAu/AGh09IgjK9Y2pkSZhW/OyZ8DVtqN8Z5Pq9GCDleM2VrbapSZfl4deEug B X-Google-Smtp-Source: AGHT+IE7+lO/nh+eyK24dV2IQYdNWKYq47gywhkkqrpuepLY9z2MMXEzO5my2vsNTJbWwB1OhkwMuQ== X-Received: by 2002:a17:90a:8c0c:b0:2e2:be64:488f with SMTP id 98e67ed59e1d1-2e2f0a6ea07mr25087823a91.6.1729191639163; Thu, 17 Oct 2024 12:00:39 -0700 (PDT) Received: from charlie.ba.rivosinc.com ([64.71.180.162]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-2e55d7bf9a8sm217766a91.13.2024.10.17.12.00.37 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 17 Oct 2024 12:00:38 -0700 (PDT) From: Charlie Jenkins Date: Thu, 17 Oct 2024 12:00:21 -0700 Subject: [PATCH v10 4/6] RISC-V: Detect unaligned vector accesses supported Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20241017-jesse_unaligned_vector-v10-4-5b33500160f8@rivosinc.com> References: <20241017-jesse_unaligned_vector-v10-0-5b33500160f8@rivosinc.com> In-Reply-To: <20241017-jesse_unaligned_vector-v10-0-5b33500160f8@rivosinc.com> To: Paul Walmsley , Palmer Dabbelt , Albert Ou , =?utf-8?q?Cl=C3=A9ment_L=C3=A9ger?= , Evan Green , Jonathan Corbet Cc: Palmer Dabbelt , linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, Charlie Jenkins , Jesse Taube X-Mailer: b4 0.13.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=15718; i=charlie@rivosinc.com; h=from:subject:message-id; bh=A1vacXhLHy/JZ4H+XAJQLSLECx28ua/pbpRywF5ZqUk=; b=owGbwMvMwCHWx5hUnlvL8Y3xtFoSQ7pg3NnZFy9Mdr+b9FHt47W//A2S6+KElzK8u3A4svTUp f3FSSvVO0pZGMQ4GGTFFFl4rjUwt97RLzsqWjYBZg4rE8gQBi5OAZjIvcsM/53nHniQ1D5vD9Of k9ERO7/enfE7jTd3he2dYHfGS44qCzMZGdZ+K5j5be65U5dm3BC3rGzmkXc7qd0W6+/hc4yv/J9 dIiMA X-Developer-Key: i=charlie@rivosinc.com; a=openpgp; fpr=7D834FF11B1D8387E61C776FFB10D1F27D6B1354 From: Jesse Taube Run an unaligned vector access to test if the system supports vector unaligned access. Add the result to a new key in hwprobe. This is useful for usermode to know if vector misaligned accesses are supported and if they are faster or slower than equivalent byte accesses. Signed-off-by: Jesse Taube Signed-off-by: Charlie Jenkins --- arch/riscv/Kconfig | 36 +++++++++ arch/riscv/include/asm/cpufeature.h | 8 +- arch/riscv/include/asm/entry-common.h | 11 --- arch/riscv/include/asm/hwprobe.h | 2 +- arch/riscv/include/asm/vector.h | 2 + arch/riscv/include/uapi/asm/hwprobe.h | 5 ++ arch/riscv/kernel/Makefile | 4 +- arch/riscv/kernel/sys_hwprobe.c | 35 ++++++++ arch/riscv/kernel/traps_misaligned.c | 125 +++++++++++++++++++++++++= +++- arch/riscv/kernel/unaligned_access_speed.c | 22 ++--- arch/riscv/kernel/vector.c | 2 +- 11 files changed, 222 insertions(+), 30 deletions(-) diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig index 2d963d4a26d7..93f9a2958de7 100644 --- a/arch/riscv/Kconfig +++ b/arch/riscv/Kconfig @@ -765,12 +765,26 @@ config THREAD_SIZE_ORDER Specify the Pages of thread stack size (from 4KB to 64KB), which also affects irq stack size, which is equal to thread stack size. =20 +config RISCV_MISALIGNED + bool + help + Embed support for detecting and emulating misaligned + scalar or vector loads and stores. + config RISCV_SCALAR_MISALIGNED bool + select RISCV_MISALIGNED select SYSCTL_ARCH_UNALIGN_ALLOW help Embed support for emulating misaligned loads and stores. =20 +config RISCV_VECTOR_MISALIGNED + bool + select RISCV_MISALIGNED + depends on RISCV_ISA_V + help + Enable detecting support for vector misaligned loads and stores. + choice prompt "Unaligned Accesses Support" default RISCV_PROBE_UNALIGNED_ACCESS @@ -822,6 +836,28 @@ config RISCV_EFFICIENT_UNALIGNED_ACCESS =20 endchoice =20 +choice + prompt "Vector unaligned Accesses Support" + depends on RISCV_ISA_V + default RISCV_PROBE_VECTOR_UNALIGNED_ACCESS + help + This determines the level of support for vector unaligned accesses. This + information is used by the kernel to perform optimizations. It is also + exposed to user space via the hwprobe syscall. The hardware will be + probed at boot by default. + +config RISCV_PROBE_VECTOR_UNALIGNED_ACCESS + bool "Probe speed of vector unaligned accesses" + select RISCV_VECTOR_MISALIGNED + depends on RISCV_ISA_V + help + During boot, the kernel will run a series of tests to determine the + speed of vector unaligned accesses if they are supported. This probing + will dynamically determine the speed of vector unaligned accesses on + the underlying system if they are supported. + +endchoice + source "arch/riscv/Kconfig.vendor" =20 endmenu # "Platform type" diff --git a/arch/riscv/include/asm/cpufeature.h b/arch/riscv/include/asm/c= pufeature.h index ccc6cf141c20..85bf1bce51e6 100644 --- a/arch/riscv/include/asm/cpufeature.h +++ b/arch/riscv/include/asm/cpufeature.h @@ -59,8 +59,8 @@ void riscv_user_isa_enable(void); #define __RISCV_ISA_EXT_SUPERSET_VALIDATE(_name, _id, _sub_exts, _validate= ) \ _RISCV_ISA_EXT_DATA(_name, _id, _sub_exts, ARRAY_SIZE(_sub_exts), _valida= te) =20 -#if defined(CONFIG_RISCV_SCALAR_MISALIGNED) bool check_unaligned_access_emulated_all_cpus(void); +#if defined(CONFIG_RISCV_SCALAR_MISALIGNED) void check_unaligned_access_emulated(struct work_struct *work __always_unu= sed); void unaligned_emulation_finish(void); bool unaligned_ctl_available(void); @@ -72,6 +72,12 @@ static inline bool unaligned_ctl_available(void) } #endif =20 +bool check_vector_unaligned_access_emulated_all_cpus(void); +#if defined(CONFIG_RISCV_VECTOR_MISALIGNED) +void check_vector_unaligned_access_emulated(struct work_struct *work __alw= ays_unused); +DECLARE_PER_CPU(long, vector_misaligned_access); +#endif + #if defined(CONFIG_RISCV_PROBE_UNALIGNED_ACCESS) DECLARE_STATIC_KEY_FALSE(fast_unaligned_access_speed_key); =20 diff --git a/arch/riscv/include/asm/entry-common.h b/arch/riscv/include/asm= /entry-common.h index 0a4e3544c877..7b32d2b08bb6 100644 --- a/arch/riscv/include/asm/entry-common.h +++ b/arch/riscv/include/asm/entry-common.h @@ -25,18 +25,7 @@ static inline void arch_exit_to_user_mode_prepare(struct= pt_regs *regs, void handle_page_fault(struct pt_regs *regs); void handle_break(struct pt_regs *regs); =20 -#ifdef CONFIG_RISCV_SCALAR_MISALIGNED int handle_misaligned_load(struct pt_regs *regs); int handle_misaligned_store(struct pt_regs *regs); -#else -static inline int handle_misaligned_load(struct pt_regs *regs) -{ - return -1; -} -static inline int handle_misaligned_store(struct pt_regs *regs) -{ - return -1; -} -#endif =20 #endif /* _ASM_RISCV_ENTRY_COMMON_H */ diff --git a/arch/riscv/include/asm/hwprobe.h b/arch/riscv/include/asm/hwpr= obe.h index ffb9484531af..1ce1df6d0ff3 100644 --- a/arch/riscv/include/asm/hwprobe.h +++ b/arch/riscv/include/asm/hwprobe.h @@ -8,7 +8,7 @@ =20 #include =20 -#define RISCV_HWPROBE_MAX_KEY 9 +#define RISCV_HWPROBE_MAX_KEY 10 =20 static inline bool riscv_hwprobe_key_is_valid(__s64 key) { diff --git a/arch/riscv/include/asm/vector.h b/arch/riscv/include/asm/vecto= r.h index be7d309cca8a..c7c023afbacd 100644 --- a/arch/riscv/include/asm/vector.h +++ b/arch/riscv/include/asm/vector.h @@ -21,6 +21,7 @@ =20 extern unsigned long riscv_v_vsize; int riscv_v_setup_vsize(void); +bool insn_is_vector(u32 insn_buf); bool riscv_v_first_use_handler(struct pt_regs *regs); void kernel_vector_begin(void); void kernel_vector_end(void); @@ -268,6 +269,7 @@ struct pt_regs; =20 static inline int riscv_v_setup_vsize(void) { return -EOPNOTSUPP; } static __always_inline bool has_vector(void) { return false; } +static __always_inline bool insn_is_vector(u32 insn_buf) { return false; } static inline bool riscv_v_first_use_handler(struct pt_regs *regs) { retur= n false; } static inline bool riscv_v_vstate_query(struct pt_regs *regs) { return fal= se; } static inline bool riscv_v_vstate_ctrl_user_allowed(void) { return false; } diff --git a/arch/riscv/include/uapi/asm/hwprobe.h b/arch/riscv/include/uap= i/asm/hwprobe.h index 1e153cda57db..34c88c15322c 100644 --- a/arch/riscv/include/uapi/asm/hwprobe.h +++ b/arch/riscv/include/uapi/asm/hwprobe.h @@ -88,6 +88,11 @@ struct riscv_hwprobe { #define RISCV_HWPROBE_MISALIGNED_SCALAR_SLOW 2 #define RISCV_HWPROBE_MISALIGNED_SCALAR_FAST 3 #define RISCV_HWPROBE_MISALIGNED_SCALAR_UNSUPPORTED 4 +#define RISCV_HWPROBE_KEY_MISALIGNED_VECTOR_PERF 10 +#define RISCV_HWPROBE_MISALIGNED_VECTOR_UNKNOWN 0 +#define RISCV_HWPROBE_MISALIGNED_VECTOR_SLOW 2 +#define RISCV_HWPROBE_MISALIGNED_VECTOR_FAST 3 +#define RISCV_HWPROBE_MISALIGNED_VECTOR_UNSUPPORTED 4 /* Increase RISCV_HWPROBE_MAX_KEY when adding items. */ =20 /* Flags */ diff --git a/arch/riscv/kernel/Makefile b/arch/riscv/kernel/Makefile index 45624c5ea86c..7f88cc4931f5 100644 --- a/arch/riscv/kernel/Makefile +++ b/arch/riscv/kernel/Makefile @@ -68,8 +68,8 @@ obj-y +=3D probes/ obj-y +=3D tests/ obj-$(CONFIG_MMU) +=3D vdso.o vdso/ =20 -obj-$(CONFIG_RISCV_SCALAR_MISALIGNED) +=3D traps_misaligned.o -obj-$(CONFIG_RISCV_SCALAR_MISALIGNED) +=3D unaligned_access_speed.o +obj-$(CONFIG_RISCV_MISALIGNED) +=3D traps_misaligned.o +obj-$(CONFIG_RISCV_MISALIGNED) +=3D unaligned_access_speed.o obj-$(CONFIG_RISCV_PROBE_UNALIGNED_ACCESS) +=3D copy-unaligned.o =20 obj-$(CONFIG_FPU) +=3D fpu.o diff --git a/arch/riscv/kernel/sys_hwprobe.c b/arch/riscv/kernel/sys_hwprob= e.c index cea0ca2bf2a2..6441baada36b 100644 --- a/arch/riscv/kernel/sys_hwprobe.c +++ b/arch/riscv/kernel/sys_hwprobe.c @@ -201,6 +201,37 @@ static u64 hwprobe_misaligned(const struct cpumask *cp= us) } #endif =20 +#ifdef CONFIG_RISCV_VECTOR_MISALIGNED +static u64 hwprobe_vec_misaligned(const struct cpumask *cpus) +{ + int cpu; + u64 perf =3D -1ULL; + + /* Return if supported or not even if speed wasn't probed */ + for_each_cpu(cpu, cpus) { + int this_perf =3D per_cpu(vector_misaligned_access, cpu); + + if (perf =3D=3D -1ULL) + perf =3D this_perf; + + if (perf !=3D this_perf) { + perf =3D RISCV_HWPROBE_MISALIGNED_VECTOR_UNKNOWN; + break; + } + } + + if (perf =3D=3D -1ULL) + return RISCV_HWPROBE_MISALIGNED_VECTOR_UNKNOWN; + + return perf; +} +#else +static u64 hwprobe_vec_misaligned(const struct cpumask *cpus) +{ + return RISCV_HWPROBE_MISALIGNED_VECTOR_UNKNOWN; +} +#endif + static void hwprobe_one_pair(struct riscv_hwprobe *pair, const struct cpumask *cpus) { @@ -229,6 +260,10 @@ static void hwprobe_one_pair(struct riscv_hwprobe *pai= r, pair->value =3D hwprobe_misaligned(cpus); break; =20 + case RISCV_HWPROBE_KEY_MISALIGNED_VECTOR_PERF: + pair->value =3D hwprobe_vec_misaligned(cpus); + break; + case RISCV_HWPROBE_KEY_ZICBOZ_BLOCK_SIZE: pair->value =3D 0; if (hwprobe_ext0_has(cpus, RISCV_HWPROBE_EXT_ZICBOZ)) diff --git a/arch/riscv/kernel/traps_misaligned.c b/arch/riscv/kernel/traps= _misaligned.c index d076dde5ad20..ef59ecfc64cb 100644 --- a/arch/riscv/kernel/traps_misaligned.c +++ b/arch/riscv/kernel/traps_misaligned.c @@ -16,6 +16,7 @@ #include #include #include +#include =20 #define INSN_MATCH_LB 0x3 #define INSN_MASK_LB 0x707f @@ -322,12 +323,37 @@ union reg_data { u64 data_u64; }; =20 -static bool unaligned_ctl __read_mostly; - /* sysctl hooks */ int unaligned_enabled __read_mostly =3D 1; /* Enabled by default */ =20 -int handle_misaligned_load(struct pt_regs *regs) +#ifdef CONFIG_RISCV_VECTOR_MISALIGNED +static int handle_vector_misaligned_load(struct pt_regs *regs) +{ + unsigned long epc =3D regs->epc; + unsigned long insn; + + if (get_insn(regs, epc, &insn)) + return -1; + + /* Only return 0 when in check_vector_unaligned_access_emulated */ + if (*this_cpu_ptr(&vector_misaligned_access) =3D=3D RISCV_HWPROBE_MISALIG= NED_VECTOR_UNKNOWN) { + *this_cpu_ptr(&vector_misaligned_access) =3D RISCV_HWPROBE_MISALIGNED_VE= CTOR_UNSUPPORTED; + regs->epc =3D epc + INSN_LEN(insn); + return 0; + } + + /* If vector instruction we don't emulate it yet */ + regs->epc =3D epc; + return -1; +} +#else +static int handle_vector_misaligned_load(struct pt_regs *regs) +{ + return -1; +} +#endif + +static int handle_scalar_misaligned_load(struct pt_regs *regs) { union reg_data val; unsigned long epc =3D regs->epc; @@ -435,7 +461,7 @@ int handle_misaligned_load(struct pt_regs *regs) return 0; } =20 -int handle_misaligned_store(struct pt_regs *regs) +static int handle_scalar_misaligned_store(struct pt_regs *regs) { union reg_data val; unsigned long epc =3D regs->epc; @@ -526,6 +552,91 @@ int handle_misaligned_store(struct pt_regs *regs) return 0; } =20 +int handle_misaligned_load(struct pt_regs *regs) +{ + unsigned long epc =3D regs->epc; + unsigned long insn; + + if (IS_ENABLED(CONFIG_RISCV_VECTOR_MISALIGNED)) { + if (get_insn(regs, epc, &insn)) + return -1; + + if (insn_is_vector(insn)) + return handle_vector_misaligned_load(regs); + } + + if (IS_ENABLED(CONFIG_RISCV_SCALAR_MISALIGNED)) + return handle_scalar_misaligned_load(regs); + + return -1; +} + +int handle_misaligned_store(struct pt_regs *regs) +{ + if (IS_ENABLED(CONFIG_RISCV_SCALAR_MISALIGNED)) + return handle_scalar_misaligned_store(regs); + + return -1; +} + +#ifdef CONFIG_RISCV_VECTOR_MISALIGNED +void check_vector_unaligned_access_emulated(struct work_struct *work __alw= ays_unused) +{ + long *mas_ptr =3D this_cpu_ptr(&vector_misaligned_access); + unsigned long tmp_var; + + *mas_ptr =3D RISCV_HWPROBE_MISALIGNED_VECTOR_UNKNOWN; + + kernel_vector_begin(); + /* + * In pre-13.0.0 versions of GCC, vector registers cannot appear in + * the clobber list. This inline asm clobbers v0, but since we do not + * currently build the kernel with V enabled, the v0 clobber arg is not + * needed (as the compiler will not emit vector code itself). If the kern= el + * is changed to build with V enabled, the clobber arg will need to be + * added here. + */ + __asm__ __volatile__ ( + ".balign 4\n\t" + ".option push\n\t" + ".option arch, +zve32x\n\t" + " vsetivli zero, 1, e16, m1, ta, ma\n\t" // Vectors of 16b + " vle16.v v0, (%[ptr])\n\t" // Load bytes + ".option pop\n\t" + : : [ptr] "r" ((u8 *)&tmp_var + 1)); + kernel_vector_end(); +} + +bool check_vector_unaligned_access_emulated_all_cpus(void) +{ + int cpu; + + if (!has_vector()) { + for_each_online_cpu(cpu) + per_cpu(vector_misaligned_access, cpu) =3D RISCV_HWPROBE_MISALIGNED_VEC= TOR_UNSUPPORTED; + return false; + } + + schedule_on_each_cpu(check_vector_unaligned_access_emulated); + + for_each_online_cpu(cpu) + if (per_cpu(vector_misaligned_access, cpu) + =3D=3D RISCV_HWPROBE_MISALIGNED_VECTOR_UNKNOWN) + return false; + + return true; +} +#else +bool check_vector_unaligned_access_emulated_all_cpus(void) +{ + return false; +} +#endif + +#ifdef CONFIG_RISCV_SCALAR_MISALIGNED + +static bool unaligned_ctl __read_mostly; + void check_unaligned_access_emulated(struct work_struct *work __always_unu= sed) { int cpu =3D smp_processor_id(); @@ -574,3 +685,9 @@ bool unaligned_ctl_available(void) { return unaligned_ctl; } +#else +bool check_unaligned_access_emulated_all_cpus(void) +{ + return false; +} +#endif diff --git a/arch/riscv/kernel/unaligned_access_speed.c b/arch/riscv/kernel= /unaligned_access_speed.c index f3508cc54f91..0b8b5e17453a 100644 --- a/arch/riscv/kernel/unaligned_access_speed.c +++ b/arch/riscv/kernel/unaligned_access_speed.c @@ -19,7 +19,8 @@ #define MISALIGNED_BUFFER_ORDER get_order(MISALIGNED_BUFFER_SIZE) #define MISALIGNED_COPY_SIZE ((MISALIGNED_BUFFER_SIZE / 2) - 0x80) =20 -DEFINE_PER_CPU(long, misaligned_access_speed); +DEFINE_PER_CPU(long, misaligned_access_speed) =3D RISCV_HWPROBE_MISALIGNED= _SCALAR_UNKNOWN; +DEFINE_PER_CPU(long, vector_misaligned_access) =3D RISCV_HWPROBE_MISALIGNE= D_VECTOR_UNSUPPORTED; =20 #ifdef CONFIG_RISCV_PROBE_UNALIGNED_ACCESS static cpumask_t fast_misaligned_access; @@ -260,23 +261,24 @@ static int check_unaligned_access_speed_all_cpus(void) kfree(bufs); return 0; } +#else /* CONFIG_RISCV_PROBE_UNALIGNED_ACCESS */ +static int check_unaligned_access_speed_all_cpus(void) +{ + return 0; +} +#endif =20 static int check_unaligned_access_all_cpus(void) { - bool all_cpus_emulated =3D check_unaligned_access_emulated_all_cpus(); + bool all_cpus_emulated; + + all_cpus_emulated =3D check_unaligned_access_emulated_all_cpus(); + check_vector_unaligned_access_emulated_all_cpus(); =20 if (!all_cpus_emulated) return check_unaligned_access_speed_all_cpus(); =20 return 0; } -#else /* CONFIG_RISCV_PROBE_UNALIGNED_ACCESS */ -static int check_unaligned_access_all_cpus(void) -{ - check_unaligned_access_emulated_all_cpus(); - - return 0; -} -#endif =20 arch_initcall(check_unaligned_access_all_cpus); diff --git a/arch/riscv/kernel/vector.c b/arch/riscv/kernel/vector.c index 682b3feee451..821818886fab 100644 --- a/arch/riscv/kernel/vector.c +++ b/arch/riscv/kernel/vector.c @@ -66,7 +66,7 @@ void __init riscv_v_setup_ctx_cache(void) #endif } =20 -static bool insn_is_vector(u32 insn_buf) +bool insn_is_vector(u32 insn_buf) { u32 opcode =3D insn_buf & __INSN_OPCODE_MASK; u32 width, csr; --=20 2.45.0