From nobody Thu Dec 18 16:19:06 2025 Received: from mail-oo1-f51.google.com (mail-oo1-f51.google.com [209.85.161.51]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id ADC1113AD38 for ; Thu, 10 Apr 2025 07:05:30 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.161.51 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1744268732; cv=none; b=KFAcaFOZ+n93OnAm//F9t3ccW8ZuHuGrZPEeyHfqVE2OUy6rh38Ineyeyw686a26RKEIBGCLrSo8JURp+Hg8ur9pOI5Hn+tHjLffcSSXlfGIwyB8Sfn1uxEYwW1+9m6IwIub18YpqOWWAH4lOtYEMirSSWTKxCyth9khzZea16I= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1744268732; c=relaxed/simple; bh=n1IKV28ehE9Wn4lJdutsk5uOI+HZ1eN54KYwMd5Wkx8=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=E9xpw7nMdCjgrM4ZHfY3gMJPB7NxvpTX5XCQMUR/uXJqnlc/IBi0syVkpM12putnd8LxVaJCkaTlbwkNvN82FESUodP3it5xHQ90y15lBXRsOP0ZIRJnB1sNDfwuy6FpH9j4qGkOvknKsQEaY+dzVP2Ph6sBOkYG6B/JpXoMN7g= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=tenstorrent.com; spf=pass smtp.mailfrom=tenstorrent.com; dkim=pass (2048-bit key) header.d=tenstorrent.com header.i=@tenstorrent.com header.b=YitfyUkF; arc=none smtp.client-ip=209.85.161.51 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=tenstorrent.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=tenstorrent.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=tenstorrent.com header.i=@tenstorrent.com header.b="YitfyUkF" Received: by mail-oo1-f51.google.com with SMTP id 006d021491bc7-6044db4b55cso212323eaf.2 for ; Thu, 10 Apr 2025 00:05:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=tenstorrent.com; s=google; t=1744268729; x=1744873529; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=qfj1lD5IvxBlvbEuJ/tOL+PNehQx5cdOKt7waVQamBc=; b=YitfyUkFZ7Ha5HFBwRtFvyRPbjn5afeOvhuEnxUlzMOGP0LRfe56Dvh1roNTr6eMVL NWiZuAVEo6EGA/P1XMfkYJbTBc8bdtDKPrpZL+HIG2KeP4Sr4bbU2TAyOajbdgvWdNdM 4qizRlE/Tnqipz9br1HNICYxN+BqQHu9a3Iw8IcBTomej0/RfP+KdMbKZSk6wndbGYtM W51Z39o9bmwOlEIOlO1ncKbY3NmMapjT7GmueYcsGROZodewIBmfI/VUEkUCA0OhqRzE zjm4bA8gKtzQtmaIiYMS55qB3JUNubjgr9hATvq0Yg9G5Lc/PNd1445UZ6OpMNzd2ed1 A8Ng== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1744268729; x=1744873529; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=qfj1lD5IvxBlvbEuJ/tOL+PNehQx5cdOKt7waVQamBc=; b=vNnJ1iJBvjruuMt/Djd23g5ABtipsPwsC/Y70yFYU5yNH08PhB3PeYmnBQ6DIqAZWG TXgEPK/Ia3tJxW3YiTOolCIBmZ9qUVJgsR0mcDtbEGT4iMze1/9Nah5ohFBrIBdtW5fH 8iXS5Ntia+B/PZCIHiClRyZG1nXPXxoFVCTzE5pZgBFQGcTrd/OYabzA40q3sOuSjtyF GcL1QpNe3FGKO0OoMNwvYqMbVPbgrMO0UQjZlAlf1mCk+MylAxJCeLbsH6OhZllVQ0Nu xbenMa6lBvUlEmMIbXRTlWfrc9SnZwzR63DqPojueSKBGikAMEN68/yTt+RTXlLKHUHf o/og== X-Forwarded-Encrypted: i=1; AJvYcCXjSamdDneyh1MH/OP45TOX55bI5ezChS/LU79nh1cnnQKAagAfDBUqrOCUMWtLUkg+RTiYGWx6wsMRxNA=@vger.kernel.org X-Gm-Message-State: AOJu0YxQAD1kcs+R2bfWgo0hHLuUecMLRnaop76LLokTrcKfI2y5H/SS 03CiJdRvvecYqjhO1K5tSvrHkOOHwaVd6epBV4C62YnjwT6RKgaUDjozXdfUiA== X-Gm-Gg: ASbGncu22nAFWUwXd3XSrr/GXBPnRmsOMBH0r9fHIpk/NJj6L0jymS018XzP4SOQQMq BiCAUtUjLrsvfHPzWiYkh4jW3HDBetj7tkSflDKNVhr2O6Ax0iKLc1Wk97hSL2+d8ivlgfkEqsI kCLUfj/kA0ss2qG1jg2hfVQGUqKWIyUHPji348lkELGyBrmNMrHjQ0OKFYLlHX29ZEd1eN6vWq8 d2s6S4hOwE/61z81hqWN9fR7YskIKigIYk5f/cstkkT/0hEKOhit1pEc4Tz5hkWs1KZeQauTIZy QSjCIbe9gRo5M2VrvYZaDeD7UqygFprbApqUhpbwK9vlaOIuwNz8s78Gv64= X-Google-Smtp-Source: AGHT+IEq3/uj00FVp1Xbt3y1cXdeUi85YNR7lcgCj8wlN2GuL4uWrvH1Xd+HddZZQveM1t1bjtbp3Q== X-Received: by 2002:a05:6808:318b:b0:3f9:3de3:c8de with SMTP id 5614622812f47-4007cc7dbacmr654153b6e.12.1744268729571; Thu, 10 Apr 2025 00:05:29 -0700 (PDT) Received: from aus-ird.tenstorrent.com ([38.104.49.66]) by smtp.gmail.com with ESMTPSA id 5614622812f47-40076282ce8sm461369b6e.9.2025.04.10.00.05.28 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 10 Apr 2025 00:05:28 -0700 (PDT) From: Cyril Bur To: palmer@dabbelt.com, aou@eecs.berkeley.edu, paul.walmsley@sifive.com, charlie@rivosinc.com, jrtc27@jrtc27.com, ben.dooks@codethink.co.uk, alex@ghiti.fr Cc: linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org, jszhang@kernel.org, syzbot+e74b94fe601ab9552d69@syzkaller.appspotmail.com Subject: [PATCH v6 1/5] riscv: save the SR_SUM status over switches Date: Thu, 10 Apr 2025 07:05:22 +0000 Message-Id: <20250410070526.3160847-2-cyrilbur@tenstorrent.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20250410070526.3160847-1-cyrilbur@tenstorrent.com> References: <20250410070526.3160847-1-cyrilbur@tenstorrent.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Ben Dooks When threads/tasks are switched we need to ensure the old execution's SR_SUM state is saved and the new thread has the old SR_SUM state restored. The issue was seen under heavy load especially with the syz-stress tool running, with crashes as follows in schedule_tail: Unable to handle kernel access to user memory without uaccess routines at virtual address 000000002749f0d0 Oops [#1] Modules linked in: CPU: 1 PID: 4875 Comm: syz-executor.0 Not tainted 5.12.0-rc2-syzkaller-00467-g0d7588ab9ef9 #0 Hardware name: riscv-virtio,qemu (DT) epc : schedule_tail+0x72/0xb2 kernel/sched/core.c:4264 ra : task_pid_vnr include/linux/sched.h:1421 [inline] ra : schedule_tail+0x70/0xb2 kernel/sched/core.c:4264 epc : ffffffe00008c8b0 ra : ffffffe00008c8ae sp : ffffffe025d17ec0 gp : ffffffe005d25378 tp : ffffffe00f0d0000 t0 : 0000000000000000 t1 : 0000000000000001 t2 : 00000000000f4240 s0 : ffffffe025d17ee0 s1 : 000000002749f0d0 a0 : 000000000000002a a1 : 0000000000000003 a2 : 1ffffffc0cfac500 a3 : ffffffe0000c80cc a4 : 5ae9db91c19bbe00 a5 : 0000000000000000 a6 : 0000000000f00000 a7 : ffffffe000082eba s2 : 0000000000040000 s3 : ffffffe00eef96c0 s4 : ffffffe022c77fe0 s5 : 0000000000004000 s6 : ffffffe067d74e00 s7 : ffffffe067d74850 s8 : ffffffe067d73e18 s9 : ffffffe067d74e00 s10: ffffffe00eef96e8 s11: 000000ae6cdf8368 t3 : 5ae9db91c19bbe00 t4 : ffffffc4043cafb2 t5 : ffffffc4043cafba t6 : 0000000000040000 status: 0000000000000120 badaddr: 000000002749f0d0 cause: 000000000000000f Call Trace: [] schedule_tail+0x72/0xb2 kernel/sched/core.c:4264 [] ret_from_exception+0x0/0x14 Dumping ftrace buffer: (ftrace buffer empty) Reported-by: syzbot+e74b94fe601ab9552d69@syzkaller.appspotmail.com Reviewed-by: Alexandre Ghiti Reviewed-by: Deepak Gupta ---[ end trace b5f8f9231dc87dda ]--- The issue comes from the put_user() in schedule_tail (kernel/sched/core.c) doing the following: asmlinkage __visible void schedule_tail(struct task_struct *prev) { ... if (current->set_child_tid) put_user(task_pid_vnr(current), current->set_child_tid); ... } the put_user() macro causes the code sequence to come out as follows: 1: __enable_user_access() 2: reg =3D task_pid_vnr(current); 3: *current->set_child_tid =3D reg; 4: __disable_user_access() The problem is that we may have a sleeping function as argument which could clear SR_SUM causing the panic above. This was fixed by evaluating the argument of the put_user() macro outside the user-enabled section in commit 285a76bb2cf5 ("riscv: evaluate put_user() arg before enabling user access")" In order for riscv to take advantage of unsafe_get/put_XXX() macros and to avoid the same issue we had with put_user() and sleeping functions we must ensure code flow can go through switch_to() from within a region of code with SR_SUM enabled and come back with SR_SUM still enabled. This patch addresses the problem allowing future work to enable full use of unsafe_get/put_XXX() macros without needing to take a CSR bit flip cost on every access. Make switch_to() save and restore SR_SUM. Reported-by: syzbot+e74b94fe601ab9552d69@syzkaller.appspotmail.com Signed-off-by: Ben Dooks Signed-off-by: Cyril Bur --- arch/riscv/include/asm/processor.h | 1 + arch/riscv/kernel/asm-offsets.c | 5 +++++ arch/riscv/kernel/entry.S | 8 ++++++++ 3 files changed, 14 insertions(+) diff --git a/arch/riscv/include/asm/processor.h b/arch/riscv/include/asm/pr= ocessor.h index 5f56eb9d114a..58fd11c89fe9 100644 --- a/arch/riscv/include/asm/processor.h +++ b/arch/riscv/include/asm/processor.h @@ -103,6 +103,7 @@ struct thread_struct { struct __riscv_d_ext_state fstate; unsigned long bad_cause; unsigned long envcfg; + unsigned long status; u32 riscv_v_flags; u32 vstate_ctrl; struct __riscv_v_ext_state vstate; diff --git a/arch/riscv/kernel/asm-offsets.c b/arch/riscv/kernel/asm-offset= s.c index 16490755304e..969c65b1fe41 100644 --- a/arch/riscv/kernel/asm-offsets.c +++ b/arch/riscv/kernel/asm-offsets.c @@ -34,6 +34,7 @@ void asm_offsets(void) OFFSET(TASK_THREAD_S9, task_struct, thread.s[9]); OFFSET(TASK_THREAD_S10, task_struct, thread.s[10]); OFFSET(TASK_THREAD_S11, task_struct, thread.s[11]); + OFFSET(TASK_THREAD_STATUS, task_struct, thread.status); =20 OFFSET(TASK_TI_CPU, task_struct, thread_info.cpu); OFFSET(TASK_TI_PREEMPT_COUNT, task_struct, thread_info.preempt_count); @@ -346,6 +347,10 @@ void asm_offsets(void) offsetof(struct task_struct, thread.s[11]) - offsetof(struct task_struct, thread.ra) ); + DEFINE(TASK_THREAD_STATUS_RA, + offsetof(struct task_struct, thread.status) + - offsetof(struct task_struct, thread.ra) + ); =20 DEFINE(TASK_THREAD_F0_F0, offsetof(struct task_struct, thread.fstate.f[0]) diff --git a/arch/riscv/kernel/entry.S b/arch/riscv/kernel/entry.S index 33a5a9f2a0d4..00bd0de9faa2 100644 --- a/arch/riscv/kernel/entry.S +++ b/arch/riscv/kernel/entry.S @@ -397,9 +397,17 @@ SYM_FUNC_START(__switch_to) REG_S s9, TASK_THREAD_S9_RA(a3) REG_S s10, TASK_THREAD_S10_RA(a3) REG_S s11, TASK_THREAD_S11_RA(a3) + + /* save the user space access flag */ + li s0, SR_SUM + csrr s1, CSR_STATUS + REG_S s1, TASK_THREAD_STATUS_RA(a3) + /* Save the kernel shadow call stack pointer */ scs_save_current /* Restore context from next->thread */ + REG_L s0, TASK_THREAD_STATUS_RA(a4) + csrs CSR_STATUS, s0 REG_L ra, TASK_THREAD_RA_RA(a4) REG_L sp, TASK_THREAD_SP_RA(a4) REG_L s0, TASK_THREAD_S0_RA(a4) --=20 2.34.1 From nobody Thu Dec 18 16:19:07 2025 Received: from mail-oi1-f177.google.com (mail-oi1-f177.google.com [209.85.167.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3A0AA2045B0 for ; Thu, 10 Apr 2025 07:05:31 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.167.177 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1744268734; cv=none; b=iDlro8wLyJoc9WzA5WVWlKBFn8M1XUNahlh/cBlwYlF1fGzMKlo2UbQnSKgyDYpqdPR7X78SBKJRsp9QH4Q0FCTbe0fvA/WlSQ5YaCfAF4IL929FlYXvxa8vaYUzVqel7dobMZt3v6asMsoVZhX3TT52MfUdxM7lDx4KC+bpbAI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1744268734; c=relaxed/simple; bh=p7gn8HCoRfmei461kdKTMrhmaoZ+R8QG+5SxtGUWr5I=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=dcmiOGoJtUrhckOjmnPUUBYLVGFrRqQKO33/xHKwu68VfypcNrdL5Ewifu2pAEfwQRrvZQZkPxUt/JRQ4tyrCWoS99GLlh2hSQzU1XveiG+jbdtBE9oLQqzrQv3uUJH91o9U83jUFJz7cNcpKX7t5ZsO8F8K+/liHaheklXJfYg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=tenstorrent.com; spf=pass smtp.mailfrom=tenstorrent.com; dkim=pass (2048-bit key) header.d=tenstorrent.com header.i=@tenstorrent.com header.b=G3BKhYOz; arc=none smtp.client-ip=209.85.167.177 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=tenstorrent.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=tenstorrent.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=tenstorrent.com header.i=@tenstorrent.com header.b="G3BKhYOz" Received: by mail-oi1-f177.google.com with SMTP id 5614622812f47-3f6a92f2328so292896b6e.0 for ; Thu, 10 Apr 2025 00:05:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=tenstorrent.com; s=google; t=1744268731; x=1744873531; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Vvw7nYWiLBIqfTSpurYIsb0A2VRmihRadD2HqJzvOMM=; b=G3BKhYOzZP4xxuWMSgqCgVjAEzHrd5QEyCSOvh8myjhiGNoF6xkDuZtW1aJOt5sppM 9Q67xiSs7O2+McVoyKWUeTbdPgwoojZYVqkluaENJQsw9Yor0ORZ1y6MlymFJ9C2z05p DAVCrTgO5E1nb66UyWSUw1scLkdl8QKD58UiDjd+2N4bGD+3JS4UXwd7cayFfL6kYoRy N9XLJd5hSeRQG5F/SvSQ5RHVSAbK1Ap7tRnVTAV1UslG1j/InDhTZOhpoDDSFhOyW8xG m8Q28j/JscHUp+PMIgUFqSyvJzZLwnscgLHmIHvjwdkG5eE6MXoa95dF9eV4Fvvxyg+p Z7Vg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1744268731; x=1744873531; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Vvw7nYWiLBIqfTSpurYIsb0A2VRmihRadD2HqJzvOMM=; b=I5VAJJTNxG8iQGlRafUHMEOmSB6CbPa/CfrFV0nFfqFZFx+oma5Vveru88/4BC1WgY fZa53SfRDoXgcRkRulLXJV9rA+OQrBXT/qOcODUss+bccAbu48A3/NMuavTu6F8Kh7el OcOrB7M+vgbdQAfugWArsIXBoGZ1y+oBIZmp3aTH4IPczXz7n5z1dOx3p1YOvv/kNiO0 COP8mdDjgVBNxREoC2XqQ0gXeye/+H3rgwG4bZ/Jl4OccmfWZ1x+7bKKOqSMlP4RzGOr o+wVJKphrPn2r8By1k9aAf0iwQfj/JmCqR5oNjp5HjEjcAVTuea8G1kqAkMIztIfG+ZZ t61w== X-Forwarded-Encrypted: i=1; AJvYcCXUjbtnRL6dGA10kPGfUSHGnBz+PmUAh01eFhelUHaFKycoiuV9yL1YqofopzW56B2j/DBGDUcroWr7hoE=@vger.kernel.org X-Gm-Message-State: AOJu0YxpNQGFWqHePXbekTKQa/ZHbJGxqYgFdDa0creA9tBfClncI8t7 fgM7A+nxZQzuk+x/8MddGUrt8M3t6Phn0Z3cykP2ev0PLqumD9UhhqZVSVUHTA== X-Gm-Gg: ASbGncsDB57xx/dW1NqNUroS9ClzWMAzw/J2NHDzgc1XbiM2COi+5FRIZYjxo6phh2r DtaQxXBKaBNRt0hl683fRPzWEGTYpfcwwcgB2b2W/uLMxrCjG9VQbfXBTPGWueNaV5fpGna390B wtVM1W/d9zILvU9avtTLapmOm0H1Fz49ifvd3IM95yrR7ga/eVmEYaoAIXktmGXN+werukvzxRt cNhVJKcgGj7aqZn0FMnRHAwf2EorCEShY4w9pzUdIXhAOsUvM0IiLsTNCfArkJoPnB5A+ODNWz2 MT7G/Gs4U75lhPncw5gPUbyjoZTthnj/2+dYYj8bT39v7d7SAFOr0aTUbUKYot8ng33xAw== X-Google-Smtp-Source: AGHT+IF/4Hhe3gfMVMHnBMPKqqrqs5V8qr1hvivIjhC40n+SJ1Uz45EXIUL/JoOL6so0taxy89Y6cQ== X-Received: by 2002:a05:6808:11c1:b0:3f9:aeb6:6e92 with SMTP id 5614622812f47-4007bfe2744mr915328b6e.9.1744268730964; Thu, 10 Apr 2025 00:05:30 -0700 (PDT) Received: from aus-ird.tenstorrent.com ([38.104.49.66]) by smtp.gmail.com with ESMTPSA id 5614622812f47-40076282ce8sm461369b6e.9.2025.04.10.00.05.29 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 10 Apr 2025 00:05:30 -0700 (PDT) From: Cyril Bur To: palmer@dabbelt.com, aou@eecs.berkeley.edu, paul.walmsley@sifive.com, charlie@rivosinc.com, jrtc27@jrtc27.com, ben.dooks@codethink.co.uk, alex@ghiti.fr Cc: linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org, jszhang@kernel.org Subject: [PATCH v6 2/5] riscv: implement user_access_begin() and families Date: Thu, 10 Apr 2025 07:05:23 +0000 Message-Id: <20250410070526.3160847-3-cyrilbur@tenstorrent.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20250410070526.3160847-1-cyrilbur@tenstorrent.com> References: <20250410070526.3160847-1-cyrilbur@tenstorrent.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Jisheng Zhang Currently, when a function like strncpy_from_user() is called, the userspace access protection is disabled and enabled for every word read. By implementing user_access_begin() and families, the protection is disabled at the beginning of the copy and enabled at the end. The __inttype macro is borrowed from x86 implementation. Signed-off-by: Jisheng Zhang Signed-off-by: Cyril Bur Reviewed-by: Alexandre Ghiti --- arch/riscv/include/asm/uaccess.h | 76 ++++++++++++++++++++++++++++++++ 1 file changed, 76 insertions(+) diff --git a/arch/riscv/include/asm/uaccess.h b/arch/riscv/include/asm/uacc= ess.h index fee56b0c8058..c9a461467bf4 100644 --- a/arch/riscv/include/asm/uaccess.h +++ b/arch/riscv/include/asm/uaccess.h @@ -61,6 +61,19 @@ static inline unsigned long __untagged_addr_remote(struc= t mm_struct *mm, unsigne #define __disable_user_access() \ __asm__ __volatile__ ("csrc sstatus, %0" : : "r" (SR_SUM) : "memory") =20 +/* + * This is the smallest unsigned integer type that can fit a value + * (up to 'long long') + */ +#define __inttype(x) __typeof__( \ + __typefits(x, char, \ + __typefits(x, short, \ + __typefits(x, int, \ + __typefits(x, long, 0ULL))))) + +#define __typefits(x, type, not) \ + __builtin_choose_expr(sizeof(x) <=3D sizeof(type), (unsigned type)0, not) + /* * The exception table consists of pairs of addresses: the first is the * address of an instruction that is allowed to fault, and the second is @@ -368,6 +381,69 @@ do { \ goto err_label; \ } while (0) =20 +static __must_check __always_inline bool user_access_begin(const void __us= er *ptr, size_t len) +{ + if (unlikely(!access_ok(ptr, len))) + return 0; + __enable_user_access(); + return 1; +} +#define user_access_begin user_access_begin +#define user_access_end __disable_user_access + +static inline unsigned long user_access_save(void) { return 0UL; } +static inline void user_access_restore(unsigned long enabled) { } + +/* + * We want the unsafe accessors to always be inlined and use + * the error labels - thus the macro games. + */ +#define unsafe_put_user(x, ptr, label) do { \ + long __err =3D 0; \ + __put_user_nocheck(x, (ptr), __err); \ + if (__err) \ + goto label; \ +} while (0) + +#define unsafe_get_user(x, ptr, label) do { \ + long __err =3D 0; \ + __inttype(*(ptr)) __gu_val; \ + __get_user_nocheck(__gu_val, (ptr), __err); \ + (x) =3D (__force __typeof__(*(ptr)))__gu_val; \ + if (__err) \ + goto label; \ +} while (0) + +#define unsafe_copy_loop(dst, src, len, type, op, label) \ + while (len >=3D sizeof(type)) { \ + op(*(type *)(src), (type __user *)(dst), label); \ + dst +=3D sizeof(type); \ + src +=3D sizeof(type); \ + len -=3D sizeof(type); \ + } + +#define unsafe_copy_to_user(_dst, _src, _len, label) \ +do { \ + char __user *__ucu_dst =3D (_dst); \ + const char *__ucu_src =3D (_src); \ + size_t __ucu_len =3D (_len); \ + unsafe_copy_loop(__ucu_dst, __ucu_src, __ucu_len, u64, unsafe_put_user, l= abel); \ + unsafe_copy_loop(__ucu_dst, __ucu_src, __ucu_len, u32, unsafe_put_user, l= abel); \ + unsafe_copy_loop(__ucu_dst, __ucu_src, __ucu_len, u16, unsafe_put_user, l= abel); \ + unsafe_copy_loop(__ucu_dst, __ucu_src, __ucu_len, u8, unsafe_put_user, la= bel); \ +} while (0) + +#define unsafe_copy_from_user(_dst, _src, _len, label) \ +do { \ + char *__ucu_dst =3D (_dst); \ + const char __user *__ucu_src =3D (_src); \ + size_t __ucu_len =3D (_len); \ + unsafe_copy_loop(__ucu_src, __ucu_dst, __ucu_len, u64, unsafe_get_user, l= abel); \ + unsafe_copy_loop(__ucu_src, __ucu_dst, __ucu_len, u32, unsafe_get_user, l= abel); \ + unsafe_copy_loop(__ucu_src, __ucu_dst, __ucu_len, u16, unsafe_get_user, l= abel); \ + unsafe_copy_loop(__ucu_src, __ucu_dst, __ucu_len, u8, unsafe_get_user, la= bel); \ +} while (0) + #else /* CONFIG_MMU */ #include #endif /* CONFIG_MMU */ --=20 2.34.1 From nobody Thu Dec 18 16:19:07 2025 Received: from mail-oi1-f176.google.com (mail-oi1-f176.google.com [209.85.167.176]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 53D2F204C39 for ; Thu, 10 Apr 2025 07:05:33 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.167.176 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1744268735; cv=none; b=hAoEVhVvjgrrbhDJT0LXQB33r0PDroAZj/9G+0Fo8ZWMzjUPqMjbFK5YEw7jhPRBPZCnpFyLP0I738PRXQs5P9hFjqnnhZcq9l5tzQb1eNF3/raebDJS9tdhgVjr7JNIuVdCcdnR3UR7TZNE14J54/Mx5820uwYC5sSepi/09jU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1744268735; c=relaxed/simple; bh=HKMnBVdHsElfI+VU/kTaLpcWor2d4s9BxLReognxMNg=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=jAyMmILhG4mkkgHPDWXE6YNmAWuSELykPxOuuLLIwF0eEUO4fQ6BBhlmnKRJxggmTR901gzNs2og8v0YMNy7vqlUCAzdSpAqiIpdYPoKGiBRWrBB40QCFFLoPflUX+5KvydKy6gqUBk2CTfXc9TO/iabRX5eAO/2muzP/Pnc550= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=tenstorrent.com; spf=pass smtp.mailfrom=tenstorrent.com; dkim=pass (2048-bit key) header.d=tenstorrent.com header.i=@tenstorrent.com header.b=HaELGWEq; arc=none smtp.client-ip=209.85.167.176 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=tenstorrent.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=tenstorrent.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=tenstorrent.com header.i=@tenstorrent.com header.b="HaELGWEq" Received: by mail-oi1-f176.google.com with SMTP id 5614622812f47-3f6a92f2328so292914b6e.0 for ; Thu, 10 Apr 2025 00:05:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=tenstorrent.com; s=google; t=1744268732; x=1744873532; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=mh8UBXGzUbnvdsif2sw+KTYhRC1BSjBoml8SqEobyIk=; b=HaELGWEqy2aRo7uxRbyFlOX3yMS8h5N7OZRrCaLqZwIyTd6yKPxz+p9OPoqt1JEWNf F9zSBPFF0pSTioJ275CYOWte3oKZ+LhyvV4XAhwgIYUByMUZSWnokzqqtLhBW3BxSmxM TPS+T9TP7eiCPRtdulqNtRgCc1p6bwgdHSxDybh/nr5UeX70wR9pAD/KIHdhUp7a6gmq F8ZSKIhVKoBOepeya+KhFB18n+rMs1z+kGD/QQ9/70nY869CDzu28BQHjkHc1AujJ6k6 ysmAQljMZQ2lCt3FfFkcQycREVe9esLuUaLhIIcqM9+ftvhyXQHiChQvYz9SslzoXnNe vS9A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1744268732; x=1744873532; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=mh8UBXGzUbnvdsif2sw+KTYhRC1BSjBoml8SqEobyIk=; b=b0HmITRHUdKKvaccpjaoa+FGkQrGiUEImubtib9U4flBbDmYrNwEOEqun7xEPOvwtm lN/0lurYC/yPIAF8vqriQEzn0QgxPFDfNhY9ZBby4GmCMxy3JyqkBej42RSqZRkt5qbG bziAFVUMI03rxWNbdaH13P4aij9sSA9VxPNH6zOqQ1IzK5rnVGxDTLgfcXv8ouS6RheX pc/+jjBiOiFYxJpBmF9Fzc6ILZRx0L1btQBRJLjk/meqpMcCa3SEVcdMcAggncnpJhKX HrrXGD7kYNLIrguEY69rdXQemr1ww5iMiEfFgW1b92j55rkAOI8q509VRJng4HJ1HbvC 6qFQ== X-Forwarded-Encrypted: i=1; AJvYcCX0ukKzXq8m+md6MvlVMMv/n71+MnSD7RbSzALN02TO9OOtmoE1TG3wICeVWnmlklovHatNd+hwVtkKlQ4=@vger.kernel.org X-Gm-Message-State: AOJu0YxY+rEv5rBus/+bSoiheCTekYRAftle9zCDEkNTCs28sv8l9Fo7 G+4kgKNM/6K+tnypFv3Pq91jXs/Sizampuxo3CqJwRmbQJ5Khv27dDd17ucVpA== X-Gm-Gg: ASbGncsIIaQSpFCvsKgYLQQJbFZ9bjLZFnFK4TdlDwRnplFMp9Dvknjhn7Co3i2E8rp bNzIi1SqeKibv0tMd4phQklYJCEky1C+hgcp46kqyyccTt7xo6ZlMxtl3HyTvPBKajQ2N02PoAY Huvx54t4nOZlHEILN4zj4iQ/9ojPA2qZq+lDs0t4/Wq/I99E7tUtGl0vQ+vWatcxk/N6hMAUVYW ccYDD8ecUZpfyWH4MHv6wXt9K1JB/L/qeP0hPd6rooNNaOQQswS6JoBURgzb7YCiAxgxO3TKRAz Ct13pnws9WNwVNR9ek2s/d01Y7fN5mubr8iRGYCL2TuSoXrR7NtabZJJv7F6O0LBAm5GWQ== X-Google-Smtp-Source: AGHT+IE/XTUu6u5msxN+Qw7e3Ta4Mxkm8pI5BxCJg/oExJOAteicyPdgbu1SRM4uzbVEJeCFnx0ZuA== X-Received: by 2002:a05:6808:2f13:b0:3f6:7677:5bef with SMTP id 5614622812f47-4007bfa52femr977035b6e.2.1744268732358; Thu, 10 Apr 2025 00:05:32 -0700 (PDT) Received: from aus-ird.tenstorrent.com ([38.104.49.66]) by smtp.gmail.com with ESMTPSA id 5614622812f47-40076282ce8sm461369b6e.9.2025.04.10.00.05.31 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 10 Apr 2025 00:05:31 -0700 (PDT) From: Cyril Bur To: palmer@dabbelt.com, aou@eecs.berkeley.edu, paul.walmsley@sifive.com, charlie@rivosinc.com, jrtc27@jrtc27.com, ben.dooks@codethink.co.uk, alex@ghiti.fr Cc: linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org, jszhang@kernel.org Subject: [PATCH v6 3/5] riscv: uaccess: use input constraints for ptr of __put_user() Date: Thu, 10 Apr 2025 07:05:24 +0000 Message-Id: <20250410070526.3160847-4-cyrilbur@tenstorrent.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20250410070526.3160847-1-cyrilbur@tenstorrent.com> References: <20250410070526.3160847-1-cyrilbur@tenstorrent.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Jisheng Zhang Putting ptr in the inputs as opposed to output may seem incorrect but this is done for a few reasons: - Not having it in the output permits the use of asm goto in a subsequent patch. There are bugs in gcc [1] which would otherwise prevent it. - Since the output memory is userspace there isn't any real benefit from telling the compiler about the memory clobber. - x86, arm and powerpc all use this technique. Link: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=3D113921 # 1 Signed-off-by: Jisheng Zhang [Cyril Bur: Rewritten commit message] Signed-off-by: Cyril Bur Reviewed-by: Alexandre Ghiti --- arch/riscv/include/asm/uaccess.h | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/arch/riscv/include/asm/uaccess.h b/arch/riscv/include/asm/uacc= ess.h index c9a461467bf4..da36057847f0 100644 --- a/arch/riscv/include/asm/uaccess.h +++ b/arch/riscv/include/asm/uaccess.h @@ -219,11 +219,11 @@ do { \ __typeof__(*(ptr)) __x =3D x; \ __asm__ __volatile__ ( \ "1:\n" \ - " " insn " %z2, %1\n" \ + " " insn " %z1, %2\n" \ "2:\n" \ _ASM_EXTABLE_UACCESS_ERR(1b, 2b, %0) \ - : "+r" (err), "=3Dm" (*(ptr)) \ - : "rJ" (__x)); \ + : "+r" (err) \ + : "rJ" (__x), "m"(*(ptr))); \ } while (0) =20 #ifdef CONFIG_64BIT @@ -236,16 +236,16 @@ do { \ u64 __x =3D (__typeof__((x)-(x)))(x); \ __asm__ __volatile__ ( \ "1:\n" \ - " sw %z3, %1\n" \ + " sw %z1, %3\n" \ "2:\n" \ - " sw %z4, %2\n" \ + " sw %z2, %4\n" \ "3:\n" \ _ASM_EXTABLE_UACCESS_ERR(1b, 3b, %0) \ _ASM_EXTABLE_UACCESS_ERR(2b, 3b, %0) \ - : "+r" (err), \ - "=3Dm" (__ptr[__LSW]), \ - "=3Dm" (__ptr[__MSW]) \ - : "rJ" (__x), "rJ" (__x >> 32)); \ + : "+r" (err) \ + : "rJ" (__x), "rJ" (__x >> 32), \ + "m" (__ptr[__LSW]), \ + "m" (__ptr[__MSW])); \ } while (0) #endif /* CONFIG_64BIT */ =20 --=20 2.34.1 From nobody Thu Dec 18 16:19:07 2025 Received: from mail-ot1-f47.google.com (mail-ot1-f47.google.com [209.85.210.47]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B50BB205E2D for ; Thu, 10 Apr 2025 07:05:34 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.47 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1744268736; cv=none; b=VhFyf4pIZ8VFG8uzFwy9ZnAZez+8Zc8W1PIzyGhD13G15+oVngoQ0MbFC3jXDWxCUkYDZW+vn4T7Iah6jnS0349GwRIRzBe6/egNMuGa0DFQXhTrQe48LI/z2u5Kt5bxj9pl3U4pUXAgv1tK1l8nbUTfM2+LRWa/fVToq5J2gpQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1744268736; c=relaxed/simple; bh=6Rd4HzTmjSi8Wj2+T4c6yX3o6/8i9wfFJ0rHfw2fL5Y=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=OR99lT3Tfc8/iJp7pXeYBwjw4jXvZun8lTN5JYQ+zhoZj7tIQ4DhpEHbBZvx+8DCFLlDr5N3vXkwDJ45lN5zG61Y9I193F/0CraMIS2li3VaOLHW19/uSXKI0S6vjU40BISp0chqf/LZQFpG6T+i0rEG8o3alG2sy93o/c0uN7M= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=tenstorrent.com; spf=pass smtp.mailfrom=tenstorrent.com; dkim=pass (2048-bit key) header.d=tenstorrent.com header.i=@tenstorrent.com header.b=LqykwAYT; arc=none smtp.client-ip=209.85.210.47 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=tenstorrent.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=tenstorrent.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=tenstorrent.com header.i=@tenstorrent.com header.b="LqykwAYT" Received: by mail-ot1-f47.google.com with SMTP id 46e09a7af769-72c1818c394so302902a34.2 for ; Thu, 10 Apr 2025 00:05:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=tenstorrent.com; s=google; t=1744268734; x=1744873534; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=DpC/LMch4vBncOIb4H+uNBUj+A3evJlilB0f5X4S7k8=; b=LqykwAYT5O0mCChVjXZhRiaVVDxoUCScgNVWukCLAaMyh/dvFdomPAp3w1DP2mjYmg mkZ0lXFspDkgoh9oSqC7x8nZlZczPudWloTiD/2/ZZSytlzKiGRy3hsABQEvnCxvgymU 0/ypkWuKuUO5RbuJOFpFDYJliYyCw6QV1l6z2HDHGjBm8FumbWlDCQXpQwPNtTZ8s1eq D46157ckE2CE9xtQlnwpvAOkyj8a6atC9dWjyc5yuS3ZxYwk8B1kQLZQWcCf8fZ2RaJT Q+DHEUx+AcgsvwwD06rBU+SKAWIhuN1iCcLfSI2RRh3kz0rgsBEPhoQPt+iMXhsJw3xi GR1A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1744268734; x=1744873534; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=DpC/LMch4vBncOIb4H+uNBUj+A3evJlilB0f5X4S7k8=; b=Omk6ufzF9wsI3YvR95a8Bdc7yvZxmUIeyShKDcENQpg7aYhzLLHJTTAQtVYwV6zFwb IgVHtSyzEewPW8z24vplOZ9DN7M+UigCt0cqElZAnS43NhF4tQ2sKb0b6qWwQ1rNUHc/ l8kw4pn1PnhgR7JgIeEP9I0F5bW+xWiXWRkz0jWkvH7ohyxGEKfRt1vP0JtAWhGNr/vt C/fXzPr3FNvHCLm6vnh0w29H3T7OAjdSzs7Dkw5FC1Y691KK0S8CcU0AnFshVGgjUuTP F4OCG2ZjCHBk/ElPINQWly/nbSa/B1Obih19RlMhV5yuhk0i9X5BqESvPMA1UZMIpoAf ALmg== X-Forwarded-Encrypted: i=1; AJvYcCX2+gF5zR4lUrQ4GeKKBLOgS0MouQAXu45L6SzlSdlfbQDzNDfxgZMWXCB9JABZCkhrDyuugKFsYAcb5qc=@vger.kernel.org X-Gm-Message-State: AOJu0YweOv5E3LfqbEfaw7EujkNrlxS+UShYfmxG8fb2L2IDL6xKGVJ8 jO1Wd/jquL8sEdOYoQCRLRkuAyux1+znqbhlDoVKhqnoTN4q6rpIbleMsmKvoA== X-Gm-Gg: ASbGncuBv67ROFpHQ9MnmQ0SHAjQczIeijA8C4D6xTwI9CjWNXN0H5HLve0OMflXcjK 9RhQW3zMUyFzNlPW3fmc+bJp+gKnCKPsiCK2DlXQYjoCtQVHDpJxVBvWz+9vCKdKe38LIXVLu3s HWvo2/U++b3nUxY64oB7x67aPvECow0XvyXrD5R3qp4gdIhglllgfu9BmVQGDW2dkZekFRCfDXA Twd9tX3QB1ywkf0xWFXhCtdx30LIboybch86Gm7tmR1RxfIBaJ9RiCr7fnCdgfuMQR+mT5QHG5Y j7uLPl3MRyAo+7CCQJx9rLRo7VQTZsEmLondzO6bAPMaKjI5OZ3sVLpUc5Y= X-Google-Smtp-Source: AGHT+IE+XtNq4lkOGfL5MMqTmIpDIasL7LqvEU62Fpm528DyNXHH0ktFXep74oD9/woVq59pGry6bA== X-Received: by 2002:a05:6808:14c2:b0:3fa:7328:b9a8 with SMTP id 5614622812f47-4007bcda67dmr907210b6e.18.1744268733620; Thu, 10 Apr 2025 00:05:33 -0700 (PDT) Received: from aus-ird.tenstorrent.com ([38.104.49.66]) by smtp.gmail.com with ESMTPSA id 5614622812f47-40076282ce8sm461369b6e.9.2025.04.10.00.05.32 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 10 Apr 2025 00:05:33 -0700 (PDT) From: Cyril Bur To: palmer@dabbelt.com, aou@eecs.berkeley.edu, paul.walmsley@sifive.com, charlie@rivosinc.com, jrtc27@jrtc27.com, ben.dooks@codethink.co.uk, alex@ghiti.fr Cc: linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org, jszhang@kernel.org Subject: [PATCH v6 4/5] riscv: uaccess: use 'asm goto' for put_user() Date: Thu, 10 Apr 2025 07:05:25 +0000 Message-Id: <20250410070526.3160847-5-cyrilbur@tenstorrent.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20250410070526.3160847-1-cyrilbur@tenstorrent.com> References: <20250410070526.3160847-1-cyrilbur@tenstorrent.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Jisheng Zhang With 'asm goto' we don't need to test the error etc, the exception just jumps to the error handling directly. Because there are no output clobbers which could trigger gcc bugs [1] the use of asm_goto_output() macro is not necessary here. Not using asm_goto_output() is desirable as the generated output asm will be cleaner. Use of the volatile keyword is redundant as per gcc 14.2.0 manual section 6.48.2.7 Goto Labels: > Also note that an asm goto statement is always implicitly considered volatile. Link: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=3D113921 # 1 Signed-off-by: Jisheng Zhang [Cyril Bur: Rewritten commit message] Signed-off-by: Cyril Bur Reviewed-by: Alexandre Ghiti --- arch/riscv/include/asm/uaccess.h | 71 +++++++++++++++----------------- 1 file changed, 33 insertions(+), 38 deletions(-) diff --git a/arch/riscv/include/asm/uaccess.h b/arch/riscv/include/asm/uacc= ess.h index da36057847f0..719c9179a751 100644 --- a/arch/riscv/include/asm/uaccess.h +++ b/arch/riscv/include/asm/uaccess.h @@ -214,61 +214,66 @@ do { \ ((x) =3D (__force __typeof__(x))0, -EFAULT); \ }) =20 -#define __put_user_asm(insn, x, ptr, err) \ +#define __put_user_asm(insn, x, ptr, label) \ do { \ __typeof__(*(ptr)) __x =3D x; \ - __asm__ __volatile__ ( \ + asm goto( \ "1:\n" \ - " " insn " %z1, %2\n" \ - "2:\n" \ - _ASM_EXTABLE_UACCESS_ERR(1b, 2b, %0) \ - : "+r" (err) \ - : "rJ" (__x), "m"(*(ptr))); \ + " " insn " %z0, %1\n" \ + _ASM_EXTABLE(1b, %l2) \ + : : "rJ" (__x), "m"(*(ptr)) : : label); \ } while (0) =20 #ifdef CONFIG_64BIT -#define __put_user_8(x, ptr, err) \ - __put_user_asm("sd", x, ptr, err) +#define __put_user_8(x, ptr, label) \ + __put_user_asm("sd", x, ptr, label) #else /* !CONFIG_64BIT */ -#define __put_user_8(x, ptr, err) \ +#define __put_user_8(x, ptr, label) \ do { \ u32 __user *__ptr =3D (u32 __user *)(ptr); \ u64 __x =3D (__typeof__((x)-(x)))(x); \ - __asm__ __volatile__ ( \ + asm goto( \ "1:\n" \ - " sw %z1, %3\n" \ + " sw %z0, %2\n" \ "2:\n" \ - " sw %z2, %4\n" \ - "3:\n" \ - _ASM_EXTABLE_UACCESS_ERR(1b, 3b, %0) \ - _ASM_EXTABLE_UACCESS_ERR(2b, 3b, %0) \ - : "+r" (err) \ - : "rJ" (__x), "rJ" (__x >> 32), \ + " sw %z1, %3\n" \ + _ASM_EXTABLE(1b, %l4) \ + _ASM_EXTABLE(2b, %l4) \ + : : "rJ" (__x), "rJ" (__x >> 32), \ "m" (__ptr[__LSW]), \ - "m" (__ptr[__MSW])); \ + "m" (__ptr[__MSW]) : : label); \ } while (0) #endif /* CONFIG_64BIT */ =20 -#define __put_user_nocheck(x, __gu_ptr, __pu_err) \ +#define __put_user_nocheck(x, __gu_ptr, label) \ do { \ switch (sizeof(*__gu_ptr)) { \ case 1: \ - __put_user_asm("sb", (x), __gu_ptr, __pu_err); \ + __put_user_asm("sb", (x), __gu_ptr, label); \ break; \ case 2: \ - __put_user_asm("sh", (x), __gu_ptr, __pu_err); \ + __put_user_asm("sh", (x), __gu_ptr, label); \ break; \ case 4: \ - __put_user_asm("sw", (x), __gu_ptr, __pu_err); \ + __put_user_asm("sw", (x), __gu_ptr, label); \ break; \ case 8: \ - __put_user_8((x), __gu_ptr, __pu_err); \ + __put_user_8((x), __gu_ptr, label); \ break; \ default: \ BUILD_BUG(); \ } \ } while (0) =20 +#define __put_user_error(x, ptr, err) \ +do { \ + __label__ err_label; \ + __put_user_nocheck(x, ptr, err_label); \ + break; \ +err_label: \ + (err) =3D -EFAULT; \ +} while (0) + /** * __put_user: - Write a simple value into user space, with less checking. * @x: Value to copy to user space. @@ -299,7 +304,7 @@ do { \ __chk_user_ptr(__gu_ptr); \ \ __enable_user_access(); \ - __put_user_nocheck(__val, __gu_ptr, __pu_err); \ + __put_user_error(__val, __gu_ptr, __pu_err); \ __disable_user_access(); \ \ __pu_err; \ @@ -373,13 +378,7 @@ do { \ } while (0) =20 #define __put_kernel_nofault(dst, src, type, err_label) \ -do { \ - long __kr_err =3D 0; \ - \ - __put_user_nocheck(*((type *)(src)), (type *)(dst), __kr_err); \ - if (unlikely(__kr_err)) \ - goto err_label; \ -} while (0) + __put_user_nocheck(*((type *)(src)), (type *)(dst), err_label) =20 static __must_check __always_inline bool user_access_begin(const void __us= er *ptr, size_t len) { @@ -398,12 +397,8 @@ static inline void user_access_restore(unsigned long e= nabled) { } * We want the unsafe accessors to always be inlined and use * the error labels - thus the macro games. */ -#define unsafe_put_user(x, ptr, label) do { \ - long __err =3D 0; \ - __put_user_nocheck(x, (ptr), __err); \ - if (__err) \ - goto label; \ -} while (0) +#define unsafe_put_user(x, ptr, label) \ + __put_user_nocheck(x, (ptr), label) =20 #define unsafe_get_user(x, ptr, label) do { \ long __err =3D 0; \ --=20 2.34.1 From nobody Thu Dec 18 16:19:07 2025 Received: from mail-oi1-f175.google.com (mail-oi1-f175.google.com [209.85.167.175]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 06DCE20ADFE for ; Thu, 10 Apr 2025 07:05:35 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.167.175 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1744268737; cv=none; b=XE2sFNsJgvWXvTH5JU1tPrSSXz1T21vMnSEi2hdeNB6EFt+3HHGp1Pk0ZldNi/w9XeQclJEb7CivJsSmWPR7hxfGWHjL1kOHLhyUnLPmvRb8SOFcmkyFNU8HoeCOb8I8p/5afi4qUbg8ALG1Z1CUH87s+QySy1CsGbwh/7Oq8D0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1744268737; c=relaxed/simple; bh=7Lt+nLMnw0X+8KO4dKZpRC2lG/LxPyHoQ5hT6u0jJf0=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=DI+dIV5yJW5UcE7ibmBmZXav+Yk7so6kBerwHFT85kkSW4779VkwxrykXfWrh0ebEEUXqV/ThxyjiM0UZiuLWoj0tkgE3jJWol0PDIRn4L2QMTCQF6tz5eV3OzN3Pewq6+xJXbfzr1BD8abl1z74JE14jGtnrZBXx19UnB562jw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=tenstorrent.com; spf=pass smtp.mailfrom=tenstorrent.com; dkim=pass (2048-bit key) header.d=tenstorrent.com header.i=@tenstorrent.com header.b=YEqj1Qxz; arc=none smtp.client-ip=209.85.167.175 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=tenstorrent.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=tenstorrent.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=tenstorrent.com header.i=@tenstorrent.com header.b="YEqj1Qxz" Received: by mail-oi1-f175.google.com with SMTP id 5614622812f47-3fe9fe62295so138968b6e.0 for ; Thu, 10 Apr 2025 00:05:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=tenstorrent.com; s=google; t=1744268735; x=1744873535; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=UI3h0NlmhfF/KG7MnQTXCVk39GvhZn/ER1oVglPpSbw=; b=YEqj1QxzccUe8Dgmt8Mp4axUrB13g1oCEagY/ZUcdOR/EmVjlpanpZWFQhEedzqz3W wVFimE4COhRiGLHzluO/5RleDPU7IOnZcNutK2deqA9dMx40bWO0xwFA6XOxABCR8Wus 6O39Gps/3eM8RCgA1Q1c5kAB8fXH4DKyX2Qx/QMDytuTlFe5xcfCPB3YH0cRPg3BUB1U ZiNRA1YAzMVchawBK39SP/tvaDR1ZeFL7HDGviahG+GPsxMg2iqzuenQM7SBDRvFl4yZ e+uJbxWYsGwuBTXlkl/MuStISjs6gKWG4Xhqy+5SZQDH3Mw3PDCoPqYH3iP8g7ay2MsI s3Ug== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1744268735; x=1744873535; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=UI3h0NlmhfF/KG7MnQTXCVk39GvhZn/ER1oVglPpSbw=; b=gSm8qIieCsCH9uy+DLlQLXXfUtZMz97dYDQFIDMKEfRGGpEH8iBIXgIwi6j0s6m4jD ugss/rKazQEEtcg5UllS0lbtdaMOUernJLxEx6ksmfSOp3HyxthZvWYEEgKXACkPWecp BGMZM/Czhy4uCfFjHWjOynS8xQkGk1ky6y/jEkcLgz+0kvJy8mjurzekg6EZezS7ZbW2 tiYeytcMNU0wmEk9vmGm2U9VDsuU52aj8Lt7qnsYyPi0X8tCfWno5dmfAT/tqflN+diF tgtdo61aRHYlzGzPD3FQI8oxPr1uSmCFtPaJsV5U39rtkMHkjPdIo3jgF7oOpc8UltWj cI2w== X-Forwarded-Encrypted: i=1; AJvYcCX30o0xmp2jAHOp8ySIo2se4o5lFkTvkFhE+4WcOM5qnLsm6h51qVHFV8Me0avJCDF3xMIOR0dqqZU+rZQ=@vger.kernel.org X-Gm-Message-State: AOJu0Yx4psI8o+kW3Ra0a7bCek2Z3EHbGHICpHRyuPsmKhjzzt/S1FJc 4nLZrPgV+xbq8MRzzZUqwSL1BMDWwPnSusR/EELsNLJ8xiNKTYii98xDZVziB68I6Plre8SZmVP 7kw== X-Gm-Gg: ASbGncsumD9Yp5/i3pXCcNJngXjVEE5Y4wI8XhhfHwfZK0B8VwDE3IjZXQK5Q3YeY4Q 6QFiRAXtd+MAj3JLSe+u5//Nn7YUKYoq62/ZexC6DCNVdjoMYBKTxUYrdzU+TBXpZb8Ty4ZohAe eBj6o+7qSMajkFSSMSSDj+heqr7zsCTenG5ieCuQmP6yV25cCUxgJ3KHjZyXb0b55em3Eisjqt0 qYoTnOmShcshaRqvVvC9ZRmjW2nsebyDyb1vwyZ0K9mK3qsft1Mql5MtUeU++rFtoVxvJM4fA96 AdJz6XYGLDqC1tif9lVo0/hkrELZ2LNIzn7Vu5UCUx3qqKtQPx/T3O7Rwq8= X-Google-Smtp-Source: AGHT+IGW0SPg26YQmBEJhc3/+38pKcYOWmw46oXnpTleszoxRuoZLFHb0CFbBs7mvDY52WpYj1W+5Q== X-Received: by 2002:a05:6808:2445:b0:3f8:debb:7683 with SMTP id 5614622812f47-4007bcde3ffmr1239917b6e.23.1744268734951; Thu, 10 Apr 2025 00:05:34 -0700 (PDT) Received: from aus-ird.tenstorrent.com ([38.104.49.66]) by smtp.gmail.com with ESMTPSA id 5614622812f47-40076282ce8sm461369b6e.9.2025.04.10.00.05.33 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 10 Apr 2025 00:05:34 -0700 (PDT) From: Cyril Bur To: palmer@dabbelt.com, aou@eecs.berkeley.edu, paul.walmsley@sifive.com, charlie@rivosinc.com, jrtc27@jrtc27.com, ben.dooks@codethink.co.uk, alex@ghiti.fr Cc: linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org, jszhang@kernel.org Subject: [PATCH v6 5/5] riscv: uaccess: use 'asm_goto_output' for get_user() Date: Thu, 10 Apr 2025 07:05:26 +0000 Message-Id: <20250410070526.3160847-6-cyrilbur@tenstorrent.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20250410070526.3160847-1-cyrilbur@tenstorrent.com> References: <20250410070526.3160847-1-cyrilbur@tenstorrent.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Jisheng Zhang With 'asm goto' we don't need to test the error etc, the exception just jumps to the error handling directly. Unlike put_user(), get_user() must work around GCC bugs [1] when using output clobbers in an asm goto statement. Link: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=3D113921 # 1 Signed-off-by: Jisheng Zhang [Cyril Bur: Rewritten commit message] Signed-off-by: Cyril Bur Reviewed-by: Alexandre Ghiti --- arch/riscv/include/asm/uaccess.h | 95 +++++++++++++++++++++++--------- 1 file changed, 68 insertions(+), 27 deletions(-) diff --git a/arch/riscv/include/asm/uaccess.h b/arch/riscv/include/asm/uacc= ess.h index 719c9179a751..87d01168f80a 100644 --- a/arch/riscv/include/asm/uaccess.h +++ b/arch/riscv/include/asm/uaccess.h @@ -96,27 +96,58 @@ static inline unsigned long __untagged_addr_remote(stru= ct mm_struct *mm, unsigne * call. */ =20 -#define __get_user_asm(insn, x, ptr, err) \ +#ifdef CONFIG_CC_HAS_ASM_GOTO_OUTPUT +#define __get_user_asm(insn, x, ptr, label) \ + asm_goto_output( \ + "1:\n" \ + " " insn " %0, %1\n" \ + _ASM_EXTABLE_UACCESS_ERR(1b, %l2, %0) \ + : "=3D&r" (x) \ + : "m" (*(ptr)) : : label) +#else /* !CONFIG_CC_HAS_ASM_GOTO_OUTPUT */ +#define __get_user_asm(insn, x, ptr, label) \ do { \ - __typeof__(x) __x; \ + long __gua_err =3D 0; \ __asm__ __volatile__ ( \ "1:\n" \ " " insn " %1, %2\n" \ "2:\n" \ _ASM_EXTABLE_UACCESS_ERR_ZERO(1b, 2b, %0, %1) \ - : "+r" (err), "=3D&r" (__x) \ + : "+r" (__gua_err), "=3D&r" (x) \ : "m" (*(ptr))); \ - (x) =3D __x; \ + if (__gua_err) \ + goto label; \ } while (0) +#endif /* CONFIG_CC_HAS_ASM_GOTO_OUTPUT */ =20 #ifdef CONFIG_64BIT -#define __get_user_8(x, ptr, err) \ - __get_user_asm("ld", x, ptr, err) +#define __get_user_8(x, ptr, label) \ + __get_user_asm("ld", x, ptr, label) #else /* !CONFIG_64BIT */ -#define __get_user_8(x, ptr, err) \ + +#ifdef CONFIG_CC_HAS_ASM_GOTO_OUTPUT +#define __get_user_8(x, ptr, label) \ + u32 __user *__ptr =3D (u32 __user *)(ptr); \ + u32 __lo, __hi; \ + asm_goto_output( \ + "1:\n" \ + " lw %0, %2\n" \ + "2:\n" \ + " lw %1, %3\n" \ + _ASM_EXTABLE_UACCESS_ERR(1b, %l4, %0) \ + _ASM_EXTABLE_UACCESS_ERR(2b, %l4, %0) \ + : "=3D&r" (__lo), "=3Dr" (__hi) \ + : "m" (__ptr[__LSW]), "m" (__ptr[__MSW]) \ + : : label); \ + (x) =3D (__typeof__(x))((__typeof__((x) - (x)))( \ + (((u64)__hi << 32) | __lo))); \ + +#else /* !CONFIG_CC_HAS_ASM_GOTO_OUTPUT */ +#define __get_user_8(x, ptr, label) \ do { \ u32 __user *__ptr =3D (u32 __user *)(ptr); \ u32 __lo, __hi; \ + long __gu8_err =3D 0; \ __asm__ __volatile__ ( \ "1:\n" \ " lw %1, %3\n" \ @@ -125,35 +156,51 @@ do { \ "3:\n" \ _ASM_EXTABLE_UACCESS_ERR_ZERO(1b, 3b, %0, %1) \ _ASM_EXTABLE_UACCESS_ERR_ZERO(2b, 3b, %0, %1) \ - : "+r" (err), "=3D&r" (__lo), "=3Dr" (__hi) \ + : "+r" (__gu8_err), "=3D&r" (__lo), "=3Dr" (__hi) \ : "m" (__ptr[__LSW]), "m" (__ptr[__MSW])); \ - if (err) \ + if (__gu8_err) { \ __hi =3D 0; \ - (x) =3D (__typeof__(x))((__typeof__((x)-(x)))( \ + goto label; \ + } \ + (x) =3D (__typeof__(x))((__typeof__((x) - (x)))( \ (((u64)__hi << 32) | __lo))); \ } while (0) +#endif /* CONFIG_CC_HAS_ASM_GOTO_OUTPUT */ + #endif /* CONFIG_64BIT */ =20 -#define __get_user_nocheck(x, __gu_ptr, __gu_err) \ +#define __get_user_nocheck(x, __gu_ptr, label) \ do { \ switch (sizeof(*__gu_ptr)) { \ case 1: \ - __get_user_asm("lb", (x), __gu_ptr, __gu_err); \ + __get_user_asm("lb", (x), __gu_ptr, label); \ break; \ case 2: \ - __get_user_asm("lh", (x), __gu_ptr, __gu_err); \ + __get_user_asm("lh", (x), __gu_ptr, label); \ break; \ case 4: \ - __get_user_asm("lw", (x), __gu_ptr, __gu_err); \ + __get_user_asm("lw", (x), __gu_ptr, label); \ break; \ case 8: \ - __get_user_8((x), __gu_ptr, __gu_err); \ + __get_user_8((x), __gu_ptr, label); \ break; \ default: \ BUILD_BUG(); \ } \ } while (0) =20 +#define __get_user_error(x, ptr, err) \ +do { \ + __label__ __gu_failed; \ + \ + __get_user_nocheck(x, ptr, __gu_failed); \ + err =3D 0; \ + break; \ +__gu_failed: \ + x =3D 0; \ + err =3D -EFAULT; \ +} while (0) + /** * __get_user: - Get a simple variable from user space, with less checking. * @x: Variable to store result. @@ -178,13 +225,16 @@ do { \ ({ \ const __typeof__(*(ptr)) __user *__gu_ptr =3D untagged_addr(ptr); \ long __gu_err =3D 0; \ + __typeof__(x) __gu_val; \ \ __chk_user_ptr(__gu_ptr); \ \ __enable_user_access(); \ - __get_user_nocheck(x, __gu_ptr, __gu_err); \ + __get_user_error(__gu_val, __gu_ptr, __gu_err); \ __disable_user_access(); \ \ + (x) =3D __gu_val; \ + \ __gu_err; \ }) =20 @@ -369,13 +419,7 @@ unsigned long __must_check clear_user(void __user *to,= unsigned long n) } =20 #define __get_kernel_nofault(dst, src, type, err_label) \ -do { \ - long __kr_err =3D 0; \ - \ - __get_user_nocheck(*((type *)(dst)), (type *)(src), __kr_err); \ - if (unlikely(__kr_err)) \ - goto err_label; \ -} while (0) + __get_user_nocheck(*((type *)(dst)), (type *)(src), err_label) =20 #define __put_kernel_nofault(dst, src, type, err_label) \ __put_user_nocheck(*((type *)(src)), (type *)(dst), err_label) @@ -401,12 +445,9 @@ static inline void user_access_restore(unsigned long e= nabled) { } __put_user_nocheck(x, (ptr), label) =20 #define unsafe_get_user(x, ptr, label) do { \ - long __err =3D 0; \ __inttype(*(ptr)) __gu_val; \ - __get_user_nocheck(__gu_val, (ptr), __err); \ + __get_user_nocheck(__gu_val, (ptr), label); \ (x) =3D (__force __typeof__(*(ptr)))__gu_val; \ - if (__err) \ - goto label; \ } while (0) =20 #define unsafe_copy_loop(dst, src, len, type, op, label) \ --=20 2.34.1