From nobody Tue Dec 16 19:46:20 2025 Received: from mail-oi1-f179.google.com (mail-oi1-f179.google.com [209.85.167.179]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 11DF01EDA2F for ; Thu, 20 Mar 2025 22:44:27 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.167.179 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1742510670; cv=none; b=DaRJQCB1lqFgDTQ0AECglznFx3i5Pv51+ZoT1epYl3de2YH/eZTmbR5baZLdFa/gT+9CO4JtVXlGipkxDldKblGe52NnijIltv4eq2FAAr6j7cnktBhrn21de3W0cgDi6BXCcJCly9XHbz2UOiBe+d2coIxyNv+j4/DnzEaeMBU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1742510670; c=relaxed/simple; bh=bvsaQf+q5zAPyQMLZEKjtaO7Zy1Z8ho7KoK1wQhVqTU=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=h019giWRcM+wCMoDH3dSX5zIFy4iBST26PsWM/YjpREPxBq0UGNKI0VVy5VIVWmswns6lfxd3LtzTuz5Bhz//TYUzOuJbj8pDmo/HQE0s3OpdTs2/dAuhYh9/d8xsxJxHLL7ItOKOQWJ2PW5idXrw/55xb9BoC9AksA1rppNC8I= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=tenstorrent.com; spf=pass smtp.mailfrom=tenstorrent.com; dkim=pass (2048-bit key) header.d=tenstorrent.com header.i=@tenstorrent.com header.b=G2/cD79N; arc=none smtp.client-ip=209.85.167.179 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=tenstorrent.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=tenstorrent.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=tenstorrent.com header.i=@tenstorrent.com header.b="G2/cD79N" Received: by mail-oi1-f179.google.com with SMTP id 5614622812f47-3fb3f4bf97aso402045b6e.2 for ; Thu, 20 Mar 2025 15:44:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=tenstorrent.com; s=google; t=1742510667; x=1743115467; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=AMSpj4Pv/tuKqFfvndPeG7a41TGSXjAKvjZxbUe88U8=; b=G2/cD79NvvFvjnDEWf2kp/6hyaXEQ36zPC1EC//tK5UOhflA7akv2Elz8Gv1mH+n58 /xRbNdesM9VDkVVIH3rG+A2+xEel7PaP8anZdKSN6sDElObz5tqrNfVb/gIEw4c7tOpl xDuvpNHYapGUCgc9xCVQwZ0/to2QGLcOlNbzgUMXdhaS8wptCSo70/WaxNdBHfyODopA d4ALSw/0L7wOSGi20m88BAOxZJUq5Z8mTh4AVTZoRtg8goHC8+YAPFrJ8M4oc7pNQWKo PHpNYXdyD8f63C1Uj+sQ+lbUrtU6BU2p6BN060v59f6V4H6ctDkzVqe04VtM9XPlYP9v ZQWw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1742510667; x=1743115467; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=AMSpj4Pv/tuKqFfvndPeG7a41TGSXjAKvjZxbUe88U8=; b=vbUw29DhTpMwSac/jdcA8rYrn3WlX1si1a72dCuy+Nm2ETfXOffDSZt8hbO4KnnKDU Tk6EzCLwVRfCJOQqgeDs7Y2Vd8Q+LB9vWT1m2bqEgpBl53m5fNcZ9tb6Vguz8G3MLe5M /Kxp4tjF0+lTgwVJLLhFYba4sBiYs1bTl6AJ2zw0igGkxbKdIcPBmv4cML9HfToZ2bKO MggWmmItfE3mUsk6e3XYJzXgs463lxdWgfKG3HLN9bexy4ykdC5c8xPUYux/TQjQ2QFd L+d8wVQ1Wpf9rCc3bFkHthhINIVnGg8u+TC5lZ4rUN2OE2brFoTXC2xvQFHnV05WVWSW ccrw== X-Forwarded-Encrypted: i=1; AJvYcCVSVEmv2admD+OZz9ABKGhSTj55dz7ibCDH7rIRNLQ8VenjfLTdYegE352lpcMKm2VHPkhQ628eR+jgO4E=@vger.kernel.org X-Gm-Message-State: AOJu0YzJgdIyodXpcYJsVs+8kSr3zWLhQBcnr1ajHue2yaR5LKrm6Xbe MBu5zfilACQwg12SNVUKx392gCaW5vU9BT+xqZz0/4Xlj4x872jZ+l7IgZBOOCDKLwTHjr4Ey17 YLA7M X-Gm-Gg: ASbGncsBX32TskAAqKvLYxWbFRQeF8EaX4GKXQzeFGJta3H8bXIq9xKXmfXcYpJ+J8p X4FlAv6zLeV3ug4MTEf8ZivYdGUBwJscBdXqn4dPaAszqTJ0lno4mUgTOZCCpTQOX0bH1VP2f76 SlF2ctv1AjXo8BhK7EVI2d81V3bsSl9myT5MZdQDsw8KJEwXDTSKl0b8irpqcu3Z9k+9XkZ+3Uq xrUrkRycSsuaNy6OqR0wAHHRmz4zaNtsDk4PxyjhH/KCbGV5/0+Gb7VfMo/3KRGh7G7JYZtJcr6 wCUTRtwORYoFnzVpOzUiyMZud4cYi+d6+AufFPKKf8rl2dWljfr4nzrZkigBOYs= X-Google-Smtp-Source: AGHT+IFd0nlvhAJDq2aqQ1KFOkNVEh4rXpWJJkMBcG9z9sJUjee94QYnGOMRYiHTi9qWadmuKL5cPg== X-Received: by 2002:a05:6808:250d:b0:3fe:b1fd:527f with SMTP id 5614622812f47-3febf70fc7amr619142b6e.1.1742510667005; Thu, 20 Mar 2025 15:44:27 -0700 (PDT) Received: from aus-ird.tenstorrent.com ([38.104.49.66]) by smtp.gmail.com with ESMTPSA id 5614622812f47-3febf6dcc09sm103524b6e.12.2025.03.20.15.44.25 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 20 Mar 2025 15:44:26 -0700 (PDT) From: Cyril Bur To: palmer@dabbelt.com, aou@eecs.berkeley.edu, paul.walmsley@sifive.com, charlie@rivosinc.com, jrtc27@jrtc27.com, ben.dooks@codethink.co.uk, alex@ghiti.fr Cc: linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org, jszhang@kernel.org, syzbot+e74b94fe601ab9552d69@syzkaller.appspotmail.com Subject: [PATCH v5 1/5] riscv: save the SR_SUM status over switches Date: Thu, 20 Mar 2025 22:44:19 +0000 Message-Id: <20250320224423.1838493-2-cyrilbur@tenstorrent.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20250320224423.1838493-1-cyrilbur@tenstorrent.com> References: <20250320224423.1838493-1-cyrilbur@tenstorrent.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Ben Dooks When threads/tasks are switched we need to ensure the old execution's SR_SUM state is saved and the new thread has the old SR_SUM state restored. The issue is seen under heavy load especially with the syz-stress tool running, with crashes as follows in schedule_tail: Unable to handle kernel access to user memory without uaccess routines at virtual address 000000002749f0d0 Oops [#1] Modules linked in: CPU: 1 PID: 4875 Comm: syz-executor.0 Not tainted 5.12.0-rc2-syzkaller-00467-g0d7588ab9ef9 #0 Hardware name: riscv-virtio,qemu (DT) epc : schedule_tail+0x72/0xb2 kernel/sched/core.c:4264 ra : task_pid_vnr include/linux/sched.h:1421 [inline] ra : schedule_tail+0x70/0xb2 kernel/sched/core.c:4264 epc : ffffffe00008c8b0 ra : ffffffe00008c8ae sp : ffffffe025d17ec0 gp : ffffffe005d25378 tp : ffffffe00f0d0000 t0 : 0000000000000000 t1 : 0000000000000001 t2 : 00000000000f4240 s0 : ffffffe025d17ee0 s1 : 000000002749f0d0 a0 : 000000000000002a a1 : 0000000000000003 a2 : 1ffffffc0cfac500 a3 : ffffffe0000c80cc a4 : 5ae9db91c19bbe00 a5 : 0000000000000000 a6 : 0000000000f00000 a7 : ffffffe000082eba s2 : 0000000000040000 s3 : ffffffe00eef96c0 s4 : ffffffe022c77fe0 s5 : 0000000000004000 s6 : ffffffe067d74e00 s7 : ffffffe067d74850 s8 : ffffffe067d73e18 s9 : ffffffe067d74e00 s10: ffffffe00eef96e8 s11: 000000ae6cdf8368 t3 : 5ae9db91c19bbe00 t4 : ffffffc4043cafb2 t5 : ffffffc4043cafba t6 : 0000000000040000 status: 0000000000000120 badaddr: 000000002749f0d0 cause: 000000000000000f Call Trace: [] schedule_tail+0x72/0xb2 kernel/sched/core.c:4264 [] ret_from_exception+0x0/0x14 Dumping ftrace buffer: (ftrace buffer empty) Reported-by: syzbot+e74b94fe601ab9552d69@syzkaller.appspotmail.com ---[ end trace b5f8f9231dc87dda ]--- The issue comes from the put_user() in schedule_tail (kernel/sched/core.c) doing the following: asmlinkage __visible void schedule_tail(struct task_struct *prev) { ... if (current->set_child_tid) put_user(task_pid_vnr(current), current->set_child_tid); ... } the put_user() macro causes the code sequence to come out as follows: 1: __enable_user_access() 2: reg =3D task_pid_vnr(current); 3: *current->set_child_tid =3D reg; 4: __disable_user_access() This means the task_pid_vnr() is being called with user-access enabled which itself is not a good idea, but that is a separate issue. Here we have a function that /might/ sleep being called with the SR_SUM and if it does, then it returns with the SR_SUM flag possibly cleared thus causing the above abort. To try and deal with this, and stop the SR_SUM leaking out into other threads (this has also been tested and see under stress. It can rarely happen but it /does/ under load) make sure the __switch_to() will save and restore the SR_SUM flag, and clear it possibly for the next thread if it does not need it. Note, test code to be supplied once other checks have been finished. There may be further issues with the mstatus flags with this, this can be discussed further once some initial testing has been done. Reported-by: syzbot+e74b94fe601ab9552d69@syzkaller.appspotmail.com Signed-off-by: Ben Dooks Signed-off-by: Cyril Bur --- arch/riscv/include/asm/processor.h | 1 + arch/riscv/kernel/asm-offsets.c | 5 +++++ arch/riscv/kernel/entry.S | 8 ++++++++ 3 files changed, 14 insertions(+) diff --git a/arch/riscv/include/asm/processor.h b/arch/riscv/include/asm/pr= ocessor.h index 5f56eb9d114a..0de05d652e0f 100644 --- a/arch/riscv/include/asm/processor.h +++ b/arch/riscv/include/asm/processor.h @@ -103,6 +103,7 @@ struct thread_struct { struct __riscv_d_ext_state fstate; unsigned long bad_cause; unsigned long envcfg; + unsigned long flags; u32 riscv_v_flags; u32 vstate_ctrl; struct __riscv_v_ext_state vstate; diff --git a/arch/riscv/kernel/asm-offsets.c b/arch/riscv/kernel/asm-offset= s.c index e89455a6a0e5..556ebcbb7e22 100644 --- a/arch/riscv/kernel/asm-offsets.c +++ b/arch/riscv/kernel/asm-offsets.c @@ -34,6 +34,7 @@ void asm_offsets(void) OFFSET(TASK_THREAD_S9, task_struct, thread.s[9]); OFFSET(TASK_THREAD_S10, task_struct, thread.s[10]); OFFSET(TASK_THREAD_S11, task_struct, thread.s[11]); + OFFSET(TASK_THREAD_FLAGS, task_struct, thread.flags); =20 OFFSET(TASK_TI_CPU, task_struct, thread_info.cpu); OFFSET(TASK_TI_FLAGS, task_struct, thread_info.flags); @@ -347,6 +348,10 @@ void asm_offsets(void) offsetof(struct task_struct, thread.s[11]) - offsetof(struct task_struct, thread.ra) ); + DEFINE(TASK_THREAD_FLAGS_RA, + offsetof(struct task_struct, thread.flags) + - offsetof(struct task_struct, thread.ra) + ); =20 DEFINE(TASK_THREAD_F0_F0, offsetof(struct task_struct, thread.fstate.f[0]) diff --git a/arch/riscv/kernel/entry.S b/arch/riscv/kernel/entry.S index 33a5a9f2a0d4..c278b3ac37b9 100644 --- a/arch/riscv/kernel/entry.S +++ b/arch/riscv/kernel/entry.S @@ -397,9 +397,17 @@ SYM_FUNC_START(__switch_to) REG_S s9, TASK_THREAD_S9_RA(a3) REG_S s10, TASK_THREAD_S10_RA(a3) REG_S s11, TASK_THREAD_S11_RA(a3) + + /* save (and disable the user space access flag) */ + li s0, SR_SUM + csrrc s1, CSR_STATUS, s0 + REG_S s1, TASK_THREAD_FLAGS_RA(a3) + /* Save the kernel shadow call stack pointer */ scs_save_current /* Restore context from next->thread */ + REG_L s0, TASK_THREAD_FLAGS_RA(a4) + csrs CSR_STATUS, s0 REG_L ra, TASK_THREAD_RA_RA(a4) REG_L sp, TASK_THREAD_SP_RA(a4) REG_L s0, TASK_THREAD_S0_RA(a4) --=20 2.34.1 From nobody Tue Dec 16 19:46:20 2025 Received: from mail-oi1-f174.google.com (mail-oi1-f174.google.com [209.85.167.174]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 43869223707 for ; Thu, 20 Mar 2025 22:44:28 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.167.174 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1742510670; cv=none; b=X+2BVyrMm30fRnY7Xs7RIl4IzK9qKWzvKr4MZvLcuYYrg6lfJmgjxJFG/kS2zqJBqUjMkhDbyBf0imiwbG7HW5ykTvpEeUE9ZjOxIsS3rKslGB3nJWts9M7Iw+LlGesr4Xl6ezUy7ep3pS+wtyz2CZSy3CYqP5vyaehfgKHunGg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1742510670; c=relaxed/simple; bh=p7gn8HCoRfmei461kdKTMrhmaoZ+R8QG+5SxtGUWr5I=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=aywzQjdhBGxoXa/mIu34ECKJUFfTk4+povjEugwVG7GtGYaLebh/vFJXqI67kCXLJs1jlXwg5ilfvtR2WCwq/fpvMvUmN42HPXsmHMuTpZ0riP/Qe8FkKOUndPfgM6OwACu+OAjDop5nqO94I4Sp2WJVYrE8jVaWCVftg9VgrrY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=tenstorrent.com; spf=pass smtp.mailfrom=tenstorrent.com; dkim=pass (2048-bit key) header.d=tenstorrent.com header.i=@tenstorrent.com header.b=VUIAL8a2; arc=none smtp.client-ip=209.85.167.174 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=tenstorrent.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=tenstorrent.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=tenstorrent.com header.i=@tenstorrent.com header.b="VUIAL8a2" Received: by mail-oi1-f174.google.com with SMTP id 5614622812f47-3f3f4890596so755580b6e.2 for ; Thu, 20 Mar 2025 15:44:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=tenstorrent.com; s=google; t=1742510668; x=1743115468; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Vvw7nYWiLBIqfTSpurYIsb0A2VRmihRadD2HqJzvOMM=; b=VUIAL8a2f7OpfpEGUQzRdhUPoA5r4jlbbhRg7IU0fhWRf2aKjtPtF47I8H8LWSGXfZ PjsBCYNga7BtCfs3S7DdT/RLOqvzYrpRy99nxyBAkobMRp5A6sZxZhfuQ7lJjXpTIQxF ZEAfF/3+l5u5uHxh3l2sh3P/dZiXrch3d0Dameuirktz1p2NC5Q6pgMGIUQYx7Hs9Bjs rqNPqOzloOgv0VsQcrEa8FybtC94F3G2xMK+PDo9GIaMj5AoRv0foY8y8R3q5mKJ8bFi Kv+UvfLLpIHrwErPmSkei0Q+wQdweCzN3RyK4xzmAl80acDBOksQqNNYKpDsyRA2/dms pBQg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1742510668; x=1743115468; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Vvw7nYWiLBIqfTSpurYIsb0A2VRmihRadD2HqJzvOMM=; b=W2DyHP2ekdNNHaFsdd2xUTSABucCH+GM5BvwmVYMDVePTPHVw5oMjA+KztDf8sfuTq rrn6wE5PCrOM629YDFoPu8uuzCZvkweDumXXTcWtoYPsSRqYRh4wEKIR87q06zPOiWlS CwTc7nwjffdGSL8swRtf6fz27NsCZFuc2VUvBLaG9mAqSx9tHwxrIfyVrLRDUm7qrAQv P/DUBFUd2y0yzptOd1u0hG2nDPOp6D8RVURpQ3azWhzSzqCK7WF+wRnUymjq2jtPxref +RDdhEFVyPNp9VOO0+elvHHkL6jZDtE/F9JgwG2INuBVsGkQP8dXPGsUtxJq+KezBlaN ZP7w== X-Forwarded-Encrypted: i=1; AJvYcCURcHNdXUvv8xKZ76IqW5sjaWokOPSYRtKtin+gsr6h+2sr2/L3THkgvSrwAsq/gassU8NZvCTHyxcZ5jk=@vger.kernel.org X-Gm-Message-State: AOJu0YxD/B64WZfzwXo2osLxnu6DjRPofnsFM22WOFL7Qywhl8DLxXBI vp47zJzw/8JXw0BVK/yC3SyV7avzMRcjNU14TZ8FcwB35MhLLmIZqeMso5QM8w== X-Gm-Gg: ASbGncv/q5f/bNGN1g7o19OXDxNDd0zQ+N6c7KVoRhOyUR04MK/MrOciGannE0uaIHf 3vOCttOk/7yDeo3I3XK46fstFwv63xQprdmdPIi4RIt6De86XZ1W30nv1sdYuCKe8z2t1b9kgwq nIKEndlBzt/xLM6+CAnUIxjlCqRa7SuKWFEf3wMYhADp9XTvp4WvTKI6Ttzuf9B4/MMxf5X3DOb wb+SgUklIA/Zdxw+Ge+ngJv0ypPgBuHGUqeHN7UUCXlYc3m1bJmURGMIUSHMQJf8gyZMt4WXWhH FhEBpwzu2s4zqf5hFc66+AoBgwzpCUoe6jJr7/EVD3k0g2I00kWUtTv+2bz3ClE= X-Google-Smtp-Source: AGHT+IFJedQeXWVCp8Wwl18/3gDvFSbrwpqc13CUW17x0vVkYsJlg/z2sLYvBni60TXj6/O4eMplxg== X-Received: by 2002:a05:6808:1495:b0:3fa:d6c:cdb8 with SMTP id 5614622812f47-3febf79528emr596590b6e.38.1742510668153; Thu, 20 Mar 2025 15:44:28 -0700 (PDT) Received: from aus-ird.tenstorrent.com ([38.104.49.66]) by smtp.gmail.com with ESMTPSA id 5614622812f47-3febf6dcc09sm103524b6e.12.2025.03.20.15.44.27 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 20 Mar 2025 15:44:27 -0700 (PDT) From: Cyril Bur To: palmer@dabbelt.com, aou@eecs.berkeley.edu, paul.walmsley@sifive.com, charlie@rivosinc.com, jrtc27@jrtc27.com, ben.dooks@codethink.co.uk, alex@ghiti.fr Cc: linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org, jszhang@kernel.org Subject: [PATCH v5 2/5] riscv: implement user_access_begin() and families Date: Thu, 20 Mar 2025 22:44:20 +0000 Message-Id: <20250320224423.1838493-3-cyrilbur@tenstorrent.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20250320224423.1838493-1-cyrilbur@tenstorrent.com> References: <20250320224423.1838493-1-cyrilbur@tenstorrent.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Jisheng Zhang Currently, when a function like strncpy_from_user() is called, the userspace access protection is disabled and enabled for every word read. By implementing user_access_begin() and families, the protection is disabled at the beginning of the copy and enabled at the end. The __inttype macro is borrowed from x86 implementation. Signed-off-by: Jisheng Zhang Signed-off-by: Cyril Bur --- arch/riscv/include/asm/uaccess.h | 76 ++++++++++++++++++++++++++++++++ 1 file changed, 76 insertions(+) diff --git a/arch/riscv/include/asm/uaccess.h b/arch/riscv/include/asm/uacc= ess.h index fee56b0c8058..c9a461467bf4 100644 --- a/arch/riscv/include/asm/uaccess.h +++ b/arch/riscv/include/asm/uaccess.h @@ -61,6 +61,19 @@ static inline unsigned long __untagged_addr_remote(struc= t mm_struct *mm, unsigne #define __disable_user_access() \ __asm__ __volatile__ ("csrc sstatus, %0" : : "r" (SR_SUM) : "memory") =20 +/* + * This is the smallest unsigned integer type that can fit a value + * (up to 'long long') + */ +#define __inttype(x) __typeof__( \ + __typefits(x, char, \ + __typefits(x, short, \ + __typefits(x, int, \ + __typefits(x, long, 0ULL))))) + +#define __typefits(x, type, not) \ + __builtin_choose_expr(sizeof(x) <=3D sizeof(type), (unsigned type)0, not) + /* * The exception table consists of pairs of addresses: the first is the * address of an instruction that is allowed to fault, and the second is @@ -368,6 +381,69 @@ do { \ goto err_label; \ } while (0) =20 +static __must_check __always_inline bool user_access_begin(const void __us= er *ptr, size_t len) +{ + if (unlikely(!access_ok(ptr, len))) + return 0; + __enable_user_access(); + return 1; +} +#define user_access_begin user_access_begin +#define user_access_end __disable_user_access + +static inline unsigned long user_access_save(void) { return 0UL; } +static inline void user_access_restore(unsigned long enabled) { } + +/* + * We want the unsafe accessors to always be inlined and use + * the error labels - thus the macro games. + */ +#define unsafe_put_user(x, ptr, label) do { \ + long __err =3D 0; \ + __put_user_nocheck(x, (ptr), __err); \ + if (__err) \ + goto label; \ +} while (0) + +#define unsafe_get_user(x, ptr, label) do { \ + long __err =3D 0; \ + __inttype(*(ptr)) __gu_val; \ + __get_user_nocheck(__gu_val, (ptr), __err); \ + (x) =3D (__force __typeof__(*(ptr)))__gu_val; \ + if (__err) \ + goto label; \ +} while (0) + +#define unsafe_copy_loop(dst, src, len, type, op, label) \ + while (len >=3D sizeof(type)) { \ + op(*(type *)(src), (type __user *)(dst), label); \ + dst +=3D sizeof(type); \ + src +=3D sizeof(type); \ + len -=3D sizeof(type); \ + } + +#define unsafe_copy_to_user(_dst, _src, _len, label) \ +do { \ + char __user *__ucu_dst =3D (_dst); \ + const char *__ucu_src =3D (_src); \ + size_t __ucu_len =3D (_len); \ + unsafe_copy_loop(__ucu_dst, __ucu_src, __ucu_len, u64, unsafe_put_user, l= abel); \ + unsafe_copy_loop(__ucu_dst, __ucu_src, __ucu_len, u32, unsafe_put_user, l= abel); \ + unsafe_copy_loop(__ucu_dst, __ucu_src, __ucu_len, u16, unsafe_put_user, l= abel); \ + unsafe_copy_loop(__ucu_dst, __ucu_src, __ucu_len, u8, unsafe_put_user, la= bel); \ +} while (0) + +#define unsafe_copy_from_user(_dst, _src, _len, label) \ +do { \ + char *__ucu_dst =3D (_dst); \ + const char __user *__ucu_src =3D (_src); \ + size_t __ucu_len =3D (_len); \ + unsafe_copy_loop(__ucu_src, __ucu_dst, __ucu_len, u64, unsafe_get_user, l= abel); \ + unsafe_copy_loop(__ucu_src, __ucu_dst, __ucu_len, u32, unsafe_get_user, l= abel); \ + unsafe_copy_loop(__ucu_src, __ucu_dst, __ucu_len, u16, unsafe_get_user, l= abel); \ + unsafe_copy_loop(__ucu_src, __ucu_dst, __ucu_len, u8, unsafe_get_user, la= bel); \ +} while (0) + #else /* CONFIG_MMU */ #include #endif /* CONFIG_MMU */ --=20 2.34.1 From nobody Tue Dec 16 19:46:20 2025 Received: from mail-oi1-f174.google.com (mail-oi1-f174.google.com [209.85.167.174]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D40EA22A4F8 for ; Thu, 20 Mar 2025 22:44:30 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.167.174 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1742510672; cv=none; b=DmswjL5LIxKBlZgCzJbXOZQoQuCp6d6gHQ25yj7tAtAiVrhgCnHEutSsXSvJbP+nb2DkQQMN4qp1CVxmh/RN5XgPEQg4GMKb4beHrd9/V2JaJjMzxo5v56/Nsjw1joExSHSQwVOaoWmbhXTqkd9PrgOvgI7NqlGEF3OhvM30Kdo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1742510672; c=relaxed/simple; bh=HKMnBVdHsElfI+VU/kTaLpcWor2d4s9BxLReognxMNg=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=qUsGKbJmD7+9/gispbypKOo5ihn+k1PRqStqvffeXuR/xg4WX3Bs9hOgMrzJOze6rZ6szZjC2O5DMBlu/78UP+Kk86o808wEAgeI4CDBOYsNwsLoXb9Drk58NOGxOJbpy0u9OyVZNULOItMb7EOB7zuU1H4wRfu5CPGukzUWEd4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=tenstorrent.com; spf=pass smtp.mailfrom=tenstorrent.com; dkim=pass (2048-bit key) header.d=tenstorrent.com header.i=@tenstorrent.com header.b=aAFzRlBe; arc=none smtp.client-ip=209.85.167.174 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=tenstorrent.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=tenstorrent.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=tenstorrent.com header.i=@tenstorrent.com header.b="aAFzRlBe" Received: by mail-oi1-f174.google.com with SMTP id 5614622812f47-3fea67e64caso803828b6e.2 for ; Thu, 20 Mar 2025 15:44:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=tenstorrent.com; s=google; t=1742510670; x=1743115470; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=mh8UBXGzUbnvdsif2sw+KTYhRC1BSjBoml8SqEobyIk=; b=aAFzRlBeXhuIoT3hwxsqqL5DcWagE+GOYw7XWHEXF52kAvMwaYFf/GQQp+0xeAMkmt Lzk1N6ARexytSbVbuZ0edssnL/Hap3xnI3CgH0MZtEeNs5gOf6724ilY79z+tWkcG03s WgYhRHCEVswwH7eCRt58Jwi/JSxrkxHRRr1QVzGPBphUGSWqSeU5Wz4JncIGRjuQ9/l8 1caF9KjCP55wSxM5n6QzLzqAP1JEfZeAbAPycJKvzlILMj98DU0+ijnlIbZzS6bKC2ey 5Qeso9YOAe5vXu+QNyon+CZSiaUMCKfnBzdyzFEgi7W7dB+vVG7PquRLtecILqmVLvVS D2wA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1742510670; x=1743115470; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=mh8UBXGzUbnvdsif2sw+KTYhRC1BSjBoml8SqEobyIk=; b=P4CGbZAdPT6KfaEkn9tG93oXg84GrtLRtYBBkdvVE3GONzjLtrOturseyTG31uOgn9 6hEd0lvk69pdjz7GNsmxbZblhvDeK+mqQdfnMAUwIaDAafMqoPldT3IhKESOMEyMwH3s 4FhbemiBY/P91PR1EDtuD7yuADAUhWJ0Zs0eJC0T0U6JfgYXoW0RK1ejio0VP0ikGcZK ENwDVs9ULlnMxoH2iuAMz4bvqZEVtLfiyz6xJ7g3qky9U7ByxY1hz5qD2fxmzotwHRi/ QBlB3ZfuuufDlUAO+/Rk6JpkbWrZ4EY+copvCnh8P5+/Ny9/cjmmyeS8V0pcfGH8hhtd 8hAg== X-Forwarded-Encrypted: i=1; AJvYcCWBx6MvtUDYQHlkEKQDT02NU7heNomLcbtXXuVi+RDNx9iro2fMvt85dxdp5g+JOPMHSpkN7QOchXo4dAA=@vger.kernel.org X-Gm-Message-State: AOJu0YwUwuKXGAw6S5mlfiTLU4wRjM5/hmN/94o/2IwNbZRzfEdp6ocs cLwLmjv83k90wNqM4QC2txkxFXctltRnCkp70XYXDbHOg1TyCxH8SVhtPr5a8A== X-Gm-Gg: ASbGnct0H6NVHegCg1Q4JZGwlTAlYJ3VK9A3t+dvbWp2b7p72OxID+SFHrElATp8k/O H4G1Ov0xj3LfqmDFZAdXeD8stDqseHX9mBxico8zw/KlppwIXwkLhtNP0QPni6cm44KYTTZQChi MgoFO6osxLR2PotTBgpz0MWeVBZz2ZgIdd5swRcrrtMkRoko2/gFA4sz74UZ/oHdj+Rm7Ip6MCO vfvE9xjd2txAIqNW2goCrUP8Ndnr+yZzDQd4/jEdASkfQM0v1K725AL4NgtTY1+5WOAMDTidrzI tgPtu2iFHfyWDFLJ3TtvlW5msztDmlVaHAf554GTYYGt4Iv6gsjOkqdJpnIpWJnOnAcCwrGXXQ= = X-Google-Smtp-Source: AGHT+IFPgB4lmDSuUaCrMpZ/2qFcRUoJ3Y+avxEgpvQKbDo7r4YIRe2aaoiARhfstm7Tp4du8Zb0Lg== X-Received: by 2002:a05:6808:1a12:b0:3fb:7ac1:512c with SMTP id 5614622812f47-3febf7481femr654995b6e.17.1742510669653; Thu, 20 Mar 2025 15:44:29 -0700 (PDT) Received: from aus-ird.tenstorrent.com ([38.104.49.66]) by smtp.gmail.com with ESMTPSA id 5614622812f47-3febf6dcc09sm103524b6e.12.2025.03.20.15.44.28 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 20 Mar 2025 15:44:28 -0700 (PDT) From: Cyril Bur To: palmer@dabbelt.com, aou@eecs.berkeley.edu, paul.walmsley@sifive.com, charlie@rivosinc.com, jrtc27@jrtc27.com, ben.dooks@codethink.co.uk, alex@ghiti.fr Cc: linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org, jszhang@kernel.org Subject: [PATCH v5 3/5] riscv: uaccess: use input constraints for ptr of __put_user() Date: Thu, 20 Mar 2025 22:44:21 +0000 Message-Id: <20250320224423.1838493-4-cyrilbur@tenstorrent.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20250320224423.1838493-1-cyrilbur@tenstorrent.com> References: <20250320224423.1838493-1-cyrilbur@tenstorrent.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Jisheng Zhang Putting ptr in the inputs as opposed to output may seem incorrect but this is done for a few reasons: - Not having it in the output permits the use of asm goto in a subsequent patch. There are bugs in gcc [1] which would otherwise prevent it. - Since the output memory is userspace there isn't any real benefit from telling the compiler about the memory clobber. - x86, arm and powerpc all use this technique. Link: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=3D113921 # 1 Signed-off-by: Jisheng Zhang [Cyril Bur: Rewritten commit message] Signed-off-by: Cyril Bur --- arch/riscv/include/asm/uaccess.h | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/arch/riscv/include/asm/uaccess.h b/arch/riscv/include/asm/uacc= ess.h index c9a461467bf4..da36057847f0 100644 --- a/arch/riscv/include/asm/uaccess.h +++ b/arch/riscv/include/asm/uaccess.h @@ -219,11 +219,11 @@ do { \ __typeof__(*(ptr)) __x =3D x; \ __asm__ __volatile__ ( \ "1:\n" \ - " " insn " %z2, %1\n" \ + " " insn " %z1, %2\n" \ "2:\n" \ _ASM_EXTABLE_UACCESS_ERR(1b, 2b, %0) \ - : "+r" (err), "=3Dm" (*(ptr)) \ - : "rJ" (__x)); \ + : "+r" (err) \ + : "rJ" (__x), "m"(*(ptr))); \ } while (0) =20 #ifdef CONFIG_64BIT @@ -236,16 +236,16 @@ do { \ u64 __x =3D (__typeof__((x)-(x)))(x); \ __asm__ __volatile__ ( \ "1:\n" \ - " sw %z3, %1\n" \ + " sw %z1, %3\n" \ "2:\n" \ - " sw %z4, %2\n" \ + " sw %z2, %4\n" \ "3:\n" \ _ASM_EXTABLE_UACCESS_ERR(1b, 3b, %0) \ _ASM_EXTABLE_UACCESS_ERR(2b, 3b, %0) \ - : "+r" (err), \ - "=3Dm" (__ptr[__LSW]), \ - "=3Dm" (__ptr[__MSW]) \ - : "rJ" (__x), "rJ" (__x >> 32)); \ + : "+r" (err) \ + : "rJ" (__x), "rJ" (__x >> 32), \ + "m" (__ptr[__LSW]), \ + "m" (__ptr[__MSW])); \ } while (0) #endif /* CONFIG_64BIT */ =20 --=20 2.34.1 From nobody Tue Dec 16 19:46:20 2025 Received: from mail-oi1-f179.google.com (mail-oi1-f179.google.com [209.85.167.179]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 2909522A81D for ; Thu, 20 Mar 2025 22:44:31 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.167.179 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1742510674; cv=none; b=dQJgWfnVvaJNoVQwGQSZS3CZXd/2OqDe8K1V2fQC3St3HS+DZqeOF6fZtDVmvet4kF6x0NwNPNaYnL5y1ZzI7WiEV8XplWKGynd4V3Cg8NT4IPUfd5DA/BbCnVMvl0tHTS/hZk5qn7eXRwTMLR3jPXNKWLioYfaIpM75GP5LkeQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1742510674; c=relaxed/simple; bh=6Rd4HzTmjSi8Wj2+T4c6yX3o6/8i9wfFJ0rHfw2fL5Y=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=SnjpxrZfTNFuMJS34Sb/O6aaMIKqGNQ+CxVqLZHUMPwwxWEr/KKRY0suNGVsb6znsfZBduG6vTuEJM2U/RqFI8VQFP31cPg+P5l09zik8u8Y6oMQKxdbP702BvY92TMzH+ek5LYXYHwEj+cN7E296HaWg12bwDkQKIOB7B8BOV8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=tenstorrent.com; spf=pass smtp.mailfrom=tenstorrent.com; dkim=pass (2048-bit key) header.d=tenstorrent.com header.i=@tenstorrent.com header.b=Dic9Oet8; arc=none smtp.client-ip=209.85.167.179 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=tenstorrent.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=tenstorrent.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=tenstorrent.com header.i=@tenstorrent.com header.b="Dic9Oet8" Received: by mail-oi1-f179.google.com with SMTP id 5614622812f47-3fea0363284so764396b6e.1 for ; Thu, 20 Mar 2025 15:44:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=tenstorrent.com; s=google; t=1742510671; x=1743115471; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=DpC/LMch4vBncOIb4H+uNBUj+A3evJlilB0f5X4S7k8=; b=Dic9Oet8Sshu46gibRue+HinZOLUFUx6Fd36fJqi7VyTKwXyzaSkbHd8FftkwrFkis 5YIoRz95N2emZX58uMqVJ8lU16TfAfYx/5TJ76dJxtdiRj23/Tr4wKWdcRLASo2U/0+7 pkHAMwBEsdzEAfPZ6bPh7Cw/MMy/EIi29dSKCJSX4bYm8Qk3RlWikA2GV2Vg1rbD//y3 HEJ3h4TwIAyTh9S7at+in1UbQy60QJsxKOasj27er5sWZJy5mWfqTkEIb5Q2eUU0vvKJ +ES1GB/LLqH1PdToZOomw6yNn5Q9qhtbcksmDMMNP6KdMUzwx0+YSUmpZTKmt/cbI63O Ocqg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1742510671; x=1743115471; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=DpC/LMch4vBncOIb4H+uNBUj+A3evJlilB0f5X4S7k8=; b=VCV0BHJVOeF182+8wgRt5AwmJ5GEvxA69osixRdsL6lcUDq1+qcH8xAiRiIfCabnh9 acF0kkRnGRJV79OPN3PtL39+Ffgy9b8I4nu5RW4sD8ZtGYty+1DMrt4H0UHiteMAo5r5 ZgIXx0HS/NEWEw1ljTe4LWQ0D4udTJMdrkUbZz89mj4TlglIdeiF0b7g8j2xcsJxAIXy vMi5jL19TYZUQ8L6q/mVhMsiIjIUgqK226EFwTHWpVIFdN9BxCg/fsb43GFK7aISLXkn BsSYkeWeOxqkFp4Nm1caY0k6nRUMkwX+Rb77zc15x4zWS5A9o4RbyjIMqvdKSB5Uej+i TYvQ== X-Forwarded-Encrypted: i=1; AJvYcCW6EHd9ULO5fuY3Bt/2rpmX9tVPojGOFGheOup8gI0nym5oDdtKBRqwLgCUGsZHrnfzzn7idQ9po3OWW0c=@vger.kernel.org X-Gm-Message-State: AOJu0YyX8O3sIVP1VBp3JWZ3tcLW3jWYXk7oX4Ufnad+GYAm6KRQWXpp CQbsO3N4Otu4wtUMLGt/EZrZzYRL+RlW3HsYaUfBw+YMlNlHJuseh8MZuAZ8LHisYafmXCaXiPH nTIyo X-Gm-Gg: ASbGncs6hROb8E8cLEygPqEVJ7uZaKa0cvgEAWtgObKKYrxPuCFK7Nh69BssKVbt3SQ 0dfIwuA/qS6kv8YXR6F3j0n8zUMhBh2QWrrqqdivxO7+4IHfVjh6KpDZEGMW8Q9gX4mj24uFoi7 yMMVIMCzP9TZnK8PXu5cWAnGdygiKX9EVG0+ZrSrCe6J7dT/nalMsvts0ebS8BBSr0yVLLgvaUw tZWWwntM1wU1jzcVpcsdgS992hCxQV3+VQ6pmhc7I5m4mT2SBdu6v5WJqCPy3wxOKT+QArAQqdw VS37eIHiYVOIzBnxbA1cP73XZT3NBP2GLcnUOiX6Mwzxv5nu2FZqrOFJGxMEp8M= X-Google-Smtp-Source: AGHT+IE2wGE2Be/0SH2O70UOpsv0bSUloPlhYCg9sxzfrbSR5xy7GPcyoBTaN4dfYwWSgd3IGSKcUA== X-Received: by 2002:a05:6808:1814:b0:3f4:12a:8ca0 with SMTP id 5614622812f47-3febeda6d98mr810784b6e.4.1742510671178; Thu, 20 Mar 2025 15:44:31 -0700 (PDT) Received: from aus-ird.tenstorrent.com ([38.104.49.66]) by smtp.gmail.com with ESMTPSA id 5614622812f47-3febf6dcc09sm103524b6e.12.2025.03.20.15.44.29 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 20 Mar 2025 15:44:30 -0700 (PDT) From: Cyril Bur To: palmer@dabbelt.com, aou@eecs.berkeley.edu, paul.walmsley@sifive.com, charlie@rivosinc.com, jrtc27@jrtc27.com, ben.dooks@codethink.co.uk, alex@ghiti.fr Cc: linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org, jszhang@kernel.org Subject: [PATCH v5 4/5] riscv: uaccess: use 'asm goto' for put_user() Date: Thu, 20 Mar 2025 22:44:22 +0000 Message-Id: <20250320224423.1838493-5-cyrilbur@tenstorrent.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20250320224423.1838493-1-cyrilbur@tenstorrent.com> References: <20250320224423.1838493-1-cyrilbur@tenstorrent.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Jisheng Zhang With 'asm goto' we don't need to test the error etc, the exception just jumps to the error handling directly. Because there are no output clobbers which could trigger gcc bugs [1] the use of asm_goto_output() macro is not necessary here. Not using asm_goto_output() is desirable as the generated output asm will be cleaner. Use of the volatile keyword is redundant as per gcc 14.2.0 manual section 6.48.2.7 Goto Labels: > Also note that an asm goto statement is always implicitly considered volatile. Link: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=3D113921 # 1 Signed-off-by: Jisheng Zhang [Cyril Bur: Rewritten commit message] Signed-off-by: Cyril Bur --- arch/riscv/include/asm/uaccess.h | 71 +++++++++++++++----------------- 1 file changed, 33 insertions(+), 38 deletions(-) diff --git a/arch/riscv/include/asm/uaccess.h b/arch/riscv/include/asm/uacc= ess.h index da36057847f0..719c9179a751 100644 --- a/arch/riscv/include/asm/uaccess.h +++ b/arch/riscv/include/asm/uaccess.h @@ -214,61 +214,66 @@ do { \ ((x) =3D (__force __typeof__(x))0, -EFAULT); \ }) =20 -#define __put_user_asm(insn, x, ptr, err) \ +#define __put_user_asm(insn, x, ptr, label) \ do { \ __typeof__(*(ptr)) __x =3D x; \ - __asm__ __volatile__ ( \ + asm goto( \ "1:\n" \ - " " insn " %z1, %2\n" \ - "2:\n" \ - _ASM_EXTABLE_UACCESS_ERR(1b, 2b, %0) \ - : "+r" (err) \ - : "rJ" (__x), "m"(*(ptr))); \ + " " insn " %z0, %1\n" \ + _ASM_EXTABLE(1b, %l2) \ + : : "rJ" (__x), "m"(*(ptr)) : : label); \ } while (0) =20 #ifdef CONFIG_64BIT -#define __put_user_8(x, ptr, err) \ - __put_user_asm("sd", x, ptr, err) +#define __put_user_8(x, ptr, label) \ + __put_user_asm("sd", x, ptr, label) #else /* !CONFIG_64BIT */ -#define __put_user_8(x, ptr, err) \ +#define __put_user_8(x, ptr, label) \ do { \ u32 __user *__ptr =3D (u32 __user *)(ptr); \ u64 __x =3D (__typeof__((x)-(x)))(x); \ - __asm__ __volatile__ ( \ + asm goto( \ "1:\n" \ - " sw %z1, %3\n" \ + " sw %z0, %2\n" \ "2:\n" \ - " sw %z2, %4\n" \ - "3:\n" \ - _ASM_EXTABLE_UACCESS_ERR(1b, 3b, %0) \ - _ASM_EXTABLE_UACCESS_ERR(2b, 3b, %0) \ - : "+r" (err) \ - : "rJ" (__x), "rJ" (__x >> 32), \ + " sw %z1, %3\n" \ + _ASM_EXTABLE(1b, %l4) \ + _ASM_EXTABLE(2b, %l4) \ + : : "rJ" (__x), "rJ" (__x >> 32), \ "m" (__ptr[__LSW]), \ - "m" (__ptr[__MSW])); \ + "m" (__ptr[__MSW]) : : label); \ } while (0) #endif /* CONFIG_64BIT */ =20 -#define __put_user_nocheck(x, __gu_ptr, __pu_err) \ +#define __put_user_nocheck(x, __gu_ptr, label) \ do { \ switch (sizeof(*__gu_ptr)) { \ case 1: \ - __put_user_asm("sb", (x), __gu_ptr, __pu_err); \ + __put_user_asm("sb", (x), __gu_ptr, label); \ break; \ case 2: \ - __put_user_asm("sh", (x), __gu_ptr, __pu_err); \ + __put_user_asm("sh", (x), __gu_ptr, label); \ break; \ case 4: \ - __put_user_asm("sw", (x), __gu_ptr, __pu_err); \ + __put_user_asm("sw", (x), __gu_ptr, label); \ break; \ case 8: \ - __put_user_8((x), __gu_ptr, __pu_err); \ + __put_user_8((x), __gu_ptr, label); \ break; \ default: \ BUILD_BUG(); \ } \ } while (0) =20 +#define __put_user_error(x, ptr, err) \ +do { \ + __label__ err_label; \ + __put_user_nocheck(x, ptr, err_label); \ + break; \ +err_label: \ + (err) =3D -EFAULT; \ +} while (0) + /** * __put_user: - Write a simple value into user space, with less checking. * @x: Value to copy to user space. @@ -299,7 +304,7 @@ do { \ __chk_user_ptr(__gu_ptr); \ \ __enable_user_access(); \ - __put_user_nocheck(__val, __gu_ptr, __pu_err); \ + __put_user_error(__val, __gu_ptr, __pu_err); \ __disable_user_access(); \ \ __pu_err; \ @@ -373,13 +378,7 @@ do { \ } while (0) =20 #define __put_kernel_nofault(dst, src, type, err_label) \ -do { \ - long __kr_err =3D 0; \ - \ - __put_user_nocheck(*((type *)(src)), (type *)(dst), __kr_err); \ - if (unlikely(__kr_err)) \ - goto err_label; \ -} while (0) + __put_user_nocheck(*((type *)(src)), (type *)(dst), err_label) =20 static __must_check __always_inline bool user_access_begin(const void __us= er *ptr, size_t len) { @@ -398,12 +397,8 @@ static inline void user_access_restore(unsigned long e= nabled) { } * We want the unsafe accessors to always be inlined and use * the error labels - thus the macro games. */ -#define unsafe_put_user(x, ptr, label) do { \ - long __err =3D 0; \ - __put_user_nocheck(x, (ptr), __err); \ - if (__err) \ - goto label; \ -} while (0) +#define unsafe_put_user(x, ptr, label) \ + __put_user_nocheck(x, (ptr), label) =20 #define unsafe_get_user(x, ptr, label) do { \ long __err =3D 0; \ --=20 2.34.1 From nobody Tue Dec 16 19:46:20 2025 Received: from mail-oi1-f177.google.com (mail-oi1-f177.google.com [209.85.167.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6686422AE7F for ; Thu, 20 Mar 2025 22:44:33 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.167.177 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1742510675; cv=none; b=BafX7FW8yT5v9ZGHcnqInMD2bUyTOCP8ffWF/MiPCiD+9zAA1V2nijLt68/vjYojzqI6dUcgrRTPfzZrz0PIxKmJ/iIzDGN2MSGGdIwZD2Uz36JJRb7QCOmFnLiVtxIUP1+6dYYBjBRAFx0CmsdtDCzLxaYDG9QFMrBaedyo4A8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1742510675; c=relaxed/simple; bh=/4Jn1Ns5m8J/PS8HTpLyGbF9ChF8Fk4/RvIEUizoI+k=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=NXeW9k53QdWYorqs58dXhqEwrJm3UUfupje5uOYt8dJzKnIlh/c4FXDSsYj124zL2fpbfLl81OUJyWJi1Tadm5G/vSQS5YJNO4Lj32vek/TIuXUhopPPowPXXEM+PxUrJMbtfLGg6aa4cKhgD8L1+GxpYNHFL19wEpsTq3HII/Q= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=tenstorrent.com; spf=pass smtp.mailfrom=tenstorrent.com; dkim=pass (2048-bit key) header.d=tenstorrent.com header.i=@tenstorrent.com header.b=DeHttRIq; arc=none smtp.client-ip=209.85.167.177 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=tenstorrent.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=tenstorrent.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=tenstorrent.com header.i=@tenstorrent.com header.b="DeHttRIq" Received: by mail-oi1-f177.google.com with SMTP id 5614622812f47-3fa0eb29cebso1475242b6e.0 for ; Thu, 20 Mar 2025 15:44:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=tenstorrent.com; s=google; t=1742510672; x=1743115472; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=aUM4cL2icpLrYe67mQWolXHTOhfxBr7CeU0SRh8tC7k=; b=DeHttRIqzYg1KebH4c5Rc2gr4ur2MPIAk3PXJCaXLfhIucGyDvriscnkhzWZAuw7g0 FvJtIcGCPsglo98tlWo+1UiOBce9r9w+GiWHPR5agnLpvUtP+xYgUyqvAIt1FooQkaPu DfcHpRyw6wC8vgHAbIYqrET/IIVZG2LUC6XDxcJRky5SZJGRmaUJWofXpnbaKSE4xDY3 PHDx745WVh5ezJ86tx+1zhLyTBgnLNdNEJ20X1jSkC783+5+gvuBcYTDKndIrXNE1wKT YKgXfb6IAeaHPpRHqiTiajCvp6INO0QiYE4B1UipDTIwsy9LtG7wWiLqo7M63s1TzGTn MYxA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1742510672; x=1743115472; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=aUM4cL2icpLrYe67mQWolXHTOhfxBr7CeU0SRh8tC7k=; b=GvUj/sSizhJD5Ba1loTbuS7i6f8W+uHh3XXhcYvfmoU2xCgVMPZ0uhSdV/CfSnehUN Q1jM7WcEbOK+/PE0RgdxFB8hvVtVI1+bj3Jm9hPLckNR8BRrX4Fx9yO9WfB/cMHmj9Mx U0+AzTgYYXwykVULGMNsILyMUE56edrDmkAQI4MBSLFE2RS7luYZ0NMMVSoVsKCi7ELM Z2ektV4HGEfKJvnrDPcooHEUkojsRs2PM3N9vF2TUNeZNZCXSYyK6c6EgImcWNZaFioZ h8slx1By5mnMmeBbA7TLxT0Np//tvL846loigteqV3jn00qyVPmKFiK1nhTqasgixpic J0hw== X-Forwarded-Encrypted: i=1; AJvYcCVGyOaRKpdrLnGRwjQTdtsd8cDmJTuKzzakC2n9a60z8OXopMF48q1fCvujgULu1YrdI3sqzZBq8nXitik=@vger.kernel.org X-Gm-Message-State: AOJu0Yw/oCCgAbV1/GplxZaDwXRV6YCF1qMuxROFLKrgjGgjJbXsLhGP uu1r7FbpFaHWQ0q/0d1qX17N/CHJaM2DyVzI19xNYKUn41jW0jonm4loxIh12g== X-Gm-Gg: ASbGncvkjAPlHpQi6p3MC4qVH42K5LPxNzcnclYgK5ER8Fdaq+EKaiUMq3YtoEznfFB K1wISDZsbjH9eGhHPB8K0F8L0Z+FQMJF3WYuAmvYG8Ko6c3EN/Gte0jzZfLX223wrPKL+dB6JK9 SNLL6+LMbMKTDnbo5/rfoe/DHDKH0mFyAG7V3wcypS+iRXOJC4+bxvYWGvLrQqr+uVPeYML0gVO KPLbHQPsr8wEUlM71NqxHTMI7N2RkcZGs0WfEQ3QcD6W8zxmp01cB+a8M3CaV+7eUxKadRZ37UT NSzPQ5p4riciVr2QArWLr/beUx5XSsrKQuvvBhtVvx6+atg2yWsXHR2/VVkzpCs= X-Google-Smtp-Source: AGHT+IHChpwGgXIo7oPDkSwy3Z8aUFaMgcTSyEt/gVcKCBKfeFe7Livn0EfqfYGjYKQnsyWXdaZjLg== X-Received: by 2002:a05:6808:444b:b0:3fa:10b4:698c with SMTP id 5614622812f47-3febeea9cf6mr1033217b6e.17.1742510672433; Thu, 20 Mar 2025 15:44:32 -0700 (PDT) Received: from aus-ird.tenstorrent.com ([38.104.49.66]) by smtp.gmail.com with ESMTPSA id 5614622812f47-3febf6dcc09sm103524b6e.12.2025.03.20.15.44.31 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 20 Mar 2025 15:44:31 -0700 (PDT) From: Cyril Bur To: palmer@dabbelt.com, aou@eecs.berkeley.edu, paul.walmsley@sifive.com, charlie@rivosinc.com, jrtc27@jrtc27.com, ben.dooks@codethink.co.uk, alex@ghiti.fr Cc: linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org, jszhang@kernel.org Subject: [PATCH v5 5/5] riscv: uaccess: use 'asm_goto_output' for get_user() Date: Thu, 20 Mar 2025 22:44:23 +0000 Message-Id: <20250320224423.1838493-6-cyrilbur@tenstorrent.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20250320224423.1838493-1-cyrilbur@tenstorrent.com> References: <20250320224423.1838493-1-cyrilbur@tenstorrent.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Jisheng Zhang With 'asm goto' we don't need to test the error etc, the exception just jumps to the error handling directly. Unlike put_user(), get_user() must work around GCC bugs [1] when using output clobbers in an asm goto statement. Link: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=3D113921 # 1 Signed-off-by: Jisheng Zhang [Cyril Bur: Rewritten commit message] Signed-off-by: Cyril Bur --- arch/riscv/include/asm/uaccess.h | 95 +++++++++++++++++++++++--------- 1 file changed, 68 insertions(+), 27 deletions(-) diff --git a/arch/riscv/include/asm/uaccess.h b/arch/riscv/include/asm/uacc= ess.h index 719c9179a751..8823471b201f 100644 --- a/arch/riscv/include/asm/uaccess.h +++ b/arch/riscv/include/asm/uaccess.h @@ -96,27 +96,58 @@ static inline unsigned long __untagged_addr_remote(stru= ct mm_struct *mm, unsigne * call. */ =20 -#define __get_user_asm(insn, x, ptr, err) \ +#ifdef CONFIG_CC_HAS_ASM_GOTO_OUTPUT +#define __get_user_asm(insn, x, ptr, label) \ + asm_goto_output( \ + "1:\n" \ + " " insn " %0, %1\n" \ + _ASM_EXTABLE_UACCESS_ERR(1b, %l2, %0) \ + : "=3D&r" (x) \ + : "m" (*(ptr)) : : label) +#else /* !CONFIG_CC_HAS_ASM_GOTO_OUTPUT */ +#define __get_user_asm(insn, x, ptr, label) \ do { \ - __typeof__(x) __x; \ + long __gua_err =3D 0; \ __asm__ __volatile__ ( \ "1:\n" \ " " insn " %1, %2\n" \ "2:\n" \ _ASM_EXTABLE_UACCESS_ERR_ZERO(1b, 2b, %0, %1) \ - : "+r" (err), "=3D&r" (__x) \ + : "+r" (__gua_err), "=3D&r" (x) \ : "m" (*(ptr))); \ - (x) =3D __x; \ + if (__gua_err) \ + goto label; \ } while (0) +#endif /* CONFIG_CC_HAS_ASM_GOTO_OUTPUT */ =20 #ifdef CONFIG_64BIT -#define __get_user_8(x, ptr, err) \ - __get_user_asm("ld", x, ptr, err) +#define __get_user_8(x, ptr, label) \ + __get_user_asm("ld", x, ptr, label) #else /* !CONFIG_64BIT */ -#define __get_user_8(x, ptr, err) \ + +#ifdef CONFIG_CC_HAS_ASM_GOTO_OUTPUT +#define __get_user_8(x, ptr, label) \ + u32 __user *__ptr =3D (u32 __user *)(ptr); \ + u32 __lo, __hi; \ + asm_goto_output( \ + "1:\n" \ + " lw %0, %2\n" \ + "2:\n" \ + " lw %1, %3\n" \ + _ASM_EXTABLE_UACCESS_ERR(1b, %l4, %0) \ + _ASM_EXTABLE_UACCESS_ERR(2b, %l4, %0) \ + : "=3D&r" (__lo), "=3Dr" (__hi) \ + : "m" (__ptr[__LSW]), "m" (__ptr[__MSW]) \ + : : label) \ + (x) =3D (__typeof__(x))((__typeof__((x) - (x)))( \ + (((u64)__hi << 32) | __lo))); \ + +#else /* !CONFIG_CC_HAS_ASM_GOTO_OUTPUT */ +#define __get_user_8(x, ptr, label) \ do { \ u32 __user *__ptr =3D (u32 __user *)(ptr); \ u32 __lo, __hi; \ + long __gu8_err =3D 0; \ __asm__ __volatile__ ( \ "1:\n" \ " lw %1, %3\n" \ @@ -125,35 +156,51 @@ do { \ "3:\n" \ _ASM_EXTABLE_UACCESS_ERR_ZERO(1b, 3b, %0, %1) \ _ASM_EXTABLE_UACCESS_ERR_ZERO(2b, 3b, %0, %1) \ - : "+r" (err), "=3D&r" (__lo), "=3Dr" (__hi) \ + : "+r" (__gu8_err), "=3D&r" (__lo), "=3Dr" (__hi) \ : "m" (__ptr[__LSW]), "m" (__ptr[__MSW])); \ - if (err) \ + if (__gu8_err) { \ __hi =3D 0; \ - (x) =3D (__typeof__(x))((__typeof__((x)-(x)))( \ + goto label; \ + } \ + (x) =3D (__typeof__(x))((__typeof__((x) - (x)))( \ (((u64)__hi << 32) | __lo))); \ } while (0) +#endif /* CONFIG_CC_HAS_ASM_GOTO_OUTPUT */ + #endif /* CONFIG_64BIT */ =20 -#define __get_user_nocheck(x, __gu_ptr, __gu_err) \ +#define __get_user_nocheck(x, __gu_ptr, label) \ do { \ switch (sizeof(*__gu_ptr)) { \ case 1: \ - __get_user_asm("lb", (x), __gu_ptr, __gu_err); \ + __get_user_asm("lb", (x), __gu_ptr, label); \ break; \ case 2: \ - __get_user_asm("lh", (x), __gu_ptr, __gu_err); \ + __get_user_asm("lh", (x), __gu_ptr, label); \ break; \ case 4: \ - __get_user_asm("lw", (x), __gu_ptr, __gu_err); \ + __get_user_asm("lw", (x), __gu_ptr, label); \ break; \ case 8: \ - __get_user_8((x), __gu_ptr, __gu_err); \ + __get_user_8((x), __gu_ptr, label); \ break; \ default: \ BUILD_BUG(); \ } \ } while (0) =20 +#define __get_user_error(x, ptr, err) \ +do { \ + __label__ __gu_failed; \ + \ + __get_user_nocheck(x, ptr, __gu_failed); \ + err =3D 0; \ + break; \ +__gu_failed: \ + x =3D 0; \ + err =3D -EFAULT; \ +} while (0) + /** * __get_user: - Get a simple variable from user space, with less checking. * @x: Variable to store result. @@ -178,13 +225,16 @@ do { \ ({ \ const __typeof__(*(ptr)) __user *__gu_ptr =3D untagged_addr(ptr); \ long __gu_err =3D 0; \ + __typeof__(x) __gu_val; \ \ __chk_user_ptr(__gu_ptr); \ \ __enable_user_access(); \ - __get_user_nocheck(x, __gu_ptr, __gu_err); \ + __get_user_error(__gu_val, __gu_ptr, __gu_err); \ __disable_user_access(); \ \ + (x) =3D __gu_val; \ + \ __gu_err; \ }) =20 @@ -369,13 +419,7 @@ unsigned long __must_check clear_user(void __user *to,= unsigned long n) } =20 #define __get_kernel_nofault(dst, src, type, err_label) \ -do { \ - long __kr_err =3D 0; \ - \ - __get_user_nocheck(*((type *)(dst)), (type *)(src), __kr_err); \ - if (unlikely(__kr_err)) \ - goto err_label; \ -} while (0) + __get_user_nocheck(*((type *)(dst)), (type *)(src), err_label) =20 #define __put_kernel_nofault(dst, src, type, err_label) \ __put_user_nocheck(*((type *)(src)), (type *)(dst), err_label) @@ -401,12 +445,9 @@ static inline void user_access_restore(unsigned long e= nabled) { } __put_user_nocheck(x, (ptr), label) =20 #define unsafe_get_user(x, ptr, label) do { \ - long __err =3D 0; \ __inttype(*(ptr)) __gu_val; \ - __get_user_nocheck(__gu_val, (ptr), __err); \ + __get_user_nocheck(__gu_val, (ptr), label); \ (x) =3D (__force __typeof__(*(ptr)))__gu_val; \ - if (__err) \ - goto label; \ } while (0) =20 #define unsafe_copy_loop(dst, src, len, type, op, label) \ --=20 2.34.1