From nobody Wed Dec 17 12:12:29 2025 Received: from mail-ot1-f48.google.com (mail-ot1-f48.google.com [209.85.210.48]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E5845191F72 for ; Tue, 18 Mar 2025 06:15:18 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.48 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1742278521; cv=none; b=ZU8MEFgQerHqk0y/zxy8cFwNO3SdnbnqHqkHCJfvTSiZSWuwr5OJiB0sDrgni+3s/LTsXWwJdY3oayeRuzneUo3rjIeign8z3FKqiZ5KW3Ar1U5923HTjGkSGeDHXH2Sm5p1tHKTU91vijrhkg9Z9FeRFB/aw+SLUQ4+PnG1gJ8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1742278521; c=relaxed/simple; bh=bvsaQf+q5zAPyQMLZEKjtaO7Zy1Z8ho7KoK1wQhVqTU=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=m+Ao8jSdJ6Ohcwsi5qrmujk8KCTsxgyPKvxh+iLoI8hfK7uesN8/0/KIw97CItvhJXFHDBjrMlz/WJnaGZOFDXVJl/8vcV2hvcu0ixBaFKUV+PO6+o9OUXhDjITK+0y3ihNaLVieShJ5JZ4HLUNz4YJWd+xjuniMjD0yzaqdIMg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=tenstorrent.com; spf=pass smtp.mailfrom=tenstorrent.com; dkim=pass (2048-bit key) header.d=tenstorrent.com header.i=@tenstorrent.com header.b=brKYd8M0; arc=none smtp.client-ip=209.85.210.48 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=tenstorrent.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=tenstorrent.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=tenstorrent.com header.i=@tenstorrent.com header.b="brKYd8M0" Received: by mail-ot1-f48.google.com with SMTP id 46e09a7af769-726819aa3fcso3235314a34.0 for ; Mon, 17 Mar 2025 23:15:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=tenstorrent.com; s=google; t=1742278518; x=1742883318; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=AMSpj4Pv/tuKqFfvndPeG7a41TGSXjAKvjZxbUe88U8=; b=brKYd8M0wtIWM34V/R7oY3j/PqonDvXIEfc8Q+7lQPC+d7WJN7uASqrUzieXgozyhJ Sb6tpDaSmq5JVSrYI+nyIF22DUtF3KAopDUP274kByuiaRZKXgkgd1HDimCrxGNZTTmi QX/vMWCPz0zK6NuIX2MEYL8/aDPVsDPIU3KEHMnNStlGdze4ufO7w48hx3kSD+a0Drro NI9LiLHD5d5mNU3aH9MRCazvBSMj4EjTp1tUI3yW6WY3GWw/UeAMvV9OM0Fc+3hspUgn FQmtGviBB72qSOGrJ+1kjTvqaLhc0Xd7NImt2LGOJrkpoIVLGqlZxVkNaju1oYng8tus ydFQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1742278518; x=1742883318; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=AMSpj4Pv/tuKqFfvndPeG7a41TGSXjAKvjZxbUe88U8=; b=TU3Me62kZfcTAbDjVtaEa78Y2LgPRiv4VRm+O0q9D/5IISsqeeAQmQdoGNXDQJSiPV LMafnrfLQmbp0ulK7KHijsCSnGRnYMZOn14exU0qqOYMUrht1jex0GP6n+VIpEeDpM3P 2NQFVuwdaNFE02Sgi5BGbBrn1C7a3dTQyFYfdQIWjALkVWQqN8AbNT13dcwJGAzHQjS/ zk9UDMhuJYAwEb0iVr241zcqZBG2pn+MKK+u2t5XvsZaD6vaTce4sMRAfcA6h0H9IXJr zQ/LAu9lqUJh3n28woqCWGvNmoJjp4x5JxsyUUKvlop1Z+dWWDhpdp3zCqJ7e+MEVPm+ 5mDQ== X-Forwarded-Encrypted: i=1; AJvYcCUxDoxDqw+QimhHpI48fozbYelFnR5v32Rl5QCvyW3FMEvneuS5Av0Z6mvVqlNmKOD55TymYkU1tXr7+3s=@vger.kernel.org X-Gm-Message-State: AOJu0Yx98vvGQ5ef6rIHcRxIgjkTOf802DPKVTntxIZs1ZdV0ZNNmrf8 RhwDRUvR1pDin+NnOQRLmsm7XGPuc5hl/mXl6vHg/GbZszXtH8bQJZmNFfrH1g== X-Gm-Gg: ASbGnctXGFPDwh4Q+G6/0yReP7LODFVlq9Zi2E1gWz36FgSzUb8iOEG2Ull41lqLMyO 9Msu7NKN5XfxOGt/HJ0dwbaCDq2KKKki7oSfEbhtkHXWzz1tVDEo68T9rbpo7MYPne/ET5D2vJq qXNbJyA4kxZuVfdCpxqLnBZqW0vtpJbdVZvVf816ulTdthXwafNmk/Lv8hUhsux0UypOELLohzc HLE3SIwYELYC5qh7uWwlzWqtsPUqq4ORgul97SdTvb97GBcOy/XrJzDTigJZkiXZbvygwWEIYSQ xxBkmYLfsj+5KYD9QVasJN9PRNPxPtQjky0L3yP+aPmkxnOBKNKWwD3TrXwsGHJDRfEdz4vHGw= = X-Google-Smtp-Source: AGHT+IE9dCEHhNSEUdS36yalwDgnQ6lSUtajzuROrLjALRXZFS2tcFhxjwVecmB7upfnJriqyJFUDw== X-Received: by 2002:a05:6830:6516:b0:72b:87bd:ad5b with SMTP id 46e09a7af769-72bbc247edcmr8104772a34.4.1742278517914; Mon, 17 Mar 2025 23:15:17 -0700 (PDT) Received: from aus-ird.tenstorrent.com ([38.104.49.66]) by smtp.gmail.com with ESMTPSA id 5614622812f47-3fcd403b882sm2051642b6e.8.2025.03.17.23.15.16 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 17 Mar 2025 23:15:17 -0700 (PDT) From: Cyril Bur To: palmer@dabbelt.com, aou@eecs.berkeley.edu, paul.walmsley@sifive.com, charlie@rivosinc.com, jrtc27@jrtc27.com, ben.dooks@codethink.co.uk, alex@ghiti.fr Cc: linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org, jszhang@kernel.org, syzbot+e74b94fe601ab9552d69@syzkaller.appspotmail.com Subject: [PATCH v4 1/5] riscv: save the SR_SUM status over switches Date: Tue, 18 Mar 2025 06:15:10 +0000 Message-Id: <20250318061514.1223111-2-cyrilbur@tenstorrent.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20250318061514.1223111-1-cyrilbur@tenstorrent.com> References: <20250318061514.1223111-1-cyrilbur@tenstorrent.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Ben Dooks When threads/tasks are switched we need to ensure the old execution's SR_SUM state is saved and the new thread has the old SR_SUM state restored. The issue is seen under heavy load especially with the syz-stress tool running, with crashes as follows in schedule_tail: Unable to handle kernel access to user memory without uaccess routines at virtual address 000000002749f0d0 Oops [#1] Modules linked in: CPU: 1 PID: 4875 Comm: syz-executor.0 Not tainted 5.12.0-rc2-syzkaller-00467-g0d7588ab9ef9 #0 Hardware name: riscv-virtio,qemu (DT) epc : schedule_tail+0x72/0xb2 kernel/sched/core.c:4264 ra : task_pid_vnr include/linux/sched.h:1421 [inline] ra : schedule_tail+0x70/0xb2 kernel/sched/core.c:4264 epc : ffffffe00008c8b0 ra : ffffffe00008c8ae sp : ffffffe025d17ec0 gp : ffffffe005d25378 tp : ffffffe00f0d0000 t0 : 0000000000000000 t1 : 0000000000000001 t2 : 00000000000f4240 s0 : ffffffe025d17ee0 s1 : 000000002749f0d0 a0 : 000000000000002a a1 : 0000000000000003 a2 : 1ffffffc0cfac500 a3 : ffffffe0000c80cc a4 : 5ae9db91c19bbe00 a5 : 0000000000000000 a6 : 0000000000f00000 a7 : ffffffe000082eba s2 : 0000000000040000 s3 : ffffffe00eef96c0 s4 : ffffffe022c77fe0 s5 : 0000000000004000 s6 : ffffffe067d74e00 s7 : ffffffe067d74850 s8 : ffffffe067d73e18 s9 : ffffffe067d74e00 s10: ffffffe00eef96e8 s11: 000000ae6cdf8368 t3 : 5ae9db91c19bbe00 t4 : ffffffc4043cafb2 t5 : ffffffc4043cafba t6 : 0000000000040000 status: 0000000000000120 badaddr: 000000002749f0d0 cause: 000000000000000f Call Trace: [] schedule_tail+0x72/0xb2 kernel/sched/core.c:4264 [] ret_from_exception+0x0/0x14 Dumping ftrace buffer: (ftrace buffer empty) Reported-by: syzbot+e74b94fe601ab9552d69@syzkaller.appspotmail.com ---[ end trace b5f8f9231dc87dda ]--- The issue comes from the put_user() in schedule_tail (kernel/sched/core.c) doing the following: asmlinkage __visible void schedule_tail(struct task_struct *prev) { ... if (current->set_child_tid) put_user(task_pid_vnr(current), current->set_child_tid); ... } the put_user() macro causes the code sequence to come out as follows: 1: __enable_user_access() 2: reg =3D task_pid_vnr(current); 3: *current->set_child_tid =3D reg; 4: __disable_user_access() This means the task_pid_vnr() is being called with user-access enabled which itself is not a good idea, but that is a separate issue. Here we have a function that /might/ sleep being called with the SR_SUM and if it does, then it returns with the SR_SUM flag possibly cleared thus causing the above abort. To try and deal with this, and stop the SR_SUM leaking out into other threads (this has also been tested and see under stress. It can rarely happen but it /does/ under load) make sure the __switch_to() will save and restore the SR_SUM flag, and clear it possibly for the next thread if it does not need it. Note, test code to be supplied once other checks have been finished. There may be further issues with the mstatus flags with this, this can be discussed further once some initial testing has been done. Reported-by: syzbot+e74b94fe601ab9552d69@syzkaller.appspotmail.com Signed-off-by: Ben Dooks Signed-off-by: Cyril Bur --- arch/riscv/include/asm/processor.h | 1 + arch/riscv/kernel/asm-offsets.c | 5 +++++ arch/riscv/kernel/entry.S | 8 ++++++++ 3 files changed, 14 insertions(+) diff --git a/arch/riscv/include/asm/processor.h b/arch/riscv/include/asm/pr= ocessor.h index 5f56eb9d114a..0de05d652e0f 100644 --- a/arch/riscv/include/asm/processor.h +++ b/arch/riscv/include/asm/processor.h @@ -103,6 +103,7 @@ struct thread_struct { struct __riscv_d_ext_state fstate; unsigned long bad_cause; unsigned long envcfg; + unsigned long flags; u32 riscv_v_flags; u32 vstate_ctrl; struct __riscv_v_ext_state vstate; diff --git a/arch/riscv/kernel/asm-offsets.c b/arch/riscv/kernel/asm-offset= s.c index e89455a6a0e5..556ebcbb7e22 100644 --- a/arch/riscv/kernel/asm-offsets.c +++ b/arch/riscv/kernel/asm-offsets.c @@ -34,6 +34,7 @@ void asm_offsets(void) OFFSET(TASK_THREAD_S9, task_struct, thread.s[9]); OFFSET(TASK_THREAD_S10, task_struct, thread.s[10]); OFFSET(TASK_THREAD_S11, task_struct, thread.s[11]); + OFFSET(TASK_THREAD_FLAGS, task_struct, thread.flags); =20 OFFSET(TASK_TI_CPU, task_struct, thread_info.cpu); OFFSET(TASK_TI_FLAGS, task_struct, thread_info.flags); @@ -347,6 +348,10 @@ void asm_offsets(void) offsetof(struct task_struct, thread.s[11]) - offsetof(struct task_struct, thread.ra) ); + DEFINE(TASK_THREAD_FLAGS_RA, + offsetof(struct task_struct, thread.flags) + - offsetof(struct task_struct, thread.ra) + ); =20 DEFINE(TASK_THREAD_F0_F0, offsetof(struct task_struct, thread.fstate.f[0]) diff --git a/arch/riscv/kernel/entry.S b/arch/riscv/kernel/entry.S index 33a5a9f2a0d4..c278b3ac37b9 100644 --- a/arch/riscv/kernel/entry.S +++ b/arch/riscv/kernel/entry.S @@ -397,9 +397,17 @@ SYM_FUNC_START(__switch_to) REG_S s9, TASK_THREAD_S9_RA(a3) REG_S s10, TASK_THREAD_S10_RA(a3) REG_S s11, TASK_THREAD_S11_RA(a3) + + /* save (and disable the user space access flag) */ + li s0, SR_SUM + csrrc s1, CSR_STATUS, s0 + REG_S s1, TASK_THREAD_FLAGS_RA(a3) + /* Save the kernel shadow call stack pointer */ scs_save_current /* Restore context from next->thread */ + REG_L s0, TASK_THREAD_FLAGS_RA(a4) + csrs CSR_STATUS, s0 REG_L ra, TASK_THREAD_RA_RA(a4) REG_L sp, TASK_THREAD_SP_RA(a4) REG_L s0, TASK_THREAD_S0_RA(a4) --=20 2.34.1 From nobody Wed Dec 17 12:12:29 2025 Received: from mail-oi1-f182.google.com (mail-oi1-f182.google.com [209.85.167.182]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1919919B3EE for ; Tue, 18 Mar 2025 06:15:19 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.167.182 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1742278521; cv=none; b=qWuVl5lCZREB43NjJdjLBDL0akgSt52F80NVNGgjJeGUDJTmC8VYunBmU6aj/vVqQCuyaE37iCOj2Amt/Fcb0du15o5742ZgKFxi/STHaa5xGQX8gg0t2e5XW8TD6VEXa9yzmxyCqOzrHKE7HiLJBok9xlE92OO+u7gt3dcMZP0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1742278521; c=relaxed/simple; bh=OyHO91umNeTvoAyNEoyZRtv9B+u3OndEt7OlSnX3SIs=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=M2IfIgEgS7MJCyEHTYeRt632UTU6Lg7/VU8iEbz5ucMyRIOpk1nn7Fv/diIWYr6j3WZLc1wJzjfYGzfShcnuvjLa0VCywUqwbJA+cvmmy1e3IPTRgnjn/FMDN4P6DPJjuscYY2j9V3v0r2RvYTXE0bjFEGG/Uh4Wlpw+XFR23C0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=tenstorrent.com; spf=pass smtp.mailfrom=tenstorrent.com; dkim=pass (2048-bit key) header.d=tenstorrent.com header.i=@tenstorrent.com header.b=O5CQAGiB; arc=none smtp.client-ip=209.85.167.182 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=tenstorrent.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=tenstorrent.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=tenstorrent.com header.i=@tenstorrent.com header.b="O5CQAGiB" Received: by mail-oi1-f182.google.com with SMTP id 5614622812f47-3f7f7b70aebso1990490b6e.2 for ; Mon, 17 Mar 2025 23:15:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=tenstorrent.com; s=google; t=1742278519; x=1742883319; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=6Rz7GbyOPq3wyNoMRzcZSnc3xOAtIOrgSgwyUJM88Lo=; b=O5CQAGiBkRIPY2xaRHQlBWo24oYqyxRLa6+4/2MEbxkQJuK61qM17oXO4el680kxUy sN7+v59wyC5Rh897bihJsTPo2tDCNyEaAy+sFEtu1qxjzX1v+I0uYKhYJiwlxbg660jT nMaxEO4cuHsJK++kGxiWsrP1tagxWnJ3T6ii8bjrXOLuf9OV0AsOulHa8xLu0JrsNTPS GUUnTSSlmnDo9Cry0VaYGijCjJmwHGEuNnf3kwUwGjGSCTI5CeSKdGQpjchOcANQlmj2 BFvdV2axSJx8Qu11LxfZ214Trj3rZaaUVm3yOaF7k2vMjAddZPHqOsmfRFx7IkZ2+AgB hrcQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1742278519; x=1742883319; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=6Rz7GbyOPq3wyNoMRzcZSnc3xOAtIOrgSgwyUJM88Lo=; b=rhIjf5hUHw02362sIqDx12rKJ2DQFDGeQ+UWl44Cwi7p4rfmua+n9bfPjeb/FiQW5j JAjh2huFdYyp6b2qbXXqotxEAmqfnXp+dKolEIs9f5sxDY1jPOAD8x15J2iaMwHvQeur TQR8DMPJlBY4lHnroAhkTkfDjzCLuuJb7v0fVrjkwyu4yCrJ7i7gPaYQBeT49SYH/gg7 4FczhJ6f6Xiq3QDZsqrP+O5E7y6D/7K/A8GfyjnBueZtOe0S6mbGKlbteOzeq/53h0Xt 9CedVBqE/i3PlA5ktr2v26nD05hVAyeOg1jQ5BHZ/hA+ZCGYzjV4+slEUqqTzBsPD4JG GaJQ== X-Forwarded-Encrypted: i=1; AJvYcCVvCUMf9c4LHD3rvNS3XLpTxaE56BeUyHnpp3K0JgiL4+nW88YKD1jaWO/Cw06nICYoC37CpNeWVdZr7aY=@vger.kernel.org X-Gm-Message-State: AOJu0Yz42VpVtpyX+MWu7JtcsQ5skT3VdPaThYdkhSXiGIxcVL3K+XUM nv4HBXIkfyp+KclIDxoR/kK3LlapsXOq1BbYwvZWuwQiRJqEMTmyaoRBtfiDSQ== X-Gm-Gg: ASbGncv999s9+X/xcgPCIWdBPIh2AAjLQKFfxxqCSVWNkvcZRRxI5efuLabu0yjEomF UECmN/0bhn4OUXf+tSA0PMx14lcjymaYcglu3tNXMaoWMwdftVq3qCzjZvHLq0rn+G8lOj5jlPk UG8n6xKEWpN6/H7hHlZ85cVevKREGVMvZIpvsNESnkwKw+MUyzYu3i774uG016zrLDz91pnlNl0 Zs/FH0Uwcq6fIjWymSbEwQgXgc7Tk1jR5sznfz9Zi1zBA0Y4idph4FhQOL4Rkx9tQsckilwdEnO iAouFKQUqCZUh/4KPmkvSj4Jql3LEp0Tl4RgmTauuwV0oO9tUoE8uhwVMr0Cybs= X-Google-Smtp-Source: AGHT+IHjTlUJ8udAjH/mkkQoHGmHBU7+KcdMu+hMBhRq6iTs6DPSX4yEs5XtvzKJa+RBjf2KhPoFlQ== X-Received: by 2002:a05:6808:3507:b0:3f8:587:dacb with SMTP id 5614622812f47-3fdee93092fmr8770534b6e.9.1742278519108; Mon, 17 Mar 2025 23:15:19 -0700 (PDT) Received: from aus-ird.tenstorrent.com ([38.104.49.66]) by smtp.gmail.com with ESMTPSA id 5614622812f47-3fcd403b882sm2051642b6e.8.2025.03.17.23.15.18 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 17 Mar 2025 23:15:18 -0700 (PDT) From: Cyril Bur To: palmer@dabbelt.com, aou@eecs.berkeley.edu, paul.walmsley@sifive.com, charlie@rivosinc.com, jrtc27@jrtc27.com, ben.dooks@codethink.co.uk, alex@ghiti.fr Cc: linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org, jszhang@kernel.org Subject: [PATCH v4 2/5] riscv: implement user_access_begin() and families Date: Tue, 18 Mar 2025 06:15:11 +0000 Message-Id: <20250318061514.1223111-3-cyrilbur@tenstorrent.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20250318061514.1223111-1-cyrilbur@tenstorrent.com> References: <20250318061514.1223111-1-cyrilbur@tenstorrent.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Jisheng Zhang Currently, when a function like strncpy_from_user() is called, the userspace access protection is disabled and enabled for every word read. By implementing user_access_begin() and families, the protection is disabled at the beginning of the copy and enabled at the end. The __inttype macro is borrowed from x86 implementation. Signed-off-by: Jisheng Zhang Signed-off-by: Cyril Bur --- arch/riscv/include/asm/uaccess.h | 76 ++++++++++++++++++++++++++++++++ 1 file changed, 76 insertions(+) diff --git a/arch/riscv/include/asm/uaccess.h b/arch/riscv/include/asm/uacc= ess.h index fee56b0c8058..d7fef81b7969 100644 --- a/arch/riscv/include/asm/uaccess.h +++ b/arch/riscv/include/asm/uaccess.h @@ -61,6 +61,19 @@ static inline unsigned long __untagged_addr_remote(struc= t mm_struct *mm, unsigne #define __disable_user_access() \ __asm__ __volatile__ ("csrc sstatus, %0" : : "r" (SR_SUM) : "memory") =20 +/* + * This is the smallest unsigned integer type that can fit a value + * (up to 'long long') + */ +#define __inttype(x) __typeof__( \ + __typefits(x, char, \ + __typefits(x, short, \ + __typefits(x, int, \ + __typefits(x, long, 0ULL))))) + +#define __typefits(x, type, not) \ + __builtin_choose_expr(sizeof(x) <=3D sizeof(type), (unsigned type)0, not) + /* * The exception table consists of pairs of addresses: the first is the * address of an instruction that is allowed to fault, and the second is @@ -368,6 +381,69 @@ do { \ goto err_label; \ } while (0) =20 +static __must_check __always_inline bool user_access_begin(const void __us= er *ptr, size_t len) +{ + if (unlikely(!access_ok(ptr, len))) + return 0; + __enable_user_access(); + return 1; +} +#define user_access_begin user_access_begin +#define user_access_end __disable_user_access + +static inline unsigned long user_access_save(void) { return 0UL; } +static inline void user_access_restore(unsigned long enabled) { } + +/* + * We want the unsafe accessors to always be inlined and use + * the error labels - thus the macro games. + */ +#define unsafe_put_user(x, ptr, label) do { \ + long __err =3D 0; \ + __put_user_nocheck(x, (ptr), __err); \ + if (__err) \ + goto label; \ +} while (0) + +#define unsafe_get_user(x, ptr, label) do { \ + long __err =3D 0; \ + __inttype(*(ptr)) __gu_val; \ + __get_user_nocheck(__gu_val, (ptr), __err); \ + (x) =3D (__force __typeof__(*(ptr)))__gu_val; \ + if (__err) \ + goto label; \ +} while (0) + +#define unsafe_copy_loop(dst, src, len, type, op, label) \ + while (len >=3D sizeof(type)) { \ + op(*(type *)(src), (type __user *)(dst), label); \ + dst +=3D sizeof(type); \ + src +=3D sizeof(type); \ + len -=3D sizeof(type); \ + } + +#define unsafe_copy_to_user(_dst, _src, _len, label) \ +do { \ + char __user *__ucu_dst =3D (_dst); \ + const char *__ucu_src =3D (_src); \ + size_t __ucu_len =3D (_len); \ + unsafe_copy_loop(__ucu_dst, __ucu_src, __ucu_len, u64, unsafe_get_user, l= abel); \ + unsafe_copy_loop(__ucu_dst, __ucu_src, __ucu_len, u32, unsafe_get_user, l= abel); \ + unsafe_copy_loop(__ucu_dst, __ucu_src, __ucu_len, u16, unsafe_get_user, l= abel); \ + unsafe_copy_loop(__ucu_dst, __ucu_src, __ucu_len, u8, unsafe_get_user, la= bel); \ +} while (0) + +#define unsafe_copy_from_user(_dst, _src, _len, label) \ +do { \ + char *__ucu_dst =3D (_dst); \ + const char __user *__ucu_src =3D (_src); \ + size_t __ucu_len =3D (_len); \ + unsafe_copy_loop(__ucu_dst, __ucu_src, __ucu_len, u64, unsafe_put_user, l= abel); \ + unsafe_copy_loop(__ucu_dst, __ucu_src, __ucu_len, u32, unsafe_put_user, l= abel); \ + unsafe_copy_loop(__ucu_dst, __ucu_src, __ucu_len, u16, unsafe_put_user, l= abel); \ + unsafe_copy_loop(__ucu_dst, __ucu_src, __ucu_len, u8, unsafe_put_user, la= bel); \ +} while (0) + #else /* CONFIG_MMU */ #include #endif /* CONFIG_MMU */ --=20 2.34.1 From nobody Wed Dec 17 12:12:29 2025 Received: from mail-oi1-f175.google.com (mail-oi1-f175.google.com [209.85.167.175]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A25A41D9A5D for ; Tue, 18 Mar 2025 06:15:21 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.167.175 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1742278523; cv=none; b=FyyvGn24qbsVDFeeI8C7aWNlRxd9VX5CqiwvYZvHB572HFeKpIrDWXEh6eoLp/690PfcN5KZniey/ts7RtsoIirNd+9Wa76CVwfJ6EYMAZNgkwoEwEIGXEXCWSJr+m3enTG+85vTUpHkNzH40hwsSal9w76VsTZNNO6ySz1vAmk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1742278523; c=relaxed/simple; bh=8+1/CTx5s8hPHAvMe5c1XoVfnjHA5kPPBw+5xNvfq30=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=NmDjb7QSFGO4l5n0gubPolSG3x2V9KRItymb5wZmRdu9zDLD9X/G2EIrIW0Lu7xTfI0rscSqqZ0h0RsQhlbvmvomdavul2xiwT1sReLDlv6BCKQPXIiqRpOJ6hfMWdcvj1/8TJ8NW5BXrwwqfzmzYoI/dWw5KTSg5uvCc5il0dU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=tenstorrent.com; spf=pass smtp.mailfrom=tenstorrent.com; dkim=pass (2048-bit key) header.d=tenstorrent.com header.i=@tenstorrent.com header.b=ET+iCnu2; arc=none smtp.client-ip=209.85.167.175 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=tenstorrent.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=tenstorrent.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=tenstorrent.com header.i=@tenstorrent.com header.b="ET+iCnu2" Received: by mail-oi1-f175.google.com with SMTP id 5614622812f47-3fea0363284so486655b6e.1 for ; Mon, 17 Mar 2025 23:15:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=tenstorrent.com; s=google; t=1742278520; x=1742883320; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=STOmRD1NGs9uSMt++nKAUKoj9Ow/rlWCVnbqWbWWLRY=; b=ET+iCnu2JhsauGSXc/nh6OcqKXvR3C/DZfFXvNFzbnD0/JnECj3nmHih6QYs2Alywx qGAKdPiX0lkScSz8wwe7wGeOl6SegokKKiHvr2W2PvJ5GJN/y+8V79nWRmO5goTAp877 wH5UDNCx69QpPSrpNxP6klERkvEnTgE+5hAHiR5Gd5AifQTCK7oRDWlXWqzv1LGqNTGN +MxLuhN+9fAH+q6BjsqSYgyy3pxgTFu2Gtu128q/zXHb/cPyOSeWTQfvTiEAejpGkARk mlXw7scND76moMakUq89OBXtb6e/mV0xNud97ZqtUWcQVQ9tWuxLDditj503XVlGsPOu zSUg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1742278520; x=1742883320; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=STOmRD1NGs9uSMt++nKAUKoj9Ow/rlWCVnbqWbWWLRY=; b=KV90ifA98uNb9HyEY6g84Yztn65iK6Hxzhj23oI1N7t468jkFR1ZKKclN0KJACnolM XcqrI63rPymOFFTptK2nmDBKCU5nghiKTba8/Jweh3hceTAlbO0QQQP1LNQToYgEJkLD nKjxqtDA6OwuONxS7wrWnXLrh2amCNWUmXwYkemDNKzJHHpEEm+k7SMJbBpLNWn/XA2n MDOtM9iU3bOygfWL2hXxSgK/lYn5Zz6jWLoJCi9oz9nu6I5hlWurZ4Q2+r+wYNpBuSa6 pgdtRiwQNrVoNxfCe2+LGB2vkpxWDMex5KkLhjOEhxNhbz9GcGxgbbX7Q67n6tiJ5Adi VHOA== X-Forwarded-Encrypted: i=1; AJvYcCWjHRGSgpd9zK007MJ94KZVK7uw5K7saiweW2duHjz5Hq0x4PuUUMHoNJdpS+Bv+eyaJsYHrJ6Zq+S1c0o=@vger.kernel.org X-Gm-Message-State: AOJu0YzRw8pclYPr2db9pfVm+deS0ovMakVbcvJ/su08KKImYdGBz9z1 v7dlQnMCmvpOprzDAUxFevVDeysrG5hcDpVeCGHWj8xNu56l7FAjGk5Z1TKbXQ== X-Gm-Gg: ASbGncvvI0n+L7cTX6uqVSshhq7ym5XCEj92OHMJ6OxXTIb61HhLCELnj+HhfxZ/M+P rc9UEdRyXB2H6CqBbCxUcNk8cK8ogsItyeCP/4FmyfxAy95AFPKULtVRXV5kSQnl6q+LPYKiid8 Uw+AJMO/jArrxgx4zDxdbEhTwoPl9Ml54HGRVPR+XcgrkWVuiC35iqE1zEeTl9WcJjVxAZv9k4d lH2xpXYV1og/jXV2y9XEwiOyzj+WONLRoU3Yjcpe6Yg3kUryxXQaAAXqJ09cJE7yFuytoVuBNdE DsoILQ3rI5Mug0757RZD3hZV/kWHB0Ogog8SJNL/MP0IWknfDYnpr5FRD/025Oc= X-Google-Smtp-Source: AGHT+IF+QEcnPvq7HOQUCyqTY14BOpN8Y8eX1PyixCC8ZH0S4NtzfP09P+KLDIccR2wqd6pOP+an5g== X-Received: by 2002:a05:6808:38c4:b0:3f8:18a3:b1cc with SMTP id 5614622812f47-3fea2482935mr1337534b6e.12.1742278520647; Mon, 17 Mar 2025 23:15:20 -0700 (PDT) Received: from aus-ird.tenstorrent.com ([38.104.49.66]) by smtp.gmail.com with ESMTPSA id 5614622812f47-3fcd403b882sm2051642b6e.8.2025.03.17.23.15.19 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 17 Mar 2025 23:15:19 -0700 (PDT) From: Cyril Bur To: palmer@dabbelt.com, aou@eecs.berkeley.edu, paul.walmsley@sifive.com, charlie@rivosinc.com, jrtc27@jrtc27.com, ben.dooks@codethink.co.uk, alex@ghiti.fr Cc: linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org, jszhang@kernel.org Subject: [PATCH v4 3/5] riscv: uaccess: use input constraints for ptr of __put_user() Date: Tue, 18 Mar 2025 06:15:12 +0000 Message-Id: <20250318061514.1223111-4-cyrilbur@tenstorrent.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20250318061514.1223111-1-cyrilbur@tenstorrent.com> References: <20250318061514.1223111-1-cyrilbur@tenstorrent.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Jisheng Zhang Putting ptr in the inputs as opposed to output may seem incorrect but this is done for a few reasons: - Not having it in the output permits the use of asm goto in a subsequent patch. There are bugs in gcc [1] which would otherwise prevent it. - Since the output memory is userspace there isn't any real benefit from telling the compiler about the memory clobber. - x86, arm and powerpc all use this technique. Link: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=3D113921 # 1 Signed-off-by: Jisheng Zhang [Cyril Bur: Rewritten commit message] Signed-off-by: Cyril Bur --- arch/riscv/include/asm/uaccess.h | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/arch/riscv/include/asm/uaccess.h b/arch/riscv/include/asm/uacc= ess.h index d7fef81b7969..180d6e21d5b5 100644 --- a/arch/riscv/include/asm/uaccess.h +++ b/arch/riscv/include/asm/uaccess.h @@ -219,11 +219,11 @@ do { \ __typeof__(*(ptr)) __x =3D x; \ __asm__ __volatile__ ( \ "1:\n" \ - " " insn " %z2, %1\n" \ + " " insn " %z1, %2\n" \ "2:\n" \ _ASM_EXTABLE_UACCESS_ERR(1b, 2b, %0) \ - : "+r" (err), "=3Dm" (*(ptr)) \ - : "rJ" (__x)); \ + : "+r" (err) \ + : "rJ" (__x), "m"(*(ptr))); \ } while (0) =20 #ifdef CONFIG_64BIT @@ -236,16 +236,16 @@ do { \ u64 __x =3D (__typeof__((x)-(x)))(x); \ __asm__ __volatile__ ( \ "1:\n" \ - " sw %z3, %1\n" \ + " sw %z1, %3\n" \ "2:\n" \ - " sw %z4, %2\n" \ + " sw %z2, %4\n" \ "3:\n" \ _ASM_EXTABLE_UACCESS_ERR(1b, 3b, %0) \ _ASM_EXTABLE_UACCESS_ERR(2b, 3b, %0) \ - : "+r" (err), \ - "=3Dm" (__ptr[__LSW]), \ - "=3Dm" (__ptr[__MSW]) \ - : "rJ" (__x), "rJ" (__x >> 32)); \ + : "+r" (err) \ + : "rJ" (__x), "rJ" (__x >> 32), \ + "m" (__ptr[__LSW]), \ + "m" (__ptr[__MSW])); \ } while (0) #endif /* CONFIG_64BIT */ =20 --=20 2.34.1 From nobody Wed Dec 17 12:12:29 2025 Received: from mail-oi1-f172.google.com (mail-oi1-f172.google.com [209.85.167.172]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1608E1DE89D for ; Tue, 18 Mar 2025 06:15:22 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.167.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1742278524; cv=none; b=sIx2CwPsgHG420n0cQOZfdfv6ofghHWQ1BFGglNy3ykMH/cnSqxPY84338eCekaGbhYRwE/iFkYVBfhKFBzstoMbvzoNxZARbM/IxslwIZ309uhWACpjAFxcj0Kqpm5rnWd3Ay0YfuCAkO1sZTIHdSfjfkohvIzK+9n54CWaBcw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1742278524; c=relaxed/simple; bh=9kSVO3zTG7zUffAvq6P8vHARSpZiIby583v5HPyggrg=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=Yb/aS1HHulhckwwCBW7WjP+DLJi8elb8/i4bzwmR5wYOEXqpmha6cucq4XaGnesl4L3kZI0exK9LVSsGagzT00Hwh+IL1NBBrmiYSjUb49R+U7+v9tf8pKnNJaX/HzG2SLCiAd6x/Sj1BlVikDDxSLY2VE6Sq1UpU1pt0+q/pAo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=tenstorrent.com; spf=pass smtp.mailfrom=tenstorrent.com; dkim=pass (2048-bit key) header.d=tenstorrent.com header.i=@tenstorrent.com header.b=X9thB2Q9; arc=none smtp.client-ip=209.85.167.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=tenstorrent.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=tenstorrent.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=tenstorrent.com header.i=@tenstorrent.com header.b="X9thB2Q9" Received: by mail-oi1-f172.google.com with SMTP id 5614622812f47-3f3da35555eso2727073b6e.1 for ; Mon, 17 Mar 2025 23:15:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=tenstorrent.com; s=google; t=1742278522; x=1742883322; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=HccPZpbZyX4ghGGLQ1GggeTzIynZ7PXEL+v/MCdb8aI=; b=X9thB2Q9W7E15werWcEEJAwZ0a4u3FXHWZWih+a6S37i7Dg49eUbdAYLwk9/0Hi3PR jrwXnKfsrOcHuhBBtQCMZU5qT/3fh+6w63IfmQiNkAMstPb0l03ZvHgGKJ+ESoRoFIiF Vxeu7O9RMwlX08iQCQZryjJ5Z/UtlV3tAzsqIETEfuQBDe0fuq3ymE4qPHr0yzTcGOi2 0+pT5OkOOqIu7D+jJhPzs0fc/xJgJ1s360CSHDHZLLoi6DGi0KZJxop4/QIMPrB0mP+k guPxE1HLTFWXIXzQS2mPTBvUtvV2SV2u7H+apg5E4OBeN1OO5BtIwlhyjtqenBWkXJHH /tBQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1742278522; x=1742883322; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=HccPZpbZyX4ghGGLQ1GggeTzIynZ7PXEL+v/MCdb8aI=; b=hjb5DORQyHBd+gTInMH9aYWBtO0sPVUYM5d8HHO1ABUW4jikt7Nb0c5/HziGiWHE2i 102teD7olf4J9jfhgJCz3nolnZyssp+DSil9g6nM4Wc3Z0W3RyYGnI1lYGjZDCsQsT48 4IAXh1P/6SSKl3BLo6dxQ3jOZPdHm2mwlf4O4seMuYC99dB3BnhzYVVvFybgnyFo7OUg Y4rRKfOkjTOhotW3JhwOT+fgLWjcj5KOh52DWuEoXKt0ckroarZxlHJSgV5+1OIrBEiE NCrMTrogq4ngeINdxT/3XoSIDLkevAYvKWbRLuW2fdKmhNuLHi5TmbTV0SsHwrng1C59 uqoQ== X-Forwarded-Encrypted: i=1; AJvYcCXOEJovc5tkIsOfmn9BcjNO0EIRo/EYs+QcZUgBtz5A6ztEwuAKJg5YOobfqqazDqNNrA+97ZXgoDoprzs=@vger.kernel.org X-Gm-Message-State: AOJu0Yxucxb7hOiE+9UW7aGqC9h/aMT+TL2M2xJf/4I7vniyCmWJCyGC 8PMhgRmGvIPdbzn0pyjINNL9zYX851lCidexMeV/+CllWDfKlnjcCz4ADxSaXw== X-Gm-Gg: ASbGncufNRRXKHb3jvNARYfSlOaydvKTy8IEg3DhiI2OCo+NnZuNlsfPFrWudRhUkZ6 nT7lCY7w7d/5F2JFlXCdrEqTXvWiJPE44tLTXU+nghunJQGbe/uC0u8duJvK/nVA6FgXuFWC7hP clfHI1WNq5+uBqvQ8jo4wssfW+6lBHyKCli4HJ6ay4QP08gNCflKJKDptjBVUaKgfH/pG74Ewp8 5MBDcxCs1LMc9GBVMyxfL6JEy5Z4QcGOLdaJKNTXxYashdTdQjJ4WhHWsGv5A51hbwZql6KtWV0 VDX36A5IVNsZDoF4rVl5QEtjBbOeJmg4MtvzZ9pbFhkbOPUa48EMu3us4nWi+28= X-Google-Smtp-Source: AGHT+IHEfvBRTZGysFXmW3RRQXGtzyNoB5Q4apt6oGZu+ZdRy6GfL5mJC1oNgLlW5pjKpJDDbIHUXw== X-Received: by 2002:a05:6808:201a:b0:3f6:7192:6aaf with SMTP id 5614622812f47-3fdeed0d9b8mr9357410b6e.22.1742278522205; Mon, 17 Mar 2025 23:15:22 -0700 (PDT) Received: from aus-ird.tenstorrent.com ([38.104.49.66]) by smtp.gmail.com with ESMTPSA id 5614622812f47-3fcd403b882sm2051642b6e.8.2025.03.17.23.15.20 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 17 Mar 2025 23:15:21 -0700 (PDT) From: Cyril Bur To: palmer@dabbelt.com, aou@eecs.berkeley.edu, paul.walmsley@sifive.com, charlie@rivosinc.com, jrtc27@jrtc27.com, ben.dooks@codethink.co.uk, alex@ghiti.fr Cc: linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org, jszhang@kernel.org Subject: [PATCH v4 4/5] riscv: uaccess: use 'asm goto' for put_user() Date: Tue, 18 Mar 2025 06:15:13 +0000 Message-Id: <20250318061514.1223111-5-cyrilbur@tenstorrent.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20250318061514.1223111-1-cyrilbur@tenstorrent.com> References: <20250318061514.1223111-1-cyrilbur@tenstorrent.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Jisheng Zhang With 'asm goto' we don't need to test the error etc, the exception just jumps to the error handling directly. Because there are no output clobbers which could trigger gcc bugs [1] the use of asm_goto_output() macro is not necessary here. Not using asm_goto_output() is desirable as the generated output asm will be cleaner. Use of the volatile keyword is redundant as per gcc 14.2.0 manual section 6.48.2.7 Goto Labels: > Also note that an asm goto statement is always implicitly considered volatile. Link: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=3D113921 # 1 Signed-off-by: Jisheng Zhang [Cyril Bur: Rewritten commit message] Signed-off-by: Cyril Bur --- arch/riscv/include/asm/uaccess.h | 71 +++++++++++++++----------------- 1 file changed, 33 insertions(+), 38 deletions(-) diff --git a/arch/riscv/include/asm/uaccess.h b/arch/riscv/include/asm/uacc= ess.h index 180d6e21d5b5..ab91dd2fa230 100644 --- a/arch/riscv/include/asm/uaccess.h +++ b/arch/riscv/include/asm/uaccess.h @@ -214,61 +214,66 @@ do { \ ((x) =3D (__force __typeof__(x))0, -EFAULT); \ }) =20 -#define __put_user_asm(insn, x, ptr, err) \ +#define __put_user_asm(insn, x, ptr, label) \ do { \ __typeof__(*(ptr)) __x =3D x; \ - __asm__ __volatile__ ( \ + asm goto( \ "1:\n" \ - " " insn " %z1, %2\n" \ - "2:\n" \ - _ASM_EXTABLE_UACCESS_ERR(1b, 2b, %0) \ - : "+r" (err) \ - : "rJ" (__x), "m"(*(ptr))); \ + " " insn " %z0, %1\n" \ + _ASM_EXTABLE(1b, %l2) \ + : : "rJ" (__x), "m"(*(ptr)) : : label); \ } while (0) =20 #ifdef CONFIG_64BIT -#define __put_user_8(x, ptr, err) \ - __put_user_asm("sd", x, ptr, err) +#define __put_user_8(x, ptr, label) \ + __put_user_asm("sd", x, ptr, label) #else /* !CONFIG_64BIT */ -#define __put_user_8(x, ptr, err) \ +#define __put_user_8(x, ptr, label) \ do { \ u32 __user *__ptr =3D (u32 __user *)(ptr); \ u64 __x =3D (__typeof__((x)-(x)))(x); \ - __asm__ __volatile__ ( \ + asm goto( \ "1:\n" \ - " sw %z1, %3\n" \ + " sw %z0, %2\n" \ "2:\n" \ - " sw %z2, %4\n" \ - "3:\n" \ - _ASM_EXTABLE_UACCESS_ERR(1b, 3b, %0) \ - _ASM_EXTABLE_UACCESS_ERR(2b, 3b, %0) \ - : "+r" (err) \ - : "rJ" (__x), "rJ" (__x >> 32), \ + " sw %z1, %3\n" \ + _ASM_EXTABLE(1b, %l4) \ + _ASM_EXTABLE(2b, %l4) \ + : : "rJ" (__x), "rJ" (__x >> 32), \ "m" (__ptr[__LSW]), \ - "m" (__ptr[__MSW])); \ + "m" (__ptr[__MSW]) : : label); \ } while (0) #endif /* CONFIG_64BIT */ =20 -#define __put_user_nocheck(x, __gu_ptr, __pu_err) \ +#define __put_user_nocheck(x, __gu_ptr, label) \ do { \ switch (sizeof(*__gu_ptr)) { \ case 1: \ - __put_user_asm("sb", (x), __gu_ptr, __pu_err); \ + __put_user_asm("sb", (x), __gu_ptr, label); \ break; \ case 2: \ - __put_user_asm("sh", (x), __gu_ptr, __pu_err); \ + __put_user_asm("sh", (x), __gu_ptr, label); \ break; \ case 4: \ - __put_user_asm("sw", (x), __gu_ptr, __pu_err); \ + __put_user_asm("sw", (x), __gu_ptr, label); \ break; \ case 8: \ - __put_user_8((x), __gu_ptr, __pu_err); \ + __put_user_8((x), __gu_ptr, label); \ break; \ default: \ BUILD_BUG(); \ } \ } while (0) =20 +#define __put_user_error(x, ptr, err) \ +do { \ + __label__ err_label; \ + __put_user_nocheck(x, ptr, err_label); \ + break; \ +err_label: \ + (err) =3D -EFAULT; \ +} while (0) + /** * __put_user: - Write a simple value into user space, with less checking. * @x: Value to copy to user space. @@ -299,7 +304,7 @@ do { \ __chk_user_ptr(__gu_ptr); \ \ __enable_user_access(); \ - __put_user_nocheck(__val, __gu_ptr, __pu_err); \ + __put_user_error(__val, __gu_ptr, __pu_err); \ __disable_user_access(); \ \ __pu_err; \ @@ -373,13 +378,7 @@ do { \ } while (0) =20 #define __put_kernel_nofault(dst, src, type, err_label) \ -do { \ - long __kr_err =3D 0; \ - \ - __put_user_nocheck(*((type *)(src)), (type *)(dst), __kr_err); \ - if (unlikely(__kr_err)) \ - goto err_label; \ -} while (0) + __put_user_nocheck(*((type *)(src)), (type *)(dst), err_label) =20 static __must_check __always_inline bool user_access_begin(const void __us= er *ptr, size_t len) { @@ -398,12 +397,8 @@ static inline void user_access_restore(unsigned long e= nabled) { } * We want the unsafe accessors to always be inlined and use * the error labels - thus the macro games. */ -#define unsafe_put_user(x, ptr, label) do { \ - long __err =3D 0; \ - __put_user_nocheck(x, (ptr), __err); \ - if (__err) \ - goto label; \ -} while (0) +#define unsafe_put_user(x, ptr, label) \ + __put_user_nocheck(x, (ptr), label) =20 #define unsafe_get_user(x, ptr, label) do { \ long __err =3D 0; \ --=20 2.34.1 From nobody Wed Dec 17 12:12:29 2025 Received: from mail-oi1-f179.google.com (mail-oi1-f179.google.com [209.85.167.179]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A13FA1DF263 for ; Tue, 18 Mar 2025 06:15:24 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.167.179 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1742278526; cv=none; b=Pv3imnNpcdthWkbkqTUR5KY+u+gcpVwl+bhKE/dQExKzVgaevhh1vBPQ7g0F+18mpadhevjVIWuhsESnbjAvAMfuUGIXFhQdzFHetHtu+cF9ReYGqCrI5bCpbGSkFVgDOezIZBzvKgrwxqQOxRu+uXW6eRWHMfDqNHTPbx0cxr0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1742278526; c=relaxed/simple; bh=pgpE4cukBlxGicXc0bp7gSGepbKSiUK/JKJ8Zip6/lw=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=dZ+QJ7Oyv0r9OvzbvqHh9ByVYNU5Orv/4nVmlYVioIHP5K2ro8dtgVMSyL+I+IcGzfe+13WmVhosgye1rBtX7LqKGbbkbguuuk15uGCTOcED+tP6wWs/dptWboMqi7CYiYdNJlgGLjhP720cvJNf4q7ZSbsaTp7fE1d05sa+8Q4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=tenstorrent.com; spf=pass smtp.mailfrom=tenstorrent.com; dkim=pass (2048-bit key) header.d=tenstorrent.com header.i=@tenstorrent.com header.b=EIAfLQ4D; arc=none smtp.client-ip=209.85.167.179 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=tenstorrent.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=tenstorrent.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=tenstorrent.com header.i=@tenstorrent.com header.b="EIAfLQ4D" Received: by mail-oi1-f179.google.com with SMTP id 5614622812f47-3f682a2c3c8so3019366b6e.1 for ; Mon, 17 Mar 2025 23:15:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=tenstorrent.com; s=google; t=1742278523; x=1742883323; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=qB88SWzbKOqS560oNmcGvhVMbrzNnGmMk8r7GYi48TU=; b=EIAfLQ4D120I2VEXwtijdORoxc3AYndf72ZtXVpQPM2PS1kWX5fm1reiKnPZFSA6EG 7iDFflLwQquabEt9lfJKvu592FwC8yW89wVkcEB6luwft1me311BRgjXv1SVh3CtaZXI 1PS+uBWN/g8FbSyf5dIdterDb0ACyoVpSz0CdksKUsnhLXoMxHAVaSvAbRJGLfIfVQx5 Acwb6uwHDb51hApllQ3vAOScBkXUooGKgFJJoj0R7RwhN9cTLEhJUqk0dYfoyU8ccp7v IX32KhwaMna18a1eUO2gqtCSmbc7Q3zLVY3U/7PzHqLrci72/FsduovouJ+IF89EsKiH aNjQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1742278523; x=1742883323; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=qB88SWzbKOqS560oNmcGvhVMbrzNnGmMk8r7GYi48TU=; b=W3ld/UpROmoa8t4WyR1cLlbVCeb9TUEmwYCggYLGgYUYtLt4sozS2KRxx+dTHnXPaj HWokz3HELVzJuvN0oSlY7gMs5HCWEzTSl7DeddBbGPAbnC33bk0nqdVq4/1GAYOvY2qW cJnQb31+xdIOzcS3bKkdeGw1Y2SCjHrECOdu/Ba0YFesM0+bLt4i/OFTfEVc1RBhl04y FBm6L/ZI6tqRHz/UFMp2xm4URL2lsmQHuivWa8E5NbN9JQG1mDaFg9BGfJHH2Z0BZLeT tHibDbGx8zuILuxou9sy4I+E1W275nXGQsE030p77EJBopeYendNmFdHMv21xDK3UAbH Cohw== X-Forwarded-Encrypted: i=1; AJvYcCXZHeGsyddhIBx8C+vxh8GHYDSsN3iQ4w2+1oDEgB4UYgZMumjpCuTx7bG8GxjbSG521vohSGUYaVjuyNc=@vger.kernel.org X-Gm-Message-State: AOJu0YytngcbNLo6Tak59jeJAbyRL8kIwirrRPRgE2s1lffoZo3sMGci oLJ0w5klRy31iYp55Im1gfQ9fNEDNjTmKJT67QaMPvVB5XzK6hFRmCMvIh9LPw== X-Gm-Gg: ASbGncsvtoxHQ1F3bY+ghtvt1D5W6DLgcKWwKHBiaGF6KlpFyJ4Iw/CuFp9qNdrL9Eo KpNVS3aiFTE/DMUeDZcm4YasSFT4tVzYkjNh6dSVIjsfxLS0iqix/U+11mUFQHb/jMgy/SAp7OJ ZQ4DBUZxCQbE86D3uhKMafE/olafxxc27YiiHvZDV8TjP3KzYYq6nvxu59+ek1ygAq3x6yQK3Vz RAIMGdGLn0B+NX85Ww2lxzP08tYYbndK9b31FdgAa5Ihz/wk63ROfVnx9qN1x21C9V9vgXT3HEl bifRJ+r5JfCrw/I4xllqX/8cZohiNB84RV4l+EdNZifNlWQuzLm3AKfILfV3fDw= X-Google-Smtp-Source: AGHT+IG5rNTfQnLxyptciRc/2MpA8tF5u7O/A4GY0S1qU3Z6Z6RSpie2GzJANFt/uJHp/UYORqcpgQ== X-Received: by 2002:a05:6808:f05:b0:3fb:3be7:ac9c with SMTP id 5614622812f47-3fdf0081260mr8892498b6e.30.1742278523672; Mon, 17 Mar 2025 23:15:23 -0700 (PDT) Received: from aus-ird.tenstorrent.com ([38.104.49.66]) by smtp.gmail.com with ESMTPSA id 5614622812f47-3fcd403b882sm2051642b6e.8.2025.03.17.23.15.22 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 17 Mar 2025 23:15:22 -0700 (PDT) From: Cyril Bur To: palmer@dabbelt.com, aou@eecs.berkeley.edu, paul.walmsley@sifive.com, charlie@rivosinc.com, jrtc27@jrtc27.com, ben.dooks@codethink.co.uk, alex@ghiti.fr Cc: linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org, jszhang@kernel.org Subject: [PATCH v4 5/5] riscv: uaccess: use 'asm_goto_output' for get_user() Date: Tue, 18 Mar 2025 06:15:14 +0000 Message-Id: <20250318061514.1223111-6-cyrilbur@tenstorrent.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20250318061514.1223111-1-cyrilbur@tenstorrent.com> References: <20250318061514.1223111-1-cyrilbur@tenstorrent.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Jisheng Zhang With 'asm goto' we don't need to test the error etc, the exception just jumps to the error handling directly. Unlike put_user(), get_user() must work around GCC bugs [1] when using output clobbers in an asm goto statement. Link: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=3D113921 # 1 Signed-off-by: Jisheng Zhang [Cyril Bur: Rewritten commit message] Signed-off-by: Cyril Bur --- arch/riscv/include/asm/uaccess.h | 95 +++++++++++++++++++++++--------- 1 file changed, 68 insertions(+), 27 deletions(-) diff --git a/arch/riscv/include/asm/uaccess.h b/arch/riscv/include/asm/uacc= ess.h index ab91dd2fa230..ef693aca5236 100644 --- a/arch/riscv/include/asm/uaccess.h +++ b/arch/riscv/include/asm/uaccess.h @@ -96,27 +96,58 @@ static inline unsigned long __untagged_addr_remote(stru= ct mm_struct *mm, unsigne * call. */ =20 -#define __get_user_asm(insn, x, ptr, err) \ +#ifdef CONFIG_CC_HAS_ASM_GOTO_OUTPUT +#define __get_user_asm(insn, x, ptr, label) \ + asm_goto_output( \ + "1:\n" \ + " " insn " %0, %1\n" \ + _ASM_EXTABLE_UACCESS_ERR(1b, %l2, %0) \ + : "=3D&r" (x) \ + : "m" (*(ptr)) : : label) +#else /* !CONFIG_CC_HAS_ASM_GOTO_OUTPUT */ +#define __get_user_asm(insn, x, ptr, label) \ do { \ - __typeof__(x) __x; \ + long __gua_err =3D 0; \ __asm__ __volatile__ ( \ "1:\n" \ " " insn " %1, %2\n" \ "2:\n" \ _ASM_EXTABLE_UACCESS_ERR_ZERO(1b, 2b, %0, %1) \ - : "+r" (err), "=3D&r" (__x) \ + : "+r" (__gua_err), "=3D&r" (x) \ : "m" (*(ptr))); \ - (x) =3D __x; \ + if (__gua_err) \ + goto label; \ } while (0) +#endif /* CONFIG_CC_HAS_ASM_GOTO_OUTPUT */ =20 #ifdef CONFIG_64BIT -#define __get_user_8(x, ptr, err) \ - __get_user_asm("ld", x, ptr, err) +#define __get_user_8(x, ptr, label) \ + __get_user_asm("ld", x, ptr, label) #else /* !CONFIG_64BIT */ -#define __get_user_8(x, ptr, err) \ + +#ifdef CONFIG_CC_HAS_ASM_GOTO_OUTPUT +#define __get_user_8(x, ptr, label) \ + u32 __user *__ptr =3D (u32 __user *)(ptr); \ + u32 __lo, __hi; \ + asm_goto_output( \ + "1:\n" \ + " lw %0, %2\n" \ + "2:\n" \ + " lw %1, %3\n" \ + _ASM_EXTABLE_UACCESS_ERR(1b, %l4, %0) \ + _ASM_EXTABLE_UACCESS_ERR(2b, %l4, %0) \ + : "=3D&r" (__lo), "=3Dr" (__hi) \ + : "m" (__ptr[__LSW]), "m" (__ptr[__MSW]) \ + : : label) \ + (x) =3D (__typeof__(x))((__typeof__((x) - (x)))( \ + (((u64)__hi << 32) | __lo))); \ + +#else /* !CONFIG_CC_HAS_ASM_GOTO_OUTPUT */ +#define __get_user_8(x, ptr, label) \ do { \ u32 __user *__ptr =3D (u32 __user *)(ptr); \ u32 __lo, __hi; \ + long __gu8_err =3D 0; \ __asm__ __volatile__ ( \ "1:\n" \ " lw %1, %3\n" \ @@ -125,35 +156,51 @@ do { \ "3:\n" \ _ASM_EXTABLE_UACCESS_ERR_ZERO(1b, 3b, %0, %1) \ _ASM_EXTABLE_UACCESS_ERR_ZERO(2b, 3b, %0, %1) \ - : "+r" (err), "=3D&r" (__lo), "=3Dr" (__hi) \ + : "+r" (__gu8_err), "=3D&r" (__lo), "=3Dr" (__hi) \ : "m" (__ptr[__LSW]), "m" (__ptr[__MSW])); \ - if (err) \ + if (__gu8_err) { \ __hi =3D 0; \ - (x) =3D (__typeof__(x))((__typeof__((x)-(x)))( \ + goto label; \ + } \ + (x) =3D (__typeof__(x))((__typeof__((x) - (x)))( \ (((u64)__hi << 32) | __lo))); \ } while (0) +#endif /* CONFIG_CC_HAS_ASM_GOTO_OUTPUT */ + #endif /* CONFIG_64BIT */ =20 -#define __get_user_nocheck(x, __gu_ptr, __gu_err) \ +#define __get_user_nocheck(x, __gu_ptr, label) \ do { \ switch (sizeof(*__gu_ptr)) { \ case 1: \ - __get_user_asm("lb", (x), __gu_ptr, __gu_err); \ + __get_user_asm("lb", (x), __gu_ptr, label); \ break; \ case 2: \ - __get_user_asm("lh", (x), __gu_ptr, __gu_err); \ + __get_user_asm("lh", (x), __gu_ptr, label); \ break; \ case 4: \ - __get_user_asm("lw", (x), __gu_ptr, __gu_err); \ + __get_user_asm("lw", (x), __gu_ptr, label); \ break; \ case 8: \ - __get_user_8((x), __gu_ptr, __gu_err); \ + __get_user_8((x), __gu_ptr, label); \ break; \ default: \ BUILD_BUG(); \ } \ } while (0) =20 +#define __get_user_error(x, ptr, err) \ +do { \ + __label__ __gu_failed; \ + \ + __get_user_nocheck(x, ptr, __gu_failed); \ + err =3D 0; \ + break; \ +__gu_failed: \ + x =3D 0; \ + err =3D -EFAULT; \ +} while (0) + /** * __get_user: - Get a simple variable from user space, with less checking. * @x: Variable to store result. @@ -178,13 +225,16 @@ do { \ ({ \ const __typeof__(*(ptr)) __user *__gu_ptr =3D untagged_addr(ptr); \ long __gu_err =3D 0; \ + __typeof__(x) __gu_val; \ \ __chk_user_ptr(__gu_ptr); \ \ __enable_user_access(); \ - __get_user_nocheck(x, __gu_ptr, __gu_err); \ + __get_user_error(__gu_val, __gu_ptr, __gu_err); \ __disable_user_access(); \ \ + (x) =3D __gu_val; \ + \ __gu_err; \ }) =20 @@ -369,13 +419,7 @@ unsigned long __must_check clear_user(void __user *to,= unsigned long n) } =20 #define __get_kernel_nofault(dst, src, type, err_label) \ -do { \ - long __kr_err =3D 0; \ - \ - __get_user_nocheck(*((type *)(dst)), (type *)(src), __kr_err); \ - if (unlikely(__kr_err)) \ - goto err_label; \ -} while (0) + __get_user_nocheck(*((type *)(dst)), (type *)(src), err_label) =20 #define __put_kernel_nofault(dst, src, type, err_label) \ __put_user_nocheck(*((type *)(src)), (type *)(dst), err_label) @@ -401,12 +445,9 @@ static inline void user_access_restore(unsigned long e= nabled) { } __put_user_nocheck(x, (ptr), label) =20 #define unsafe_get_user(x, ptr, label) do { \ - long __err =3D 0; \ __inttype(*(ptr)) __gu_val; \ - __get_user_nocheck(__gu_val, (ptr), __err); \ + __get_user_nocheck(__gu_val, (ptr), label); \ (x) =3D (__force __typeof__(*(ptr)))__gu_val; \ - if (__err) \ - goto label; \ } while (0) =20 #define unsafe_copy_loop(dst, src, len, type, op, label) \ --=20 2.34.1